Could the build farm alert the maintainer that the first CL breaks builder?

At the moment that process is manual. However, you can expect an issue to be filed including a mention of the appropriate subteam(s) of @‌golang/port-maintainers.

Most port maintainers are doing the porting work as a hobby in their spare time. Understand the status is already a lot of investments.

Indeed. That's partly why we're thinking in terms of 2 or more maintainers per port, and part of why a port broken in release 1.N would not be removed until at least release 1.(N+1) — that gives the maintainers a full release cycle to catch up in the event of a serious problem, and with more than one maintainer it is more likely that at least one maintainer will have the time to look into a regression in any given release cycle.

On the other hand, if we were to require Go developers to always make their code work on secondary ports, then those secondary ports would no longer be “secondary”: the main difference between a “first-class” port and a “secondary” one is whether “all contributors” are responsible for fixing them. Removing that distinction would mean that we would not be able to accept nearly so many secondary ports in the Go repo to begin with, because the Go team would have to budget for the full cost (including the cost to other contributors) of maintaining those ports vs. the benefit of having them in the tree.

that would substantially raise the effort required to contribute even non-port-specific changes

I agree a lot with the vision of not requiring always work for every secondary port. However, in practice, we do not know well enough whether a change is "port-specific". For example, CL 256917 is specifically for darwin/arm64. Without further notice, it implicitly breaks ios/arm64. If developers are informed that only responsible for their ports. This type of break may be amplified over time. We might end up with different implementations on different ports. Therefore I think "best-effort" port responsibility may be better than "encouraged" port policy in this context.

I'd like to share an experience report, as a contributor who ran into related tests failing on a secondary port.

if a corresponding builder is online and functioning

My own experience has been that "builder is online and functioning" isn't enough to make sure my code works well on a port. I had a hard time with the linux/riscv64 port last fall, with a typical delay of 3 hours between when I asked for trybots/slowbots to when a builder for that port would get to the part of all.bash that tested my changes. The relevant tests didn't fail 100% of the time, which made it even harder to see if I was fixing the problem or not.

Gomote might have helped, but new access requests for that are not currently being granted (since May 2021). The only way I saw to get a tight feedback loop (which I saw as a necessary part of building trust in my code on the first-class ports) was by buying a RISCV64 single-board computer of my own. I didn't find RISCV64 machines available for rent from a cloud provider, and it took about two weeks for the SBC to arrive in the part of the world where I live. But after that, I was able to get my code working well on the port without too much trouble.

It looks like the Gomote security work is moving forward, so maybe that's a good path for Go contributors to get fast feedback on whether a broken secondary port is giving them early warning about a subtle bug that can affect multiple platforms. But without it, I think that "builder is online and functioning, is only available to run all.bash, and gives results after three hours" is too low a bar.

As the maintainer of the netbsd-arm and netbsd-arm64 builder, the way it currently works is fairly stressful. I maintain the machines, which are kind of fickle as they are ARM SBCs. But then I also get a near-daily stream of issue CCs on test failures.

As a contributor in my spare time (I am a Googler but not on the Go team), this creates a lot of stress for me, since there is always an implied threat in these messages. The threat is "if YOU don't fix this, or get someone else to fix this, we'll remove the port you are actively using from the tree". This is not a hypothetical concern! @bcmills literally filed issue #50451 to propose dropping the netbsd-arm64 port -- thankfully this was declined eventually.

Running builders and maintaining support is thankless enough as it is. I fear that by making requirements more stringent, you are just going to lose the contributors that keep platforms alive.

@bsiegert, I'm sorry that the way it currently works ends up causing stress.

FWIW, it's been quite stressful for me too, which is part of why I've been spending a large chunk of my time over the past year fixing platform-independent bugs that happen to surface on these builders — for example, just yesterday I noticed #53160 on one of the -bsiegert builders, triaged it down to a platform-independent issue in the test, and mailed and merged a fix.

But it's a lot more work (for me and others) to triage and fix the platform-independent bugs when there are also platform-specific bugs adding noise to the process, and it really doesn't feel great to do the legwork to collect details on a platform-specific bug (as in #50138 (comment)) only to have that information go apparently-unused because nobody has enough time to actively maintain the port.

As I see it, part of the point of the proposed changes is to alleviate the stress on both sides.

• The two-maintainer requirement should help to distribute load over multiple maintainers. (If you-specifically don't have the bandwidth to fix an issue, there ought to be someone else who can help!)
• The process to mark a port as broken (and guidance about what that means) should reduce the need to ping maintainers about failure modes that nobody is actively working on, and provide a clearer timeline for when a fix is expected — namely, a full 6-month release cycle after the port is marked broken — before anyone even considers removing the port.
• The explicit policy of notifying maintainers when a port breaks should help to prevent platform-independent bugs from being glossed over just because they happen to reproduce more readily on secondary builders. (If someone from the Go project is triaging the failures enough to determine who to notify, it's more likely that they'll notify an appropriate package maintainer instead of just ignoring the failure altogether.)

I would suggest that it depends on how invasive the fixes are, and at what point in the freeze the work begins.

If the fix only affects source code specific to the broken port, then no freeze exception should be needed in the first place. (The port is already broken, so there is little risk to the release.)

If the fix involves modest cross-platform changes but the work is started fairly early in the freeze, then a freeze exception should pretty much always be granted.

If the fix involves more invasive cross-platform changes, or the work is started late in the freeze (such that it wouldn't be finished by the first RC), then a freeze exception should generally be declined (but may be decided on a case-by-case basis).

This means it is possible for someone to make a change that breaks a secondary port right before the freeze, which could take significant time to fix.

If we expect the fix to take significant time, I would expect the developer making the breaking change to discuss the plan ahead of time to avoid that kind of situation.

Changes that are that disruptive (and that difficult to fix) should be landing at the beginning of the cycle (see the early-in-cycle label), not the end, and I think you could reasonably make a case for rolling back a change for one release cycle if it breaks an otherwise-healthy port without advance discussion in a way that requires a lot of work to fix.

(That said, most of the changes that break ports are also not that difficult to fix for someone with knowledge of the platform and a day or two to debug.)

The flip-side of that is: I don't think we should roll back a late change merely due to breaking a port, especially if nobody has done the work to triage how much work it will be to fix. For an actively-maintained port, we should be able to count on at least a day or two of debugging from the port maintainers in between the start of the freeze and the RC if the need arises.

The ppc64le/ppc64 builders are maintained by the Go team and we don't have access to those machines. There have been several cases where there have been failures on the builders that we can't reproduce on systems here which has made it hard to resolve those issues. Are there any plans to change the maintainers of the builders for some cases or will they continue to be maintained by the Go team?

Hi @laboger! I think if IBM wanted to run the PowerPC64 machines hosted at OSU instead of the Go team, we would not object to that, but I'm also not sure we need to go that far. We can get you and anyone else at IBM at least ssh access to the machines for debugging those kinds of machine-specific problems. I'll mail you off list to find out who we should include. Thanks!

(I do think it's worth sending an email to maintainers and golang-dev before we mark a port as broken, though, in case the GitHub issue notifications have been dropped or overlooked.)

I want to get back to this part of the proposal:

The Go team currently publishes binaries for the following secondary ports, which we would stop doing if we adopt this part of the proposal:
-freebsd/386
-freebsd/amd64
-linux/ppc64le
-linux/s390x
-windows/arm64
We encourage port maintainers to publish them instead; we can link to pages from https://go.dev/dl as appropriate.
Note: when and if reliable testing hardware is available, the Go team is likely to promote windows/arm64 to be a first class port.

As I posted above, a common use in scripts and Dockerfiles is to pull the binary directly from the golang.org/dl location. As I read this, golang.org/dl would now contain a link to the location where the secondary ports would host their binaries so existing curl commands like above would no longer work. Other comments in this issue make it sound like we could provide you with the binary to post as before. Can this be clarified?

Also in the wording it says "if we adopt this part of the proposal" which sounds like this is not for sure.

In any case, we would hope to have some lead time if this is something we have to do. Also does this mean you will remove all the old binaries as well?

That is a good question. The general goal here is to draw a line between binary releases that the Go team will build and ones that the team will not build. Right now that line seems a bit random, and honestly I'm not sure how we decided it. It seems simpler to just draw the line around the first class ports. That has the advantage of clarifying that the Go release team isn't required to maintain the builders for the other ports with binary releases.

That said, there is a separate issue which is where other binary releases will live. I did not actually intend for golang.org/dl/RELEASE to forward to a binary release hosted elsewhere. It's important that the binary releases be trusted, exactly because people do treat them as a relatively trusted resource. And that's not necessarily consistent with forwarding to some other host. What I was thinking was more like a link on dl.golang.org saying "look here for releases for GOOS/GOARCH".

That said, a different approach might be to continue to host binary releases for some secondary ports, but to expect secondary port maintainers to build and update those releases. I don't actually know myself where those builds are stored today, so I don't know whether that is reasonable.

If that is feasible, and we did that, would that address your concern? Thanks.

We should be able to take over building the release binaries, and it would be great if they could be found at the same location as now to avoid breaking users who have scripts or Dockerfiles that reference the existing location.

Thanks for the clarification.

I got some pushback on that internally, because it would mean that golang.org/dl would be hosting hard-to-verify binaries and as such go against the new reviewing rules. But it may be possible for the release team to simply cross-compile release builds for arbitrary targets, with any cgo elements filled in on the user's machine with the user's C compiler.

Is there a way to have a builder that doesn't report status back to the go build dashboard?

Generally we don't triage builders marked with known-issues, so IMO it would be fine to add a builder at HEAD and just leave it marked with an issue ~forever.

(I would only consider the port overall to be broken if all of its builders are broken or unreliable — having one builder intentionally unstable seems fine as long as there is another builder working reliably.)

If not, I can figure out a way to run the tests out of band, perhaps with a cron entry.

FWIW, running src/all.rc in a loop at head with GO_BUILDER_NAME set in the environment ought to at least give a reasonable approximation, although some of the batching and timeout behaviors may differ slightly from the real buildlet.

@oridb I think it would be useful if you could run a plan9/amd64 builder running the 9front pc64 kernel on your side. It would provide more coverage, since the current plan9/amd64 builder is running Bell Labs' 9k kernel.

@bcmills I think it would be a good idea to try to run the plan9/386 builder on GCE again, if you are interested. I can provide an updated Plan 9 image.

Running Plan 9 builders on GCE seems like a good idea to me — that would at least eliminate the builder hardware as a variable.

I'm not very involved with the builder deployment at this point; maybe start a discussion on #29801 with @golang/release to see what's viable?

windows-amd64-unified is a builder for an active-but-incomplete GOEXPERIMENT, and thus not a broken port

Is there a reason this builder isn't listed in https://farmer.golang.org/builders?