As mentioned in #55090 (reply in thread) it also seems like that was the intent. rust editions (mentioned in that comment) also seem like really clear precedent for this. I wonder if anyone can weigh in on how well that's working in the rust ecosystesm.

A significant detail about Rust editions is that they cannot typically change the standard library, and instead focus on the language. I gather this is because the library code is linked once into the program and shared between all callers.

This is in turn because editions are a per-crate decision rather than a main-crate decision. This avoids splitting the ecosystem by allowing multiple language versions to exist in the program at once.

The current treatment of the go directive, e.g. activating type parameters in 1.18, seems to roughly match the intent of Rust additions. This further idea of making the main program's selected version also affect library behavior doesn't have precedent in Rust editions as far as I know.

I don't mean to say that it isn't a good idea just because Rust doesn't do it! I think it is an interesting idea to investigate, and given that we're talking about a small number of very specific changes made for exceptional reasons only (as opposed to becoming a broad way to evolve the standard library in incompatible ways) it seems nice and pragmatic to me.

Indeed. That's exactly right: the //go:debug lines would set the default values for specific settings, which could then be overridden individually by $GODEBUG.

Indeed, and I think security deprecations are generally different from security fixes.

That does also raise a question about security point releases. Suppose the LookPath behavior was bad enough that it needed to get fixed in a dot release, but it still needs backwards compatibility support. Now the implicit go.mod line behavior isn't really enough to provide stability, because go.mod specifies only the major Go version.

Does this mean that some classes of security fixes (those with backwards compatibility issues) are only possible to make in major Go releases?

Does this mean that some classes of security fixes (those with backwards compatibility issues) are only possible to make in major Go releases?

Yes, and I believe that is true already. One of the hard rules of point releases is it really must not break anyone, because we never want someone to be unable to add an urgent security fix due to some breakage in that same point release or an earlier one in the sequence. That applies to the security fixes themselves too.

LookPath is a good example. There was a reported bug affecting go toolchain programs, and we fixed the bug by making the LookPath change in a forked copy of os/exec specifically for those programs. We left the toolchain-wide fix for a major Go release precisely because of the compatibility issue.

The same is true of net.ParseIP. We decided it was an important security-hardening fix but on balance inappropriate for a point release because of the potential for breakage.

It's hard for me to think of a security problem that would be so critical that it must be fixed in a point release and simultaneously so broad that the fix fundamentally must break unaffected user programs as collateral damage. To date I believe we've always found a way to avoid such a fix, and I think the onus is on those of us preparing security releases to continue to do that.

Thanks. I think this is an important enough point that, if this gets written up as a proposal, it should be called out explicitly.

Thanks. Yes, indeed. I'm not sure I had seen this Perl thing before, but it's also similar to what Rust is doing with editions. Enabling this kind of backward compatibility story was definitely one of the reasons we put the go version into the go.mod in the first place. (We didn't have the GODEBUG things in mind at the time, just language versions, which we already use it for.)

It would change, but as you note that change is #55092. I think we have to consider them at the same time, because they are two sides of the same coin, but it also made sense to split the discussions, because the two different directions have different answers. I think we could do this idea without #55092, but it is a stronger story if we include #55092 too.

If you want to borrow a term from the FreeBSD ports system, 'knob' could work, though that might not be ideal in some varieties of English. Other possibilities would be 'switch' and 'tweak'.

GOBACKWARDS? It's a bit tongue-in-cheek, but it reasonable implies you're enabling a previous/older behaviour.

Alternatively, GOBEHAVIOR or GOCAPABILITY might have some potential, and could be used for both gating a new experiment in a release as well as an old code path.

Alternatively, GOBEHAVIOR or GOCAPABILITY might have some potential, and could be used for both gating a new experiment in a release as well as an old code path.

This is exactly what we don't want to use it for (and why I excluded GOFEATURE), because it has the potential for splitting the ecosystem like Python 2 vs Python 3. It is very important that this only be used for things that very very very few people will need. Otherwise it's too large a break and shouldn't be done at all.

I agree the name is hard, as I was planning to put a suggestion in my original message but couldn't think of a good one at the time. :)

The best I've got right now is GOCOMPAT, but maybe GODEBUG is nice just in that it looks like it shouldn't be used!

Kubernetes adopts new Go versions relatively quickly we even tend to jump on RCs these days, this isn't about Kubernetes avoiding fixing something ...

But when Go makes breaking changes to say stdlib crypto it is problematic for the our minor version branches as end users are expecting no breaking changes within the lifecycle of those minor versions, and as evidenced in the examples in the original post here sometimes upgrading Go fundamentally makes breaking changes no matter what Kubernetes does.

This is about providing a way for projects like us to upgrade to the latest go without shipping breaking behavioral changes to our end users.

This is about providing a way for projects like us to upgrade to the latest go without shipping breaking behavioral changes to our end users.

Right. I'm saying that maybe we shouldn't do that, for the reasons I gave. Kube could wait until its next major version release to upgrade Go and absorb the breaking changes, if that's what Kube's versioning policy requires. This problem affects a tiny percentage of Go users, according to @rsc, so I'm questioning the value, cost, and complexity of making this change to module semantics and the build system just to accommodate them when they already have a viable workaround, and things are working as intended.

if "things are working as intended", then please update https://go.dev/doc/go1compat to reflect the reality.

Having worked on some of these failures in the kubernetes code base when we "jump on RCs" i think that the go1compat is no longer valid.

No, this is not just about kubernetes alone. other folks are affected too, see example in #51991

Either we work towards making go1compat true or accept that it is false and drop it. let's not pretend please.

Kube could wait until its next major version release to upgrade Go and absorb the breaking changes, if that's what Kube's versioning policy requires.

The versioning policy is just semver. Kubernetes minor versions just last longer than Go's, by user demand.

Users need longer support cycles for their clusters with no breaking changes in on order to run critical infrastructure.

Thus far that requires sticking to eventually unsupported go versions that are not receiving any patches, or sometimes it can be worked around (e.g. by forking the standard library and patching usage to use the forked package).

This problem affects a tiny percentage of Go users, according to @rsc,

I don't see a statement to that effect. I do see this (emphasis mine):

It turns out that Go's been averaging about one Kubernetes-breaking change per year for the past few years. I don't think Kubernetes is an outlier here: I expect most large projects have similar experiences. Once per year is not high, but it's not zero either, and our goal with Go 1 compatibility is zero.

I think rsc is pretty equipped to determine if the complexity is justified for the Go project.

This problem affects a tiny percentage of Go users, according to @rsc

I don't see a statement to that effect.

Here you go (from this thread):

Continuing the SHA1 example, changing the default to not accept SHA1 certificates makes Go more secure and breaks hardly anyone. The benefits for the 99.99% of users far outweigh the inconvenience for the 0.01%. Should we use semver and start calling it Go 2.0.0 because we no longer accept SHA1 certificates? That makes everyone expect a breakage and then for almost everyone, nothing breaks. That makes no sense.

I think we could probably commit to saying that any flag stays for at least two years. As I noted, some will stay forever. And also any flag should have usage metrics exported (counting executions that would have run differently with the default behavior, when that behavior is actively suppressed), which should make it easy for deployments of Kubernetes or other systems to watch for when they can safely stop using the flag.

I really like the idea of runtime metrics for hits to non-default GODEBUG code paths. This helps identify uses that should be fixed, and allows gathering data if there are uses that can't be fixed that weren't anticipated and should inform speed of removal of the compatibility switch.

I think this comment belongs on #55092..?

I prefer the main package override approach, since that gives flexibility if specific binaries within the module need to override defaults differently in more fine-grained ways.

Should we then move the go version there as well?
It seems silly to say "for the definitive answer, see A and then B" (;-))

More importantly, should we have multiple conflicting defaults?

It seems silly to say "for the definitive answer, see A and then B"

Unfortunately that is already the case anyway.

If my go.mod says go 1.18 but I have a file with a constraint //go:build !go1.18, that file will today only be used when building with Go 1.17 and lower, so it would be at least incoherent to use a Go 1.18 feature in that file.

Similarly, if my go.mod says go 1.18 but I have a file with a constraint //go:build go1.19, that file will be used only when building with Go 1.19 and above, even though the module overall uses Go 1.18 language semantics.

(#30791 is closely related.)

Ow! Sorry to hear that, I know from experience how much "fun" it can be to maintain (;-))

Can you expand on the mindset/goal behind wanting to update go while at the same not wanting the new go version behaviors?

Go-based projects need a way to ship maintenance patch updates rebuilt with go versions that fix go runtime vulnerabilities, without picking up go runtime deprecations or significant runtime default behavior changes (e.g. ParseIP changes, default disablement of SHA-1 support, default client change from http/1 to http/2).

Currently, the minor version updates required to get go vulnerability fixes also regularly bring the latter.

Go-based projects need a way to ship maintenance patch updates rebuilt with go versions that fix go runtime vulnerabilities, without picking up go runtime deprecations or significant runtime default behavior changes

Indeed. A Go toolset has effectively one year of maintenance attached to it with the N-1 model. A lot of projects, particularly at the corporate level, end up with servicing lifetimes much longer than that. At Microsoft a 3 year model is quite common. Those projects want to be shipping minimal changes throughout the lifetime of the product but taking a new Go toolset can force them to make unwanted significant changes

At Sun we found that bug fixes and deprecation interacted badly. We originally used semantic versioning at the whole-file level, and ran into exactly the kind of conflicts you describe. Fortunately you're in effect dealing with the bugs at the interface level, so you don't just deadlock.

(If you're interested, this is also related to the version problem that Mr Cox described in https://research.swtch.com/version-sat. My take on it is https://leaflessca.wordpress.com/2017/02/12/dll-hell-and-avoiding-an-np-complete-problem/, and Sun's was David J. Brown's https://www.usenix.org/legacy/publications/library/proceedings/als00/2000papers/papers/full_papers/browndavid/browndavid_html/
This stuff used to be my life (;-))

Go-based projects need a way to ship maintenance patch updates rebuilt with go versions that fix go runtime vulnerabilities, without picking up go runtime deprecations or significant runtime default behavior changes (e.g. ParseIP changes, default disablement of SHA-1 support, default client change from http/1 to http/2).

Makes sense 👍

Excluding http/2 change, the changes listed as an example were made for security reasons which make this discussion more difficult. The distinction is more/less clarified here #55090 (reply in thread)

Given the effort required to maintain behaviors that were deemed insecure and the security risk of maintaining those behaviors I think that kind of backwards compatibility should be short lived, as in, 3 point releases (deprecated/hidden in three releases, removed on the forth).

If we're talking about a significant number of the GODDEBUG (GOBUG?) settings being for security problems, then we probably need to spend some time considering using the rewriting tools to either provide corrections directly, or to provide blocks of candidate corrections.
At Sun we did the latter, to help as much as we could to avoid someone "depending on the bug" (:-()

Yes: a go mod features command is a clean way to ensure you get predictable behavior. I'd probably put the whole table in as a go mod list option, like git stash list, but that's a niggle.

go mod edit sounds like the beginning of a go idiom. Let's have less magic and more idioms: elegant ones!

One example of a breaking change: the change in 1.13 strconv behavior to allow underscores as separators introduced regressions in some of our parsing code where underscore was used to separate fields in a URL format.

I don't understand this concern. When we test libraries the library is the main module, so wouldn't it still have control over the versions of the toolchain used for its tests?

Right, but the library then gets used in other projects, which may choose a different version as the main module/package.

True, but how is that different than the status quo?

Oh, it's not. My point is that, since the proposal is formalizing "Go version semantics", I wonder what we could do to surface potential incompatibilities at build time before they could cause run-time breakages down the line.

As a strawman idea: a bit of static analysis that could show a warning if a library uses exec.LookPath with go 1.17 but the main module then overrides the version with go 1.19, thus potentially breaking the library.

Thanks for the example. I agree that change may have been a mistake.