#131947 and #131869 hit this

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/131947/pull-kubernetes-e2e-kind-ipv6/1925971218247192576
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/131869/pull-kubernetes-e2e-kind-ipv6/1925942985422278656

#131883 was pretty recent on the kubernetes/kubernetes changes side of things. It doesn't appear to have flaked on this though. https://prow.k8s.io/pr-history/?org=kubernetes&repo=kubernetes&pr=131883

Other commits don't stand out.

Not sure about the infra yet, but that sounds more likely at the moment.

https://github.com/kubernetes/test-infra/commits/master/ nothing obvious here?

Or in https://github.com/kubernetes/k8s.io/commits/main/

Maybe the cluster itself. These ran in gke-prow-build-pool5 .... nodes in k8s-infra-prow-build which are not the new nodepool experiment @upodroid and @ameukam have been working on.

I don't think we've had other changes to that cluster lately, it could've auto-upgraded maybe.

If we ignore the compat-verison jobs which have had other issues we get a clearer picture:
https:/go.k8s.io/triagel?pr=1&job=kind&test=SynchronizedBeforeSuite&xjob=compatibility

There's a lone failure in the alpha-beta-features job on tuesday the 20th that isn't clearly related, the rest start since 4:00 UTC-7 (US Pacific) on the 23rd.

Upgrade logs might align:

{
  "insertId": "1m0nhfte82p86",
  "jsonPayload": {
    "operation": "operation-1747999572322-1bde5da1-4f3f-4340-9c30-ad42e3f6cdf2",
    "@type": "type.googleapis.com/google.container.v1beta1.UpgradeEvent",
    "resource": "projects/k8s-infra-prow-build/locations/us-central1/clusters/prow-build/nodePools/pool5-20210928124956061000000001",
    "currentVersion": "1.32.2-gke.1297002",
    "operationStartTime": "2025-05-23T11:26:12.322248554Z",
    "resourceType": "NODE_POOL",
    "targetVersion": "1.32.3-gke.1785003"
  },
  "resource": {
    "type": "gke_nodepool",
    "labels": {
      "location": "us-central1",
      "nodepool_name": "pool5-20210928124956061000000001",
      "cluster_name": "prow-build",
      "project_id": "k8s-infra-prow-build"
    }
  },
  "timestamp": "2025-05-23T11:26:25.010880914Z",
  "severity": "NOTICE",
  "logName": "projects/k8s-infra-prow-build/logs/container.googleapis.com%2Fnotifications",
  "receiveTimestamp": "2025-05-23T11:26:25.026034909Z"
}

I think this is a kernel issue with the node pool.

SIG K8s Infra is looking to migrate to COS + cgroup v2 + C4 VMs (this is running on Ubuntu + v1 + N1), but we were still testing this.

Looking into CI node pool down/up-grade.
/assign

/triage accepted
/sig k8s-infra

gcloud container node-pools rollback pool5-20210928124956061000000001 --cluster=prow-build --project=k8s-infra-prow-build --region=us-central1 is currently running.

Specifically there appear to be netfilter UDP bug(s) that affected Ubuntu and COS (and others, it's an upstream kernel issue), COS has a patched release already available but I'm not sure Ubuntu does. AFAICT the upstream issue in Ubuntu is still open tracking one of the kernel reports.

EDIT: There's a workaround available for the known impact to GKE clusters with intranode visibility, the kind angle is new ...

https://bugzilla.netfilter.org/show_bug.cgi?id=1795
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2109889
https://bugzilla.netfilter.org/show_bug.cgi?id=1797
EDIT: this is possibly different, there was a known issue impacting this version that pointed to these, I haven't had a chance to root-cause, I'm focused on getting CI green inbetween other things right now ...

this is impacting ~all kind e2e jobs

If you see it fail at SynchronizedBeforeSuite it's probably this, the logs will have an error like:

[ERROR] plugin/errors: 2 7730850699321325609.2949657682776755114. HINFO: dial udp 172.18.0.1:53: connect: network is unreachable

NOTE: we're heading into a 3 day weekend here in the US.

I think this might be IPV6 only, digging through more of the failures.

kernel version upgraded from 6.8.0-1019 to 6.8.0-1022

Tentatively the COS + C4 + cgroup v2 nodepool (pool6....) is good. https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/126563/pull-kubernetes-e2e-kind-ipv6/1926052312141271040

To migrate a job:

1. Set the taint and tolerations as in canary job followups test-infra#34841 (this will run it on the nodepool)
2. Drop the preset-kind-volume-mounts label as in canary job followups test-infra#34841 (we really don't want to be bind-mounting cgroups anymore, never should have)

The operation to rollback the main (pool5...) nodepool upgrade is still pending some capacity issues are slowing it down so the pool is only partially rolled back.

This can be checked like:
gcloud container operations describe operation-1748028469601-078a0d5e-5797-4c6c-a6cb-6e3ac98b04d4 --region=us-central1 --project=k8s-infra-prow-build

The job updated above (pull-kubernetes-e2e-kind-ipv6) seems to be working reliably.

https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/pr-logs/directory/pull-kubernetes-e2e-kind-ipv6