You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[ upstream commit 398d55c ]
As reported in [1], Go's HTTP2 client < 1.16 had some serious bugs which
could result in lost connections to kube-apiserver. Worse than this was
that the client couldn't recover.
In the case of CoreDNS the loose of connectivity to kube-apiserver was
even not logged. I have validated this by adding the following rule on
the node which was running the CoreDNS pod (6443 port as the socket-lb
was doing the service xlation):
iptables -I FORWARD 1 -m tcp --proto tcp --src $CORE_DNS_POD_IP \
--dport=6443 -j DROP
After upgrading CoreDNS to the one which was compiled with Go >= 1.16,
the pod was not only logging the errors, but also was able to recover
from them in a fast way. An example of such an error:
W1126 12:45:08.403311 1 reflector.go:436]
pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: watch
of *v1.Endpoints ended with: an error on the server ("unable to
decode an event from the watch stream: http2: client connection
lost") has prevented the request from succeeding
To determine the min vsn bump, I was using the following:
for i in 1.7.0 1.7.1 1.8.0 1.8.1 1.8.2 1.8.3 1.8.4; do
docker run --rm -ti "k8s.gcr.io/coredns/coredns:v$i" \
--version
done
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
CoreDNS-1.7.1
linux/amd64, go1.15.2, aa82ca6
CoreDNS-1.8.0
linux/amd64, go1.15.3, 054c9ae
k8s.gcr.io/coredns/coredns:v1.8.1 not found: manifest unknown:
k8s.gcr.io/coredns/coredns:v1.8.2 not found: manifest unknown:
CoreDNS-1.8.3
linux/amd64, go1.16, 4293992
CoreDNS-1.8.4
linux/amd64, go1.16.4, 053c4d5
Hopefully, the bumped version will fix the CI flakes in which a service
domain name is not available after 7min. In other words, CoreDNS is not
able to resolve the name which means that it hasn't received update from
the kube-apiserver for the service.
[1]: kubernetes/kubernetes#87615 (comment)
Signed-off-by: Martynas Pumputis <m@lambda.lt>
Signed-off-by: nathanjsweet <nathanjsweet@pm.me>
0 commit comments