Page MenuHomePhabricator
Create Task
Maniphest T359641

[infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
Open, In Progress, MediumPublic
Actions

Description

K8s

https://v1-27.docs.kubernetes.io/blog/2023/04/11/kubernetes-v1-27-release/

Working etherpad: https://etherpad.wikimedia.org/p/k8s-1.26-to-1.27-upgrade
Persistent wiki page: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes/1.26_to_1.27_notes

1.26 - > 1.27 retrospective post-mortem: https://docs.google.com/document/d/1_Mudr7sOxT-tTw1Dj76FtI7EXtJMpYsih8pxliuQbU8/edit

Workgroup page: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Ongoing_Efforts/Toolforge_Upgrade_Workgroup/Upgrades_Overview

Components

Pre-k8s upgrade

should be upgraded (potentially not blocking, tests pass without them upgrading)
can be upgraded

Post-k8s upgrade

need upgrading
can be upgraded
  • calico from 3.26 to 3.28 (manually patched into our calico repo https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements)
  • builds-api
    • github.com/tektoncd/pipeline from 0.40.2, to 0.59.3 depending on the tekton upgrade
    • k8s.io deps to 0.27.X (if possible)
  • volume-admission, k8s.io deps to 0.27.X
  • registry-admission, k8s.io deps to 0.27.X
  • ingress-admission, k8s.io deps to 0.27.X

Details

Other Assignee
dcaro
SubjectRepoBranchLines +/-
[wmcs-cookbook] update toolsbeta-test-k8s-control vmscloud/wmcs-cookbooksmain+0 -0
aptrepo: add k8s 1.27 reposoperations/puppetproduction+11 -11
kyverno.copy_images_to_registry: add missing kyverno-cli imagecloud/wmcs-cookbooksmain+1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT364293 [infra,k8s] Move to kubernetes PAVs and drop kyverno
 OpenSlst2020T372697 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.31
 OpenaborreroT362869 [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30)
 OpenfnegriT362868 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29
 OpendcaroT362867 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28
 In ProgressRaymond_NdibeT359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27
 ResolvedSlst2020T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26
 ResolvedaborreroT316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25
 ResolvedaborreroT307651 Upgrade Toolforge Kubernetes to version 1.24
 ResolvedtaaviT298005 Upgrade Toolforge Kubernetes to version 1.23
 ResolvedtaaviT286856 Upgrade Toolforge Kubernetes to latest 1.22
 ResolvedtaaviT280436 Migrate Toolforge admission webhooks to use the admissionregistration.k8s.io/v1 API
 ResolvedtaaviT286857 Update maintain-kubeusers to certificates/v1 api
 Resolved• BstormT289390 Certificate generation is broken in toolsbeta
 ResolvedtaaviT292238 Figure out certificate generation for admission webhooks before we lose the certificates/v1beta1
 ResolvedtaaviT329453 Deploy cert-manager to Toolforge
 ResolvedtaaviT292706 tools-webservice should use ingress v1
 ResolvedtaaviT292771 upgrade to ingress-nginx 1.0
 ResolvedrookT291589 Upgrade paws jupyterhub
 ResolvedtaaviT295190 Upgrade all third-party Toolforge Kubernetes components to versions supporting Kubernetes 1.22
 ResolvedtaaviT306296 Ensure k8s-status tool will work on Kubernetes 1.22
 ResolvedrookT308172 Upgrade PAWS to Kubernetes 1.21
 ResolvedtaaviT282942 Upgrade Toolforge Kubernetes to latest 1.21
 ResolvedrookT280402 Upgrade Toolforge Kubernetes to latest 1.20
 ResolvedtaaviT280340 Upgrade Toolforge Kubernetes to latest 1.19
 ResolvedtaaviT280299 Upgrade Toolforge Kubernetes to latest 1.18
 ResolvedtaaviT280300 Validate that maintain-kubeusers works with k8s 1.18
 ResolvedtaaviT280301 Refresh certs that are not controlled by kubeadm
 ResolvedtaaviT280302 Upgrade PAWS Kubernetes to the latest 1.18 release
 ResolvedtaaviT264221 Upgrade the nginx ingress controller in Toolforge (and likely PAWS)
 Resolved• BstormT282087 Support cinder or expanded ephemeral disk worker nodes on Toolforge Kubernetes
 ResolvedtaaviT284353 Fix prometheus monitoring for Toolforge Ingress
 ResolvedtaaviT280342 Upgrade Calico to 3.18
 ResolvedtaaviT280360 Update ingress-admission-controller to support Ingress from networking.k8s.io/v1
 ResolvedtaaviT279106 Establish replacement for PodPresets in Toolforge Kubernetes
 Resolved• BstormT292672 Issue creating pods after migration away from PodPresets
 DeclinedNoneT290531 Weird issue with the wmcs-k8s-node-upgrade.py script
 ResolvedrookT291976 Upgrade toolsbeta to k8s 1.20
 ResolvedtaaviT292698 Upgrade calico to 3.21
 ResolvedBUG REPORTNoneT308189 Toolforge jobs stopped getting scheduled around the same time as the Toolforge k8s cluster upgrade
 DuplicateNoneT308203 Create an alert for K8s cronjobs
 ResolvedtaaviT308204 toolforge-jobs should set startingDeadlineSeconds by default
 ResolvedtaaviT308205 Re-enable CronJobControllerV2
 OpenNoneT308381 toolforge: Scrape Kubernetes controller-manager and apiserver metrics into Prometheus
 ResolvedrookT308274 test PAWS in k8s 1.21.8
 ResolvedtaaviT329869 Remove tool RBAC rules for APIs removed in Kubernetes 1.22
 ResolvedBUG REPORTtaaviT331572 maintain-kubeusers container in CrashLoopBackoff preventing new tool creation after 'user-maintainer' ClusterRole changes
 ResolvedtaaviT331619 toolforge: rbac: change existing roles to reference PSP in the policy group
 ResolvedtaaviT334380 Toolforge k8s: kube-scheduler permissions errors
 ResolvedtaaviT334382 Toolforge k8s: kube-controller-manager permission errors
 ResolvedtaaviT329677 Upgrade Calico to 3.24
 ResolvedtaaviT284656 Toolforge k8s: Migrate workers to Containerd and Bookworm
 ResolvedtaaviT349795 Upgrade cadvisor
 ResolvedtaaviT350227 toolforge prometheus servers OOMing
 ResolvedtaaviT357901 Request increased server-group-members quota for tools
 ResolvedaborreroT358476 toolforge k8s: some static pods needs manual restart
 StalleddcaroT314665 Toolforge: Replace all bastion with grid-less bookworm based bastion hosts
 StalleddcaroT314664 [infra] Decommission the Grid Engine infrastructure
 ResolvedkomlaT313405 Migrate remaining tools off Gridengine
 DeclinedNoneT316124 Make Grid Engine tooling emit deprecation warnings
 ResolvedtaaviT329619 Toolforge: decide what to do with tools that have 'stretch' in their service manifests
 ResolvedAndrewT353351 Provide tools for disabling the grid for specific tools
 ResolvedAndrewT353642 Relocate disable-tool-archive-dbs.service
 ResolvedtaaviT358333 Remove toolschecker grid engine checks
 ResolvedtaaviT359934 [infra] Archive grid engine related infrastructure tools
 ResolvedtaaviT359935 [infra] Archive grid engine related Gerrit repositories
 StalleddcaroT356377 [toolforge] simplify calling the different toolforge apis from within the containers
 StalledSlst2020T356262 [jobs-cli,builds-cli,toolforge-cli,webservice] Consolidate the Toolforge CLIs
 In ProgressSlst2020T356261 [toolforge-cli,jobs-cli,builds-cli,envvars-cli] Explore OpenAPI SDK tooling for client consolidation
 ResolvedSlst2020T354745 [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client
 ResolvedSlst2020T358100 [toolforge API] expose all backend APIs OpenAPI specs
 ResolvedaborreroT356523 [jobs-api] introduce OpenAPI to jobs framework
 ResolvedSlst2020T362299 [api-gateway] Add a python server to serve consolidated openapi docs
 OpenNoneT369693 [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images
 ResolvedBUG REPORTdcaroT369569 `webservice` requires effective user to be the tool user and listed in NSS passwd data
 ResolvedBUG REPORTdcaroT369573 `toolforge jobs` requires current user to be the tool user and listed in NSS passwd data
 OpenBUG REPORT• bd808T360488 Missing Perl packages on dev.toolforge.org for anomiebot workflows
 ResolveddcaroT371556 Please re-install "joe"
 ResolvedaborreroT359619 toolforge: prepare deb packages for k8s 1.24
 ResolvedaborreroT359620 toolforge: upgrade k8s etcd nodes to debian bookworm
 ResolvedaborreroT359638 [toolsbeta,infra] upgrade kubernetes to 1.24
 ResolvedaborreroT359639 toolforge: verify etcd version is compatible with k8s 1.24
 ResolvedaborreroT359798 refresh kube-state-metrics version for k8s 1.24
 ResolvedaborreroT359927 [cookbook] wmcs.toolforge.k8s.prepare_upgrade: be more flexible checking for deb package components
 ResolvedaborreroT279110 [infra] Replace PodSecurityPolicy in Toolforge Kubernetes
 OpenNoneT335131 Toolforge: replace admission controllers with an existing policy admin project
 ResolvedaborreroT362050 toolforge: review pod templates for PSP replacement
 ResolvedaborreroT362966 lima-kilo: replicate sssd setup from Toolforge
 ResolvedaborreroT362233 Decision Request - Toolforge policy agent
 ResolvedaborreroT362872 Decision Request - Toolforge policy agent enforcement model
 ResolvedaborreroT363347 toolforge lima-kilo: PodSecurityPolicy admission is disabled
 OpenaborreroT362967 lima-kilo: container image caching
 ResolvedaborreroT363482 toolforge lima-kilo: refresh maintain-kubeusers test data
 ResolvedaborreroT364113 toolforge: identify and cache in our container registry all kyverno images
 ResolvedaborreroT364297 [k8s,infra] track PSP migration plan
 ResolvedaborreroT364312 [maintain-kubeusers,infra,k8s]: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies)
 ResolvedaborreroT366564 toolforge: new maintain-kubeusers takes long time to loop over all the accounts to reconcile them
 ResolvedaborreroT366598 maintain-kubeusers: metrics, monitoring and alerting
 ResolvedaborreroT367332 toolforge maintain-kubeusers backtrace
 ResolvedAndrewT367348 Incident: 2024-06-12 toolforge k8s control plane
 DuplicatedcaroT367349 Fix HA proxy load-balancer health check monitor to not poll nodes where the API is not responding
 ResolvedaborreroT367350 [k8s,infra] Verify that kyverno policies are evaluated only for namespaced resources
 ResolvedaborreroT367386 [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways
 ResolvedaborreroT367388 [k8s,infra] consider scaling the k8s control plane
 ResolvedaborreroT367389 [k8s,infra,alerting] improve HAproxy and k8s apiserver interaction
 DeclinedaborreroT367950 Decision Request - Toolforge pod security via custom admission webhook
 DeclinedaborreroT367952 toolforge: drop kyverno
 DeclinedaborreroT367985 toolforge: create a new custom admission webhook to handle pod security settings
 ResolvedaborreroT368044 Toolforge: redeploy kyverno after the outage
 ResolvedaborreroT368141 toolforge: kyverno: change policies to Enforce
 ResolvedaborreroT368142 Toolforge: drop PodSecurityPolicy
 OpenaborreroT368135 kyverno: explore change from per-namespace policy resource to a single ClusterPolicy resource
 DuplicateNoneT368512 toolforge: maintain-kubeusers crashes if LDAP server terminates session
 ResolvedaborreroT368515 toolforge: kyverno: enable monitoring
 ResolvedSlst2020T329671 toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.25
 ResolvedaborreroT349197 [infra] Remove TTLAfterFinished from config before upgrade to 1.25
 ResolvedaborreroT355881 [toolforge,infra] Fix deprecated Kubelet flags
 ResolvedaborreroT369163 toolforge: prepare deb packages for k8s 1.25
 ResolvedaborreroT369164 toolforge: review k8s API usage by custom components for 1.25 upgrade
 ResolvedSlst2020T369165 toolforge: upgrade lima-kilo for kubernetes 1.25
 ResolvedSlst2020T369166 toolforge: refresh kubernetes cookbooks for the 1.25 upgrade
 ResolvedaborreroT369167 toolforge: upgrade toolsbeta to k8s 1.25
 ResolvedaborreroT369168 toolsbeta: upgrade control plane nodes to k8s 1.25
 ResolvedaborreroT369170 toolsbeta: upgrade data plane nodes to k8s 1.25
 ResolvedaborreroT369171 toolforge: upgrade worker nodes to k8s 1.25
 ResolvedaborreroT369172 toolforge: upgrade control plane nodes to k8s 1.25
 ResolvedSlst2020T370244 [lima-kilo, k8s] Upgrade Kubernetes in lima-kilo to version 1.26
 ResolvedSlst2020T370046 toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26
 ResolvedSlst2020T370246 [infra,k8s] prepare deb packages for k8s 1.26
 ResolvedSlst2020T370247 [infra,k8s] review k8s API usage by custom components for 1.26 upgrade
 ResolvedSlst2020T370248 [infra,k8s] Upgrade Toolsbeta to k8s 1.26
 ResolvedSlst2020T370249 [infra,k8s] Upgrade Tools to k8s version 1.26
 ResolvedAndrewT349207 [infra] Upgrade Toolforge K8s etcd nodes to Bullseye
 OpenNoneT370245 [infra,k8s] remove deprecated kubelet flags before 1.27 upgrade
 ResolveddcaroT373972 [infra,k8s,kyverno] Toolforge Kyverno low policy resources tools
 In ProgressdcaroT374908 [builds-builder,builds-api] Upgrade tekton

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Tue, Sep 17, 1:49 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T01:49:10Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 (T359641)

Stashbot added a comment.Tue, Sep 17, 1:50 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T01:50:50Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 (T359641)

Stashbot added a comment.Tue, Sep 17, 1:54 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T01:54:26Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 (T359641)

Stashbot added a comment.Tue, Sep 17, 1:57 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T01:57:22Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 (T359641)

Stashbot added a comment.Tue, Sep 17, 1:57 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T01:57:36Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 (T359641)

Stashbot added a comment.Tue, Sep 17, 1:58 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T01:58:15Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)

Stashbot added a comment.Tue, Sep 17, 1:58 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T01:58:22Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)

Stashbot added a comment.Tue, Sep 17, 2:04 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:04:52Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 (T359641)

Stashbot added a comment.Tue, Sep 17, 2:05 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:05:11Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 (T359641)

Stashbot added a comment.Tue, Sep 17, 2:12 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:12:35Z] <raymond-ndibe@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster (T359641)

Stashbot added a comment.Tue, Sep 17, 2:15 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:15:22Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)

Stashbot added a comment.Tue, Sep 17, 2:22 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:22:33Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)

Stashbot added a comment.Tue, Sep 17, 2:29 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:29:28Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)

Stashbot added a comment.Tue, Sep 17, 2:36 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:36:06Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)

Stashbot added a comment.Tue, Sep 17, 2:40 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:40:07Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 (T359641)

Stashbot added a comment.Tue, Sep 17, 2:40 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:40:14Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 (T359641)

Stashbot added a comment.Tue, Sep 17, 2:50 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:50:30Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)

Stashbot added a comment.Tue, Sep 17, 2:57 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T02:57:32Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)

Stashbot added a comment.Tue, Sep 17, 3:04 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T03:04:40Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 (T359641)

Stashbot added a comment.Tue, Sep 17, 3:08 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T03:08:20Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 (T359641)

Stashbot added a comment.Tue, Sep 17, 3:18 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T03:18:01Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 (T359641)

Stashbot added a comment.Tue, Sep 17, 3:19 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T03:18:59Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 (T359641)

Stashbot added a comment.Tue, Sep 17, 3:19 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T03:19:51Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 (T359641)

Stashbot added a comment.Tue, Sep 17, 3:20 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T03:20:49Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 (T359641)

Stashbot added a comment.Tue, Sep 17, 3:23 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T03:23:11Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 (T359641)

Stashbot added a comment.Tue, Sep 17, 3:24 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T03:24:19Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 (T359641)

Maintenance_bot removed a project: Patch-For-Review.Tue, Sep 17, 3:30 AM
dcaro changed the status of subtask T374908: [builds-builder,builds-api] Upgrade tekton from Open to In Progress.Tue, Sep 17, 8:12 AM
dcaro added a comment.Tue, Sep 17, 8:31 AM

There were a couple worker nodes this morning having puppet issues, one of them was stuck with the dpkg error we saw in the last upgrade too:

root@tools-k8s-worker-nfs-65:~# run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for tools-k8s-worker-nfs-65.tools.eqiad1.wikimedia.cloud
Info: Applying configuration version '(8afe6a6e23) David Caro - typos: add colud to the list'
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python3-ldap' returned 100: E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.
Error: /Stage[main]/Profile::Ssh::Server/Package[python3-ldap]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python3-ldap' returned 100: E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. (corrective)

Solved it by running the command it suggests:

root@tools-k8s-worker-nfs-65:~# dpkg --configure -a
root@tools-k8s-worker-nfs-65:~# run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for tools-k8s-worker-nfs-65.tools.eqiad1.wikimedia.cloud
Info: Applying configuration version '(8afe6a6e23) David Caro - typos: add colud to the list'
Notice: /Stage[main]/Profile::Ssh::Server/Package[python3-ldap]/ensure: created (corrective)
Notice: Applied catalog in 9.11 seconds

The other seems to have certificate issues:

rroot@tools-k8s-worker-nfs-75:~# run-puppet-agent
Info: Using environment 'production'
Error: certificate verify failed [self-signed certificate in certificate chain for CN=Puppet CA: tools-puppetmaster-01.tools.eqiad.wmflabs]
Info: Loading facts
Error: Could not retrieve catalog from remote server: certificate verify failed [self-signed certificate in certificate chain for CN=Puppet CA: tools-puppetmaster-01.tools.eqiad.wmflabs]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: certificate verify failed [self-signed certificate in certificate chain for CN=Puppet CA: tools-puppetmaster-01.tools.eqiad.wmflabs]

Looking

Stashbot added a comment.Tue, Sep 17, 8:32 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T08:32:42Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud (T359641)

Stashbot added a comment.Tue, Sep 17, 8:35 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T08:35:25Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud (T359641)

dcaro added a comment.Tue, Sep 17, 8:35 AM

Running the refresh certs cookbook did the trick:

dcaro@urcuchillay$ cookbooks wmcs.vps.refresh_puppet_certs --fqdn tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud --task-id T359641

It was though missing a lot of setup, so I'm looking to see if the worker is behaving as expected.

Stashbot added a comment.Tue, Sep 17, 8:40 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T08:40:20Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 (T359641)

Stashbot added a comment.Tue, Sep 17, 8:40 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T08:40:49Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud (T359641)

Stashbot added a comment.Tue, Sep 17, 8:41 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T08:41:24Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 (T359641)

Stashbot added a comment.Tue, Sep 17, 8:43 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T08:43:32Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud (T359641)

Stashbot added a comment.Tue, Sep 17, 8:43 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T08:43:40Z] <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 (T359641)

Stashbot added a comment.Tue, Sep 17, 8:46 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-17T08:46:35Z] <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 (T359641)

dcaro added a comment.Tue, Sep 17, 8:46 AM

I found tools-k8s-worker-nfs-75 and 70 that were missing a full puppet run (I'm guessing after creation?), they had a bunch of stuck pods in 'ContainerCreating' status, fix the certs, ran puppet again (and did a lot of changes), and rebooted them, and now they are back online.

dcaro added a comment.Tue, Sep 17, 9:35 AM

It seems also that when removing the old workers, the certs were not cleaned up properly (maybe they were removed by hand?), so I ran this on the puppetsevrer:

root@tools-puppetserver-01:~# for host in $(grep -o 'cert_name="[^"]*' /var/lib/prometheus/node.d/openstack_stale_puppet_certs.prom  | cut -d'"' -f2); do ping -c1 -w1 "$host" && { echo "SKIPPING: $host is alive"; continue; }; puppetserver ca clean --certname "$host"; done
taavi subscribed.Thu, Sep 19, 9:49 AM

Anything left to do here? Or can this task be closed?

dcaro added a comment.Thu, Sep 19, 12:01 PM
In T359641#10159899, @taavi wrote:

Anything left to do here? Or can this task be closed?

We were waiting mostly to see if it was stable, as it was a bumpy upgrade, maybe @Raymond_Ndibe is wants to upgrade something else in the list still, though that can be done in a different task too.

Stashbot added a comment.Thu, Sep 19, 10:35 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T22:35:37Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Thu, Sep 19, 10:35 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T22:35:43Z] <wmbot~raymondndibe@wmf3402> Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 (T359641)

Stashbot added a comment.Thu, Sep 19, 10:35 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T22:35:47Z] <wmbot~raymondndibe@wmf3402> END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) (T359641)

Stashbot added a comment.Thu, Sep 19, 10:36 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T22:36:09Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Thu, Sep 19, 10:36 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T22:36:11Z] <wmbot~raymondndibe@wmf3402> Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 (T359641)

Stashbot added a comment.Thu, Sep 19, 10:36 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T22:36:20Z] <wmbot~raymondndibe@wmf3402> END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) (T359641)

Stashbot added a comment.Thu, Sep 19, 10:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T22:37:30Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Thu, Sep 19, 10:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T22:37:32Z] <wmbot~raymondndibe@wmf3402> Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 (T359641)

Stashbot added a comment.Thu, Sep 19, 10:38 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T22:38:08Z] <wmbot~raymondndibe@wmf3402> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)

Stashbot added a comment.Thu, Sep 19, 11:11 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T23:11:49Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Thu, Sep 19, 11:11 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T23:11:53Z] <wmbot~raymondndibe@wmf3402> Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 (T359641)

Stashbot added a comment.Thu, Sep 19, 11:12 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T23:12:25Z] <wmbot~raymondndibe@wmf3402> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)

Stashbot added a comment.Thu, Sep 19, 11:17 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T23:17:02Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Thu, Sep 19, 11:17 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T23:17:06Z] <wmbot~raymondndibe@wmf3402> Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 (T359641)

Stashbot added a comment.Thu, Sep 19, 11:17 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T23:17:08Z] <wmbot~raymondndibe@wmf3402> END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) (T359641)

Stashbot added a comment.Fri, Sep 20, 12:25 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T00:25:14Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T359641)

Stashbot added a comment.Fri, Sep 20, 12:30 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T00:30:52Z] <wmbot~raymondndibe@wmf3402> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T359641)

Stashbot added a comment.Fri, Sep 20, 12:32 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T00:32:30Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T359641)

Stashbot added a comment.Fri, Sep 20, 12:39 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T00:39:29Z] <wmbot~raymondndibe@wmf3402> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T359641)

Raymond_Ndibe updated the task description. (Show Details)Fri, Sep 20, 12:42 AM
CodeReviewBot added a comment.Fri, Sep 20, 4:11 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/520

[toolforge-deploy] upgrade wmcs-k8s-metrics

Stashbot added a comment.Fri, Sep 20, 4:54 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T16:54:38Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Fri, Sep 20, 4:54 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T16:54:42Z] <wmbot~raymondndibe@wmf3402> Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 (T359641)

Stashbot added a comment.Fri, Sep 20, 4:56 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T16:56:37Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Fri, Sep 20, 4:56 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T16:56:39Z] <raymond-ndibe@cloudcumin1001> Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 (T359641)

Stashbot added a comment.Fri, Sep 20, 4:57 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T16:57:03Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)

Stashbot added a comment.Fri, Sep 20, 4:59 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T16:59:34Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Fri, Sep 20, 4:59 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T16:59:36Z] <raymond-ndibe@cloudcumin1001> Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 (T359641)

Stashbot added a comment.Fri, Sep 20, 4:59 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T16:59:46Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)

Stashbot added a comment.Fri, Sep 20, 5:02 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:02:18Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Fri, Sep 20, 5:02 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:02:20Z] <raymond-ndibe@cloudcumin1001> Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 (T359641)

Stashbot added a comment.Fri, Sep 20, 5:02 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:02:32Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)

Stashbot added a comment.Fri, Sep 20, 5:03 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:03:08Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Fri, Sep 20, 5:03 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:03:10Z] <raymond-ndibe@cloudcumin1001> Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 (T359641)

Stashbot added a comment.Fri, Sep 20, 5:04 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:04:02Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)

Stashbot added a comment.Fri, Sep 20, 5:04 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:04:38Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Fri, Sep 20, 5:04 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:04:41Z] <raymond-ndibe@cloudcumin1001> Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 (T359641)

Stashbot added a comment.Fri, Sep 20, 5:04 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:04:51Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)

Stashbot added a comment.Fri, Sep 20, 5:05 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:05:57Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

Stashbot added a comment.Fri, Sep 20, 5:06 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:05:59Z] <raymond-ndibe@cloudcumin1001> Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 (T359641)

Stashbot added a comment.Fri, Sep 20, 5:06 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T17:06:02Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)

CodeReviewBot added a project: Patch-For-Review.Fri, Sep 20, 5:16 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/10

[calico] upgrade to 3.28.2

CodeReviewBot added a comment.Fri, Sep 20, 5:18 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/10

[calico] upgrade to 3.28.2

CodeReviewBot added a comment.Fri, Sep 20, 5:19 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/524

calico: bump to 0.0.10-20240920171811-afd7f481

CodeReviewBot added a comment.Fri, Sep 20, 5:25 PM

raymond-ndibe closed https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/524

calico: bump to 0.0.10-20240920171811-afd7f481

CodeReviewBot added a comment.Fri, Sep 20, 5:27 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/525

calico: bump to 0.0.11-20240920172624-a95e9e60

CodeReviewBot added a comment.Fri, Sep 20, 5:27 PM

raymond-ndibe closed https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/525

calico: bump to 0.0.11-20240920172624-a95e9e60

CodeReviewBot added a comment.Fri, Sep 20, 5:30 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12

[calico] upgrade to 3.28.2

CodeReviewBot added a comment.Fri, Sep 20, 7:05 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12

[calico] upgrade to 3.28.2

CodeReviewBot added a comment.Fri, Sep 20, 7:07 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/526

calico: bump to 0.0.12-20240920190545-a342cb41

CodeReviewBot added a comment.Fri, Sep 20, 7:12 PM

raymond-ndibe closed https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/526

calico: bump to 0.0.12-20240920190545-a342cb41

CodeReviewBot added a comment.Fri, Sep 20, 7:13 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/14

[calico] upgrade to 3.28.2

CodeReviewBot added a comment.Fri, Sep 20, 7:15 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/14

[calico] upgrade to 3.28.2

CodeReviewBot added a comment.Fri, Sep 20, 7:16 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/527

calico: bump to 0.0.14-20240920191533-f94f2f8d

CodeReviewBot added a comment.Fri, Sep 20, 8:13 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/527

calico: bump to 0.0.14-20240920191533-f94f2f8d

Raymond_Ndibe updated the task description. (Show Details)Fri, Sep 20, 8:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy