Following the deployment of the latest refactor of maintain-kubeusers in T364312: [maintain-kubeusers,infra,k8s]: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies), we need to make sure we have the metrics, monitoring and alerting that we need.
As of this writing, I don't think we even have any alert if the daemon is not running.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| In Progress | Raymond_Ndibe | T362867 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 | |||
| Resolved | Raymond_Ndibe | T359641 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 | |||
| Restricted Task | |||||
| Resolved | Slst2020 | T327025 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.26 | |||
| Resolved | aborrero | T316107 [infra,k8s] Upgrade Toolforge Kubernetes to version 1.25 | |||
| Resolved | aborrero | T279110 [infra] Replace PodSecurityPolicy in Toolforge Kubernetes | |||
| Resolved | aborrero | T364297 [k8s,infra] track PSP migration plan | |||
| Resolved | aborrero | T364312 [maintain-kubeusers,infra,k8s]: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies) | |||
| Resolved | aborrero | T366598 maintain-kubeusers: metrics, monitoring and alerting |
aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/12
alerts: add maintain-kubeusers
aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/12
alerts: add maintain-kubeusers
we have basic alerts and a basic dashboard https://grafana-rw.wmcloud.org/d/E9HyRBySk/maintain-kubeusers?from=now-6h&to=now&orgId=1 (thanks to @dcaro)
aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/13
maintain-kubeusers: print the time in the hang alert
aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/35
cli: initialize maintain_kubeusers_run_finished prometheus metrics
aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/13
maintain-kubeusers: print the time in the hang alert
aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/14
maintain-kubeusers: MaintainKubeusersHang: adjust alert 'for' value
aborrero closed https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/35
cli: initialize maintain_kubeusers_run_finished prometheus metrics
aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/14
maintain-kubeusers: MaintainKubeusersHang: adjust alert 'for' value