Kubernetes incidents, explained
Each guide covers one failure from the SnowOps Night Shift week: what it looks like from outside, the kubectl commands that find the root cause, the fix, and how to stop it coming back. The incidents were recorded on a real cluster.
- Week 1, MondayCrashLoopBackOff after a config changeWhy is my Kubernetes pod in CrashLoopBackOff right after a config change?
- Week 1, TuesdayService returns 503 while every pod is RunningWhy does my Kubernetes Service return 503 when all the pods are Running and Ready?
- Week 1, WednesdayOOMKilled and exit code 137Why does my Kubernetes container keep restarting with OOMKilled and exit code 137?
- Week 1, ThursdayPods fail to start after a Secret changeWhy do my Kubernetes pods not start after a Secret key was renamed?
- Week 1, FridayHealthy pods restarted by a liveness probeWhy does Kubernetes keep restarting a healthy pod because of its liveness probe?
- Week 2, MondaySlow service with low CPU and no restartsWhy is my Kubernetes service slow when no pod is crashing? Could it be CPU throttling?
- Week 2, TuesdayRollout stuck on ImagePullBackOffWhy is my Kubernetes rollout stuck with ImagePullBackOff?
- Week 2, WednesdayPods stuck Pending with insufficient CPUWhy are my Kubernetes pods stuck in Pending with insufficient CPU?