SnowOps Night Shift

Kubernetes incidents, explained

Each guide covers one failure from the SnowOps Night Shift week: what it looks like from outside, the kubectl commands that find the root cause, the fix, and how to stop it coming back. The incidents were recorded on a real cluster.

  1. Week 1, MondayCrashLoopBackOff after a config changeWhy is my Kubernetes pod in CrashLoopBackOff right after a config change?
  2. Week 1, TuesdayService returns 503 while every pod is RunningWhy does my Kubernetes Service return 503 when all the pods are Running and Ready?
  3. Week 1, WednesdayOOMKilled and exit code 137Why does my Kubernetes container keep restarting with OOMKilled and exit code 137?
  4. Week 1, ThursdayPods fail to start after a Secret changeWhy do my Kubernetes pods not start after a Secret key was renamed?
  5. Week 1, FridayHealthy pods restarted by a liveness probeWhy does Kubernetes keep restarting a healthy pod because of its liveness probe?
  6. Week 2, MondaySlow service with low CPU and no restartsWhy is my Kubernetes service slow when no pod is crashing? Could it be CPU throttling?
  7. Week 2, TuesdayRollout stuck on ImagePullBackOffWhy is my Kubernetes rollout stuck with ImagePullBackOff?
  8. Week 2, WednesdayPods stuck Pending with insufficient CPUWhy are my Kubernetes pods stuck in Pending with insufficient CPU?