SnowOps Night Shift

Why is my Kubernetes pod in CrashLoopBackOff right after a config change?

A pod shows CrashLoopBackOff when its container keeps exiting and the kubelet waits longer between each restart. The logs of the previous container, from kubectl logs with --previous, show why the process exited. When the trouble starts right after a Deployment change, compare the environment and arguments in the Deployment with what the app expects. In this incident an invalid PORT value stopped the app at startup.

What it looks like

Commands that find the cause

  1. kubectl logs -n checkout-api <pod-name> --previous

    --previous prints the last run of a container that already exited, which is where a crash leaves its message. Read the final lines first: normal is a clean startup followed by request lines.

  2. kubectl get deploy checkout-api -n checkout-api -o yaml

    The pod template is under spec.template.spec.containers: image, ports, env, resources and probes. Every new pod is built from it, so read the values it sets and compare related ones with each other.

  3. kubectl get pods -n checkout-api

    Read STATUS, READY and RESTARTS together. Normal is Running, READY n/n and a RESTARTS count that stays put. A STATUS that is not Running, or a RESTARTS count that grows between looks, tells you which pod to inspect next.

  4. kubectl get rs -n checkout-api

    One row per ReplicaSet with DESIRED, CURRENT and READY. Normal is one current set at full strength and older sets at 0; during a rollout the old set shrinks as the new one becomes READY.

  5. kubectl get events -n checkout-api --sort-by=.lastTimestamp

    Newest events are last. Each row has a TYPE, REASON, OBJECT and MESSAGE. Normal rows are routine (Scheduled, Pulled, Created, Started); Warning rows are what Kubernetes itself noticed.

Root cause

A config change set PORT to 80800 on the checkout-api Deployment, so the app could not listen, exited at startup and left no pod serving.

The fix

Correct the PORT value in the Deployment:

kubectl set env deploy/checkout-api -n checkout-api PORT=8080

How to prevent it

Validate environment values in CI and again when the app starts, so a bad value fails loudly with a clear message. Keep maxUnavailable low on rolling updates so one bad change cannot replace every working pod.

Run it on your own cluster

SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.

More Kubernetes incidents