Why is my Kubernetes pod in CrashLoopBackOff right after a config change?
A pod shows CrashLoopBackOff when its container keeps exiting and the kubelet waits longer between each restart. The logs of the previous container, from kubectl logs with --previous, show why the process exited. When the trouble starts right after a Deployment change, compare the environment and arguments in the Deployment with what the app expects. In this incident an invalid PORT value stopped the app at startup.
What it looks like
- STATUS reads CrashLoopBackOff and the RESTARTS count keeps climbing.
- The Deployment has no ready replicas, so the service has nothing to send traffic to.
- The previous container's logs end during startup, before any request line.
- The trouble began right after a config or Deployment change.
Commands that find the cause
kubectl logs -n checkout-api <pod-name> --previous--previous prints the last run of a container that already exited, which is where a crash leaves its message. Read the final lines first: normal is a clean startup followed by request lines.
kubectl get deploy checkout-api -n checkout-api -o yamlThe pod template is under spec.template.spec.containers: image, ports, env, resources and probes. Every new pod is built from it, so read the values it sets and compare related ones with each other.
kubectl get pods -n checkout-apiRead STATUS, READY and RESTARTS together. Normal is Running, READY n/n and a RESTARTS count that stays put. A STATUS that is not Running, or a RESTARTS count that grows between looks, tells you which pod to inspect next.
kubectl get rs -n checkout-apiOne row per ReplicaSet with DESIRED, CURRENT and READY. Normal is one current set at full strength and older sets at 0; during a rollout the old set shrinks as the new one becomes READY.
kubectl get events -n checkout-api --sort-by=.lastTimestampNewest events are last. Each row has a TYPE, REASON, OBJECT and MESSAGE. Normal rows are routine (Scheduled, Pulled, Created, Started); Warning rows are what Kubernetes itself noticed.
Root cause
A config change set PORT to 80800 on the checkout-api Deployment, so the app could not listen, exited at startup and left no pod serving.
The fix
Correct the PORT value in the Deployment:
kubectl set env deploy/checkout-api -n checkout-api PORT=8080How to prevent it
Validate environment values in CI and again when the app starts, so a bad value fails loudly with a clear message. Keep maxUnavailable low on rolling updates so one bad change cannot replace every working pod.
Run it on your own cluster
SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.