Why do my Kubernetes pods not start after a Secret key was renamed?
When a container reads an environment variable from a Secret key that does not exist, Kubernetes cannot create the container and the pod stays in an error state such as CreateContainerConfigError. The pod's events name the missing key, and kubectl describe secret lists the keys the Secret has now. Running pods keep working until they are replaced, which is why the failure often shows up hours after the change. In this incident a rotation renamed the key.
What it looks like
- New pods never become Ready and show a config error status instead of Running.
- The pod's events say a key could not be found in the Secret.
- The application logs are empty, because the container never started.
- The failure appeared when nodes or pods were replaced, not when the Secret changed.
Commands that find the cause
kubectl describe pod -n checkout-api <pod-name>Read State and Reason under the container, then the Events at the bottom. A Waiting state carries a Reason, and the event messages say in words what Kubernetes could not do and which object it was looking for.
kubectl describe secret checkout-api-secret -n checkout-apiData lists each key in the Secret with its size in bytes, never the value. Type and Name are at the top. Read the key names carefully: references to them have to match exactly.
kubectl get pods -n checkout-apiRead STATUS, READY and RESTARTS together. Running with READY n/n is normal. Compare a STATUS that is not Running with its RESTARTS count: a count of 0 and a count that climbs describe different things.
kubectl logs -n checkout-api <pod-name>The container's own output for its current run. Normal is startup lines followed by request lines. If kubectl prints an error or nothing, ask what that says about how far the container got.
kubectl get events -n checkout-api --sort-by=.lastTimestampNewest last; read TYPE, REASON and MESSAGE. Warning rows name the object involved and what failed. Normal rows are Scheduled, Pulled, Created and Started.
Root cause
The rotation renamed the Secret key api-token to API_TOKEN, so when the nodes were replaced the new pods could not find it and never started.
The fix
Point the Secret reference at the key the Secret has now:
kubectl patch deploy checkout-api -n checkout-api -p '{"spec":{"template":{"spec":{"containers":[{"name":"checkout-api","env":[{"name":"API_TOKEN","valueFrom":{"secretKeyRef":{"name":"checkout-api-secret","key":"API_TOKEN"}}}]}]}}}}'How to prevent it
Treat Secret key names as an interface: change them in the same release that updates every Deployment reading them. Roll out Secret changes to a canary pod before the whole fleet.
Run it on your own cluster
SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.