SnowOps Night Shift

Why do my Kubernetes pods not start after a Secret key was renamed?

When a container reads an environment variable from a Secret key that does not exist, Kubernetes cannot create the container and the pod stays in an error state such as CreateContainerConfigError. The pod's events name the missing key, and kubectl describe secret lists the keys the Secret has now. Running pods keep working until they are replaced, which is why the failure often shows up hours after the change. In this incident a rotation renamed the key.

What it looks like

Commands that find the cause

  1. kubectl describe pod -n checkout-api <pod-name>

    Read State and Reason under the container, then the Events at the bottom. A Waiting state carries a Reason, and the event messages say in words what Kubernetes could not do and which object it was looking for.

  2. kubectl describe secret checkout-api-secret -n checkout-api

    Data lists each key in the Secret with its size in bytes, never the value. Type and Name are at the top. Read the key names carefully: references to them have to match exactly.

  3. kubectl get pods -n checkout-api

    Read STATUS, READY and RESTARTS together. Running with READY n/n is normal. Compare a STATUS that is not Running with its RESTARTS count: a count of 0 and a count that climbs describe different things.

  4. kubectl logs -n checkout-api <pod-name>

    The container's own output for its current run. Normal is startup lines followed by request lines. If kubectl prints an error or nothing, ask what that says about how far the container got.

  5. kubectl get events -n checkout-api --sort-by=.lastTimestamp

    Newest last; read TYPE, REASON and MESSAGE. Warning rows name the object involved and what failed. Normal rows are Scheduled, Pulled, Created and Started.

Root cause

The rotation renamed the Secret key api-token to API_TOKEN, so when the nodes were replaced the new pods could not find it and never started.

The fix

Point the Secret reference at the key the Secret has now:

kubectl patch deploy checkout-api -n checkout-api -p '{"spec":{"template":{"spec":{"containers":[{"name":"checkout-api","env":[{"name":"API_TOKEN","valueFrom":{"secretKeyRef":{"name":"checkout-api-secret","key":"API_TOKEN"}}}]}]}}}}'

How to prevent it

Treat Secret key names as an interface: change them in the same release that updates every Deployment reading them. Roll out Secret changes to a canary pod before the whole fleet.

Run it on your own cluster

SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.

More Kubernetes incidents