SnowOps Night Shift

Why is my Kubernetes rollout stuck with ImagePullBackOff?

ImagePullBackOff means the kubelet could not pull the container image: the tag does not exist, the name is wrong or the registry credentials are missing. The new pods wait while the old ones keep serving, so a rollout stalls instead of failing outright. Check the image in the Deployment against what was pushed, read the pod's events, and use kubectl rollout undo to restore the last good release. In this incident the release named a tag that was never pushed.

What it looks like

Commands that find the cause

  1. kubectl describe pod -n checkout-api <pod-name>

    Read the container State and Reason, then the Events at the bottom. Events record each attempt in order, with the object and the reason in the MESSAGE.

  2. kubectl get deploy checkout-api -n checkout-api -o jsonpath='{.spec.template.spec.containers[0].image}'

    Prints one field of the Deployment: the image its pod template asks for, as repository:tag. Compare it with the image of a pod that is serving.

  3. kubectl rollout status deploy/checkout-api -n checkout-api --timeout=10s

    Prints progress and ends with 'successfully rolled out' once every new pod is ready. If it is still waiting when the timeout ends, the rollout has not finished.

  4. kubectl get pods -n checkout-api

    Read STATUS and AGE. During a rollout old and new pods sit side by side; normal ends with only new pods, all Running and READY n/n.

  5. kubectl get events -n checkout-api --sort-by=.lastTimestamp

    Newest events are last. Each row has a TYPE, REASON, OBJECT and MESSAGE. Normal rows are routine (Scheduled, Pulled, Created, Started); Warning rows are what Kubernetes itself noticed.

Root cause

The Deployment pointed at an image tag that was never pushed, so the new pods could not pull it and the release stalled behind the old pods.

The fix

Roll back to the last good release:

kubectl rollout undo deploy/checkout-api -n checkout-api

How to prevent it

Push the image and update the manifest in one pipeline, and block the deploy until the registry confirms the tag exists. Keep a rollback command ready in the runbook.

Run it on your own cluster

SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.

More Kubernetes incidents