Why is my Kubernetes rollout stuck with ImagePullBackOff?
ImagePullBackOff means the kubelet could not pull the container image: the tag does not exist, the name is wrong or the registry credentials are missing. The new pods wait while the old ones keep serving, so a rollout stalls instead of failing outright. Check the image in the Deployment against what was pushed, read the pod's events, and use kubectl rollout undo to restore the last good release. In this incident the release named a tag that was never pushed.
What it looks like
- New pods show ErrImagePull or ImagePullBackOff while old pods stay Running.
- kubectl rollout status times out instead of finishing.
- Pod events report a failed image pull for the new tag.
- A release was just applied and the new version never arrived.
Commands that find the cause
kubectl describe pod -n checkout-api <pod-name>Read the container State and Reason, then the Events at the bottom. Events record each attempt in order, with the object and the reason in the MESSAGE.
kubectl get deploy checkout-api -n checkout-api -o jsonpath='{.spec.template.spec.containers[0].image}'Prints one field of the Deployment: the image its pod template asks for, as repository:tag. Compare it with the image of a pod that is serving.
kubectl rollout status deploy/checkout-api -n checkout-api --timeout=10sPrints progress and ends with 'successfully rolled out' once every new pod is ready. If it is still waiting when the timeout ends, the rollout has not finished.
kubectl get pods -n checkout-apiRead STATUS and AGE. During a rollout old and new pods sit side by side; normal ends with only new pods, all Running and READY n/n.
kubectl get events -n checkout-api --sort-by=.lastTimestampNewest events are last. Each row has a TYPE, REASON, OBJECT and MESSAGE. Normal rows are routine (Scheduled, Pulled, Created, Started); Warning rows are what Kubernetes itself noticed.
Root cause
The Deployment pointed at an image tag that was never pushed, so the new pods could not pull it and the release stalled behind the old pods.
The fix
Roll back to the last good release:
kubectl rollout undo deploy/checkout-api -n checkout-apiHow to prevent it
Push the image and update the manifest in one pipeline, and block the deploy until the registry confirms the tag exists. Keep a rollback command ready in the runbook.
Run it on your own cluster
SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.