Why are my Kubernetes pods stuck in Pending with insufficient CPU?
A pod stays Pending when the scheduler finds no node with enough unreserved capacity for its requests. The pod's events show a FailedScheduling warning that names the resource it ran out of. If a single pod requests more CPU than any node can allocate, waiting never helps: lower the request or add larger nodes. In this incident a launch change raised the CPU request above every node's allocatable CPU.
What it looks like
- Some replicas stay Pending, so a rollout stalls partway, such as three of four ready.
- Events show FailedScheduling and mention insufficient CPU.
- Every node reports Ready, so nothing is wrong with the nodes themselves.
- The change that raised the resource request shipped shortly before.
Commands that find the cause
kubectl describe pod -n checkout-api <pod-name>Read Status, Node, the container's resource Requests and Limits, then the Events at the bottom. A pod that has not been placed shows Node: <none>, and the scheduler's events say what it checked.
kubectl get deploy checkout-api -n checkout-api -o yamlThe container's resources block holds requests and limits. Compare its cpu and memory requests with the allocatable figures of the nodes.
kubectl get deploy,rs,pods -n checkout-apiThe Deployment's READY column is ready/desired. Below it come each ReplicaSet and each pod with its STATUS. Normal is READY equal to desired and every pod Running.
kubectl get events -n checkout-api --sort-by=.lastTimestampNewest last; read the REASON and MESSAGE of Warning rows and the object each belongs to. Scheduler and kubelet events cover placing and starting pods.
kubectl top nodesCPU(cores) and CPU% per node: what is used right now, not what is reserved. The scheduler places pods by requests, so a low use figure does not decide whether a pod fits.
Root cause
The launch change raised the checkout-api CPU request above what any node can allocate, so the new pods could not be scheduled and the rollout stalled at three of four.
The fix
Give the container a CPU request the nodes can hold:
kubectl set resources deploy/checkout-api -n checkout-api --requests=cpu=100m --limits=cpu=500mHow to prevent it
Compare new requests against node allocatable CPU and memory in review or CI. Keep a rollout strategy that holds the old pods until the new ones are placed.
Run it on your own cluster
SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.