SnowOps Night Shift

Why are my Kubernetes pods stuck in Pending with insufficient CPU?

A pod stays Pending when the scheduler finds no node with enough unreserved capacity for its requests. The pod's events show a FailedScheduling warning that names the resource it ran out of. If a single pod requests more CPU than any node can allocate, waiting never helps: lower the request or add larger nodes. In this incident a launch change raised the CPU request above every node's allocatable CPU.

What it looks like

Commands that find the cause

  1. kubectl describe pod -n checkout-api <pod-name>

    Read Status, Node, the container's resource Requests and Limits, then the Events at the bottom. A pod that has not been placed shows Node: <none>, and the scheduler's events say what it checked.

  2. kubectl get deploy checkout-api -n checkout-api -o yaml

    The container's resources block holds requests and limits. Compare its cpu and memory requests with the allocatable figures of the nodes.

  3. kubectl get deploy,rs,pods -n checkout-api

    The Deployment's READY column is ready/desired. Below it come each ReplicaSet and each pod with its STATUS. Normal is READY equal to desired and every pod Running.

  4. kubectl get events -n checkout-api --sort-by=.lastTimestamp

    Newest last; read the REASON and MESSAGE of Warning rows and the object each belongs to. Scheduler and kubelet events cover placing and starting pods.

  5. kubectl top nodes

    CPU(cores) and CPU% per node: what is used right now, not what is reserved. The scheduler places pods by requests, so a low use figure does not decide whether a pod fits.

Root cause

The launch change raised the checkout-api CPU request above what any node can allocate, so the new pods could not be scheduled and the rollout stalled at three of four.

The fix

Give the container a CPU request the nodes can hold:

kubectl set resources deploy/checkout-api -n checkout-api --requests=cpu=100m --limits=cpu=500m

How to prevent it

Compare new requests against node allocatable CPU and memory in review or CI. Keep a rollout strategy that holds the old pods until the new ones are placed.

Run it on your own cluster

SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.

More Kubernetes incidents