Why is my Kubernetes service slow when no pod is crashing? Could it be CPU throttling?
A CPU limit lets the kernel pause a container once it has used its quota for the current scheduling period. The container is never killed, so there are no restarts or errors, only added latency on every request. Compare kubectl top with the limit in the Deployment, and look at the throttling metrics. In this incident a cost review set the limit below what a pod needs under load.
What it looks like
- Latency, especially p99, rises while pods stay Running with no restarts.
- CPU use sits flat at the limit instead of following load.
- Error rates stay low, so nothing alerts on failures.
- The slowdown began after a change to resource limits.
Commands that find the cause
kubectl top pod -n checkout-apiCPU(cores) is what each pod used over the last sample window. Compare it with the pod's CPU request and limit, and run it a few times: readings that sit on one value are worth a closer look.
kubectl describe pod -n checkout-api <pod-name>Under the container, Limits and Requests show cpu and memory as set in the spec. Read them next to what top showed. The Events at the bottom list what Kubernetes did.
kubectl get pods -n checkout-apiRead STATUS, READY and RESTARTS. Running, READY n/n and a RESTARTS count of 0 mean the pods are up and stable. That says nothing about how quickly they answer.
kubectl top nodesCPU(cores) and CPU% per node, and the same for memory. A node near 100% CPU is short for everything on it; a low percentage means CPU is not scarce at node level.
kubectl get deploy checkout-api -n checkout-api -o yamlThe container's resources block is where CPU and memory requests and limits live. The values here apply to every new pod.
Root cause
The cost review set the checkout-api CPU limit below what a pod needs under load, so the kernel throttled it and p99 latency rose to seconds.
The fix
Give checkout-api a CPU limit with headroom:
kubectl set resources deploy/checkout-api -n checkout-api --requests=cpu=50m --limits=cpu=200mHow to prevent it
Set CPU limits from measured load, or rely on requests and leave the limit out. Watch container_cpu_cfs_throttled_seconds_total so throttling shows on a graph before it shows in support tickets.
Run it on your own cluster
SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.