SnowOps Night Shift

Why is my Kubernetes service slow when no pod is crashing? Could it be CPU throttling?

A CPU limit lets the kernel pause a container once it has used its quota for the current scheduling period. The container is never killed, so there are no restarts or errors, only added latency on every request. Compare kubectl top with the limit in the Deployment, and look at the throttling metrics. In this incident a cost review set the limit below what a pod needs under load.

What it looks like

Commands that find the cause

  1. kubectl top pod -n checkout-api

    CPU(cores) is what each pod used over the last sample window. Compare it with the pod's CPU request and limit, and run it a few times: readings that sit on one value are worth a closer look.

  2. kubectl describe pod -n checkout-api <pod-name>

    Under the container, Limits and Requests show cpu and memory as set in the spec. Read them next to what top showed. The Events at the bottom list what Kubernetes did.

  3. kubectl get pods -n checkout-api

    Read STATUS, READY and RESTARTS. Running, READY n/n and a RESTARTS count of 0 mean the pods are up and stable. That says nothing about how quickly they answer.

  4. kubectl top nodes

    CPU(cores) and CPU% per node, and the same for memory. A node near 100% CPU is short for everything on it; a low percentage means CPU is not scarce at node level.

  5. kubectl get deploy checkout-api -n checkout-api -o yaml

    The container's resources block is where CPU and memory requests and limits live. The values here apply to every new pod.

Root cause

The cost review set the checkout-api CPU limit below what a pod needs under load, so the kernel throttled it and p99 latency rose to seconds.

The fix

Give checkout-api a CPU limit with headroom:

kubectl set resources deploy/checkout-api -n checkout-api --requests=cpu=50m --limits=cpu=200m

How to prevent it

Set CPU limits from measured load, or rely on requests and leave the limit out. Watch container_cpu_cfs_throttled_seconds_total so throttling shows on a graph before it shows in support tickets.

Run it on your own cluster

SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.

More Kubernetes incidents