SnowOps Night Shift

Why does Kubernetes keep restarting a healthy pod because of its liveness probe?

When a liveness probe fails the configured number of times in a row, the kubelet kills the container and starts it again, whether or not the app is actually broken. A probe that points at a path the app does not serve fails every time. The pod's events show the failed probe followed by a kill, and the app's own logs look clean. In this incident a values update changed the probe path to one the app does not serve.

What it looks like

Commands that find the cause

  1. kubectl describe pod -n checkout-api <pod-name>

    Read Last State (Reason and Exit Code), Restart Count, and the Liveness and Readiness lines, then the Events at the bottom. Exit Code 0 is a clean exit and 1 and up a failure. Events record what the kubelet did and why.

  2. kubectl get events -n checkout-api --sort-by=.lastTimestamp

    Newest last. The kubelet records its own actions here, such as probe failures and restarts, with the reason in the MESSAGE. Normal rows are Scheduled, Pulled, Created and Started.

  3. kubectl get pods -n checkout-api

    Read STATUS, READY and RESTARTS. Normal is Running, READY n/n and a RESTARTS count that stays put. A pod that is Running while its count keeps rising is being stopped and started again.

  4. kubectl logs -n checkout-api <pod-name> --previous

    The last run of a container that already stopped. Read the final lines first, and note the paths and status codes of the requests. Normal is startup lines followed by 2xx requests.

  5. kubectl get deploy checkout-api -n checkout-api -o yaml

    Probes sit under the container: livenessProbe, readinessProbe and startupProbe, each with a path or command, port, delay, period and failureThreshold. Compare a probe's path and port with what the app serves.

Root cause

A values update changed the liveness probe path from /health to /healthz, which the app does not serve, so the kubelet kept killing healthy pods.

The fix

Point the liveness probe at the path the app serves:

kubectl patch deploy checkout-api -n checkout-api --type=json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/path","value":"/health"}]'

How to prevent it

Test the probe path against a running container before it ships. Keep the liveness check cheap and separate from readiness, and review probe changes as carefully as code.

Run it on your own cluster

SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.

More Kubernetes incidents