SnowOps Night Shift

Why does my Kubernetes Service return 503 when all the pods are Running and Ready?

A Service sends traffic only to pods whose labels match its selector. If the selector matches no pod, the Service has no endpoints and every request fails, even though the pods are healthy. Compare kubectl get endpoints for the Service with kubectl get pods --show-labels, then check the selector in the Service's YAML. In this incident the selector had been changed to a label no pod carried.

What it looks like

Commands that find the cause

  1. kubectl get endpoints checkout-api -n checkout-api

    ENDPOINTS lists the pod IP:port pairs that sit behind the Service. Normal is one address per ready pod, so compare the count with the pods you have.

  2. kubectl get svc checkout-api -n checkout-api -o yaml

    spec.selector holds the labels the Service uses to pick pods, and spec.ports says where it sends traffic. A Service matches pods that carry every key and value in its selector.

  3. kubectl get pods -n checkout-api

    Read READY, STATUS and RESTARTS. Running with READY n/n and a low, steady RESTARTS count is normal and says the containers started. It does not say whether requests can reach them.

  4. kubectl get pods -n checkout-api --show-labels

    The LABELS column lists each pod's labels as key=value pairs. Read them against the keys and values of a selector: they have to match exactly, key by key.

  5. kubectl get events -n checkout-api --sort-by=.lastTimestamp

    Newest last, with TYPE, REASON and MESSAGE. Events record things Kubernetes did or noticed (scheduling, pulls, probe failures, restarts). A quiet list means none of those happened, which is not the same as healthy.

Root cause

The checkout-api Service selector was changed to app.kubernetes.io/name: checkout-api-v2, which no pod carries, so its endpoints emptied and nothing was left to serve.

The fix

Point the Service selector back at the pods' label:

kubectl patch svc checkout-api -n checkout-api -p '{"spec":{"selector":{"app.kubernetes.io/name":"checkout-api"}}}'

How to prevent it

Define the selector labels in one place, such as a shared chart helper, so the Service and the Deployment cannot drift apart. Check that endpoints are populated as part of every rollout.

Run it on your own cluster

SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.

More Kubernetes incidents