Why does my Kubernetes container keep restarting with OOMKilled and exit code 137?
Exit code 137 with the reason OOMKilled means the kernel stopped the container for using more memory than its limit. kubectl describe pod shows it under Last State, next to the container's limits and requests. Raise the limit to fit the real peak use with some headroom, or find the leak if memory grows without bound. In this incident the limit was set below what the app needs under load.
What it looks like
- RESTARTS climbs while the logs show no error, because the process was killed from outside.
- Last State is Terminated with the reason OOMKilled and exit code 137.
- On the memory graph the line meets the limit just before each restart.
- Failures arrive with traffic and ease off when load drops.
Commands that find the cause
kubectl describe pod -n checkout-api <pod-name>Read the Containers section: State, Last State with its Reason and Exit Code, Restart Count, Limits and Requests. Exit Code 0 is a clean exit, 1 and up is a failure, and 137 is a kill from outside. The Events at the bottom list what Kubernetes did.
kubectl get deploy checkout-api -n checkout-api -o yamlThe resources block under the container holds requests (what the scheduler reserves) and limits (the most the container may use); the dashed line on the memory panel is that limit. Normal is a request the app was sized for and a limit above it with some room; the peak on the panel is only the ceiling the container hit.
kubectl get pods -n checkout-apiRead STATUS, READY and RESTARTS. Normal is Running, READY n/n and a RESTARTS count that stays put. A count that climbs between looks means containers are being stopped and started again.
kubectl logs -n checkout-api <pod-name> --previousPrints the last run of a container that already stopped; read the final lines first. Normal is startup lines followed by request lines.
kubectl top pods -n checkout-apiCurrent CPU(cores) and MEMORY(bytes) per pod. A single reading at rest tells little: compare it with the limits while the service is under load, and look again a few times.
Root cause
The checkout-api memory limit was set below what the app needs under load, so the kernel killed the container (exit code 137) whenever traffic arrived.
The fix
Raise the memory limit and restore the request:
kubectl set resources deploy/checkout-api -n checkout-api --limits=memory=128Mi --requests=memory=32MiHow to prevent it
Size limits from measured peak use under load, and set requests near typical use. Alert on restart counts and on memory close to the limit, so a squeeze shows up before users see it.
Run it on your own cluster
SnowOps Labs, the free hands-on Kubernetes lab, injects this incident into a cluster you run yourself, so you can work through it with your own tools. Open the lab page for this incident.