Kubernetes OOMKilled (exit code 137): causes and fixes

OOMKilled

OOMKilled shows up in the STATUS column of kubectl get pods for a moment after a container dies, then the pod goes back to Running with one more restart. If it keeps happening, the pod settles into CrashLoopBackOff. In kubectl describe pod the evidence is in the container’s last state:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

The message is precise. A process in this container asked for more memory than the container’s limit allows, and the kernel killed it. What it can’t tell you is whether the limit is too low, the app is leaking, or the runtime inside the container doesn’t know about the limit at all. This page covers how to tell those apart, and the fix for each.

What OOMKilled and exit code 137 mean

When the kubelet starts a container, it passes the memory limit to the container runtime, which writes it into the container’s Linux cgroup. The resource management docs describe what happens next: if the container tries to allocate more than the limit, the kernel’s out-of-memory subsystem activates and usually stops one of the processes in the container that tried to allocate. If that process is PID 1 and the container can be restarted, Kubernetes restarts it. The container’s last state then shows the reason OOMKilled, and kubectl describe nodes records the kill too, with a line like Warning OOMKilling Memory cgroup out of memory: Kill process 4481 (stress).

The exit code comes from the signal. The OOM killer sends SIGKILL, which is signal 9 and can’t be caught or blocked. The Bash manual gives the convention: a process killed by signal N exits with 128 + N. So 128 + 9 = 137. Because SIGKILL can’t be handled, the app writes no final log line, which is why the previous logs often stop mid-sentence.

Two details of the kill depend on the node:

• cgroup v2 kills the whole container. Since Kubernetes v1.28, on cgroup v2 nodes the kubelet sets memory.oom.group on container cgroups, so the changelog says all processes in the cgroup are killed together. A worker process going over the limit takes the main process down with it.
• v1.32 added a way back. The kubelet setting singleProcessOOMKill stops the kubelet from setting memory.oom.group, so processes are killed one at a time, as on cgroup v1. On cgroup v2 it defaults to false.

The memory request plays no part in the kill. The memory task docs say a container can go over its request when the node has memory free, but can never go over its limit.

Reading exit code 137 correctly

Exit code 137 alone doesn’t prove an OOM kill. It means SIGKILL, and three different things send one. Check which you have before you touch a limit.

kubectl describe pod <pod>
kubectl get pod <pod> -o jsonpath='{range .status.containerStatuses[*]}{.name}{" "}{.lastState.terminated.reason}{" "}{.lastState.terminated.exitCode}{"\n"}{end}'

• Reason OOMKilled, exit code 137. The container hit its own limit. The rest of this page is about this case.
• Reason Error, exit code 137. Something sent SIGKILL without an OOM. When a pod stops, the Pod Lifecycle docs say the kubelet sends the stop signal, waits for the grace period (30 seconds by default), and then the runtime sends SIGKILL to anything still running. A failed liveness probe also kills the container. Here the fix is in graceful shutdown or the probe, and memory limits won’t help. The CrashLoopBackOff page covers the probe case.
• Pod status Failed, reason Evicted. This is a node problem, covered below.

In a pod with several containers, read the reason for each one. A sidecar such as a log shipper or a proxy can be the container that hits its limit, and its restarts can take the pod’s readiness down with it. The memory task docs describe the pod’s request and limit as the sums of its containers’ values, but each limit is enforced on its own container, so one small sidecar limit is enough to cause OOM kills while the main app has room to spare.

Why the container goes over its memory limit

Cause 1: the limit is below what the app really uses

How to tell it’s yours: the kills happen under normal load, usage climbs to a plateau close to the limit, and the plateau is the same after each restart.

kubectl top pod <pod> --containers

kubectl top needs the metrics-server or another resource metrics API in the cluster, and the docs warn that metrics may be missing for a few minutes after a pod starts. It samples, so a short spike can kill the container without ever showing in top.

Fix:

1. Find the real peak. Watch kubectl top pod --containers under production load, or read the container memory metric in your monitoring tool over a few days, including the busiest hour.
2. Set the request to normal usage and the limit above the peak. For a quick change:

   kubectl set resources deployment/api -c=api --requests=memory=512Mi --limits=memory=1Gi

   Then put the same values in the manifest so the next deploy doesn’t undo them:

   containers:
   - name: api
     image: registry.example.com/api:1.4.2
     resources:
       requests:
         memory: "512Mi"
       limits:
         memory: "1Gi"

3. Count memory-backed volumes. The kubelet tracks a tmpfs emptyDir as container memory, so files written there count toward the limit. Give the volume a sizeLimit and include it in your sizing.
4. Check a node can fit it. Pods are scheduled by requests, so a request bigger than any node can offer leaves the pod Pending.

Cause 2: a memory leak

How to tell it’s yours: usage grows steadily from each start and never levels off, and the time between kills is roughly the same. Raising the limit only stretches that time.

Fix:

1. Confirm the shape. Plot the container’s memory since the last restart. A straight line or staircase up to the limit is a leak. A plateau is cause 1.
2. Take heap snapshots at two points in the climb and compare them. Node.js can write one as it nears the heap limit with --heapsnapshot-near-heap-limit; the JVM has -XX:+HeapDumpOnOutOfMemoryError. Point the output at a volume that outlives the container.
3. Look first at what grows with traffic: unbounded in-memory caches, maps keyed by user or request, listeners that are added and never removed, and buffers held for retries.
4. While you fix the code, raise the limit enough to cut how often it restarts, and note the date so the extra headroom gets removed once the leak is gone.
5. Ship the fix and compare the memory curve over the same number of hours as before. A curve that now flattens confirms the leak is gone; one that climbs more slowly means there is a second one.

Leaks often arrive with a deploy. If the kills started right after a release, compare that release’s diff with the previous one before you open a profiler: a new cache, a new dependency or a change to how requests are buffered is a good first suspect. kubectl rollout undo deployment/<name> buys you time while you look.

Cause 3: the runtime’s heap isn’t sized to the limit

How to tell it’s yours: the app runs on Node.js, the JVM or another runtime with its own heap limit, and the kill comes with no out-of-memory error from the runtime itself. The runtime thought it had room left; the kernel disagreed.

A runtime that sizes its heap from the machine, or from a fixed default, can grow past the container limit before its garbage collector works hard. Set the heap explicitly and leave room under the limit for everything that isn’t heap: thread stacks, native buffers, code and the runtime itself.

Fix for Node.js:

1. Set --max-old-space-size in MiB. The Node.js docs say that as usage approaches it, V8 spends more time on garbage collection, and give the example of 1536 on a machine with 2 GiB to leave memory for other uses.
2. Pass it through NODE_OPTIONS so it applies whatever the start command is:

   containers:
   - name: api
     image: registry.example.com/api:1.4.2
     env:
     - name: NODE_OPTIONS
       value: "--max-old-space-size=1536"
     resources:
       limits:
         memory: "2Gi"

Fix for the JVM:

1. Use -XX:MaxRAMPercentage, which sets the heap as a percentage of the memory the JVM sees. The java command docs give a default of 25 percent and an example of 75.
2. Set it with JAVA_TOOL_OPTIONS:

   env:
   - name: JAVA_TOOL_OPTIONS
     value: "-XX:MaxRAMPercentage=75"

3. Keep the percentage well below 100. The JVM also uses memory outside the heap, and all of it counts toward the container limit.

Cause 4: the node ran out of memory

This one looks different, and it needs a different fix. When the node itself runs low, the kubelet evicts whole pods to protect it. The node-pressure eviction docs give the default hard threshold on Linux as memory.available<100Mi. An evicted pod’s phase is set to Failed, its reason is Evicted, and the message comes from the kubelet’s eviction code:

Status:   Failed
Reason:   Evicted
Message:  The node was low on resource: memory. Container api was using 1843Mi, request is 512Mi, has larger consumption of memory.

How to tell it’s yours: the pod status says Evicted, other pods on the same node may fail at the same time, and kubectl describe node <node> shows the MemoryPressure condition.

The kubelet evicts pods whose usage is over their request first, ordered by priority and then by how far over they are. With hard thresholds, it uses a 0 second grace period. If memory drops too fast for the kubelet to act, the kernel’s OOM killer acts on the node. It kills the container with the highest score after the kubelet’s oom_score_adj: -997 for Guaranteed pods, 1000 for BestEffort, and a value in between for Burstable pods based on their request. Containers in low QoS pods that use a lot of memory compared with their requests go first.

Fix:

1. Set memory requests on every container so the scheduler doesn’t pack the node past its real capacity.
2. For critical workloads, set requests equal to limits on every container. That gives the Guaranteed QoS class, and the docs say those pods are never evicted because of another pod’s usage.
3. Reserve memory for the system with the kubelet’s --kube-reserved and --system-reserved flags, as the docs suggest.

Confirming the OOM kills have stopped

1. Watch the restart count:

   kubectl get pod <pod> -w

   It should stay flat through your busiest period.
2. Watch memory under load:

   kubectl top pod <pod> --containers

   Usage should level off with clear room below the limit. A line that keeps rising means the leak is still there.
3. Check the last state again. kubectl describe pod <pod> should show the same Last State timestamp as before your change, or none at all for a new pod.

Keeping OOMKilled from coming back

• Size from data, on a schedule. The Vertical Pod Autoscaler is installed separately. With updateMode: "Off" it only writes recommendations to the VPA object’s .status, so you can read them and change requests yourself:

  apiVersion: autoscaling.k8s.io/v1
  kind: VerticalPodAutoscaler
  metadata:
    name: api
  spec:
    targetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: api
    updatePolicy:
      updateMode: "Off"

  Then read them with kubectl describe vpa api.
• Alert before the kill. Alert when a container’s memory stays above a set share of its limit, and on any restart whose reason is OOMKilled.
• Tie heap flags to limits. When someone changes limits.memory, the Node or JVM heap setting should change in the same pull request.
• Test with production-sized data. Many leaks only show with real traffic and real payload sizes.

Outside Kubernetes the same kill happens in plain Docker. docker run -m sets the limit, with a minimum of 6m, and docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}}' <container> prints true 137 after a kill.

Finding the cause of an OOM kill with Polylane

Polylane connects a cluster through an agent you install with Helm, with read-only RBAC limited to get, list and watch. Its agents inspect pods, deployments and nodes and read logs and events, and OOM kills are among the failures it investigates and writes up, starting from the rollout that came before them.

Running on Kubernetes? See how Polylane monitors Kubernetes in production.