If you've worked with Kubernetes for some time, you've probably come across a pod that suddenly restarts with the reason OOMKilled and an exit code of 137. It usually happens without much warning. One moment the application is running normally, and the next moment Kubernetes terminates the container and starts a new one.
The first reaction is often to increase the memory limit and redeploy the application. Sometimes that works, but in many cases the same issue returns after a few hours or days because the actual problem was never investigated.
In most production environments, OOMKilled isn't the real problem. It's an indication that the application consumed more memory than it was allowed to use. Our objective shouldn't be to simply allocate more memory. Instead, we should understand why the application exceeded its limit in the first place.
In this article, we will understand what OOMKilled means, why Kubernetes reports Exit Code 137, how to investigate memory-related issues and the best practices to prevent them from happening again.
What Does OOMKilled Mean?
OOM stands for Out Of Memory.
Every container running in Kubernetes has resource limits that define the maximum amount of memory it can consume. If the application crosses that limit, the Linux kernel immediately terminates the process to protect the node from running out of memory.
Kubernetes detects that the container has stopped unexpectedly and restarts it according to the pod's restart policy.
If the application continues exceeding the memory limit after every restart, the pod may eventually enter a CrashLoopBackOff state.
Understanding this behaviour is important because Kubernetes isn't killing the application. The operating system is. Kubernetes simply reports what happened and starts a new container.
Understanding Exit Code 137
When a Linux process exits, it returns an exit code.
For an OOMKilled container, Kubernetes usually reports something similar to the following:
Last State:
Terminated
Reason:
OOMKilled
Exit Code:
137
Exit Code 137 indicates that the process was terminated using the SIGKILL signal.
In Kubernetes, this almost always means one of the following:
The application exceeded its configured memory limit.
The Linux Out Of Memory Killer terminated the process.
Kubernetes restarted the container after it exited.
Whenever we see Exit Code 137, memory usage should be the first thing we investigate.
Confirm That the Pod Was OOMKilled
The easiest way to verify the reason is by describing the pod.
kubectl describe pod <pod-name>
Look for the Last State section.
Last State:
Terminated
Reason:
OOMKilled
Exit Code:
137
If both the reason and exit code match the output above, we've confirmed that memory exhaustion caused the container to terminate.
Before making any configuration changes, we should understand why it happened.
Review the Resource Configuration
The next step is checking the memory requests and limits configured for the container.
Run:
kubectl get pod <pod-name> -o yaml
Locate the resources section.
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
There is often confusion between requests and limits.
A memory request determines the amount of memory Kubernetes reserves for the container during scheduling.
A memory limit defines the maximum memory the application is allowed to consume.
Once the application crosses that limit, the Linux kernel terminates the process immediately.
Setting memory limits too low is one of the most common reasons for OOMKilled pods.
Check Current Memory Usage
Before increasing memory limits, we should understand how much memory the application is actually consuming.
If Metrics Server is installed, run:
kubectl top pod
or
kubectl top pod <pod-name>
Example:
NAME CPU(cores) MEMORY(bytes)
payment-api 210m 985Mi
If the container has a memory limit of 1Gi and is already consuming around 985Mi, even a small increase in workload may cause the application to exceed its limit.
Monitoring memory usage gives us a much clearer picture than simply guessing.
Check the Previous Logs
When Kubernetes restarts a container, the current logs may only contain startup information.
The actual problem usually exists in the previous container.
Run:
kubectl logs <pod-name> --previous
Depending on the application, we may find messages related to:
Memory allocation failures
Large file uploads
Garbage collection warnings
Cache growth
Unexpected spikes in workload
Checking the previous logs has helped us identify the root cause of many production incidents.
Common Reasons for OOMKilled
Although every application behaves differently, most OOMKilled incidents fall into a few common categories.
Memory Leaks
Applications that continuously allocate memory without releasing it eventually consume all available memory.
Initially everything appears normal.
After running for several hours or days, memory usage gradually increases until the container reaches its configured limit.
Monitoring memory growth over time usually helps identify this pattern.
Processing Large Files
Applications processing PDFs, images, videos or large Excel files often require much more memory than expected.
If the entire file is loaded into memory, temporary spikes can exceed the configured limit.
Whenever possible, processing files in smaller chunks significantly reduces memory consumption.
Loading Large Datasets
Another common mistake is loading an entire dataset into memory before processing it.
Instead of loading thousands of records at once, processing them in batches or using streaming techniques keeps memory usage under control.
Traffic Spikes
Applications that perform well under normal traffic may consume considerably more memory during peak usage.
Higher request volumes often result in additional objects being created, more active database connections and larger caches.
Without sufficient memory planning, the container eventually exceeds its limit.
Incorrect Resource Configuration
Sometimes the application itself isn't the problem.
The configured memory limit simply doesn't reflect the application's actual workload.
Development and testing environments often use much smaller datasets than production, making it difficult to estimate realistic memory requirements.
Should We Simply Increase the Memory Limit?
Increasing the memory limit may stop the restarts temporarily, but it shouldn't be the first solution.
Before changing resource limits, it's worth asking a few questions.
Did the issue start after a recent deployment?
Has application traffic increased?
Are larger files being processed?
Has a new feature introduced additional memory usage?
Is memory usage continuously increasing over time?
Answering these questions often leads us to the actual root cause instead of masking the problem.
Review Recent Deployments
If the application was working correctly yesterday but started failing after a deployment, compare the recent changes.
Run:
kubectl rollout history deployment <deployment-name>
Recent code changes may have introduced:
Larger in-memory caches
Additional background workers
New libraries
Bigger response payloads
Changes in data processing
If production is impacted, rolling back to the previous deployment can restore service while the investigation continues.
Best Practices to Prevent OOMKilled
Preventing memory issues is much easier than troubleshooting them during an outage.
Some practices that have worked well across production environments include:
Configure realistic memory requests and limits.
Continuously monitor CPU and memory usage.
Process large files in batches.
Stream large datasets whenever possible.
Optimise application caching.
Load test applications before production releases.
Review memory consumption after every major deployment.
Configure Horizontal Pod Autoscaler where appropriate.
Small improvements in memory management often make a significant difference to application stability.
Common Mistakes
These are some of the mistakes we see most frequently while investigating OOMKilled incidents.
Increasing memory limits without identifying the root cause.
Ignoring historical memory usage.
Deploying applications without load testing.
Assuming Kubernetes is responsible for application crashes.
Forgetting to review previous container logs.
Using the same resource configuration for every environment.
Avoiding these mistakes usually makes troubleshooting much faster.
OOMKilled is one of the most common Kubernetes issues, but it's also one of the easiest to diagnose once we understand what Exit Code 137 represents.
Rather than treating OOMKilled as the actual problem, we should treat it as an indication that the application consumed more memory than its configured limit.
A structured troubleshooting process always produces better results than making configuration changes based on assumptions. Confirm the reason using kubectl describe, review the configured resource limits, analyse memory usage, inspect the previous logs and compare recent deployments before increasing memory.
In many cases, Kubernetes has already provided everything we need to identify the root cause. We simply need to collect the information in the right order and let the evidence guide our investigation.
