Hardcoded API Keys in Source Code: How to Find, Fix and Prevent Credential Leaks

During one of our recent architectural assessments, we were reviewing a client's application with a broader objective than just understanding the architecture.

As part of the assessment, we were also looking at some of the security and operational practices around the application. While going through the codebases, we came across something that immediately caught our attention: there were API keys and other credentials present directly in the source code.

Finding a key in a codebase may look like a small issue, especially if the application is working correctly. But from a security perspective, it can become a much bigger problem depending on what the key provides access to, where the code is stored and who can access the repository.

We highlighted the finding as part of our assessment.

The client later rotated the affected keys based on our recommendation.

This is a good example of why architectural assessments should not only focus on application structure and technology choices. We should also look for security practices that can create operational risks later.

In this article, we will look at why hardcoded credentials are a problem, how we can identify them, what should happen after finding one, and how we can prevent similar issues from reaching the codebase again.

Why Are Hardcoded API Keys a Problem?

An API key is effectively a credential.

Depending on the system, it may provide access to APIs, cloud services, databases, third-party platforms or other resources.

When the key is stored directly in source code, anyone who can access that code may potentially access the credential as well.

For example:

String API_KEY = "xxxxxxxxxxxxxxxx";

Or:

const apiKey = "xxxxxxxxxxxxxxxx";

The problem becomes even more serious when the repository is accessible to a larger engineering team, external contractors or third-party systems.

If the repository is public, the situation becomes significantly more serious because the credential may already be exposed to the internet.

Even when the repository is private, we shouldn't assume that the credential is safe forever.

Repositories get copied.

Code gets forked.

Developers download source code to their machines.

Backups are created.

CI/CD systems access repositories.

The more places the source code exists, the more difficult it becomes to control the exposure of a credential embedded inside it.

How Do We Find Hardcoded Credentials?

The first step is identifying whether credentials are present in the codebase.

A simple search can sometimes reveal obvious cases.

For example:

grep -Rni "api_key" .

We can search for other common patterns:

grep -Rni "apikey" .
grep -Rni "api-key" .
grep -Rni "password" .
grep -Rni "secret" .

However, simple text searches aren't enough.

Developers may use different variable names, encoded values or configuration formats.

For example:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
CLIENT_SECRET
PRIVATE_KEY
AUTH_TOKEN
DATABASE_PASSWORD
ACCESS_TOKEN

This is why dedicated secret-scanning tools are useful.

Tools such as Gitleaks, TruffleHog and other security scanners can look for patterns that resemble credentials and secrets.

We can also use Trivy for secret scanning as part of our security checks.

For example:

trivy fs --scanners secret .

The objective isn't to blindly treat every finding as a confirmed credential. The result needs to be reviewed and validated.

Finding a Secret Doesn't Mean the Investigation Is Finished

This is probably the most important part.

Suppose we find an API key in a source file.

Removing the line from the code doesn't solve the problem.

We first need to understand whether the credential was actually exposed and whether it is still active.

A useful investigation starts with questions such as:

• What system does the key provide access to?
• What permissions does the key have?
• Is the key still active?
• Who has access to the repository?
• Is the repository public or private?
• Was the key committed to Git?
• Does the key exist in Git history?
• Was the key copied into another repository?
• Is the key present in build artifacts or deployment packages?
• Could the key have been exposed through logs?

The answers determine the severity and the next steps.

Don't Just Delete the Key

One of the most common mistakes is simply removing the credential from the source file.

For example, changing:

const apiKey = "xxxxxxxxxxxxxxxx";

to:

const apiKey = "";

doesn't invalidate the original credential.

If someone already has the key, deleting it from the current version of the source code doesn't take away their access.

The credential itself needs to be revoked or rotated.

Rotate the Credential

Once we confirm that a credential has been exposed, the next step should generally be to rotate or revoke it.

The exact process depends on the system that issued the credential.

For example, if the key belongs to a third-party API, we should use that provider's credential management system to revoke the existing key and generate a new one.

The new credential should then be stored outside the source code.

In our assessment, this was the action taken by the client after we highlighted the finding. The affected keys were rotated based on our recommendation.

This is an important distinction between identifying a security issue and actually reducing the security risk.

What About Git History?

There is another important detail that is easy to miss.

Removing the credential from the latest version of the source code doesn't necessarily remove it from Git history.

Consider this sequence:

Commit 1
API key added

Commit 2
Application changes

Commit 3
API key removed

The key may no longer exist in the latest version of the code, but it may still exist in Commit 1.

Anyone with access to the repository history may still be able to retrieve it.

This is why we should treat an exposed credential as compromised even if the key has already been removed from the current source code.

Rotating the credential is therefore much more important than simply removing the text from the repository.

Store Secrets Outside the Source Code

Once we've removed the hardcoded credential, we need somewhere appropriate to store it.

The application should retrieve the secret from a secure configuration or secret-management system at runtime.

Depending on the environment, this could be:

• Cloud secret management services

• HashiCorp Vault

• Kubernetes Secrets

• CI/CD secret stores

• Environment-specific configuration systems

The important principle is that credentials shouldn't be part of the application source code.

For example, instead of:

const apiKey = "xxxxxxxxxxxxxxxx";

the application should retrieve the value from its runtime configuration.

const apiKey = process.env.API_KEY;

The exact implementation will depend on the application and deployment environment.

Be Careful With Environment Variables

Moving a secret from source code to an environment variable is an improvement, but environment variables aren't automatically a complete secrets-management solution.

We still need to consider:

• Who can view the environment?
• Where is the environment variable configured?
• Is it visible in CI/CD logs?
• Is it stored securely?
• Can developers retrieve it unnecessarily?
• Is it exposed through debugging or error messages?

The goal should be controlled access to secrets, not simply moving the secret from one location to another.

Add Secret Scanning to CI/CD

Finding a credential during an architectural assessment is useful.

Finding it automatically before the code is merged is even better.

We can introduce secret scanning into the CI/CD pipeline.

For example:

Developer
   ↓
Commit
   ↓
Pull Request
   ↓
Secret Scan
   ↓
Build
   ↓
Tests
   ↓
Security Scan
   ↓
Deploy

If a potential credential is detected, the pipeline can stop and require the issue to be reviewed.

This moves security closer to the developer and reduces the chance that credentials make their way into production repositories.

Scan More Than Just the Current Code

A common mistake is scanning only the current working directory.

For repositories with a long history, we should also consider Git history.

A secret may have been committed months ago and removed later.

Depending on the tool we're using, we can scan repository history to identify credentials that may have existed in previous commits.

This is especially important when performing a security or architectural assessment on an existing application.

Common Places Where Secrets Hide

During assessments, we shouldn't look only for obvious API key variables.

Secrets can appear in many places.

Some common examples include:

• Source code
• Configuration files
• .env files
• Dockerfiles
• Docker Compose files
• Kubernetes manifests
• CI/CD configuration
• Infrastructure as Code
• Scripts
• Documentation
• Test configuration
• Sample configuration files

Even documentation can accidentally contain a real credential if developers copy production configuration while creating examples.

What About Configuration Files?

Configuration files are another common place for credentials.

For example:

database:
  username: admin
  password: mypassword

This might not look like source code, but it can create exactly the same security problem.

Configuration should therefore be included in security scanning and code reviews.

Don't Ignore Test Credentials

Test environments also deserve attention.

Teams sometimes assume that test credentials are harmless because they don't provide access to production.

That isn't always true.

A test credential may still provide access to customer information, internal systems or paid third-party services.

We should understand what every credential can access rather than assuming that a credential is safe simply because it belongs to a non-production environment.

A Practical Response Process

When we discover a credential in a codebase, the following process provides a good starting point.

1. Identify the Credential

Determine what the credential belongs to and what system it can access.

2. Assess the Exposure

Check where the credential exists and who could potentially access it.

3. Check Whether It Is Active

An old or revoked credential may not create the same level of risk as an active credential.

4. Review Its Permissions

A read-only API key is different from a credential with administrative access.

5. Rotate or Revoke It

If the credential is active and exposed, rotate or revoke it as quickly as possible.

6. Remove It From the Code

Remove the credential from the current source code and configuration.

7. Review Git History

Determine whether the credential exists in previous commits.

8. Move the Secret to Proper Secret Management

Use the appropriate secret-management mechanism for the environment.

9. Add Preventive Controls

Introduce secret scanning into developer workflows and CI/CD.

This process helps us move from simply identifying a security finding to actually reducing the risk.

Common Mistakes

There are a few mistakes we should avoid.

Deleting the Credential and Moving On

Removing the credential from the latest source code doesn't invalidate it.

We need to rotate or revoke the credential.

Assuming Private Repositories Are Safe

Private repositories reduce exposure, but they don't eliminate it.

Access should still be controlled.

Storing Secrets in Configuration Files

Moving a password from source code into a committed configuration file doesn't solve the underlying problem.

The configuration file is still part of the codebase.

Putting Secrets in CI/CD Logs

Even when credentials aren't stored in source code, they can accidentally appear in build logs.

We should make sure sensitive variables are masked and never printed.

Giving Credentials More Permissions Than Necessary

If an application only needs read access to a service, the credential shouldn't have administrative privileges.

The principle of least privilege should apply to application credentials as well.

What We Should Check During an Architectural Assessment

When performing an architectural assessment, security shouldn't be limited to reviewing authentication and network diagrams.

A practical assessment should also include questions around:

• Where are application secrets stored?
• How are credentials managed?
• Who can access production secrets?
• Are secrets stored in source control?
• Is Git history scanned?
• Are secrets scanned during CI/CD?
• How are credentials rotated?
• What happens when a credential is compromised?
• Are application credentials granted excessive permissions?
• Are secrets exposed in logs or monitoring systems?

These questions can uncover risks that may not be visible from architecture diagrams alone.

Trivy in CI/CD: How to Add Vulnerability Scanning to Your Pipeline

Security testing shouldn't start after an application reaches production.

If we are already using CI/CD to build, test and deploy our applications, it makes sense to introduce security checks into the same process. This allows us to identify vulnerabilities before they become production problems.

One of the open-source tools we can use for this is Trivy.

Trivy can scan container images, source code repositories and filesystems for vulnerabilities. It can also identify configuration problems, secrets and other security-related issues.

In this article, we will see how we can introduce Trivy into a CI/CD pipeline, starting with a simple local scan and then moving towards using it as a security gate in our pipeline.

What Is Trivy?

Trivy is an open-source security scanner maintained by Aqua Security.

It is commonly used for scanning container images, but it can do much more than that. Depending on how we use it, Trivy can scan:

• Container images
• Filesystems
• Git repositories
• Infrastructure as Code
• Kubernetes configurations
• Dependencies
• Secrets
• Licenses

For this article, we will focus mainly on container image vulnerability scanning because it is one of the easiest ways to introduce security scanning into an existing CI/CD process.

Why Add Vulnerability Scanning to CI/CD?

Let's consider a typical application pipeline.

The application is compiled, tests are executed, a Docker image is created and the image is pushed to a container registry. Eventually, that image is deployed to Kubernetes or another production environment.

The problem is that the container image may contain vulnerable operating system packages or application dependencies.

If we only discover those vulnerabilities after deployment, fixing them becomes more complicated.

Instead, we can scan the image before it is pushed or deployed.

A simple pipeline can therefore look like this:

Code
  ↓
Build
  ↓
Unit Tests
  ↓
Build Docker Image
  ↓
Trivy Scan
  ↓
Push Image
  ↓
Deploy

If the vulnerability scan fails, the pipeline stops and the vulnerable image doesn't move further through the deployment process.

This is the basic idea behind adding security into CI/CD.

Step 1: Install Trivy

There are several ways to install Trivy depending on the operating system and environment.

For Ubuntu, we can install it using the official repository.

sudo apt-get install wget gnupg

Add the repository signing key:

wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | \
gpg --dearmor | \
sudo tee /usr/share/keyrings/trivy.gpg > /dev/null

Add the Trivy repository:

echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] \
https://aquasecurity.github.io/trivy-repo/deb \
generic main" | \
sudo tee /etc/apt/sources.list.d/trivy.list

Update the package list:

sudo apt-get update

Install Trivy:

sudo apt-get install trivy

Verify the installation:

trivy --version

We should see the installed Trivy version in the output.

The installation method may change over time, so it is worth checking the current Trivy documentation if we are setting this up on a new machine.

Step 2: Scan a Docker Image

Once Trivy is installed, we can immediately start scanning container images.

For example:

trivy image nginx:latest

Trivy will download the image if it isn't already available locally and scan its packages for known vulnerabilities.

The output will contain information such as:

Library        Vulnerability     Severity
openssl        CVE-XXXX-XXXXX    HIGH
curl           CVE-XXXX-XXXXX    MEDIUM
libxyz         CVE-XXXX-XXXXX    CRITICAL

The exact results will depend on the image version and the vulnerabilities known at the time of the scan.

This is already useful, but we probably don't want every vulnerability to stop our pipeline.

Step 3: Scan Only High and Critical Vulnerabilities

In a CI/CD environment, we usually need to decide which vulnerabilities should block a deployment.

We can filter the results by severity:

trivy image --severity HIGH,CRITICAL nginx:latest

This allows us to focus on vulnerabilities that require immediate attention.

However, filtering the displayed results alone doesn't necessarily make the pipeline fail. We need to explicitly configure the exit code.

Step 4: Make the Pipeline Fail

This is where Trivy becomes useful as a CI/CD security gate.

Consider this command:

trivy image \
  --severity HIGH,CRITICAL \
  --exit-code 1 \
  nginx:latest

The --exit-code 1 option tells Trivy to return a non-zero exit code when vulnerabilities matching the selected criteria are found.

CI/CD systems generally treat a non-zero exit code as a failed step.

So the behaviour becomes:

No HIGH/CRITICAL vulnerabilities
        ↓
Pipeline continues

HIGH/CRITICAL vulnerability found
        ↓
Trivy returns exit code 1
        ↓
Pipeline fails

This is the important difference between simply running a security scan and actually making security part of our deployment process.

Step 5: Scan Our Own Docker Image

Instead of scanning a public image, let's assume our pipeline builds an image called:

myapp:1.0.0

We can scan it using:

trivy image --severity HIGH,CRITICAL myapp:1.0.0

For CI/CD, we can use:

trivy image \
  --severity HIGH,CRITICAL \
  --exit-code 1 \
  myapp:1.0.0

If the scan passes, the pipeline can continue with the next stage.

If the scan finds a HIGH or CRITICAL vulnerability, the pipeline stops.

Step 6: Ignore Vulnerabilities That Don't Have a Fix

This is one area where we need to be careful.

A vulnerability may be known, but there may not yet be a fixed package available.

If we fail the pipeline for every vulnerability regardless of whether a fix exists, developers may quickly start treating the security pipeline as an obstacle rather than a useful control.

Trivy allows us to ignore vulnerabilities for which no fix is currently available.

trivy image \
  --severity HIGH,CRITICAL \
  --ignore-unfixed \
  --exit-code 1 \
  myapp:1.0.0

This means we focus the pipeline gate on vulnerabilities where a fix is available.

That doesn't mean unfixed vulnerabilities should be ignored forever. They should still be tracked and reviewed.

Step 7: Scanning the Source Code

Trivy isn't limited to container images.

We can also scan a project directory:

trivy fs .

We can focus on vulnerabilities:

trivy fs \
  --scanners vuln \
  .

We can also scan for secrets:

trivy fs \
  --scanners secret \
  .

This can help detect accidentally committed credentials, tokens and other sensitive information.

For example, a developer might accidentally commit a configuration file containing an API key.

A source scan gives us another opportunity to detect the problem before the code reaches production.

Step 8: Adding Trivy to a CI/CD Pipeline

Now we can bring everything together.

A simplified pipeline looks like this:

Checkout Code
      ↓
Build Application
      ↓
Run Tests
      ↓
Build Docker Image
      ↓
Trivy Vulnerability Scan
      ↓
Push Image
      ↓
Deploy

The important part is where we place the security scan.

We should scan the exact image that we are planning to deploy.

For example:

docker build -t myapp:$BUILD_ID .

Then:

trivy image \
  --severity HIGH,CRITICAL \
  --ignore-unfixed \
  --exit-code 1 \
  myapp:$BUILD_ID

If the scan passes:

docker push myapp:$BUILD_ID

The deployment can then use that exact image.

This gives us a simple security gate before the artifact moves to the next environment.

Step 9: Generate a Report

Sometimes we don't just want the pipeline to pass or fail. We also want a report that developers and security teams can review.

Trivy supports different output formats.

For example:

trivy image \
  --format json \
  --output trivy-report.json \
  myapp:$BUILD_ID

We can then publish the generated report as a CI/CD pipeline artifact.

This is useful because the pipeline result tells us that something failed, while the report tells us what actually needs to be fixed.

Step 10: Don't Make Every Vulnerability a Pipeline Failure

This is where security implementation requires some judgement.

If we configure the pipeline to fail for every LOW, MEDIUM, HIGH and CRITICAL vulnerability from day one, there is a good chance the pipeline will become difficult to use.

A better approach is to establish a security policy.

For example:

LOW       → Report
MEDIUM    → Report and Track
HIGH      → Review / Block
CRITICAL  → Block

The exact policy will depend on the application and organisation.

For internet-facing applications, we may choose a stricter policy. For internal applications, we may initially use a more gradual approach.

The important thing is to define the policy rather than letting every security finding become an emergency.

Managing Exceptions

There will be situations where a vulnerability needs to be accepted temporarily.

Trivy supports ignore files for this purpose.

For example:

.trivyignore

We can place vulnerability IDs in the file that we have deliberately reviewed and accepted.

However, this should be used carefully.

A common mistake is to keep adding vulnerabilities to .trivyignore simply because they are causing pipeline failures.

That defeats the purpose of having the security scan in the first place.

Every exception should have a reason, an owner and, ideally, an expiry or review date.

Common Mistakes

There are a few mistakes we should avoid when introducing Trivy into CI/CD.

Scanning Only in Production

If we scan only after deployment, we have already allowed the vulnerable artifact into our environment.

Security scanning is more useful when it happens before deployment.

Blocking Everything Immediately

Introducing a security gate without understanding the current vulnerability baseline can cause hundreds of existing issues to break the pipeline.

It is often better to establish a baseline first and then gradually increase the enforcement level.

Ignoring Unfixed Vulnerabilities Forever

Using --ignore-unfixed can make the pipeline more practical, but it shouldn't become an excuse to forget about those vulnerabilities.

The vulnerability may receive a fix later.

Ignoring the Docker Base Image

Many vulnerabilities come from the base image itself.

For example:

FROM ubuntu:latest

The application code may be perfectly secure while the underlying image contains vulnerable packages.

Keeping the base image updated is therefore an important part of container security.

Treating the Scan as the Final Security Check

Trivy is a valuable security tool, but it doesn't replace a complete security program.

Application security also includes:

• Secure coding
• Dependency management
• Secrets management
• Access control
• Network security
• Authentication
• Authorization
• Infrastructure security
• Runtime monitoring

Trivy should be one layer in the overall security process.

Where Should We Run the Scan?

There isn't one universal answer.

A practical approach is to scan at multiple stages.

For example:

Developer Machine
        ↓
Source / Dependency Scan
        ↓
CI Build
        ↓
Container Image Scan
        ↓
Container Registry
        ↓
Deployment
        ↓
Runtime Monitoring

The earlier we identify a problem, the cheaper it usually is to fix.

A vulnerability found during development is much easier to address than one discovered after a production deployment.

How to Set Up Let’s Encrypt SSL Certificate with Nginx

If we are hosting a web application on a Linux server, enabling HTTPS is one of the first things we should take care of before exposing the application to users.

One of the easiest ways to do this is by using Let’s Encrypt. It provides free SSL/TLS certificates, and with Certbot we can automate both certificate installation and renewal.

In this article, we will go step by step through the process of configuring a Let’s Encrypt certificate with Nginx. We will also look at certificate renewal and some common issues that can prevent the certificate from being generated or renewed successfully.

What We Are Going to Set Up

We will configure Nginx to serve our application over HTTPS using a certificate issued by Let’s Encrypt.

The setup will include:

• A domain pointing to our server

• Nginx configured as the web server or reverse proxy

• A Let’s Encrypt SSL certificate

• HTTPS access on port 443

• HTTP to HTTPS redirection

• Automatic certificate renewal

Prerequisites

For this guide, we will assume that:

• We have a Linux server.

• Ubuntu is being used as the operating system.

• Nginx is installed or can be installed.

• We have a domain name.

• We have access to the DNS configuration for the domain.

• We have sudo access to the server.

• Ports 80 and 443 are accessible from the internet.

For the examples below, we will use:

Domain: example.com
Server IP: 203.0.113.10

Replace these values with the actual domain and server details.

Step 1: Point the Domain to the Server

Before requesting an SSL certificate, our domain needs to resolve to the server where Nginx is running.

In the DNS configuration, create an A record.

Type: A
Name: @
Value: 203.0.113.10

If we also want to support www.example.com, we can create another record:

Type: A
Name: www
Value: 203.0.113.10

Depending on the DNS provider, the interface will look different, but the concept remains the same.

We should verify that DNS is resolving correctly before moving forward.

From the server or our local machine, run:

nslookup example.com

or:

dig example.com

The returned IP address should match our server's public IP.

If DNS is not resolving correctly, there is no point proceeding with Certbot yet. Let's Encrypt needs to verify that we control the domain.

Step 2: Install Nginx

If Nginx isn't already installed, we can install it using:

sudo apt update
sudo apt install nginx -y

Once the installation is complete, check the service:

sudo systemctl status nginx

We should see that the service is running.

We can also verify the configuration:

sudo nginx -t

A successful configuration check should return something similar to:

syntax is ok
test is successful

Now open the domain in a browser:

http://example.com

At this stage, we should get the Nginx default page or our application, depending on how Nginx has been configured.

Step 3: Configure Nginx for the Domain

Before requesting the certificate, we should configure a server block for our domain.

Create a configuration file:

sudo nano /etc/nginx/sites-available/example.com

Add the following:

server {
    listen 80;
    listen [::]:80;

    server_name example.com www.example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The proxy_pass value should point to the application running behind Nginx.

For example, if our application is running on port 3000:

proxy_pass http://127.0.0.1:3000;

If the application is running on another server, we can use that server's address instead.

Step 4: Enable the Nginx Configuration

Create a symbolic link to enable the configuration:

sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/

Then test the configuration:

sudo nginx -t

If everything looks correct, reload Nginx:

sudo systemctl reload nginx

Now verify that the domain is accessible:

http://example.com

We should make sure the application works correctly over HTTP before moving to HTTPS.

Step 5: Install Certbot

Now we can install Certbot and the Nginx plugin.

sudo apt update
sudo apt install certbot python3-certbot-nginx -y

Verify the installation:

certbot --version

We should get the installed Certbot version as the output.

Certbot will communicate with Let's Encrypt, request the certificate and configure Nginx for HTTPS.

Step 6: Request the Let's Encrypt Certificate

Now we can request the certificate.

Run:

sudo certbot --nginx -d example.com -d www.example.com

Certbot will ask for some information, including an email address and agreement to the terms of service.

It will then communicate with Let's Encrypt and perform domain validation.

If the validation succeeds, Certbot will obtain the certificate and update the Nginx configuration.

One of the advantages of using the Nginx plugin is that we don't have to manually copy certificate paths into the Nginx configuration.

Step 7: Redirect HTTP to HTTPS

During the Certbot setup, we may be asked whether HTTP traffic should be redirected to HTTPS.

Choose the redirect option if we want all HTTP traffic to automatically use HTTPS.

After enabling the redirect, users visiting:

http://example.com

will automatically be redirected to:

https://example.com

This ensures that users always access the application through an encrypted connection.

Step 8: Verify the HTTPS Configuration

Open the following URL in a browser:

https://example.com

The browser should show the secure connection indicator.

We can also check the certificate from the command line:

openssl s_client -connect example.com:443 -servername example.com

This provides information about the certificate and TLS connection.

We can also check the Nginx configuration:

sudo nginx -t

If everything is configured correctly, Nginx should report:

syntax is ok
test is successful

Step 9: Check the Certificate

Certbot provides a convenient command to check the certificates currently installed on the server.

sudo certbot certificates

Example output:

Certificate Name: example.com
Domains: example.com www.example.com
Expiry Date: ...
Certificate Path: /etc/letsencrypt/live/example.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/example.com/privkey.pem

Certbot normally stores certificates under:

/etc/letsencrypt/

We generally shouldn't manually modify files inside this directory because Certbot manages the certificate lifecycle.

Step 10: Configure Automatic Renewal

Let's Encrypt certificates are short-lived and need to be renewed regularly.

The good part is that Certbot can handle renewal automatically.

First, check whether the Certbot renewal timer is active:

sudo systemctl status certbot.timer

We can also check the timers on the system:

systemctl list-timers | grep certbot

If the timer is configured correctly, Certbot will periodically check whether the certificate needs renewal.

Step 11: Test Certificate Renewal

We shouldn't wait until the certificate is about to expire to find out that automatic renewal doesn't work.

We can perform a dry run:

sudo certbot renew --dry-run

This performs a simulated renewal process without replacing the existing certificate.

A successful test should indicate that the renewal simulation completed successfully.

This is something worth testing after the initial setup and whenever we make significant changes to the Nginx or DNS configuration.

Common Problems

The installation itself is usually straightforward. Most problems occur during domain validation, Nginx configuration or certificate renewal.

Problem 1: Domain Doesn't Point to the Server

If Certbot reports that domain validation failed, the first thing we should check is DNS.

Run:

dig example.com

Make sure the returned IP address points to the correct server.

If DNS was changed recently, we may also need to wait for the DNS changes to propagate.

Problem 2: Port 80 Is Not Accessible

Let's Encrypt commonly needs to reach the server through HTTP during certificate validation.

Make sure port 80 is open.

On Ubuntu with UFW:

sudo ufw status

If required:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

If the server is running in a cloud environment, we also need to check the cloud firewall or security group.

Opening the port in UFW isn't enough if the cloud firewall is blocking the traffic.

Problem 3: Nginx Configuration Test Fails

If this command fails:

sudo nginx -t

we should fix the Nginx configuration before running Certbot.

The error message usually contains the configuration file and line number where the problem exists.

We can inspect the complete configuration using:

sudo nginx -T

This prints the complete Nginx configuration currently being loaded.

Problem 4: Too Many Redirects

After enabling HTTPS, we may sometimes encounter a redirect loop.

For example, the browser keeps switching between HTTP and HTTPS.

This usually happens when there is another reverse proxy or load balancer in front of Nginx and the original protocol isn't being handled correctly.

We should check:

• Nginx redirect rules

• Reverse proxy configuration

• Load balancer configuration

• X-Forwarded-Proto headers

We should avoid adding multiple layers of HTTPS redirects without understanding how traffic flows through the infrastructure.

Problem 5: Certificate Renewal Fails

If automatic renewal fails, first check the existing certificates:

sudo certbot certificates

Then test renewal manually:

sudo certbot renew --dry-run

Check the Certbot logs if the problem isn't obvious:

sudo ls /var/log/letsencrypt/

The logs usually provide enough information to identify whether the issue is DNS, Nginx, port accessibility or certificate validation.

Useful Commands

Here are some commands we can keep handy when troubleshooting a Let's Encrypt and Nginx setup.

Check Nginx status:

sudo systemctl status nginx

Test Nginx configuration:

sudo nginx -t

Reload Nginx:

sudo systemctl reload nginx

Check certificates:

sudo certbot certificates

Test renewal:

sudo certbot renew --dry-run

Check Certbot timer:

systemctl list-timers | grep certbot

Check HTTPS certificate:

openssl s_client -connect example.com:443 -servername example.com

Check DNS:

dig example.com

A Few Things We Should Keep in Mind

Getting the certificate is only one part of enabling HTTPS.

We should also make sure that:

• HTTP redirects to HTTPS.

• Port 443 is accessible.

• The certificate covers all required domains.

• Automatic renewal is working.

• Nginx configuration remains valid.

• Application URLs use HTTPS where required.

• Mixed-content issues are not introduced in the application.

HTTPS should be treated as part of the application's infrastructure rather than simply a certificate that we install once and forget about.

How We Set Up Centralized Logging in Kubernetes with Grafana Loki

As Kubernetes environments grow, troubleshooting applications becomes increasingly difficult if we rely only on kubectl logs. While the command works well for individual pods, it becomes inefficient when applications are distributed across multiple namespaces, deployments and worker nodes.

In a production environment, pods are constantly being created, restarted and terminated. Once a pod is deleted, its logs are often lost unless they have been collected and stored somewhere centrally.

This is where a centralized logging solution becomes essential.

In this article, we will set up centralized logging in Kubernetes using Grafana, Loki and Promtail. We will also look at some of the common issues that teams encounter during installation and how to resolve them.

Why Do We Need Centralized Logging?

For a small cluster with only a few applications, checking logs using kubectl logs may be sufficient.

kubectl logs <pod-name>

However, as more applications are deployed, finding logs quickly becomes difficult.

Some common challenges include:

• Pods restart frequently.

• Multiple replicas generate separate logs.

• Applications run across different namespaces.

• Logs disappear when pods are recreated.

• Searching across multiple applications becomes time consuming.

A centralized logging solution solves these problems by collecting logs from every node and storing them in one place.

Why Grafana, Loki and Promtail?

There are several logging solutions available for Kubernetes, including the ELK Stack.

For our Kubernetes environments, we chose Grafana Loki because it is lightweight, easy to deploy and integrates seamlessly with Grafana.

The architecture is simple.

Application Pods
        │
        ▼
Promtail (DaemonSet)
        │
        ▼
      Loki
        │
        ▼
     Grafana

Each component has a specific responsibility.

• Promtail collects logs from every Kubernetes node.

• Loki stores the logs.

• Grafana provides a user interface to search and visualize them.

Prerequisites

Before starting the installation, make sure the following requirements are met.

• Kubernetes cluster is running.

• Helm is installed.

• StorageClass is available.

• You have cluster administrator access.

• A namespace exists for monitoring.

Create the namespace if it doesn't already exist.

kubectl create namespace monitoring

Step 1: Add the Grafana Helm Repository

First, add the official Helm repository.

helm repo add grafana https://grafana.github.io/helm-charts

helm repo update

This downloads the latest Helm charts for Grafana, Loki and Promtail.

Step 2: Install Loki Stack

Install the Loki Stack using Helm.

helm install loki grafana/loki-stack \
--namespace monitoring

Depending on the chart version, Helm deploys components such as:

• Loki

• Promtail

• Grafana (optional)

• Service Accounts

• ConfigMaps

Wait for the installation to complete before moving to the next step.

Step 3: Verify the Installation

Check whether all pods are running successfully.

kubectl get pods -n monitoring

A healthy deployment should look similar to this.

NAME                                   READY   STATUS
loki-0                                 1/1     Running
promtail-xxxxx                         1/1     Running
grafana-xxxxxxxx                       1/1     Running

If any pod is stuck in Pending or CrashLoopBackOff, investigate the issue before proceeding.

Step 4: Access Grafana

If an Ingress has not been configured yet, use port forwarding.

kubectl port-forward svc/grafana 3000:80 -n monitoring

Open the browser.

http://localhost:3000

Log in using the administrator credentials.

After logging in, navigate to:

Connections → Data Sources

Verify that Loki is configured as a data source.

Step 5: Verify Loki Connectivity

Before exploring logs, make sure Grafana can communicate with Loki.

Open the Loki data source and click "Save & Test".

If everything is configured correctly, Grafana displays a success message.

If the connection fails, verify:

• Loki service name

• Namespace

• Service port

• DNS resolution

• Network policies

Most connectivity issues are caused by an incorrect service URL.

Step 6: Explore Logs

Open the Explore page in Grafana.

Select the Loki data source.

Run a simple LogQL query.

{namespace="default"}

This displays logs for all pods in the default namespace.

To filter logs for a specific application:

{app="payment-api"}

Search for error messages.

{namespace="production"} |= "ERROR"

You can also filter by container name.

{container="nginx"}

LogQL makes searching Kubernetes logs much easier compared to manually checking individual pods.

Common Issues During Installation

Although the installation process is straightforward, there are a few issues that commonly appear in production environments.

Grafana Cannot Connect to Loki

One issue we encountered was Grafana failing to connect to Loki even though both pods were running.

The first thing to verify is the Loki service.

kubectl get svc -n monitoring

Confirm that the service name matches the URL configured in the Grafana data source.

Also verify that both services exist in the same namespace.

Most connectivity problems are caused by an incorrect service URL or namespace mismatch.

Promtail Pods Remain Pending

Promtail usually runs as a DaemonSet and schedules one pod on every worker node.

If Promtail remains in the Pending state, check the pod description.

kubectl describe pod <promtail-pod> -n monitoring

Common reasons include:

• Insufficient memory

• Taints

• Node selectors

• Missing tolerations

In our environment, Promtail couldn't be scheduled because the worker node didn't have enough available memory.

Increasing node capacity resolved the issue.

Loki Pod Keeps Restarting

If Loki repeatedly restarts, inspect the logs.

kubectl logs <loki-pod> -n monitoring

Typical causes include:

• Storage permission issues

• Persistent Volume problems

• Invalid Helm configuration

• Missing storage class

Checking the logs usually identifies the problem quickly.

No Logs Are Appearing

Sometimes every pod appears healthy, but Grafana still doesn't display any logs.

Check whether Promtail is collecting logs.

kubectl logs <promtail-pod> -n monitoring

Also verify:

• Promtail is running on every node.

• The correct namespace is selected.

• Labels match the LogQL query.

• Loki is receiving log entries.

Most "missing logs" issues are related to label mismatches rather than Loki itself.

Useful LogQL Queries

Display logs from a namespace.

{namespace="default"}

Display logs for a deployment.

{app="payment-api"}

Show only error messages.

{namespace="production"} |= "ERROR"

Show warning messages.

{namespace="production"} |= "WARN"

Search for a specific exception.

{namespace="production"} |= "NullPointerException"

These simple queries make troubleshooting much faster than manually checking logs from individual pods.

Best Practices

After deploying centralized logging, consider the following recommendations.

• Configure persistent storage for Loki.

• Set appropriate retention policies.

• Configure CPU and memory requests.

• Configure resource limits.

• Label workloads consistently.

• Secure Grafana with authentication.

• Create dashboards for frequently used queries.

• Configure alerts for critical application errors.

Following these practices keeps the logging platform reliable as the Kubernetes cluster grows.

Centralized logging is one of the first observability tools every Kubernetes environment should have. While kubectl logs is useful for quick debugging, it becomes increasingly difficult to troubleshoot distributed applications as the number of services grows.

Grafana, Loki and Promtail provide a lightweight and scalable logging solution that integrates naturally with Kubernetes. Once the platform is configured, developers can search logs across namespaces, deployments and containers from a single interface, making production troubleshooting much faster.

In our experience, most installation issues are related to configuration rather than the tools themselves. Taking a few extra minutes to verify connectivity, resource allocation and service configuration during setup saves a significant amount of troubleshooting later.

OOMKilled in Kubernetes: Understanding Exit Code 137 and How to Fix It

If you've worked with Kubernetes for some time, you've probably come across a pod that suddenly restarts with the reason OOMKilled and an exit code of 137. It usually happens without much warning. One moment the application is running normally, and the next moment Kubernetes terminates the container and starts a new one.

The first reaction is often to increase the memory limit and redeploy the application. Sometimes that works, but in many cases the same issue returns after a few hours or days because the actual problem was never investigated.

In most production environments, OOMKilled isn't the real problem. It's an indication that the application consumed more memory than it was allowed to use. Our objective shouldn't be to simply allocate more memory. Instead, we should understand why the application exceeded its limit in the first place.

In this article, we will understand what OOMKilled means, why Kubernetes reports Exit Code 137, how to investigate memory-related issues and the best practices to prevent them from happening again.

What Does OOMKilled Mean?

OOM stands for Out Of Memory.

Every container running in Kubernetes has resource limits that define the maximum amount of memory it can consume. If the application crosses that limit, the Linux kernel immediately terminates the process to protect the node from running out of memory.

Kubernetes detects that the container has stopped unexpectedly and restarts it according to the pod's restart policy.

If the application continues exceeding the memory limit after every restart, the pod may eventually enter a CrashLoopBackOff state.

Understanding this behaviour is important because Kubernetes isn't killing the application. The operating system is. Kubernetes simply reports what happened and starts a new container.

Understanding Exit Code 137

When a Linux process exits, it returns an exit code.

For an OOMKilled container, Kubernetes usually reports something similar to the following:

Last State:
  Terminated

Reason:
  OOMKilled

Exit Code:
  137

Exit Code 137 indicates that the process was terminated using the SIGKILL signal.

In Kubernetes, this almost always means one of the following:

• The application exceeded its configured memory limit.

• The Linux Out Of Memory Killer terminated the process.

• Kubernetes restarted the container after it exited.

Whenever we see Exit Code 137, memory usage should be the first thing we investigate.

Confirm That the Pod Was OOMKilled

The easiest way to verify the reason is by describing the pod.

kubectl describe pod <pod-name>

Look for the Last State section.

Last State:
  Terminated

Reason:
  OOMKilled

Exit Code:
  137

If both the reason and exit code match the output above, we've confirmed that memory exhaustion caused the container to terminate.

Before making any configuration changes, we should understand why it happened.

Review the Resource Configuration

The next step is checking the memory requests and limits configured for the container.

Run:

kubectl get pod <pod-name> -o yaml

Locate the resources section.

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"

  limits:
    memory: "1Gi"
    cpu: "500m"

There is often confusion between requests and limits.

A memory request determines the amount of memory Kubernetes reserves for the container during scheduling.

A memory limit defines the maximum memory the application is allowed to consume.

Once the application crosses that limit, the Linux kernel terminates the process immediately.

Setting memory limits too low is one of the most common reasons for OOMKilled pods.

Check Current Memory Usage

Before increasing memory limits, we should understand how much memory the application is actually consuming.

If Metrics Server is installed, run:

kubectl top pod

or

kubectl top pod <pod-name>

Example:

NAME              CPU(cores)   MEMORY(bytes)
payment-api       210m         985Mi

If the container has a memory limit of 1Gi and is already consuming around 985Mi, even a small increase in workload may cause the application to exceed its limit.

Monitoring memory usage gives us a much clearer picture than simply guessing.

Check the Previous Logs

When Kubernetes restarts a container, the current logs may only contain startup information.

The actual problem usually exists in the previous container.

Run:

kubectl logs <pod-name> --previous

Depending on the application, we may find messages related to:

• Memory allocation failures

• Large file uploads

• Garbage collection warnings

• Cache growth

• Unexpected spikes in workload

Checking the previous logs has helped us identify the root cause of many production incidents.

Common Reasons for OOMKilled

Although every application behaves differently, most OOMKilled incidents fall into a few common categories.

Memory Leaks

Applications that continuously allocate memory without releasing it eventually consume all available memory.

Initially everything appears normal.

After running for several hours or days, memory usage gradually increases until the container reaches its configured limit.

Monitoring memory growth over time usually helps identify this pattern.

Processing Large Files

Applications processing PDFs, images, videos or large Excel files often require much more memory than expected.

If the entire file is loaded into memory, temporary spikes can exceed the configured limit.

Whenever possible, processing files in smaller chunks significantly reduces memory consumption.

Loading Large Datasets

Another common mistake is loading an entire dataset into memory before processing it.

Instead of loading thousands of records at once, processing them in batches or using streaming techniques keeps memory usage under control.

Traffic Spikes

Applications that perform well under normal traffic may consume considerably more memory during peak usage.

Higher request volumes often result in additional objects being created, more active database connections and larger caches.

Without sufficient memory planning, the container eventually exceeds its limit.

Incorrect Resource Configuration

Sometimes the application itself isn't the problem.

The configured memory limit simply doesn't reflect the application's actual workload.

Development and testing environments often use much smaller datasets than production, making it difficult to estimate realistic memory requirements.

Should We Simply Increase the Memory Limit?

Increasing the memory limit may stop the restarts temporarily, but it shouldn't be the first solution.

Before changing resource limits, it's worth asking a few questions.

• Did the issue start after a recent deployment?

• Has application traffic increased?

• Are larger files being processed?

• Has a new feature introduced additional memory usage?

• Is memory usage continuously increasing over time?

Answering these questions often leads us to the actual root cause instead of masking the problem.

Review Recent Deployments

If the application was working correctly yesterday but started failing after a deployment, compare the recent changes.

Run:

kubectl rollout history deployment <deployment-name>

Recent code changes may have introduced:

• Larger in-memory caches

• Additional background workers

• New libraries

• Bigger response payloads

• Changes in data processing

If production is impacted, rolling back to the previous deployment can restore service while the investigation continues.

Best Practices to Prevent OOMKilled

Preventing memory issues is much easier than troubleshooting them during an outage.

Some practices that have worked well across production environments include:

• Configure realistic memory requests and limits.

• Continuously monitor CPU and memory usage.

• Process large files in batches.

• Stream large datasets whenever possible.

• Optimise application caching.

• Load test applications before production releases.

• Review memory consumption after every major deployment.

• Configure Horizontal Pod Autoscaler where appropriate.

Small improvements in memory management often make a significant difference to application stability.

Common Mistakes

These are some of the mistakes we see most frequently while investigating OOMKilled incidents.

• Increasing memory limits without identifying the root cause.

• Ignoring historical memory usage.

• Deploying applications without load testing.

• Assuming Kubernetes is responsible for application crashes.

• Forgetting to review previous container logs.

• Using the same resource configuration for every environment.

Avoiding these mistakes usually makes troubleshooting much faster.

OOMKilled is one of the most common Kubernetes issues, but it's also one of the easiest to diagnose once we understand what Exit Code 137 represents.

Rather than treating OOMKilled as the actual problem, we should treat it as an indication that the application consumed more memory than its configured limit.

A structured troubleshooting process always produces better results than making configuration changes based on assumptions. Confirm the reason using kubectl describe, review the configured resource limits, analyse memory usage, inspect the previous logs and compare recent deployments before increasing memory.

In many cases, Kubernetes has already provided everything we need to identify the root cause. We simply need to collect the information in the right order and let the evidence guide our investigation.

Why Your Kubernetes Pod Keeps Restarting: A Complete Debugging Guide

If you've been working with Kubernetes for any length of time, you've probably encountered a pod that refuses to stay running. One moment it's starting successfully, and the next moment it's restarting. After a few attempts, Kubernetes reports a CrashLoopBackOff status, leaving many engineers wondering what went wrong.

The first reaction is often to delete the pod and hope Kubernetes creates a healthy replacement. While this may appear to solve the problem temporarily, it rarely fixes the actual issue. More importantly, deleting the pod too quickly can remove valuable information that would have helped identify the root cause.

The good news is that Kubernetes almost always tells us why a pod is restarting. We simply need to know where to look and follow a structured troubleshooting approach instead of making assumptions.

In this article, we will walk through the exact process we can use to diagnose and resolve Kubernetes pods that keep restarting.

Step 1: Check the Pod Status

Whenever a pod starts restarting, our first objective is to understand what Kubernetes already knows about it.

The first command we should run is:

kubectl get pods

Example output:

NAME                    READY   STATUS             RESTARTS   AGE
payment-api-65d8fd      0/1     CrashLoopBackOff   12         18m

Pay close attention to these three columns:

• STATUS

• RESTARTS

• AGE

A restart count of 10 or 20 immediately tells us the application has been crashing repeatedly and Kubernetes has been trying to recover it.

Some common pod statuses include:

Status	Meaning
Running	Application is healthy
Pending	Waiting to be scheduled
CrashLoopBackOff	Container keeps crashing after startup
ImagePullBackOff	Unable to pull the container image
ErrImagePull	Image download failed
Completed	Job completed successfully
Terminating	Pod is shutting down

Once we've confirmed the pod is restarting, the next step is to gather more information.

Step 2: Describe the Pod

One of the most useful commands in Kubernetes troubleshooting is:

kubectl describe pod <pod-name>

Many engineers immediately jump to application logs, but kubectl describe provides far more context.

It includes information such as:

• Current container state

• Previous container state

• Restart count

• Mounted volumes

• Environment variables

• Probe failures

• Scheduling information

• Events

The Events section at the bottom deserves special attention because Kubernetes records everything significant that happened to the pod.

For example:

Warning  Unhealthy
Readiness probe failed

Warning  BackOff
Back-off restarting failed container

In many situations, the Events section already points us in the right direction before we even inspect the application logs.

Step 3: Read the Application Logs

Once we've reviewed the pod details, it's time to inspect the application logs.

kubectl logs <pod-name>

If the pod has already restarted, don't forget to check the logs from the previous container instance:

kubectl logs <pod-name> --previous

This is one of the most overlooked commands in Kubernetes.

When a container crashes and restarts, the current logs may only show startup messages. The actual exception often exists only in the previous container's logs. We've solved many production issues simply by checking kubectl logs --previous.

Now that we've gathered the basic information, we can start identifying the actual reason behind the restarts.

Common Causes of Restarting Pods

In most production environments, restarting pods usually fall into one of the following categories.

CrashLoopBackOff

This is probably the most common Kubernetes error.

A CrashLoopBackOff doesn't tell us what failed—it simply tells us Kubernetes keeps restarting the container because it exits shortly after starting.

Common reasons include:

• Application exceptions

• Missing environment variables

• Database connection failures

• Invalid configuration

• Missing ConfigMaps

• Missing Secrets

• Startup script failures

For example:

Error: Database connection refused

The application exits.

Kubernetes restarts it.

The application crashes again.

Eventually Kubernetes delays each restart attempt and reports CrashLoopBackOff.

It's important to remember that CrashLoopBackOff is a symptom, not the root cause.

The real reason is almost always available in the application logs.

OOMKilled

Another common reason for pod restarts is an out-of-memory condition.

If the application consumes more memory than allowed, the Linux kernel terminates the container.

We can verify this using:

kubectl describe pod <pod-name>

Look for something similar to:

Last State:
Terminated

Reason:
OOMKilled

Typical causes include:

• Memory leaks

• Processing large files

• Image or PDF conversion

• Loading large datasets into memory

• Incorrect resource limits

Many teams immediately increase the memory limit and redeploy the application.

While that may stop the restarts temporarily, it's always worth understanding why the application is consuming excessive memory before increasing resources.

Readiness Probe Failures

A readiness probe tells Kubernetes whether the application is ready to receive traffic.

Example:

readinessProbe:
  httpGet:
    path: /health
    port: 8080

If the readiness probe fails:

• The pod continues running.

• Kubernetes stops routing traffic to it.

Common reasons include:

• Wrong endpoint

• Wrong port

• Slow application startup

• Database not yet available

• Dependent services still starting

Readiness failures usually indicate that the application isn't fully initialised yet.

Liveness Probe Failures

Unlike readiness probes, liveness probes determine whether the application is still healthy.

If a liveness probe keeps failing, Kubernetes assumes the application is unhealthy and restarts it automatically.

Example:

livenessProbe:
  httpGet:
    path: /health
    port: 8080

One common mistake is configuring the liveness probe too aggressively.

For example, if an application requires 60 seconds to initialise but the liveness probe starts after only 15 seconds, Kubernetes may repeatedly kill the container before it has a chance to finish starting.

ImagePullBackOff

Sometimes the container never starts because Kubernetes cannot download the container image.

Possible reasons include:

• Incorrect image name

• Wrong image tag

• Private registry authentication failure

• Registry connectivity issues

• Image does not exist

The quickest way to investigate is:

kubectl describe pod <pod-name>

Again, the Events section usually contains the exact reason for the image pull failure.

Configuration Problems

Configuration errors are another common cause of restarting pods.

Examples include:

• Missing ConfigMaps

• Missing Secrets

• Incorrect environment variables

• Invalid file paths

• Missing certificates

• Invalid application configuration

Even a simple typo in an environment variable can prevent an application from starting successfully.

Step 4: Check Resource Usage

Sometimes the problem isn't the application itself.

The Kubernetes node may be under heavy resource pressure.

We can check resource usage using:

kubectl top pod

kubectl top node

These commands help identify:

• High CPU usage

• High memory usage

• Resource spikes

• Node pressure

If these commands don't work, ensure the Kubernetes Metrics Server is installed.

Step 5: Review Cluster Events

Cluster events often provide additional context that application logs cannot.

Run:

kubectl get events --sort-by=.metadata.creationTimestamp

Look for messages related to:

• Failed scheduling

• Failed mounts

• Failed image pulls

• Probe failures

• Resource exhaustion

Events frequently tell the complete story of what happened before the pod entered its restart cycle.

Step 6: Review Recent Deployments

If the application was working yesterday but started restarting after a deployment, don't ignore the possibility that the latest release introduced the issue.

Review the deployment history:

kubectl rollout history deployment payment-api

If required, roll back to the previous version:

kubectl rollout undo deployment payment-api

Rolling back isn't the final solution, but it can restore service quickly while we continue investigating the root cause.

A Simple Troubleshooting Checklist

Whenever we encounter a restarting pod, following a consistent process saves time and avoids unnecessary guesswork.

1. Check the pod status.

2. Describe the pod.

3. Review the current logs.

4. Review the previous logs.

5. Check the Events section.

6. Verify CPU and memory usage.

7. Inspect readiness and liveness probes.

8. Check for image pull errors.

9. Review recent deployments.

Following the same sequence every time helps us diagnose problems much faster.

Common Mistakes

Over the years, we've seen the same mistakes repeated in many Kubernetes environments.

• Deleting the pod before collecting logs.

• Ignoring the Events section.

• Increasing memory without understanding the root cause.

• Restarting deployments repeatedly instead of investigating.

• Forgetting to use kubectl logs --previous.

• Assuming Kubernetes is the problem when the application itself is crashing.

Avoiding these mistakes can significantly reduce troubleshooting time.

Kubernetes doesn't restart containers without a reason. Every restart is a symptom of an underlying issue, whether it's an application crash, resource exhaustion, configuration error, failed health check or infrastructure problem.

The goal isn't to memorise dozens of Kubernetes commands. Instead, we should develop a structured troubleshooting approach that helps us identify the root cause quickly and consistently.

The next time you see a pod stuck in CrashLoopBackOff, resist the temptation to delete it immediately. Start by checking the pod status, describing the pod, reviewing the logs and inspecting the Events section. In most cases, Kubernetes has already provided enough information to lead us to the solution.