Hardcoded API Keys in Source Code: How to Find, Fix and Prevent Credential Leaks

During one of our recent architectural assessments, we were reviewing a client's application with a broader objective than just understanding the architecture.

As part of the assessment, we were also looking at some of the security and operational practices around the application. While going through the codebases, we came across something that immediately caught our attention: there were API keys and other credentials present directly in the source code.

Finding a key in a codebase may look like a small issue, especially if the application is working correctly. But from a security perspective, it can become a much bigger problem depending on what the key provides access to, where the code is stored and who can access the repository.

We highlighted the finding as part of our assessment.

The client later rotated the affected keys based on our recommendation.

This is a good example of why architectural assessments should not only focus on application structure and technology choices. We should also look for security practices that can create operational risks later.

In this article, we will look at why hardcoded credentials are a problem, how we can identify them, what should happen after finding one, and how we can prevent similar issues from reaching the codebase again.

Why Are Hardcoded API Keys a Problem?

An API key is effectively a credential.

Depending on the system, it may provide access to APIs, cloud services, databases, third-party platforms or other resources.

When the key is stored directly in source code, anyone who can access that code may potentially access the credential as well.

For example:

String API_KEY = "xxxxxxxxxxxxxxxx";

Or:

const apiKey = "xxxxxxxxxxxxxxxx";

The problem becomes even more serious when the repository is accessible to a larger engineering team, external contractors or third-party systems.

If the repository is public, the situation becomes significantly more serious because the credential may already be exposed to the internet.

Even when the repository is private, we shouldn't assume that the credential is safe forever.

Repositories get copied.

Code gets forked.

Developers download source code to their machines.

Backups are created.

CI/CD systems access repositories.

The more places the source code exists, the more difficult it becomes to control the exposure of a credential embedded inside it.

How Do We Find Hardcoded Credentials?

The first step is identifying whether credentials are present in the codebase.

A simple search can sometimes reveal obvious cases.

For example:

grep -Rni "api_key" .

We can search for other common patterns:

grep -Rni "apikey" .
grep -Rni "api-key" .
grep -Rni "password" .
grep -Rni "secret" .

However, simple text searches aren't enough.

Developers may use different variable names, encoded values or configuration formats.

For example:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
CLIENT_SECRET
PRIVATE_KEY
AUTH_TOKEN
DATABASE_PASSWORD
ACCESS_TOKEN

This is why dedicated secret-scanning tools are useful.

Tools such as Gitleaks, TruffleHog and other security scanners can look for patterns that resemble credentials and secrets.

We can also use Trivy for secret scanning as part of our security checks.

For example:

trivy fs --scanners secret .

The objective isn't to blindly treat every finding as a confirmed credential. The result needs to be reviewed and validated.

Finding a Secret Doesn't Mean the Investigation Is Finished

This is probably the most important part.

Suppose we find an API key in a source file.

Removing the line from the code doesn't solve the problem.

We first need to understand whether the credential was actually exposed and whether it is still active.

A useful investigation starts with questions such as:

• What system does the key provide access to?
• What permissions does the key have?
• Is the key still active?
• Who has access to the repository?
• Is the repository public or private?
• Was the key committed to Git?
• Does the key exist in Git history?
• Was the key copied into another repository?
• Is the key present in build artifacts or deployment packages?
• Could the key have been exposed through logs?

The answers determine the severity and the next steps.

Don't Just Delete the Key

One of the most common mistakes is simply removing the credential from the source file.

For example, changing:

const apiKey = "xxxxxxxxxxxxxxxx";

to:

const apiKey = "";

doesn't invalidate the original credential.

If someone already has the key, deleting it from the current version of the source code doesn't take away their access.

The credential itself needs to be revoked or rotated.

Rotate the Credential

Once we confirm that a credential has been exposed, the next step should generally be to rotate or revoke it.

The exact process depends on the system that issued the credential.

For example, if the key belongs to a third-party API, we should use that provider's credential management system to revoke the existing key and generate a new one.

The new credential should then be stored outside the source code.

In our assessment, this was the action taken by the client after we highlighted the finding. The affected keys were rotated based on our recommendation.

This is an important distinction between identifying a security issue and actually reducing the security risk.

What About Git History?

There is another important detail that is easy to miss.

Removing the credential from the latest version of the source code doesn't necessarily remove it from Git history.

Consider this sequence:

Commit 1
API key added

Commit 2
Application changes

Commit 3
API key removed

The key may no longer exist in the latest version of the code, but it may still exist in Commit 1.

Anyone with access to the repository history may still be able to retrieve it.

This is why we should treat an exposed credential as compromised even if the key has already been removed from the current source code.

Rotating the credential is therefore much more important than simply removing the text from the repository.

Store Secrets Outside the Source Code

Once we've removed the hardcoded credential, we need somewhere appropriate to store it.

The application should retrieve the secret from a secure configuration or secret-management system at runtime.

Depending on the environment, this could be:

• Cloud secret management services

• HashiCorp Vault

• Kubernetes Secrets

• CI/CD secret stores

• Environment-specific configuration systems

The important principle is that credentials shouldn't be part of the application source code.

For example, instead of:

const apiKey = "xxxxxxxxxxxxxxxx";

the application should retrieve the value from its runtime configuration.

const apiKey = process.env.API_KEY;

The exact implementation will depend on the application and deployment environment.

Be Careful With Environment Variables

Moving a secret from source code to an environment variable is an improvement, but environment variables aren't automatically a complete secrets-management solution.

We still need to consider:

• Who can view the environment?
• Where is the environment variable configured?
• Is it visible in CI/CD logs?
• Is it stored securely?
• Can developers retrieve it unnecessarily?
• Is it exposed through debugging or error messages?

The goal should be controlled access to secrets, not simply moving the secret from one location to another.

Add Secret Scanning to CI/CD

Finding a credential during an architectural assessment is useful.

Finding it automatically before the code is merged is even better.

We can introduce secret scanning into the CI/CD pipeline.

For example:

Developer
   ↓
Commit
   ↓
Pull Request
   ↓
Secret Scan
   ↓
Build
   ↓
Tests
   ↓
Security Scan
   ↓
Deploy

If a potential credential is detected, the pipeline can stop and require the issue to be reviewed.

This moves security closer to the developer and reduces the chance that credentials make their way into production repositories.

Scan More Than Just the Current Code

A common mistake is scanning only the current working directory.

For repositories with a long history, we should also consider Git history.

A secret may have been committed months ago and removed later.

Depending on the tool we're using, we can scan repository history to identify credentials that may have existed in previous commits.

This is especially important when performing a security or architectural assessment on an existing application.

Common Places Where Secrets Hide

During assessments, we shouldn't look only for obvious API key variables.

Secrets can appear in many places.

Some common examples include:

• Source code
• Configuration files
• .env files
• Dockerfiles
• Docker Compose files
• Kubernetes manifests
• CI/CD configuration
• Infrastructure as Code
• Scripts
• Documentation
• Test configuration
• Sample configuration files

Even documentation can accidentally contain a real credential if developers copy production configuration while creating examples.

What About Configuration Files?

Configuration files are another common place for credentials.

For example:

database:
  username: admin
  password: mypassword

This might not look like source code, but it can create exactly the same security problem.

Configuration should therefore be included in security scanning and code reviews.

Don't Ignore Test Credentials

Test environments also deserve attention.

Teams sometimes assume that test credentials are harmless because they don't provide access to production.

That isn't always true.

A test credential may still provide access to customer information, internal systems or paid third-party services.

We should understand what every credential can access rather than assuming that a credential is safe simply because it belongs to a non-production environment.

A Practical Response Process

When we discover a credential in a codebase, the following process provides a good starting point.

1. Identify the Credential

Determine what the credential belongs to and what system it can access.

2. Assess the Exposure

Check where the credential exists and who could potentially access it.

3. Check Whether It Is Active

An old or revoked credential may not create the same level of risk as an active credential.

4. Review Its Permissions

A read-only API key is different from a credential with administrative access.

5. Rotate or Revoke It

If the credential is active and exposed, rotate or revoke it as quickly as possible.

6. Remove It From the Code

Remove the credential from the current source code and configuration.

7. Review Git History

Determine whether the credential exists in previous commits.

8. Move the Secret to Proper Secret Management

Use the appropriate secret-management mechanism for the environment.

9. Add Preventive Controls

Introduce secret scanning into developer workflows and CI/CD.

This process helps us move from simply identifying a security finding to actually reducing the risk.

Common Mistakes

There are a few mistakes we should avoid.

Deleting the Credential and Moving On

Removing the credential from the latest source code doesn't invalidate it.

We need to rotate or revoke the credential.

Assuming Private Repositories Are Safe

Private repositories reduce exposure, but they don't eliminate it.

Access should still be controlled.

Storing Secrets in Configuration Files

Moving a password from source code into a committed configuration file doesn't solve the underlying problem.

The configuration file is still part of the codebase.

Putting Secrets in CI/CD Logs

Even when credentials aren't stored in source code, they can accidentally appear in build logs.

We should make sure sensitive variables are masked and never printed.

Giving Credentials More Permissions Than Necessary

If an application only needs read access to a service, the credential shouldn't have administrative privileges.

The principle of least privilege should apply to application credentials as well.

What We Should Check During an Architectural Assessment

When performing an architectural assessment, security shouldn't be limited to reviewing authentication and network diagrams.

A practical assessment should also include questions around:

• Where are application secrets stored?
• How are credentials managed?
• Who can access production secrets?
• Are secrets stored in source control?
• Is Git history scanned?
• Are secrets scanned during CI/CD?
• How are credentials rotated?
• What happens when a credential is compromised?
• Are application credentials granted excessive permissions?
• Are secrets exposed in logs or monitoring systems?

These questions can uncover risks that may not be visible from architecture diagrams alone.

Trivy in CI/CD: How to Add Vulnerability Scanning to Your Pipeline

Security testing shouldn't start after an application reaches production.

If we are already using CI/CD to build, test and deploy our applications, it makes sense to introduce security checks into the same process. This allows us to identify vulnerabilities before they become production problems.

One of the open-source tools we can use for this is Trivy.

Trivy can scan container images, source code repositories and filesystems for vulnerabilities. It can also identify configuration problems, secrets and other security-related issues.

In this article, we will see how we can introduce Trivy into a CI/CD pipeline, starting with a simple local scan and then moving towards using it as a security gate in our pipeline.

What Is Trivy?

Trivy is an open-source security scanner maintained by Aqua Security.

It is commonly used for scanning container images, but it can do much more than that. Depending on how we use it, Trivy can scan:

• Container images
• Filesystems
• Git repositories
• Infrastructure as Code
• Kubernetes configurations
• Dependencies
• Secrets
• Licenses

For this article, we will focus mainly on container image vulnerability scanning because it is one of the easiest ways to introduce security scanning into an existing CI/CD process.

Why Add Vulnerability Scanning to CI/CD?

Let's consider a typical application pipeline.

The application is compiled, tests are executed, a Docker image is created and the image is pushed to a container registry. Eventually, that image is deployed to Kubernetes or another production environment.

The problem is that the container image may contain vulnerable operating system packages or application dependencies.

If we only discover those vulnerabilities after deployment, fixing them becomes more complicated.

Instead, we can scan the image before it is pushed or deployed.

A simple pipeline can therefore look like this:

Code
  ↓
Build
  ↓
Unit Tests
  ↓
Build Docker Image
  ↓
Trivy Scan
  ↓
Push Image
  ↓
Deploy

If the vulnerability scan fails, the pipeline stops and the vulnerable image doesn't move further through the deployment process.

This is the basic idea behind adding security into CI/CD.

Step 1: Install Trivy

There are several ways to install Trivy depending on the operating system and environment.

For Ubuntu, we can install it using the official repository.

sudo apt-get install wget gnupg

Add the repository signing key:

wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | \
gpg --dearmor | \
sudo tee /usr/share/keyrings/trivy.gpg > /dev/null

Add the Trivy repository:

echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] \
https://aquasecurity.github.io/trivy-repo/deb \
generic main" | \
sudo tee /etc/apt/sources.list.d/trivy.list

Update the package list:

sudo apt-get update

Install Trivy:

sudo apt-get install trivy

Verify the installation:

trivy --version

We should see the installed Trivy version in the output.

The installation method may change over time, so it is worth checking the current Trivy documentation if we are setting this up on a new machine.

Step 2: Scan a Docker Image

Once Trivy is installed, we can immediately start scanning container images.

For example:

trivy image nginx:latest

Trivy will download the image if it isn't already available locally and scan its packages for known vulnerabilities.

The output will contain information such as:

Library        Vulnerability     Severity
openssl        CVE-XXXX-XXXXX    HIGH
curl           CVE-XXXX-XXXXX    MEDIUM
libxyz         CVE-XXXX-XXXXX    CRITICAL

The exact results will depend on the image version and the vulnerabilities known at the time of the scan.

This is already useful, but we probably don't want every vulnerability to stop our pipeline.

Step 3: Scan Only High and Critical Vulnerabilities

In a CI/CD environment, we usually need to decide which vulnerabilities should block a deployment.

We can filter the results by severity:

trivy image --severity HIGH,CRITICAL nginx:latest

This allows us to focus on vulnerabilities that require immediate attention.

However, filtering the displayed results alone doesn't necessarily make the pipeline fail. We need to explicitly configure the exit code.

Step 4: Make the Pipeline Fail

This is where Trivy becomes useful as a CI/CD security gate.

Consider this command:

trivy image \
  --severity HIGH,CRITICAL \
  --exit-code 1 \
  nginx:latest

The --exit-code 1 option tells Trivy to return a non-zero exit code when vulnerabilities matching the selected criteria are found.

CI/CD systems generally treat a non-zero exit code as a failed step.

So the behaviour becomes:

No HIGH/CRITICAL vulnerabilities
        ↓
Pipeline continues

HIGH/CRITICAL vulnerability found
        ↓
Trivy returns exit code 1
        ↓
Pipeline fails

This is the important difference between simply running a security scan and actually making security part of our deployment process.

Step 5: Scan Our Own Docker Image

Instead of scanning a public image, let's assume our pipeline builds an image called:

myapp:1.0.0

We can scan it using:

trivy image --severity HIGH,CRITICAL myapp:1.0.0

For CI/CD, we can use:

trivy image \
  --severity HIGH,CRITICAL \
  --exit-code 1 \
  myapp:1.0.0

If the scan passes, the pipeline can continue with the next stage.

If the scan finds a HIGH or CRITICAL vulnerability, the pipeline stops.

Step 6: Ignore Vulnerabilities That Don't Have a Fix

This is one area where we need to be careful.

A vulnerability may be known, but there may not yet be a fixed package available.

If we fail the pipeline for every vulnerability regardless of whether a fix exists, developers may quickly start treating the security pipeline as an obstacle rather than a useful control.

Trivy allows us to ignore vulnerabilities for which no fix is currently available.

trivy image \
  --severity HIGH,CRITICAL \
  --ignore-unfixed \
  --exit-code 1 \
  myapp:1.0.0

This means we focus the pipeline gate on vulnerabilities where a fix is available.

That doesn't mean unfixed vulnerabilities should be ignored forever. They should still be tracked and reviewed.

Step 7: Scanning the Source Code

Trivy isn't limited to container images.

We can also scan a project directory:

trivy fs .

We can focus on vulnerabilities:

trivy fs \
  --scanners vuln \
  .

We can also scan for secrets:

trivy fs \
  --scanners secret \
  .

This can help detect accidentally committed credentials, tokens and other sensitive information.

For example, a developer might accidentally commit a configuration file containing an API key.

A source scan gives us another opportunity to detect the problem before the code reaches production.

Step 8: Adding Trivy to a CI/CD Pipeline

Now we can bring everything together.

A simplified pipeline looks like this:

Checkout Code
      ↓
Build Application
      ↓
Run Tests
      ↓
Build Docker Image
      ↓
Trivy Vulnerability Scan
      ↓
Push Image
      ↓
Deploy

The important part is where we place the security scan.

We should scan the exact image that we are planning to deploy.

For example:

docker build -t myapp:$BUILD_ID .

Then:

trivy image \
  --severity HIGH,CRITICAL \
  --ignore-unfixed \
  --exit-code 1 \
  myapp:$BUILD_ID

If the scan passes:

docker push myapp:$BUILD_ID

The deployment can then use that exact image.

This gives us a simple security gate before the artifact moves to the next environment.

Step 9: Generate a Report

Sometimes we don't just want the pipeline to pass or fail. We also want a report that developers and security teams can review.

Trivy supports different output formats.

For example:

trivy image \
  --format json \
  --output trivy-report.json \
  myapp:$BUILD_ID

We can then publish the generated report as a CI/CD pipeline artifact.

This is useful because the pipeline result tells us that something failed, while the report tells us what actually needs to be fixed.

Step 10: Don't Make Every Vulnerability a Pipeline Failure

This is where security implementation requires some judgement.

If we configure the pipeline to fail for every LOW, MEDIUM, HIGH and CRITICAL vulnerability from day one, there is a good chance the pipeline will become difficult to use.

A better approach is to establish a security policy.

For example:

LOW       → Report
MEDIUM    → Report and Track
HIGH      → Review / Block
CRITICAL  → Block

The exact policy will depend on the application and organisation.

For internet-facing applications, we may choose a stricter policy. For internal applications, we may initially use a more gradual approach.

The important thing is to define the policy rather than letting every security finding become an emergency.

Managing Exceptions

There will be situations where a vulnerability needs to be accepted temporarily.

Trivy supports ignore files for this purpose.

For example:

.trivyignore

We can place vulnerability IDs in the file that we have deliberately reviewed and accepted.

However, this should be used carefully.

A common mistake is to keep adding vulnerabilities to .trivyignore simply because they are causing pipeline failures.

That defeats the purpose of having the security scan in the first place.

Every exception should have a reason, an owner and, ideally, an expiry or review date.

Common Mistakes

There are a few mistakes we should avoid when introducing Trivy into CI/CD.

Scanning Only in Production

If we scan only after deployment, we have already allowed the vulnerable artifact into our environment.

Security scanning is more useful when it happens before deployment.

Blocking Everything Immediately

Introducing a security gate without understanding the current vulnerability baseline can cause hundreds of existing issues to break the pipeline.

It is often better to establish a baseline first and then gradually increase the enforcement level.

Ignoring Unfixed Vulnerabilities Forever

Using --ignore-unfixed can make the pipeline more practical, but it shouldn't become an excuse to forget about those vulnerabilities.

The vulnerability may receive a fix later.

Ignoring the Docker Base Image

Many vulnerabilities come from the base image itself.

For example:

FROM ubuntu:latest

The application code may be perfectly secure while the underlying image contains vulnerable packages.

Keeping the base image updated is therefore an important part of container security.

Treating the Scan as the Final Security Check

Trivy is a valuable security tool, but it doesn't replace a complete security program.

Application security also includes:

• Secure coding
• Dependency management
• Secrets management
• Access control
• Network security
• Authentication
• Authorization
• Infrastructure security
• Runtime monitoring

Trivy should be one layer in the overall security process.

Where Should We Run the Scan?

There isn't one universal answer.

A practical approach is to scan at multiple stages.

For example:

Developer Machine
        ↓
Source / Dependency Scan
        ↓
CI Build
        ↓
Container Image Scan
        ↓
Container Registry
        ↓
Deployment
        ↓
Runtime Monitoring

The earlier we identify a problem, the cheaper it usually is to fix.

A vulnerability found during development is much easier to address than one discovered after a production deployment.

How to Set Up Let’s Encrypt SSL Certificate with Nginx

If we are hosting a web application on a Linux server, enabling HTTPS is one of the first things we should take care of before exposing the application to users.

One of the easiest ways to do this is by using Let’s Encrypt. It provides free SSL/TLS certificates, and with Certbot we can automate both certificate installation and renewal.

In this article, we will go step by step through the process of configuring a Let’s Encrypt certificate with Nginx. We will also look at certificate renewal and some common issues that can prevent the certificate from being generated or renewed successfully.

What We Are Going to Set Up

We will configure Nginx to serve our application over HTTPS using a certificate issued by Let’s Encrypt.

The setup will include:

• A domain pointing to our server

• Nginx configured as the web server or reverse proxy

• A Let’s Encrypt SSL certificate

• HTTPS access on port 443

• HTTP to HTTPS redirection

• Automatic certificate renewal

Prerequisites

For this guide, we will assume that:

• We have a Linux server.

• Ubuntu is being used as the operating system.

• Nginx is installed or can be installed.

• We have a domain name.

• We have access to the DNS configuration for the domain.

• We have sudo access to the server.

• Ports 80 and 443 are accessible from the internet.

For the examples below, we will use:

Domain: example.com
Server IP: 203.0.113.10

Replace these values with the actual domain and server details.

Step 1: Point the Domain to the Server

Before requesting an SSL certificate, our domain needs to resolve to the server where Nginx is running.

In the DNS configuration, create an A record.

Type: A
Name: @
Value: 203.0.113.10

If we also want to support www.example.com, we can create another record:

Type: A
Name: www
Value: 203.0.113.10

Depending on the DNS provider, the interface will look different, but the concept remains the same.

We should verify that DNS is resolving correctly before moving forward.

From the server or our local machine, run:

nslookup example.com

or:

dig example.com

The returned IP address should match our server's public IP.

If DNS is not resolving correctly, there is no point proceeding with Certbot yet. Let's Encrypt needs to verify that we control the domain.

Step 2: Install Nginx

If Nginx isn't already installed, we can install it using:

sudo apt update
sudo apt install nginx -y

Once the installation is complete, check the service:

sudo systemctl status nginx

We should see that the service is running.

We can also verify the configuration:

sudo nginx -t

A successful configuration check should return something similar to:

syntax is ok
test is successful

Now open the domain in a browser:

http://example.com

At this stage, we should get the Nginx default page or our application, depending on how Nginx has been configured.

Step 3: Configure Nginx for the Domain

Before requesting the certificate, we should configure a server block for our domain.

Create a configuration file:

sudo nano /etc/nginx/sites-available/example.com

Add the following:

server {
    listen 80;
    listen [::]:80;

    server_name example.com www.example.com;

    location / {
        proxy_pass http://127.0.0.1:8000;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The proxy_pass value should point to the application running behind Nginx.

For example, if our application is running on port 3000:

proxy_pass http://127.0.0.1:3000;

If the application is running on another server, we can use that server's address instead.

Step 4: Enable the Nginx Configuration

Create a symbolic link to enable the configuration:

sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/

Then test the configuration:

sudo nginx -t

If everything looks correct, reload Nginx:

sudo systemctl reload nginx

Now verify that the domain is accessible:

http://example.com

We should make sure the application works correctly over HTTP before moving to HTTPS.

Step 5: Install Certbot

Now we can install Certbot and the Nginx plugin.

sudo apt update
sudo apt install certbot python3-certbot-nginx -y

Verify the installation:

certbot --version

We should get the installed Certbot version as the output.

Certbot will communicate with Let's Encrypt, request the certificate and configure Nginx for HTTPS.

Step 6: Request the Let's Encrypt Certificate

Now we can request the certificate.

Run:

sudo certbot --nginx -d example.com -d www.example.com

Certbot will ask for some information, including an email address and agreement to the terms of service.

It will then communicate with Let's Encrypt and perform domain validation.

If the validation succeeds, Certbot will obtain the certificate and update the Nginx configuration.

One of the advantages of using the Nginx plugin is that we don't have to manually copy certificate paths into the Nginx configuration.

Step 7: Redirect HTTP to HTTPS

During the Certbot setup, we may be asked whether HTTP traffic should be redirected to HTTPS.

Choose the redirect option if we want all HTTP traffic to automatically use HTTPS.

After enabling the redirect, users visiting:

http://example.com

will automatically be redirected to:

https://example.com

This ensures that users always access the application through an encrypted connection.

Step 8: Verify the HTTPS Configuration

Open the following URL in a browser:

https://example.com

The browser should show the secure connection indicator.

We can also check the certificate from the command line:

openssl s_client -connect example.com:443 -servername example.com

This provides information about the certificate and TLS connection.

We can also check the Nginx configuration:

sudo nginx -t

If everything is configured correctly, Nginx should report:

syntax is ok
test is successful

Step 9: Check the Certificate

Certbot provides a convenient command to check the certificates currently installed on the server.

sudo certbot certificates

Example output:

Certificate Name: example.com
Domains: example.com www.example.com
Expiry Date: ...
Certificate Path: /etc/letsencrypt/live/example.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/example.com/privkey.pem

Certbot normally stores certificates under:

/etc/letsencrypt/

We generally shouldn't manually modify files inside this directory because Certbot manages the certificate lifecycle.

Step 10: Configure Automatic Renewal

Let's Encrypt certificates are short-lived and need to be renewed regularly.

The good part is that Certbot can handle renewal automatically.

First, check whether the Certbot renewal timer is active:

sudo systemctl status certbot.timer

We can also check the timers on the system:

systemctl list-timers | grep certbot

If the timer is configured correctly, Certbot will periodically check whether the certificate needs renewal.

Step 11: Test Certificate Renewal

We shouldn't wait until the certificate is about to expire to find out that automatic renewal doesn't work.

We can perform a dry run:

sudo certbot renew --dry-run

This performs a simulated renewal process without replacing the existing certificate.

A successful test should indicate that the renewal simulation completed successfully.

This is something worth testing after the initial setup and whenever we make significant changes to the Nginx or DNS configuration.

Common Problems

The installation itself is usually straightforward. Most problems occur during domain validation, Nginx configuration or certificate renewal.

Problem 1: Domain Doesn't Point to the Server

If Certbot reports that domain validation failed, the first thing we should check is DNS.

Run:

dig example.com

Make sure the returned IP address points to the correct server.

If DNS was changed recently, we may also need to wait for the DNS changes to propagate.

Problem 2: Port 80 Is Not Accessible

Let's Encrypt commonly needs to reach the server through HTTP during certificate validation.

Make sure port 80 is open.

On Ubuntu with UFW:

sudo ufw status

If required:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

If the server is running in a cloud environment, we also need to check the cloud firewall or security group.

Opening the port in UFW isn't enough if the cloud firewall is blocking the traffic.

Problem 3: Nginx Configuration Test Fails

If this command fails:

sudo nginx -t

we should fix the Nginx configuration before running Certbot.

The error message usually contains the configuration file and line number where the problem exists.

We can inspect the complete configuration using:

sudo nginx -T

This prints the complete Nginx configuration currently being loaded.

Problem 4: Too Many Redirects

After enabling HTTPS, we may sometimes encounter a redirect loop.

For example, the browser keeps switching between HTTP and HTTPS.

This usually happens when there is another reverse proxy or load balancer in front of Nginx and the original protocol isn't being handled correctly.

We should check:

• Nginx redirect rules

• Reverse proxy configuration

• Load balancer configuration

• X-Forwarded-Proto headers

We should avoid adding multiple layers of HTTPS redirects without understanding how traffic flows through the infrastructure.

Problem 5: Certificate Renewal Fails

If automatic renewal fails, first check the existing certificates:

sudo certbot certificates

Then test renewal manually:

sudo certbot renew --dry-run

Check the Certbot logs if the problem isn't obvious:

sudo ls /var/log/letsencrypt/

The logs usually provide enough information to identify whether the issue is DNS, Nginx, port accessibility or certificate validation.

Useful Commands

Here are some commands we can keep handy when troubleshooting a Let's Encrypt and Nginx setup.

Check Nginx status:

sudo systemctl status nginx

Test Nginx configuration:

sudo nginx -t

Reload Nginx:

sudo systemctl reload nginx

Check certificates:

sudo certbot certificates

Test renewal:

sudo certbot renew --dry-run

Check Certbot timer:

systemctl list-timers | grep certbot

Check HTTPS certificate:

openssl s_client -connect example.com:443 -servername example.com

Check DNS:

dig example.com

A Few Things We Should Keep in Mind

Getting the certificate is only one part of enabling HTTPS.

We should also make sure that:

• HTTP redirects to HTTPS.

• Port 443 is accessible.

• The certificate covers all required domains.

• Automatic renewal is working.

• Nginx configuration remains valid.

• Application URLs use HTTPS where required.

• Mixed-content issues are not introduced in the application.

HTTPS should be treated as part of the application's infrastructure rather than simply a certificate that we install once and forget about.