As Kubernetes environments grow, troubleshooting applications becomes increasingly difficult if we rely only on kubectl logs. While the command works well for individual pods, it becomes inefficient when applications are distributed across multiple namespaces, deployments and worker nodes.
In a production environment, pods are constantly being created, restarted and terminated. Once a pod is deleted, its logs are often lost unless they have been collected and stored somewhere centrally.
This is where a centralized logging solution becomes essential.
In this article, we will set up centralized logging in Kubernetes using Grafana, Loki and Promtail. We will also look at some of the common issues that teams encounter during installation and how to resolve them.
Why Do We Need Centralized Logging?
For a small cluster with only a few applications, checking logs using kubectl logs may be sufficient.
kubectl logs <pod-name>
However, as more applications are deployed, finding logs quickly becomes difficult.
Some common challenges include:
Pods restart frequently.
Multiple replicas generate separate logs.
Applications run across different namespaces.
Logs disappear when pods are recreated.
Searching across multiple applications becomes time consuming.
A centralized logging solution solves these problems by collecting logs from every node and storing them in one place.
Why Grafana, Loki and Promtail?
There are several logging solutions available for Kubernetes, including the ELK Stack.
For our Kubernetes environments, we chose Grafana Loki because it is lightweight, easy to deploy and integrates seamlessly with Grafana.
The architecture is simple.
Application Pods
│
▼
Promtail (DaemonSet)
│
▼
Loki
│
▼
Grafana
Each component has a specific responsibility.
Promtail collects logs from every Kubernetes node.
Loki stores the logs.
Grafana provides a user interface to search and visualize them.
Prerequisites
Before starting the installation, make sure the following requirements are met.
Kubernetes cluster is running.
Helm is installed.
StorageClass is available.
You have cluster administrator access.
A namespace exists for monitoring.
Create the namespace if it doesn't already exist.
kubectl create namespace monitoring
Step 1: Add the Grafana Helm Repository
First, add the official Helm repository.
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
This downloads the latest Helm charts for Grafana, Loki and Promtail.
Step 2: Install Loki Stack
Install the Loki Stack using Helm.
helm install loki grafana/loki-stack \
--namespace monitoring
Depending on the chart version, Helm deploys components such as:
Loki
Promtail
Grafana (optional)
Service Accounts
ConfigMaps
Wait for the installation to complete before moving to the next step.
Step 3: Verify the Installation
Check whether all pods are running successfully.
kubectl get pods -n monitoring
A healthy deployment should look similar to this.
NAME READY STATUS
loki-0 1/1 Running
promtail-xxxxx 1/1 Running
grafana-xxxxxxxx 1/1 Running
If any pod is stuck in Pending or CrashLoopBackOff, investigate the issue before proceeding.
Step 4: Access Grafana
If an Ingress has not been configured yet, use port forwarding.
kubectl port-forward svc/grafana 3000:80 -n monitoring
Open the browser.
http://localhost:3000
Log in using the administrator credentials.
After logging in, navigate to:
Connections → Data Sources
Verify that Loki is configured as a data source.
Step 5: Verify Loki Connectivity
Before exploring logs, make sure Grafana can communicate with Loki.
Open the Loki data source and click "Save & Test".
If everything is configured correctly, Grafana displays a success message.
If the connection fails, verify:
Loki service name
Namespace
Service port
DNS resolution
Network policies
Most connectivity issues are caused by an incorrect service URL.
Step 6: Explore Logs
Open the Explore page in Grafana.
Select the Loki data source.
Run a simple LogQL query.
{namespace="default"}
This displays logs for all pods in the default namespace.
To filter logs for a specific application:
{app="payment-api"}
Search for error messages.
{namespace="production"} |= "ERROR"
You can also filter by container name.
{container="nginx"}
LogQL makes searching Kubernetes logs much easier compared to manually checking individual pods.
Common Issues During Installation
Although the installation process is straightforward, there are a few issues that commonly appear in production environments.
Grafana Cannot Connect to Loki
One issue we encountered was Grafana failing to connect to Loki even though both pods were running.
The first thing to verify is the Loki service.
kubectl get svc -n monitoring
Confirm that the service name matches the URL configured in the Grafana data source.
Also verify that both services exist in the same namespace.
Most connectivity problems are caused by an incorrect service URL or namespace mismatch.
Promtail Pods Remain Pending
Promtail usually runs as a DaemonSet and schedules one pod on every worker node.
If Promtail remains in the Pending state, check the pod description.
kubectl describe pod <promtail-pod> -n monitoring
Common reasons include:
Insufficient memory
Taints
Node selectors
Missing tolerations
In our environment, Promtail couldn't be scheduled because the worker node didn't have enough available memory.
Increasing node capacity resolved the issue.
Loki Pod Keeps Restarting
If Loki repeatedly restarts, inspect the logs.
kubectl logs <loki-pod> -n monitoring
Typical causes include:
Storage permission issues
Persistent Volume problems
Invalid Helm configuration
Missing storage class
Checking the logs usually identifies the problem quickly.
No Logs Are Appearing
Sometimes every pod appears healthy, but Grafana still doesn't display any logs.
Check whether Promtail is collecting logs.
kubectl logs <promtail-pod> -n monitoring
Also verify:
Promtail is running on every node.
The correct namespace is selected.
Labels match the LogQL query.
Loki is receiving log entries.
Most "missing logs" issues are related to label mismatches rather than Loki itself.
Useful LogQL Queries
Display logs from a namespace.
{namespace="default"}
Display logs for a deployment.
{app="payment-api"}
Show only error messages.
{namespace="production"} |= "ERROR"
Show warning messages.
{namespace="production"} |= "WARN"
Search for a specific exception.
{namespace="production"} |= "NullPointerException"
These simple queries make troubleshooting much faster than manually checking logs from individual pods.
Best Practices
After deploying centralized logging, consider the following recommendations.
Configure persistent storage for Loki.
Set appropriate retention policies.
Configure CPU and memory requests.
Configure resource limits.
Label workloads consistently.
Secure Grafana with authentication.
Create dashboards for frequently used queries.
Configure alerts for critical application errors.
Following these practices keeps the logging platform reliable as the Kubernetes cluster grows.
Centralized logging is one of the first observability tools every Kubernetes environment should have. While kubectl logs is useful for quick debugging, it becomes increasingly difficult to troubleshoot distributed applications as the number of services grows.
Grafana, Loki and Promtail provide a lightweight and scalable logging solution that integrates naturally with Kubernetes. Once the platform is configured, developers can search logs across namespaces, deployments and containers from a single interface, making production troubleshooting much faster.
In our experience, most installation issues are related to configuration rather than the tools themselves. Taking a few extra minutes to verify connectivity, resource allocation and service configuration during setup saves a significant amount of troubleshooting later.
No comments:
Post a Comment