Visualizing Log Data with Grafana, Loki, and Promtail

What is Loki and Promtail?

Loki: Loki is a log aggregation system developed by Grafana Labs. It allows you to store and query your application and infrastructure logs in a scalable and cost-effective way.

Promtail: Promtail is an agent developed by Grafana Labs to ship logs to Loki. It performs the following functions:

• Discovers targets (e.g. Pods in Kubernetes) that generate logs

• Labels the log streams with metadata

• Ships the logs to a Loki instance

Promtail can collect logs from:

• Local log files

• The systemd journal (on Linux)

• The Docker logging driver

• Syslog

Promtail uses a configuration file to specify:

• Which log files to tail

• Labels to attach to log streams

• The Loki instance to send logs to

Some of the benefits of using Loki and Promtail together are:

• Logs are centrally collected and stored in Loki

• Logs are indexed and can be queried using LogQL

• Logs can be visualized in Grafana dashboards

• Logs can be stored for long periods of time

• Logs can be collected from multiple sources and locations

Task: Create a dashboard using Grafana with the integrations of Loki and Promtail.

Here's the blog to install Grafana in your EC2 instance.

We'll install Loki and Promtail using Docker, let's install Docker first.

sudo apt-get update
sudo apt install docker.io
sudo usermod -aG docker $USER
sudo reboot

Download Loki Config:

Use the following command to download the Loki configuration file:

mkdir grafana_configs
cd grafana_configs
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

Download Promtail Config

Download the Promtail configuration file using the command below in grafana_configs directory:

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail

Run Loki Docker container using the below command.

docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml

Edit the inbound rule in the security group of the ec2 instance to allow port 3100.

Copy public-ip of instance and paste in browser on https:<public-ip>:3100/ready and check loki is ready..?

You can also see the metrics which means the logs which is the sole purpose of loki to collect use /metrics

Run the Promtail Docker container using the below command and check with docker ps to see the container status.

 sudo docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml

Add Data source in Grafana

• Now, navigate to the Grafana webapp and on the homepage choose the add data source option.

• Provide the HTTP URL as below to connect the loki data source to Grafana so that loki will send the logs to grafana.

• localhost:3100

Checking logs in Loki:

• Click on explore in the below screenshot after adding the data source.

• In the label filters, we can choose job and varlogs which is generally the path /var/log/*log in the backend to show all the system logs.

• Click on the run query in the above screenshot to execute and show all the system logs as below.

Now we have to Create a Dashboard:

• Let’s add the log to the dashboard by choosing the option from the above screenshot location.

• Now the system logs are added to the grafana dashboard. Let’s add some more by clicking on visualization to add some graphs as shown below screenshot.

• In Label filters choose job and varlogs and line contains to error to show all the lines with error and select the duration to show all the lines with error in the logs.

• Similarly, let’s check the error lines in grafana log that is placed in /var/log/grafana/grafana.log

• To accomplish the objective of displaying the Grafana log, we must specify the Grafana log path in the promtail config YAML file within the target section, as illustrated below.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
  - targets:
      - localhost
    labels:
      job: grafanalogs
      __path__: /var/log/grafana/*log

• After edit promtail_config.yaml file we have to restart our promtail docker container

• We can now choose the label filters to set the job and grafana logs with the line contains and visualization option to view in a graphical manner. We can add this to our dashboard.

• Install nginx

• Use the proper label filters to show an aggregate sum of words repeating nginx while installing. This can be achieved by setting the varlogs as label filters.

• We can see now the complete grafana dashboard.

Thank you for enjoying my DevOps blog! Your positive response fuels my passion to dive deeper into technology and innovation.

Stay tuned for more captivating DevOps articles, where we'll explore this dynamic field together. Follow me on Hashnode and connect on LinkedIn (https://www.linkedin.com/in/som-shanker-pandey/) for the latest updates and discussions.