How to Build a Logging Pipeline with OpenTelemetry, Grafana Loki, and Grafana in Docker Compose

“`html

How to Build a Logging Pipeline with OpenTelemetry, Grafana Loki, and Grafana in Docker Compose

Building a logging pipeline is essential for monitoring applications and systems in real-time. In this post, we will guide you through creating a logging pipeline using OpenTelemetry, Grafana Loki, and Grafana, all orchestrated using Docker Compose. This method allows for easy deployment and management of your logging infrastructure.

Prerequisites

Before proceeding, ensure you have the following:

• Docker and Docker Compose installed on your machine.
• Basic knowledge of Docker and YAML configuration files.

Step 1: Setting Up Your Docker Compose Configuration

First, create a directory for your project and a `docker-compose.yml` file within it. This file will define how to run your services.

version: '3.8'

services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - loki

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config:/etc/loki

  opentelemetry:
    image: otel/opentelemetry-collector:latest
    ports:
      - "4317:4317"
    volumes:
      - ./otel-config.yaml:/etc/otel-config.yaml
    command:
      -config=/etc/otel-config.yaml

networks:
  default:
    driver: bridge

Step 2: Loki Configuration

Create a directory named loki-config and add a file local-config.yaml inside it. This file will configure Loki’s options.

auth_enabled: false

server:
  http_listen_port: 3100

frontend:
  compress_response: true

ingester:
  wal:
    enabled: true
    dir: /loki/wal

schema_config:
  configs:
    - from: 2020-10-10
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    shared_store: filesystem

Step 3: OpenTelemetry Configuration

Next, create the otel-config.yaml file to set up the OpenTelemetry Collector:

receivers:

  otlp:

    protocols:

      grpc:

        endpoint: "0.0.0.0:4317"

exporters:

  loki:

    endpoint: "http://loki:3100/loki/api