Combining Grafana + Prometheus + ELK for Spring Boot Observability

Effective observability in Spring Boot microservices demands a robust setup for tracking metrics, monitoring logs, and visualizing data. A single tool often cannot provide full clarity, which makes integrating Grafana + Prometheus + ELK (Elasticsearch, Logstash, Kibana) a winning combination for a complete observability stack.

This guide dives into how you can leverage Prometheus for metrics collection, ELK for log management, and Grafana to create unified panels that track both. We’ll cover setting up metrics and logs, correlating latency with error logs, and creating dashboards that give a holistic view of service health.

Table of Contents

1. Why Combine Grafana, Prometheus, and ELK?
2. Using Prometheus for Metrics and ELK for Logs
3. Creating Grafana Panels for Logs and Metrics
4. Correlating Latency Spikes with Error Logs
5. Building Unified Service Dashboards
6. Official Documentation Links
7. Summary

Why Combine Grafana, Prometheus, and ELK?

Each tool in this observability stack serves a unique purpose, and combining them creates a comprehensive system for monitoring, troubleshooting, and improving your Spring Boot applications.

• Prometheus:
  • A time-series database designed to collect and query metrics like request rates, memory usage, or CPU performance.
  • Works seamlessly with Spring Boot Actuator to expose metrics.
• ELK Stack:
  • Manages application logs for real-time analysis.
  • Elasticsearch indexes and searches log data.
  • Logstash processes and enriches logs.
  • Kibana creates dashboards for log exploration.
• Grafana:
  • Centralizes visualization of both metrics and logs.
  • Enables correlation of key activities like system latencies and error trends.
  • Provides actionable insights through unified dashboards.

With Prometheus and ELK handling different data types (metrics vs. logs), Grafana brings it all together with powerful visualization capabilities.

Using Prometheus for Metrics and ELK for Logs

Setting Up Prometheus for Metrics

Prometheus collects and stores metrics exposed from your Spring Boot applications.

Step 1. Add Prometheus Dependency in Spring Boot

Include the Spring Boot Actuator and Micrometer Prometheus Registry dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Step 2. Configure Prometheus in application.properties

Enable Actuator’s Prometheus support and expose /actuator/prometheus:

management.endpoints.web.exposure.include=prometheus
management.endpoint.prometheus.enabled=true

Step 3. Run Prometheus and Scrape Metrics

Download and run Prometheus:

docker run -p 9090:9090 prom/prometheus

Edit the prometheus.yml configuration file to scrape your application metrics:

scrape_configs:
  - job_name: 'spring-boot-app'
    static_configs:
      - targets: ['localhost:8080']

Access Prometheus GUI:

http://localhost:9090

Setting Up ELK for Logs

The ELK stack is perfect for capturing, searching, and visualizing Spring Boot logs.

Step 1. Add Logback Appender for Logstash

Add the logstash-logback-encoder dependency:

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>7.3</version>
</dependency>

Configure logback-spring.xml to send logs to Logstash:

<configuration>
    <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>localhost:5044</destination>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>

    <root level="INFO">
        <appender-ref ref="LOGSTASH" />
    </root>
</configuration>

Step 2. Set Up the ELK Stack in Docker

Create a docker-compose.yml file to run Elasticsearch, Logstash, and Kibana:

version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.5.0
    ports:
      - "9200:9200"

  logstash:
    image: docker.elastic.co/logstash/logstash:8.5.0
    ports:
      - "5044:5044"
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf

  kibana:
    image: docker.elastic.co/kibana/kibana:8.5.0
    ports:
      - "5601:5601"

Define logstash.conf to process incoming logs:

input {
  tcp {
    port => 5044
    codec => json
  }
}
output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "spring-logs-%{+yyyy.MM.dd}"
  }
}

Start the stack:

docker-compose up -d

Test the Setup

1. Verify Prometheus scraping on http://localhost:9090.
2. Check Kibana logs under http://localhost:5601.

Creating Grafana Panels for Logs and Metrics

Grafana connects to both Prometheus and Elasticsearch, letting you build dashboards that visualize metrics and logs side by side.

Step 1. Add Prometheus and Elasticsearch Data Sources

1. Navigate to Settings > Data Sources in Grafana.
2. Add a Prometheus data source:
   • URL: http://localhost:9090
3. Add an Elasticsearch data source:
   • URL: http://localhost:9200
   • Index pattern: spring-logs-*
   • Time field: @timestamp

Step 2. Create Panels for Metrics

1. Create a Time Series panel for Prometheus metrics:
   • Query example: http_server_requests_seconds_count
   • Group by status code or endpoint.
2. Use Grafana’s visualization tools to add legends, colors, or annotations for better insight.

Step 3. Create Panels for Logs

1. Create a Logs panel using Elasticsearch.
2. Filter logs for a specific service_name or level: service_name:"order-service" AND level:"ERROR"

Combine metrics and logs on one dashboard for a holistic view.

Correlating Latency Spikes with Error Logs

Discovering the relationship between performance degradation and errors is critical for debugging microservices.

Step 1. Monitor Latency Metrics

Query latency from Prometheus:

http_server_requests_seconds_max{endpoint="/api/orders"}

This tracks maximum response times for the /api/orders endpoint.

Step 2. Filter Error Logs in Elasticsearch

Filter logs for errors during the same time period:

service_name:"order-service" AND level:"ERROR"

Step 3. Overlay Logs and Metrics

Use Grafana to overlay logs and metrics:

• Add a Time Series for latency (Prometheus).
• Add a Logs panel (Elasticsearch) to view errors side by side.

This correlation pinpoints issues like specific requests triggering errors.

Building Unified Service Dashboards

Combining metrics and logs allows you to create unified dashboards tracking overall service health.

Include Panels for:

1. Error Count: Track error trends from Elasticsearch logs.
2. Latency Trends: Visualize latency from Prometheus.
3. Request Volume: Query Prometheus for request rates.
4. Log Insights: Display raw logs or filter critical logs for visibility.

Enable Refresh Intervals

Set dashboards to refresh every 5s or 10s for real-time monitoring.

Share Dashboards

Export dashboards to share with your team or integrate them with alerting systems like Slack.

Official Documentation Links

1. Prometheus Documentation
2. ELK Stack Documentation
3. Grafana Documentation

Summary

Combining Grafana, Prometheus, and ELK provides unparalleled observability for Spring Boot applications. With metrics from Prometheus, logs from ELK, and visualizations from Grafana, you achieve complete visibility into system health and performance.

Key Takeaways:

1. Prometheus for Metrics: Monitor request rates, latencies, and CPU usage.
2. ELK for Logs: Centralize and analyze logs for debugging.
3. Grafana Visualization: Build dashboards unifying metrics and logs.
4. Correlations: Quickly correlate latency spikes with error occurrences.

Implement this stack today to transform how your team monitors and maintains Spring Boot microservices!