Monitoring Microservices (Spring Boot + Micrometer + Prometheus + Grafana)

In this article, we will see how to use Micrometer, Actuator, Prometheus, and Grafana to collect, monitor, and alert about microservices' metrics.

Micrometer

Micrometer is a dimensional-first metrics collection facade that allows the application to time, count, and gauge its code with a vendor-neutral API.

It is the metrics collection facility included in Spring Boot 2's actuator.

Micrometer integration with Spring Boot enables a developer to choose one or more monitoring systems to use today, and change his mind later as the need changes without requiring a rewrite of the custom metrics instrumentation.

Spring Boot 2 autoconfigures quite a few metrics out of the box: JVM metrics (memory, GC statistics, thread utilization, classes), CPU usage, request latencies, uptime, Tomcat usage, HikariCP pool metrics, etc.

It provides a vendor-neutral metrics collection API and implementations for a variety of monitoring systems: Datadog, New Relic, JMX, Prometheus, Elasticsearch, CloudWatch, etc.

Spring Boot 2 configures a composite "MeterRegistry" to which any number of registry implementations can be added, allowing us to ship application metrics to more than one monitoring system.

The simplest form of the registry is "SimpleMeterRegistry," but "CompositeMeterRegistry" allows multiple registries to be added.

All a developer needs to do is include a vendor-specific micrometer dependency in the application.

Prometheus

Prometheus is a popular open-source time-series database for collecting and storing time-series data such as performance metrics.

Prometheus uses a pull-based approach for getting metrics. The application doesn't need to know where Prometheus is located or even be concerned if Prometheus has died.

Grafana

Grafana is a multi-platform, open-source analytics and interactive visualisation web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

Actuator

Actuator is used to expose operational information about the production-ready application—health, metrics, info, dump, env, etc. It uses HTTP endpoints or JMX beans to enable us to interact with it.

Code Changes

Let's create two microservices, "order-service" and "catalog-service," where "order-service" calls a REST API exposed by "catalog-service."

1) Include Micrometre and Actuator

1.1) Dependencies

The "micrometer-registry-prometheus" dependency allows the metrics collected by Micrometer to be exposed in a Prometheus way.

        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

1.2) Config

The "management.endpoints.web.exposure.include" property tells Spring Boot's Actuator which endpoints it should expose.

management.endpoints.web.exposure.include=health,info,prometheus

Now a number of metrics will be exposed automatically on the actuator endpoint: http://localhost:8081/actuator/prometheus.

2) Add a custom metric

Although the setup so far provides quite a few metrics out of the box, we can also add our own custom metrics.

Micrometer can publish different types of metrics: gauge, histogram, and timer.

Let's add a custom metric to "catalog-service." It will calculate how long the "getProduct(...)" methods take to execute and report that as a metric. We will be using the Micrometer "Timer" metric type.

2.1) Dependencies

In order to add a custom timer, we need to add the "spring-boot-starter-aop" dependency to the project.

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>

2.2) Code changes

If you register the "TimedAspect" bean in the spring context, it will allow Micrometer to add a timer to custom methods.

Next, we need to add the @Timed annotation to the method we want to time.

2.3) Test

In order to check if the custom metric is populating in the actuator response, let's hit the "catalog-service" API, i.e., http://localhost:8081/product/101, and check the actuator endpoint at http://localhost:8081/actuator/prometheus.

Pull metrics in Prometheus

One of the easiest ways to start Prometheus for testing is to run it as a Docker container: https://hub.docker.com/r/prom/prometheus.

1) Configure

Let's add rules in a "prometheus.yml" to scrape metrics from "order-service" (http://localhost:8080/actuator/prometheus) and "catalog-service" (http://localhost:8081/actuator/prometheus) every 5 seconds.

We have used "host.docker.internal" in place of "localhost" so that the "Prometheus" instance can connect to the Docker host from inside a Docker container.

2) Run

Next, we need to run Prometheus and pass this configuration to it.

docker run \
    -p 9090:9090 \
    -v ~/Desktop/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus

Let's observe the metrics in Prometheus by pointing to: http://localhost:9090 > Status > Targets.

Go to the "Graph" tab and search for the metric "process_cpu_usage":

Visualize in a Grafana Dashboard

Although Prometheus has a decent UI, Grafana's dashboards are more powerful.

One of the easiest ways to start Grafana for testing is to run it as a Docker container: https://hub.docker.com/r/grafana/grafana.

docker run -d --name=grafana -p 3000:3000 grafana/grafana

Let's point to Grafana: http://localhost:3000/

Use the default admin user credentials as admin/admin.

1) Configure

Let's add Prometheus as a data source: http://host.docker.internal:9090/

Click "Save" and check at the end of the screen; you should see a message that the data source has been connected.

2) Import a Predefined Dashboard

Since we have a Spring Boot app, we can take advantage of one of the pre-configured dashboards for Spring Boot.

Go to Dashboards > New Dashboard > Import.

We can either search for the dashboard or just enter the template id into the Import field (put the id as 12900) and click "load."

Choose Prometheus as the data source and click Import.

3) Visualize

Great! we can now have a complete Spring Boot dashboard:

Source code: GitHub