Profiling a Spring Boot application with Pyroscope

Profiling of a software application is a type of dynamic analysis to measures its behavior while it's running. This process involves gathering metrics such as execution time, memory consumption, and function calls to pinpoint performance bottlenecks, memory leaks, and other inefficiencies.

The objective of profiling is to identify which sections of a program need optimization, whether it's to enhance its speed or reduce its memory usage.

In Pyroscope, profiling encompasses various types that target specific dimensions of application performance analysis, such as CPU usage, memory allocation, or thread synchronization.

For Java, Pyroscope supports the following profiling types:

1) CPU Profiling

CPU profiling measures the amount of CPU time consumed by various parts of your application code. You may get similar insights as with metrics, but with profiling, you gain more detailed information about the specific cause of a spike in CPU usage at the code level.

In the Pyroscope Flame graph, the width of blocks represents the CPU time consumed by each function.

2) Memory allocation Profiling (Alloc Objects & Alloc Space)

Memory allocation profiling monitors the quantity and frequency of memory allocations made by the application. Excessive or inefficient memory allocation can result in memory leaks and high garbage collection overhead, thereby impacting application efficiency.

A memory leak can occur due to improper memory management within a function. This can be identified by observing the timeline and noticing a gradual increase in memory allocations that persist without reduction. Such a pattern is a clear indicator of a memory leak.

Although memory leaks can be detected through metrics or out-of-memory (OOM) error logs, profiling provides more detailed insights into the specific function responsible for the memory allocation causing the leak, down to the code level.

Imagine your application experiences a sudden spike in memory usage. Without profiling, you would have to navigate from the memory spike to investigating code or making educated guesses about the cause.

However, with profiling, you can utilize the flame graph and table to pinpoint precisely which function is primarily responsible for the spike. This often manifests as a single node occupying a significantly wider width in the flame graph.

3) Block profiling (Lock Count and Lock Duration)

Block profiling measures the frequency and duration of blocking operations, during which a thread is paused or delayed.

Understanding Pyroscope UI

Pyroscope collects profiling data continuously, offering a time-centric perspective that enables querying of performance data from any historical point.

In its user interface, Pyroscope enhances observability tools by delivering a querying experience akin to Prometheus.

The Pyroscope Tag Explorer allows users to navigate and analyze profiling data based on specific tags. This feature helps in organizing and filtering profiling data efficiently.

The Single View page in Pyroscope’s UI is tailored for in-depth profile analysis. Here, you can explore a single flame graph with multiple viewing options and functionalities.

The Table View breaks down the profiling data into a sortable table format. Selecting "Top Table" displays the table and hides the flame graph.

The sandwich view allows you to identify that a function called throughout many functions in the codebase is the culprit.

The Comparison page enables side-by-side comparison of profiles based on different label sets, distinct time periods, or both. This feature is highly valuable for understanding the impact of changes or differences between two distinct queries of your application.

The Diff page serves as an extension of the Comparison page, offering a crucial tool for visually highlighting the differences between two profiling datasets in a more straightforward manner.

'Self' refers to the resource usage (such as CPU time or memory allocation) directly attributed to a specific function or code segment, excluding the resources used by its sub-functions or calls.

How to read a Pyroscope flame graph?

Horizontally, the flame graph represents 100% of the time that the application was running. The width of each node indicates the amount of time spent in that particular function.

Vertically, the nodes in the flame graph illustrate the hierarchy of functions, showing which functions were called and the time spent in each.

Nodes positioned below represent functions called from those higher-level functions, indicating the time spent in each of them.

Profiling a Spring Boot application with Pyroscope

Install and run the Pyroscope serve

To install and run the Pyroscope server, use these docker commands:

% docker pull grafana/pyroscope:latest
% docker network create pyroscope-demo
% docker run --rm --name pyroscope --network=pyroscope-demo -p 4040:4040 grafana/pyroscope:latest

Check if pyroscope is up and running

curl localhost:4040/ready
read

Check if the Pyroscope UI is running on http://localhost:4040/.

Create a Spring Boot CRUD Application

This is the Spring Boot app which we will profile using Pyroscope. Here is the GitHub code for the same.

You can start Pyroscope either from your application's code or attach it as a Java agent. In this article, we will start Pyroscope as a Java agent. For that, we need the Pyroscope.jar, which can be downloaded from here: Pyroscope agent.

Pyroscope configurations

Pyroscope searches for configuration in multiple sources: system properties, environment variables, and pyroscope.properties. Here are some useful Pyroscope configurations:

1) PYROSCOPE_FORMAT: Sets the profiler output format. The default is collapsed, but in order to support multiple formats it must be set to jfr.

2) PYROSCOPE_PROFILER_EVENT: Sets the profiler event. With JFR format enabled, this event refers to one of the possible CPU profiling events: itimer, cpu, wall. The default is itimer.

3) PYROSCOPE_PROFILER_ALLOC: Sets the allocation threshold to register the events, in bytes (equivalent to --alloc= in async-profiler). The default value is "" - empty string, which means that allocation profiling is disabled. Setting it to 0 will register every event, causing significant CPU and network overhead, making it not suitable for production environments. It is recommended to set a starting value of 512k and adjusting it as needed.

4) PYROSCOPE_PROFILER_LOCK: Sets the lock threshold to register the events, in nanoseconds (equivalent to --lock= in async-profiler). The default value is "" - empty string, which means that lock profiling is disabled. Setting it to 0 will register every event, causing significant CPU and network overhead, making it not suitable for production environments. It is recommended to set a starting value of 10ms and adjusting it as needed.

5) PYROSCOPE_LOG_LEVEL: Determines the level of verbosity for Pyroscope's logger. Available options include debug, info, warn, and error. The default value is set to info.

Start a Spring Boot application with the Pyroscope agent

To start a Spring Boot application with the Pyroscope agent attached, you can add the Pyroscope Java agent as a JVM argument when running your application.

% export PYROSCOPE_APPLICATION_NAME=spring-boot-crud
% export PYROSCOPE_SERVER_ADDRESS=http://localhost:4040
% export PYROSCOPE_FORMAT=jfr
% export PYROSCOPE_PROFILER_EVENT=cp
% export PYROSCOPE_PROFILER_ALLOC=0
% export PYROSCOPE_PROFILER_LOCK=0
% export PYROSCOPE_LOG_LEVEL=debug

% java -javaagent:pyroscope.jar -jar spring-boot-postgresql-jpa-crud-rest-0.0.1-SNAPSHOT.jar

The Spring Boot application should be up and running. You can access it here - Swagger UI. Additionally, the Pyroscope agent is running behind the scenes.

Let's check the Pyroscope server UI to confirm if the agent has started sending data.

Make a few calls (Optional)

This is not necessary in an actual production app, but for a local setup, let's create a few calls to the Spring Boot app using a load test tool. This will generate some meaningful data in Pyroscope for us to analyze.

We are using Locus.io, open source load testing tool, and here is the Python script used:

Read more about - Load Testing using Locust.io

Test Profiling Data in Pyroscope UI

Once you've generated profiling data, navigate to the Pyroscope UI in your web browser - http://localhost:4040/.

Pyroscope CPU profiling allows you to analyze the CPU usage of your application by monitoring the amount of CPU time consumed by different parts of your codebase.

The Pyroscope Focus View is a feature that allows users to zoom in on a specific area of interest within a flame graph or other visualization.

To search for application classes in the Pyroscope UI, tilize the search functionality provided by Pyroscope.