Circuit Breaker Pattern with Resilience4J in a Spring Boot Application

Mar 31, 2024

Resilience4j is a lightweight fault tolerance library that draws inspiration from Netflix Hystrix but is specifically crafted for functional programming.

The library offers higher-order functions, known as decorators, designed to augment any functional interface, lambda expression, or method reference with features such as Circuit Breaker, Rate Limiter, Retry, or Bulkhead.

These functionalities can be seamlessly integrated within a project, class, or even applied to a single method.

It's possible to layer multiple decorators on any functional interface, lambda expression, or method reference, allowing for versatile and customizable fault tolerance.

While numerous annotation-based implementations exist online, this article focuses solely on the reactive approach using router predicates and router functions.

How Circuit Breaker Pattern works?

In general, a circuit breaker functions as an automatic electrical switch, safeguarding an electrical circuit from potential damage caused by overload (such as a lightning strike) or a short circuit. Its primary purpose is to halt the flow of current upon detecting a defect, thereby protecting the electrical appliances in your home.

The Circuit Breaker design pattern is employed to cease the request and response process if a service is not functioning as intended.

Failures can arise from various issues, including network issues, a malfunctioning router or switch, stress on the downstream service due to excess load, or low memory/CPU, resulting in the return of error codes or the introduction of latency. In a microservices environment, numerous components can cause a request to fail either fully or partially.

The Circuit Breaker is implemented through a finite state machine with three normal states: CLOSED, OPEN, and HALF_OPEN, as well as two special states: DISABLED and FORCED_OPEN.

If exceptions/errors or timeouts persist, the circuit breaker transitions to the OPEN state, causing requests to fail-fast without reaching the stressed service. This, in turn, allows the stressed service some time to recover.

After a specified time period, the circuit breaker transitions to the HALF_OPEN state, permitting a limited number of calls to reach the previously stressed downstream service to pass/fail. If these requests succeed, the circuit breaker returns to normal operation (CLOSED state). If failures persist, the HALF_OPEN state restarts.

The Circuit Breaker utilizes a sliding window to store and aggregate the outcomes of calls, offering the choice between a count-based sliding window and a time-based sliding window.

In a count-based sliding window, if the window size is, for instance, 10, the circular array (sliding window) always has 10 buckets for measurement.

Conversely, in a time-based sliding window of, say, 10 seconds, each bucket aggregates the outcomes of all calls occurring in a certain second.

The transition from CLOSED to OPEN occurs when the failure rate equals or exceeds a configurable threshold, such as when more than 50% of recorded calls have failed (i.e., 5 out of 10).

By default, all exceptions count as failures, but one can define a list of exceptions that should count as failures. All other exceptions are considered successes unless they are explicitly ignored.

The Circuit Breaker also transitions from CLOSED to OPEN when the percentage of slow calls equals or exceeds a configurable threshold, e.g., when more than 50% (5 out of 10) recorded calls took longer than 5 seconds.

Both the failure rate and slow call rate can only be calculated if a minimum number of calls were recorded. For example, if the minimum required calls are 7, then at least 7 calls must be recorded before the failure rate can be calculated. If only 6 calls have been evaluated, the Circuit Breaker will not trip open, even if all 6 calls have failed.

When in the OPEN state, the Circuit Breaker rejects calls with a CallNotPermittedException. After a specified wait time has elapsed, the Circuit Breaker transitions from OPEN to HALF_OPEN, allowing a configurable number of calls to check if the stressed service is still unavailable or has become available again. Further calls are rejected with a CallNotPermittedException until all permitted calls have completed.

If the failure rate or slow call rate remains equal or greater than the configured threshold, the state changes back to OPEN. If both rates are below the threshold, the state changes back to CLOSED.

In other two special states states, no Circuit Breaker events (apart from state transitions) are generated, and no metrics are recorded. The only way to exit these states is to trigger a state transition or reset the Circuit Breaker.

Read More: Circuit Breaker Pattern in Microservices

Resilience4j + Spring BOOT

The objective is to develop:

1) A straightforward Spring Boot REST service, named "inventory-service," featuring a solitary REST API - inventory/{productId}. This service will be invoked by another client Spring Boot REST service, denoted as "order-service."

2) The "order-service" will include the "resilience4j" dependency and incorporate all the associated logic for the Circuit Breaker.

If, during the invocation of "inventory/{productId}" from the "inventory-service," exceptions occur and the error rate surpasses the allowed threshold, the "order-service" will promptly reject inventory/{productId} requests directed to the "inventory-service" for a configurable duration.

This pause allows the "inventory-service" a specified time to recover.

1) Inventory Service "inventory-service"

This Spring Boot service acts as a simple API Server featuring a lone REST API - inventory/{productId}.

1.1) Create a Spring Boot Project

You can use Spring Initializr to generate a basic project structure.

Dependecnies

You should have following dependencies in your pom.xml file.

1.2) Create a 'Inventory' Entity

Create a "Inventory" class/record to represent "inventory" in your application.

1.3) Create a Controller

Create a controller class to handle HTTP requests.

1.4) Update application.yml

The server port is set to 8081, in this case, the application would be accessible at: http://localhost:8081.

1.5) Run Application

Open your project in an IDE (IntelliJ or Eclipse) or use the command line to navigate to your project directory. Run the application:

mvn spring-boot:run

The application will start, and you can access it at: http://localhost:8081.

1.5) Test Your API

Open the Swagger UI in the browser to test your APIs. It provides a convenient way to input parameters, make requests, and view responses.

Source Code: GitHub

2) Order Service (order-service)

The "order-service" will include the "resilience4j" dependency and incorporate all the relevant logic for the Circuit Breaker.

Initially, the circuit breaker remains in the "CLOSED" state, allowing all calls from the "order-service" to the "inventory-service" to pass through.

Should the "order-service," while calling inventory/{productId} from the "inventory-service," encounter exceptions and surpass the allowed error rate threshold, the circuit breaker will shift to the "OPEN" state. Consequently, the circuit breaker will promptly reject inventory/{productId} requests sent from the "order-service" to the "inventory-service" for a configurable duration, providing the "inventory-service" time to recover.

After a specific configurable time has elapsed, the circuit breaker will transition to the "HALF_OPEN" state. In this state, the circuit breaker permits the "order-service" to dispatch a configurable number of calls to the "inventory-service" to assess its recovery status.

If a configurable percentage of calls are successful during this period, the circuit breaker will move from "HALF_OPEN" to "OPEN." Otherwise, if the calls are not successful, the state will shift from "HALF_OPEN" back to "CLOSED."

2.1) Create a Spring Boot Project

You can use Spring Initializr to generate a basic project structure.

Dependecnies

Two dependencies, namely actuator and aop, are essential for recording resilience4j metrics, monitoring health status, and enabling aspect orientation.

Given that the project employs Webflux, it is necessary to include the resilience4j-reactor dependency.

Ensure that the following dependencies are present in your pom.xml file.

2.2) Create a 'Inventory' Entity

Create a "Inventory" class or record for serializing and deserializing "Inventory" data from the API Server ("inventory-service").

2.3) Create a HTTP Client

Create an HTTP WebClient for accessing APIs from the "inventory-service." To delve deeper into REST Client functionality in Spring Boot, refer to this.

The "retry" properties enable exponential backoff, configuring the back-off multiplier to "2" and the wait duration to "5" seconds. This setup allows the retry mechanism to pause for 5 seconds, multiplied by a factor of 2, before attempting another retry. The maximum number of retries is set to "3".

2.4) Circuit Breaker Configuration

Configuring Resilience4j in a Spring Boot project can be accomplished through two approaches:

1) Properties file (application.properties/yaml) or

2) By "Java configuration"

It is advisable to opt for properties file configuration unless a customized configuration is required in your project. The configuration specified in the properties file takes precedence over the Java configuration.

Nevertheless, in this application, we will utilize Java configuration to gain a deeper understanding and facilitate the integration of additional functionalities if necessary.

A sliding window size (slidingWindowSize) of "10" is utilized to track call outcomes when the circuit breaker is closed. A minimum number of calls (minimumNumberOfCalls) set at "5" are required per sliding window period before the CircuitBreaker can calculate the error rate or slow call rate.

Upon reaching a failure rate threshold (failureRateThreshold) of "50%", where the failure rate is equal to or greater than the threshold (in this case, 50% of 5, which equals 3), the CircuitBreaker transitions to the OPEN state and initiates short-circuiting of calls.

Additional properties include:

waitDurationInOpenState: the duration of wait in the OPEN state before automatically transitioning into a HALF-OPEN state.

permittedNumberOfCallsInHalfOpenState: the number of calls allowed in the half-open state.

As specified by the "recordFailurePredicate," the circuit breaker will only open when any of the defined exceptions/errors on the predicate are recorded and meet or exceed the threshold.

By default, retry has a lower priority than the circuit breaker. For the "retry" functionality, the application will listen for the "onRetry" event, which occurs when the application is attempting to retry the call, request, or action and log it.

Regarding the "circuit breaker," the application will be attentive to events such as:

1) "onCallNotPermitted": when the circuit is in the open state and no action/call is allowed.

2) "onSuccess": when the action/call is successful.

3) "onError": when there is a recorded error on the action/call being performed.

4) "onIgnoredError": when there is an error that is not recorded on the action/call being performed.

5) "onReset": when the circuit breaker is reset.

6) "onStateTransition": when the circuit breaker transitions from one state to another.

2.5) Properties

Enabling the property "management.health.circuitbreakers.enabled: true" incorporates CircuitBreaker information, including status, state, thresholds, etc., into the actuator health information response.

This information can be accessed through the /actuator/health endpoint.

2.6) Service

Let's incorporate the CircuitBreaker in our service class when retrieving inventory information from the "inventory-service."

2.7) Order Entity

This is a simple Java record that represents the "order" entity.

2.8) Controller

This is a basic controller featuring only a single endpoint.

2.9) Run Application

Open your project in an IDE (IntelliJ or Eclipse) or use the command line to navigate to your project directory. Run the application:

mvn spring-boot:run

The application will start, and you can access it at: http://localhost:8080.

2.10) Test

Open the Swagger UI in the browser to test your APIs. It provides a convenient way to input parameters, make requests, and view responses.

Attempt to modify the "inventory-service" endpoint by introducing various exceptions, sending a 200 response, and exceeding the timeout by incorporating delays. Observe the behavior of the Circuit Breaker. One potential outcome is illustrated below:

Source Code: GitHub