Building fault-tolerant microservices with the @Fallback annotation

duration 20 minutes

You’ll explore how to manage the impact of failures using MicroProfile Fault Tolerance by adding fallback behavior to microservice dependencies.

What you’ll learn

You will learn how to use MicroProfile (MP) Fault Tolerance to build resilient microservices that reduce the impact from failure and ensure continued operation of services.

MP Fault Tolerance provides a simple and flexible solution to build fault-tolerant microservices. Fault tolerance leverages different strategies to guide the execution and result of logic. As stated in the MicroProfile website, retry policies, bulkheads, and circuit breakers are popular concepts in this area. They dictate whether and when executions take place, and fallbacks offer an alternative result when an execution does not complete successfully.

The application that you will be working with is an inventory service, which collects, stores, and returns the system properties. It uses the system service to retrieve the system properties for a particular host. You will add fault tolerance to the inventory service so that it reacts accordingly when the system service is unavailable.

You will use the @Fallback annotations from the MicroProfile Fault Tolerance specification to define criteria for when to provide an alternative solution for a failed execution.

Getting started

The fastest way to work through this guide is to clone the Git repository and use the projects that are provided inside:

git clone https://github.com/openliberty/guide-microprofile-fallback.git
cd guide-microprofile-fallback

The start directory contains the starting project that you will build upon.

The finish directory contains the finished project, which is what you will build.

Try what you’ll build

The finish directory in the root of this guide contains the finished implementation for the application. You can give it a try before you proceed with building your own.

To try out the application, navigate to the finish directory and run the following command:

mvn clean install liberty:start-server

Point your browser to the http://localhost:9080/inventory/systems/localhost URL, which accesses the inventory service with a localhost host name. You see the system properties for this host. When you visit this URL, some of these system properties, such as the OS name and user name, are automatically stored in the inventory.

Now, open the resources/CustomConfigSource.json file and change the io_openliberty_guides_system_inMaintenance property from false to true and save the file. You do not need to restart the server. Next, return to your browser and point back to the http://localhost:9080/inventory/systems/localhost URL. The fallback mechanism is triggered because the system service is now in maintenance. You see the cached properties for this localhost.

When you are done checking out the application, go to the CustomConfigSource.json file again and change the io_openliberty_guides_system_inMaintenance property from true to false to set this condition back to its original value.

Stop the Open Liberty server:

mvn liberty:stop-server

Now, navigate back to the start directory to begin.

Enabling fault tolerance

The MicroProfile Fault Tolerance API was added as a dependency to your pom.xml file. Look for the dependency with the org.eclipse.microprofile.fault-tolerance artifact ID. Adding this dependency allows you to use the fault tolerance policies in your microservices.

You can also find the mpFaultTolerance-1.0 feature in your src/main/liberty/config/server.xml server configuration, which turns on MicroProfile Fault Tolerance capabilities in Open Liberty.

To easily work through this guide, the two provided microservices are set up to run on the same server. To simulate the availability of the services and then to enable fault tolerance, dynamic configuration with MicroProfile Configuration is used so that you can easily take one service or the other down for maintenance. If you want to learn more about setting up dynamic configuration, see Configuring microservices.

The following two steps set up the dynamic configuration on the system service and its client. You can move on to the next section, which adds the fallback mechanism on the inventory service.

First, the src/main/java/io/openliberty/guides/system/SystemResource.java class has the inMaintenance() condition, which determines that the system properties are returned only if you set the io_openliberty_guides_system_inMaintenance configuration property to false in the resources/CustomConfigSource.json file. Otherwise, the service returns a Status.SERVICE_UNAVAILABLE message, which makes it unavailable.

Next, the src/main/java/io/openliberty/guides/inventory/client/SystemClient.java class makes a request to the system service and checks for the response. The system service may return a Status.SERVICE_UNAVAILABLE message, which is a 503 status code. This code indicates that the server being called is unable to handle the request because of a temporary overload or scheduled maintenance, which would likely be alleviated after some delay. To simulate that the system is unavailable, an IOException is thrown.

The public getProperties() method in this class calls the getPropertiesHelper() method. The InventoryManager class calls this public method. You will look into the InventoryManager class in more detail in the next section.

Adding the @Fallback annotation

The inventory service is now able to recognize that the system service was taken down for maintenance. An IOException is thrown to simulate the system service is unavailable. Now, set a fallback method to deal with this failure.

Create the InventoryManager class in the src/main/java/io/openliberty/guides/inventory/InventoryManager.java file:

package io.openliberty.guides.inventory;

import java.io.IOException;
import java.util.Properties;
import javax.enterprise.context.ApplicationScoped;
import org.eclipse.microprofile.faulttolerance.Fallback;
import io.openliberty.guides.inventory.model.InventoryList;

@ApplicationScoped
public class InventoryManager {

  private InventoryList invList = new InventoryList();
  private InventoryUtils invUtils = new InventoryUtils();

  @Fallback(fallbackMethod = "fallbackForGet")
  public Properties get(String hostname) throws IOException {
    Properties properties = invUtils.getProperties(hostname);

    if (properties != null) {
      invList.addToInventoryList(hostname, properties);
    }
    return properties;
  }

  public Properties fallbackForGet(String hostname) {
    Properties properties = invList.findHost(hostname);
    if (properties == null) {
      Properties msgProp = new Properties();
      msgProp.setProperty(hostname, "System is not found in the inventory");
      return msgProp;
    }
    return properties;
  }

  public InventoryList list() {
    return invList;
  }
}

The @Fallback annotation dictates a method to call when the original method encounters a failed execution. In this example, use the fallbackForGet method.

The fallbackForGet() method, which is the designated fallback method for the original get() method, checks to see if the system’s properties exist in the inventory. If the system properties entry is not found in the inventory, the method prints out a warning message in the browser. Otherwise, this method returns the cached property values from the inventory.

You successfully set up your microservice to have fault tolerance capability.

Building and running the application

To build the application, run the Maven install phase from the command line in the start directory:

mvn install

This command builds the application and creates a .war file in the target directory. It also configures and installs Open Liberty into the target/liberty/wlp directory.

Next, run the Maven liberty:start-server goal:

mvn liberty:start-server

This goal starts an Open Liberty server instance. Your Maven pom.xml is already configured to start the application in this server instance.

When the server is running, point your browser to the http://localhost:9080/inventory/systems/localhost URL. You receive the system properties of your local JVM from the inventory service. Next, point your browser to the system service URL, which is located at http://localhost:9080/system/properties, to retrieve the system properties for the specific localhost. Notice that the results from the two URLs are identical because the inventory service gets its results from calling the system service.

You can test the fault tolerance mechanism of your microservices by dynamically changing the io_openliberty_guides_system_inMaintenance property value to true in the resources/CustomConfigSource.json file, which turns the system service in maintenance. After saving the file, go back to your browser and refresh to the http://localhost:9080/inventory/systems/localhost URL to view the cached version of the properties. The fallbackForGet() method, which is the designated fallback method, is called when the system service is not available. The cached system properties contain only the OS name and user name key and value pairs.

To see that the system service is down, point your browser to the http://localhost:9080/system/properties URL again. You see that the service displays a 503 HTTP response code.

When you’re done, change the io_openliberty_guides_system_inMaintenance property value back to false in the resources/CustomConfigSource.json file.

If you make changes to the code, use the Maven compile goal to rebuild the application and have the running Open Liberty server pick them up automatically. Make sure your application is using loose config for this to work:

mvn compile

To stop the Open Liberty server, run the Maven liberty:stop-server goal:

mvn liberty:stop-server

Testing the application

You can test your application manually, but automated tests ensure code quality because they trigger a failure whenever a code change introduces a defect. JUnit and the JAX-RS Client API provide a simple environment for you to write tests.

Create the FaultToleranceTest class in the src/test/java/it/io/openliberty/guides/faulttolerance/FaultToleranceTest.java file:

package it.io.openliberty.guides.faulttolerance;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import javax.json.JsonObject;
import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.core.Response;
import org.apache.cxf.jaxrs.provider.jsrjsonp.JsrJsonpProvider;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import it.io.openliberty.guides.utils.TestUtils;

public class FaultToleranceTest {

    private Response response;
    private Client client;

    @Before
    public void setup() {
        client = ClientBuilder.newClient();
        client.register(JsrJsonpProvider.class);
    }

    @After
    public void teardown() {
        client.close();
        response.close();
    }

    @Test
    public void testFallbackForGet() throws InterruptedException {
        response = TestUtils.getResponse(client,
                                         TestUtils.INVENTORY_LOCALHOST_URL);
        assertResponse(TestUtils.baseUrl, response);
        JsonObject obj = response.readEntity(JsonObject.class);
        int propertiesSize = obj.size();
        TestUtils.changeSystemProperty(TestUtils.SYSTEM_MAINTENANCE_FALSE,
                                       TestUtils.SYSTEM_MAINTENANCE_TRUE);
        Thread.sleep(3000);
        response = TestUtils.getResponse(client,
                                         TestUtils.INVENTORY_LOCALHOST_URL);
        assertResponse(TestUtils.baseUrl, response);
        obj = response.readEntity(JsonObject.class);
        int propertiesSizeFallBack = obj.size();
        assertTrue("The total number of properties from the @Fallback method "
                + "is not smaller than the number from the system service, as expected.",
                   propertiesSize > propertiesSizeFallBack);
        TestUtils.changeSystemProperty(TestUtils.SYSTEM_MAINTENANCE_TRUE,
                                       TestUtils.SYSTEM_MAINTENANCE_FALSE);
    }

    private void assertResponse(String url, Response response) {
        assertEquals("Incorrect response code from " + url, 200,
                     response.getStatus());
    }
}

The @Before and @After annotations indicate that this method executes before or after the other test case. These methods are generally used to perform any setup and teardown tasks. In this case, the setup method creates a JAX-RS client, which makes HTTP requests to the inventory service. This client must also be registered with a JSON-P provider to process JSON resources. The teardown method simply destroys this client instance as well as the HTTP responses.

The testFallbackForGet() test case sends a request to the inventory service to get the systems properties for a host name before and after the system service becomes unavailable. Then, it asserts outputs from the two requests to ensure that they are different from each other.

The @Test annotation indicates that the method automatically executes when your test class runs.

In addition, a few endpoint tests have been included for you to test the basic functionality of the inventory and system services. If a test failure occurs, then you might have introduced a bug into the code.

Running the tests

If the server is still running from the previous steps, stop it using the Maven liberty:stop-server goal from command line in the start directory:

mvn liberty:stop-server

Then, verify that the tests pass using the Maven verify goal:

mvn verify

It may take some time before build is complete. If the tests pass, you will see a similar output to the following:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.system.SystemEndpointTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.937 sec - in it.io.openliberty.guides.system.SystemEndpointTest
Running it.io.openliberty.guides.inventory.InventoryEndpointTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.396 sec - in it.io.openliberty.guides.inventory.InventoryEndpointTest
Running it.io.openliberty.guides.faulttolerance.FaultToleranceTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.517 sec - in it.io.openliberty.guides.faulttolerance.FaultToleranceTest

Results :

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

To see if the tests detect a failure, comment out the changeSystemProperty() methods in the FaultToleranceTest.java file. Rerun the Maven build. You see a test failure occur for the testFallbackForGet() test case.

Great work! You’re done!

You just learned how to build a fallback mechanism for a microservice with MicroProfile Fault Tolerance and wrote a test to validate it.

You can try one of the related MicroProfile guides. They demonstrate technologies that you can learn and expand on what you built here.

Contribute to this guide

Is something missing or broken? Raise an issue, or send us a pull request.