Enabling distributed tracing in microservices with Jaeger

duration 20 minutes
New

Prerequisites:

Explore how to enable and customize tracing of JAX-RS and non-JAX-RS methods by using MicroProfile OpenTracing and Jaeger.

What you’ll learn

You will learn how to enable automatic tracing for JAX-RS methods and create custom tracers for non-JAX-RS methods by using MicroProfile OpenTracing.

OpenTracing is a standard API for instrumenting microservices for distributed tracing. Distributed tracing helps troubleshoot microservices by examining and logging requests as they propagate through a distributed system, allowing developers to tackle the otherwise difficult task of debugging these requests. Without a distributed tracing system in place, analyzing the workflows of operations becomes difficult, particularly in regard to pinpointing when and by whom a request is received or when a response is sent back.

Tracer and Span are two critical types in the OpenTracing specification. The Span type is the primary building block of a distributed trace, representing an individual unit of work done in a distributed system. The Trace type in OpenTracing can be thought of as a directed acyclic graph (DAG) of Spans, where the edges between Spans are called References. The Tracer interface creates Spans and Traces and understands how to serialize and deserialize their metadata across process boundaries.

MicroProfile OpenTracing enables distributed tracing in microservices. The MicroProfile OpenTracing specification doesn’t address the problem of defining, implementing, or configuring the underlying distributed tracing system. Rather, the specification makes it easier to instrument services with distributed tracing given an existing distributed tracing system.

Jaeger is an open source distributed tracing system that is compatible with the OpenTracing specification. Jaeger also provides an implementation of Tracer in the client package that is compatible with MicroProfile OpenTracing.

You’ll configure the provided inventory and system services to use Jaeger for distributed tracing with MicroProfile OpenTracing. You’ll run these services in two separate JVMs made of two server instances to demonstrate tracing in a distributed environment. If all the components were run on a single server, then any logging software would be sufficient.

Additional prerequisites

Before you begin, deploy the Jaeger all-in-one executable file to start the Jaeger tracing system. The Jaeger all-in-one executable file is configured for quick local testing. You can find information about the Jaeger server and instructions for starting the all-in-one executable file in the Jaeger documentation.

Before you proceed, make sure that your Jaeger server is up and running. Jaeger can be found at the http://localhost:16686 URL.

Getting started

The fastest way to work through this guide is to clone the Git repository and use the projects that are provided inside:

git clone https://github.com/openliberty/guide-microprofile-opentracing-jaeger.git
cd guide-microprofile-opentracing-jaeger

The start directory contains the starting project that you will build upon.

The finish directory contains the finished project that you will build.

Try what you’ll build

The finish directory in the root of this guide contains the finished application. Give it a try before you proceed.

Open a command-line session and navigate to the finish/inventory directory. Run the following Maven goal to build the inventory service and deploy it to Open Liberty:

mvn liberty:run

Open another command-line session and navigate to the finish/system directory. Run the following Maven goal to build the system service and deploy it to Open Liberty:

mvn liberty:run

After you see the following message in both command-line sessions, both of your services are ready:

The defaultServer server is ready to run a smarter planet.

Make sure that your Jaeger server is running and point your browser to the http://localhost:9081/inventory/systems/localhost URL. When you visit this endpoint, you make two GET HTTP requests, one to the system service and one to the inventory service. Both of these requests are configured to be traced, so a new trace is recorded in Jaeger.

To view the traces, go to the http://localhost:16686 URL. You can view the traces for the inventory or system services under the Search tab in the upper left of the page. Select the services in the Select a service menu and click the Find Traces button at the end of the section.

If you only see the jaeger-query listed in the dropdown, you might need to wait a little longer and refresh the page to see the application services.

View the traces for inventory. You’ll see the following trace:

Trace result


The trace has four spans, three from inventory and one from system. Click the trace to view its details. Under Service & Operation, you see the spans in this trace. You can inspect each span by clicking it to reveal more detailed information, such as the time at which a request was received and the time at which a response was sent back.

Verify that there are three spans from inventory and one span from system:

Finished application’s trace


After you’re finished reviewing the application, stop the Open Liberty servers by pressing CTRL+C in the command-line sessions where you ran the system and inventory services. Alternatively, you can run the following goals from the finish directory in another command-line session:

 mvn -pl system liberty:stop
 mvn -pl inventory liberty:stop

Building the application

You need to start the services to see basic traces appear in Jaeger.

When you run Open Liberty in development mode, known as dev mode, the server listens for file changes and automatically recompiles and deploys your updates whenever you save a new change.

Open a command-line session and navigate to the start/inventory directory. Run the following Maven goal to start the inventory service in dev mode:

mvn liberty:dev

Open a command-line session and navigate to the start/system directory. Run the following Maven goal to start the system service in dev mode:

mvn liberty:dev

After you see the following message, your application server in dev mode is ready:

Press the Enter key to run tests on demand.

Dev mode holds your command-line session to listen for file changes. Open another command-line session to continue, or open the project in your editor.

When the servers start, you can find the system and inventory services at the following URLs:

Enabling existing Tracer implementation

To collect traces across your systems, you need to implement the OpenTracing Tracer interface. Jaeger provides a Tracer implementation for the Jaeger server in the jaeger-client package.

This package is already added as a dependency for you in your pom.xml file. It’s downloaded and installed automatically into each service when you run a Maven build.

Configuring the Jaeger client

In a development environment, it is important that every trace is sampled. When every trace is sampled, all spans are available in the Jaeger UI.

The JAEGER_SAMPLER_TYPE and JAEGER_SAMPLER_PARAM environment variables are set as Open Liberty configuration properties to sample all traces.

The const value for JAEGER_SAMPLER_TYPE environment variable configures the Jaeger client sampler to make the same sampling decision for each trace, based on the sampler parameter. If the sampler parameter is 1, it samples all traces. If the sampler parameter is 0, it doesn’t sample any traces.

The 1 value for JAEGER_SAMPLER_PARAM variable configures the Jaeger sampler to sample all traces.

In a production environment, this configuration might cause a lot of overhead on the application and a lower sampling rate can be used. The different values for client sampling configuration can be found in the sampling documentation.

Similarly, in a production environment, Jaeger might not be running in the same host as the application. In this case, set the hostname of the Jaeger server to the JAEGER_AGENT_HOST environment variable and set the port that communicates with the Jaeger host to the JAEGER_AGENT_PORT environment variable.

You can view the configuration environment variables at the Jaeger Java client documentation.

pom.xml

  1<?xml version='1.0' encoding='utf-8'?>
  2<project xmlns="http://maven.apache.org/POM/4.0.0"
  3xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  4xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
  5http://maven.apache.org/xsd/maven-4.0.0.xsd">
  6    <modelVersion>4.0.0</modelVersion>
  7
  8    <groupId>io.openliberty.guides</groupId>
  9    <artifactId>inventory</artifactId>
 10    <version>1.0-SNAPSHOT</version>
 11    <packaging>war</packaging>
 12
 13    <properties>
 14        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
 15        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
 16        <maven.compiler.source>1.8</maven.compiler.source>
 17        <maven.compiler.target>1.8</maven.compiler.target>
 18
 19        <!-- OpenLiberty runtime -->
 20        <liberty.var.system.http.port>9080</liberty.var.system.http.port>
 21        <liberty.var.default.http.port>9081</liberty.var.default.http.port>
 22        <liberty.var.default.https.port>9444</liberty.var.default.https.port>
 23
 24        <!-- Jaeger configuration -->
 25        <!-- tag::jaegerClientConfig[] -->
 26        <!-- tag::samplerTypeConfig[] -->
 27        <liberty.env.JAEGER_SAMPLER_TYPE>const</liberty.env.JAEGER_SAMPLER_TYPE>
 28        <!-- end::samplerTypeConfig[] -->
 29        <!-- tag::samplerParamConfig[] -->
 30        <liberty.env.JAEGER_SAMPLER_PARAM>1</liberty.env.JAEGER_SAMPLER_PARAM>
 31        <!-- end::samplerParamConfig[] -->
 32        <!-- end::jaegerClientConfig[] -->
 33    </properties>
 34
 35    <dependencies>
 36        <!-- Provided dependencies -->
 37        <dependency>
 38            <groupId>jakarta.platform</groupId>
 39            <artifactId>jakarta.jakartaee-api</artifactId>
 40            <version>8.0.0</version>
 41            <scope>provided</scope>
 42        </dependency>
 43        <dependency>
 44            <groupId>org.eclipse.microprofile</groupId>
 45            <artifactId>microprofile</artifactId>
 46            <version>3.3</version>
 47            <type>pom</type>
 48            <scope>provided</scope>
 49        </dependency>
 50        <!-- For Jaeger -->
 51        <!-- tag::jaeger[] -->
 52        <dependency>
 53            <groupId>io.jaegertracing</groupId>
 54            <artifactId>jaeger-client</artifactId>
 55            <version>0.34.0</version>
 56        </dependency>
 57        <dependency>
 58            <groupId>org.slf4j</groupId>
 59            <artifactId>slf4j-api</artifactId>
 60            <version>1.7.30</version>
 61        </dependency>
 62        <dependency>
 63            <groupId>org.slf4j</groupId>
 64            <artifactId>slf4j-jdk14</artifactId>
 65            <version>1.7.30</version>
 66        </dependency>
 67        <!-- end::jaeger[] -->
 68        <!-- For tests -->
 69        <dependency>
 70            <groupId>org.junit.jupiter</groupId>
 71            <artifactId>junit-jupiter</artifactId>
 72            <version>5.6.2</version>
 73            <scope>test</scope>
 74        </dependency>
 75        <dependency>
 76            <groupId>org.apache.cxf</groupId>
 77            <artifactId>cxf-rt-rs-client</artifactId>
 78            <version>3.3.6</version>
 79            <scope>test</scope>
 80        </dependency>
 81        <dependency>
 82            <groupId>org.apache.cxf</groupId>
 83            <artifactId>cxf-rt-rs-extension-providers</artifactId>
 84            <version>3.3.6</version>
 85            <scope>test</scope>
 86        </dependency>
 87        <dependency>
 88            <groupId>org.glassfish</groupId>
 89            <artifactId>javax.json</artifactId>
 90            <version>1.1.4</version>
 91            <scope>test</scope>
 92        </dependency>
 93    </dependencies>
 94
 95    <build>
 96        <finalName>${project.artifactId}</finalName>
 97        <plugins>
 98            <plugin>
 99                <groupId>org.apache.maven.plugins</groupId>
100                <artifactId>maven-war-plugin</artifactId>
101                <version>3.2.3</version>
102            </plugin>
103
104            <!-- Liberty plugin -->
105            <plugin>
106                <groupId>io.openliberty.tools</groupId>
107                <artifactId>liberty-maven-plugin</artifactId>
108                <version>3.2.1</version>
109            </plugin>
110
111            <!-- Plugin to run unit tests -->
112            <plugin>
113                <groupId>org.apache.maven.plugins</groupId>
114                <artifactId>maven-surefire-plugin</artifactId>
115                <version>2.22.2</version>
116            </plugin>
117
118            <!-- Plugin to run functional tests -->
119            <plugin>
120                <groupId>org.apache.maven.plugins</groupId>
121                <artifactId>maven-failsafe-plugin</artifactId>
122                <version>2.22.2</version>
123                <configuration>
124                    <systemPropertyVariables>
125                        <sys.http.port>${liberty.var.system.http.port}</sys.http.port>
126                        <inv.http.port>${liberty.var.default.http.port}</inv.http.port>
127                    </systemPropertyVariables>
128                </configuration>
129            </plugin>
130        </plugins>
131    </build>
132</project>

Enabling and disabling distributed tracing

The MicroProfile OpenTracing feature enables tracing of all JAX-RS methods by default. To further control and customize these traces, use the @Traced annotation to enable and disable tracing of particular methods. You can also inject a custom Tracer object to create and customize spans.

This feature is already enabled in the inventory and system configuration files.

Enabling distributed tracing without code instrumentation

Because tracing of all JAX-RS methods is enabled by default, you only need to enable the MicroProfile OpenTracing feature in the server.xml file to see some basic traces in Jaeger.

The OpenTracing API is exposed as a third-party API in Open Liberty. To add the visibility of OpenTracing APIs to the application, add third-party to the types of API packages that this class loader supports. Instead of explicitly configuring a list of API packages that includes third-party, set the +third-party value to the apiTypeVisibility attribute in the <classLoader /> configuration. This configuration adds third-party to the default list of API package types that are supported.

server.xml

 1<server description="Sample Liberty server">
 2
 3    <featureManager>
 4        <feature>jaxrs-2.1</feature>
 5        <feature>jsonp-1.1</feature>
 6        <feature>cdi-2.0</feature>
 7        <feature>mpConfig-1.4</feature>
 8        <!-- tag::mpOpenTracing[] -->
 9        <feature>mpOpenTracing-1.3</feature>
10        <!-- end::mpOpenTracing[] -->
11    </featureManager>
12
13    <httpEndpoint httpPort="${default.http.port}" httpsPort="${default.https.port}"
14                  id="defaultHttpEndpoint" host="*" />
15
16    <webApplication location="inventory.war" contextRoot="/">
17        <!-- enable visibility to third party apis -->
18        <!-- tag::thirdParty[] -->
19        <classloader apiTypeVisibility="+third-party"/>
20        <!-- end::thirdParty[] -->
21    </webApplication>
22
23</server>

Make sure that your services are running. Then, point your browser to any of the services' endpoints and check your Jaeger server for traces.

Enabling explicit distributed tracing

Use the @Traced annotation to define explicit span creation for specific classes and methods. If you place the annotation on a class, then the annotation is automatically applied to all methods within that class. If you place the annotation on a method, then the annotation overrides the class annotation if one exists.

The @Traced annotation can be configured with the following two parameters:

  • The value=[true|false] parameter indicates whether a particular class or method is traced. For example, while all JAX-RS methods are traced by default, you can disable their tracing by using the @Traced(false) annotation. This parameter is set to true by default.

  • The operationName=<Span name> parameter indicates the name of the span that is assigned to the method that is traced. If you omit this parameter, the span is named with the <package name>.<class name>.<method name> format. If you use this parameter at a class level, then all methods within that class have the same span name unless they are explicitly overridden by another @Traced annotation.

Update the InventoryManager class.
inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java

Enable tracing of the list() non-JAX-RS method by updating @Traced as shown.

Go to the http://localhost:9081/inventory/systems URL and check your Jaeger server at the http://localhost:16686 URL. If you have the Jaeger UI open from a previous step, refresh the page. Select the inventory traces and click the Find Traces button.

You see a new trace record that is two spans long. One span is for the listContents() JAX-RS method in the InventoryResource class, and the other span is for the list() method in the InventoryManager class.

Verify that you see the following spans:

Explicit trace span


InventoryManager.java

 1//tag::copyright[]
 2/*******************************************************************************
 3* Copyright (c) 2017, 2020 IBM Corporation and others.
 4* All rights reserved. This program and the accompanying materials
 5* are made available under the terms of the Eclipse Public License v1.0
 6* which accompanies this distribution, and is available at
 7* http://www.eclipse.org/legal/epl-v10.html
 8*
 9* Contributors:
10*     IBM Corporation - Initial implementation
11*******************************************************************************/
12//end::copyright[]
13package io.openliberty.guides.inventory;
14
15import java.util.ArrayList;
16import java.util.Properties;
17import io.openliberty.guides.inventory.client.SystemClient;
18import io.openliberty.guides.inventory.model.InventoryList;
19import javax.enterprise.context.ApplicationScoped;
20import javax.inject.Inject;
21import java.util.List;
22import java.util.Collections;
23
24import org.eclipse.microprofile.opentracing.Traced;
25
26import io.opentracing.Scope;
27import io.opentracing.Tracer;
28import io.openliberty.guides.inventory.model.*;
29
30@ApplicationScoped
31// tag::InventoryManager[]
32public class InventoryManager {
33
34    private List<SystemData> systems = Collections.synchronizedList(new ArrayList<>());
35    private SystemClient systemClient = new SystemClient();
36    // tag::customTracer[]
37    @Inject Tracer tracer;
38    // end::customTracer[]
39
40    public Properties get(String hostname) {
41        systemClient.init(hostname, 9080);
42        Properties properties = systemClient.getProperties();
43
44        return properties;
45    }
46
47    public void add(String hostname, Properties systemProps) {
48        Properties props = new Properties();
49        props.setProperty("os.name", systemProps.getProperty("os.name"));
50        props.setProperty("user.name", systemProps.getProperty("user.name"));
51
52        SystemData system = new SystemData(hostname, props);
53        // tag::Add[]
54        if (!systems.contains(system)) {
55            // tag::Try[]
56            // tag::addSpan[]
57            try (Scope childScope = tracer.buildSpan("add() Span")
58                                              .startActive(true)) {
59            // end::addSpan[]
60                // tag::addToInvList[]
61                systems.add(system);
62                // end::addToInvList[]
63            // tag::endTry[]
64            }
65            // end::endTry[]
66            // end::Try[]
67        }
68        // end::Add[]
69    }
70
71    // tag::Traced[]
72    @Traced(operationName = "InventoryManager.list")
73    // end::Traced[]
74    // tag::list[]
75    public InventoryList list() {
76        return new InventoryList(systems);
77    }
78    // end::list[]
79
80    int clear() {
81        int propertiesClearedCount = systems.size();
82        systems.clear();
83        return propertiesClearedCount;
84    }
85}
86// end::InventoryManager[]

InventoryResource.java

 1// tag::copyright[]
 2/*******************************************************************************
 3 * Copyright (c) 2017, 2020 IBM Corporation and others.
 4 * All rights reserved. This program and the accompanying materials
 5 * are made available under the terms of the Eclipse Public License v1.0
 6 * which accompanies this distribution, and is available at
 7 * http://www.eclipse.org/legal/epl-v10.html
 8 *
 9 * Contributors:
10 *     IBM Corporation - Initial implementation
11 *******************************************************************************/
12// end::copyright[]
13package io.openliberty.guides.inventory;
14
15import java.util.Properties;
16import javax.enterprise.context.RequestScoped;
17import javax.inject.Inject;
18import javax.ws.rs.*;
19import javax.ws.rs.core.MediaType;
20import javax.ws.rs.core.Response;
21
22import org.eclipse.microprofile.opentracing.Traced;
23
24import io.openliberty.guides.inventory.model.InventoryList;
25
26@RequestScoped
27@Path("/systems")
28// tag::InventoryResource[]
29public class InventoryResource {
30
31    @Inject InventoryManager manager;
32
33    @GET
34    @Path("/{hostname}")
35    @Produces(MediaType.APPLICATION_JSON)
36    // tag::getPropertiesForHost[]
37    public Response getPropertiesForHost(@PathParam("hostname") String hostname) {
38        Properties props = manager.get(hostname);
39        if (props == null) {
40            return Response.status(Response.Status.NOT_FOUND)
41                           .entity("{ \"error\" : \"Unknown hostname or the system service "
42                           + "may not be running on " + hostname + "\" }")
43                           .build();
44        }
45        manager.add(hostname, props);
46        return Response.ok(props).build();
47    }
48    // end::getPropertiesForHost[]
49
50    @GET
51    // tag::Traced-false[]
52    @Traced(false)
53    // end::Traced-false[]
54    @Produces(MediaType.APPLICATION_JSON)
55    // tag::listContents[]
56    public InventoryList listContents() {
57        return manager.list();
58    }
59    // end::listContents[]
60
61    @DELETE
62    @Produces(MediaType.APPLICATION_JSON)
63    public Response clearContents() {
64        int cleared = manager.clear();
65
66        if (cleared == 0) {
67            return Response.status(Response.Status.NOT_MODIFIED)
68                    .build();
69        }
70        return Response.status(Response.Status.OK)
71                .build();
72    }
73}
74// end::InventoryResource[]

Disable automatic distributed tracing

You can use the @Traced annotation with a value of false to disable automatic distributed tracing of JAX-RS methods.

Update the InventoryResource class.
inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java

Disable tracing of the listContents() JAX-RS method by setting @Traced(false).

Go to the http://localhost:9081/inventory/systems URL and check your Jaeger server at the http://localhost:16686 URL. If you have the Jaeger UI open from a previous step, refresh the page. Select the inventory traces and click the Find Traces button. You see a new trace record that is just one span long for the remaining list() method in the InventoryManager class.

Verify that you see the following span:

Disable trace span


InventoryManager.java

 1//tag::copyright[]
 2/*******************************************************************************
 3* Copyright (c) 2017, 2020 IBM Corporation and others.
 4* All rights reserved. This program and the accompanying materials
 5* are made available under the terms of the Eclipse Public License v1.0
 6* which accompanies this distribution, and is available at
 7* http://www.eclipse.org/legal/epl-v10.html
 8*
 9* Contributors:
10*     IBM Corporation - Initial implementation
11*******************************************************************************/
12//end::copyright[]
13package io.openliberty.guides.inventory;
14
15import java.util.ArrayList;
16import java.util.Properties;
17import io.openliberty.guides.inventory.client.SystemClient;
18import io.openliberty.guides.inventory.model.InventoryList;
19import javax.enterprise.context.ApplicationScoped;
20import javax.inject.Inject;
21import java.util.List;
22import java.util.Collections;
23
24import org.eclipse.microprofile.opentracing.Traced;
25
26import io.opentracing.Scope;
27import io.opentracing.Tracer;
28import io.openliberty.guides.inventory.model.*;
29
30@ApplicationScoped
31// tag::InventoryManager[]
32public class InventoryManager {
33
34    private List<SystemData> systems = Collections.synchronizedList(new ArrayList<>());
35    private SystemClient systemClient = new SystemClient();
36    // tag::customTracer[]
37    @Inject Tracer tracer;
38    // end::customTracer[]
39
40    public Properties get(String hostname) {
41        systemClient.init(hostname, 9080);
42        Properties properties = systemClient.getProperties();
43
44        return properties;
45    }
46
47    public void add(String hostname, Properties systemProps) {
48        Properties props = new Properties();
49        props.setProperty("os.name", systemProps.getProperty("os.name"));
50        props.setProperty("user.name", systemProps.getProperty("user.name"));
51
52        SystemData system = new SystemData(hostname, props);
53        // tag::Add[]
54        if (!systems.contains(system)) {
55            // tag::Try[]
56            // tag::addSpan[]
57            try (Scope childScope = tracer.buildSpan("add() Span")
58                                              .startActive(true)) {
59            // end::addSpan[]
60                // tag::addToInvList[]
61                systems.add(system);
62                // end::addToInvList[]
63            // tag::endTry[]
64            }
65            // end::endTry[]
66            // end::Try[]
67        }
68        // end::Add[]
69    }
70
71    // tag::Traced[]
72    @Traced(operationName = "InventoryManager.list")
73    // end::Traced[]
74    // tag::list[]
75    public InventoryList list() {
76        return new InventoryList(systems);
77    }
78    // end::list[]
79
80    int clear() {
81        int propertiesClearedCount = systems.size();
82        systems.clear();
83        return propertiesClearedCount;
84    }
85}
86// end::InventoryManager[]

InventoryResource.java

 1// tag::copyright[]
 2/*******************************************************************************
 3 * Copyright (c) 2017, 2020 IBM Corporation and others.
 4 * All rights reserved. This program and the accompanying materials
 5 * are made available under the terms of the Eclipse Public License v1.0
 6 * which accompanies this distribution, and is available at
 7 * http://www.eclipse.org/legal/epl-v10.html
 8 *
 9 * Contributors:
10 *     IBM Corporation - Initial implementation
11 *******************************************************************************/
12// end::copyright[]
13package io.openliberty.guides.inventory;
14
15import java.util.Properties;
16import javax.enterprise.context.RequestScoped;
17import javax.inject.Inject;
18import javax.ws.rs.*;
19import javax.ws.rs.core.MediaType;
20import javax.ws.rs.core.Response;
21
22import org.eclipse.microprofile.opentracing.Traced;
23
24import io.openliberty.guides.inventory.model.InventoryList;
25
26@RequestScoped
27@Path("/systems")
28// tag::InventoryResource[]
29public class InventoryResource {
30
31    @Inject InventoryManager manager;
32
33    @GET
34    @Path("/{hostname}")
35    @Produces(MediaType.APPLICATION_JSON)
36    // tag::getPropertiesForHost[]
37    public Response getPropertiesForHost(@PathParam("hostname") String hostname) {
38        Properties props = manager.get(hostname);
39        if (props == null) {
40            return Response.status(Response.Status.NOT_FOUND)
41                           .entity("{ \"error\" : \"Unknown hostname or the system service "
42                           + "may not be running on " + hostname + "\" }")
43                           .build();
44        }
45        manager.add(hostname, props);
46        return Response.ok(props).build();
47    }
48    // end::getPropertiesForHost[]
49
50    @GET
51    // tag::Traced-false[]
52    @Traced(false)
53    // end::Traced-false[]
54    @Produces(MediaType.APPLICATION_JSON)
55    // tag::listContents[]
56    public InventoryList listContents() {
57        return manager.list();
58    }
59    // end::listContents[]
60
61    @DELETE
62    @Produces(MediaType.APPLICATION_JSON)
63    public Response clearContents() {
64        int cleared = manager.clear();
65
66        if (cleared == 0) {
67            return Response.status(Response.Status.NOT_MODIFIED)
68                    .build();
69        }
70        return Response.status(Response.Status.OK)
71                .build();
72    }
73}
74// end::InventoryResource[]

Injecting a custom Tracer object

The MicroProfile OpenTracing specification also makes the underlying OpenTracing Tracer instance available for use. You can access the configured Tracer by injecting it into a bean by using the @Inject annotation from the Contexts and Dependency Injections API.

Inject the Tracer object into the inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java file. Then, use it to define a new child scope in the add() call.

Replace the InventoryManager class.
inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java

InventoryManager.java

 1//tag::copyright[]
 2/*******************************************************************************
 3* Copyright (c) 2017, 2020 IBM Corporation and others.
 4* All rights reserved. This program and the accompanying materials
 5* are made available under the terms of the Eclipse Public License v1.0
 6* which accompanies this distribution, and is available at
 7* http://www.eclipse.org/legal/epl-v10.html
 8*
 9* Contributors:
10*     IBM Corporation - Initial implementation
11*******************************************************************************/
12//end::copyright[]
13package io.openliberty.guides.inventory;
14
15import java.util.ArrayList;
16import java.util.Properties;
17import io.openliberty.guides.inventory.client.SystemClient;
18import io.openliberty.guides.inventory.model.InventoryList;
19import javax.enterprise.context.ApplicationScoped;
20import javax.inject.Inject;
21import java.util.List;
22import java.util.Collections;
23
24import org.eclipse.microprofile.opentracing.Traced;
25
26import io.opentracing.Scope;
27import io.opentracing.Tracer;
28import io.openliberty.guides.inventory.model.*;
29
30@ApplicationScoped
31// tag::InventoryManager[]
32public class InventoryManager {
33
34    private List<SystemData> systems = Collections.synchronizedList(new ArrayList<>());
35    private SystemClient systemClient = new SystemClient();
36    // tag::customTracer[]
37    @Inject Tracer tracer;
38    // end::customTracer[]
39
40    public Properties get(String hostname) {
41        systemClient.init(hostname, 9080);
42        Properties properties = systemClient.getProperties();
43
44        return properties;
45    }
46
47    public void add(String hostname, Properties systemProps) {
48        Properties props = new Properties();
49        props.setProperty("os.name", systemProps.getProperty("os.name"));
50        props.setProperty("user.name", systemProps.getProperty("user.name"));
51
52        SystemData system = new SystemData(hostname, props);
53        // tag::Add[]
54        if (!systems.contains(system)) {
55            // tag::Try[]
56            // tag::addSpan[]
57            try (Scope childScope = tracer.buildSpan("add() Span")
58                                              .startActive(true)) {
59            // end::addSpan[]
60                // tag::addToInvList[]
61                systems.add(system);
62                // end::addToInvList[]
63            // tag::endTry[]
64            }
65            // end::endTry[]
66            // end::Try[]
67        }
68        // end::Add[]
69    }
70
71    // tag::Traced[]
72    @Traced(operationName = "InventoryManager.list")
73    // end::Traced[]
74    // tag::list[]
75    public InventoryList list() {
76        return new InventoryList(systems);
77    }
78    // end::list[]
79
80    int clear() {
81        int propertiesClearedCount = systems.size();
82        systems.clear();
83        return propertiesClearedCount;
84    }
85}
86// end::InventoryManager[]

This try block is called a try-with-resources statement, meaning that the childScope object is closed at the end of the statement. It’s good practice to define custom spans inside such statements. Otherwise, any exceptions that are thrown before the span closes will leak the active span.

Go to the http://localhost:9081/inventory/systems/localhost URL and check your Jaeger server at the http://localhost:16686 URL. If you have the Jaeger UI open from a previous step, refresh the page. Select the inventory traces and click the Find Traces button.

Verify that there are three spans from inventory and one span from system:

Trace with custom span


This simple example shows what you can do with the injected Tracer object. More configuration options are available to you, including setting a timestamp for when a span was created and destroyed. However, these options require an implementation of their own, which doesn’t come as a part of the Jaeger user feature that is provided. In a real-world scenario, implement all the OpenTracing interfaces that you deem necessary, which might include the SpanBuilder interface. You can use this interface for span creation and customization, including setting timestamps.

SystemResource.java

 1// tag::copyright[]
 2/*******************************************************************************
 3 * Copyright (c) 2017, 2020 IBM Corporation and others.
 4 * All rights reserved. This program and the accompanying materials
 5 * are made available under the terms of the Eclipse Public License v1.0
 6 * which accompanies this distribution, and is available at
 7 * http://www.eclipse.org/legal/epl-v10.html
 8 *
 9 * Contributors:
10 *     IBM Corporation - Initial implementation
11 *******************************************************************************/
12// end::copyright[]
13package io.openliberty.guides.system;
14
15import java.util.Properties;
16
17// CDI
18import javax.enterprise.context.RequestScoped;
19import javax.ws.rs.GET;
20// JAX-RS
21import javax.ws.rs.Path;
22import javax.ws.rs.Produces;
23import javax.ws.rs.core.MediaType;
24
25@RequestScoped
26@Path("properties")
27public class SystemResource {
28
29  @GET
30  @Produces(MediaType.APPLICATION_JSON)
31  // tag::Properties[]
32  public Properties getProperties() {
33    return System.getProperties();
34  }
35  // end::Properties[]
36}

InventoryResource.java

 1// tag::copyright[]
 2/*******************************************************************************
 3 * Copyright (c) 2017, 2020 IBM Corporation and others.
 4 * All rights reserved. This program and the accompanying materials
 5 * are made available under the terms of the Eclipse Public License v1.0
 6 * which accompanies this distribution, and is available at
 7 * http://www.eclipse.org/legal/epl-v10.html
 8 *
 9 * Contributors:
10 *     IBM Corporation - Initial implementation
11 *******************************************************************************/
12// end::copyright[]
13package io.openliberty.guides.inventory;
14
15import java.util.Properties;
16import javax.enterprise.context.RequestScoped;
17import javax.inject.Inject;
18import javax.ws.rs.*;
19import javax.ws.rs.core.MediaType;
20import javax.ws.rs.core.Response;
21
22import org.eclipse.microprofile.opentracing.Traced;
23
24import io.openliberty.guides.inventory.model.InventoryList;
25
26@RequestScoped
27@Path("/systems")
28// tag::InventoryResource[]
29public class InventoryResource {
30
31    @Inject InventoryManager manager;
32
33    @GET
34    @Path("/{hostname}")
35    @Produces(MediaType.APPLICATION_JSON)
36    // tag::getPropertiesForHost[]
37    public Response getPropertiesForHost(@PathParam("hostname") String hostname) {
38        Properties props = manager.get(hostname);
39        if (props == null) {
40            return Response.status(Response.Status.NOT_FOUND)
41                           .entity("{ \"error\" : \"Unknown hostname or the system service "
42                           + "may not be running on " + hostname + "\" }")
43                           .build();
44        }
45        manager.add(hostname, props);
46        return Response.ok(props).build();
47    }
48    // end::getPropertiesForHost[]
49
50    @GET
51    // tag::Traced-false[]
52    @Traced(false)
53    // end::Traced-false[]
54    @Produces(MediaType.APPLICATION_JSON)
55    // tag::listContents[]
56    public InventoryList listContents() {
57        return manager.list();
58    }
59    // end::listContents[]
60
61    @DELETE
62    @Produces(MediaType.APPLICATION_JSON)
63    public Response clearContents() {
64        int cleared = manager.clear();
65
66        if (cleared == 0) {
67            return Response.status(Response.Status.NOT_MODIFIED)
68                    .build();
69        }
70        return Response.status(Response.Status.OK)
71                .build();
72    }
73}
74// end::InventoryResource[]

Testing the services

No automated tests are provided to verify the correctness of the traces. Manually verify these traces by viewing them on the Jaeger server.

A few tests are included for you to test the basic functionality of the services. If a test failure occurs, then you might have introduced a bug into the code.

Running the tests

Since you started Open Liberty in dev mode, run the tests for the system and inventory services by pressing the enter/return key in the command-line sessions where you started the services.

When you are done checking out the services, exit dev mode by pressing CTRL+C in the shell sessions where you ran the system and inventory services, or by typing q and then pressing the enter/return key.

Finally, stop the Jaeger service that you started in the Additional prerequisites section.

Great work! You’re done!

You just used MicroProfile OpenTracing in Open Liberty to customize how and which traces are delivered to Jaeger.

Try out one of the related MicroProfile guides. These guides demonstrate more technologies that you can learn to expand on what you built in this guide.

Guide Attribution

Enabling distributed tracing in microservices with Jaeger by Open Liberty is licensed under CC BY-ND 4.0

Copied to clipboard
Copy code block
Copy file contents

Prerequisites:

Nice work! Where to next?

What did you think of this guide?

Extreme Dislike Dislike Like Extreme Like

What could make this guide better?

Raise an issue to share feedback

Create a pull request to contribute to this guide

Need help?

Ask a question on Stack Overflow

Like Open Liberty? Star our repo on GitHub.

Star