git clone https://github.com/openliberty/guide-microprofile-opentracing.git
cd guide-microprofile-opentracing
Enabling distributed tracing in microservices with Zipkin
Prerequisites:
Explore how to enable and customize tracing of JAX-RS and non-JAX-RS methods by using MicroProfile OpenTracing and the Zipkin tracing system.
What you’ll learn
You will learn how to enable automatic tracing for JAX-RS methods as well as create custom tracers for non-JAX-RS methods by using MicroProfile OpenTracing.
OpenTracing is a standard API for instrumenting microservices for distributed tracing. Distributed tracing helps troubleshoot microservices by examining and logging requests as they propagate through a distributed system, allowing developers to tackle the otherwise difficult task of debugging these requests. Without a distributed tracing system in place, analyzing the workflows of operations becomes difficult, particularly in regard to pinpointing when and by whom a request is received, as well as when a response is sent back.
MicroProfile OpenTracing enables distributed tracing in microservices without adding any explicit distributed tracing code to the application. Note that the MicroProfile OpenTracing specification does not address the problem of defining, implementing, or configuring the underlying distributed tracing system. Rather, the specification makes it easy to instrument services with distributed tracing given an existing distributed tracing system.
You will configure the provided inventory
and system
services to use distributed tracing with MicroProfile OpenTracing. You will run these services in two separate JVMs made of two server instances to demonstrate tracing in a distributed environment. If all the components were to run on a single server, then any logging software would do the trick.
Getting started
The fastest way to work through this guide is to clone the Git repository and use the projects that are provided inside:
The start
directory contains the starting project that you will build upon.
The finish
directory contains the finished project that you will build.
For this guide, use Zipkin as your distributed tracing system. You can find the installation instructions for Zipkin at the Zipkin quickstart page. You are not required to use Zipkin, but keep in mind that you might need more instructions that are not listed here if you choose to use another tracing system.
Before you proceed, make sure that your Zipkin server is up and running. By default, Zipkin can be found at the http://localhost:9411 URL.
Try what you’ll build
The finish
directory in the root directory of this guide contains two services that are configured to use MicroProfile OpenTracing. Give them a try before you proceed.
To try out the services, navigate to the finish
directory and run the Maven install
phase to build the services
cd finish
mvn install
then, run the Maven liberty:start-server
goal to start them in two Open Liberty servers:
mvn liberty:start-server
Make sure that your Zipkin server is running and point your browser to the http://localhost:9081/inventory/systems/localhost URL. When you visit this endpoint, you make two GET HTTP requests, one to the system
service and one to the inventory
service. Both of these requests are configured to be traced, so a new trace will be recorded in Zipkin. Visit the http://localhost:9411 URL or another location where you configured Zipkin to run and sort the traces by newest first. Verify that this new trace contains three spans with the following names:
get:io.openliberty.guides.inventory.inventoryresource.getpropertiesforhost
get:io.openliberty.guides.system.systemresource.getproperties
add() span
You can inspect each span by clicking it to reveal more detailed information, such as the time at which the request was received and the time at which a response was sent back.
If you examine the other traces, you might notice a red trace entry, which happens when an error is caught by the span. In this case, since one of the tests accesses the /inventory/systems/badhostname
endpoint, which is invalid, an error is thrown. This behavior is expected.
When you are done checking out the services, stop both Open Liberty servers using the Maven liberty:stop-server
goal:
mvn liberty:stop-server
Running the services
Navigate to the start
directory to begin.
You’ll need to start the services to see basic traces appear in Zipkin. So, before you proceed, build and start the provided system
and inventory
services in the starting project by running the Maven install
goal:
mvn install
then, run the liberty:start-server
goal:
mvn liberty:start-server
When the servers start, you can find the system
and inventory
services at the following URLs:
Existing Tracer implementation
To collect traces across your systems, you need to implement the OpenTracing Tracer
interface. For this guide, you can access a bare-bones Tracer
implementation for the Zipkin server in the form of a user feature for Open Liberty.
This feature is already configured for you in your pom.xml
and server.xml
files. It will be downloaded and installed automatically into each service when you run a Maven build. You can find the opentracingZipkin
feature enabled in your server.xml
file.
The download-maven-plugin
Maven plug-in in your pom.xml
is responsible for downloading and installing the feature.
If you want to install this feature yourself, see Enabling distributed tracing in the IBM Knowledge Centre.
server.xml
1<server description="Sample Liberty server">
2
3 <featureManager>
4 <feature>jaxrs-2.1</feature>
5 <feature>jsonp-1.1</feature>
6 <feature>cdi-2.0</feature>
7 <!-- tag::mpOpenTracing[] -->
8 <feature>mpOpenTracing-1.3</feature>
9 <!-- end::mpOpenTracing[] -->
10 <!-- tag::zipkinUsr[] -->
11 <feature>usr:opentracingZipkin-0.31</feature>
12 <!-- end::zipkinUsr[] -->
13 </featureManager>
14
15 <opentracingZipkin host="localhost"/>
16
17 <httpEndpoint httpPort="${default.http.port}" httpsPort="${default.https.port}"
18 id="defaultHttpEndpoint" host="*" />
19
20 <webApplication location="inventory-service.war" contextRoot="/">
21 <!-- enable visibility to third party apis -->
22 <classloader apiTypeVisibility="api,ibm-api,spec,stable,third-party"/>
23 </webApplication>
24
25</server>
pom.xml
1<?xml version='1.0' encoding='utf-8'?>
2<project xmlns="http://maven.apache.org/POM/4.0.0"
3xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
5http://maven.apache.org/xsd/maven-4.0.0.xsd">
6 <modelVersion>4.0.0</modelVersion>
7
8 <parent>
9 <groupId>io.openliberty.guides</groupId>
10 <artifactId>guide-microprofile-opentracing</artifactId>
11 <version>1.0-SNAPSHOT</version>
12 </parent>
13
14 <artifactId>inventory-service</artifactId>
15 <packaging>war</packaging>
16
17 <properties>
18 <!-- Plugins -->
19 <version.exec-maven-plugin>1.6.0</version.exec-maven-plugin>
20 </properties>
21
22 <dependencies>
23 <!-- Open Liberty features -->
24 <dependency>
25 <groupId>io.openliberty.features</groupId>
26 <artifactId>jaxrs-2.1</artifactId>
27 <type>esa</type>
28 </dependency>
29 <dependency>
30 <groupId>io.openliberty.features</groupId>
31 <artifactId>jsonp-1.1</artifactId>
32 <type>esa</type>
33 </dependency>
34 <dependency>
35 <groupId>io.openliberty.features</groupId>
36 <artifactId>cdi-2.0</artifactId>
37 <type>esa</type>
38 </dependency>
39 <dependency>
40 <groupId>io.openliberty.features</groupId>
41 <artifactId>mpOpenTracing-1.3</artifactId>
42 <type>esa</type>
43 </dependency>
44 <!-- For tests -->
45 <dependency>
46 <groupId>junit</groupId>
47 <artifactId>junit</artifactId>
48 </dependency>
49 <dependency>
50 <groupId>org.apache.cxf</groupId>
51 <artifactId>cxf-rt-rs-client</artifactId>
52 </dependency>
53 <dependency>
54 <groupId>org.apache.cxf</groupId>
55 <artifactId>cxf-rt-rs-extension-providers</artifactId>
56 </dependency>
57 <dependency>
58 <groupId>org.glassfish</groupId>
59 <artifactId>javax.json</artifactId>
60 </dependency>
61 <dependency>
62 <groupId>org.apache.commons</groupId>
63 <artifactId>commons-lang3</artifactId>
64 </dependency>
65 </dependencies>
66
67 <profiles>
68 <!-- Start system service before running tests -->
69 <profile>
70 <id>setup-test</id>
71 <activation>
72 <activeByDefault>true</activeByDefault>
73 </activation>
74 <build>
75 <plugins>
76 <plugin>
77 <groupId>org.codehaus.mojo</groupId>
78 <artifactId>exec-maven-plugin</artifactId>
79 <version>${version.exec-maven-plugin}</version>
80 <executions>
81 <execution>
82 <id>test-start-system-service</id>
83 <goals>
84 <goal>exec</goal>
85 </goals>
86 <phase>pre-integration-test</phase>
87 <configuration>
88 <workingDirectory>../system</workingDirectory>
89 <executable>mvn${mvn.extension}</executable>
90 <arguments>
91 <argument>liberty:start-server</argument>
92 </arguments>
93 </configuration>
94 </execution>
95 </executions>
96 </plugin>
97 </plugins>
98 </build>
99 </profile>
100 <!-- Stop system service after running tests -->
101 <profile>
102 <id>teardown-test</id>
103 <activation>
104 <activeByDefault>false</activeByDefault>
105 </activation>
106 <build>
107 <plugins>
108 <plugin>
109 <groupId>org.codehaus.mojo</groupId>
110 <artifactId>exec-maven-plugin</artifactId>
111 <version>${version.exec-maven-plugin}</version>
112 <executions>
113 <execution>
114 <id>test-stop-system-service</id>
115 <goals>
116 <goal>exec</goal>
117 </goals>
118 <phase>post-integration-test</phase>
119 <configuration>
120 <workingDirectory>../system</workingDirectory>
121 <executable>mvn${mvn.extension}</executable>
122 <arguments>
123 <argument>liberty:stop-server</argument>
124 </arguments>
125 </configuration>
126 </execution>
127 </executions>
128 </plugin>
129 </plugins>
130 </build>
131 </profile>
132 </profiles>
133
134 <build>
135 <plugins>
136 <plugin>
137 <groupId>org.apache.maven.plugins</groupId>
138 <artifactId>maven-war-plugin</artifactId>
139 </plugin>
140
141 <!-- Liberty plugin -->
142 <plugin>
143 <groupId>net.wasdev.wlp.maven.plugins</groupId>
144 <artifactId>liberty-maven-plugin</artifactId>
145 <configuration>
146 <configFile>src/main/liberty/config/server.xml</configFile>
147 <include>usr</include>
148 <bootstrapProperties>
149 <default.http.port>${inv.service.http.port}</default.http.port>
150 <default.https.port>${inv.service.https.port}</default.https.port>
151 <system.http.port>${sys.service.http.port}</system.http.port>
152 </bootstrapProperties>
153 </configuration>
154 <executions>
155 <execution>
156 <id>install-apps</id>
157 <configuration>
158 <appsDirectory>apps</appsDirectory>
159 <stripVersion>true</stripVersion>
160 <installAppPackages>project</installAppPackages>
161 <looseApplication>true</looseApplication>
162 </configuration>
163 </execution>
164 <execution>
165 <id>install-feature</id>
166 <configuration>
167 <skip>true</skip>
168 </configuration>
169 </execution>
170 </executions>
171 </plugin>
172
173 <!-- For downloading files -->
174 <!-- tag::download[] -->
175 <plugin>
176 <groupId>com.googlecode.maven-download-plugin</groupId>
177 <artifactId>download-maven-plugin</artifactId>
178 <version>${version.download-maven-plugin}</version>
179 <executions>
180 <execution>
181 <id>install-tracer</id>
182 <phase>prepare-package</phase>
183 <goals>
184 <goal>wget</goal>
185 </goals>
186 <configuration>
187 <url>${zipkin.usr.feature}</url>
188 <unpack>true</unpack>
189 <outputDirectory>
190 ${project.build.directory}/liberty/wlp/usr
191 </outputDirectory>
192 </configuration>
193 </execution>
194 </executions>
195 </plugin>
196 <!-- end::download[] -->
197
198 <!-- Plugin to run unit tests -->
199 <plugin>
200 <groupId>org.apache.maven.plugins</groupId>
201 <artifactId>maven-surefire-plugin</artifactId>
202 </plugin>
203
204 <!-- Plugin to run functional tests -->
205 <plugin>
206 <groupId>org.apache.maven.plugins</groupId>
207 <artifactId>maven-failsafe-plugin</artifactId>
208 </plugin>
209 </plugins>
210 </build>
211</project>
Enabling distributed tracing
The MicroProfile OpenTracing feature enables tracing of all JAX-RS methods by default. To further control and customize these traces, use the @Traced
annotation to enable and disable tracing of particular methods. You can also inject a custom Tracer
object to create and customize spans.
Enabling distributed tracing without code instrumentation
Because tracing of all JAX-RS methods is enabled by default, you need only to enable MicroProfile OpenTracing
feature and the Zipkin
user feature in the server.xml
file to see some basic traces in Zipkin.
Both of these features are already enabled in the inventory
and system
configuration files.
server.xml
1<server description="Sample Liberty server">
2
3 <featureManager>
4 <feature>jaxrs-2.1</feature>
5 <feature>jsonp-1.1</feature>
6 <feature>cdi-2.0</feature>
7 <!-- tag::mpOpenTracing[] -->
8 <feature>mpOpenTracing-1.3</feature>
9 <!-- end::mpOpenTracing[] -->
10 <!-- tag::zipkinUsr[] -->
11 <feature>usr:opentracingZipkin-0.31</feature>
12 <!-- end::zipkinUsr[] -->
13 </featureManager>
14
15 <opentracingZipkin host="localhost"/>
16
17 <httpEndpoint httpPort="${default.http.port}" httpsPort="${default.https.port}"
18 id="defaultHttpEndpoint" host="*" />
19
20 <webApplication location="inventory-service.war" contextRoot="/">
21 <!-- enable visibility to third party apis -->
22 <classloader apiTypeVisibility="api,ibm-api,spec,stable,third-party"/>
23 </webApplication>
24
25</server>
Make sure that your services are running. Then, simply point your browser to any of their endpoints and check your Zipkin server for traces.
Enabling explicit distributed tracing
Use the @Traced
annotation to define explicit span creation for specific classes and methods. If you place the annotation on a class, then it’s automatically applied to all methods within that class. If you place the annotation on a method, then it overrides the class annotation if one exists.
The @Traced
annotation can be configured with the following two parameters:
The
value=[true|false]
parameter indicates whether or not a particular class or method is traced. For example, while all JAX-RS methods are traced by default, you can disable their tracing by using the@Traced(false)
annotation. This parameter is set totrue
by default.The
operationName=<Span name>
parameter indicates the name of the span that is assigned to the particular method that is traced. If you omit this parameter, the span will be named with the following form:<package name>.<class name>.<method name>
. If you use this parameter at a class level, then all methods within that class will have the same span name unless they are explicitly overridden by another@Traced
annotation.
Update theInventoryManager
class.inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java
Enable tracing of the list()
non-JAX-RS method by updating @Traced
as shown.
InventoryManager.java
Next, run the following command from the start
directory to recompile your services.
mvn compile
Point to the http://localhost:9081/inventory/systems URL, check your Zipkin server, and sort the traces by newest first. You see a new trace record that is two spans long with one span for the listContents()
JAX-RS method in the InventoryResource
class and another span for the list()
method in the InventoryManager
class. Verify that these spans have the following names:
get:io.openliberty.guides.inventory.inventoryresource.listcontents
inventorymanager.list
Update theInventoryResource
classinventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java
Disable tracing of the listContents()
JAX-RS method by setting @Traced(false)
.
InventoryResource.java
Again, run the mvn compile
command from the start
directory to recompile your services:
mvn compile
Point to the http://localhost:9081/inventory/systems URL, check your Zipkin server, and sort the traces by newest first. You see a new trace record that is just one span long for the remaining list()
method in the InventoryManager
class. Verify that this span has the following name:
inventorymanager.list
Injecting a custom Tracer object
The MicroProfile OpenTracing specification also makes the underlying OpenTracing Tracer
instance available for use. You can access the configured Tracer
by injecting it into a bean by using the @Inject
annotation from the Contexts and Dependency Injections API.
Inject the Tracer
object into the inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java
file. Then, use it to define a new child scope in the add()
call.
Replace theInventoryManager
class.inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java
InventoryManager.java
The try
block that you see here is called a try-with-resources
statement, meaning that the childScope
object is closed at the end of the statement. It’s good practice to define custom spans inside such statements. Otherwise, any exceptions that are thrown before the span is closed will leak the active span.
Next, run the following command from the start
directory to recompile your services.
mvn compile
Point to the http://localhost:9081/inventory/systems/localhost URL, check your Zipkin server, and sort the traces by newest first. You see two new trace records, one for the system
service and one for the inventory
service. The system
trace contains one span for the getProperties()
method in the SystemResource
class. The inventory
trace contains two spans. The first span is for the getPropertiesForHost()
method in the InventoryResource
class. The second span is the custom span that you created around the add()
call. Verify that all of these spans have the following names:
The system
trace:
get:io.openliberty.guides.system.systemresource.getproperties
The inventory
trace:
get:io.openliberty.guides.inventory.inventoryresource.getpropertiesforhost
add() span
This simple example shows what you can do with the injected Tracer
object. More configuration options are available to you, including setting a timestamp for when a span was created and destroyed. However, these options require an implementation of their own, which does not come as a part of the Zipkin user feature that is provided. In a real-world scenario, implement all the OpenTracing interfaces that you deem necessary, which might include the SpanBuilder
interface. You can use this interface for span creation and customization, including setting timestamps.
SystemResource.java
InventoryResource.java
Testing the services
No automated tests are provided to verify the correctness of the traces. Manually verify these traces by viewing them on the Zipkin server.
A few tests are included for you to test the basic functionality of the services. If a test failure occurs, then you might have introduced a bug into the code. These tests will run automatically as a part of the Maven build process when you run the mvn install
command. You can also run these tests separately from the build by using the mvn verify
command, but first make sure that the servers are stopped.
When you are done checking out the services, stop the server by using the Maven liberty:stop-server
goal:
mvn liberty:stop-server
Great work! You’re done!
You have just used MicroProfile OpenTracing in Open Liberty to customize how and which traces are delivered to Zipkin.
Feel free to try one of the related MicroProfile guides. They demonstrate additional technologies that you can learn to expand on top of what you built here.
Guide Attribution
Enabling distributed tracing in microservices with Zipkin by Open Liberty is licensed under CC BY-ND 4.0
Prerequisites:
Nice work! Where to next?
What did you think of this guide?




Thank you for your feedback!
What could make this guide better?
Raise an issue to share feedback
Create a pull request to contribute to this guide
Need help?
Ask a question on Stack Overflow