Enabling distributed tracing in microservices with OpenTelemetry and Jaeger
Prerequisites:
Distributed tracing helps teams keep track of requests between microservices. MicroProfile Telemetry adopts OpenTelemetry tracing, so you can observe requests across your distributed systems.
What you’ll learn
The complexity of microservices architecture can make it more difficult to understand how services depend on or affect each other and to identify sources of latency or inaccuracies.
One way to increase the observability of an application is by emitting traces. OpenTelemetry is a set of APIs, SDKs, tooling, and integrations designed to create and manage telemetry data such as traces, metrics, and logs. MicroProfile Telemetry adopts OpenTelemetry so your Java applications can benefit from both manual and automatic traces.
Traces represent requests, which can contain multiple operations or spans. Each span comprises a name, time-related data, log messages, and metadata that describe what occurred during a transaction. Spans are associated with a context, which identifies the request within which the span occurred. Developers can then follow a single request between services through a potentially complex distributed system. Exporters send the data that MicroProfile Telemetry collects to Jaeger so you can visualize and monitor the generated spans.
The diagram shows multiple services, which is where distributed tracing is valuable. However, for simplicity, in this guide, you’ll configure only the system
and inventory
services to use Jaeger for distributed tracing with MicroProfile Telemetry. You’ll run these services in two separate JVMs made of two Open Liberty instances to demonstrate tracing in a distributed environment.
Additional prerequisites
Before you begin, deploy the Jaeger all-in-one executable file to start the Jaeger tracing system. The Jaeger all-in-one executable file is configured for quick local testing. Jaeger 1.46 or later is recommended. You can find information about the Jaeger server and instructions for starting the all-in-one executable file in the Jaeger documentation.
Before you proceed, make sure that your Jaeger server is up and running by visiting the http://localhost:16686 URL.
Getting started
The fastest way to work through this guide is to clone the Git repository and use the projects that are provided inside:
git clone https://github.com/openliberty/guide-microprofile-telemetry-jaeger.git
cd guide-microprofile-telemetry-jaeger
The start
directory contains the starting project that you will build upon.
The finish
directory contains the finished project that you will build.
Before you begin, make sure you have all the necessary prerequisites.
Try what you’ll build
The finish
directory in the root of this guide contains the finished application. Give it a try before you proceed.
Open a command-line session and navigate to the finish/system
directory. Run the following Maven goal to build the system
service and deploy it to Open Liberty:
mvn liberty:run
Open another command-line session and navigate to the finish/inventory
directory. Run the following Maven goal to build the inventory
service and deploy it to Open Liberty:
mvn liberty:run
After you see the following message in both command-line sessions, both of your services are ready:
The defaultServer server is ready to run a smarter planet.
Make sure that your Jaeger server is running and navigate your browser to the http://localhost:9081/inventory/systems/localhost URL.
When you visit this endpoint, you make two GET HTTP requests, one to the system
service and another to the inventory
service. Both of these requests are configured to be traced, so a new trace is recorded in Jaeger. To view the traces, go to the http://localhost:16686 URL.
You can view the traces for the system
or inventory
services under the Search tab. If you see only the jaeger-query option in the drop-down menu, wait a little longer and refresh the page to see the application services.
Select the services in the Select A Service menu and click the Find Traces button at the end of the section. You will see the following result:
The trace has five spans, four from the inventory
service and one from the system
service. Click the trace to view its details. Under Service & Operation, you see the spans in this trace. You can inspect each span by clicking it to reveal more detailed information, such as the times that a request was received and a response was sent.
After you’re finished reviewing the application, stop the Open Liberty instances by pressing CTRL+C
in the command-line sessions where you ran the system
and inventory
services. Alternatively, you can run the following goals from the finish
directory in another command-line session:
mvn -pl system liberty:stop mvn -pl inventory liberty:stop
Building the application
You need to start the services to see basic traces appear in Jaeger.
When you run Open Liberty in dev mode, dev mode listens for file changes and automatically recompiles and deploys your updates whenever you save a new change.
Open a command-line session and navigate to the start/system
directory. Run the following Maven goal to start the system
service in dev mode:
mvn liberty:dev
Open a command-line session and navigate to the start/inventory
directory. Run the following Maven goal to start the inventory
service in dev mode:
mvn liberty:dev
After you see the following message, your Liberty instance is ready in dev mode:
************************************************************** * Liberty is running in dev mode.
Dev mode holds your command-line session to listen for file changes. Open another command-line session to continue, or open the project in your editor.
When the runtime instances start, you can find the system
and inventory
services at the following URLs:
Enabling Telemetry implementation
Navigate to the start
directory to begin.
MicroProfile Telemetry allows you to observe traces without modifying the source code in your Jakarta RESTful applications. You can enable the mpTelemetry
feature in the server.xml
configuration file.
Replace theserver.xml
file of the system service:system/src/main/liberty/config/server.xml
system/server.xml
1<server description="system service">
2
3 <featureManager>
4 <feature>cdi-4.0</feature>
5 <feature>jsonb-3.0</feature>
6 <feature>jsonp-2.1</feature>
7 <feature>restfulWS-3.1</feature>
8 <!-- tag::mpTelemetry[] -->
9 <feature>mpTelemetry-1.1</feature>
10 <!-- end::mpTelemetry[] -->
11 </featureManager>
12
13 <httpEndpoint httpPort="${http.port}"
14 httpsPort="${https.port}"
15 id="defaultHttpEndpoint" host="*" />
16
17 <webApplication location="guide-microprofile-telemetry-system.war"
18 contextRoot="/" />
19
20</server>
The mpTelemetry
feature is now enabled in the server.xml
of the system
service.
Replace theserver.xml
file of the inventory service:inventory/src/main/liberty/config/server.xml
inventory/server.xml
The mpTelemetry
feature is now enabled in the server.xml
of the inventory
service.
By default, MicroProfile Telemetry tracing is off. To enable any tracing aspects, specify the otel
properties in the MicroProfile configuration file.
Create themicroprofile-config.properties
file of the system service:system/src/main/resources/META-INF/microprofile-config.properties
system/microprofile-config.properties
1# tag::service[]
2otel.service.name=system
3# end::service[]
4# tag::disabled[]
5otel.sdk.disabled=false
6# end::disabled[]
The MicroProfile properties file sets the otel.service.name
property with the system
service name and sets the otel.sdk.disabled
property to false
to enable tracing.
Replace themicroprofile-config.properties
file of the inventory service:inventory/src/main/resources/META-INF/microprofile-config.properties
inventory/microprofile-config.properties
1io.openliberty.guides.inventory.client.SystemClient/mp-rest/url=http://localhost:9080/system
2# tag::otel[]
3otel.service.name=inventory
4otel.sdk.disabled=false
5# end::otel[]
Similarly, specify the otel
properties for the inventory
service.
For more information about these and other Telemetry properties, see the MicroProfile Config properties for MicroProfile Telemetry documentation.
To run the system
and inventory
services, simply navigate your browser to the http://localhost:9081/inventory/systems/localhost URL. To view the traces, go to the http://localhost:16686 URL.
You can view the traces for the system
or inventory
services under the Search tab. Select the services in the Select A Service menu and click the Find Traces button at the end of the section. You’ll see the result as:
Verify that there are two spans from the inventory
service and one span from the system
service. Click the trace to view its details.
Enabling explicit distributed tracing
Automatic instrumentation only instruments Jakarta RESTful web services and MicroProfile REST clients. To get further spans on other operations, such as database calls, you can add manual instrumentation to the source code.
Enabling OpenTelemetry APIs
The MicroProfile Telemetry feature has been enabled to trace all REST endpoints by default in the previous section. To further control and customize traces, use the @WithSpan
annotation to enable particular methods. You can also inject a Tracer
object to create and customize spans.
Replace thepom.xml
Maven project file of the inventory service:inventory/pom.xml
pom.xml
1<?xml version='1.0' encoding='utf-8'?>
2<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
3http://maven.apache.org/xsd/maven-4.0.0.xsd">
4 <modelVersion>4.0.0</modelVersion>
5
6 <groupId>io.openliberty.guides</groupId>
7 <artifactId>guide-microprofile-telemetry-inventory</artifactId>
8 <version>1.0-SNAPSHOT</version>
9 <packaging>war</packaging>
10
11 <properties>
12 <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
13 <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
14 <maven.compiler.source>11</maven.compiler.source>
15 <maven.compiler.target>11</maven.compiler.target>
16
17 <!-- OpenLiberty runtime -->
18 <liberty.var.system.http.port>9080</liberty.var.system.http.port>
19 <liberty.var.http.port>9081</liberty.var.http.port>
20 <liberty.var.https.port>9444</liberty.var.https.port>
21 </properties>
22
23 <dependencies>
24 <!-- Provided dependencies -->
25 <dependency>
26 <groupId>jakarta.platform</groupId>
27 <artifactId>jakarta.jakartaee-api</artifactId>
28 <version>10.0.0</version>
29 <scope>provided</scope>
30 </dependency>
31 <dependency>
32 <groupId>org.eclipse.microprofile</groupId>
33 <artifactId>microprofile</artifactId>
34 <version>6.1</version>
35 <type>pom</type>
36 <scope>provided</scope>
37 </dependency>
38 <!-- tag::opentelemetry[] -->
39 <dependency>
40 <groupId>io.opentelemetry</groupId>
41 <artifactId>opentelemetry-api</artifactId>
42 <version>1.39.0</version>
43 <scope>provided</scope>
44 </dependency>
45 <dependency>
46 <groupId>io.opentelemetry.instrumentation</groupId>
47 <artifactId>opentelemetry-instrumentation-annotations</artifactId>
48 <version>2.5.0</version>
49 <scope>provided</scope>
50 </dependency>
51 <!-- end::opentelemetry[] -->
52 <!-- For tests -->
53 <dependency>
54 <groupId>org.junit.jupiter</groupId>
55 <artifactId>junit-jupiter</artifactId>
56 <version>5.11.1</version>
57 <scope>test</scope>
58 </dependency>
59 <dependency>
60 <groupId>org.jboss.resteasy</groupId>
61 <artifactId>resteasy-client</artifactId>
62 <version>6.2.10.Final</version>
63 <scope>test</scope>
64 </dependency>
65 <dependency>
66 <groupId>org.jboss.resteasy</groupId>
67 <artifactId>resteasy-json-binding-provider</artifactId>
68 <version>6.2.10.Final</version>
69 <scope>test</scope>
70 </dependency>
71 <dependency>
72 <groupId>org.glassfish</groupId>
73 <artifactId>jakarta.json</artifactId>
74 <version>2.0.1</version>
75 <scope>test</scope>
76 </dependency>
77 <dependency>
78 <groupId>org.eclipse</groupId>
79 <artifactId>yasson</artifactId>
80 <version>3.0.4</version>
81 <scope>test</scope>
82 </dependency>
83 </dependencies>
84
85 <build>
86 <finalName>${project.artifactId}</finalName>
87 <plugins>
88 <plugin>
89 <groupId>org.apache.maven.plugins</groupId>
90 <artifactId>maven-war-plugin</artifactId>
91 <version>3.4.0</version>
92 </plugin>
93
94 <!-- Liberty plugin -->
95 <plugin>
96 <groupId>io.openliberty.tools</groupId>
97 <artifactId>liberty-maven-plugin</artifactId>
98 <version>3.10.3</version>
99 </plugin>
100
101 <plugin>
102 <groupId>org.apache.maven.plugins</groupId>
103 <artifactId>maven-failsafe-plugin</artifactId>
104 <version>3.5.0</version>
105 <configuration>
106 <systemPropertyVariables>
107 <sys.http.port>${liberty.var.system.http.port}</sys.http.port>
108 <inv.http.port>${liberty.var.http.port}</inv.http.port>
109 </systemPropertyVariables>
110 </configuration>
111 </plugin>
112 </plugins>
113 </build>
114</project>
The OpenTelemetry API and OpenTelemetry Instrumentation Annotations must be provided as dependencies to your build path. The pom.xml
now includes two io.opentelemetry
dependencies.
Replace theserver.xml
file of the inventory service:inventory/src/main/liberty/config/server.xml
inventory/server.xml
1<server description="inventory service">
2
3 <featureManager>
4 <feature>cdi-4.0</feature>
5 <feature>jsonb-3.0</feature>
6 <feature>jsonp-2.1</feature>
7 <feature>restfulWS-3.1</feature>
8 <feature>mpConfig-3.1</feature>
9 <!-- tag::mpTelemetry[] -->
10 <feature>mpTelemetry-1.1</feature>
11 <!-- end::mpTelemetry[] -->
12 </featureManager>
13
14 <httpEndpoint httpPort="${http.port}"
15 httpsPort="${https.port}"
16 id="defaultHttpEndpoint" host="*" />
17
18 <webApplication location="guide-microprofile-telemetry-inventory.war"
19 contextRoot="/">
20 <!-- tag::thirdPartyComment[] -->
21 <!-- enable visibility to third party apis -->
22 <!-- end::thirdPartyComment[] -->
23 <!-- tag::thirdParty[] -->
24 <classloader apiTypeVisibility="+third-party"/>
25 <!-- end::thirdParty[] -->
26 </webApplication>
27
28</server>
The OpenTelemetry APIs are exposed as third-party APIs in Open Liberty. To add the visibility of OpenTelemetry APIs to the application, add third-party
to the types of API packages that this class loader supports. Instead of explicitly configuring a list of API packages that includes third-party
, set the +third-party
value to the apiTypeVisibility
attribute in the classLoader
configuration. This configuration adds third-party
to the default list of API package types that are supported.
Enabling tracing in Jakarta CDI beans
You can trace your Jakarta CDI beans by annotating their methods with a @WithSpan
annotation.
Replace theInventoryManager
class:inventory/src/main/java/io/openliberty/guides/inventory/InventoryManager.java
InventoryManager.java
The list()
and add()
methods are annotated with the @WithSpan
annotation, which can accept an optional parameter that functions as the span name. In this example, the default span name assigned to the list()
method is automatically generated through the instrumentation. You can also specify a custom span name. For example, Inventory Manager Add
is specified as the span name for the add()
method. The OpenTelemetry instrumentation provides a new span for each method. You can now collect and trace the spans across different services.
Optionally, you can include parameters and their values in the span by using the @SpanAttribute
annotation. For example, the @SpanAttribute
annotation specifies hostname
as the attribute name for the host
parameter , which helps trace the parameter within the add
span.
To learn more about how to use OpenTelemetry annotations to instrument code, see the OpenTelemetry Annotations documentation.
Now, you can check out the traces that are generated by the @WithSpan
annotation. Visit the http://localhost:9081/inventory/systems URL and then your Jaeger server at the http://localhost:16686 URL.
Select the inventory
service and click the Find Traces button at the end of the section. You’ll see the result as:
Verify that there are two spans from the inventory
service. Click the trace to view its details. You’ll see the InventoryManager.list
span that is created by the @WithSpan
annotation.
To check out the information generated by the @SpanAttribute
annotation, visit the http://localhost:9081/inventory/systems/localhost URL and then your Jaeger server at the http://localhost:16686 URL.
Select the inventory
service and click the Find Traces button at the end of the section. You will see the following result:
Verify that there are three spans from the inventory
service and one span from the system
service. Click the trace to view its details.
Click the Inventory Manager Add
span and its Tags
. You can see the hostname
tag with the localhost
value that is created by the @SpanAttribute
annotation.
Injecting a custom Tracer object
The MicroProfile Telemetry specification makes the underlying OpenTelemetry Tracer instance available. The configured Tracer is accessed by injecting it into a bean. You can use it to instrument your code to create traces.
Replace theInventoryResource
class:inventory/src/main/java/io/openliberty/guides/inventory/InventoryResource.java
InventoryResource.java
To access the Tracer, the @Inject
annotation from the Contexts and Dependency Injections API injects the Tracer into a bean.
Before the InventoryManager
calls the system
service, it creates and starts a span called the GettingProperties
by using the spanBuilder()
and startSpan()
Tracer APIs.
When you start a span, you must also end it by calling end()
on the span. If you don’t end a span, it won’t be recorded at all and won’t show up in Jaeger. This code ensures that end()
is always called by including it in a finally
block.
After you start the span, make it current with the makeCurrent()
call. Making a span current means that any new spans created in the same thread, either automatically by Open Liberty or manually by calling the API, will use this span as their parent span.
The makeCurrent()
call returns a Scope
. Make sure to always close the Scope
, which stops the span from being current and makes the previous span current again. Use a try-with-resources
block, which automatically closes the Scope
at the end of the block.
Use the addEvent()
Span API to create an event when the properties are received and an event when it fails to get the properties from the system
service. Use the end()
Span API to mark the GettingProperties
span as completed.
To check out the traces that contain the GettingProperties
span, visit the http://localhost:9081/inventory/systems/localhost URL and then your Jaeger server at the http://localhost:16686 URL.
Select the inventory
service and click the Find Traces button at the end of the section. You’ll see the result:
Verify that there are four spans from the inventory
service and one span from the system
service. Click the trace to view its details. You’ll see the GettingProperties
span.
To check out the event adding to the GettingProperties
span, visit the http://localhost:9081/inventory/systems/unknown URL and then your Jaeger server at the http://localhost:16686 URL.
Select the inventory
service and click the Find Traces button at the end of the section. You will see the following result:
There are two spans from the inventory
service. Click the trace to view its details. You’ll see the GettingProperties
span. Click the GettingProperties
span and its Logs
. You can see the Cannot get properties
message.
To learn more about how to use OpenTelemetry APIs to instrument code, see the OpenTelemetry Manual Instrumentation documentation.
Testing the application
Manually verify the traces by inspecting them on the Jaeger server. You will find some tests included to test the basic functionality of the services. If any of the tests fail, you might have introduced a bug into the code.
Running the tests
Since you started Open Liberty in dev mode, run the tests for the system
and inventory
services by pressing the enter/return
key in the command-line sessions where you started the services.
When you are done checking out the services, exit dev mode by pressing CTRL+C
in the shell sessions where you ran the system
and inventory
services.
Finally, stop the Jaeger
service that you started in the Additional prerequisites section.
Great work! You’re done!
You just used MicroProfile Telemetry in Open Liberty to customize how and which traces are delivered to Jaeger.
Try out one of the related MicroProfile guides. These guides demonstrate more technologies that you can learn to expand on what you built in this guide.
Guide Attribution
Enabling distributed tracing in microservices with OpenTelemetry and Jaeger by Open Liberty is licensed under CC BY-ND 4.0
Prerequisites:
Nice work! Where to next?
What did you think of this guide?
Thank you for your feedback!
What could make this guide better?
Raise an issue to share feedback
Create a pull request to contribute to this guide
Need help?
Ask a question on Stack Overflow