Enable distributed tracing with MicroProfile Telemetry

In microservice applications, sources of latency or inaccuracy can be difficult to determine because relationships and dependencies among the constituent services are not always obvious. MicroProfile Telemetry helps you collect data on the paths that application requests take through services.

One way to increase observability of an application is by emitting traces. Traces represent requests and consist of multiple spans. A span represents a single operation in a request. It includes a name, time-related data, log messages, and metadata to collect data about what happens during a transaction.

MicroProfile Telemetry is based on the OpenTelemetry project, a collection of open source vendor-independent tools, APIs, and SDKs for creating and managing trace data. You can enable MicroProfile Telemetry for Open Liberty by adding the MicroProfile Telemetry feature to your server.xml file and configuring the feature to connect to your distributed trace service.

MicroProfile Telemetry replaces MicroProfile OpenTracing. For more information about migrating your applications from MicroProfile OpenTracing to MicroProfile Telemetry, see Differences between MicroProfile Telemetry 1.0 and MicroProfile OpenTracing 3.0.

The following sections explain how to prepare your Open Liberty runtime and application code to use MicroProfile Telemetry.

Configuring Open Liberty to use MicroProfile Telemetry

To enable MicroProfile Telemetry in your Open Liberty runtime, you must add the MicroProfile Telemetry feature to your server.xml file and specify MicroProfile Config properties to configure how MicroProfile Telemetry collects and exports traces.

  1. Add a version of the MicroProfile Telemetry feature to your server.xml file.

  2. Enable tracing and configure a trace storage system by specifying MicroProfile Config properties.

    You can configure how MicroProfile Telemetry collects and exports traces by specifying configuration properties in any of the config sources that are available to MicroProfile Config. If you choose to set these configuration properties by using environment variables, make the key name uppercase and convert any punctuation to underscores. For example, the otel.sdk.disabled=false property is equivalent to the OTEL_SDK_DISABLED=false environment variable.

    1. OpenTelemetry is disabled by default. To enable the generation of traces, set the otel.sdk.disabled=false property.

    2. Configure a trace storage system by specifying an exporter definition that includes the exporter type and the endpoint to connect to.

      For example, to use a Jaeger server, you might add configuration similar to the following example to your bootstrap.properties file:

      otel.sdk.disabled=false
otel.traces.exporter=otlp
otel.exporter.otlp.endpoint=http://localhost:4317/

      Alternatively, to use a Zipkin server, you might add configuration similar to the following example to your bootstrap.properties file:

      otel.sdk.disabled=false
otel.traces.exporter=zipkin
otel.exporter.zipkin.endpoint=http://localhost:9411/api/v2/spans

    3. Optionally, set other MicroProfile Config properties to configure trace details.

      For example, if you want to export traces to Open Liberty log files, set the following property:

      otel.traces.exporter=logging

      For more information about the available properties, see MicroProfile Config properties: MicroProfile Telemetry.

  3. Depending on how you choose to instrument your application code for tracing, further configuration might be required.

    For more information, see the following section.

Code instrumentation

After you enable MicroProfile Telemetry and configure a trace server, you can instrument tracing in your application code by using one of the following methods:

Automatic instrumentation

With automatic instrumentation, you can observe traces without modifying the source code in your applications. All you need to do is configure runtime as described in Configuring Open Liberty to use MicroProfile Telemetry. However, automatic instrumentation is available only for JAX-RS and Jakarta RESTful web service applications.

In Open Liberty version 23.0.0.11 and later, spans are automatically generated for incoming HTTP requests, including static files, servlets, and JSPs.

Manual instrumentation

Automatic instrumentation is available only for JAX-RS and Jakarta RESTful web service applications. To create spans for other operations, such as database calls, you can add manual instrumentation to the source code for those operations by using the OpenTelemetry API. However, before you manually instrument your code, you must complete the following prerequisites.

  1. Enable MicroProfile Telemetry, as described in Configuring Open Liberty to use MicroProfile Telemetry.

  2. Enable third-party APIs for your application by adding the following code in your server.xml file:

    <webApplication id="app-name" location="app-name.war">
    <classloader apiTypeVisibility="+third-party"/>
</webApplication>

  3. Add the opentelemetry API and OpenTelemetry instrumentation annotations as a provided dependency to your build path. For example, with Maven, add the following code to your pom.xml file.

        <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-api</artifactId>
        <version>1.19.0</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry.instrumentation</groupId>
        <artifactId>opentelemetry-instrumentation-annotations</artifactId>
        <version>1.19.0-alpha</version>
        <scope>provided</scope>
    </dependency>

After you complete those prerequisites, you’re ready to instrument your code. The following examples show configuration options with the OpenTelemetry API.

  • Add extra information, such as the user ID, to the current span. Any information that you add to a span is visible when you look at traces on your trace server.

    private static final AttributeKey<String> USER_ID_ATTR = AttributeKey.stringKey("userId");

@Inject private Span currentSpan;

@GET
public String myMethod() {
    ...
    currentSpan.setAttribute(USER_ID_ATTR, getUserId());
    ...
}

  • Create a subspan around a particular operation, such as querying a database. This subspan shows you how long it took and the order in which it occurred relative to other spans.

    @Inject private Tracer tracer;

@GET
public String myMethod() {
    ...
    Span newSpan = tracer.spanBuilder("QueryDatabase").startSpan();
    try (Scope s = newSpan.makeCurrent()) {
        queryDatabase();
    } finally {
        newSpan.end();
    }
    ...
}

  • Annotate methods in any Jakarta CDI beans by using the @WithSpan annotation. This annotation creates a new Span and establishes any required relationships with the current trace context. You can annotate method parameters with the @SpanAttribute annotation to indicate which method parameters are part of the trace, as shown in the following example.

    @ApplicationScoped
class SpanBean {

    @WithSpan("name")
    void spanName() {
       ...
    }

    @WithSpan
    void spanArgs(@SpanAttribute(value = "arg") String arg) {
       ...
    }
}

Considerations for using manual instrumentation

The following important considerations apply to manual instrumentation.

  • You must call the .end() method on any span you create, otherwise the span is not recorded.

  • The current span is used as the parent for any new spans that are created. Therefore, when you create a span, you usually also want to make it current. However, you must close the Scope instance that is returned by the Span.makeCurrent() method. You can close a Scope instance by specifying a try-with-resources block, as shown in the previous example for creating a subspan.

  • Because Liberty supports per-application configuration, it does not support GlobalOpenTelemetry. Using that class does not produce any telemetry data.

  • If you set any properties by using environment variables, including those in the server.env file, the keys must be in uppercase and all punctuation must be replaced by an underscore. Values must be written normally.

For more information, see the OpenTelemetry manual instrumentation documentation. However, remember when you use the MicroProfile Telemetry feature in Open Liberty, you must obtain the OpenTelemetry and Tracer objects by injecting them, not by creating your own. Furthermore, be aware that this documentation includes information for the OpenTelemetry Metrics and Logging APIs, which are not supported by MicroProfile Telemetry.

Agent instrumentation

The OpenTelemetry Instrumentation for Java project provides a Java agent JAR file that can be attached to any Java 8+ application. This file dynamically injects bytecode that adds telemetry support to popular open source libraries and frameworks. If you are using any of the supported libraries in your application, you can use this agent with Open Liberty to instrument them.

To enable the Java agent on your Open Liberty runtime, download the latest agent version from OpenTelemetry and add the following line to your jvm.options file.

-javaagent: path/to/opentelemetry-javaagent.jar

You can configure the agent with environment variables and system properties. You can find a list of supported libraries and frameworks in the OpenTelemetry Java instrumentation documentation.

Limitations of agent instrumentation

The OpenTelemetry Java agent is a tool that is provided by the OpenTelemetry project. Although it is compatible with Open Liberty, it is a separate project and is subject to the following limitations.

  • Configuration works differently when you use the agent. Configuration of the agent is well documented, but the following aspects are different from configuration without the agent:

    • Configuration is shared between all applications that are deployed to the server.

    • Configuration properties are only read from system properties and environment variables. They are not read from MicroProfile Config configuration sources.

    • Because the agent reads its configuration early in the startup process, system properties are not read from the bootstrap.properties file. Alternatively, you can set system properties in the jvm.options file by using the following syntax: -Dname=value

    • Implementations of SPI extensions within applications are ignored. For more information, see the agent documentation for providing SPI extensions.

  • When you use the agent, it takes over the instrumentation of REST calls and methods that are annotated with @WithSpan. As a result, the created spans might be slightly different.

  • The agent is not compatible with Java 2 security.

  • Open Liberty uses many open source libraries internally. Some of these libraries might be automatically instrumented by the agent.

Troubleshooting MicroProfile Telemetry

The following information can help you determine the cause of common problems and error messages.

Previous spans are incorrectly shown as current or parent spans

If the Scope instance is not closed correctly, the context and baggage values of previous spans might remain when the next operation executes. Alternatively, the current span might remain and be picked up as the parent of the next operation that executes.

Always close the Scope instance when you exit an operation. This configuration stops the span from being current and makes the previous span current again. Use a try-with-resources block, which automatically closes the Scope instance at the end of the block, as shown in the following example:

Span span = tracer.spanBuilder("PerformingOperation").startSpan();
try (Scope scope = span.makeCurrent()) {
    ...
} finally {
    span.end();
}
You receive the CWMOT5100I message that tracing is disabled

If you enable the mpTelemetry-1.1 or mpTelemetry-1.0 feature, you must also set the otel.sdk.disabled=false property in any of the configuration sources that are accessible through MicroProfile Config to enable tracing.

You receive the CWMOT5003W message that the application attempted to acquire MicroProfile Telemetry after shut down

Review the application to see why it attempted to use MicroProfile Telemetry after it shut down. Actions that might trigger MicroProfile Telemetry include calling a method that is annotated with @WithSpan or making a request with a JAX-RS Client or MP Rest Client.