Enable observability with MicroProfile Telemetry
With MicroProfile Telemetry 2.0 and later, you can manage logs, metrics, and traces in a standardized way by using the OpenTelemetry protocol. MicroProfile Telemetry is based on the OpenTelemetry project, a collection of open source vendor-independent tools, APIs, and SDKs for creating and managing logs, metrics, and trace data.
The following sections explain how to prepare your Open Liberty runtime and application code to use MicroProfile Telemetry.
Enabling MicroProfile Telemetry for Open Liberty
To enable MicroProfile Telemetry in your Open Liberty runtime, you must add the MicroProfile Telemetry feature to your server.xml
file and enable the OpenTelemetry SDK. OpenTelemetry is disabled by default. Depending on your needs, you can configure OpenTelemetry SDK instances at the runtime level or the application level. In most cases, runtime-level configuration is preferred as it includes both runtime-level telemetry and application-specific telemetry.
- Runtime-level configuration
Runtime-level configuration collects and emits telemetry from the runtime and all applications. As all telemetry from the runtime and applications is labeled as coming from the same service, this configuration works best when you are running a single application per Liberty runtime.
You can configure your runtime instance in a few different sources, including your
jvm.options
,bootstrap.properties
, orserver.env
files. You do not need to modify your application configuration to configure your runtime instance.- Application-level configuration
Configuring application-level OpenTelemetry instances gives you more fine-grained control over the data you collect for each application. This option also supports configuration compatibility with MicroProfile Telemetry 1.1 and earlier, which created only application-level SDK instances. Runtime-level data is not collected or emitted by application-level instances.
You can configure your application-level OpenTelemetry instances in the
microprofile-config.properties
file or theappProperties
attribute for the application element in theserver.xml
file. When you create application-level instances, any system and server environment properties overwrite your application-level configuration.
For more information about configuration sources and their ordinal values, see Default configuration sources.
To enable MicroProfile Telemetry to collect and emit logs, metrics, and traces, complete the following steps.
Enable the MicroProfile Telemetry feature in your
server.xml
file. To export metrics or logs, you must enablempTelemetry-2.0
or later.Set the
otel.sdk.disabled=false
property at the runtime level or the application level.At runtime initialization, if the
otel.sdk.disabled
property is set tofalse
, the runtime-level OpenTelemetry SDK instance is created. If the runtime instance is not enabled andotel.sdk.disabled=false
is specified at the application level, an application-level instance is created during application initialization.To enable the OpenTelemetry SDK at the runtime level, set the
otel.sdk.disabled=false
property as a system property, for example, in thebootstrap.properties
file:otel.sdk.disabled=false otel.service.name=RT1
The
otel.service.name
property creates a name for any telemetry that this OpenTelemetry SDK instance collects and emits.Alternatively, you can set runtime-level configuration properties by using environment variables in your
server.env
file. For any property definition, make the key name uppercase and convert any punctuation to underscores. For example, theotel.sdk.disabled=false
property is equivalent to theOTEL_SDK_DISABLED=false
environment variable.To enable the OpenTelemetry SDK at the application level, set the
otel.sdk.disabled=false
property in themicroprofile-config.properties
file of each application.Application 1
microprofile-config.properties
file:otel.sdk.disabled=false otel.service.name=A1
Application 2
microprofile-config.properties
file:otel.sdk.disabled=false otel.service.name=A2
This configuration creates all telemetry from Application 1 with the service name
A1
, and from Application 2 with the service nameA2
. It omits all runtime-level telemetry.Optionally, use a combination of application-level and runtime-level configuration.
Regardless of whether a runtime instance is created, any configuration in the system properties and server environment variables takes precedence over application-level configuration. Therefore, you can configure shared application configuration at the runtime level and make application-specific changes in your application configuration.
For example, you can set the OpenTelemetry exporter at the runtime-level in the
bootstrap.properties
file:otel.logs.exporter=myExporter
Then, enable an application-level OpenTelemetry SDK instance in the
microprofile-config.properties
file for each application in the runtime:otel.sdk.disabled=false otel.service.name=<serviceName>
This configuration enables an application-level SDK instance and custom service name for each configured application, while all applications in the runtime use the
myExporter
OpenTelemetry exporter.
Configuring Open Liberty to use MicroProfile Telemetry to collect traces
One way to increase observability of an application is by emitting traces. Traces represent requests and consist of multiple spans. A span represents a single operation in a request. It includes a name, time-related data, log messages, and metadata about what happens during a transaction.
MicroProfile Telemetry replaces MicroProfile OpenTracing. For more information about migrating your applications from MicroProfile OpenTracing to MicroProfile Telemetry, see Differences between MicroProfile Telemetry 1.0 and MicroProfile OpenTracing 3.0.
Enable the MicroProfile Telemetry feature and specify a MicroProfile Config property or environment variable to enable the OpenTelemetry SDK.
For more information, see Enabling MicroProfile Telemetry for Open Liberty
Configure a trace storage system by specifying an exporter definition that includes the exporter type and endpoint.
For example, to use a Jaeger server or to forward data to an OpenTelemetry collector, you might add configuration similar to the following example to your
bootstrap.properties
file:otel.sdk.disabled=false otel.traces.exporter=otlp otel.exporter.otlp.endpoint=http://localhost:4317/
Alternatively, to use a Zipkin server, you might add configuration similar to the following example to your
bootstrap.properties
file:otel.sdk.disabled=false otel.traces.exporter=zipkin otel.exporter.zipkin.endpoint=http://localhost:9411/api/v2/spans
Optionally, set other MicroProfile Config properties to configure trace details.
For example, if you want to export traces to Open Liberty log files, set the following property:
otel.traces.exporter=console
For more information about the available properties, see MicroProfile Config properties: MicroProfile Telemetry.
Depending on how you choose to instrument your application code for tracing, further configuration might be required.
For more information, see Code instrumentation for MicroProfile Telemetry tracing.
Configuring Open Liberty to use MicroProfile Telemetry to collect logs
To enable MicroProfile Telemetry to collect and export logs in your Open Liberty runtime, add the MicroProfile Telemetry feature 2.0 or later to your server.xml
file and enable the OpenTelemetry SDK. Optionally, you can specify MicroProfile Config properties to configure how MicroProfile Telemetry collects and exports logs.
Enable the MicroProfile Telemetry feature 2.0 or later and specify a MicroProfile Config property or environment variable to enable the OpenTelemetry SDK.
For more information, see Enabling MicroProfile Telemetry for Open Liberty.
Optionally, configure the log sources that MicroProfile Telemetry collects and where the logs are exported.
You can configure how MicroProfile Telemetry collects and exports logs by specifying configuration properties in any of the config sources that are available to MicroProfile Config, or by specifying environment variables.
Enable MicroProfile Telemetry to collect logs from different sources in the Open Liberty runtime environment.
Configure the
source
attribute for thempTelemetry
element with a comma-separated list of log sources:<mpTelemetry source="message, trace, ffdc"/>
The
mpTelemetry
configuration element is optional. If you do not specify it, or if you do not include thesource
attribute, the default configuration source ismessage
. For more information, see Collect logs from a specified source.Change the log exporter that MicroProfile Telemetry uses.
By default, all OpenTelemetry data is exported to OTLP. You can change this setting by specifying the
otel.logs.exporter
property.For example, to send logs to the
console.log
file for debugging purposes, you might add configuration similar to the following example to yourbootstrap.properties
file:otel.sdk.disabled=false otel.logs.exporter=console
If you set this property to
console
, all the logs are exported to standard out (stdout
) or theconsole.log
file. Theconsole.log
file contains the usual logs, along with duplicate OpenTelemetry-mapped logs. This setting is only for debugging purposes because theconsole.log
file does not roll over and might affect performance if it gets too large. If you set this property tonone
, no logs are exported.Configure MicroProfile Telemetry to use the OpenTelemetry Batch LogRecord Processor.
By default, the
SimpleLogRecordProcessor
is enabled, so the records are sent immediately. However, if you want to send the records in batches, you can also configure the following logging-specific Batch LogRecord Processor properties or environment variables:otel.blrp.schedule.delay
orOTEL_BLRP_SCHEDULE_DELAY
otel.blrp.max.queue.size
orOTEL_BLRP_MAX_QUEUE_SIZE
otel.blrp.max.export.batch.size
orOTEL_BLRP_MAX_EXPORT_BATCH_SIZE
otel.blrp.export.timeout
orOTEL_BLRP_EXPORT_TIMEOUT
For more information, see the OpenTelemetry Batch LogRecord Processor documentation.
For more information about the available properties, see MicroProfile Config properties: MicroProfile Telemetry.
For information about Liberty message event fields for MicroProfile Telemetry, see MicroProfile Telemetry log events reference list.
Configuring Open Liberty to use MicroProfile Telemetry to collect metrics
To enable MicroProfile Telemetry to collect and export metrics in your Open Liberty runtime, add the MicroProfile Telemetry 2.0 feature to your server.xml
file and enable the OpenTelemetry SDK. Optionally, you can specify MicroProfile Config properties to configure how MicroProfile Telemetry collects and exports metrics.
Enable the MicroProfile Telemetry feature 2.0 or later and specify a MicroProfile Config property or environment variable to enable the OpenTelemetry SDK.
For more information, see Enabling MicroProfile Telemetry for Open Liberty
Optionally, change the log exporter that MicroProfile Telemetry uses.
By default, all OpenTelemetry data is exported to OTLP. You can change this setting by specifying the
otel.metrics.exporter
property or theOTEL_METRICS_EXPORTER
environment variable.For example, to export metrics to Open Liberty log files, specify the following property:
otel.metrics.exporter=console
Optionally, change the metric export interval.
By default, metric data is exported at an interval of 60 seconds. To modify the export interval, specify the
otel.metric.export.interval
property or theOTEL_METRIC_EXPORT_INTERVAL
environment variable. Specify the value in milliseconds. For more information, see Periodic exporter MetricReader in the OpenTelemetry documentation.
Depending on how you choose to instrument your application code for metrics, further configuration might be required. For information about defining your own metrics, see Define custom MicroProfile Telemetry metrics.
For a list of metrics that are available for Open Liberty when you enable the MicroProfile Telemetry feature 2.0 or later, see MicroProfile Telemetry metrics reference list.
Troubleshooting MicroProfile Telemetry
The following information can help you determine the cause of common problems and error messages.
- Previous spans are incorrectly shown as current or parent spans
If the
Scope
instance is not closed correctly, the context and baggage values of previous spans might remain when the next operation executes. Alternatively, the current span might remain and be picked up as the parent of the next operation that executes.Always close the
Scope
instance when you exit an operation. This configuration stops the span from being current and makes the previous span current again. Use atry-with-resources
block, which automatically closes theScope
instance at the end of the block, as shown in the following example:Span span = tracer.spanBuilder("PerformingOperation").startSpan(); try (Scope scope = span.makeCurrent()) { ... } finally { span.end(); }
- You receive the
CWMOT5100I
message that tracing is disabled If you enable the
mpTelemetry-1.1
ormpTelemetry-1.0
feature, you must also set theotel.sdk.disabled=false
property in any of the configuration sources that are accessible through MicroProfile Config to enable tracing.- You receive the CWMOT5003W message that the application attempted to acquire MicroProfile Telemetry after shut down
Review the application to see why it attempted to use MicroProfile Telemetry after it shut down. Actions that might trigger MicroProfile Telemetry include calling a method that is annotated with
@WithSpan
or making a request with a JAX-RS Client or MP Rest Client.- You receive either of the CWMOT5006W or CWMOT5007 warning message that conflicting configuration is specified for otel.sdk.disabled
Specify the settings to enable or disable OpenTelemetry instances by using either environment variables or MicroProfile Config sources, but not both. If you see these warnings, the other MicroProfile Config source to look at is your
server.xml
file.