Microservice observability with metrics

Open Liberty uses MicroProfile Metrics to provide metrics that describe the internal state of many Open Liberty components. MicroProfile Metrics provides a /metrics endpoint from which you can access all metrics that are emitted by an Open Liberty runtime and deployed applications.

When MicroProfile Metrics is enabled and an application is running, you can view metric data from any browser by accessing the /metrics endpoint. An example of this endpoint is https://localhost:9443/metrics, where 9443 is the port number for your application.

You can narrow the scope of the metrics by using the scope query parameter with the /metrics endpoint to specify the metric registry scope. For example, to view the metric data for the base metric registry, use the /metrics?scope=base endpoint. This pattern is used to access all runtime-provided scopes (base, application, and vendor) and any user-defined metric registry scopes.

For MicroProfile Metrics 4.0 and earlier, only runtime-provided scopes are available and path parameters are used to narrow the scope of metric data. The endpoints are /metrics/base, /metrics/application, and /metrics/vendor.

For more information about metrics endpoints, see MicroProfile Metrics and the metrics endpoint.

By default, metric data is emitted in Prometheus format. Operations teams can gather the metrics and store them in a database by using tools like Prometheus. They can then visualize the metric data for analysis in dashboards such as Grafana. Starting in version 5.0, MicroProfile Metrics uses embedded Micrometer metrics technology so you can send your metrics data to the third-party monitoring tool of your choice.

For a list of metrics that are available in Open Liberty, see the Metrics reference list.

Adding metrics to your applications

You can use MicroProfile Metrics with Open Liberty by enabling the MicroProfile Metrics feature in your server.xml file. To add metrics to your applications, you must create and register metrics with the default application registry or your own user-defined metric registry scope. When metrics are registered, they’re known to the system and can be reported on from the /metrics endpoint. The simplest way to add metrics to your application is by using annotations. MicroProfile Metrics defines annotations that you can use to quickly build metrics into your code.

User-defined metric registry scopes

Starting in MicroProfile Metrics 5.0, in addition to the default application metric registry scope, you can define your own metric registry scope to register metrics to. Define your own scope either by specifying the scope parameter for metric annotations or by injecting a MetricRegistry class with the RegistryScope annotation and specifying the scope parameter. If you don’t define a value for the scope parameter in the metric annotation or RegistryScope annotation, the default scope is application.

The following example uses the @Counted metric annotation to define a customScope user-defined scope.

@Counted(name="customCounter" description="This counter resides in a user-defined metric registry", scope="customScope")
public String countingThings() {
    return "1 2 3 4..!";
}

The following example injects a MetricRegistry class that uses the customScope user-defined scope to instrument metrics with the Java API.

@Inject
@RegistryScope(scope="customScope")
MetricRegistry customMetricRegistry;

Metric Tags

With MicroProfile metrics, you can logically group metrics that track similar statistics by instrumenting the metrics with tags. Tags consist of a name and a value. You can group multiple metrics of the same type together by instrumenting them with tags of the same name but different values to group them together.

For example, when you use counters, you can create multiple counters that are named methodCounter that return the count of the method that is identified by the tag.

@Counted(name="methodCounter", tags={"method=foo"})
public void foo() {
    //foo something
}

@Counted(name="methodCounter", tags={"method=bar"})
public void bar() {
    //bar something
}

However, in MicroProfile Metrics 5.0 and later, the set of tag names that are used must be identical across all metric IDs with the same metric name. Otherwise, an IllegalArgumentException is thrown. The final methodCounter instance in the following example has an inconsistent extraTag tag name and leads to an IllegalArgumentException.

@Counted(name="methodCounter", tags={"method=foo"})
public void foo() {
    //foo something
}

@Counted(name="methodCounter", tags={"method=bar"})
public void bar() {
    //bar something
}
@Counted(name="methodCounter", tags={"method=bad", "extraTag=bad"})
public void bad() {
    // bad!
}

Metric types and annotations

The following sections describe metric types that are available and how their corresponding annotations are used. Some metric types are only available in certain versions of MicroProfile Metrics and others might behave slightly differently in MicroProfile Metrics 5.0 than in earlier versions. Details are provided in the section for each metric type. For more information, see Differences between MicroProfile Metrics 5.0 and 4.0.

Timer

A timer metric aggregates timing durations in nanoseconds and provides duration statistics.

Use the @Timed annotation to mark a constructor or method as a timer. The timer tracks how frequently the annotated object is started and how long the invocations take to complete, as shown in the following example.

@POST
@Path("/creditcard")
@Timed(
    name="donateAmountViaCreditCard.timer",
    description = "Donations that were made using a credit card")
public String donateAmountViaCreditCard(@FormParam("amount") Long amount, @FormParam("card") String card) {

    if (processCard(card, amount))
        return "Thanks for donating!";

    return "Sorry, please try again.";
}

Starting in MicroProfile Metrics 5.0, you can adjust the percentile precision of the Timer metrics by using the mp.metrics.smallrye.timer.precision MicroProfile Config property. The property accepts a value from 1 to 5 and is defaulted to 3 if no value is specified. A greater value results in more exact percentile calculations, but at a greater memory cost. For more information, see MicroProfile Config properties: MicroProfile Metrics.

Histogram

A histogram is a metric that calculates the distribution of a value. It provides the following information:

  • Maximum, median, and mean values

  • The value at the 50th, 75th, 95th, 98th, 99th, 99.9th percentile

  • A count of the number of values

  • Standard deviation for the value (MicroProfile Metrics 1.0-4.0 only)

Note: When you view the Prometheus-formatted metric data for a histogram, the mean value is not included.

The histogram metric does not have an annotation. To record a value in the histogram, you must call the histogram.update(long value) method with the value that you want to record. The current state, or snapshot, of the histogram can be retrieved by using the getSnapshot() method. Histograms in MicroProfile Metrics support only integer or long values.

The following example illustrates a histogram that is used to store the value of donations. This example provides the administrator with an idea of the distribution of donation amounts:

Metadata donationDistributionMetadata = Metadata.builder()
              .withName("donationDistribution")                             // name
              .withDescription("The distribution of the donation amounts")  // description
              .withUnit("Dollars")                                          // units
              .build();
Histogram donationDistribution = registry.histogram(donationDistributionMetadata);
public void addDonation(Long amount) {
    totalDonations += amount;
    donations.add(amount);
    donationDistribution.update(amount);
}

For this example, the following response is generated from the REST endpoints in Prometheus format:

# HELP donationDistribution_Dollars The distribution of the donation amounts
# TYPE donationDistribution_Dollars summary
donationDistribution_Dollars{mp_scope="application",tier="integration",quantile="0.5",} 431.248046875
donationDistribution_Dollars{mp_scope="application",tier="integration",quantile="0.75",} 695.498046875
donationDistribution_Dollars{mp_scope="application",tier="integration",quantile="0.95",} 914.498046875
donationDistribution_Dollars{mp_scope="application",tier="integration",quantile="0.98",} 977.498046875
donationDistribution_Dollars{mp_scope="application",tier="integration",quantile="0.99",} 991.498046875
donationDistribution_Dollars{mp_scope="application",tier="integration",quantile="0.999",} 1000.498046875
donationDistribution_Dollars_count{mp_scope="application",tier="integration",} 203.0
donationDistribution_Dollars_sum{mp_scope="application",tier="integration",} 91850.0
# HELP donationDistribution_Dollars_max The distribution of the donation amounts
# TYPE donationDistribution_Dollars_max gauge
donationDistribution_Dollars_max{mp_scope="application",tier="integration",} 1000.0

Starting in MicroProfile Metrics 5.0, you can adjust the percentile precision of the Histogram metrics by using the mp.metrics.smallrye.histogram.precision MicroProfile Config property. The property accepts a value from 1 to 5 and is defaulted to 3 if no value is specified. A greater value results in more exact percentile calculations, but at a greater memory cost. For more information, see MicroProfile Config properties: MicroProfile Metrics.

In MicroProfile 4.0 and earlier, the following JSON response is also available from the REST endpoints:

{
  "com.example.samples.donationapp.DonationManager.donationDistribution": {
      "count": 203,
      "max": 102,
      "mean": 19.300015535407777,
      "min": 3,
      "p50": 5.0,
      "p75": 24.0,
      "p95": 83.0,
      "p98": 93.0,
      "p99": 101.0,
      "p999": 102.0,
      "stddev": 26.35464238355834
  }
}

Counter

A counter metric keeps an incremental count. The initial value of the counter is set to zero, and the metric increments each time that an annotated element is started.

Use the @Counted annotation to mark a method, constructor, or type as a counter. The counter increments monotonically, counting total invocations of the annotated method:

@GET
@Path("/no")
@Counted(name="no", description="Number of people that declined to donate.")
public String noDonation() {
    return "Maybe next time!";
}

Gauge

You implement a gauge metric so that the gauge can be sampled to obtain a particular value. For example, you might use a gauge to measure CPU temperature or disk usage.

Use the @Gauge annotation to mark a method as a gauge:

@Gauge(
    name="donations",
    description="Total amount of money raised for charity!",
    unit = "dollars",
    absolute=true)
public Long getTotalDonations(){
    return totalDonations;
}

Meter (available only in MicroProfile Metrics 1.0-4.0)

A meter metric tracks throughput. This metric provides the following information:

  • The mean throughput

  • The exponentially weighted moving average throughput at 1-minute, 5-minute, and 15-minute marks

  • A count of the number of measurements

Use the @Metered annotation to mark a constructor or method as a meter. The meter counts the invocations of the annotated constructor or method and tracks how frequently they are called.

@Metered(displayName="Rate of donations", description="Rate of incoming donations (the instances not the amount)")
public void addDonation(Long amount) {
    totalDonations += amount;
    donations.add(amount);
    donationDistribution.update(amount);
}

Concurrent gauge (available only in MicroProfile Metrics 2.0-4.0)

A concurrent gauge metric counts the concurrent invocations of an annotated element. This metric also tracks the high and low watermarks of each invocation. This metric type is removed starting in MicroProfile Metrics 5.0.

Use the @ConcurrentGauge annotation to mark a method as a concurrent gauge. The concurrent gauge increments when the annotated method is called and decrements when the annotated method returns, counting current invocations of the annotated method:

@GET
@Path("/livestream");
@ConcurrentGauge(name = "liveStreamViewers", displayName="Donation live stream viewers", description="Number of active viewers for the donation live stream")
public void donationLiveStream() {
    launchLiveStreamConnection();
}

Simple timer (available only in MicroProfile Metrics 2.3-4.0)

A simple timer metric tracks the elapsed timing duration and invocation counts. This type of metric is available beginning in MicroProfile Metrics 2.3. It is removed starting in MicroProfile Metrics 5.0. The simple timer is a lightweight alternative to the performance-heavy timer metric. Beginning in MicroProfile Metrics 3.0, the simple timer metric also tracks the largest and smallest recorded duration of the previous complete minute. A complete minute is defined as 00:00:00.000 seconds to 00:00:59.999 seconds.

Use the @SimplyTimed annotation to mark a method, constructor, or type as a simple timer. The simple timer tracks how frequently the annotated object is started and how long the invocations take to complete:

@GET
@Path("/weather");
@SimplyTimed(name = "weatherSimplyTimed", displayName="Weather data", description="Provides weather data in JSON")
public JSON getWeatherData() {
    retrieveWeatherData();
}

See also