Microservice observability with metrics

Building observability into microservices externalizes the internal status of a system to enable operations teams to monitor microservice systems more effectively. MicroProfile Metrics provides a /metrics endpoint from which you can access all metrics that are emitted by an Open Liberty server and deployed applications.

You can write an application to produce metrics that operations teams use when the application is running in production. When an application is running, you can view your metric data from any browser by accessing a /metrics endpoint, for example, https://localhost:9443/metrics, where 9443 is the port number for your application. You can narrow down the scope of the metric data by accessing the /metrics/base, /metrics/application, and /metrics/vendor endpoints.

By default, metric data is emitted in Prometheus format. Metric data can be retrieved in JSON format by configuring the Accept header of your request to the application/json value. A GET request returns a list of metrics, and an OPTIONS request returns a list of metrics with their metadata. Operations teams can gather the metrics and store them in a database by using tools like Prometheus. They can then visualize the metric data for analysis in dashboards such as Grafana. For a list of metrics that are available in Open Liberty, see the metrics reference list.

Adding metrics to your applications

You can use MicroProfile Metrics with Open Liberty by enabling the MicroProfile Metrics feature in your server.xml file. To add metrics to your applications, you must create and register metrics with the application registry so that they’re known to the system and can be reported on from the /metrics endpoint. The simplest way to add metrics to your application is by using annotations. MicroProfile Metrics defines annotations that you can use to quickly build metrics into your code. These metrics ultimately provide transparency for operations teams into how services are running.

Metrics and annotations

The following sections describe metric types that are available with MicroProfile Metrics and how their corresponding annotations are used:

Concurrent gauge

A concurrent gauge metric counts the concurrent invocations of an annotated element. This metric also tracks the high and low watermarks of each invocation.

Use the @ConcurrentGauge annotation to mark a method as a concurrent gauge. The concurrent gauge increments when the annotated method is called and decrements when the annotated method returns, counting current invocations of the annotated method:

@GET
@Path("/livestream");
@ConcurrentGauge(name = "liveStreamViewers", displayName="Donation live stream viewers", description="Number of active viewers for the donation live stream")
public void donationLiveStream() {
    launchLiveStreamConnection();
}

Counter

A counter metric keeps an incremental count. The initial value of the counter is set to zero, and the metric increments each time that an annotated element is started.

Use the @Counted annotation to mark a method, constructor, or type as a counter. The counter increments monotonically, counting total invocations of the annotated method:

@GET
@Path("/no")
@Counted(name="no", displayName="No donation count", description="Number of people that declined to donate.")
public String noDonation() {
    return "Maybe next time!";
}

Gauge

You implement a gauge metric so that the gauge can be sampled to obtain a particular value. For example, you might use a gauge to measure CPU temperature or disk usage.

Use the @Gauge annotation to mark a method as a gauge:

@Gauge(
    name="donations",
    displayName="Total Donations",
    description="Total amount of money raised for charity!",
    unit = "dollars",
    absolute=true)
public Long getTotalDonations(){
    return totalDonations;
}

Meter

A meter metric tracks throughput. This metric provides the following information:

  • The mean throughput

  • The exponentially weighted moving average throughput at 1-minute, 5-minute, and 15-minute marks

  • A count of the number of measurements

Use the @Metered annotation to mark a constructor or method as a meter. The meter counts the invocations of the annotated constructor or method and tracks how frequently they are called:

@Metered(displayName="Rate of donations", description="Rate of incoming donations (the instances not the amount)")
public void addDonation(Long amount) {
    totalDonations += amount;
    donations.add(amount);
    donationDistribution.update(amount);
}

Simple timer

A simple timer metric tracks the elapsed timing duration and invocation counts. This type of metric is available beginning in MicroProfile Metrics 2.3. The simple timer is a lightweight alternative to the performance-heavy timer metric. Beginning in MicroProfile Metrics 3.0, the simple timer metric also tracks the largest and smallest recorded duration of the previous complete minute. A complete minute is defined as 00:00:00.000 seconds to 00:00:59.999 seconds.

Use the @SimplyTimed annotation to mark a method, constructor, or type as a simple timer. The simple timer tracks how frequently the annotated object is started and how long the invocations take to complete:

@GET
@Path("/weather");
@SimplyTimed(name = "weatherSimplyTimed", displayName="Weather data", description="Provides weather data in JSON")
public JSON getWeatherData() {
    retrieveWeatherData();
}

Timer

A timer metric aggregates timing durations in nanoseconds, and provides duration and throughput statistics.

Use the @Timed annotation to mark a constructor or method as a timer. The timer tracks how frequently the annotated object is started and how long the invocations take to complete:

@POST
@Path("/creditcard")
@Timed(
    name="donateAmountViaCreditCard.timer",
    displayName="Donations Via Credit Cards",
    description = "Donations that were made using a credit card")
public String donateAmountViaCreditCard(@FormParam("amount") Long amount, @FormParam("card") String card) {

    if (processCard(card, amount))
        return "Thanks for donating!";

    return "Sorry, please try again.";
}

Histogram

A histogram is a metric that calculates the distribution of a value. It provides the following information:

  • Maximum, minimum, median and mean values

  • The value at the 50th, 75th, 95th, 98th, 99th, 99.9th percentile

  • A count of the number of values

  • Standard deviation for the value

The histogram metric does not have an annotation. To record a value in the histogram, you must call the histogram.update(long value) method with the value that you want to record. The current state, or snapshot, of the histogram can be retrieved by using the getSnapshot() method. Histograms in MicroProfile Metrics only support integer or long values.

Examples of histograms include the distribution of payload sizes that are retrieved or for an onboarding survey that collects the distribution of household income.

The following example illustrates a histogram that is used to store the value of donations. This example provides the administrator with an idea of the distribution of donation amounts:

Metadata donationDistributionMetadata = Metadata.builder()
              .withName("donationDistribution")                             // name
              .withDisplayName("Donation Distribution")                     // display name
              .withDescription("The distribution of the donation amounts")  // description
              .withType(MetricType.HISTOGRAM)                               //type
              .withUnit("Dollars")                                          //units
              .build();
Histogram donationDistribution = registry.histogram(donationDistributionMetadata);
public void addDonation(Long amount) {
    totalDonations += amount;
    donations.add(amount);
    donationDistribution.update(amount);
}

For this example, the following response is generated from the REST endpoints:

{
  "com.example.samples.donationapp.DonationManager.donationDistribution": {
      "count": 203,
      "max": 102,
      "mean": 19.300015535407777,
      "min": 3,
      "p50": 5.0,
      "p75": 24.0,
      "p95": 83.0,
      "p98": 93.0,
      "p99": 101.0,
      "p999": 102.0,
      "stddev": 26.35464238355834
  }
}

These types of metrics are available to add to your applications to make them observable. In production, operations teams can use these metrics to monitor the application, along with metrics that are automatically emitted from the JVM and the Open Liberty server runtime. If you’re interested in learning more about using MicroProfile Metrics to build observability into your microservices, see the Open Liberty guide for Providing metrics from a microservice.

See also