Checking the health of microservices on Kubernetes

duration 20 minutes

Prerequisites:

Learn how to check the health of microservices on Kubernetes by setting up startup, liveness, and readiness probes to inspect MicroProfile Health Check endpoints.

What you’ll learn

You will learn how to create health check endpoints for your microservices. Then, you will configure Kubernetes to use these endpoints to keep your microservices running smoothly.

MicroProfile Health allows services to report their health, and it publishes the overall health status to defined endpoints. If a service reports UP, then it’s available. If the service reports DOWN, then it’s unavailable. MicroProfile Health reports an individual service status at the endpoint and indicates the overall status as UP if all the services are UP. A service orchestrator can then use the health statuses to make decisions.

Kubernetes provides startup, liveness, and readiness probes that are used to check the health of your containers. These probes can check certain files in your containers, check a TCP socket, or make HTTP requests. MicroProfile Health exposes startup, liveness, and readiness endpoints on your microservices. Kubernetes polls these endpoints as specified by the probes to react appropriately to any change in the microservice’s status. Read the Adding health reports to microservices guide to learn more about MicroProfile Health.

The two microservices you will work with are called system and inventory. The system microservice returns the JVM system properties of the running container and it returns the pod’s name in the HTTP header making replicas easy to distinguish from each other. The inventory microservice adds the properties from the system microservice to the inventory. This demonstrates how communication can be established between pods inside a cluster.

Additional prerequisites

Before you begin, you need a containerization software for building containers. Kubernetes supports various container runtimes. You will use Docker in this guide. For Docker installation instructions, refer to the official Docker documentation.

Use Docker Desktop, where a local Kubernetes environment is pre-installed and enabled. If you do not see the Kubernetes tab, then upgrade to the latest version of Docker Desktop.

Complete the setup for your operating system:

After you complete the Docker setup instructions for your operating system, ensure that Kubernetes (not Swarm) is selected as the orchestrator in Docker Preferences.

Use Docker Desktop, where a local Kubernetes environment is pre-installed and enabled. If you do not see the Kubernetes tab, then upgrade to the latest version of Docker Desktop.

Complete the setup for your operating system:

After you complete the Docker setup instructions for your operating system, ensure that Kubernetes (not Swarm) is selected as the orchestrator in Docker Preferences.

You will use Minikube as a single-node Kubernetes cluster that runs locally in a virtual machine. Make sure you have kubectl installed. If you need to install kubectl, see the kubectl installation instructions. For Minikube installation instructions, see the Minikube documentation.

Getting started

The fastest way to work through this guide is to clone the Git repository and use the projects that are provided inside:

git clone https://github.com/openliberty/guide-kubernetes-microprofile-health.git
cd guide-kubernetes-microprofile-health

The start directory contains the starting project that you will build upon.

The finish directory contains the finished project that you will build.

Before you begin, make sure you have all the necessary prerequisites.

Starting and preparing your cluster for deployment

Start your Kubernetes cluster.

Start your Docker Desktop environment.

Ensure that Kubernetes is running on Docker Desktop and that the context is set to docker-desktop.

Run the following command from a command-line session:

minikube start

Next, validate that you have a healthy Kubernetes environment by running the following command from the active command-line session.

kubectl get nodes

This command should return a Ready status for the master node.

You do not need to do any other step.

Run the following command to configure the Docker CLI to use Minikube’s Docker daemon. After you run this command, you will be able to interact with Minikube’s Docker daemon and build new images directly to it from your host machine:

eval $(minikube docker-env)

Adding health checks to the inventory microservice

Navigate to start directory to begin.

Create the InventoryStartupCheck class.
inventory/src/main/java/io/openliberty/guides/inventory/InventoryStartupCheck.java

InventoryStartupCheck.java

 1// tag::copyright[]
 2/*******************************************************************************
 3 * Copyright (c) 2022 IBM Corporation and others.
 4 * All rights reserved. This program and the accompanying materials
 5 * are made available under the terms of the Eclipse Public License v1.0
 6 * which accompanies this distribution, and is available at
 7 * http://www.eclipse.org/legal/epl-v10.html
 8 *
 9 * Contributors:
10 *     IBM Corporation - Initial implementation
11 *******************************************************************************/
12// end::copyright[]
13// tag::InventoryStartupCheck[]
14package io.openliberty.guides.inventory;
15
16import java.lang.management.ManagementFactory;
17import com.sun.management.OperatingSystemMXBean;
18import jakarta.enterprise.context.ApplicationScoped;
19import org.eclipse.microprofile.health.Startup;
20import org.eclipse.microprofile.health.HealthCheck;
21import org.eclipse.microprofile.health.HealthCheckResponse;
22
23// tag::Startup[]
24@Startup
25// end::Startup[]
26@ApplicationScoped
27public class InventoryStartupCheck implements HealthCheck {
28
29    @Override
30    public HealthCheckResponse call() {
31        OperatingSystemMXBean bean = (com.sun.management.OperatingSystemMXBean)
32        ManagementFactory.getOperatingSystemMXBean();
33        double cpuUsed = bean.getSystemCpuLoad();
34        String cpuUsage = String.valueOf(cpuUsed);
35        return HealthCheckResponse.named(InventoryResource.class
36                                            .getSimpleName() + " Startup Check")
37                                            .status(cpuUsed < 0.95).build();
38    }
39}
40
41// end::InventoryStartupCheck[]

A health check for startup allows applications to define startup probes that verify whether deployed application is fully initialized before the liveness probe takes over. This check is useful for applications that require additional startup time on their first initialization. The @Startup annotation must be applied on a HealthCheck implementation to define a startup check procedure. Otherwise, this annotation is ignored. This startup check verifies that the cpu usage is below 95%. If more than 95% of the cpu is used, a status of DOWN is returned.

Create the InventoryLivenessCheck class.
inventory/src/main/java/io/openliberty/guides/inventory/InventoryLivenessCheck.java

InventoryLivenessCheck.java

 1// tag::copyright[]
 2/*******************************************************************************
 3 * Copyright (c) 2019, 2022 IBM Corporation and others.
 4 * All rights reserved. This program and the accompanying materials
 5 * are made available under the terms of the Eclipse Public License v1.0
 6 * which accompanies this distribution, and is available at
 7 * http://www.eclipse.org/legal/epl-v10.html
 8 *
 9 * Contributors:
10 *     IBM Corporation - Initial implementation
11 *******************************************************************************/
12// end::copyright[]
13package io.openliberty.guides.inventory;
14
15import jakarta.enterprise.context.ApplicationScoped;
16
17import java.lang.management.MemoryMXBean;
18import java.lang.management.ManagementFactory;
19
20import org.eclipse.microprofile.health.Liveness;
21import org.eclipse.microprofile.health.HealthCheck;
22import org.eclipse.microprofile.health.HealthCheckResponse;
23
24// tag::Liveness[]
25@Liveness
26// end::Liveness[]
27@ApplicationScoped
28public class InventoryLivenessCheck implements HealthCheck {
29
30  @Override
31  public HealthCheckResponse call() {
32      MemoryMXBean memBean = ManagementFactory.getMemoryMXBean();
33      long memUsed = memBean.getHeapMemoryUsage().getUsed();
34      long memMax = memBean.getHeapMemoryUsage().getMax();
35
36      return HealthCheckResponse.named(InventoryResource.class.getSimpleName()
37                                      + " Liveness Check")
38                                .status(memUsed < memMax * 0.9).build();
39  }
40}

A health check for liveness allows third party services to determine whether the application is running. If this procedure fails, the application can be stopped. The @Liveness annotation must be applied on a HealthCheck implementation to define a Liveness check procedure. Otherwise, this annotation is ignored. This liveness check verifies that the heap memory usage is below 90% of the maximum memory. If more than 90% of the maximum memory is used, a status of DOWN is returned.

The inventory microservice is healthy only when the system microservice is available. To add this check to the /health/ready endpoint, create a class that is annotated with the @Readiness annotation and implements the HealthCheck interface. A Health Check for readiness allows third party services to know whether the application is ready to process requests. The @Readiness annotation must be applied on a HealthCheck implementation to define a readiness check procedure. Otherwise, this annotation is ignored.

Create the InventoryReadinessCheck class.
inventory/src/main/java/io/openliberty/guides/inventory/InventoryReadinessCheck.java

InventoryReadinessCheck.java

 1// tag::copyright[]
 2/*******************************************************************************
 3 * Copyright (c) 2019, 2022 IBM Corporation and others.
 4 * All rights reserved. This program and the accompanying materials
 5 * are made available under the terms of the Eclipse Public License v1.0
 6 * which accompanies this distribution, and is available at
 7 * http://www.eclipse.org/legal/epl-v10.html
 8 *
 9 * Contributors:
10 *     IBM Corporation - Initial implementation
11 *******************************************************************************/
12// end::copyright[]
13package io.openliberty.guides.inventory;
14
15import jakarta.enterprise.context.ApplicationScoped;
16import jakarta.inject.Inject;
17import jakarta.ws.rs.client.Client;
18import jakarta.ws.rs.client.ClientBuilder;
19
20import org.eclipse.microprofile.config.inject.ConfigProperty;
21import org.eclipse.microprofile.health.Readiness;
22import org.eclipse.microprofile.health.HealthCheck;
23import org.eclipse.microprofile.health.HealthCheckResponse;
24
25// tag::Readiness[]
26@Readiness
27// end::Readiness[]
28@ApplicationScoped
29public class InventoryReadinessCheck implements HealthCheck {
30
31    private static final String READINESS_CHECK = InventoryResource.class
32                                                .getSimpleName()
33                                                + " Readiness Check";
34
35    @Inject
36    @ConfigProperty(name = "SYS_APP_HOSTNAME")
37    private String hostname;
38
39    public HealthCheckResponse call() {
40        if (isSystemServiceReachable()) {
41            return HealthCheckResponse.up(READINESS_CHECK);
42        } else {
43            return HealthCheckResponse.down(READINESS_CHECK);
44        }
45    }
46
47    private boolean isSystemServiceReachable() {
48        try {
49            Client client = ClientBuilder.newClient();
50            client
51                .target("http://" + hostname + ":9080/system/properties")
52                .request()
53                .post(null);
54
55            return true;
56        } catch (Exception ex) {
57            return false;
58        }
59    }
60}

This health check verifies that the system microservice is available at http://system-service:9080/. The system-service host name is accessible only from inside the cluster; you can’t access it yourself. If it’s available, then it returns an UP status. Similarly, if it’s unavailable then it returns a DOWN status. When the status is DOWN, the microservice is considered to be unhealthy.

The health checks for the system microservice were already been implemented. The system microservice was set up to become unhealthy for 60 seconds when a specific endpoint is called. This endpoint has been provided for you to observe the results of an unhealthy pod and how Kubernetes reacts.

Configuring startup, liveness, and readiness probes

You will configure Kubernetes startup, liveness, and readiness probes. Startup probes determine whether your application is fully initialized. Liveness probes determine whether a container needs to be restarted. Readiness probes determine whether your application is ready to accept requests. If it’s not ready, no traffic is routed to the container.

Create the kubernetes configuration file.
kubernetes.yaml

kubernetes.yaml

  1apiVersion: apps/v1
  2kind: Deployment
  3metadata:
  4  name: system-deployment
  5  labels:
  6    app: system
  7spec:
  8  replicas: 2
  9  selector:
 10    matchLabels:
 11      app: system
 12  template:
 13    metadata:
 14      labels:
 15        app: system
 16    spec:
 17      containers:
 18      - name: system-container
 19        image: system:1.0-SNAPSHOT
 20        ports:
 21        - containerPort: 9080
 22        # system probes
 23        startupProbe:
 24          httpGet:
 25            # tag::start1[]
 26            path: /health/started
 27            # end::start1[]
 28            port: 9080
 29        livenessProbe:
 30          httpGet:
 31            # tag::live1[]
 32            path: /health/live
 33            # end::live1[]
 34            port: 9080
 35          # tag::delay1[]
 36          initialDelaySeconds: 60
 37          # end::delay1[]
 38          # tag::period1[]
 39          periodSeconds: 10
 40          # end::period1[]
 41          # tag::timeout1[]
 42          timeoutSeconds: 3
 43          # end::timeout1[]
 44          # tag::threshold1[]
 45          failureThreshold: 1
 46          # end::threshold1[]
 47        readinessProbe:
 48          httpGet:
 49            # tag::ready1[]
 50            path: /health/ready
 51            # end::ready1[]
 52            port: 9080
 53          # tag::delay2[]
 54          initialDelaySeconds: 30
 55          # end::delay2[]
 56          # tag::period2[]
 57          periodSeconds: 10
 58          # end::period2[]
 59          # tag::timeout2[]
 60          timeoutSeconds: 3
 61          # end::timeout2[]
 62          # tag::threshold2[]
 63          failureThreshold: 1
 64          # end::threshold2[]
 65---
 66apiVersion: apps/v1
 67kind: Deployment
 68metadata:
 69  name: inventory-deployment
 70  labels:
 71    app: inventory
 72spec:
 73  selector:
 74    matchLabels:
 75      app: inventory
 76  template:
 77    metadata:
 78      labels:
 79        app: inventory
 80    spec:
 81      containers:
 82      - name: inventory-container
 83        image: inventory:1.0-SNAPSHOT
 84        ports:
 85        - containerPort: 9080
 86        env:
 87        - name: SYS_APP_HOSTNAME
 88          value: system-service
 89        # inventory probes
 90        startupProbe:
 91          httpGet:
 92            # tag::start2[]
 93            path: /health/started
 94            # end::start2[]
 95            port: 9080
 96        livenessProbe:
 97          httpGet:
 98            # tag::live2[]
 99            path: /health/live
100            # end::live2[]
101            port: 9080
102          # tag::delay3[]
103          initialDelaySeconds: 60
104          # end::delay3[]
105          # tag::period3[]
106          periodSeconds: 10
107          # end::period3[]
108          # tag::timeout3[]
109          timeoutSeconds: 3
110          # end::timeout3[]
111          # tag::threshold3[]
112          failureThreshold: 1
113          # end::threshold3[]
114        readinessProbe:
115          httpGet:
116            # tag::ready2[]
117            path: /health/ready
118            # end::ready2[]
119            port: 9080
120          # tag::delay4[]
121          initialDelaySeconds: 30
122          # end::delay4[]
123          # tag::period4[]
124          periodSeconds: 10
125          # end::period4[]
126          # tag::timeout4[]
127          timeoutSeconds: 3
128          # end::timeout4[]
129          # tag::threshold4[]
130          failureThreshold: 1
131          # end::threshold4[]
132---
133apiVersion: v1
134kind: Service
135metadata:
136  name: system-service
137spec:
138  type: NodePort
139  selector:
140    app: system
141  ports:
142  - protocol: TCP
143    port: 9080
144    targetPort: 9080
145    nodePort: 31000
146---
147apiVersion: v1
148kind: Service
149metadata:
150  name: inventory-service
151spec:
152  type: NodePort
153  selector:
154    app: inventory
155  ports:
156  - protocol: TCP
157    port: 9080
158    targetPort: 9080
159    nodePort: 32000

The startup, liveness, and readiness probes are configured for the containers that are running the system and inventory microservices.

The startup probes are configured to poll the /health/started endpoint. The startup probe determines whether a container is started.

The liveness probes are configured to poll the /health/live endpoint. The liveness probes determine whether a container needs to be restarted. The initialDelaySeconds field defines the duration that the probe waits before it starts to poll so that it does not make requests before the server is started. The periodSeconds option defines how often the probe polls the given endpoint. The timeoutSeconds option defines how many seconds before the probe times out. The failureThreshold option defines how many times the probe fails before the state changes from ready to not ready.

The readiness probes are configured to poll the /health/ready endpoint. The readiness probe determines the READY status of the container, as seen in the kubectl get pods output. Similar to the liveness probes, the readiness probes also define initialDelaySeconds, periodSeconds, timeoutSeconds, and failureThreshold.

Deploying the microservices

To build these microservices, navigate to the start directory and run the following command.

mvn package

Run the following command to download or update to the latest Open Liberty Docker image:

docker pull icr.io/appcafe/open-liberty:full-java11-openj9-ubi

Next, run the docker build commands to build container images for your application:

docker build -t system:1.0-SNAPSHOT system/.
docker build -t inventory:1.0-SNAPSHOT inventory/.

The -t flag in the docker build command allows the Docker image to be labeled (tagged) in the name[:tag] format. The tag for an image describes the specific image version. If the optional [:tag] tag is not specified, the latest tag is created by default.

When the builds succeed, run the following command to deploy the necessary Kubernetes resources to serve the applications.

kubectl apply -f kubernetes.yaml

Use the following command to view the status of the pods. There will be two system pods and one inventory pod, later you’ll observe their behavior as the system pods become unhealthy.

kubectl get pods
NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          59s
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          59s
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          59s

Wait until the pods are ready. After the pods are ready, you will make requests to your services.

The default host name for Docker Desktop is localhost.

The default host name for minikube is 192.168.99.100. Otherwise it can be found using the minikube ip command.

Navigate to http://[hostname]:31000/system/properties and observe a response containing JVM system properties. Replace [hostname] with the IP address or host name of your Kubernetes cluster. The readiness probe ensures the READY state won’t be 1/1 until the container is available to accept requests. Without a readiness probe, you might notice an unsuccessful response from the server. This scenario can occur when the container has started, but the application server hasn’t fully initialized. With the readiness probe, you can be certain the pod will only accept traffic when the microservice has fully started.

Similarly, navigate to http://[hostname]:32000/inventory/systems/system-service and observe that the request is successful.

Changing the ready state of the system microservice

An unhealthy endpoint has been provided under the system microservice to set it to an unhealthy state. The unhealthy state causes the readiness probe to fail. A request to the unhealthy endpoint puts the service in an unhealthy state as a simulation.

Navigate to http://[hostname]:31000/system/unhealthy to invoke the unhealthy endpoint by running the following curl command:

curl http://[hostname]:31000/system/unhealthy

Run the following command to view the state of the pods:

kubectl get pods
NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          1m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          1m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          1m

You will notice that one of the two system pods is no longer in the ready state. Navigate to http://[hostname]:31000/system/properties. Your request is successful because you have two replicas and one is still healthy.

Observing the effects on the inventory microservice

Wait until the system pod is ready again. Make two requests to http://[hostname]:31000/system/unhealthy. If you see the same pod name twice, repeat the request until you see that the second pod is unhealthy. You might see the same pod twice due to a delay between when a pod becomes unhealthy and when the readiness probe notices it. Therefore, traffic might still be routed to the unhealthy service for approximately 5 seconds. Continue to observe the output of kubectl get pods. You will see both pods are no longer ready. During this process, the readiness probe for the inventory microservice will also fail. Observe that it’s no longer in the ready state either.

First, both system pods will no longer be ready because the readiness probe failed.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     0/1       Running   0          5m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          5m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          5m

Next, the inventory pod is no longer ready because the readiness probe failed. The probe failed because system-service is now unavailable.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     0/1       Running   0          6m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          6m
inventory-deployment-cf8f564c6-nctcr   0/1       Running   0          6m

Then, the system pods will start to become healthy again after 60 seconds.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          7m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          7m
inventory-deployment-cf8f564c6-nctcr   0/1       Running   0          7m
NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          7m
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          7m
inventory-deployment-cf8f564c6-nctcr   0/1       Running   0          7m

Finally, you will see all of the pods have recovered.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          8m
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          8m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          8m

Testing the microservices

Run the tests by running the following command and appropriately substituting [hostname] for the correct value.

mvn failsafe:integration-test -Dsystem.service.root=[hostname]:31000 -Dinventory.service.root=[hostname]:32000

A few tests are included for you to test the basic functions of the microservices. If a test fails, then you might have introduced a bug into the code. Wait for all pods to be in the ready state before you run the tests.

When the tests succeed, you should see output similar to the following in your console.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.system.SystemEndpointIT
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.65 s - in it.io.openliberty.guides.system.SystemEndpointIT

Results:

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.inventory.InventoryEndpointIT
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.542 s - in it.io.openliberty.guides.inventory.InventoryEndpointIT

Results:

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

Tearing down the environment

To remove all of the resources created during this guide, run the following command to delete all of the resources that you created.

kubectl delete -f kubernetes.yaml

Nothing more needs to be done for Docker Desktop.

Perform the following steps to return your environment to a clean state.

  1. Point the Docker daemon back to your local machine:

    eval $(minikube docker-env -u)
  2. Stop your Minikube cluster:

    minikube stop
  3. Delete your cluster:

    minikube delete

Great work! You’re done!

You have used MicroProfile Health and Open Liberty to create endpoints that report on your microservice’s status. Then, you observed how Kubernetes uses the /health/started, /health/live, and /health/ready endpoints to keep your microservices running smoothly.

Guide Attribution

Checking the health of microservices on Kubernetes by Open Liberty is licensed under CC BY-ND 4.0

Copied to clipboard
Copy code block
Copy file contents

Prerequisites:

Nice work! Where to next?

What did you think of this guide?

Extreme Dislike Dislike Like Extreme Like

What could make this guide better?

Raise an issue to share feedback

Create a pull request to contribute to this guide

Need help?

Ask a question on Stack Overflow

Like Open Liberty? Star our repo on GitHub.

Star