What you’ll learn
Additional prerequisites
Getting started
Starting and preparing your cluster for deployment
Adding health checks to the inventory microservice
Configuring startup, liveness, and readiness probes
Deploying the microservices
Changing the ready state of the system microservice

Observing the effects on the inventory microservice

Testing the microservices
Tearing down the environment
Nice work! Where to next?

Checking the health of microservices on Kubernetes

20 minutes

Learn how to check the health of microservices on Kubernetes by setting up startup, liveness, and readiness probes to inspect MicroProfile Health Check endpoints.

Prerequisites:

Table of contents

What you’ll learn

You will learn how to create health check endpoints for your microservices. Then, you will configure Kubernetes to use these endpoints to keep your microservices running smoothly.

MicroProfile Health allows services to report their health, and it publishes the overall health status to defined endpoints. If a service reports UP, then it’s available. If the service reports DOWN, then it’s unavailable. MicroProfile Health reports an individual service status at the endpoint and indicates the overall status as UP if all the services are UP. A service orchestrator can then use the health statuses to make decisions.

Kubernetes provides startup, liveness, and readiness probes that are used to check the health of your containers. These probes can check certain files in your containers, check a TCP socket, or make HTTP requests. MicroProfile Health exposes startup, liveness, and readiness endpoints on your microservices. Kubernetes polls these endpoints as specified by the probes to react appropriately to any change in the microservice’s status. Read the Adding health reports to microservices guide to learn more about MicroProfile Health.

The two microservices you will work with are called system and inventory. The system microservice returns the JVM system properties of the running container and it returns the pod’s name in the HTTP header making replicas easy to distinguish from each other. The inventory microservice adds the properties from the system microservice to the inventory. This demonstrates how communication can be established between pods inside a cluster.

Additional prerequisites

Before you begin, you need a containerization software for building containers. Kubernetes supports various container runtimes. You will use Docker in this guide. For Docker installation instructions, refer to the official Docker documentation.

WINDOWS

MAC

LINUX

Use Docker Desktop, where a local Kubernetes environment is pre-installed and enabled. If you do not see the Kubernetes tab, then upgrade to the latest version of Docker Desktop.

Complete the setup for your operating system:

Set up Docker for Windows.

After you complete the Docker setup instructions for your operating system, ensure that Kubernetes (not Swarm) is selected as the orchestrator in Docker Preferences.

Getting started

The fastest way to work through this guide is to clone the Git repository and use the projects that are provided inside:

Copied to clipboard

git clone https://github.com/openliberty/guide-kubernetes-microprofile-health.git
cd guide-kubernetes-microprofile-health

The start directory contains the starting project that you will build upon.

The finish directory contains the finished project that you will build.

Before you begin, make sure you have all the necessary prerequisites.

Starting and preparing your cluster for deployment

Start your Kubernetes cluster.

WINDOWS

MAC

LINUX

Start your Docker Desktop environment.

Ensure that Kubernetes is running on Docker Desktop and that the context is set to docker-desktop.

Next, validate that you have a healthy Kubernetes environment by running the following command from the active command-line session.

Copied to clipboard

kubectl get nodes

This command should return a Ready status for the master node.

WINDOWS

MAC

LINUX

You do not need to do any other step.

Adding health checks to the inventory microservice

Navigate to start directory to begin.

Create the InventoryStartupCheck class.

Copied to clipboard

inventory/src/main/java/io/openliberty/guides/inventory/InventoryStartupCheck.java

A health check for startup allows applications to define startup probes that verify whether deployed application is fully initialized before the liveness probe takes over. This check is useful for applications that require additional startup time on their first initialization. The @Startup annotation must be applied on a HealthCheck implementation to define a startup check procedure. Otherwise, this annotation is ignored. This startup check verifies that the cpu usage is below 95%. If more than 95% of the cpu is used, a status of DOWN is returned.

Create the InventoryLivenessCheck class.

Copied to clipboard

inventory/src/main/java/io/openliberty/guides/inventory/InventoryLivenessCheck.java

A health check for liveness allows third party services to determine whether the application is running. If this procedure fails, the application can be stopped. The @Liveness annotation must be applied on a HealthCheck implementation to define a Liveness check procedure. Otherwise, this annotation is ignored. This liveness check verifies that the heap memory usage is below 90% of the maximum memory. If more than 90% of the maximum memory is used, a status of DOWN is returned.

The inventory microservice is healthy only when the system microservice is available. To add this check to the /health/ready endpoint, create a class that is annotated with the @Readiness annotation and implements the HealthCheck interface. A Health Check for readiness allows third party services to know whether the application is ready to process requests. The @Readiness annotation must be applied on a HealthCheck implementation to define a readiness check procedure. Otherwise, this annotation is ignored.

Create the InventoryReadinessCheck class.

Copied to clipboard

inventory/src/main/java/io/openliberty/guides/inventory/InventoryReadinessCheck.java

This health check verifies that the system microservice is available at http://system-service:9090/. The system-service host name is accessible only from inside the cluster; you can’t access it yourself. If it’s available, then it returns an UP status. Similarly, if it’s unavailable then it returns a DOWN status. When the status is DOWN, the microservice is considered to be unhealthy.

The health checks for the system microservice were already been implemented. The system microservice was set up to become unhealthy for 60 seconds when a specific endpoint is called. This endpoint has been provided for you to observe the results of an unhealthy pod and how Kubernetes reacts.

Configuring startup, liveness, and readiness probes

You will configure Kubernetes startup, liveness, and readiness probes. Startup probes determine whether your application is fully initialized. Liveness probes determine whether a container needs to be restarted. Readiness probes determine whether your application is ready to accept requests. If it’s not ready, no traffic is routed to the container.

Create the kubernetes configuration file.

Copied to clipboard

kubernetes.yaml

The startup, liveness, and readiness probes are configured for the containers that are running the system and inventory microservices.

The startup probes are configured to poll the /health/started endpoint. The startup probe determines whether a container is started.

The liveness probes are configured to poll the /health/live endpoint. The liveness probes determine whether a container needs to be restarted. The initialDelaySeconds field defines the duration that the probe waits before it starts to poll so that it does not make requests before the server is started. The periodSeconds option defines how often the probe polls the given endpoint. The timeoutSeconds option defines how many seconds before the probe times out. The failureThreshold option defines how many times the probe fails before the state changes from ready to not ready.

The readiness probes are configured to poll the /health/ready endpoint. The readiness probe determines the READY status of the container, as seen in the kubectl get pods output. Similar to the liveness probes, the readiness probes also define initialDelaySeconds, periodSeconds, timeoutSeconds, and failureThreshold.

Deploying the microservices

To build these microservices, navigate to the start directory and run the following command.

Copied to clipboard

mvn package

Next, run the docker build commands to build container images for your application:

Copied to clipboard

docker build -t system:1.0-SNAPSHOT system/.
docker build -t inventory:1.0-SNAPSHOT inventory/.

The -t flag in the docker build command allows the Docker image to be labeled (tagged) in the name[:tag] format. The tag for an image describes the specific image version. If the optional [:tag] tag is not specified, the latest tag is created by default.

When the builds succeed, run the following command to deploy the necessary Kubernetes resources to serve the applications.

Copied to clipboard

kubectl apply -f kubernetes.yaml

Use the following command to view the status of the pods. There will be two system pods and one inventory pod, later you’ll observe their behavior as the system pods become unhealthy.

Copied to clipboard

kubectl get pods

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          59s
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          59s
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          59s

Wait until the pods are ready. After the pods are ready, you will make requests to your services.

WINDOWS

MAC

LINUX

The default host name for Docker Desktop is localhost.

Navigate to http://[hostname]:31000/system/properties and observe a response containing JVM system properties. Replace [hostname] with the IP address or host name of your Kubernetes cluster. The readiness probe ensures the READY state won’t be 1/1 until the container is available to accept requests. Without a readiness probe, you might notice an unsuccessful response from the server. This scenario can occur when the container has started, but the application server hasn’t fully initialized. With the readiness probe, you can be certain the pod will only accept traffic when the microservice has fully started.

Similarly, navigate to http://[hostname]:32000/inventory/systems/system-service and observe that the request is successful.

Changing the ready state of the system microservice

An unhealthy endpoint has been provided under the system microservice to set it to an unhealthy state. The unhealthy state causes the readiness probe to fail. A request to the unhealthy endpoint puts the service in an unhealthy state as a simulation.

Navigate to http://[hostname]:31000/system/unhealthy to invoke the unhealthy endpoint by running the following curl command:

Copied to clipboard

curl http://[hostname]:31000/system/unhealthy

Run the following command to view the state of the pods:

Copied to clipboard

kubectl get pods

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          1m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          1m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          1m

You will notice that one of the two system pods is no longer in the ready state. Navigate to http://[hostname]:31000/system/properties. Your request is successful because you have two replicas and one is still healthy.

Observing the effects on the inventory microservice

Wait until the system pod is ready again. Make two requests to http://[hostname]:31000/system/unhealthy. If you see the same pod name twice, repeat the request until you see that the second pod is unhealthy. You might see the same pod twice due to a delay between when a pod becomes unhealthy and when the readiness probe notices it. Therefore, traffic might still be routed to the unhealthy service for approximately 5 seconds. Continue to observe the output of kubectl get pods. You will see both pods are no longer ready. During this process, the readiness probe for the inventory microservice will also fail. Observe that it’s no longer in the ready state either.

First, both system pods will no longer be ready because the readiness probe failed.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     0/1       Running   0          5m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          5m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          5m

Next, the inventory pod is no longer ready because the readiness probe failed. The probe failed because system-service is now unavailable.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     0/1       Running   0          6m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          6m
inventory-deployment-cf8f564c6-nctcr   0/1       Running   0          6m

Then, the system pods will start to become healthy again after 60 seconds.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          7m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          7m
inventory-deployment-cf8f564c6-nctcr   0/1       Running   0          7m

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          7m
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          7m
inventory-deployment-cf8f564c6-nctcr   0/1       Running   0          7m

Finally, you will see all of the pods have recovered.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          8m
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          8m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          8m

Testing the microservices

Run the tests by running the following command and appropriately substituting [hostname] for the correct value.

Copied to clipboard

mvn failsafe:integration-test -Dsystem.service.root=[hostname]:31000 -Dinventory.service.root=[hostname]:32000

A few tests are included for you to test the basic functions of the microservices. If a test fails, then you might have introduced a bug into the code. Wait for all pods to be in the ready state before you run the tests.

When the tests succeed, you should see output similar to the following in your console.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.system.SystemEndpointIT
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.65 s - in it.io.openliberty.guides.system.SystemEndpointIT

Results:

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.inventory.InventoryEndpointIT
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.542 s - in it.io.openliberty.guides.inventory.InventoryEndpointIT

Results:

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

Tearing down the environment

To remove all of the resources created during this guide, run the following command to delete all of the resources that you created.

Copied to clipboard

kubectl delete -f kubernetes.yaml

WINDOWS

MAC

LINUX

Nothing more needs to be done for Docker Desktop.

13package io.openliberty.guides.inventory;
14
15import java.lang.management.ManagementFactory;
16import com.sun.management.OperatingSystemMXBean;
17import jakarta.enterprise.context.ApplicationScoped;
18import org.eclipse.microprofile.health.Startup;
19import org.eclipse.microprofile.health.HealthCheck;
20import org.eclipse.microprofile.health.HealthCheckResponse;
21
23@Startup
25@ApplicationScoped
26public class InventoryStartupCheck implements HealthCheck {
27
28    @Override
29    public HealthCheckResponse call() {
30        OperatingSystemMXBean bean = (com.sun.management.OperatingSystemMXBean)
31        ManagementFactory.getOperatingSystemMXBean();
32        double cpuUsed = bean.getSystemCpuLoad();
33        String cpuUsage = String.valueOf(cpuUsed);
34        return HealthCheckResponse.named(InventoryResource.class
35                                            .getSimpleName() + " Startup Check")
36                                            .status(cpuUsed < 0.95).build();
37    }
38}
39
41

12package io.openliberty.guides.inventory;
13
14import jakarta.enterprise.context.ApplicationScoped;
15
16import java.lang.management.MemoryMXBean;
17import java.lang.management.ManagementFactory;
18
19import org.eclipse.microprofile.health.Liveness;
20import org.eclipse.microprofile.health.HealthCheck;
21import org.eclipse.microprofile.health.HealthCheckResponse;
22
24@Liveness
26@ApplicationScoped
27public class InventoryLivenessCheck implements HealthCheck {
28
29  @Override
30  public HealthCheckResponse call() {
31      MemoryMXBean memBean = ManagementFactory.getMemoryMXBean();
32      long memUsed = memBean.getHeapMemoryUsage().getUsed();
33      long memMax = memBean.getHeapMemoryUsage().getMax();
34
35      return HealthCheckResponse.named(InventoryResource.class.getSimpleName()
36                                      + " Liveness Check")
37                                .status(memUsed < memMax * 0.9).build();
38  }
39}40

12package io.openliberty.guides.inventory;
13
14import jakarta.enterprise.context.ApplicationScoped;
15import jakarta.inject.Inject;
16import jakarta.ws.rs.client.Client;
17import jakarta.ws.rs.client.ClientBuilder;
18
19import org.eclipse.microprofile.config.inject.ConfigProperty;
20import org.eclipse.microprofile.health.Readiness;
21import org.eclipse.microprofile.health.HealthCheck;
22import org.eclipse.microprofile.health.HealthCheckResponse;
23
25@Readiness
27@ApplicationScoped
28public class InventoryReadinessCheck implements HealthCheck {
29
30    private static final String READINESS_CHECK = InventoryResource.class
31                                                .getSimpleName()
32                                                + " Readiness Check";
33
34    @Inject
35    @ConfigProperty(name = "SYS_APP_HOSTNAME")
36    private String hostname;
37
38    public HealthCheckResponse call() {
39        if (isSystemServiceReachable()) {
40            return HealthCheckResponse.up(READINESS_CHECK);
41        } else {
42            return HealthCheckResponse.down(READINESS_CHECK);
43        }
44    }
45
46    private boolean isSystemServiceReachable() {
47        try {
48            Client client = ClientBuilder.newClient();
49            client
50                .target("http://" + hostname + ":9090/system/properties")
51                .request()
52                .post(null);
53
54            return true;
55        } catch (Exception ex) {
56            return false;
57        }
58    }
59}60

  1apiVersion: apps/v1
  2kind: Deployment
  3metadata:
  4  name: system-deployment
  5  labels:
  6    app: system
  7spec:
  8  replicas: 2
  9  selector:
 10    matchLabels:
 11      app: system
 12  strategy:
 13    type: RollingUpdate
 14    rollingUpdate:
 15      maxUnavailable: 1
 16      maxSurge: 1
 17  template:
 18    metadata:
 19      labels:
 20        app: system
 21    spec:
 22      containers:
 23      - name: system-container
 24        image: system:1.0-SNAPSHOT
 25        ports:
 26        - containerPort: 9090
 27        # system probes
 28        startupProbe:
 29          httpGet:
 31            path: /health/started
 33            port: 9090
 34        livenessProbe:
 35          httpGet:
 37            path: /health/live
 39            port: 9090
 41          initialDelaySeconds: 60
 44          periodSeconds: 10
 47          timeoutSeconds: 3
 50          failureThreshold: 1
 52        readinessProbe:
 53          httpGet:
 55            path: /health/ready
 57            port: 9090
 59          initialDelaySeconds: 30
 62          periodSeconds: 10
 65          timeoutSeconds: 3
 68          failureThreshold: 1
 70---
 71apiVersion: apps/v1
 72kind: Deployment
 73metadata:
 74  name: inventory-deployment
 75  labels:
 76    app: inventory
 77spec:
 78  selector:
 79    matchLabels:
 80      app: inventory
 81  strategy:
 82    type: RollingUpdate
 83    rollingUpdate:
 84      maxUnavailable: 1
 85      maxSurge: 1
 86  template:
 87    metadata:
 88      labels:
 89        app: inventory
 90    spec:
 91      containers:
 92      - name: inventory-container
 93        image: inventory:1.0-SNAPSHOT
 94        ports:
 95        - containerPort: 9090
 96        env:
 97        - name: SYS_APP_HOSTNAME
 98          value: system-service
 99        # inventory probes
100        startupProbe:
101          httpGet:
103            path: /health/started
105            port: 9090
106        livenessProbe:
107          httpGet:
109            path: /health/live
111            port: 9090
113          initialDelaySeconds: 60
116          periodSeconds: 10
119          timeoutSeconds: 3
122          failureThreshold: 1
124        readinessProbe:
125          httpGet:
127            path: /health/ready
129            port: 9090
131          initialDelaySeconds: 30
134          periodSeconds: 10
137          timeoutSeconds: 3
140          failureThreshold: 1
142---
143apiVersion: v1
144kind: Service
145metadata:
146  name: system-service
147spec:
148  type: NodePort
149  selector:
150    app: system
151  ports:
152  - protocol: TCP
153    port: 9090
154    targetPort: 9090
155    nodePort: 31000
156---
157apiVersion: v1
158kind: Service
159metadata:
160  name: inventory-service
161spec:
162  type: NodePort
163  selector:
164    app: inventory
165  ports:
166  - protocol: TCP
167    port: 9090
168    targetPort: 9090
169    nodePort: 32000170

Prerequisites:

Nice work! Where to next?

Nice work! You have used MicroProfile Health and Open Liberty to create endpoints that report on your microservice’s status. Then, you observed how Kubernetes uses the /health/started, /health/live, and /health/ready endpoints to keep your microservices running smoothly.

Checking the health of microservices on Kubernetes by Open Liberty is licensed under CC BY-ND 4.0