Checking the health of microservices on Kubernetes

duration 20 minutes

Prerequisites:

Learn how to check the health of microservices on Kubernetes by setting up readiness and liveness probes to inspect MicroProfile Health Check endpoints.

What you’ll learn

You will learn how to create health check endpoints for your microservices. Then, you will configure Kubernetes to use these endpoints to keep your microservices running smoothly.

MicroProfile Health allows services to report their health, and it publishes the overall health status to defined endpoints. If a service reports UP, then it’s available. If the service reports DOWN, then it’s unavailable. MicroProfile Health reports an individual service status at the endpoint and indicates the overall status as UP if all the services are UP. A service orchestrator can then use the health statuses to make decisions.

Kubernetes provides liveness and readiness probes that are used to check the health of your containers. These probes can check certain files in your containers, check a TCP socket, or make HTTP requests. MicroProfile Health exposes readiness and liveness endpoints on your microservices. Kubernetes polls these endpoints as specified by the probes to react appropriately to any change in the microservice’s status. Read the Adding health reports to microservices guide to learn more about MicroProfile Health.

The two microservices you will work with are called system and inventory. The system microservice returns the JVM system properties of the running container and it returns the pod’s name in the HTTP header making replicas easy to distinguish from each other. The inventory microservice adds the properties from the system microservice to the inventory. This demonstrates how communication can be established between pods inside a cluster.

Additional prerequisites

Before you begin, you need a containerization software for building containers. Kubernetes supports various container runtimes. You will use Docker in this guide. For Docker installation instructions, refer to the official Docker documentation.

Use Docker Desktop, where a local Kubernetes environment is pre-installed and enabled. If you do not see the Kubernetes tab, then upgrade to the latest version of Docker Desktop.

Complete the setup for your operating system:

After you complete the Docker setup instructions for your operating system, ensure that Kubernetes (not Swarm) is selected as the orchestrator in Docker Preferences.

Use Docker Desktop, where a local Kubernetes environment is pre-installed and enabled. If you do not see the Kubernetes tab, then upgrade to the latest version of Docker Desktop.

Complete the setup for your operating system:

After you complete the Docker setup instructions for your operating system, ensure that Kubernetes (not Swarm) is selected as the orchestrator in Docker Preferences.

You will use Minikube as a single-node Kubernetes cluster that runs locally in a virtual machine. For Minikube installation instructions, see the Minikube documentation. Be sure to read the Requirements section, as different operating systems require different prerequisites to run Minikube.

Getting started

The fastest way to work through this guide is to clone the Git repository and use the projects that are provided inside:

git clone https://github.com/openliberty/guide-kubernetes-microprofile-health.git
cd guide-kubernetes-microprofile-health

The start directory contains the starting project that you will build upon.

The finish directory contains the finished project that you will build.

Starting and preparing your cluster for deployment

Start your Kubernetes cluster.

Start your Docker Desktop environment.

Run the following command from a command line:

minikube start

Next, validate that you have a healthy Kubernetes environment by running the following command from the command line.

kubectl get nodes

This command should return a Ready status for the master node.

You do not need to do any other step.

Run the following command to configure the Docker CLI to use Minikube’s Docker daemon. After you run this command, you will be able to interact with Minikube’s Docker daemon and build new images directly to it from your host machine:

eval $(minikube docker-env)

Adding health checks to the inventory microservice

Navigate to start directory to begin.

The inventory microservice should be healthy only when system is available. To add this check to the /health/ready endpoint, you will create a class that is annotated with the @Readiness annotation and implements the HealthCheck interface.

Create InventoryReadinessCheck class.
inventory/src/main/java/io/openliberty/guides/inventory/InventoryReadinessCheck.java

InventoryReadinessCheck.java

 1package io.openliberty.guides.inventory;
 2
 3import javax.enterprise.context.ApplicationScoped;
 4import javax.inject.Inject;
 5import javax.ws.rs.client.Client;
 6import javax.ws.rs.client.ClientBuilder;
 7
 8import org.eclipse.microprofile.config.inject.ConfigProperty;
 9import org.eclipse.microprofile.health.Readiness;
10import org.eclipse.microprofile.health.HealthCheck;
11import org.eclipse.microprofile.health.HealthCheckResponse;
12import org.eclipse.microprofile.health.HealthCheckResponseBuilder;
13
14@Readiness
15@ApplicationScoped
16public class InventoryReadinessCheck implements HealthCheck {
17    @Inject
18    @ConfigProperty(name = "SYS_APP_HOSTNAME")
19    private String hostname;
20
21    public HealthCheckResponse call() {
22        HealthCheckResponseBuilder builder = HealthCheckResponse.named(hostname);
23        if (isSystemServiceReachable()) {
24            builder = builder.up();
25        } else {
26            builder = builder.down();
27        }
28
29        return builder.build();
30    }
31
32    private boolean isSystemServiceReachable() {
33        try {
34            Client client = ClientBuilder.newClient();
35            client
36                .target("http://" + hostname + ":9080/system/properties")
37                .request()
38                .post(null);
39
40            return true;
41        } catch (Exception ex) {
42            return false;
43        }
44    }
45}

This health check verifies that the system microservice is available at http://system-service:9080/. The system-service host name is only accessible from inside the cluster, you can’t access it yourself. If it’s available, then it returns an UP status. Similarly, if it’s unavailable then it returns a DOWN status. When the status is DOWN, the microservice is considered to be unhealthy.

Create InventoryLivenessCheck class.
inventory/src/main/java/io/openliberty/guides/inventory/InventoryLivenessCheck.java

InventoryLivenessCheck.java

 1package io.openliberty.guides.inventory;
 2
 3import javax.enterprise.context.ApplicationScoped;
 4
 5import java.lang.management.MemoryMXBean;
 6import java.lang.management.ManagementFactory;
 7
 8import org.eclipse.microprofile.health.Liveness;
 9import org.eclipse.microprofile.health.HealthCheck;
10import org.eclipse.microprofile.health.HealthCheckResponse;
11
12@Liveness
13@ApplicationScoped
14public class InventoryLivenessCheck implements HealthCheck {
15
16  @Override
17  public HealthCheckResponse call() {
18      MemoryMXBean memBean = ManagementFactory.getMemoryMXBean();
19      long memUsed = memBean.getHeapMemoryUsage().getUsed();
20      long memMax = memBean.getHeapMemoryUsage().getMax();
21
22      return HealthCheckResponse.named(InventoryResource.class.getSimpleName() + "Liveness")
23                                .withData("memory used", memUsed)
24                                .withData("memory max", memMax)
25                                .state(memUsed < memMax * 0.9).build();
26  }
27}

This liveness check verifies that the heap memory usage is below 90% of the maximum memory. If more than 90% of the maximum memory is used, a status of DOWN will be returned.

The health checks for the system microservice were already been implemented. The system microservice was set up to become unhealthy for 60 seconds when a specific endpoint is called. This endpoint has been provided for you to observe the results of an unhealthy pod and how Kubernetes reacts.

Configuring readiness and liveness probes

You will configure Kubernetes readiness and liveness probes. Readiness probes are responsible for determining that your application is ready to accept requests. If it’s not ready, traffic won’t be routed to the container. Liveness probes are responsible for determining when a container needs to be restarted.

Create the kubernetes configuration file.
kubernetes.yaml

kubernetes.yaml

  1apiVersion: apps/v1
  2kind: Deployment
  3metadata:
  4  name: system-deployment
  5  labels:
  6    app: system
  7spec:
  8  replicas: 2
  9  selector:
 10    matchLabels:
 11      app: system
 12  template:
 13    metadata:
 14      labels:
 15        app: system
 16    spec:
 17      containers:
 18      - name: system-container
 19        image: system:1.0-SNAPSHOT
 20        ports:
 21        - containerPort: 9080
 22        # system probes
 23        readinessProbe:
 24          httpGet:
 25            path: /health/ready
 26            port: 9080
 27          initialDelaySeconds: 30
 28          periodSeconds: 10
 29          timeoutSeconds: 3
 30          failureThreshold: 1
 31        livenessProbe:
 32          httpGet:
 33            path: /health/live
 34            port: 9080
 35          initialDelaySeconds: 60
 36          periodSeconds: 10
 37          timeoutSeconds: 3
 38          failureThreshold: 1
 39---
 40apiVersion: apps/v1
 41kind: Deployment
 42metadata:
 43  name: inventory-deployment
 44  labels:
 45    app: inventory
 46spec:
 47  selector:
 48    matchLabels:
 49      app: inventory
 50  template:
 51    metadata:
 52      labels:
 53        app: inventory
 54    spec:
 55      containers:
 56      - name: inventory-container
 57        image: inventory:1.0-SNAPSHOT
 58        ports:
 59        - containerPort: 9080
 60        env:
 61        - name: SYS_APP_HOSTNAME
 62          value: system-service
 63        # inventory probe
 64        readinessProbe:
 65          httpGet:
 66            path: /health/ready
 67            port: 9080
 68          initialDelaySeconds: 30
 69          periodSeconds: 10
 70          timeoutSeconds: 3
 71          failureThreshold: 1
 72        livenessProbe:
 73          httpGet:
 74            path: /health/live
 75            port: 9080
 76          initialDelaySeconds: 60
 77          periodSeconds: 10
 78          timeoutSeconds: 3
 79          failureThreshold: 1
 80---
 81apiVersion: v1
 82kind: Service
 83metadata:
 84  name: system-service
 85spec:
 86  type: NodePort
 87  selector:
 88    app: system
 89  ports:
 90  - protocol: TCP
 91    port: 9080
 92    targetPort: 9080
 93    nodePort: 31000
 94---
 95apiVersion: v1
 96kind: Service
 97metadata:
 98  name: inventory-service
 99spec:
100  type: NodePort
101  selector:
102    app: inventory
103  ports:
104  - protocol: TCP
105    port: 9080
106    targetPort: 9080
107    nodePort: 32000

The readiness and liveness probes are configured for the containers running the system and inventory microservices.

The readiness probes are configured to poll the /health/ready endpoint. The readiness probe determines the READY status of the container as seen in the kubectl get pods output. The initialDelaySeconds field defines how long the probe should wait before it starts to poll so the probe does not start making requests before the server has started. The failureThreshold option defines how many times the probe should fail before the state should be changed from ready to not ready. The timeoutSeconds option defines how many seconds before the probe times out. The periodSeconds option defines how often the probe should poll the given endpoint.

The liveness probes are configured to poll the /health/live endpoint. The liveness probes determine when a container needs to be restarted. Similar to the readiness probes, the liveness probes also define initialDelaySeconds, failureThreshold, timeoutSeconds, and periodSeconds.

Deploying the microservices

To build these microservices, navigate to the start directory and run the following command.

mvn package

Next, run the docker build commands to build container images for your application:

docker build -t system:1.0-SNAPSHOT system/.
docker build -t inventory:1.0-SNAPSHOT inventory/.

The -t flag in the docker build command allows the Docker image to be labeled (tagged) in the name[:tag] format. The tag for an image describes the specific image version. If the optional [:tag] tag is not specified, the latest tag is created by default.

When the builds succeed, run the following command to deploy the necessary Kubernetes resources to serve the applications.

kubectl apply -f kubernetes.yaml

Use the following command to view the status of the pods. There will be two system pods and one inventory pod, later you’ll observe their behavior as the system pods become unhealthy.

kubectl get pods
NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          59s
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          59s
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          59s

Wait until the pods are ready. After the pods are ready, you will make requests to your services.

The default host name for Docker Desktop is localhost.

The default host name for minikube is 192.168.99.100. Otherwise it can be found using the minikube ip command.

Navigate to http://[hostname]:31000/system/properties and observe a response containing JVM system properties. Replace [hostname] with the IP address or host name of your Kubernetes cluster. The readiness probe ensures the READY state won’t be 1/1 until the container is available to accept requests. Without a readiness probe, you may notice an unsuccessful response from the server. This scenario can occur when the container has started, but the application server hasn’t fully initialized. With the readiness probe, you can be certain the pod will only accept traffic when the microservice has fully started.

Similarly, navigate to http://[hostname]:32000/inventory/systems/system-service and observe that the request is successful.

Changing the ready state of the system microservice

An endpoint has been provided under the system microservice to set it to an unhealthy state in the health check. The unhealthy state will cause the readiness probe to fail. Use the curl command to invoke this endpoint by making a POST request to http://[hostname]:31000/system/unhealthy — if curl is unavailable then use a tool such as Postman.

curl -X POST http://[hostname]:31000/system/unhealthy

Run the following command to view the state of the pods:

kubectl get pods
NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          59s
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          59s
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          59s
NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          1m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          1m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          1m

You will notice that one of the two system pods is no longer in the ready state. Navigate to http://[hostname]:31000/system/properties. Observe that your request will still be successful because you have two replicas and one is still healthy.

Observing the effects on the inventory microservice

Wait until the system pod is ready again. Make two POST requests to http://[hostname]:31000/system/unhealthy. If you see the same pod name twice, make the request again until you see that the second pod has been made unhealthy. You may see the same pod twice because there’s a delay between a pod becoming unhealthy and the readiness probe noticing it. Therefore, traffic may still be routed to the unhealthy service for approximately 5 seconds. Continue to observe the output of kubectl get pods. You will see both pods are no longer ready. During this process, the readiness probe for the inventory microservice will also fail. Observe it’s no longer in the ready state either.

First, both system pods will no longer be ready because the readiness probe failed.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     0/1       Running   0          5m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          5m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          5m

Next, the inventory pod is no longer ready because the readiness probe failed. The probe failed because system-service is now unavailable.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     0/1       Running   0          6m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          6m
inventory-deployment-cf8f564c6-nctcr   0/1       Running   0          6m

Then, the system pods will start to become healthy again after 60 seconds.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          6m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          6m
inventory-deployment-cf8f564c6-nctcr   0/1       Running   0          6m

Finally, you will see all of the pods have recovered.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          6m
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          6m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          6m

Testing the microservices

Run the tests by running the following command and appropriately substituting [hostname] for the correct value.

mvn verify -Dcluster.ip=[hostname]

A few tests are included for you to test the basic functions of the microservices. If a test failure occurs, then you might have introduced a bug into the code. To run the tests, wait for all pods to be in the ready state before proceeding further.

When the tests succeed, you should see output similar to the following in your console.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.system.SystemEndpointTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.65 s - in it.io.openliberty.guides.system.SystemEndpointTest

Results:

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.inventory.InventoryEndpointTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.542 s - in it.io.openliberty.guides.inventory.InventoryEndpointTest

Results:

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Tearing down the environment

To remove all of the resources created during this guide, run the following command to delete all of the resources that you created.

kubectl delete -f kubernetes.yaml

Nothing more needs to be done for Docker Desktop.

Perform the following steps to return your environment to a clean state.

  1. Point the Docker daemon back to your local machine:

    eval $(minikube docker-env -u)
  2. Stop your Minikube cluster:

    minikube stop
  3. Delete your cluster:

    minikube delete

Great work! You’re done!

You have used MicroProfile Health and Open Liberty to create endpoints that report on your microservice’s status. Then, you observed how Kubernetes uses the /health/ready and /health/live endpoints to keep your microservices running smoothly.

Guide Attribution

Checking the health of microservices on Kubernetes by Open Liberty is licensed under CC BY-ND 4.0

Copied to clipboard
Copy code block
Copy file contents

Prerequisites:

Nice work! Where to next?

What did you think of this guide?

Extreme Dislike Dislike Like Extreme Like

What could make this guide better?

Raise an issue to share feedback

Create a pull request to contribute to this guide

Need help?

Ask a question on Stack Overflow

Like Open Liberty? Star our repo on GitHub.

Star