Checking the health of microservices on Kubernetes

duration 20 minutes
Git clone to get going right away:
git clone https://github.com/OpenLiberty/guide-kubernetes-microprofile-health.git
Copy Github clone command

Learn how to check the health of microservices on Kubernetes by setting up readiness probes to inspect MicroProfile Health Check endpoints.

What you’ll learn

You will learn how to create a health check endpoint for your microservices. Then, you will configure Kubernetes to use this endpoint to keep your microservices running smoothly.

MicroProfile Health allows services to report their health, and it publishes the overall health status to a defined endpoint. If a service reports UP, then it’s available. If the service reports DOWN, then it’s unavailable. MicroProfile Health reports an individual service status at the endpoint and indicates the overall status as UP if all the services are UP. A service orchestrator can then use the health statuses to make decisions.

Kubernetes provides liveness and readiness probes that are used to check the health of your containers, you will work with readiness probes. These probes can check certain files in your containers, check a TCP socket, or make HTTP requests. MicroProfile Health exposes a health endpoint on your microservices. Kubernetes polls the endpoint as specified by the probes to react appropriately to any change in the microservice’s status. Read the Adding health reports to microservices guide to learn more about MicroProfile Health.

The two microservices you will work with are called system and inventory. The system microservice returns the JVM system properties of the running container and it returns the pod’s name in the HTTP header making replicas easy to distinguish from each other. The inventory microservice adds the properties from the system microservice to the inventory. This demonstrates how communication can be established between pods inside a cluster.

Prerequisites

Before you begin, have the following tools installed:

First, you will need a containerization software for building containers. Kubernetes supports a variety of container types. You will use Docker in this guide. For installation instructions, refer to the official Docker documentation.

WINDOWS | MAC

Use Docker Desktop, where a local Kubernetes environment is pre-installed and enabled. If you do not see the Kubernetes tab then you have an older version of Docker Desktop; upgrade to the latest version.

Complete the setup for your operating system:

  • Set up Docker for Windows. On the Docker for Windows General Setting page, ensure that the option Expose daemon on tcp://localhost:2375 without TLS is enabled. This is required by the dockerfile-maven part of the build.

  • Set up Docker for Mac.

  • After following one of the sets of instructions, ensure that Kubernetes (not Swarm) is selected as the orchestrator in Docker Preferences.

LINUX

You will use Minikube as a single-node Kubernetes cluster that runs locally in a virtual machine. For Minikube installation instructions see the minikube installation instructions. Make sure to pay attention to the requirements as they vary by platform.

Getting started

The fastest way to work through this guide is to clone the Git repository and use the projects that are provided inside:

git clone https://github.com/openliberty/guide-kubernetes-microprofile-health.git
cd guide-kubernetes-microprofile-health

The start directory contains the starting project that you will build upon.

The finish directory contains the finished project that you will build.

Starting and preparing your cluster for deployment

Start your Kubernetes cluster.

WINDOWS | MAC

Start your Docker Desktop environment.

LINUX

Run the following command from a command line:

minikube start

Next, validate that you have a healthy Kubernetes environment by running the following command from the command line.

kubectl get nodes

This command should return a Ready status for the master node.

WINDOWS | MAC

You do not need to do any other step.

LINUX

Run the following command to configure the Docker CLI to use Minikube’s Docker daemon. After you run this command, you will be able to interact with Minikube’s Docker daemon and build new images directly to it from your host machine:

eval $(minikube docker-env)

Adding a health check to the inventory microservice

Navigate to start directory to begin.

The inventory microservice should only be healthy when system is available. To add this check to the /health endpoint, you will create a class implementing the HealthCheck interface.

Create InventoryHealth class.
inventory/src/main/java/io/openliberty/guides/inventory/InventoryHealth.java

InventoryHealth.java

 1package io.openliberty.guides.inventory;
 2
 3import java.net.URL;
 4import java.util.Properties;
 5
 6import javax.enterprise.context.ApplicationScoped;
 7import javax.inject.Inject;
 8import javax.ws.rs.client.Client;
 9import javax.ws.rs.client.ClientBuilder;
10
11import org.eclipse.microprofile.config.inject.ConfigProperty;
12import org.eclipse.microprofile.health.Health;
13import org.eclipse.microprofile.health.HealthCheck;
14import org.eclipse.microprofile.health.HealthCheckResponse;
15import org.eclipse.microprofile.health.HealthCheckResponseBuilder;
16import org.eclipse.microprofile.rest.client.RestClientBuilder;
17
18import io.openliberty.guides.inventory.client.SystemClient;
19
20@Health
21@ApplicationScoped
22public class InventoryHealth implements HealthCheck {
23    @Inject
24    @ConfigProperty(name = "SYS_APP_HOSTNAME")
25    private String hostname;
26
27    public HealthCheckResponse call() {
28        HealthCheckResponseBuilder builder = HealthCheckResponse.named(hostname);
29        if (isSystemServiceReachable()) {
30            builder = builder.up();
31        } else {
32            builder = builder.down();
33        }
34
35        return builder.build();
36    }
37
38    private boolean isSystemServiceReachable() {
39        try {
40            Client client = ClientBuilder.newClient();
41            client
42                .target("http://" + hostname + ":9080/system/properties")
43                .request()
44                .post(null);
45
46            return true;
47        } catch (Exception ex) {
48            return false;
49        }
50    }
51}

This health check verifies that the system microservice is available at http://system-service:9080/. The system-service host name is only accessible from inside the cluster, you can’t access it yourself. If it’s available, then it returns an UP status. Similarly, if it’s unavailable then it returns a DOWN status. When the status is DOWN, the microservice is considered to be unhealthy.

The health check for the system microservice has already been implemented. It has been set up to become unhealthy for 60 seconds when a specific endpoint is called. This endpoint has been provided for you to observe the results of an unhealthy pod and how Kubernetes reacts.

Configuring readiness probes

You will configure Kubernetes readiness probes. Readiness probes are responsible for determining that your application is ready to accept requests. If it’s not ready, traffic won’t be routed to the container.

Create the kubernetes configuration file.
kubernetes.yaml

kubernetes.yaml

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: system-deployment
 5  labels:
 6    app: system
 7spec:
 8  replicas: 2
 9  selector:
10    matchLabels:
11      app: system
12  template:
13    metadata:
14      labels:
15        app: system
16    spec:
17      containers:
18      - name: system-container
19        image: system:1.0-SNAPSHOT
20        ports:
21        - containerPort: 9080
22        # system probe
23        readinessProbe:
24          httpGet:
25            path: /health
26            port: 9080
27          initialDelaySeconds: 15
28          periodSeconds: 5
29          failureThreshold: 1
30---
31apiVersion: apps/v1
32kind: Deployment
33metadata:
34  name: inventory-deployment
35  labels:
36    app: inventory
37spec:
38  selector:
39    matchLabels:
40      app: inventory
41  template:
42    metadata:
43      labels:
44        app: inventory
45    spec:
46      containers:
47      - name: inventory-container
48        image: inventory:1.0-SNAPSHOT
49        ports:
50        - containerPort: 9080
51        env:
52        - name: SYS_APP_HOSTNAME
53          value: system-service
54        # inventory probe
55        readinessProbe:
56          httpGet:
57            path: /health
58            port: 9080
59          initialDelaySeconds: 15
60          periodSeconds: 5
61          failureThreshold: 1
62---
63apiVersion: v1
64kind: Service
65metadata:
66  name: system-service
67spec:
68  type: NodePort
69  selector:
70    app: system
71  ports:
72  - protocol: TCP
73    port: 9080
74    targetPort: 9080
75    nodePort: 31000
76---
77apiVersion: v1
78kind: Service
79metadata:
80  name: inventory-service
81spec:
82  type: NodePort
83  selector:
84    app: inventory
85  ports:
86  - protocol: TCP
87    port: 9080
88    targetPort: 9080
89    nodePort: 32000

The readiness probes are configured for the containers running the system and inventory microservices.

The readiness probes are configured to poll the /health endpoint. The readiness probe determines the READY status of the container as seen in the kubectl get pods output. The initialDelaySeconds field defines how long the probe should wait before it starts to poll so the probe does not start making requests before the server has started. The failureThreshold option defines how many times the probe should fail before the state should be changed from ready to not ready. The periodSeconds option defines how often the probe should poll the given endpoint.

Deploying the microservices

To build these microservices, navigate to the start directory and run the following command.

mvn package

When the build succeeds, run the following command to deploy the necessary Kubernetes resources to serve the applications.

kubectl apply -f kubernetes.yaml

Use the following command to view the status of the pods. There will be two system pods and one inventory pod, later you’ll observe their behavior as the system pods become unhealthy.

kubectl get pods
NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          59s
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          59s
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          59s

Wait until the pods are ready. After the pods are ready, you will make requests to your services.

WINDOWS | MAC

The default host name for Docker Desktop is localhost.

LINUX

The default host name for minikube is 192.168.99.100. Otherwise it can be found using the minikube ip command.

Navigate to http://[hostname]:31000/system/properties and observe a response containing JVM system properties. Replace [hostname] with the IP address or host name of your Kubernetes cluster. The readiness probe ensures the READY state won’t be 1/1 until the container is available to accept requests. Without a readiness probe, you may notice an unsuccessful response from the server. This scenario can occur when the container has started, but the application server hasn’t fully initialized. With the readiness probe, you can be certain the pod will only accept traffic when the microservice has fully started.

Similarly, navigate to http://[hostname]:32000/inventory/systems/system-service and observe that the request is successful.

Changing the ready state of the system microservice

An endpoint has been provided under the system microservice to set it to an unhealthy state in the health check. The unhealthy state will cause the readiness probe to fail. Use the curl command to invoke this endpoint by making a POST request to http://[hostname]:31000/system/unhealthy — if curl is unavailable then use a tool such as Postman.

curl -X POST http://[hostname]:31000/system/unhealthy

Run the following command to view the state of the pods:

kubectl get pods
NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          59s
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          59s
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          59s
NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          1m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          1m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          1m

You will notice that one of the two system pods is no longer in the ready state. Navigate to http://[hostname]:31000/system/properties. Observe that your request will still be successful because you have two replicas and one is still healthy.

Observing the effects on the inventory microservice

Wait until the system pod is ready again. Make two POST requests to http://[hostname]:31000/system/unhealthy. If you see the same pod name twice, make the request again until you see that the second pod has been made unhealthy. You may see the same pod twice because there’s a delay between a pod becoming unhealthy and the readiness probe noticing it. Therefore, traffic may still be routed to the unhealthy service for approximately 5 seconds. Continue to observe the output of kubectl get pods. You will see both pods are no longer ready. During this process, the readiness probe for the inventory microservice will also fail. Observe it’s no longer in the ready state either.

First, both system pods will no longer be ready because the readiness probe failed.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     0/1       Running   0          5m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          5m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          5m

Next, the inventory pod is no longer ready because the readiness probe failed. The probe failed because system-service is now unavailable.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     0/1       Running   0          6m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          6m
inventory-deployment-cf8f564c6-nctcr   0/1       Running   0          6m

Then, the system pods will start to become healthy again after 60 seconds.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          6m
system-deployment-694c7b74f7-lrlf7     0/1       Running   0          6m
inventory-deployment-cf8f564c6-nctcr   0/1       Running   0          6m

Finally, you will see all of the pods have recovered.

NAME                                   READY     STATUS    RESTARTS   AGE
system-deployment-694c7b74f7-hcf4q     1/1       Running   0          6m
system-deployment-694c7b74f7-lrlf7     1/1       Running   0          6m
inventory-deployment-cf8f564c6-nctcr   1/1       Running   0          6m

Testing the microservices

Run the tests by running the following command and appropriately substituting [hostname] for the correct value.

mvn verify -Ddockerfile.skip=true -Dcluster.ip=[hostname]

A few tests are included for you to test the basic functions of the microservices. If a test failure occurs, then you might have introduced a bug into the code. To run the tests, wait for all pods to be in the ready state before proceeding further.

When the tests succeed, you should see output similar to the following in your console.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.system.SystemEndpointTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.65 s - in it.io.openliberty.guides.system.SystemEndpointTest

Results:

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.inventory.InventoryEndpointTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.542 s - in it.io.openliberty.guides.inventory.InventoryEndpointTest

Results:

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Tearing down the environment

To remove all of the resources created during this guide, run the following command to delete all of the resources that you created.

kubectl delete -f kubernetes.yaml

WINDOWS | MAC

Nothing more needs to be done for Docker Desktop.

LINUX

Perform the following steps to return your environment to a clean state.

  1. Point the Docker daemon back to your local machine:

    eval $(minikube docker-env -u)
  2. Stop your Minikube cluster:

    minikube stop
  3. Delete your cluster:

    minikube delete

Great work! You’re done!

You have used MicroProfile Health and Open Liberty to create an endpoint that reports on your microservice’s status. Then, you observed how Kubernetes uses the /health endpoint to keep your microservices running smoothly.

Guide Attribution

Checking the health of microservices on Kubernetes by Open Liberty is licensed under CC BY-ND 4.0

Copied to clipboard
Copy code block
Copy file contents
Git clone this repo to get going right away:
git clone https://github.com/OpenLiberty/guide-kubernetes-microprofile-health.git
Copy github clone command
Copied to clipboard

Nice work! Where to next?

What did you think of this guide?

Extreme Dislike Dislike Like Extreme Like

What could make this guide better?

Raise an issue to share feedback

Create a pull request to contribute to this guide

Need help?

Ask a question on Stack Overflow

Like Open Liberty? Star our repo on GitHub.

Star