Checking the health of microservices on Kubernetes

duration 20 minutes

Learn how to check the health of microservices on Kubernetes by setting up readiness probes to inspect MicroProfile Health Check endpoints.

What you’ll learn

You will learn how to create a health check endpoint for your microservices. Then, you will configure Kubernetes to use this endpoint to keep your microservices running smoothly.

MicroProfile Health allows services to report their health, and it publishes the overall health status to a defined endpoint. A service reports UP if it is available and reports DOWN if it is unavailable. MicroProfile Health reports an individual service status at the endpoint and indicates the overall status as UP if all the services are UP. A service orchestrator can then use the health statuses to make decisions.

Kubernetes provides liveness and readiness probes that are used to check the health of your containers, you will work with readiness probes. These probes can check certain files in your containers, check a TCP socket, or make HTTP requests. MicroProfile Health exposes a health endpoint on your microservices. Kubernetes polls the endpoint as specified by the probes to react appropriately to any change in the microservice’s status. Read the Adding health reports to microservices guide to learn more about MicroProfile Health.

The two microservices you will work with are called name and ping. The name microservice displays a brief greeting and the name of the container that it runs in. The ping microservice pings the Kubernetes Service that encapsulates the pod running the name microservice. The ping microservice demonstrates how communication can be established between pods inside a cluster.

Prerequisites

Before you begin, make sure to have the following tools installed:

First, you will need a containerization software for building containers. Kubernetes supports a variety of container types. You will use Docker in this guide. For installation instructions, see https://docs.docker.com/install/.

Windows | Mac

Use the Docker Desktop, where a local Kubernetes environment is pre-installed and enabled. If you do not see the Kubernetes tab then you have an older version of Docker Desktop; upgrade to the latest version.

Linux

You will use Minikube as a single-node Kubernetes cluster that runs locally in a virtual machine. For Minikube installation instructions see https://github.com/kubernetes/minikube. Make sure to read the "Requirements" section as different operating systems require different prerequisites to get Minikube running.

Getting started

The fastest way to work through this guide is to clone the Git repository and use the projects that are provided inside:

git clone https://github.com/openliberty/guide-kubernetes-microprofile-health.git
cd guide-kubernetes-microprofile-health

The start directory contains the starting project that you will build upon.

The finish directory contains the finished project, which is what you will build.

Starting and preparing your cluster for deployment

Start up your Kubernetes cluster.

Windows | Mac

Start your Docker Desktop environment.

Linux

Run the following command from a command line:

minikube start

Next, validate that you have a healthy Kubernetes environment by running the following command from the command line.

kubectl get nodes

This should return a Ready status for the master node.

Windows | Mac

You do not need to do any other step.

Linux

Run the following command to configure the Docker CLI to use Minikube’s Docker daemon. After running this command, you will be able to interact with Minikube’s Docker daemon and build new images directly to it from your host machine:

eval $(minikube docker-env)

Adding a health check to the ping microservice

Navigate to start directory to begin.

The ping microservice should only be healthy when name is available. To add this check to the /health endpoint, you will create a class implementing the HealthCheck interface.

Create the PingHealth class in ping/src/main/java/io/openliberty/guides/ping/PingHealth.java:

package io.openliberty.guides.ping;

import java.net.URL;

import javax.enterprise.context.ApplicationScoped;
import javax.inject.Inject;

import org.eclipse.microprofile.config.inject.ConfigProperty;
import org.eclipse.microprofile.health.Health;
import org.eclipse.microprofile.health.HealthCheck;
import org.eclipse.microprofile.health.HealthCheckResponse;
import org.eclipse.microprofile.health.HealthCheckResponseBuilder;
import org.eclipse.microprofile.rest.client.RestClientBuilder;

import io.openliberty.guides.ping.client.NameClient;
import io.openliberty.guides.ping.client.UnknownUrlException;

@Health
@ApplicationScoped
public class PingHealth implements HealthCheck {
    @Inject
    @ConfigProperty(name = "NAME_HOSTNAME")
    private String hostname;

    public HealthCheckResponse call() {
        HealthCheckResponseBuilder builder = HealthCheckResponse.named(hostname);
        if (isNameServiceReachable()) {
            builder = builder.up();
        } else {
            builder = builder.down();
        }

        return builder.build();
    }

    private boolean isNameServiceReachable() {
        try {
            NameClient client = RestClientBuilder
                .newBuilder()
                .baseUrl(new URL("http://" + hostname + ":9080/api"))
                .register(UnknownUrlException.class)
                .build(NameClient.class);

            client.getContainerName();
            return true;
        } catch (Exception ex) {
            return false;
        }
    }
}

This health check verifies that the name microservice is available at http://name-service:9080/api. The name-service host name is only accessible from inside the cluster, you can’t access it yourself. If it’s available, then it returns an UP status. Similarly, if it’s unavailable then it returns a DOWN status. When the status is DOWN the microservice is considered to be unhealthy.

The health check for the name microservice has already been implemented. It has been setup to become unhealthy for 60 seconds when a specific endpoint is called. This endpoint has been provided for you to observe the results of an unhealthy pod and how Kubernetes reacts.

Configuring readiness probes

You will configure Kubernetes readiness probes. Readiness probes are responsible for determining that your application is ready to accept requests. If it’s not ready, traffic won’t be routed to the container.

Create the kubernetes.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: name-deployment
  labels:
    app: name
spec:
  replicas: 2
  selector:
    matchLabels:
      app: name
  template:
    metadata:
      labels:
        app: name
    spec:
      containers:
      - name: name-container
        image: name:1.0-SNAPSHOT
        ports:
        - containerPort: 9080
        # name probe
        readinessProbe:
          httpGet:
            path: /health
            port: 9080
          initialDelaySeconds: 15
          periodSeconds: 5
          failureThreshold: 1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ping-deployment
  labels:
    app: ping
spec:
  selector:
    matchLabels:
      app: ping
  template:
    metadata:
      labels:
        app: ping
    spec:
      containers:
      - name: ping-container
        image: ping:1.0-SNAPSHOT
        ports:
        - containerPort: 9080
        env:
        - name: NAME_HOSTNAME
          value: name-service
        # ping probe
        readinessProbe:
          httpGet:
            path: /health
            port: 9080
          initialDelaySeconds: 15
          periodSeconds: 5
          failureThreshold: 1
---
apiVersion: v1
kind: Service
metadata:
  name: name-service
spec:
  type: NodePort
  selector:
    app: name
  ports:
  - protocol: TCP
    port: 9080
    targetPort: 9080
    nodePort: 31000
---
apiVersion: v1
kind: Service
metadata:
  name: ping-service
spec:
  type: NodePort
  selector:
    app: ping
  ports:
  - protocol: TCP
    port: 9080
    targetPort: 9080
    nodePort: 32000

The readiness probes are configured for the containers running the name and ping microservices.

The readiness probes are configured to poll the /health endpoint. The readiness probe determines the READY status of the container as seen in the kubectl get pods output. The initialDelaySeconds field defines how long the probe should wait before it starts to poll so the probe does not start making requests before the server has started. The failureThreshold option defines how many times the probe should fail before the state should be changed from ready to not ready. The periodSeconds option defines how often the probe should poll the given endpoint.

Deploying the microservices

To build these microservices, navigate to the start directory and run the following command.

mvn package

When the build succeeds, run the following command to deploy the necessary Kubernetes resources to serve the applications.

kubectl apply -f kubernetes.yaml

Use the following command to view the status of the pods. There will be two name pods and one ping pod, later you’ll observe their behaviour as the name pods become unhealthy.

kubectl get pods
NAME                               READY     STATUS    RESTARTS   AGE
name-deployment-694c7b74f7-hcf4q   1/1       Running   0          59s
name-deployment-694c7b74f7-lrlf7   1/1       Running   0          59s
ping-deployment-cf8f564c6-nctcr    1/1       Running   0          59s

Wait until the pods are ready. After the pods are ready, you will make requests to your services.

Windows | Mac

The default host name for Docker Desktop is localhost.

Linux

The default host name for minikube is 192.168.99.100. Otherwise it can be found using the minikube ip command.

Navigate to http://[hostname]:31000/api/name and observe a response similar to Hello! I’m container name-deployment-5f868854bf-2rhdq. Replace [hostname] with the IP address or host name of your Kubernetes cluster. The readiness probe ensures the READY state won’t be 1/1 until the container is available to accept requests. Without a readiness probe, you may notice an unsuccessful response from the server. This scenario can occur when the container has started, but the application server hasn’t fully initialized. With the readiness probe, you can be certain the pod will only accept traffic when the microservice has fully started.

Similarly, navigate to http://[hostname]:32000/api/ping/name-service and observe a response with the content pong.

Changing the ready state of the name microservice

An endpoint has been provided under the name microservice to set it to an unhealthy state in the health check. The unhealthy state will cause the readiness probe to fail. Use the curl command to invoke this endpoint by making a POST request to http://[hostname]:31000/api/name/unhealthy — if curl is unavailable then use a tool such as Postman.

curl -X POST http://[hostname]:31000/api/name/unhealthy

Run the following command to view the state of the pods:

kubectl get pods
NAME                               READY     STATUS    RESTARTS   AGE
name-deployment-694c7b74f7-hcf4q   1/1       Running   0          59s
name-deployment-694c7b74f7-lrlf7   1/1       Running   0          59s
ping-deployment-cf8f564c6-nctcr    1/1       Running   0          59s
NAME                               READY     STATUS    RESTARTS   AGE
name-deployment-694c7b74f7-hcf4q   1/1       Running   0          1m
name-deployment-694c7b74f7-lrlf7   0/1       Running   0          1m
ping-deployment-cf8f564c6-nctcr    1/1       Running   0          1m

You will notice that one of the two name pods is no longer in the ready state. Navigate to http://[hostname]:31000/api/name. Observe that your request will still be successful because you have two replicas and one is still healthy.

Observing the effects on the ping microservice

Wait until the name pod is ready again. Make two POST requests to http://[hostname]:31000/api/name/unhealthy. If you see the same pod name twice, make the request again until you see that the second pod has been made unhealthy. You may see the same pod twice because there’s a delay between a pod becoming unhealthy and the readiness probe noticing it. Therefore, traffic may still be routed to the unhealthy service for approximately 5 seconds. Continue to observe the output of kubectl get pods. You will see both pods are no longer ready. During this process, the readiness probe for the ping microservice will also fail. Observe it’s no longer in the ready state either.

First, both name pods will no longer be ready because the readiness probe failed.

NAME                               READY     STATUS    RESTARTS   AGE
name-deployment-694c7b74f7-hcf4q   0/1       Running   0          5m
name-deployment-694c7b74f7-lrlf7   0/1       Running   0          5m
ping-deployment-cf8f564c6-nctcr    1/1       Running   0          5m

Next, the ping pod is no longer ready because the readiness probe failed. The probe failed because name-service is now unavailable.

NAME                               READY     STATUS    RESTARTS   AGE
name-deployment-694c7b74f7-hcf4q   0/1       Running   0          6m
name-deployment-694c7b74f7-lrlf7   0/1       Running   0          6m
ping-deployment-cf8f564c6-nctcr    0/1       Running   0          6m

Then, the name pods will start to recover.

NAME                               READY     STATUS    RESTARTS   AGE
name-deployment-694c7b74f7-hcf4q   1/1       Running   0          6m
name-deployment-694c7b74f7-lrlf7   0/1       Running   0          6m
ping-deployment-cf8f564c6-nctcr    0/1       Running   0          6m

Finally, you will see all of the pods have recovered.

NAME                               READY     STATUS    RESTARTS   AGE
name-deployment-694c7b74f7-hcf4q   1/1       Running   0          6m
name-deployment-694c7b74f7-lrlf7   1/1       Running   0          6m
ping-deployment-cf8f564c6-nctcr    1/1       Running   0          6m

Testing the microservices

Run the tests by running the following command and appropriately substituting [hostname] for the correct value.

mvn verify -Ddockerfile.skip=true -Dcluster.ip=[hostname]

The tests verify that the health endpoints for the name and ping microservices respond with HTTP status 200 indicating that they are healthy.

When the tests succeed, you should see output similar to the following in your console.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.name.NameEndpointTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.39 sec - in it.io.openliberty.guides.name.NameEndpointTest

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running it.io.openliberty.guides.ping.PingEndpointTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.43 sec - in it.io.openliberty.guides.ping.PingEndpointTest

Results :

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

Tearing down the environment

To remove all of the resources created during this guide, run the following command to delete all of the resources that you created.

kubectl delete -f kubernetes.yaml

Windows | Mac

Nothing more needs to be done for Docker Desktop.

Linux

Perform the following two steps to return your environment to a clean state.

Firstly, point the Docker daemon back to your local machine:

eval $(minikube docker-env -u)

Then, stop your Minikube cluster:

minikube stop

Finally, delete your cluster:

minikube delete

Great work! You’re done!

You have used MicroProfile Health to create an endpoint that reports on your microservice’s status. Then, you observed how Kubernetes uses the /health endpoint to keep your microservices running smoothly.

Contribute to this guide

Is something missing or broken? Raise an issue, or send us a pull request.