When we have both MicroProfile and Istio Fault Tolerance capabilities enabled, there is a compounding effect that may be unexpected. If both MicroProfile and Istio set their own Retry policies on a service, the maximum number of retries that occur is not equivalent to either of the number of retries specified in MicroProfile or Istio. The number of retries set by MicroProfile and Istio are actually multiplied.
If you want to use Istio as your service mesh and only its fault tolerance capabilities, you can turn off MicroProfile Fault Tolerance by adding a property. This configuration avoids any overlap in behaviours.
MicroProfile Fault Tolerance offers a config property
MP_Fault_Tolerance_NonFallback_Enabled that disables all MicroProfile Fault Tolerance capabilities except fallback. If
MP_Fault_Tolerance_NonFallback_Enabled is set to false, only the
@Fallback behaviour is enabled. The other behaviours specified by the MicroProfile Fault Tolerance annotations, including
@Retry, won’t take effect.
You will define the
MP_Fault_Tolerance_NonFallback_Enabled config property in a ConfigMap. ConfigMaps store configuration settings about a Kubernetes pod. This configuration is loaded into the pod as an environment variable that is used by the pod’s containers. The environment variables are defined in the pod’s specification by using the
envFrom field. To learn more about ConfigMaps, check out the Configuring microservices running in Kubernetes guide.
MP_Fault_Tolerance_NonFallback_Enabled config property to disable the retries performed by MicroProfile, so that only Istio performs retries.
5 name: sys-app-gateway
8 istio: ingressgateway
10 - port:
11 number: 80
12 name: http
13 protocol: HTTP
15 - "system.example.com"
16 - "inventory.example.com"
22 name: system-service
24 app: system
27 - port: 9080
28 name: http
30 app: system
35 name: inventory-service
37 app: inventory
40 - port: 9081
41 name: http
43 app: inventory
48 name: system-deployment
50 app: system
54 app: system
58 app: system
61 - name: system-container
62 image: system:1.0-SNAPSHOT
64 - containerPort: 9080
69 name: inventory-deployment
71 app: inventory
75 app: inventory
79 app: inventory
82 - name: inventory-container
83 image: inventory:1.0-SNAPSHOT
85 - containerPort: 9081
88 - configMapRef:
89 # tag::configName2
90 name: inventory-config
91 # end::configName2
100 name: inventory-config
104 MP_Fault_Tolerance_NonFallback_Enabled: "false"
ConfigMap into the
services.yaml file, and set the
MP_Fault_Tolerance_NonFallback_Enabled config property to false. Add the
envFrom field to inject the
ConfigMap with the
MP_Fault_Tolerance_NonFallback_Enabled property into your pods.
name of the
ConfigMap, which is
inventory-config, becomes the environment variable
name that is specified in the
Deploy your microservices again to turn off all MicroProfile Fault Tolerance capabilities, except fallback:
kubectl replace --force -f services.yaml
Wait until all of your deployments are ready and available. Run the
kubectl get pods command to get the new
[system-pod-name]. Pause the
system service pod to simulate that the service is unavailable:
kubectl exec -it [system-pod-name] /opt/ol/wlp/bin/server pause
Make a request to the service by using
curl -H Host:inventory.example.com http://localhost/inventory/systems/system-service -I
curl command is unavailable, then use Postman.
curl -H Host:inventory.example.com http://`minikube ip`:31380/inventory/systems/system-service -I
system service is unavailable, the request still returns a
503 response code. This time, however, the request was retried several times with Istio, without any retries from MicroProfile.
See the number of times that the service was retried:
kubectl logs [system-pod-name] -c istio-proxy | grep -c system-service:9080
kubectl logs [system-pod-name] -c istio-proxy | find /C "system-service:9080"
You will see the following output:
The above command returns 15, indicating that a total of 15 requests are made to the
system service. Since MicroProfile’s Retry policy is disabled, only Istio’s retries are performed.
4 name: system-virtual-service
7 - "system.example.com"
9 - sys-app-gateway
11 - route:
12 - destination:
14 number: 9080
15 host: system-service
20 name: inventory-virtual-service
23 - "inventory.example.com"
25 - sys-app-gateway
28 - route:
29 - destination:
31 number: 9081
32 host: inventory-service
36 attempts: 4
39 retryOn: 5xx