Istio from fooling to proficient: one-half live microservices

Posted May 27, 20209 min read


Author | Sheng Dong Alibaba Cloud After-sales Technical Expert

<Follow Alibaba Cloud Native Public Account, reply to investigation and download e-book>

The book "In-depth Kubernetes" brings together 12 technical articles to help you understand the 6 core principles at a time, understand the basic theory, and learn the gorgeous operation of 6 typical problems at a time!

Istio is the future! Basically, I believe that students who have some slight judgments about cloud native technology trends will have this consciousness. The logic behind it is actually relatively simple:when container clusters, especially Kubernetes, become the de facto standard, applications will inevitably become more complicated, and service governance will definitely become a strong demand.

The current status of Istio is that there are many people talking, but very few people use them. Therefore, the articles that we can see have a lot of reason, and very little experience of stepping on the pit. Alibaba Cloud after-sales team as a front-line team, sharing experience in troubleshooting, we are duty-bound. In this article, I will talk to you about a simple Istio problem troubleshooting process.

Half live microservices

The problem is this, users install Istio in their test cluster and deploy the bookinfo application to get started with Istio according to the official documentation. After deployment, the user executes the kubectl get pods command and finds that only one-half of all pods are READY.

# kubectl get pods
details-v1-68868454f5-94hzd 1/2 Running 0 1m
productpage-v1-5cb458d74f-28nlz 1/2 Running 0 1m
ratings-v1-76f4c9765f-gjjsc 1/2 Running 0 1m
reviews-v1-56f6855586-dplsf 1/2 Running 0 1m
reviews-v2-65c9df47f8-zdgbw 1/2 Running 0 1m
reviews-v3-6cf47594fd-cvrtf 1/2 Running 0 1m

If we have never paid attention to the READY column, we may have two doubts:what does 2 mean here, and what exactly does 1/2 mean.

Simply put, the READY column here gives the Readiness of each Pod's internal container, which is the ready state. The kubelet on each cluster node will confirm the Readiness of the corresponding container according to the definition of the Readiness rules of the container itself, which are tcp, http or exec, respectively.

More specifically, as a process running on each node, kubelet uses tcp/http(node network namespace to Pod network namespace) to access the interface defined by the container, or execute the command defined by exec in the namespace of the container. To determine if the container is ready.


Here, 2 indicates that there are two containers in these pods, and 1/2 means that only one container in each pod is ready, that is, it passes the Readiness test. Regarding this point, we will talk about it in the next section. Here we first look at why there is a container in all pods that is not ready.

Use the kubectl tool to pull the layout template of the first details pod. You can see that there are only two containers in this pod, and only one readiness probe is defined. For containers with no readiness probe defined, kubelet believes that as long as the processes in the container start running, the container will be in a ready state. So 1/2 ready Pods means that there is a container with a readiness probe defined, which fails the kubelet test.

What failed the readiness probe test was the istio-proxy container. Its readiness probe rules are defined as follows:


We log in to the node where the pod is located, use the curl tool to simulate kubelet access to the uri below, and test the ready state of istio-proxy.

# curl -v
* About to connect() to port 15020(# 0)
* Trying ...
* Connected to port 15020(# 0)
> GET/healthz/ready HTTP/1.1
> User-Agent:curl/7.29.0
> Host:
> Accept:*/*>
<HTTP/1.1 503 Service Unavailable <Date:Fri, 30 Aug 2019 16:43:50 GMT
Connection # 0 to host left intact

Big picture that can't get around

In the previous section, we described the phenomenon of the problem, but one problem remains, why is the number of containers in the Pod 2. Although each Pod essentially has at least two containers:one is a placeholder container pause, and the other is a real working container, but when we use the kubectl command to get the Pod list, the READY column does not include the pause container.

The other container here is actually the core concept of the service grid, sidercar. In fact, this container is called sidecar, in a sense it can not reflect the essence of this container. The sidecar container is essentially a reverse proxy. It was originally a load balancer for a pod to access other service backend pods.


However, when we carry a reverse proxy "on the fly" for each Pod in the cluster, the Pod and reverse proxy become a service mesh. As shown in the classic picture below. This picture is really a bit difficult to draw, so it can only be borrowed, not around.


Therefore, the sidecar mode is actually a "bring your own communicator" mode. What is more interesting here is that when we bind sidecar and Pod together, sidecar plays the role of reverse proxy when forwarding outgoing traffic, and can do more than reverse proxy when receiving incoming traffic thing. We will discuss this in other articles.

Istio implements a service mesh based on Kubernetes. The sidecar container used by Isito is the one mentioned in the first section, and there is no ready container. So this problem is actually inside the service mesh, all sidecar containers are not ready.

Agent and agent life cycle management

As we saw in the previous section, each Pod in Istio comes with a reverse proxy sidecar. The problem we encountered was that all the sidecars were not ready. We also see that readiness probe defines that the way to judge the sidecar container is ready is to access the following interface:

http://<pod ip>:15020/healthz/ready

Next, let's take a closer look at the Pod, and the composition and principle of its sidecar. In the service grid, in addition to the container that handles the business itself, a Pod also has the sidecar container istio-proxy. Under normal circumstances, istio-proxy will start two processes:pilot-agent and Envoy.

As shown in the following figure, Envoy is actually an agent responsible for functions such as traffic management. Data flows in and out of the business container must pass through Envoy; and the pilot-agent is responsible for maintaining the static configuration of Envoy and managing the life cycle of Envoy. The dynamic configuration part here will be expanded in the next section.


We can use the following command to enter Pod's istio-proxy container for further investigation. A little trick here is that we can enter the istio-proxy container in privileged mode as user 1337, so that we can use commands such as iptables that can only be run in privileged mode.

docker exec -ti -u 1337 --privileged <istio-proxy container id> bash

The 1337 user here is actually a user istio-proxy with the same name defined in the sidecar image. By default, the sidecar container uses this user. If we do not use the user option u in the above command, the privileged mode is actually granted to the root user, so after we enter the container, we need to switch to the root user to execute the privileged command.

After entering the container, we use the netstat command to view the monitoring. We will find that the port 15020 that reads the readiness probe is actually a pilot-agent process.

istio-proxy @ details-v1-68868454f5-94hzd:/$netstat -lnpt
Active Internet connections(only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0* LISTEN 19/envoy
tcp 0 0* LISTEN 19/envoy
tcp 0 0* LISTEN-
tcp6 0 0 :::15020 :::* LISTEN 1/pilot-agent

When we access the readiness probe interface inside istio-proxy, we will also get a 503 error.

Readiness check implementation

After understanding the sidecar agent and the pilot-agent process that manages the agent's life cycle, we can think a little bit about how the pilot-agent should implement the healthz/ready interface. Obviously, if this interface returns OK, it not only means that the pilot-agent is ready, but also must ensure that the agent is working.

In fact, the implementation of the pilot-agent readiness check interface is exactly the same. After receiving this request, this interface will call the server \ _info interface of the proxy Envoy. The IP used for the call is Localhost. This is very easy to understand, because this is the same Pod internal process communication. The port used is Envoy's proxyAdminPort, which is 15000.


With the above knowledge prepared, let's take a look at the log of the istio-proxy container. In fact, in the container log, an error has been repeatedly output, this error is divided into two parts, of which Envoy proxy is NOT ready This part is the information output by the pilot agent in response to the healthz/ready interface, that is, the Envoy proxy does not Ready; and the remaining config not received from Pilot(is Pilot running?):Cds updates:0 successful, 0 rejected; lds updates:0 successful, 0 rejected This part is when pilot-agent accesses server \ _info through proxyAdminPort The information brought back, it seems that Envoy has no way to get the configuration from Pilot.

Envoy proxy is NOT ready:config not received from Pilot(is Pilot running?):Cds updates:0 successful, 0 rejected; lds updates:0 successful, 0 rejected.

At this point, I suggest you go back and look at the illustrations in the previous section. In the previous section, we selectively ignored the dotted line from Pilot to Envoy, which is the dynamic configuration. The error reported here is actually that Envoy failed to obtain the dynamic configuration from the control plane Pilot.

Control plane and data plane

So far, this problem has been very clear. Before further analyzing the problem, I will talk about my understanding of the control plane and the data plane. The control plane data plane mode can be said to be everywhere. We give two extreme examples here.

The first example is the DHCP server. We all know that computers in the local area network can obtain IP addresses by configuring DHCP. In this example, the DHCP server is managed in a unified manner and dynamically assigns IP addresses to computers in the network. The DHCP server here is the control plane, and The computer that gets the IP is the data plane.

The second example is the script of the movie, and the performance of the movie. The script can be regarded as the control plane, and the performance of the movie, including every dialogue of the actor, the layout of the movie scene, etc., can be regarded as the data plane.

The reason why I think this is two extremes, because in the first example, the control plane affects only one attribute of the computer, while in the second example, the control plane is almost a complete abstraction and copy of the data plane, affecting All aspects of the data plane. The control plane of the Istio service grid is closer to the second example, as shown in the following figure:


Pilot, the control plane of Istio, uses the gRPC protocol to expose the interface istio-pilot.istio-system:15010. The reason why Envoy cannot obtain the dynamic configuration from Pilot is that the cluster DNS is not available in all Pods.

Simple reason

The reason for this problem is actually relatively simple. In the sidecar container istio-proxy, Envoy cannot access Pilot because the cluster DNS cannot resolve the service name istio-pilot.istio-system. In the container, the DNS server configured by resolv.conf is, which is the cluster's default kube-dns service address.

istio-proxy @ details-v1-68868454f5-94hzd:/$cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local localdomain

But the customer deleted and rebuilt the kube-dns service, and did not specify the service IP, which led to the fact that the cluster DNS address changed, which is why all sidecars cannot access Pilot.

# kubectl get svc -n kube-system
kube-dns ClusterIP <none> 53/UDP, 53/TCP 5d

Finally, by modifying the kube-dns service, specifying the IP address, the sidecar returns to normal.

# kubectl get pods
details-v1-68868454f5-94hzd 2/2 Running 0 6d
nginx-647d5bf6c5-gfvkm 2/2 Running 0 2d
nginx-647d5bf6c5-wvfpd 2/2 Running 0 2d
productpage-v1-5cb458d74f-28nlz 2/2 Running 0 6d
ratings-v1-76f4c9765f-gjjsc 2/2 Running 0 6d
reviews-v1-56f6855586-dplsf 2/2 Running 0 6d
reviews-v2-65c9df47f8-zdgbw 2/2 Running 0 6d
reviews-v3-6cf47594fd-cvrtf 2/2 Running 0 6d

in conclusion

This is actually a relatively simple problem, and the troubleshooting process is actually only a few minutes. But writing this article, I feel a bit like watching the twelve hours of Chang'an, just a few minutes of the investigation process, writing the principle behind the completeness, the cause and effect, but it took several hours. This is the first article in Istio's article, and I hope it will be helpful when you troubleshoot problems.

The third cloud native webinar invites you to participate

On May 28th, Alibaba Cloud technical experts will bring you "How to bring stable and efficient deployment capabilities for cloud native applications?" , Which will introduce the core deployment problems encountered by Alibaba economies in the process of large-scale application on the cloud, the corresponding solutions adopted, and how these solutions are precipitated as generalized capabilities and output open source, how to help users on Alibaba Cloud improve applications Deployment efficiency and stability.

Listeners can obtain the following benefits:

Understand the practical experience of large-scale cloud application in Ali economies, and how to solve the problem that the native K8s workload does not meet the needs of the scenario;
As an external user, how to experience and use the application deployment and release capabilities precipitated by Shangyun in the Shangali economy;
Demonstrate how Alibaba can achieve DaemonSet high-availability grayscale upgrade for large-scale K8s clusters(open source soon!)

Click the link to schedule a live broadcast:

" Alibaba Cloud native focus on micro service, Serverless, Containers, Service Mesh and other technical fields, focusing on cloud-native popular technology trends, cloud-native large-scale landing practices, and being the public account that understands cloud-native developers best