4 types of containers supported by k8s

Posted Jun 7, 20209 min read

As of now k8s1.18, k8s has supported four types of containers:standard container, sidecar container, init container, and ephemeral container. In this article, we will introduce in detail the characteristics of these four containers have been used scenarios.

Ephemeral container

Temporary containers differ from other containers in that they lack the guarantee of resources or execution, and they never restart automatically, so they are not suitable for building applications. Temporary containers are described using the same ContainerSpec section as regular containers, but many fields are incompatible and not allowed.

Temporary containers are created using a special ephemeralcontainers processor in the API, rather than directly added to the pod.spec section, so it is not possible to use kubectl edit to add a temporary container.

As with regular containers, once a temporary container is added to a pod, it cannot be changed or deleted.

Why do we need Ephemeral containers?

We know that the advantage of containers is that they provide all the necessary dependencies to run isolated processes by using immutable methods. By adding only the required dependencies to the image, the container can reduce the attack surface and provide faster startup and deployment. Using the "distroless" method to build container images(based on scratch), the container image has been taken to a new level by including only compiled application binaries. Unlike ordinary container images, they are not based on any kind of Linux distribution, so they do not contain any other binaries and tools that can be executed by kubectl exec for troubleshooting. This determines that the container helps provide a safe and reliable runtime environment, but it is also difficult to debug when problems occur.

In this case, the temporary container comes into play. They implement the function of attaching the debug container to the main process, which you can then use to debug any type of problem. The debugging container can be based on any image, so it can be customized according to your needs. You can build your own debug image, which contains special debug binaries or only tools like curl, OpenSSL and MongoDB client. However, you can also choose a Linux distribution(such as Ubuntu) or just run the Busybox image, both of which already contain many useful tools.

How to use temporary container?

The temporary container is an alpha feature, so it is disabled by default. You will need to activate the following function doors to use them:

  • Temporary container
  • PodShareProcessNamespace(beta version in v1.16, so it is enabled by default)

The examples in this section demonstrate how temporary containers appear in the API. Generally, you can use the kubectl plugin to troubleshoot and automate these steps.

Temporary containers are created using Pod's ephemeralcontainers sub-resource and can be displayed using the kubectl --raw command. First describe that the temporary container is added as an EphemeralContainers list:

{
    "apiVersion":"v1",
    "kind":"EphemeralContainers",
    "metadata":{
            "name":"example-pod"
    },
    "ephemeralContainers":[{
        "command":[
            "sh"
       ],
        "image":"busybox",
        "imagePullPolicy":"IfNotPresent",
        "name":"debugger",
        "stdin":true,
        "tty":true,
        "terminationMessagePolicy":"File"
    }]
}

Use the following command to update the running temporary container example-pod:

kubectl replace --raw /api/v1/namespaces/default/pods/example-pod/ephemeralcontainers -f ec.json

This will return a new list of temporary containers:

{
   "kind":"EphemeralContainers",
   "apiVersion":"v1",
   "metadata":{
      "name":"example-pod",
      "namespace":"default",
      "selfLink":"/api/v1/namespaces/default/pods/example-pod/ephemeralcontainers",
      "uid":"a14a6d9b-62f2-4119-9d8e-e2ed6bc3a47c",
      "resourceVersion":"15886",
      "creationTimestamp":"2019-08-29T06:41:42Z"
   },
   "ephemeralContainers":[
      {
         "name":"debugger",
         "image":"busybox",
         "command":[
            "sh"
        ],
         "resources":{

         },
         "terminationMessagePolicy":"File",
         "imagePullPolicy":"IfNotPresent",
         "stdin":true,
         "tty":true
      }
  ]
}

You can use the following command to view the status of the newly created temporary container:

kubectl describe pod example-pod
...
Ephemeral Containers:
  debugger:
    Container ID:docker://cf81908f149e7e9213d3c3644eda55c72efaff67652a2685c1146f0ce151e80f
    Image:busybox
    Image ID:docker-pullable://busybox@sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70
    Port:<none>
    Host Port:<none>
    Command:
      sh
    State:Running
      Started:Thu, 29 Aug 2019 06:42:21 +0000
    Ready:False
    Restart Count:0
    Environment:<none>
    Mounts:<none>
...

You can use the following command to connect to the new temporary container:

kubectl attach -it example-pod -c debugger

If process namespace sharing is enabled, you can view the processes in all containers of the pod. For example, after running the above attach operation, run the ps operation in the debugger container:

# Run this shell command in the "debugger" temporary container
ps auxww

After running the command, the output is similar to:

PID USER TIME COMMAND
    1 root 0:00 /pause
    6 root 0:00 nginx:master process nginx -g daemon off;
   11 101 0:00 nginx:worker process
   12 101 0:00 nginx:worker process
   13 101 0:00 nginx:worker process
   14 101 0:00 nginx:worker process
   15 101 0:00 nginx:worker process
   16 101 0:00 nginx:worker process
   17 101 0:00 nginx:worker process
   18 101 0:00 nginx:worker process
   19 root 0:00 /pause
   24 root 0:00 sh
   29 root 0:00 ps auxww

Init container

In Kubernetes, an init container is a container that starts and executes before other containers in the same Pod. It is designed to perform initialization logic for the main application hosted on the Pod. For example, create necessary user accounts, perform database migration, create database structure, etc.

Some considerations should be considered when creating an initcontainer:

  • They are always executed before other containers in the Pod. Therefore, they should not contain complex logic that takes a long time to complete. The startup script is usually small and concise. If you find that you want to add too much logic to the initialization container, you should consider moving some of it to the application container itself.
  • The initialization container is started and executed in sequence. Unless one initialization container is successfully executed, the next initialization container will not be executed. Therefore, if the startup task is very long, you can consider dividing it into multiple steps, each of which is handled by an initialization container so that you know which steps failed.
  • If any initialization of the container fails, the entire Pod will be restarted(unless you set restartPolicy to Never). Restarting the Pod means re-executing all containers again, including any initialization containers. Therefore, you may need to ensure that the startup logic allows multiple executions without causing duplication. For example, if the database migration has been completed, you should simply ignore the execution of the migration command again.
  • The initialization container is a good choice to delay application initialization until one or more dependencies are available. For example, if your application depends on an API that imposes an API request rate limit, you may have to wait for a while before receiving a response from that API. Implementing this logic in the application container can be complicated; it needs to be used in conjunction with health and readiness probes. A simpler method is to create an initialization container that will not exit successfully until the API is ready. The application container will only start after the initialization container has successfully completed its work.
  • Initialization containers cannot use health and ready probes like application containers. The reason is that they have to start and exit successfully, just as Jobs and CronJobs behave.
  • All containers on the same Pod share the same volume and network. You can use this feature to share data between the application and its initialization container.

As we just discussed, the init container always starts before other application containers on the same pod. As a result, the scheduler gives higher priority to the resources and restrictions of the init container. This behavior must be carefully considered because it may lead to undesirable consequences. For example, if you have an initialization container and an application container, and set the initialization container's resources and limits to be higher than the application container's resources and limits, then only if there is one available node that satisfies the initialization, the whole is scheduled Pod container requirements. In other words, even if there is an unused node in which to run the application container, if the initialization container has a higher resource prerequisite that the node can handle, the Pod will not be deployed to that node. Therefore, you should be as strict as possible when defining requests and restrictions for initializing containers. As a best practice, unless absolutely necessary, do not set these parameters higher than the values of the application container.

Standard container and Sidecar container

Before k8s1.18, there was no difference between these two containers from the perspective of k8s management. It's just a functional distinction.

Advantages of using sidecar container(modular)

  • Accelerate application development because containers can be reused between teams and even larger communities
  • Organize expert knowledge, because everyone collaborates on a containerized implementation that reflects best practices, rather than countless different containers of their own production with roughly the same function
  • Enable agile teams, because the container boundary is the natural boundary, and it is the contract of the team s responsibility
  • Provide separation of concerns and focus on specific functions to reduce spaghetti dependency and unpredictable components

Generally speaking, the Sidecar container is mainly reflected in the following 4 roles:

  • Agent. For example, Envoy in Istio.

Through this Sidecar model, the agent can intercept the traffic entering and leaving the main container so that Istio can extract a large number of signals about the traffic behavior as attributes. Istio can use these attributes to execute strategic decisions and send them to the monitoring system to provide information about the behavior of the entire grid.

The Sidecar proxy model also allows you to add Istio functionality to existing deployments without having to restructure or rewrite code.

  • Adapter. The adapter container standardizes the output. Consider the task of monitoring N different applications. Each application can be built using different ways to export monitoring data.(For example, JMX, StatsD, application-specific statistics), but each monitoring system expects the monitoring data it collects to have a consistent and unified data model. By using the composite container's adapter pattern, you can create heterogeneous monitoring data from different systems into a unified representation by creating a Pod that groups application containers with adapters that know how to convert. Also, since these Pods share the namespace and file system, the coordination of these two containers is very straightforward.

  • Enhanced main container function. Sidecar containers extend and enhance the "main" container, they can use existing containers and make them better. For example, consider a container running an Nginx web server. Add another container that synchronizes the file system with the git repository, share the file system between these containers, and you have built Git Push-to-deploy. But you have done this in a modular way, where the git synchronizer can be built by different teams and can be reused on many different web servers(Apache, Python, Tomcat, etc.). Because of this modularity, you only need to write and test the git synchronizer once, and you can reuse it in many applications. And, if someone writes it, you don t even need to do it.
  • Realize auxiliary functions. This kind of scene usually appears in DevOps. For example, deploy the log collection component in the way of Sidecar to achieve the purpose of collecting logs, or deploy a Sidecar component to monitor configuration changes from the configuration center and update the local configuration in real time.

Life cycle

All problems with Sidecar containers are related to the relevance of the container life cycle. Since there is no difference between regular containers in Pod, it is not possible to control which container starts or terminates first, but running the Sidecar container first is usually a requirement for the application container to run correctly.

Starting from version 1.18, the built-in Sidecar function of K8S will ensure that the Sidecar container is up and running before the start of normal business processes. That is, by changing the startup life cycle of the pod, the sidecar container is started after the init container is completed, and the business is started when the sidecar container is ready. The container guarantees the sequence from the starting process.

By changing the container.lifecycle.type in the Pod specification to mark the container as Sidecar type:Sidecar, the default is Standard, as follows:

apiVersion:v1
kind:Pod
metadata:
  name:bookings-v1-b54bc7c9c-v42f6
  labels:
    app:demoapp
spec:
  containers:
  -name:bookings
    image:banzaicloud/allspark:0.1.1
    ...
  -name:istio-proxy
    image:docker.io/istio/proxyv2:1.4.3
    lifecycle:
      type:Sidecar
    ...

to sum up

This article briefly introduces four types of containers:standard container, sidecar container, init container, and ephemeral container. With the increasing popularity of kubernetes, we need to fully grasp the principles and usage methods of several types of containers in order to better serve our business.

In addition, the Sidecar container will become a new way of software delivery in the future. According to Dapr and other teams, different teams provide their own functional containers, and then selectively inject Sidecar into the main business container to achieve decoupling.