How to solve the security problem of Kubernetes? 4 ways to make a list of programs for you
Posted May 26, 2020 • 11 min read
Introduction:Security in Kubernetes is a multi-dimensional problem that must be solved from various perspectives to be perfect. This article will list the security list for readers from four aspects.
Translator of this article:"Open Source Village OSV"
Kubernetes, after faster adoption and more contributions from the community, is increasingly climbing to new heights. However, security is still one of the important aspects of Kubernetes, initially discouraging early adopters. Compared with virtual machines, many people doubt the security of containers and Kubernetes, and then canceled the containers. But people gradually began to believe that containers and Kubernetes are now as secure as physical and virtual machines.
Security in Kubernetes is a practice, not just a feature. Security is a multi-dimensional problem that must be solved from many different angles. Security in Kubernetes cannot be covered in an article, but the following list covers the main security areas that should be reviewed throughout the stack.
Security must be a first-class citizen of any organization's DevOps process(often called DevSecOps). With DevSecOps, security issues are embedded as part of the DevOps pipeline from the beginning. DevSecOps practices can automate most security issues and provide a series of security checks during development.
The DevSecOps open plan led by Nicolas M. Chaillan, Air Force Chief Software Officer, is a good example of the adoption and importance of DevSecOps in an organization. Detailed information has been posted on the Air Force's website for other organizations to use DevSecOps in depth. These DevSecOps practices can provide organizations with a way to embed security protections in their Kubernetes infrastructure. The security of Kubernetes can be defined as the following four aspects:infrastructure, Kubernetes, containers and applications, as shown in the figure below.
- Protect Infrastructure
Infrastructure-level security is usually the most basic task, but it is also the largest task. However, it is often ignored during development. When building an application, it is important to keep in mind the security of the infrastructure, because it affects how the application is architected.
Infrastructure security itself has many aspects:
Kubernetes deployment is mainly microservices, all microservices are talking to each other or communicating with external applications and services. It is important to limit network traffic only to the necessary traffic, and understand that microservices can be short-lived and can be moved between nodes in the cluster. You need to consider all aspects of network design to develop a secure network.
(1) Isolation of control traffic:Kubernetes control plane traffic must be isolated from data plane traffic. This is not only for security reasons, but also to prevent data traffic from affecting Kubernetes control plane traffic. Without isolation, traffic from the data plane may overshadow traffic from the control plane and cause temporary service interruption.
(2) Isolate storage traffic:Similarly, you need to isolate storage traffic from regular data and control the traffic so that the infrastructure storage service does not consume or shut down application networks, and vice versa.
(3) Network segmentation:Kubernetes hides the infrastructure from users. Developers should keep this fact and multi-tenancy in mind when designing networks. The underlying network infrastructure must also support layer 2 based
VLAN segmentation and layer 3 VXLAN-based segmentation to isolate traffic between various tenants or applications. Both segmentation techniques can be used as needed.
(4) Quality of service:In a shared network infrastructure, noisy "neighbors" are a big problem. Importantly, the basic network infrastructure can guarantee the service level specified for each Pod or tenant, while ensuring that the traffic of one Pod does not affect other Pods. Network virtualization technologies such as SR-IOV help to provide virtualization isolation on a shared infrastructure.
(5) Network strategy, firewall and ACL:We will discuss the application-level network access control in more detail later, but the network should have a lower level of access control at the hardware level and should better control the shared environment Traffic.
For any organization, storage is a critical part of security. Hackers usually look for confidential data stored in application storage, such as credit card information or personally identifiable information(PII). Developers using Kubernetes should consider the following form of storage-level security implementation.
(1) Self-encrypting drive:A basic form of storage security is a self-encrypting drive. With these drives, encryption is transferred to the disk itself, where the data is encrypted as it is written to the disk. This ensures that if someone can physically access the disk drive, they will not be able to access the data.
(2) Volume encryption:In a shared infrastructure, Kubernetes CSI manages the life cycle of volumes. This can isolate users from the underlying storage infrastructure. Volume encryption ensures that individual volumes are protected to prevent access from unwanted elements.
(3) Quality of service:In a shared storage infrastructure, a large number of I/O applications may affect the performance of other applications. It is important that the basic storage infrastructure should have
Tenants' ability to provide a guaranteed level of service. Similarly, SR-IOV helps to provide storage isolation at the PCI level, and each tenant uses a separate queue.
- Host and operating system(OS)
The next level of infrastructure is the physical or virtual host itself. Operators will want to protect the foundation by:
(1) Strengthen the operating system:The site reliability engineer(SRE) should follow the general security guidelines to protect the host operating system and strengthen the operating system to avoid any further changes. SRE should also apply
Firewall, port blocking and other standard best practice security measures. Regular security updates and patches must be applied as soon as they are available. Hackers and intruders often exploit known vulnerabilities.
(2) Enable kernel security:Kernel security modules such as SELinux and AppArmor define access control for applications, processes, and files on the system.
(3) Audit logging:Organizations using Kubernetes should implement audit logging, which can not only help monitor the system, but also help debug and find traces of security vulnerabilities.
(4) Rotation vouchers:User credentials must be rotated frequently, and strict safety guidelines must be followed to avoid being cracked or stolen.
(5) Lock the node:After configuring and setting the node in the Kubernetes cluster, the operating system should be deleted. Apart from patches and upgrades, there is no need to install or configure any new content. All nodes must be locked and can only be accessed by the super administrator.
(6) CIS conformance:CIS(Internet Security Center) provides conformance testing to ensure that all best practices have been implemented. Review your host settings and pass compliance tests to ensure compliance.
- Host-level access management
The weakest point to enter the Kubernetes cluster is the node itself. Because Kubernetes greatly isolates users from the underlying nodes, it is important to control access to the nodes.
(1) Strict access:The organization should be careful to limit root/admin access to the node to a very limited set of trusted users.
(2) Establish lock:Even for non-root users, ideally, direct login to developers should be restricted and restricted to access through the Kubernetes API server. In order to avoid any threat to the Kubernetes services running on the nodes, all nodes should be locked.
(3) Isolate Kubernetes nodes:Kubernetes nodes must be located on an isolated network and must not be directly exposed to the public network. If possible, it should not even be directly exposed to the company network. This is only possible when Kubernetes control and data traffic are isolated. Otherwise, both business flows flow through the same pipeline, and open access to the data plane means open access to the control plane. Ideally, the node should be configured to only accept connections from the master node on the designated port(through a network access control list).
(4) Master node:Master node access must be controlled by a network access control list, limited to the set of IP addresses required to manage the cluster.
- Protect Kubernetes
With the infrastructure locked, the next level of security is the Kubernetes installation itself. In a typical open source Kubernetes installation, because they are not all turned on by default, many of them require manual configuration.
- Security etcd
Etcd is a highly available key-value store that is used as a backup storage for all cluster data in Kubernetes. It contains all the status, secrets and information of Kubernetes, which means protecting etcd
Is often important.
(1) As mentioned earlier, the nodes in etcd should be locked with minimum access rights.
(2) Ideally, the drive containing etcd data should be encrypted.
(3) Access to etcd must be limited to the master.
(4) Ideally, etcd communication should pass TLS.
- Ensure access to the Kubernetes cluster
Kubernetes allows enterprises to use standard identity and access control solutions, but they require integration with the environment and are not provided by default. Access control can be broken down into the following components.
(1) Authentication:Users must be authenticated before they can access the Kubernetes API. Kubernetes provides various authentication modules-including client certificates, passwords, ordinary tokens, boot tokens, and JWT tokens(for service accounts). However, actual user management and authentication are not part of Kubernetes. For production environments, organizations will need external user management and authentication plug-ins or Kubernetes platforms that support these functions. Integration with LDAP, Active Directory or other identity provider solutions is very important.
(2) Authorization:After verifying the user's identity(that is, allowing him to connect to the Kubernetes cluster), the next step is authorization to determine the access rights to the requested resource. Kubernetes supports multiple authorization modules, such as attribute-based access control(ABAC), role-based access control(RBAC) and Webhooks. RBAC is one of the most popular authorized plugins because it allows fine-grained control of a single Kubernetes resource in a multi-tenant environment.
(3) Admission control:Admission control hooks allow organizations to intercept and control Kubernetes requests after users have been authenticated and authorized to access the requested resources. The best example of admission control is resource quotas, which allow organizations to control resource consumption. Access to the Kubernetes API server must also be protected through TLS.
- Security policy
Kubernetes provides some configurable policies that can be defined by the user. These should conform to corporate practice, but are not enabled by default.
A pod security strategy is an access control plug-in that ensures that the pod is only recognized when certain safety guidelines are followed. Strategies that can be defined include restricting the creation of privileged Pods, preventing containers from running as root or restricting the use of certain namespaces, networks, or volumes. The network strategy is implemented by the Container Network Interface(CNI) plug-in, which controls how Pod groups are allowed to communicate with each other and other network endpoints. Setting network policies is very important, because by default, pods are non-isolated(they accept traffic from any source).
Kubernetes provides quality of service(QoS) guarantees for computing resources(CPU and memory) to avoid problems with too many neighbors or lack of resources, but it does not provide QoS for I/O(storage and network). Diamanti and other hyper-converged platforms have added support for I/O QoS.
- Workload isolation and multi-tenancy
In a multi-tenant environment, each tenant or group of tenants must have a separate namespace to isolate workload and data from each other. CNI, CSI, and authentication plug-ins need to support these separations and boundaries so that they remain consistent throughout the stack.
- Protecting the container
Containers need to be protected during development and runtime. There are many great resources for protecting containers, including this article, but here are some key elements:
- Container image security
All running containers are based on image files, which can be downloaded from open libraries such as Docker Hub or can be passed from one team to another. It is important to know the source and internal content of the image. All of these initiatives should be part of the organization s DevOps process to automate and ensure safety.
(1) Image vulnerability scanning:You must use tools such as Aqua, Twistlock, Sysdig, and Clair to scan the container image being built to find known vulnerabilities. These tools parse the packages and dependencies in the image to find known vulnerabilities.
(2) Image signature:The organization should also implement a strict admission control policy, allowing only images signed by the company's notary. TUF and Notary are useful tools for signing container images and maintaining container content trust systems.
(3) Restrict privileges:In addition, organizations should avoid using root users in container images and prevent privilege escalation. Users inside the container must have the necessary minimum level of operating system privileges to achieve the container's goals.
- Container runtime
The container runtime is a program installed in the operating system. Today, most environments use Docker, and CIS benchmarks are available. Seccomp can be used to reduce the attack surface, and newer runtimes(such as CRI-O) have additional built-in security features.
- Run the container
Twistlock, Aqua and Sysdig and many other tools also provide continuous monitoring and threat prevention for runtime vulnerabilities by monitoring network and system calls. These tools also have the ability to intercept and block these unwanted calls or communications and enforce security policies.
- Protect the application
Finally, after protecting the infrastructure, Kubernetes, and containers, it is still important to protect the application itself.
- Application access
(1) TLS for Kubernetes ingress:The most common way to expose your application outside the cluster is to use an ingress controller, such as Envoy or NGINX. All external access to the entrance controller must pass TLS, and communication between the entrance controller and the application container should also use TLS, although in some cases it is not necessary to use TLS, depending on the network design and company security policy .
(2) All contents in encrypted transmission:Except for a few cases, the default behavior should be all contents in encrypted transmission. Even behind the company firewall, it is recommended to encrypt network traffic between containers. Many service meshes such as Istio and Linkerd provide mTLS options to automatically encrypt traffic in Kubernetes clusters.
(1) Network:Service meshes such as Istio, Linkerd and Consul provide many layer 7 network functions, which can limit and control the traffic between multiple tenants.
(2) Port:It is important to disclose only the ports necessary for communication with the application on the application/container.
- Application enhancement
Many DevOp practices should be built into the CI/CD pipeline to ensure application security and follow best practices. Some examples are:
(1) Regularly analyze the source code to ensure that it follows best practices to avoid vulnerabilities and threats. There are many tools available, such as Veracode and Synopsys.
(2) Most developers rely on third-party applications and libraries to build their applications and microservices. Scan code dependencies regularly for new vulnerabilities to ensure that they do not threaten the security of the application.
(3) Continuously test whether your application is affected by common attack practices, such as SQL injection, DDoS attacks, etc. There are various dynamic analysis tools available to assist you.
Security is always the top priority of the organization. But traditionally, security is an independent team in an organization that works in an island that is independent of the development process. Developers usually focus on the application, and the security team will intervene at the end of the development cycle. However, because the development team ignored the critical security strategy, the security team delayed the process, which may derail the deployment. This unhealthy interaction between the security and development teams not only led to fragile software development, but also caused many of the latest bugs and unexpected delays in production.
In the new era of containers and Kubernetes, it is very important to have a strong security practice automation. From the beginning, security should be included in the development cycle. With security entrenched in the DevOps process, DevSecOps is now in focus. The challenge is that many items outlined in the above list must be manually configured in multiple domains. The lack of one of these may put the entire application and company at risk.
Application security remains the responsibility of the developer. But other security functions related to infrastructure, platform and Kubernetes can be solved by modern hyper-convergence methods such as Diamanti platform. The Diamanti platform is a full-stack hardware and software platform for Kubernetes. It has built in many of the security features mentioned in this article, thereby alleviating the pain of implementing these features for the organization itself. This can help you easily set up the DevSecOps pipeline so that you can focus on application development.