In this article, we will delve into some best practices, tips, and tricks for securing your containers and Kubernetes (K8S) clusters with some configuration file examples.
We will cover:
Container security refers to the practice of securing containerized applications and the infrastructure that supports them. Due to the ephemeral nature of containers and the dynamic deployments and scaling, container security measures should be automated and baked into every stage of the software development life cycle (SDLC).
A Hardened deployment should cover all areas involved with the deployment from the infrastructure to the application. This includes:
- Build Pipeline
- Container images
- Container registries
- Container host machines
- Container runtimes (e.g., Docker or containerd)
- Container platforms and orchestrators (e.g., Kubernetes)
As with any technology, there are potential security risks associated with the use of containers, and securing your container runtime environment is essential for ensuring the security of your containerized applications. Out-of-the-box deployments are not secure by default, and hardening your deployment can be complex.
Taking a holistic approach and implementing the recommendations detailed below where possible will harden our deployment and give you a much better overall security posture.
- Use trusted base images
- Keep images up to date
- Reduce the attack surface
- Limit container privileges
- Implement access controls
- Scan images for vulnerabilities
- Implement network security
- Monitor container activity
- Train your team
Let’s look at each area in turn, and some suggested solutions!
You can also check out how Spacelift brings the benefits of CI/CD to infrastructure management. You can use Spacelift to deploy changes to Kubernetes with GitOps while benefiting from robust security policies and automated compliance checks. Spacelift also works with other infrastructure as code (IaC) providers, so you can use similar techniques to manage every component of your stack.
Avoid using images from unknown sources, as they may contain malicious code. When possible, use container images that are provided by the official repositories of the software vendor or organization. These images are usually tested, verified, and maintained by the vendor or organization, ensuring their quality and security.
Always check the image’s source code repository, its maintainers, and its community to ensure that it is reliable and trustworthy. Make sure signed images are used, which use cryptographic signatures to ensure that the image has not been tampered with or modified since it was signed by the author. Signed images can help protect against malicious attacks and supply chain attacks.
Keep your container images up to date with the latest security patches and software updates to prevent known vulnerabilities from being exploited. Establish a regular schedule for updating your container images. Depending on the frequency of updates released by the software vendor or organization, consider scheduling updates weekly, bi-weekly, or monthly.
You can also consider automating the updates. To help with this, always choose container image registries that support automatic updates and rollbacks, making it easier to manage image updates at scale.
If your cluster is running on Amazon EKS, you can use managed node groups to automatically manage and update the underlying EC2 instances used to run your Kubernetes workloads. This helps to ensure that your nodes are running the latest security updates.
Use minimalistic images with only the necessary software installed. This reduces the attack surface and minimizes the potential for security breaches. Using lightweight images will also increase your container start-up time and performance.
There are many images out there that aim to meet this goal, each with its own strengths and weaknesses, depending on your specific use case.
Popular choices include:
- Alpine Linux: A lightweight Linux distribution that weighs in at only 5 MB. It’s a popular choice for Docker images due to its small size.
- Ubuntu Minimal: This is a minimalistic version of Ubuntu that’s designed for use in containers. It includes only the essential packages needed to run Ubuntu and weighs in at around 50 MB.
- Scratch: This is a special Docker image that’s essentially an empty container. It doesn’t have a package manager or any system utilities, but it can be useful for building truly minimalistic images from scratch.
- BusyBox: This is a minimalistic Linux distribution that includes a variety of common Unix utilities, such as ls, cp, and grep. It’s only about 2 MB, making it a good choice for small Docker images.
- Tiny Core Linux: Another lightweight distribution popular for container images. It’s only 16 MB in size and includes a minimalistic graphical user interface.
Containers should be run with minimal privileges, such as non-root user mode, and limit their access to the host system and other containers. Running containers as root users can give the containers too much power and access to the host system.
User namespaces can be used to map container users to different users on the host system through UID mapping.
By using UID mapping with user namespaces, you can isolate container users from the host system while still allowing them to have access to the host’s resources. This can be useful for running containers with different privilege levels and reducing the risk of privilege escalation attacks.
- Create a user called jack with a UID of 9999.
sudo useradd -u 9999 jack
- Create a new user namespace for the container using the “newuidmap” and “newgidmap” utilities. This will map the container’s root user (UID 0) to the host user “jack” (UID 9999).
sudo unshare --user echo "0 $(id -u) 1" | sudo tee /proc/self/uid_map echo "0 $(id -g) 1" | sudo tee /proc/self/gid_map exit
- Start the container using the new user namespace using the
userns=hostoption, which tells Docker to use the host user namespace for the container.
docker run --rm -it --userns=host alpine sh
- Verify that the container’s root user is mapped to the host user “jack” using the
su jack id
Note: Popular tools like AppArmor or SELinux can also be used to restrict container access to sensitive system resources. Secure computing mode (or seccomp) is a Linux kernel feature that can be used to limit the system calls that containers can make.
Implement access controls and limit container access to only those who need it. Use authentication and authorization mechanisms to ensure that only authorized users can access the containers.
Kubernetes RBAC (Role-Based Access Control) is a security feature that allows administrators to control access to resources in a Kubernetes cluster based on a user’s role or job function. With RBAC, administrators can define roles, role bindings, and service accounts that grant specific permissions to users or groups of users.
The key components of Kubernetes RBAC:
- Roles: A role is a set of permissions that define what actions a user or group of users can perform on a specific set of resources. In this example, a role named
pod-readeris defined with permissions to
listpods in the
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: my-namespace name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "watch", "list"]
- Role Bindings: A role binding is a mapping between a role and one or more users, groups of users, or service accounts. In this example, a role binding is created that binds the
pod-readerrole to the user
my-namespacenamespace, and grants
alicepermissions to read pods in the
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-pods namespace: my-namespace subjects: - kind: User name: alice apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io
- Service Accounts: A service account is an identity that is used by a pod to access other Kubernetes resources. Service accounts can be granted specific permissions using role bindings. The example below shows how to create a service account called
my-service-accountand how to use it to create role bindings to grant permissions to access resources in a namespace.
apiVersion: v1 kind: ServiceAccount metadata: name: my-service-account namespace: my-namespace
apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: my-role-binding namespace: my-namespace roleRef: kind: Role name: my-role apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: my-service-account
If your cluster is running on AKS, you can use Azure Active Directory (AAD) integration to control access to AKS resources and enforce strong authentication and authorization policies. If you are using Amazon EKS, use IAM.
Use a container image scanning tool to detect vulnerabilities and ensure that your images are free from security risks before deployment. These tools can be integrated into the CI/CD pipeline to automatically scan images as they are built and deployed, ensuring that only secure images are used in production environments.
Popular tools include:
- Trivy: Trivy is an open-source image scanner that is designed to be lightweight and fast. The tool checks container images for known vulnerabilities in both operating system packages and application dependencies.
- Anchore Engine: Anchore Engine is an open-source tool that provides container image analysis and policy-based evaluations. The tool checks container images against customizable policies and can provide notifications when policy violations are detected.
- Aqua Security: Aqua Security provides a container security platform that includes an image scanning tool. The tool checks container images for known vulnerabilities, malware, and other security threats.
- Clair: Clair is an open-source container image scanner that is integrated with many popular container registries like Docker Hub, Quay, and Google Container Registry. The tool checks container images for known vulnerabilities and provides detailed reports on any issues found.
- Sysdig Secure: Sysdig Secure is a container security platform that includes an image scanning tool. The tool checks container images for vulnerabilities, malware, and configuration issues.
Implement network security controls to prevent unauthorized access to your containers. Use network segmentation to isolate containers and limit network access to only the necessary services.
Kubernetes network policies can be used to define rules for incoming and outgoing traffic to pods in your cluster by limiting traffic to specific ports and protocols and restricting traffic to specific pods based on labels. TLS encryption can be implemented for communication between pods, and use SSL/TLS certificates can be used to secure communication between the API server and other components in your cluster. Load balancers and other tools can be used to limit ingress traffic to your cluster.
You can also implement a service mesh like Istio or Linkerd to secure traffic between microservices in your cluster. Service meshes provide a centralized control plane for traffic management and can help secure communication between services using mTLS encryption.
Don’t forget to secure your network if using a cloud provider to host your cluster as a PaaS service, for example with Azure Kubernetes Service (AKS), you should use network security groups (NSGs) to control inbound and outbound traffic to and from AKS and restrict traffic to only the necessary ports and protocols. For Amazon Elastic Kubernetes Service (EKS), use VPC security groups.
Monitor logs and metrics to detect anomalies and respond to security incidents. Implement container monitoring to detect suspicious activity and potential security breaches. Security tools like cAdvisor or kube-state-metrics can be used to collect resource usage metrics.
Popular container monitoring tools like Prometheus and Grafana can be used to monitor the performance and activity of containers running in your cluster. These tools can help you detect issues with resource usage, application performance, and other important metrics.
Use network monitoring tools like Wireshark or tcpdump to capture and analyze network traffic between containers. This can help you detect potential security threats or performance issues.
If you are using AKS, you can use Azure Monitor to monitor resources and detect security issues and Azure Log Analytics to collect and analyze log data from AKS resources. For Amazon EKS, use Amazon CloudWatch to monitor and Amazon CloudTrail for logging.
Lastly, you should take steps to educate your team on container security best practices and train them on how to identify and respond to security threats.
Following container security best practices should be part of the training after an overview of containerization and related security risks. Hands-on training, regular security assessments, and continuous learning should help keep your team up to date with the latest security trends and techniques.
Effective container security involves a holistic approach that encompasses not only the containers themselves but also the underlying infrastructure, networks, and user access controls. By implementing robust container security practices, you can ensure that your applications are protected from potential threats and vulnerabilities.
The Most Flexible CI/CD Automation Tool
Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.