In this article, we will delve into some best practices, tips, and tricks for securing your containers and Kubernetes (K8S) clusters with some configuration file examples.
We will cover:
Container security refers to the practice of securing containerized applications and the infrastructure that supports them. Due to the ephemeral nature of containers and the dynamic deployments and scaling, container security measures should be automated and baked into every stage of the software development life cycle (SDLC).
As containerization plays a significant role in today’s software development, container security becomes more and more crucial for organizations.
Containers share the same OS kernel, so if a container is compromised, there’s a risk that an attacker could exploit potential vulnerabilities to affect the host system and other containers.
Because containers are created from images, and these images can contain vulnerabilities, an attacker exploiting these can gain access to other parts of the system. Containers often require secrets to operate, and improper management of these secrets can expose them to unauthorized access.
Ensuring security is paramount to protect sensitive data, maintain operational integrity, and safeguard against potential breaches that could have severe consequences.
Container security architecture is a comprehensive framework designed to protect containers throughout their lifecycle. This framework encompasses several key components, each focusing on different aspects of container security. Let’s look into the primary component:
- Container images security
- Registry security
- Orchestrators security
- Container engine security
Container images security
To implement container image security, you need:
- Vulnerability scanning – scanning images as part of your build process makes a lot of sense for identifying known vulnerabilities, and failing your builds when these vulnerabilities are encountered is a must for keeping your containers secure
- Minimal base image – containers are lightweight, and they are lightweight for security reasons too. By using a minimal base image that includes only the necessary components for your applications to run, you reduce the potential attack surface by eliminating unnecessary packages, thus eliminating potential vulnerabilities
- Image signing – you should only use images from a known source
Registry security
Container registries are responsible for storing container images after they are built. Securing the registry involves:
- Implementing least privilege access – strict access control to manage who can push and pull images will solve many potential security issues
- Regular registry scan – scanning images shouldn’t be done only when you are building the image but also throughout the lifecycle of that image
- Protect against interception – you should encrypt your images during transport to and from the registry
Orchestrators security
Nowadays, containers are used mostly with an orchestrator. The most popular orchestrator is Kubernetes, and there are many things you can secure for your orchestrator:
- RBAC – ensure users get the least privileged access
- Network policies – implement network policies to control the traffic flow
- Secrets management – manage secrets securely, rotate them, and have controlled access to them
Container engine security
The container engine is responsible for running containers on the host machine, and securing it involves:
- Implementing runtime security – monitor and protect containers in real-time (anomaly detection, blocking suspicious events)
- Host security – ensure users get the least privileged access to the host that runs the containers
- Resource limitations – set resource limits to prevent distributed denial of service (DDoS) attacks
The most common security challenges, alongside their risks and some potential solutions, are:
| Common security challenge | Risks | Solution |
| Vulnerabilities in container images | Attackers exploiting these can gain access to other parts of the system | Implement security vulnerability scanning as part of your build process and your registry |
| Insecure container registries | Unauthorized access | Ensure the least privileged access and encrypt your images during transport |
| Misconfigurations | Create vulnerabilities that can be exploited to get unauthorized access or service unavailability | Ensure the build and deployment process of your images and containers are fairly tested & regular audits |
| Secrets management | Data breaches | Use dedicated secrets management tools and encrypt secrets in transit and at rest |
| Network issues | Without appropriate segmentation, malicious traffic can increase an attack’s surface | Implement network policies and use service meshes |
| The ephemeral nature of containers | Makes it hard to detect issues in real-time | Adopt runtime security solutions |
The most important container security best practices:
- Use trusted base images
- Keep images up to date
- Reduce the attack surface
- Limit container privileges
- Implement access controls
- Scan images for vulnerabilities
- Implement network security
- Monitor container activity
- Compliance and auditing
- Train your team
Let’s look at each area in turn, and some suggested solutions!
You can also check out how Spacelift brings the benefits of CI/CD to infrastructure management. You can use Spacelift to deploy changes to Kubernetes with GitOps while benefiting from robust security policies and automated compliance checks. Spacelift also works with other infrastructure as code (IaC) providers, so you can use similar techniques to manage every component of your stack.
Avoid using images from unknown sources, as they may contain malicious code. When possible, use container images that are provided by the official repositories of the software vendor or organization. These images are usually tested, verified, and maintained by the vendor or organization, ensuring their quality and security.
Always check the image’s source code repository, its maintainers, and its community to ensure that it is reliable and trustworthy. Make sure signed images are used, which use cryptographic signatures to ensure that the image has not been tampered with or modified since it was signed by the author. Signed images can help protect against malicious attacks and supply chain attacks.
Keep your container images up to date with the latest security patches and software updates to prevent known vulnerabilities from being exploited. Establish a regular schedule for updating your container images. Depending on the frequency of updates released by the software vendor or organization, consider scheduling updates weekly, bi-weekly, or monthly.
You can also consider automating the updates. To help with this, always choose container image registries that support automatic updates and rollbacks, making it easier to manage image updates at scale.
If your cluster is running on Amazon EKS, you can use managed node groups to automatically manage and update the underlying EC2 instances used to run your Kubernetes workloads. This helps to ensure that your nodes are running the latest security updates.
Use minimalistic images with only the necessary software installed. This reduces the attack surface and minimizes the potential for security breaches. Using lightweight images will also increase your container start-up time and performance.
There are many images out there that aim to meet this goal, each with its own strengths and weaknesses, depending on your specific use case.
Popular choices include:
- Alpine Linux: A lightweight Linux distribution that weighs in at only 5 MB. It’s a popular choice for Docker images due to its small size.
- Ubuntu Minimal: This is a minimalistic version of Ubuntu that’s designed for use in containers. It includes only the essential packages needed to run Ubuntu and weighs in at around 50 MB.
- Scratch: This is a special Docker image that’s essentially an empty container. It doesn’t have a package manager or any system utilities, but it can be useful for building truly minimalistic images from scratch.
- BusyBox: This is a minimalistic Linux distribution that includes a variety of common Unix utilities, such as ls, cp, and grep. It’s only about 2 MB, making it a good choice for small Docker images.
- Tiny Core Linux: Another lightweight distribution popular for container images. It’s only 16 MB in size and includes a minimalistic graphical user interface.
Containers should be run with minimal privileges, such as non-root user mode, and limit their access to the host system and other containers. Running containers as root users can give the containers too much power and access to the host system.
User namespaces can be used to map container users to different users on the host system through UID mapping.
By using UID mapping with user namespaces, you can isolate container users from the host system while still allowing them to have access to the host’s resources. This can be useful for running containers with different privilege levels and reducing the risk of privilege escalation attacks.
- Create a user called jack with a UID of 9999.
sudo useradd -u 9999 jack- Create a new user namespace for the container using the “newuidmap” and “newgidmap” utilities. This will map the container’s root user (UID 0) to the host user “jack” (UID 9999).
sudo unshare --user
echo "0 $(id -u) 1" | sudo tee /proc/self/uid_map
echo "0 $(id -g) 1" | sudo tee /proc/self/gid_map
exit- Start the container using the new user namespace using the
userns=hostoption, which tells Docker to use the host user namespace for the container.
docker run --rm -it --userns=host alpine sh- Verify that the container’s root user is mapped to the host user “jack” using the
idcommand:
su jack
idNote: Popular tools like AppArmor or SELinux can also be used to restrict container access to sensitive system resources. Secure computing mode (or seccomp) is a Linux kernel feature that can be used to limit the system calls that containers can make.
Implement access controls and limit container access to only those who need it. Use authentication and authorization mechanisms to ensure that only authorized users can access the containers.
Kubernetes RBAC (Role-Based Access Control) is a security feature that allows administrators to control access to resources in a Kubernetes cluster based on a user’s role or job function. With RBAC, administrators can define roles, role bindings, and service accounts that grant specific permissions to users or groups of users.
The key components of Kubernetes RBAC:
- Roles: A role is a set of permissions that define what actions a user or group of users can perform on a specific set of resources. In this example, a role named
pod-readeris defined with permissions toget,watch, andlistpods in themy-namespacenamespace.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: my-namespace
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]- Role Bindings: A role binding is a mapping between a role and one or more users, groups of users, or service accounts. In this example, a role binding is created that binds the
pod-readerrole to the useralicein themy-namespacenamespace, and grantsalicepermissions to read pods in themy-namespacenamespace.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-pods
namespace: my-namespace
subjects:
- kind: User
name: alice
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io- Service Accounts: A service account is an identity that is used by a pod to access other Kubernetes resources. Service accounts can be granted specific permissions using role bindings. The example below shows how to create a service account called
my-service-accountand how to use it to create role bindings to grant permissions to access resources in a namespace.
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-service-account
namespace: my-namespaceapiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: my-role-binding
namespace: my-namespace
roleRef:
kind: Role
name: my-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: my-service-accountIf your cluster is running on AKS, you can use Azure Active Directory (AAD) integration to control access to AKS resources and enforce strong authentication and authorization policies. If you are using Amazon EKS, use IAM.
Use a container image scanning tool to detect vulnerabilities and ensure that your images are free from security risks before deployment. These tools can be integrated into the CI/CD pipeline to automatically scan images as they are built and deployed, ensuring that only secure images are used in production environments.
Popular tools include:
- Astra Security: Astra Security’s vulnerability scanner provides automated and manual security scans for cloud-based containers. The tool conducts 9300+ tests to detect vulnerabilities in container configurations and images during its runtime.
- Trivy: Trivy is an open-source image scanner that is designed to be lightweight and fast. The tool checks container images for known vulnerabilities in both operating system packages and application dependencies.
- Anchore Engine: Anchore Engine is an open-source tool that provides container image analysis and policy-based evaluations. The tool checks container images against customizable policies and can provide notifications when policy violations are detected.
- Aqua Security: Aqua Security provides a container security platform that includes an image scanning tool. The tool checks container images for known vulnerabilities, malware, and other security threats.
- Clair: Clair is an open-source container image scanner that is integrated with many popular container registries like Docker Hub, Quay, and Google Container Registry. The tool checks container images for known vulnerabilities and provides detailed reports on any issues found.
- Sysdig Secure: Sysdig Secure is a container security platform that includes an image scanning tool. The tool checks container images for vulnerabilities, malware, and configuration issues.
Implement network security controls to prevent unauthorized access to your containers. Use network segmentation to isolate containers and limit network access to only the necessary services.
Kubernetes network policies can be used to define rules for incoming and outgoing traffic to pods in your cluster by limiting traffic to specific ports and protocols and restricting traffic to specific pods based on labels. TLS encryption can be implemented for communication between pods, and use SSL/TLS certificates can be used to secure communication between the API server and other components in your cluster. Load balancers and other tools can be used to limit ingress traffic to your cluster.
You can also implement a service mesh like Istio or Linkerd to secure traffic between microservices in your cluster. Service meshes provide a centralized control plane for traffic management and can help secure communication between services using mTLS encryption.
Don’t forget to secure your network if using a cloud provider to host your cluster as a PaaS service, for example with Azure Kubernetes Service (AKS), you should use network security groups (NSGs) to control inbound and outbound traffic to and from AKS and restrict traffic to only the necessary ports and protocols. For Amazon Elastic Kubernetes Service (EKS), use VPC security groups.
Monitor logs and metrics to detect anomalies and respond to security incidents. Implement container monitoring to detect suspicious activity and potential security breaches. Security tools like cAdvisor or kube-state-metrics can be used to collect resource usage metrics.
Popular container monitoring tools like Prometheus and Grafana can be used to monitor the performance and activity of containers running in your cluster. These tools can help you detect issues with resource usage, application performance, and other important metrics.
Use network monitoring tools like Wireshark or tcpdump to capture and analyze network traffic between containers. This can help you detect potential security threats or performance issues.
If you are using AKS, you can use Azure Monitor to monitor resources and detect security issues and Azure Log Analytics to collect and analyze log data from AKS resources. For Amazon EKS, use Amazon CloudWatch to monitor and Amazon CloudTrail for logging.
Addressing compliance and auditing in a containerized environment can be hard to do. These processes should incorporate both technological solutions and process-oriented strategies. Due to their ephemeral nature and the dynamic environments in which they operate, containers have unique challenges in ensuring compliance with regulatory standards and conducting effective audits.
You should implement automated compliance checks using automated tools (such as Chef InSpec or Aqua Security) against the CIS (Center for Internet Security) benchmark, HIPAA (Health Insurance Portability and Accountability Act), and GDPR (General Data Protection Regulation) requirements. These automated tools can provide real-time alerts and reports on compliance status.
Lastly, you should take steps to educate your team on container security best practices and train them on how to identify and respond to security threats.
Following container security best practices should be part of the training after an overview of containerization and related security risks. Hands-on training, regular security assessments, and continuous learning should help keep your team up to date with the latest security trends and techniques.
Effective container security involves a holistic approach that encompasses not only the containers themselves but also the underlying infrastructure, networks, and user access controls. By implementing robust container security practices, you can ensure that your applications are protected from potential threats and vulnerabilities.
The Most Flexible CI/CD Automation Tool
Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.
