James Walker is the founder of Heron Web, a UK-based software development studio providing bespoke solutions for SMEs. He's experienced in delivering custom software using engineering workflows built around modern DevOps methodologies.

All post by James Walker - DevOps Expert

James Walker

Author title

Using generic CI/CD tools for your IaC automation? 🤖⚙️

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

Transform your IaC management to scale
securely, efficiently, and productively
into the future.

Blog

What Are Terraform Modules and How to Use Them: Tutorial

Terraform Functions, Expressions, Loops (Examples)

Managing Terraform State - Best Practices & Examples

Terraform For Loop - Expression Overview with Examples

What is Terraform? What is it Used For, How it Works

44 Ansible Best Practices to Follow [Tips & Tricks]

How to Stop and Remove Docker Containers

How to Create a CI/CD Pipeline with Docker [Tutorial]

CI/CD Pipeline : Everything You Need To Know

Kubernetes CI/CD Pipelines - 7 Best Practices and Tools

How to Deploy Infrastructure in CI/CD Using Terraform (Pipeline)

Terraform with Azure DevOps CI/CD Pipelines - Tutorial

Terragrunt Tutorial - Getting Started & Examples

Pulumi Pricing – Editions Overview 2024

73 Most Useful DevOps Tools to Use in 2024

Top 10 Terraform Enterprise Alternatives for 2024

Terraform Cloud/Enterprise Pricing - Tiers Overview 2024

16 Most Useful Infrastructure as Code (IaC) Tools for 2024

Terraform Cheat Sheet - 23 Terraform CLI Commands & Examples

ebook-global-banner

ebook global banner

Product overview

How it works

Integrations

Product

Solutions

Standardize and control infrastructure provisioning and configuration

cicd

CICD

cicd-2

Infrastructure Governance

Implement and automate secure, collaborative workflows

team

Team

team-2

Collaboration

Make it easy for developers to provision and configure with a single workflow

speed-2

Speed

speed-3

Developer Velocity

Scale your infrastructure safely and efficiently with an end-to-end workflow

productivity

Productivity

productivity-2

Infrastructure at Scale

Use cases

Financial

Technology

Industries

Request a meeting

Enjoy an action-packed day at AWS Summit London 2024, where the latest in cloud innovation comes to life.

ebook-pic

Practitioner’s Guide to Scaling IaC

Check out the Practitioner's Guide to Scaling IaC

Docs

Pricing

Resources

Learn more about Spacelift and infrastructure best practices

blog

blog-2

partners

Partners

partners-2

events

Events

events-2

Insights and quick guides to get you started

cheatsheets

Cheatsheets

cheatsheets-3

Ebooks & Cheatsheets

users

Users

users-2

Case studies

33-what-makes-spacelift-secure

What Makes Spacelift Secure

About Us

Contact Us

About

cheatsheets-2

Test

See more

Test webinar name

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Platform Engineering Guides and Tutorials | Spacelift Blog

Platform engineering

Flavius Dinu

In this article, you’ll learn what platform engineering is, the main responsibilities of a platform engineer, and the value platform engineering teams bring.  We’ll cover some best practices for implementing platform engineering and explain how they benefit your software workflows.
What we will cover:
<ol>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-platform-engineering#what-is-platform-engineering">What is platform engineering?</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-platform-engineering#the-principles-of-platform-engineering">The principles of platform engineering</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-platform-engineering#benefits-of-platform-engineering">Benefits of platform engineering</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-platform-engineering#what-do-platform-engineers-do">What do platform engineers do?</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-platform-engineering#how-to-build-a-modern-platform-engineering-team">How to build a modern platform engineering team?</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-platform-engineering#types-of-platform-engineering-tools">Platform engineering tools</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-platform-engineering#platform-engineering-best-practices">Platform engineering best practices</a></li>
<li aria-level="1"><a href="https://spacelift.io/blog/what-is-platform-engineering#common-misconceptions-about-platform-engineering-">Common misconceptions about platform engineering</a></li>
</ol>

Platform engineering is the process of designing and implementing toolchains that streamline software development and delivery by providing a unified, self-service platform for developers. The platform, called an “<a href="https://spacelift.io/blog/what-is-an-internal-developer-platform" target="_blank" rel="noopener">Internal Developer Platform</a>,” acts as a bridge between developers and the infrastructure, streamlining complex tasks that would be impractical for individual developers to handle independently.
<a href="https://spacelift.io/blog/self-service-infrastructure" target="_blank" rel="noopener">Self-service access</a> is a defining characteristic of effective platform engineering, enabling developers to utilize necessary capabilities without relying on other teams, like Operations and Infrastructure. This autonomy reduces bottlenecks, such as the need for approval to create new staging environments, allowing developers to instantly start isolated environments with a simple command, thereby maintaining productivity without deep infrastructure knowledge.

platform-engineering

platform engineering

<h3>Why use platform engineering?</h3>
Platform engineering will benefit your organization in many ways: it streamlines and standardizes the development and deployment process, enhances developer velocity, improves reliability and performance, and increases the overall scalability of your infrastructure and applications.

Platform engineering combines several technologies and methodologies to produce a holistic development experience:
<ul>
<li style="font-weight: 400;" aria-level="1">Automation and IaC — Infrastructure should be automated and reproducible. <a href="https://spacelift.io/blog/infrastructure-as-code-tools" target="_blank" rel="noopener">IaC tools</a> are used to define what the platform should look like and enable new instances to be created on-demand. Manual actions are kept to a minimum to eliminate friction points in the development flow.</li>
<li style="font-weight: 400;" aria-level="1">Focus on efficiency — The platform should be designed to solve the most common challenges encountered by developers. Focus on supporting the unique needs of your teams instead of trying to recreate functionality that already works well. Rolling your own CI/CD or source control system is unlikely to be beneficial, for example, but providing a mechanism that mirrors a snapshot of your production infrastructure into a fresh staging environment could save developers hours each week.</li>
<li style="font-weight: 400;" aria-level="1">Self-service access — Every part of the platform should be an asset that developers can freely utilize. You’re providing a toolbox of controls for developers to use as they see fit. Avoid prescribing specific usage patterns, as individual engineers may work in slightly different ways.</li>
<li style="font-weight: 400;" aria-level="1">Continual evolution — The platform should be continually developed using the same product-driven mindset you apply to customer-facing functionality. The “customer” happens to be internal developers, but it’s still vital that improvements are implemented promptly, so the platform effectively meets their needs. This ensures developers stay productive over a sustained period of time.</li>
</ul>
Each one of these principles revolves around simplifying the development experience. It’s the job of platform teams to listen to development teams, then provide the toolchains they require.

Organizations can be hesitant to invest in platform engineering. A common concern is whether the engineers working on the platform would be better utilized within the main product team. Here are four reasons why you should commit to platform engineering.
<h3>Accelerating development</h3>
Internal platforms accelerate development. Automated processes and self-service infrastructure help keep developers productive. They’re able to continually move forwards on the product features prioritized by the business.
Once a feature is ready, the development team can spin up a new test environment to autonomously verify the change. The platform can perform automated tests and then ship the feature to customers in production while developers start work on the next task. This reduces your time to market without compromising on quality.
<h3>Promoting focus and specialization</h3>
Developers should be able to focus on what they’re best at: development. Modern infrastructure, CI/CD, and distributed deployment systems are dedicated specialisms. Developers don’t need to be experts in these fields and may sometimes struggle to understand them.
Platform teams enable developers to stay productive by concentrating on building new software. The platform engineering team can be staffed with experts skilled in relevant topics such as IaC, CI/CD, and PaaS solutions. Advancements in both development and infrastructure will proceed more quickly as each individual will be specialized in their role.
<h3>Ensuring tools and processes continually develop</h3>
Development processes need to evolve as your product grows. Over time, your stack expands with additional technologies and new requirements. You might introduce a new storage system, require more comprehensive end-to-end tests, or have to comply with another regulatory standard.
Platform engineering ensures your toolchain develops in tandem. Without it, you have to make ad-hoc workflow adjustments, which can be poorly documented and difficult to maintain. Moreover, developers often lack the time to make the optimizations they require. This perpetuates the use of inefficient practices after they’ve been recognized as a bottleneck. Providing devs with access to a platform engineering team allows frustrations to be addressed without delaying the product’s release schedule.
<h3>Improving developer experience (DevEx)</h3>
Platform engineering improves DevEx through regulated self-service infrastructure. If your developers require infrastructure provisioning to test their applications, they can easily use the self-service mechanism the platform team provides, and without needing any other input from them, they will be able to test their application. This reduces the time spent on testing new features, resulting in faster deployment and, thus, faster time to market.

<ul>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-tools">Top Most Useful CI/CD Tools for 2023</a></li>
<li><a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Deploy your Infrastructure in CI/CD using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Deploy Your Infrastructure in CI/CD Using Terraform</a></li>
</ul>

Platform engineers handle tasks that help application developers work more efficiently, such as preparing CI/CD pipelines, setting up staging environments, and configuring Infrastructure as Code (IaC) to automate cloud resource provisioning.
Platform engineers create new tools and workflows for developers to use. They produce an integrated environment for building, testing, and iterating upon changes. 
<h3>Roles and responsibilities in platform engineering teams</h3>
Platform engineers have several responsibilities. They’ll discuss challenges with development teams and then act upon their insights to build internal platforms. This can include the following tasks:
<ul>
<li style="font-weight: 400;" aria-level="1">Configuring IaC tools to provision new infrastructure on-demand.</li>
<li style="font-weight: 400;" aria-level="1">Working with existing infrastructure and operations teams to “narrow the gap” between dev and prod.</li>
<li style="font-weight: 400;" aria-level="1">Implementing and maintaining <a href="https://spacelift.io/blog/ci-cd-pipeline" target="_blank" rel="noopener">CI/CD pipelines</a> that automate inefficient workflows.</li>
<li style="font-weight: 400;" aria-level="1">Creating bespoke internal tools to accommodate org-specific workflows, enforce security policies, and maintain compliance with regulatory standards.</li>
<li style="font-weight: 400;" aria-level="1">Building, maintaining and documenting custom APIs, CLIs, and web UIs that expose the platform’s functionality. This could be an API that exposes the number of errors in each environment or a CLI that pushes local code straight to a new sandbox.</li>
</ul>
<h4>Building Internal Developer Platforms (IDPs)</h4>
The platform team work often <a href="https://platformengineering.org/talks-library/how-we-built-an-internal-paas-at-adevinta" target="_blank" rel="nofollow noopener">culminates in</a> the form of internal Developer Platforms (IDPs). Developers get to deploy their applications onto infrastructure using a fully automated workflow that requires no specialist knowledge. This platform coordinates complex procedures such as creating cloud resources, deploying containerized microservices, setting up networking, and seeding any test data the developer requires.
<h4>Does a platform engineer code?</h4>
Platform engineers need to code, especially because all the things they do should be reusable. Even if they are using a declarative <a href="https://spacelift.io/blog/infrastructure-as-code-tools" target="_blank" rel="noopener">IaC tool</a> such as OpenTofu, they need to know how to use loops, conditionals, and modules to create reusable components

platform-engineering-team-2

platform engineering team

Your platform engineers should work closely with SREs, Cloud Architects, and Security Engineers to build a successful platform. Platform engineers should be experts in IaC (OpenTofu, Terraform, Pulumi, etc.) and Cloud technologies (AWS/Azure/GCP depending on the cloud provider the organization is using). 
Read more: <a href="https://spacelift.io/blog/how-to-implement-platform-engineering" target="_blank" rel="noopener">How to implement platform engineering?</a> and <a href="https://spacelift.io/blog/how-to-build-a-platform-engineering-team">How to build a platform engineering team?</a>
In addition, they should ensure the platform&#8217;s reliability, scalability, and performance while ensuring the necessary standardization and governance. This means they should have expertise in CI/CD, monitoring, observability, governance, and compliance tools.

Platform engineering encompasses various areas, including cloud services, containers, automation, monitoring, and more, so it offers many tools:
<table>
<tbody>
<tr>
<td>Type of tool</td>
<td>Examples</td>
</tr>
<tr>
<td>Cloud Services</td>
<td>AWS, Microsoft Azure, Google Cloud, Oracle Cloud, etc.</td>
</tr>
<tr>
<td>Version Control Systems</td>
<td>GitHub, GitLab, BitBucket</td>
</tr>
<tr>
<td>Infrastructure as Code (IaC)</td>
<td>Terraform, OpenTofu, Pulumi, AWS CloudFormation</td>
</tr>
<tr>
<td>IaC Management Platforms</td>
<td>Spacelift</td>
</tr>
<tr>
<td>Containerization</td>
<td>Docker, Podman</td>
</tr>
<tr>
<td>Orchestration </td>
<td>Kubernetes, Docker Swarm</td>
</tr>
<tr>
<td>Configuration Management</td>
<td>Ansible, Chef, Puppet</td>
</tr>
<tr>
<td>CI/CD</td>
<td>Jenkins, GitHub Actions, CircleCI, GitLab CI</td>
</tr>
<tr>
<td>Monitoring</td>
<td>Prometheus, Grafana, ELK Stack (Elastic Search, Logstash, Kibana)</td>
</tr>
<tr>
<td>Security</td>
<td>Open Policy Agent, AWS Secrets Manager, Vault</td>
</tr>
<tr>
<td>Programming Languages</td>
<td>Python, Golang, Bash, Powershell</td>
</tr>
</tbody>
</table>

You need to consider many aspects when you adopt platform engineering. Let’s take a look at some of the best practices:
<ol>
<li>Store your code in a VCS – This improves collaboration, tracks changes, and can facilitate reverting to a previous state when errors appear.</li>
<li>Adopt IaC – Automate and provision your infrastructure to reduce manual repetitive costs and minimize human errors.</li>
<li>Implement CI/CD – Reduce deployment times, automate builds/tests, and ensure reliability.</li>
<li>Increase Observability – Monitor the health and performance of your applications and infrastructure. Ensure logs are collected and easily accessible for troubleshooting.</li>
<li>Improve Security – Use the least privilege principle, manage secrets securely, and scan regularly for security vulnerabilities.</li>
<li>Build for scale – Design your infrastructure for scaling out rather than scaling up (this will ensure you add more instances to your workload, rather than more resources for one instance) and also design for failure by implementing high-availability and disaster recovery mechanisms.</li>
<li>Optimize usage – Ensure your resources are used efficiently and optimize costs.</li>
<li>Improve documentation and knowledge-sharing – Promote a culture of collaboration between teams and share the necessary resources to understand the architecture, configurations, and processes.</li>
</ol>

Automation, testing, monitoring, and IaC: these characteristics are already familiar <a href="https://spacelift.io/blog/who-is-devops-engineer" target="_blank" rel="noopener">to DevOps practitioners</a> and SREs, so what sets platform engineering apart?
<h3>Platform engineering vs DevOps</h3>
Platform engineering shouldn’t be viewed as an alternative to DevOps. It’s more accurate to treat it as an implementation of DevOps concepts and philosophies. The overarching aim of DevOps is to simultaneously improve software quality and throughput using new tools, processes, and collaboration frameworks. Platform engineering is an example of what this looks like in practice.
Providing developers with self-service access to infrastructure shortens the feedback loop and reduces complexity. This enables more focused work on forwards-facing tasks relevant to your business aims. Platform engineering accelerates the development cycle, which achieves the objectives expressed by DevOps.
In summary, platform engineering sits separately from DevOps but is usually part of a DevOps strategy. Seen from the other side, DevOps is more than just platform engineering, as a complete DevOps flow will extend beyond internal development tasks to deliver and manage code in production.
Learn more: <a href="https://spacelift.io/blog/platform-engineering-vs-devops" target="_blank" rel="noopener">Platform Engineering vs. DevOps – Key Differences</a>.
<h3>Platform engineering vs SRE</h3>
Platform engineering neighbors <a href="https://spacelift.io/blog/sre-vs-devops" target="_blank" rel="noopener">Site Reliability Engineering (SRE)</a>, too. SRE&#8217;s main purpose is to preserve the stability of your production environments. These teams use objective data-driven targets such as <a href="https://cloud.google.com/blog/products/devops-sre/sre-fundamentals-slis-slas-and-slos" target="_blank" rel="nofollow noopener">SLAs and SLOs</a> to identify when incidents materially affect your customers or your business. SRE then manages the incident resolution, analyzes what went wrong, and implements changes to prevent the problem from recurring.
Because platform engineering looks at internal systems, it doesn’t directly overlap with SRE. SRE produces infrastructure that’s optimized for highly reliable operations. Platform engineering creates assets that facilitate high-velocity development.
Information should be shared between the disciplines, though, as insights from one field are often valuable to the other. Difficulties encountered while setting up an internal workflow could reveal opportunities to simplify production infrastructure, for example. The idea is not to silo off the concerns but instead keep them <a href="https://www.cncf.io/blog/2022/11/02/is-platform-engineering-putting-an-end-to-devops-and-sre" target="_blank" rel="nofollow noopener">as complementary</a> philosophies that you can iterate upon.

Spacelift offers all the mechanisms required to build a successful platform:
<h3>1. Policies</h3>
With <a href="https://docs.spacelift.io/concepts/policy" target="_blank" rel="noopener">policies</a>, you can control what kind of resources people can create, what kind of parameters these resources can have, build custom policies for third-party tools you integrate into your workflow, control how many approvals you need for runs, and more:

platform-engineering-spacelift-policies

platform engineering spacelift policies

In the above example, we are enforcing a couple of mandatory tags for our resources (Name, env, and owner).
<h3>2. Stack dependencies</h3>
With <a href="https://docs.spacelift.io/concepts/stack/stack-dependencies" target="_blank" rel="noopener">stack dependencies</a>, you can build dependencies between your configurations, and even share outputs between them. You don’t have any constraint to the number of dependencies you want to create, and whenever a parent configuration finishes a run successfully, it will trigger runs to its children. As <a href="https://spacelift.io/integrations" target="_blank" rel="noopener">Spacelift supports multiple infrastructure tools</a>, you can build dependencies between them, so a parent stack can use OpenTofu for example, and a child stack can use Kubernetes.

platform-engineering-spacelift-stack-dependencies

platform engineering spacelift stack dependencies

<h3>3. Blueprints</h3>
<a href="https://docs.spacelift.io/concepts/blueprint" target="_blank" rel="noopener">Blueprints</a> enable you to configure every aspect of your stack, including governance and compliance. With blueprints, you can create self-service infrastructure, and by this your developer velocity will increase considerably.

spacelift-blueprints-platform-engineering

spacelift blueprints platform engineering

<h3>4. Cloud integrations</h3>
Static credentials are easily intercepted and can be used with malicious intent. Spacelift understands that, so it offers you the ability to <a href="https://docs.spacelift.io/integrations/cloud-providers" target="_blank" rel="noopener">integrate natively with AWS, Microsoft Azure, and Google Cloud</a> to generate dynamic credentials. Based on the roles you are using, these integrations can offer as few or as many permissions as you want:

platform-engineering-cloud-integrations

platform engineering cloud integrations

<h3>5. Spaces</h3>
Spaces help you implement RBAC, and give partial admin rights to your users.

spacelift-spaces-platform-engineering

spacelift spaces platform engineering

In the above example, if you give a user admin rights to <a href="https://docs.spacelift.io/concepts/spaces" target="_blank" rel="noopener">the resources space</a>, and no other rights, he will have all permissions to the resources and production space, but he won’t be able to even view resources in other spaces.
<h3>6. Contexts</h3>
<a href="https://docs.spacelift.io/concepts/configuration/context.html" target="_blank" rel="noopener">Contexts</a> are logical containers that can be shared between multiple configurations and contain environment variables, mounted files, and lifecycle hooks, making it easier to ensure reusability and idempotency.

spacelift-contexts-platform-engineering

spacelift contexts platform engineering

<h3>7. Drift detection and optional remediation</h3>
<a href="https://spacelift.io/blog/drift-detection" target="_blank" rel="noopener">Infrastructure drift</a> can be one of the worst problems you can have because if, for example, you fix something manually and then apply the code again at a later time, you will reintroduce the bug into your configuration.
<a href="https://spacelift.io/drift-detection" target="_blank" rel="noopener">Spacelift offers a drift detection mechanism</a> that runs a schedule that informs you about drift and can optionally remediate it:

spacelift-drift-platform-engineering

spacelift drift platform engineering

<h3>8. Resources view</h3>
With Spacelift, you can <a href="https://docs.spacelift.io/concepts/resources.html" target="_blank" rel="noopener">see all the resources</a> that have been deployed into your Spacelift account (based on the permissions you have), details about them and their health:

platform-engineering-resource-view

platform engineering resource view

There are other features that Spacelift offers that enable you to enhance your platform’s capabilities.

Platform engineering is the practice of building and maintaining internal toolchains that support software delivery workflows. A dedicated platform team implements tools and processes with self-service capabilities that enhance developer productivity and provide them with access to infrastructure.
Companies that pursue platform engineering can acquire a competitive edge in their field, since platform engineering team binds the goal of aligning development practices with business priorities. Development teams can work more autonomously without constantly waiting for approvals from infrastructure admins, which means more time spent adding new features to their products.
Platform engineering doesn’t mean you have to create everything from scratch. Take a look at <a href="https://spacelift.io/" target="_blank" rel="noopener">Spacelift</a> to start building the internal platform you need. Spacelift offers collaborative <a href="https://spacelift.io/ci-cd-for-infrastructure" target="_blank" rel="noopener">CI/CD for your infrastructure</a> and workflow automation, letting you unlock developer freedom while retaining precise security guardrails.

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

44 Best Books for DevOps Engineers to Read in 2024

What Is Infrastructure Automation & How Does It Work?

Platform engineering definition. Learn what platform engineers do, how to build your team, and how it benefits your software workflows.

143-platform-engineering

What Is Platform Engineering?

What is Platform Engineering? Role, Principles & Benefits

Want to level up your Kubernetes knowledge? Explore our tutorials and step-by-step guides written by Kubernetes experts. Learn Kubernetes best practices and solutions to the most difficult problems.

Kubernetes Guides and Tutorials | Spacelift Blog

Kubernetes

<a href="https://kubernetes.io" target="_blank" rel="nofollow noopener">Kubernetes</a> is an open-source container orchestrator that automates container deployment, scaling, and administration tasks. It provides helpful abstraction layers for running containerized workloads in production.
Kubernetes is a distributed system. It horizontally scales containers across multiple physical hosts termed Nodes. This produces fault-tolerant deployments which adapt to conditions such as Node resource pressure, instability, and elevated external traffic levels. If one Node suffers an outage, Kubernetes can reschedule your containers onto neighboring healthy Nodes.
The system provides auto-scaling functionality that <a href="https://kubernetes.io/blog/2016/07/autoscaling-in-kubernetes" target="_blank" rel="nofollow noopener">changes deployment replica counts automatically</a> as load grows. Managed Kubernetes services such as Google GKE and Amazon EKS support <a href="https://kubernetes.io/blog/2016/07/autoscaling-in-kubernetes" target="_blank" rel="nofollow noopener">dynamic Node creation too</a>. This helps you ride out traffic spikes by dynamically increasing your cluster’s capacity without requiring manual intervention.
All this functionality means the Kubernetes architecture is relatively complex. Several different components work together to create a functioning cluster. In this article, we&#8217;ll look at each major item in turn so you can better understand how Kubernetes works.

Kubernetes aims to lower the management overhead of running fleets of containers in production. It achieves this by pooling multiple compute Nodes into one logical platform termed a &#8220;<a href="https://spacelift.io/blog/kubernetes-cluster" target="_blank" rel="noopener">cluster</a>.&#8221; Deploying workloads to your Kubernetes cluster automatically starts containers on one or more of the Nodes.
<h3>1) Workloads</h3>
Kubernetes includes several abstraction layers that you use to define your application. Here are some of <a href="https://kubernetes.io/docs/concepts/workloads" target="_blank" rel="nofollow noopener">the most common workload objects</a> you&#8217;ll encounter:
<ul>
<li style="font-weight: 400;" aria-level="1">Pod &#8211; <a href="https://kubernetes.io/docs/concepts/workloads/pods" target="_blank" rel="nofollow noopener">Pods</a> are the fundamental compute unit in Kubernetes. A Pod is a group of one or more containers that share the same specification. All the containers in the Pod are scheduled to the same host.</li>
<li style="font-weight: 400;" aria-level="1">Deployment &#8211; A <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment" target="_blank" rel="nofollow noopener">Deployment</a> wraps the lower-level <a href="https://kubernetes.io/docs/concepts/workloads/controllers/replicaset" target="_blank" rel="nofollow noopener">ReplicaSet</a> object. It guarantees a certain number of replicas of a Pod will be running in your cluster. Deployments also provide declarative updates for Pods; you describe the desired state, and the Deployment will automatically add, replace, and remove Pods to achieve it.</li>
<li style="font-weight: 400;" aria-level="1">Service &#8211; <a href="https://kubernetes.io/docs/concepts/services-networking/service" target="_blank" rel="nofollow noopener">Services</a> expose Pods as a network service. You use services to permit access to Pods, either within your cluster via automatic service discovery, or externally through an <a href="https://spacelift.io/blog/kubernetes-ingress" target="_blank" rel="noopener">Ingress</a>. (Read more: <a href="https://spacelift.io/blog/kubernetes-service" target="_blank" rel="noopener">What is a Kubernetes Service?</a>)</li>
<li style="font-weight: 400;" aria-level="1">Job &#8211; A <a href="https://kubernetes.io/docs/concepts/workloads/controllers/job" target="_blank" rel="nofollow noopener">Job</a> starts one or more Pods and waits for them to successfully terminate. Kubernetes also provides <a href="https://spacelift.io/blog/kubernetes-cronjob" target="_blank" rel="noopener">CronJobs</a> to automatically create Jobs on a recurring schedule.</li>
</ul>
There are other kinds of workload too, such as <a href="https://spacelift.io/blog/kubernetes-daemonset" target="_blank" rel="noopener">DaemonSets</a> and <a href="https://spacelift.io/blog/kubernetes-statefulset" target="_blank" rel="noopener">StatefulSets</a>. DaemonSets replicate a Pod to every Node in your cluster, while StatefulSets provide persistent replica identities.
See <a href="https://spacelift.io/blog/statefulset-vs-deployment" target="_blank" rel="noopener">StatefulSet vs. Deployment</a>.
<h3>2) Control Plane</h3>
The Kubernetes Control Plane is your cluster&#8217;s management surface. It stores the cluster&#8217;s state, monitors for changes, and applies any actions that are required. Actions could be initiated by users via the API or in response to Node events, such as increased memory pressure.
The control plane gets created automatically when you deploy a cluster. It often runs on a dedicated Node, ensuring it&#8217;s isolated from your workloads for maximum performance and security. This is not mandatory, though – the machine that runs the control plane can also be used as a regular Node.
The control plane is a collective term for many different components. Together they provide everything needed to administer your cluster but not actually start and run containers. Those functions are provided by Node-level software, which we&#8217;ll see in the next section.
API Server (kube-apiserver)
The API server is the control plane component that exposes the Kubernetes REST API. You&#8217;re using this API whenever you run a command <a href="https://spacelift.io/blog/kubernetes-cheat-sheet" target="_blank" rel="noopener">with Kubectl</a>. You&#8217;ll lose management access to your cluster when the API server fails, but your workloads won’t necessarily be affected.
Controller Manager (kube-controller-manager)
Much of Kubernetes is built upon the <a href="https://kubernetes.io/docs/concepts/architecture/controller" target="_blank" rel="nofollow noopener">controller pattern</a>. A controller is a loop that continually monitors your cluster and performs actions when certain events occur. The Deployment controller launches new Pods when you create a Deployment object, for example.
The controller manager oversees all the controllers in your cluster. It starts their processes and ensures they&#8217;re operational the whole time that your cluster&#8217;s running.
Scheduler (kube-scheduler)
The scheduler is responsible for placing newly created Pods onto the Nodes in your cluster. The scheduling process works by first filtering out Nodes that can&#8217;t host the Pod, then scoring each eligible Node to identify the most suitable placement.
Nodes could be filtered out because of insufficient CPU or memory, inability to satisfy the Pod&#8217;s <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node" target="_blank" rel="nofollow noopener">affinity rules</a>, or other factors such as being <a href="https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node" target="_blank" rel="nofollow noopener">cordoned</a> for maintenance. The scoring process prioritizes Pods that satisfy non-mandatory conditions like <a href="https://kubernetes.io/blog/2017/03/advanced-scheduling-in-kubernetes" target="_blank" rel="nofollow noopener">preferred affinities</a>. If several Nodes appear to be equally suitable, Kubernetes will try to evenly distribute your Pods across them.
Etcd
<a href="https://etcd.io" target="_blank" rel="nofollow noopener">Etcd</a> is a distributed key-value storage system. Kubernetes stores your cluster&#8217;s state within etcd. It holds every API object, including config values and sensitive data you store in <a href="https://kubernetes.io/docs/concepts/configuration/configmap" target="_blank" rel="nofollow noopener">ConfigMaps</a> and <a href="https://spacelift.io/blog/kubernetes-secrets" target="_blank" rel="noopener">Secrets</a>.
Etcd is the most security-critical control plane component. Successfully compromising it would permit full access to your Kubernetes data. It&#8217;s important that etcd receives adequate hardware resources, too, as any starvation <a href="https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd" target="_blank" rel="nofollow noopener">can affect the performance</a> and stability of your entire cluster.
Cloud Controller Manager
The Cloud Controller Manager integrates Kubernetes with your cloud provider&#8217;s platform. It facilitates interactions between your cluster and its outside environment. This component is involved whenever Kubernetes objects change your cloud account, such as by <a href="https://spacelift.io/blog/kubernetes-load-balancer" target="_blank" rel="noopener">provisioning a load balancer</a>, adding a block storage volume, or creating a virtual machine to act as a Node.
<h3>3) Nodes</h3>
Nodes are the physical or virtual machines that host the Pods in your cluster. Although it&#8217;s possible to run a cluster with a single Node, production environments should include several so you can horizontally scale your resources and achieve high availability.
Nodes join the cluster using a token that&#8217;s issued by the control plane. Once the Node&#8217;s been admitted, the control plane will start scheduling new Pods to it. Each Node runs several software components to start containers and maintain communication with the control plane.
Learn <a href="https://spacelift.io/blog/kubectl-delete-pod" target="_blank" rel="noopener">how to delete Pods from a Kubernetes Node.</a>
Kubelet
<a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet" target="_blank" rel="nofollow noopener">Kubelet</a> is the Node-level process that acts as the control plane&#8217;s agent. It periodically checks in with the control plane to report the state of the Node’s workloads. The control plane can reach out to Kubelet when it wants to schedule a new Pod onto the Node.
Kubelet&#8217;s also responsible for running Pod containers. It will pull the images required by newly scheduled Pods, then start containers to produce the desired state. Once the containers are up, Kubelet monitors them to ensure they remain healthy.
Read more about <a href="https://spacelift.io/blog/kubernetes-imagepullpolicy" target="_blank" rel="noopener">Kubernetes Image Pull Policy</a>.
Kube-Proxy
The <a href="https://kubernetes.io/docs/concepts/overview/components/#kube-proxy" target="_blank" rel="nofollow noopener">kube-proxy</a> component facilitates network communications between the Nodes in your cluster. It automatically applies and maintains networking rules so that Pods exposed by Services are able to reach each other. If kube-proxy fails, Pods on that Node won&#8217;t be reachable over the network.
Container Runtime
Each Node requires a<a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes" target="_blank" rel="nofollow noopener"> CRI-compatible runtime</a> so it can start your containers. The <a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd" target="_blank" rel="nofollow noopener">containerd runtime</a> is the most popular option, but alternatives such as<a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cri-o" target="_blank" rel="nofollow noopener"> CRI-O</a> and <a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker" target="_blank" rel="nofollow noopener">Docker Engine</a> can be used instead. The runtime uses operating system features such as <a href="https://man7.org/linux/man-pages/man7/cgroups.7.html" target="_blank" rel="nofollow noopener">cgroups</a> to achieve containerization.

Below you can find an example of the Kubernetes architecture diagram:

kubernetes-architecture-diagram

kubernetes architecture diagram

<ul>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-best-practices" target="_blank" rel="noopener">15 Kubernetes Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/provision-kubernetes-cluster-terraform-spacelift" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Provision Kubernetes Cluster with Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Provision Kubernetes Cluster with Terraform</a></li>
</ul>

Kubernetes is highly extensible, so you can customize it to suit your environment. Although the control plane and Node-level software stacks are the most important, several other aspects of the architecture are significant too.
<h3>Networking</h3>
<a href="https://spacelift.io/blog/kubernetes-networking" target="_blank" rel="noopener">Kubernetes networking</a> uses a plugin-based approach. A <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins" target="_blank" rel="nofollow noopener">CNI-compatible networking plugin</a> must be installed to allow Pods to reach each other. Most popular Kubernetes distributions include a plugin for you, but you&#8217;ll have to manually install a solution such as <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins" target="_blank" rel="nofollow noopener">Calico</a> or <a href="https://github.com/flannel-io/flannel" target="_blank" rel="nofollow noopener">Flannel</a> when you <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm" target="_blank" rel="nofollow noopener">deploy a cluster</a> from scratch.
<h3>Storage</h3>
Storage provisioning can work very differently depending on your cloud provider. <a href="https://kubernetes.io/docs/concepts/storage/storage-classes" target="_blank" rel="nofollow noopener">Storage Classes</a> provide a consistent interface for accessing different types of storage in your workloads. You can add storage classes to save data to different platforms, such as a local volume on a Node’s filesystem or your cloud platform&#8217;s block storage volumes.
Kubernetes also has an external dependency on a <a href="https://docs.docker.com/registry/" target="_blank" rel="nofollow noopener">container registry</a>. You&#8217;ll need somewhere central to store your container images. You can run a registry inside your cluster, but this is not included with the default Kubernetes distribution.
<h3>Custom Functionality</h3>
You can add your own Kubernetes abstractions with <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources" target="_blank" rel="nofollow noopener">custom resource definitions</a> (CRDs). CRDs extend the API with support for your own data structures.
You can build the functionality around your <a href="https://spacelift.io/blog/kubernetes-crd" target="_blank" rel="noopener">CRDs</a> by writing <a href="https://spacelift.io/blog/kubernetes-operator" target="_blank" rel="noopener">controllers and operators</a>. These facilitate advanced automated workflows, such as automatically provisioning a database when you add a PostgresDatabaseConnection object to your cluster. They use the same fundamental concepts as built-in functionality: you author a control loop that watches for new objects and performs tasks when they occur.

Complexity and many moving parts create the potential for security problems. Hardening Kubernetes is a substantial topic: while the system purports to be production-ready, in practice, you need to take several manual actions to fully protect yourself.
Enabling <a href="https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data" target="_blank" rel="nofollow noopener">etcd encryption</a> is one essential step. Your cluster&#8217;s data isn&#8217;t encrypted by default so passwords and certificates in secrets are stored in plain text. It&#8217;s also important to secure the API server, avoid running other software on your Nodes, and ensure you use features like <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies" target="_blank" rel="nofollow noopener">networking policies</a> to fully isolate your workloads from each other. You can learn how to strengthen your cluster in <a href="https://spacelift.io/blog/kubernetes-security" target="_blank" rel="noopener">our Kubernetes security guide</a>.

Kubernetes requires the control plane components to remain available at all times. While the system is naturally resilient to Node failures, a problem in the control plane will affect the whole system. Its central role makes it a potential weak point in the architecture.
You can mitigate this risk by <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability" target="_blank" rel="nofollow noopener">running multiple replicas</a> of each component. This produces a highly available control plane that&#8217;s distributed across several Nodes, similarly to your Pods. It takes more time to set up but provides additional safety for production clusters. If you&#8217;re using a managed Kubernetes service, check with your provider to see if a high-availability control plane option is available.
If you need more help managing the complexities and compliance challenges of using Kubernetes check out <a href="https://spacelift.io" target="_blank" rel="noopener">Spacelift</a>. It brings with it a GitOps flow, so your Kubernetes Deployments are synced with your Kubernetes Stacks, and pull requests show you a preview of what they&#8217;re planning to change. It also has an extensive selection of policies, which lets you automate compliance checks and build complex multi-stack workflows. You can also use Spacelift to mix and match Terraform, Pulumi, AWS CloudFormation, and Kubernetes Stacks and have them talk to one another. Find out more about <a href="https://docs.spacelift.io/vendors/kubernetes/" target="_blank" rel="noopener" data-mf-id="1901">how Spacelift works with Kubernetes.</a>

Kubernetes is a complete platform for deploying, scaling, and managing distributed systems using containers. With powerful functionality comes to a complex and potentially confusing architecture. Kubernetes clusters are formed from several pieces, each assigned its own vital role.
In this article, you&#8217;ve learned how the fundamental components, such as Kubelet, kube-scheduler, etcd, and the API server combine to create an operational cluster. You can now continue your Kubernetes journey by deploying your own environment, <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm" target="_blank" rel="nofollow noopener">either from scratch with Kubeadm</a> or using a managed cloud service such as <a href="https://aws.amazon.com/eks" target="_blank" rel="nofollow noopener">Amazon EKS</a>, <a href="https://cloud.google.com/kubernetes-engine" target="_blank" rel="nofollow noopener">Google GKE</a>, or <a href="https://github.com/digitalocean/DOKS" target="_blank" rel="nofollow noopener">DigitalOcean DOKS</a>. You can get all the information you need to use Kubernetes in <a href="https://spacelift.io/blog/kubernetes-tutorial" target="_blank" rel="noopener">our beginners guide</a>, or try some of the <a href="https://spacelift.io/blog/kubectl-exec" target="_blank" rel="noopener">advanced topics</a> elsewhere on our blog.

<div class="PostBanner_text__05nV_">
<div class="WysiwigContent_wysiwigContent__duACp">
Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities s for infrastructure management.
</div>
</div>

Terraform vs. Kubernetes : Key Differences and Comparison

How to View Kubernetes Pod Logs Files With Kubectl

17 Kubernetes Best Practices Every Developer Should Know

Learn what are the different components of Kubernetes architecture that work together to create a functioning Kubernetes cluster.

103-kubernetes_architecture

What Is Kubernetes Architecture?

What Is Kubernetes Architecture? - Components Overview 

<a href="https://kubernetes.io" target="_blank" rel="nofollow noopener">Kubernetes</a> is the most popular orchestrator for deploying and scaling containerized applications in production. While Kubernetes makes it easy to start new workloads, this convenience comes at a cost: the system isn&#8217;t secure by default, so your clusters and users could be at risk.
Container attacks are <a href="https://www.paloaltonetworks.com/blog/prisma-cloud/6-common-kubernetes-attacks">on the rise</a> and insecure Kubernetes installations are a magnet for attackers. Fortunately you can protect your clusters by adhering to a few best practices. You can run containers in production without security issues, if you consciously implement protections around your deployments.

These steps help harden your environment to minimize your attack surface and defend against incoming threats. Implementing them all will give you the greatest protection by restricting network activity, encrypting data at rest, and preventing vulnerable workloads from reaching your cluster.
<ol>
<li><a href="https://spacelift.io/blog/kubernetes-security#1-use-rbac">Use RBAC</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-security#2-protect-the-control-plane">Protect the Control Plane</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-security#3-harden-your-nodes">Harden Your Nodes</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-security#4-add-network-security-policies">Add Network Security Policies</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-security#5-use-podlevel-security-features">Use Pod-Level Security Features</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-security#6-harden-your-workloads">Harden Your Workloads</a></li>
</ol>
You can also take a look at how <a href="https://spacelift.io" target="_blank" rel="noopener" data-mf-id="1896">Spacelift</a> helps you manage the complexities and compliance challenges of using Kubernetes. Anything that can be run via kubectl can be run within a Spacelift stack. Find out more about <a href="https://docs.spacelift.io/vendors/kubernetes/" target="_blank" rel="noopener" data-mf-id="1901">how Spacelift works with Kubernetes</a>, and get started on your journey by <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener" data-mf-id="1906">creating a free trial account</a>.

<a href="https://spacelift.io/blog/kubernetes-rbac" target="_blank" rel="noopener">Role-Based Access Control (RBAC)</a> is a built-in Kubernetes feature. It lets you control what individual users and service accounts can do by assigning them one or more roles. Each role allows a combination of actions, such as creating and listing Pods but not deleting them.
You should use RBAC to assign appropriate roles to each user and <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account" target="_blank" rel="nofollow noopener">service account</a> that interacts with your cluster. Developers may not need as many roles as operators and administrators, while CI/CD systems and Pod service accounts can be granted the bare minimum permissions needed to run their jobs.
RBAC protects your cluster if credentials are lost or stolen. An attacker who acquires a token for an account will be restricted to just the roles you&#8217;ve specifically assigned.
Roles must be as granular as possible to have the greatest security effect. Over-privileged roles, configured with too many permissions, are a risk because they grant attackers extra capabilities without providing any benefit to the legitimate user.

The Kubernetes control plane is responsible for managing all cluster-level operations. It exposes the API server, schedules new Pods onto Nodes, and stores the system&#8217;s current state. Breaching the control plane could give attackers control of your cluster.
Implementing these strategies will help lockdown the control plane and limit the effects of any compromise that occurs.
<ol>
<li style="font-weight: 400;" aria-level="1">Restrict access to etcd. Kubernetes uses <a href="https://etcd.io" target="_blank" rel="noopener">etcd</a> to store your cluster&#8217;s data. This includes credentials, certificates, and the values of ConfigMaps and Secrets you create. The central positioning of etcd makes it an attractive target for attackers. You should isolate access to it <a href="https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/#restrict-access-to-etcd" target="_blank" rel="nofollow noopener">behind a firewall</a> which only your Kubernetes components can reach through. You can achieve this by running etcd on a dedicated Node and using a network policy engine like <a href="https://www.tigera.io/project-calico" target="_blank" rel="nofollow noopener">Calico</a> to enforce traffic rules.</li>
<li style="font-weight: 400;" aria-level="1">Enable etcd encryption. Data within etcd is not encrypted by default. This option can be turned on by <a href="https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data" target="_blank" rel="nofollow noopener">specifying an encryption provider</a> when you start the Kubernetes API server. If you&#8217;re using a managed Kubernetes service in the cloud, check with your provider to see if encryption is already active and whether you can enable it. Encryption will help protect the credentials, secrets, and other sensitive information within your cluster if the control plane is successfully compromised.</li>
<li>Set up external API server authentication. The Kubernetes API server is usually configured with simple certificate-based authentication. This makes it challenging to configure and maintain large user cohorts. Integrating Kubernetes with your existing <a href="https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/#api-authentication" target="_blank" rel="nofollow noopener">OAuth or LDAP provider</a> tightens security by separating user management from the control plane itself. You can use your provider&#8217;s existing controls to block malicious authentication attempts and enforce login policies such as multi-factor authentication. <a href="https://kubernetes.io/docs/reference/access-authn-authz/kubelet-authn-authz/#kubelet-authentication" target="_blank" rel="nofollow noopener">Anonymous Kubelet authentication</a> should be disabled too. This will block requests to Kubelet from sources other than the Kubernetes API server instance that it’s connected to. Set the <code>--anonymous-auth=false</code> flag when you start Kubelet if you’re maintaining your own cluster.</li>
</ol>

<ul>
<li><a href="https://spacelift.io/blog/kubernetes-best-practices" target="_blank" rel="noopener">15 Kubernetes Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/provision-kubernetes-cluster-terraform-spacelift" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Provision Kubernetes Cluster with Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Provision Kubernetes Cluster with Terraform</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
</ul>

Don&#8217;t forget your Nodes when you&#8217;re securing your environment. Basic misconfiguration at the Node-level could compromise your entire cluster in a worst case scenario. Gaining access to a Node grants attackers <a href="https://labs.withsecure.com/publications/attacking-kubernetes-through-kubelet" target="_blank" rel="nofollow noopener">privileged access to the Kubernetes API server</a> through abuse of the Kubelet worker process that all Nodes run.
Securing Nodes used for Kubernetes is no different from protecting any other production server. You should monitor system logs regularly and keep the OS updated with new security patches, kernel revisions, and CPU microcode packages as they become available.
It&#8217;s best to dedicate your Nodes to Kubernetes. Avoid running other workloads directly on a Node, particularly network-exposed software, which could give attackers a foothold. Lockdown external access to sensitive protocols <a href="https://www.redhat.com/sysadmin/locking-down-sshd" target="_blank" rel="nofollow noopener">such as SSH</a>.

Kubernetes defaults to allowing all Pods to communicate freely with each other. Compromising one Pod could let bad actors inspect its surroundings and then move laterally into other workloads. Breaching the Pod that serves your website allows an attacker to direct traffic straight to your database Pods, for example.
<a href="https://spacelift.io/blog/kubernetes-network-policy" target="_blank" rel="noopener">Kubernetes Network Policies</a> defend against this risk by giving you precise control over the situations when Pods are allowed to communicate. You can specify at the Pod-level whether <a href="https://spacelift.io/blog/kubernetes-ingress" target="_blank" rel="noopener">Ingress</a> and Egress are allowed based on the other Pod&#8217;s identity, namespace, and IP address range. This lets you prevent access to sensitive services from containers that shouldn&#8217;t need to reach them.
Network policies are a Kubernetes resource type that you can apply to your cluster using <a href="https://spacelift.io/blog/yaml" target="_blank" rel="noopener">YAML files</a>. Here&#8217;s a simple example Policy that targets Pods with an app-component: database label:

This policy declares that incoming traffic to your database Pods can only originate from other Pods in the same <a href="https://spacelift.io/blog/kubernetes-namespaces" target="_blank" rel="noopener">namespace</a> with the app-component: api label. Nefarious requests made from your frontend web server&#8217;s Pod labeled as app-component: frontend will be rejected at the network-level.
It’s possible to set up a <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/#default-deny-all-ingress-traffic" target="_blank" rel="nofollow noopener">default namespace-level network policy</a> to guard against Pods being accidentally omitted from your rules. Using an empty <code>podSelector</code> field will apply the Policy to every Pod in the namespace:

This policy prevents both Ingress and Egress traffic to the namespace’s Pods, unless a more specific rule overrides it. You can block incoming or outbound traffic without affecting the other by changing the types listed under <code>spec.policyTypes</code>.

Besides using networking policies to prevent unwanted traffic flows, Kubernetes has a few other Pod-level capabilities that protect your cluster and applications from vulnerabilities in each other.
All Pods should be assigned a <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context" target="_blank" rel="nofollow noopener">security context</a> that defines their privileges. You can use this mechanism to require that containers run with restricted <a href="https://man7.org/linux/man-pages/man7/capabilities.7.html" target="_blank" rel="nofollow noopener">Linux capabilities</a>, avoid the use of HostPorts, and run with AppArmor, SELinux, and Seccomp enabled, among <a href="https://kubernetes.io/docs/concepts/security/pod-security-standards" target="_blank" rel="nofollow noopener">other controls</a>. The security context can also define the user and group to run containers as:

This security context makes containers in the Pod run as the user with UID 1000 while preventing privilege escalation. The <code>runAsNonRoot: true</code> declaration enforces that the container will run as a non-root user, while <code>runAsUser: 1000</code> requires it to run with UID 1000. You can omit the UID and use <code>runAsNonRoot</code> alone in situations where it doesn’t matter which user runs the container, provided it’s not root.
The security context shown above also stipulates that containers will run with a read-only root filesystem. Setting <code>securityContext.readOnlyRootFilesystem</code> to <code>true</code> prevents workloads inside the container from writing to the container filesystem. It limits what attackers can achieve if the container is compromised as they’ll be unable to persist malicious binaries or tamper with existing files.
Security contexts can also be set at the container-level, as <code>spec.containers[].securityContext</code>, overriding any constraints set on their Pod. This allows you to further harden individual containers, or relax the rules for administrative workloads.
<a href="https://kubernetes.io/blog/2022/08/25/pod-security-admission-stable" target="_blank" rel="nofollow noopener">Pod Security admission rules</a>, a replacement for the older <a href="https://kubernetes.io/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future" target="_blank" rel="nofollow noopener">PodSecurityPolicy system</a>, allow you to enforce minimum security standards for the Pods in your cluster. This mechanism will reject any Pods that violate the configured <a href="https://kubernetes.io/docs/concepts/security/pod-security-standards" target="_blank" rel="nofollow noopener">Pod security standard</a>, such as by omitting securityContext settings, binding HostPorts, or using HostPath volume mounts.
Policies can be enabled at the namespace or cluster-level. It&#8217;s good practice to use this system across your clusters. It guarantees that Pods with potentially dangerous weaknesses are prevented from running until you address their policy issues. If you need to run a Pod with elevated capabilities, you can use the <a href="https://kubernetes.io/docs/concepts/security/pod-security-standards/#privileged" target="_blank" rel="nofollow noopener">Privileged profile</a> to make your intentions explicit.

Kubernetes security isn&#8217;t all about your cluster. Applications need to be secure before they&#8217;re deployed, which means taking some basic steps to protect your container images.
First run <a href="https://snyk.io/learn/application-security/sast-vs-dast" target="_blank" rel="nofollow noopener">automated security scanning tools</a> to detect vulnerabilities in your code. Next, use a <a href="https://www.cisecurity.org/insights/blog/how-to-layer-secure-docker-containers-with-hardened-images" target="_blank" rel="nofollow noopener">hardened base image</a> and layer in your source. Finally, scan your built image with an analyzer like <a href="https://github.com/quay/clair" target="_blank" rel="nofollow noopener">Clair</a> or <a href="https://github.com/aquasecurity/trivy" target="_blank" rel="nofollow noopener">Trivy</a> to identify outdated OS packages and known CVEs. You should rebuild the image to incorporate mitigations if issues are present.
In security-critical situations, think about starting <a href="https://hub.docker.com/_/scratch" target="_blank" rel="nofollow noopener">from scratch</a> so you can have certainty about what&#8217;s present in your containers. This lets you assemble your entire filesystem without relying on an upstream image that could contain lurking threats.
It&#8217;s also vital to use the security mechanisms that Kubernetes provides. Sensitive data such as database passwords, API keys, and certificates shouldn&#8217;t reside in plain-text <a href="https://spacelift.io/blog/kubernetes-configmap" target="_blank" rel="noopener">ConfigMaps</a>, or be hardcoded into container filesystems, for example. Kubernetes <a href="https://spacelift.io/blog/kubernetes-secrets" target="_blank" rel="noopener">Secrets</a> let you store these values securely, independently of your Pods. They don’t encrypt values by default, though, so it’s vital to <a href="https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data" target="_blank" rel="nofollow noopener">enable etcd encryption</a> wherever they’re used. Secrets can also<a href="https://secrets-store-csi-driver.sigs.k8s.io/concepts.html#provider-for-the-secrets-store-csi-driver" target="_blank" rel="nofollow noopener"> integrate with external datastores</a> to save credentials outside your cluster.

Using these best practices helps you adhere to the <a href="https://spacelift.io/blog/cloud-native-security" target="_blank" rel="noopener">4C model of cloud-native security</a>. This simple mental picture sets out four security layers to protect yourself, each of them common words prefixed with &#8220;C&#8221;:
<ul>
<li style="font-weight: 400;" aria-level="1">Cloud &#8211; Vulnerabilities in your cloud infrastructure, such as not enabling 2FA for your Azure, AWS, or Google Cloud accounts, give attackers a route to all your resources. Protect yourself by regularly auditing your environment and choosing a reputable provider with a good compliance record.</li>
<li style="font-weight: 400;" aria-level="1">Cluster &#8211; Apply recommendations such as etcd encryption, RBAC controls, and Node isolation to protect your cluster from attack. Cluster-level compromise will expose all your applications and their data.</li>
<li style="font-weight: 400;" aria-level="1">Container &#8211; Individual containers can be strengthened by using hardened base images, scanning for vulnerabilities, and avoiding use of privileged capabilities. Malware inside a single container might be able to break out to access other resources and the host Node.</li>
<li style="font-weight: 400;" aria-level="1">Code &#8211; Code inside containers should be audited, scanned, and probed as it&#8217;s created to identify any weaknesses. Don&#8217;t underestimate attackers: while Kubernetes provides strong container isolation when configured correctly, weaknesses in your code could let intruders exploit zero-day vulnerabilities to escape the container and control your cluster.</li>
</ul>
Applying security protections across all four segments will give you the greatest protection. Only targeting a few areas could create weaknesses that let attackers move down the 4C pyramid and then laterally across your resources.
Read more about <a href="https://spacelift.io/blog/container-security" target="_blank" rel="noopener">Container security best practices and solutions</a>.

Kubernetes makes it easy to start and run containers but using plain images in a fresh cluster can be a security risk. Your workloads and clusters need to be hardened to make them safe for critical production environments. While it can be tempting to skip these steps, you&#8217;ll be vulnerable to exploitation if bad actors find your cluster.
The steps we&#8217;ve shared above will help you use Kubernetes securely by following the 4C model of Cloud, Cluster, Container, and Code. Attackers can manipulate weaknesses in any of these areas to cause a security incident.
Although the techniques listed here are good starting points, they&#8217;re not an exhaustive list of measures you can take. You can uncover additional improvement opportunities by using automated tools like <a href="https://github.com/kubescape/kubescape" target="_blank" rel="nofollow noopener">Kubescape</a>. This policy-based cluster scanner detects Kubernetes misconfigurations, security vulnerabilities, and container image risks in a single scan of your cluster.
There&#8217;s growing industry interest in strengthening Kubernetes deployments, including from U.S. government bodies. The <a href="https://media.defense.gov/2022/Aug/29/2003066362/-1/-1/0/CTR_KUBERNETES_HARDENING_GUIDANCE_1.2_20220829.PDF" target="_blank" rel="nofollow noopener">NSA/CISA</a> Kubernetes hardening guide and the Center for Internet Security&#8217;s <a href="https://www.cisecurity.org/benchmark/kubernetes" target="_blank" rel="nofollow noopener">Kubernetes security benchmark</a> are two references you can use to find threats in security-critical situations. We&#8217;ve also got more posts on the <a href="https://spacelift.io/blog/kubernetes" target="_blank" rel="noopener">Spacelift blog</a> about Kubernetes security, such as <a href="https://spacelift.io/blog/kubernetes-secrets" target="_blank" rel="noopener">how to use secrets</a> to store sensitive data in your cluster.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities s for infrastructure management.

Troubleshoot and Fix Kubernetes CrashLoopBackoff Status

See the Kubernetes Security best practices to harden your environment to minimize your attack surface and defend against incoming threats.

129-kubernetes-security-best-practices

Kubernetes Security Best Practices

Kubernetes Security : 6 Best Practices for 4C Security Model

<a title="Kubernetes" href="https://kubernetes.io" target="_blank" rel="nofollow noopener">Kubernetes</a> is the most popular orchestrator for deploying and scaling containerized systems. You can use Kubernetes to reliably build and distribute your applications in the cloud.
In this beginner&#8217;s guide, you&#8217;ll learn what Kubernetes can do and how to get started running your own containerized solutions.
<ol>
<li><a href="https://spacelift.io/blog/kubernetes-tutorial#what-is-kubernetes">What Is Kubernetes?</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-tutorial#kubernetes-features">Kubernetes Features</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-tutorial#how-kubernetes-works">How Kubernetes Works</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-tutorial#installation-and-setup">Installation and Setup</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-tutorial#kubernetes-basic-terms-and-concepts">Kubernetes Basic Terms and Concepts</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-tutorial#using-kubectl-to-interact-with-kubernetes">Using Kubectl to Interact With Kubernetes</a></li>
</ol>

Kubernetes is an open-source system that automates container deployment tasks. It was originally developed at Google but is now maintained as part of the <a href="https://www.cncf.io" target="_blank" rel="nofollow noopener"> Cloud Native Computing Foundation (CNCF).</a>
Kubernetes has risen to prominence because it solves many of the challenges around using containers in production. It makes it easy to launch limitless container replicas, distribute them across multiple physical hosts, and set up networking so users can reach your service.
Most developers begin their container journey with <a href="https://spacelift.io/blog/docker-tutorial" target="_blank" rel="noopener"> Docker</a>. While this is a comprehensive tool, it&#8217;s relatively low-level and relies on CLI commands that interact with one container at a time. Kubernetes provides much higher-level abstractions for defining applications and their infrastructure using declarative schemas you can collaborate on.
Check out <a href="https://spacelift.io/blog/docker-vs-kubernetes" target="_blank" rel="noopener">Docker vs. Kubernetes Comparison</a>.

Kubernetes has a comprehensive feature set that includes a full spectrum of capabilities for running containers and associated infrastructure:
<ul>
<li>Automated rollouts, scaling, and rollbacks &#8211; Kubernetes automatically creates the specified number of replicas, distributes them onto suitable hardware, and takes action to reschedule your containers if a node goes down. You can instantly scale the number of replicas on-demand or in response to changing conditions such as CPU usage.</li>
<li>Service discovery, load balancing, and network ingress &#8211; Kubernetes provides a complete networking solution that covers internal service discovery and public container exposure.</li>
<li>Stateless and stateful applications &#8211; While Kubernetes initially focused on stateless containers, it&#8217;s now also got <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset" target="_blank" rel="nofollow noopener"> built-in objects</a> to represent stateful apps too. You can run any kind of application in Kubernetes</li>
<li>Storage management &#8211; Persistent storage is abstracted by a consistent interface that works across providers, whether in the cloud, on a network share, or on a local filesystem.</li>
<li>Declarative state &#8211; Kubernetes uses object manifests in YAML files to define the state you want to create in your cluster. Applying a manifest instructs Kubernetes to automatically transition the cluster to the target state. You don&#8217;t have to manually script the changes you want to see.</li>
<li>Works across environments &#8211; Kubernetes can be used in the cloud, at the edge, or on your developer workstation. Many different distributions are available to match different use cases. Major cloud providers like AWS and Google Cloud offer managed Kubernetes services, while single-node distributions such as <a href="https://minikube.sigs.k8s.io/docs" target="_blank" rel="nofollow noopener">Minikube</a> and <a href="https://k3s.io" target="_blank" rel="nofollow noopener">K3s</a> are great for local use.</li>
<li>Highly extensible &#8211; Kubernetes packs in a lot of functionality, but you can add even more using extensions. You can create <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources" target="_blank" rel="nofollow noopener">custom object types</a>, controllers, and <a href="https://spacelift.io/blog/kubernetes-operator" target="_blank" rel="noopener">operators</a> to support your own workloads.</li>
</ul>
With so many features available, Kubernetes is ideal for any situation where you want to deploy containers with declarative configuration.
Check out our <a href="https://spacelift.io/blog/openshift-vs-kubernetes" target="_blank" rel="noopener">OpenShift vs. Kubernetes comparison</a>.

Kubernetes has a reputation for complexity because it has several moving parts. Understanding the basics of how they fit together will help you start out on your Kubernetes journey.
A Kubernetes environment is termed a cluster. It includes one or more nodes. A node is simply a machine that will run your containers. It could be physical hardware or a VM.
As well as nodes, the cluster also has a control plane. The control plane coordinates the entire cluster&#8217;s operations. It schedules new containers onto available nodes and provides the API server that you interact with. It&#8217;s possible to run a cluster with <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability" target="_blank" rel="nofollow noopener">multiple control plane instances</a> to create a highly available setup with greater resiliency.
Here are the most important Kubernetes components:
<ul>
<li>kube-apiserver &#8211; This is the part of the control plane that runs the API server. It&#8217;s the only way to interact with a running Kubernetes cluster. You can issue commands to the API server using the <a href="https://kubernetes.io/docs/tasks/tools" target="_blank" rel="nofollow noopener">Kubectl CLI</a> or an HTTP client.</li>
<li>kube-controller-manager &#8211; The controller manager starts and runs Kubernetes&#8217; built-in controllers. A <a href="https://kubernetes.io/docs/concepts/architecture/controller" target="_blank" rel="nofollow noopener">controller</a> is essentially an event loop that applies actions after changes in your cluster. They create, scale, and delete objects in response to events such as an API request or increased load.</li>
<li>kube-scheduler &#8211; The scheduler assigns new Pods (containers) onto the nodes in your cluster. It establishes which nodes can fulfill the Pod&#8217;s requirements, then selects the most optimal placement to maximize performance and reliability.</li>
<li>kubelet &#8211; Kubelet is a worker process that runs on each of your nodes. It maintains communication with the Kubernetes control plane to receive its instructions. Kubelet is responsible for pulling container images and starting containers in response to scheduling requests.</li>
<li>kube-proxy &#8211; Proxy is another component found on individual nodes. It configures the host&#8217;s networking system so traffic can reach the services in your cluster.</li>
</ul>
<a href="https://kubernetes.io/docs/tasks/tools" target="_blank" rel="nofollow noopener">Kubectl</a> is usually the final piece in a functioning Kubernetes environment. You&#8217;ll need this CLI to interact with your cluster and its objects. Once your cluster&#8217;s set up, you can also install the <a href="https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard" target="_blank" rel="nofollow noopener">official dashboard</a> or a third-party solution to control Kubernetes from a GUI.

<ul>
<li><a href="https://spacelift.io/blog/provision-kubernetes-cluster-terraform-spacelift" target="_blank" rel="noopener">How to Provision Kubernetes Cluster with Terraform</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-best-practices" target="_blank" rel="noopener">17 Kubernetes Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/terraform-vs-kubernetes" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Terraform vs. Kubernetes : Key Differences and Comparison&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Terraform vs. Kubernetes Comparison</a></li>
</ul>

There are many different ways to get started with Kubernetes because of the range of distributions on offer. Creating a cluster <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm" target="_blank" rel="nofollow noopener">using the official distribution</a> is relatively involved so most people use a packaged solution like <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm" target="_blank" rel="nofollow noopener">Minikube</a>, <a href="https://microk8s.io" target="_blank" rel="nofollow noopener">MicroK8s</a>, <a href="https://k3s.io" target="_blank" rel="nofollow noopener">K3s</a>, or <a href="https://kind.sigs.k8s.io" target="_blank" rel="nofollow noopener">Kind</a>.
Check out <a href="https://spacelift.io/blog/install-kubernetes" target="_blank" rel="noopener">how to install Kubernetes</a> using these four different methods.
We&#8217;ll use K3s for this tutorial. It&#8217;s an ultra-lightweight Kubernetes distribution that bundles all the Kubernetes components into a single binary. Unlike other options, there&#8217;s no dependencies to install or heavy VMs to run. It also includes the Kubectl CLI that you&#8217;ll use to issue <a href="https://spacelift.io/blog/kubernetes-cheat-sheet" target="_blank" rel="noopener">Kubernetes commands</a>.
Running the following command will install K3s on your machine:

It automatically downloads the latest available Kubernetes release and registers a system service for K3s.
After installation, run the following command to copy the auto-generated Kubectl config file into your .kube directory:

Now tell K3s to use this <a href="https://spacelift.io/blog/kubeconfig" target="_blank" rel="noopener">config file</a> by running the following command:

You can add this line to your <code>~/.profile</code> or <code>~/.bashrc</code> file to automatically apply the change after you login.
Next run this command:

In this beginners Kubernetes tutorial, you'll learn what Kubernetes can do and how to get started running your own containerized solutions.

James Walker

What is Platform Engineering? Role, Principles & Benefits

Get the newsletter

Recent posts

What Is Kubernetes Architecture? – Components Overview

Kubernetes Security : 6 Best Practices for 4C Security Model

Kubernetes Tutorial for Beginners: Basic Concepts

Share your data and download the guide