Want to level up your Kubernetes knowledge? Explore our tutorials and step-by-step guides written by Kubernetes experts. Learn Kubernetes best practices and solutions to the most difficult problems.

Kubernetes Guides and Tutorials | Spacelift Blog

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Kubernetes

In this post, I will run through some best practices to use when using Kubernetes (K8s).
As the most popular container orchestration system, K8s is the de-facto standard for the modern cloud engineer to get to grips with. K8s is a notoriously complex system to use and maintain, so getting a good grasp of what you should and should not be doing, and knowing what is possible will get your deployment off to a solid start.
These recommendations cover common issues within 3 broad categories, application development, governance, and cluster configuration.
<h3 id="b48f" class="lq lr ip bn ls lt lu lv lw lx ly lz ma kn mb mc md kr me mf mg kv mh mi mj mk gt">Kubernetes Best Practices:</h3>
<ol class="">
<li id="d51a" class="ml mm ip kg b kh mn kk mo kn mp kr mq kv mr kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#use-namespaces" rel="">Use namespaces</a></li>
<li id="10ef" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#use-readiness-and-liveness-probes">Use readiness and liveness probes</a></li>
<li id="39ca" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#use-autoscaling">Use resource requests and limits</a></li>
<li id="b033" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#deploy-your-pods-as-part-of-a-deployment-daemonset-replicaset-or-statefulset-across-nodes">Deploy your Pods as part of a Deployment, DaemonSet, ReplicaSet or StatefulSet across nodes.</a></li>
<li id="257b" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#use-multiple-nodes">Use multiple nodes</a></li>
<li id="3469" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#use-rolebased-access-control-rbac">Use Role-based access control (RBAC)</a></li>
<li id="3bde" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#host-your-kubernetes-cluster-externally-use-a-cloud-service">Host your Kubernetes cluster externally (use a cloud service)</a></li>
<li id="8020" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#upgrade-your-kubernetes-version">Upgrade your Kubernetes version</a></li>
<li id="aceb" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#monitor-your-cluster-resources-and-audit-policy-logs">Monitor your cluster resources and audit policy logs</a></li>
<li id="1c77" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#use-a-version-control-system">Use a version control system</a></li>
<li id="5eae" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#use-a-gitbased-workflow-gitops">Use a Git-based workflow (GitOps)</a></li>
<li id="00e8" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#reduce-the-size-of-your-containers">Reduce the size of your containers</a></li>
<li id="5459" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#organize-your-objects-with-labels">Organize your objects with labels</a></li>
<li id="8a71" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#use-network-policies">Use network policies</a></li>
<li id="eecf" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph=""><a href="https://spacelift.io/blog/kubernetes-best-practices#use-a-firewall">Use a firewall</a></li>
</ol>

Namespaces in K8s are important to utilize in order to organize your objects, create logical partitions within your cluster, and for security purposes. By default, there are 3 namespaces in a K8s cluster, <code class="gs nl nm nn no b">default</code>, <code class="gs nl nm nn no b">kube-public</code> and <code class="gs nl nm nn no b">kube-system</code>.
RBAC can be used to control access to particular namespaces in order to limit the access of a group to control the blast-radius of any mistakes that might occur, for example, a group of developers may only have access to a namespace called <code class="gs nl nm nn no b">dev</code> , and have no access to the <code class="gs nl nm nn no b">production</code> namespace. The ability to limit different teams to different namespaces can be valuable to avoid duplicated work or resource conflict.
LimitRange objects can also be configured against namespaces to define the standard size for a container deployed in the namespace. ResourceQuotas can also be used to limit the total resource consumption of all containers inside a Namespace. Network policies can be used against namespaces to limit traffic between pods.

Readiness and Liveness probes are essentially types of health checks. These are another very important concept to utilize in K8s.
Readiness probes ensure that requests to a pod are only directed to it when the pod is ready to serve requests. If it is not ready, then requests are directed elsewhere. It is important to define the readiness probe for each container, as there are no default values set for these in K8s. For example, if a pod takes 20 seconds to start and the readiness probe was missing, then any traffic directed to that pod during the startup time would cause a failure. Readiness probes should be independent and not take into account any dependencies on other services, such as a backend database or caching service.
Liveness probes test if the application is running in order to mark it as healthy. For example, a particular path of a web app could be tested to ensure it is responding. If not, the pod will not be marked as healthy and the probe failure will cause the <code class="gs nl nm nn no b">kubelet</code>to launch a new pod, which will then be tested again. This type of probe is used as a recovery mechanism in case the process becomes unresponsive.

Where it is appropriate, autoscaling can be employed to dynamically adjust the number of pods (horizontal pod autoscaler), the amount of resources consumed by the pods (vertical autoscaler), or the number of nodes in the cluster (cluster autoscaler), depending on the demand for the resources.
The horizontal pod autoscaler can also scale a replication controller, replica set, or stateful set based on CPU demand.
Using scaling also brings some challenges, such as not storing persistent data in the container&#8217;s local filesystem, as this would prevent horizontal autoscaling. Instead, a PersistentVolume could be used.
The cluster autoscaler is useful when highly variable workloads exist on the cluster that may require different amounts of resources at different times based on demand. Removing unused nodes automatically is also a great way to save money!

Resource requests and limits (minimum and maximum amount of resources that can be used in a container) should be set to avoid a container starting without the required resources assigned, or the cluster running out of available resources.
Without limits, pods can utilize more resources than required, causing the total available resources to be reduced which may cause a problem with other applications on the cluster. Nodes may crash, and new pods may not be able to be placed corrected by the scheduler.
Without requests, if the application cannot be assigned enough resources, it may fail when attempting to start or perform erratically.
Resource requests and limits define the amount of CPU and Memory available in millicores and mebibytes. Note that if your process goes over the memory limit, the process is terminated, so it may not always be appropriate to set this in all cases. If your container goes over the CPU limit, the process is throttled.

A single pod should never be run individually. To improve fault tolerance, instead, they should always be part of a Deployment, DaemonSet, ReplicaSet or StatefulSet. The pods can then be deployed across nodes using anti-affinity rules in your deployments to avoid all pods being run on a single node, which may cause downtime if it was to go down.

Running K8s on a single node is not a good idea if you want to build in fault tolerance. Multiple nodes should be employed in your cluster so workloads can be spread between them.

Using RBAC in your K8s cluster is essential to properly secure your system. Users, Groups, and Service accounts can be assigned permissions to perform permitted actions on a particular namespace (a Role), or to the entire cluster (ClusterRole). Each role can have multiple permissions. To tie the defined roles to the users, groups, or service accounts, RoleBinding or ClusterRoleBinding objects are used.
RBAC roles should be set up to grant using the principle of least privilege, i.e. only permissions that are required are granted. For example, the admins group may have access to all resources, and your operators group may be able to deploy, but not be able to read secrets.

Hosting a K8s cluster on your own hardware can be a complex undertaking. Cloud services offer K8s clusters as platform as a service (PaaS), such as AKS (Azure Kubernetes Service) on Azure, or EKS (Amazon Elastic Kubernetes Service) on Amazon Web Services. Taking advantage of this means the underlying infrastructure will be managed by your cloud provider, and tasks around scaling your cluster, such as adding and removing nodes can be much more easily achieved, leaving your engineers to the management of what is running on the K8s cluster itself.

As well as introducing new features, new K8s versions also include vulnerability and security fixes, which make it important to run an up-to-date version of K8s on your cluster. Support for older versions will likely not be as good as newer ones.
Migrating to a new version should be treated with caution however as certain features can be depreciated, as well as new ones added. Also, the apps running on your cluster should be checked that they are compatible with the newer targeted version before upgrading.

Monitoring the components in the K8s control plane is important to keep resource consumption under control. The control plane is the core of K8s, these components keep the system running and so are vital to correct K8s operations. Kubernetes API, kubelet, etcd, controller-manager, kube-proxy and kube-dns make up the control plane.
Control plane components can output metrics in a format that can be used by Prometheus, the most common K8s monitoring tool.
Automated monitoring tools should be used rather than manually managing alerts.
Audit logging in K8s can be turned on whilst starting the kube-apiserver to enable deeper investigation using the tools of your choice. The audit.log will detail all requests made to the K8s API and should be inspected regularly for any issues that might be a problem on the cluster. The Kubernetes cluster default policies are defined in the audit-policy.yaml file and can be amended as required.
A log aggregation tool such as Azure Monitor can be used to send logs to a log analytics workspace from AKS for future interrogation using Kusto queries. On AWS Cloudwatch can be used. Third-party tools also provide deeper monitoring functionality such as Dynatrace and Datadog.
Finally, a defined retention period should be in place for the logs, around 30–45 days is common.

K8s configuration files should be controlled in a version control system (VCS). This enables a raft of benefits, including increased security, enabling an audit trail of changes, and will increase the stability of the cluster. Approval gates should be put in place for any changes made so the team can peer-review the changes before they are committed to the main branch.

Successful deployments of K8s require thought on the workflow processes used by your team. Using a git-based workflow enables automation through the use of CI/CD (Continuous Integration / Continuous Delivery) pipelines, which will increase application deployment efficiency and speed. CI/CD will also provide an audit trail of deployments. Git should be the single source of truth for all automation and will enable unified management of the K8s cluster. You can also consider using a dedicated infrastructure delivery platform such as <a class="au lp" href="https://spacelift.io/?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bkubernetes_best_practices%7D" target="_blank" rel="noopener">Spacelift</a>, which recently introduced Kubernetes support.

Smaller image sizes will help speed up your builds and deployments and reduce the amount of resources the containers consumed on your K8s cluster. Uneccesery packages should be removed where possible, and small OS distribution images such as Alpine should be favored. Smaller images can be pulled faster than larger images, and consume less storage space.
Following this approach will also provide security benefits as there will be fewer potential vectors of attack for malicious actors.

K8s labels are key-value pairs that are attached to objects in order to organize your cluster resources. Labels should be meaningful metadata that provide a mechanism to track how different components in the K8s system interact.
Recommended labels for pods in the <a class="au lp" href="https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/" target="_blank" rel="noopener ugc nofollow">official K8s documentation</a> include <code class="gs nl nm nn no b">name, instance, version, component, part-of,</code> and <code class="gs nl nm nn no b">managed-by</code>.
Labels can also be used in a similar way to using tags in a cloud environment on resources in order to track things related to the business, such as object <code class="gs nl nm nn no b">ownership</code>, and the <code class="gs nl nm nn no b">environment</code>an object should belong to.
Also recommended is to use labels to detail security requirements, including <code class="gs nl nm nn no b">confidentiality</code> and <code class="gs nl nm nn no b">compliance</code>.

Network policies should be employed to restrict traffic between objects in the K8s cluster. By default, all containers can talk to each other in the network, something that presents a security risk if malicious actors gain access to a container, allowing them to traverse objects in the cluster. Network policies can control traffic at the IP and port level, similar to the concept of security groups in cloud platforms to restrict access to resources. Typically, all traffic should be denied by default, then allow rules should be put in place to allow required traffic.

As well as using network policies to restrict internal traffic on your K8s cluster, you should also put a firewall in front of your K8s cluster in order to restrict requests to the API server from the outside world. IP addresses should be whitelisted and open ports restricted.

Following the best practices listed in this article when designing, running, and maintaining your Kubernetes cluster will put you on the path to success on your modern application journey!

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities s for infrastructure management.

How to Provision an EKS Kubernetes Cluster with Terraform

Infrastructure as Code : Best Practices, Benefits & Examples

Initial Availability of Kubernetes Integration in Spacelift

Follow the best practices for Kubernetes listed in this article for painless and efficient maintenance of your K8s cluster.

73-kubernetes_best_practices_1

Kubernetes Best Practices

15 Kubernetes Best Practices Every Developer Should Know

Provision a Kubernetes Cluster with Spacelift and Terraform

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Product

By now, you have probably heard about the initial availability of Kubernetes integration in Spacelift. You can read more about the announcement <a href="https://spacelift.io/blog/spacelift-kubernetes-integration-beta" target="_blank" rel="noopener">here</a> and review the documentation.
Spacelift, like many of our customers, utilizes Kubernetes in a variety of ways. Before the integration, we operated Kubernetes or Public worker pools, testing and hosting the Spacelift platform. Now that the Kubernetes integration is initially available, we thought it would be good to provide a simple way for you to provision a Kubernetes cluster on AWS utilizing Terraform with Spacelift.
This blog will guide you through the process of deploying a Terraform based deployment of an EKS cluster with Spacelift. Below is an example of the cloned Terraform repository.

image2-18

devrel-k8s-worker-cluster

Start by cloning the repository found on <a href="https://github.com/hashicorp/learn-terraform-provision-eks-cluster" target="_blank" rel="nofollow noopener">GitHub</a> and commit to your respective VCS provider. We made a few changes, but that was only to help with the naming and possible confusion in our dev environment. Otherwise, utilizing the repository as-is will work.

In Spacelift, click the Add Stack button to create a Stack.
<h3>Integrate VCS</h3>
<ul>
<li style="font-weight: 400;" aria-level="1">Select the repository you created in the initial step, as seen in the picture.</li>
</ul>

image11-5

Integrate VCS

<h3>Configure Backend</h3>
For this example, I will keep the default selections.

image7-7

Configure Backend

<h3>Define Behavior</h3>
If you need to modify the configuration, select the arrow next to Show Advanced Options. You can find out more about these options by reading the <a href="https://docs.spacelift.io/concepts/stack/creating-a-stack#define-behavior" target="_blank" rel="nofollow noopener">documentation</a>.

image4-15

Define Behavior

<h3>Name Stack</h3>
Name the stack and provide any labels or descriptions.

image8-7

Name Stack

To provision the EKS cluster, this Stack requires integration with AWS. 
Navigate to the Settings, Integrations page and select the dropdown arrow to access the following selection screen:

image1-18

image1

The following links will help you set up the necessary integration with your Cloud Provider of choice.
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/integrations/cloud-providers/aws" target="_blank" rel="nofollow noopener">AWS</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/integrations/cloud-providers/azure" target="_blank" rel="nofollow noopener">Azure</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/integrations/cloud-providers/gcp" target="_blank" rel="nofollow noopener">GCP</a></li>
</ul>
To simplify things, I decided to utilize an AWS role that already exists with the necessary permissions to provide the essential resources associated with this Terraform example.

image9-4

To Trigger a Run, select Trigger on the right side of the Stacks view.

image3-16

Trigger a Run

<h3>Trigger Run Status</h3>
<h4>Planning</h4>
After you manually trigger the Run in the Stack view, Spacelift will deploy a runner image, initialize the Cloud Provider, and run a Terraform Plan based upon the contents of the Terraform code.
After a successful planning phase, you can check the log to see the planned changes.

image5-12

Trigger Run Status - Planning

To confirm the Triggered run, click the CONFIRM button at the top of the screen.
<h4>Finished Deployment</h4>
The following screen highlights the Finished Run and output from a successful deployment to your Kubernetes cluster.

image6-7

Trigger Run Status - Finished Deployment

To review the outputs associated with this EKS cluster deployment, navigate to the outputs page. You can find all pertinent information about this cluster.

image12-4

Outputs

You have completed the deployment of an EKS cluster via Terraform with Spacelift!

Get started on your journey by signing up for a <a href="https://spacelift.io/free-trial?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bprovision_kubernetes_cluster_terraform_spacelift%7D" target="_blank" rel="noopener">Free Trial</a> and taking it for a spin around the world! Sign-up is as easy as authenticating with the provider of your choice.

Spacelift effectively manages Terraform state, more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and includes many more features.

How to Deploy your Infrastructure in CI/CD using Terraform

Managing Azure Resources using Spacelift

Running Spacelift CI/CD workers in Kubernetes using DinD

Now that the Kubernetes integration is initially available learn how to complete the deployment of an EKS cluster via Terraform with Spacelift!

70-kubernetes_cluster_with_terraform_and_spacelift

At its most basic form, <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code (IaC)</a> is the provisioning and lifecycle management of infrastructure through code instead of manual processes. Over time, the code part has evolved, and each language is either imperative or declarative. The IaC tooling landscape has matured over the years. We have plenty of options to choose from; open-source and commercial. Over the past several years, a majority of the infrastructure related developments is summed up in a single word or ecosystem… Kubernetes!
Kubernetes is nothing new to Spacelift. The platform utilizes Kubernetes for infrastructure, our Public workers run on Kubernetes, and Spacelift support’s our customer&#8217;s deployment of <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">Private workers</a> on Kubernetes via <a href="https://helm.sh/" target="_blank" rel="nofollow noopener">Helm</a>.

image1-17

Kubernetes (Beta)

Kustomize drives Kubernetes support in Spacelift with native support in kubectl. The screenshot below highlights the ability to run kubectl from a Task within a given Stack.

image4-14

Tasks

The team is very excited to see what our customers and community decide to do with the newest integration. The following screenshot is a sample of the output from Spacelift during a Kubernetes Planning phase.

image2-17

Kubernetes Planning phase

Get started on your journey by signing up for a <a href="https://spacelift.io/free-trial?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bspacelift_kubernetes_integration_beta%7D" target="_blank" rel="noopener">Free Trial</a> and taking it for a spin worldwide! Sign-up is as easy as authenticating with the provider of your choice.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Achieving the Atlantis Workflow with Spacelift

How a Specialized Management Platform Can Improve Your IaC

Kubernetes is nothing new to Spacelift, but the team is very excited to see what our customers and community decide to do with the newest integration.

71-initial_kubernetes_integration

Running Spacelift CICD Workers in Kubernetes Using DinD

Check out the Spacelift Engineering blog to discover internal technical architecture posts written by Spacelift's very own engineers. 

Engineering | Spacelift Blog

Engineering

For a while now at Spacelift we’ve been hearing from users that they’d like to be able to run <a href="https://docs.spacelift.io/concepts/worker-pools" target="_blank" rel="noopener">Spacelift worker pools</a> in their Kubernetes clusters. As a first step towards that we decided to investigate whether we could run workers in Kubernetes using a Docker-in-Docker sidecar container. The rest of this post gives a rough overview of the architecture of Spacelift workers, and also explains how the sidecar container strategy works.
I’d just like to say thanks to <a href="https://medium.com/@v.m_hs" target="_blank" rel="nofollow noopener">Vadym Martsynovskyy</a> for his great article about running <a href="https://medium.com/hootsuite-engineering/building-docker-images-inside-kubernetes-42c6af855f25" target="_blank" rel="nofollow noopener">Jenkins agents in Kubernetes</a>. In that article he outlines the general approach taken here, and also describes some of the pros and cons of using Docker-in-Docker. It’s definitely well worth a read.

Before going into more detail, it’s worth explaining what a Spacelift worker is, and giving a quick overview of our architecture. At Spacelift, we provide a hybrid-SaaS model like many other CI/CD systems. What this means in practice is that the control plane that handles scheduling runs is managed by us, but the execution of runs happens in a separate process called a worker. We provide a public worker pool, but also allow you to self-host your own workers, providing complete control over your infrastructure.
Perhaps confusingly, if you follow the instructions to create a private worker pool, you’ll notice that it involves downloading something called the “launcher”. The launcher is responsible for communicating with the Spacelift mothership (control plane), and creating and destroying worker containers using Docker for each Spacelift run:

image2-5

Spacelift mothership (control plane)

As you can see, the launcher process needs access to a Docker daemon in order to create a container for each run.

We found that the launcher architecture maps very nicely to Kubernetes using a sidecar container to run the Docker daemon. For each worker in the pool, we need to create a Kubernetes Pod with two containers: one for the launcher; and another for Docker. The following shows a stripped down Deployment definition for creating a worker pool with 4 workers:

The launcher communicates with the Docker-in-Docker sidecar container via TCP, which is configured via the <code>DOCKER_HOST</code> environment variable for the launcher container. This works because the containers in a Kubernetes Pod share the same <a href="https://kubernetes.io/docs/concepts/workloads/pods/#pod-networking" target="_blank" rel="nofollow noopener">IP address and port space, and can communicate with each other via localhost</a>.
In addition, a shared volume called <code>launcher-storage</code> is mounted into each container. This is used by the launcher to store the workspaces for runs, along with other things like cached tool binaries (for example Terraform). The Docker sidecar needs access to the run workspaces in order to mount them into the worker containers, and it also uses that volume to store its image cache.
The last thing worth pointing out is that the <code>dind</code> container sets <code>securityContext.privileged</code> to <code>true</code>. This is required for Docker-in-Docker to function correctly.

The Spacelift launcher is distributed as a statically linked binary, making it simple to build an image. As you can see, the Dockerfile simply copies the launcher into an Alpine container, and then sets the startup command:

This image is rebuilt and published to our public ECR repository any time the launcher binary is updated.

We provide <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a> to make it really easy for users to deploy worker pools to AWS, Azure and GCP, and we wanted to provide a similar experience for Kubernetes. To achieve this, we created a Helm chart that makes it simple to deploy workers to Kubernetes clusters: <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">https://github.com/spacelift-io/spacelift-workerpool-k8s</a>.
Assuming you’ve already configured your worker pool in Spacelift and have access to the credentials for the pool, deploying this chart to your cluster simply requires two steps.
First, add the Spacelift Helm chart repository and update your local chart cache:

Next, install the chart:

Replace <code>&lt;worker-pool-token&gt;</code> and <code>&lt;worker-pool-private-key&gt;</code> with your own credentials, and make sure to base64-encode the private key.
If all goes well, you should be able to view the pods for your worker pool using <code>kubectl get pods</code>:

You should also be able to view the workers in your pool in Spacelift:

image1-5

Bitbucket DC Local Pool

When implementing in a production environment, you may want to use an alternative approach to providing credentials, and may want to configure a custom storage volume for the worker Pods. For more information about configuring the chart, check out the README in the <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">chart GitHub repo</a>.

If you’re interested in running Spacelift workers in Kubernetes, we’d welcome any feedback about the approach, contributions to the Helm chart, and also any issues you encounter so that we can make improvements. Also, if you aren’t already using Spacelift but are interested in trying it, you can signup and start your free evaluation of Spacelift <a href="https://spacelift.io/free-trial?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bci_cd_workers_in_kubernetes_dind%7D" target="_blank" rel="noopener">here</a>.

5 Ways to Manage Terraform at Scale - Best Practices

Building a Secure CI/CD Integration with Azure

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

This article is an overview of the architecture of Spacelift workers with an explanation of how the sidecar container strategy works.

49-running_spacelift_cicd_workers_in_kubernetes_using_dind-3

provision an AWS EKS Kubernetes cluster with Terraform

Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Terraform

AWS EKS provides managed Kubernetes clusters as a service. You’re on AWS and want to avoid getting into the details of setting up a Kubernetes cluster from scratch? EKS is the way to go!
In this guide, you will learn how to provision an AWS EKS Kubernetes cluster with Terraform. Let&#8217;s start with the basics.

First, locally install Terraform:

as well as the AWS CLI:

and kubectl:

If you’re on a different operating system, please find the respective installation instructions here:
<ul>
<li style="font-weight: 400;" aria-level="1">Terraform: <a href="https://learn.hashicorp.com/tutorials/terraform/install-cli" target="_blank" rel="nofollow noopener">https://learn.hashicorp.com/tutorials/terraform/install-cli</a></li>
<li style="font-weight: 400;" aria-level="1">awscli: <a href="https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html" target="_blank" rel="nofollow noopener">https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html</a></li>
<li style="font-weight: 400;" aria-level="1">kubectl: <a href="https://kubernetes.io/docs/tasks/tools/" target="_blank" rel="nofollow noopener">https://kubernetes.io/docs/tasks/tools/</a></li>
</ul>

Now, you’ll need to configure your AWS CLI with access credentials to your AWS account. You can do this by running

and providing your Access Key ID and Secret Access Key. You will also need to add the region. For the purposes of this guide, we will use us-east-2. Terraform will later use these credentials to provision your AWS resources.

You can now clone a repository which contains everything you need to set up EKS:

Inside you’ll see a few files, the main one being eks-cluster.tf:

It uses the EKS Terraform module to set up an EKS cluster with 2 worker groups (the actual nodes running your workloads): one with a single medium machine, and one with two small machines.

You can now create all of those resources using Terraform. First, run

to initialize the Terraform workspace and download any <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">modules</a> and providers which are used.
In order to do a dry run of the changes to be made, run

This will show you that 51 resources will be added, as well as their relevant details. You can then run terraform apply with the resulting plan, in order to actually provision the resources:

This may take a few minutes to finish. You might get a “timed out” error, in which case just repeat both the terraform plan and terraform apply steps.
In the end you will get a list of outputs with their respective values printed out. Make note of your cluster_name.

In order to use kubectl, which is the main tool to interact with a Kubernetes cluster, you have to give it credentials to your EKS Kubernetes cluster. You can do this by running

Make sure to replace &lt;output.cluster_name&gt; with the relevant value from your Terraform apply outputs.

You can now view the nodes of your cluster by running

This command is so long because it displays custom columns, thanks to which we can indeed see that there are 2 smaller nodes, and 1 bigger node.
Let’s deploy an Nginx instance to see if the cluster is working correctly.

You can see the status of it by running:

And finally set up a tunnel from your computer to this pod:

If you open <a href="http://localhost:3000" target="_blank" rel="nofollow noopener">http://localhost:3000</a> in your browser, you should see the web server greet you:

welcome to ngix

In order to destroy the resources we’ve created in this session, run

This may again take up to a few minutes. You can read more about it here: <a href="https://spacelift.io/blog/how-to-destroy-terraform-resources" target="_blank" rel="noopener">How to Delete Resources from Terraform Using the Destroy Command.</a>

I hope this guide helped you on your Kubernetes journey on AWS! If you want more help managing your Terraform state file, building more complex workflows based on Terraform, and managing AWS credentials per run, instead of using a static pair on your local machine, check out <a href="https://spacelift.io/?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bhow_to_provision_aws_eks_kubernetes_cluster_with_terraform%7D" target="_blank" rel="noopener">Spacelift</a>. We’d love to have you!

Spacelift helps manage Terraform state, build more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and many more.

Importing Existing Infrastructure into Terraform - Step by Step

Learn how to provision an AWS EKS Kubernetes cluster with Terraform. See a full step-by-step tutorial.

provision-an-aws-eks-kubernetes-cluster-with-terraform

How Specialized Solution Can Improve Your IaC

General | Spacelift Blog

General

The introduction of Infrastructure as Code or IaC has transformed the way you can provision and deploy high-performance cloud-based IT infrastructures.
IaC tools, such as Terraform, have been integrated into DevOps toolchains, saving DevOps IaC teams from the excessive manual effort.
While these tools undoubtedly help accelerate building IT infrastructures, their limitations can impact DevOps’ ability to optimize and improve control of their IaC processes supporting future business needs.
In this article, you’ll learn about IT infrastructure limitations that IaC DevOps teams deal with on a daily basis and how Spacelift is able to get past them.

One of the most frequent challenges while using more generic IaC tools is the non-intuitive workflow driven by its reliance on pull requests. Some solutions offer multiple workspaces, but the result can be fragile and nondeterministic. 
Since there’s no concept of mapping projects to branches or tags, anyone commenting on an approved pull request can deploy arbitrary code to production, even if the approval was meant for a short-lived experimental environment.
Spacelift does not depend on pull requests. It is mostly driven by push and tag events, so building a sophisticated Git flow is much easier. Spacelift reports the outcome of its jobs as commit status checks, allowing you to block merging the code on a failing Spacelift check.
Triggering a run can be customized using <a href="https://docs.spacelift.io/concepts/policy/git-push-policy" target="_blank" rel="noopener">Git push policies</a>. Thanks to that, Spacelift can provide the same level of comfort and security to teams using one project per repository and those using mono repo with hundreds of interdependent projects. You can read more about our approach to VCS integration <a href="https://docs.spacelift.io/integrations/source-control/github#tracked-branches" target="_blank" rel="noopener">here</a>.

The majority of generic tools don’t offer access control models but rely on comments on pull requests to drive infrastructure deployments. While it is usually fine when a single repository drives a single Terraform project, it becomes a huge liability for more complex scenarios.
Spacelift ships with a sophisticated mechanism allowing administrators to declare <a href="https://docs.spacelift.io/concepts/policy/login-policy" target="_blank" rel="noopener">who can log in</a> (and under what circumstances) and what their <a href="https://docs.spacelift.io/concepts/policy/stack-access-policy" target="_blank" rel="noopener">level of access</a> to each of the managed projects should be. Even our <a href="https://docs.spacelift.io/integrations/slack" target="_blank" rel="noopener">Slack integration</a> can be subject to policy controls, allowing an admin to grant access to a project <a href="https://docs.spacelift.io/integrations/slack#managing-access-to-stacks-with-policies" target="_blank" rel="noopener">based on Slack-specific data</a> (think team, channel, user, etc.).

One thing that’s not in scope for most IaC solutions is the way to ensure that your infrastructure is compliant with industry best practices and your company policies.
Spacelift puts policy-as-code in the center of its value proposition and builds a consistent, <a href="https://docs.spacelift.io/concepts/policy" target="_blank" rel="noopener">robust policy framework</a> on top of <a href="https://www.openpolicyagent.org/" target="_blank" rel="nofollow noopener">Open Policy Agent</a>. Apart from providing a comprehensive automated change review and ensuring compliance of your <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy" target="_blank" rel="noopener">Terraform changes</a>, Spacelift uses the same approach to allow you to declare rules around the <a href="https://docs.spacelift.io/concepts/policy/login-policy" target="_blank" rel="noopener">account</a> and <a href="https://docs.spacelift.io/concepts/policy/stack-access-policy" target="_blank" rel="noopener">project access</a>, <a href="https://docs.spacelift.io/concepts/policy/git-push-policy" target="_blank" rel="noopener">handling push notifications</a>, <a href="https://docs.spacelift.io/concepts/policy/run-initialization-policy" target="_blank" rel="noopener">starting runs</a> and <a href="https://docs.spacelift.io/concepts/policy/task-run-policy" target="_blank" rel="noopener">triggering tasks</a>, and creating <a href="https://docs.spacelift.io/concepts/policy/trigger-policy" target="_blank" rel="noopener">relationships between projects</a>.

Handling interdependencies between projects has always been Terraform’s Achilles’ heel. The usual approach to this problem is adding another layer of abstraction in the form of a Terraform wrapper like <a href="https://terragrunt.gruntwork.io/" target="_blank" rel="nofollow noopener">Terragrunt</a>. But it’s only a partial solution as it breaks the problem into smaller chunks.
<a href="https://docs.spacelift.io/concepts/policy/trigger-policy" target="_blank" rel="noopener">Spacelift’s trigger policies</a> on the other hand provide a smart, declarative automation layer on top of vanilla Terraform. They allow you to plug into state changes of individual projects and declare dependencies that should be resolved, following the changes that have just been applied. Read more <a href="https://docs.spacelift.io/concepts/policy/trigger-policy" target="_blank" rel="noopener">here</a> to discover other exciting possibilities.

Another problem to solve externally when using some of the generic tools is authoring and maintaining reusable <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a> for your organization. Terraform is flexible in allowing modules to come from <a href="https://www.terraform.io/docs/modules/sources.html" target="_blank" rel="nofollow noopener">various sources</a>, but ensuring confidential access, as well as testing and versioning, are left to you, the user.
Until now, the golden standard in that regard has been the <a href="https://www.terraform.io/docs/cloud/registry/index.html" target="_blank" rel="nofollow noopener">private module registry from HashiCorp</a>. But Spacelift offers much more. Far from being just a glorified package manager, Spacelift adds a <a href="https://docs.spacelift.io/vendors/terraform/module-registry" target="_blank" rel="noopener">full CI solution for Terraform modules</a>, out of the box and free of charge. You can thus ensure that your private modules are healthy before you distribute them to the rest of your organization.

If you manage a single or a handful of rarely changing projects, it’s likely that you just set your IaC up once and forget about it. But in a more dynamic environment, where microservices come and go, new environments proliferate and new product teams require their own Terraform workspaces. The need to configure it each and every time become a major nuisance, putting a lot of pressure on your DevOps team.
Enter Spacelift. In Spacelift, much of the configuration can be handled by the project owners themselves—you can add Terraform and/or <a href="https://docs.spacelift.io/concepts/configuration/environment" target="_blank" rel="noopener">environment variables</a> and mount files (even inject Terraform code!) programmatically or through the GUI without the need for administrative privileges or changing the central server configuration. For administrators, adding new projects requires minimal hassle since there’s no need to set up webhooks or change any YAML. And it can all be done programmatically using Terraform.

What comes as a pleasant surprise to users of generic CI tools, Spacelift entities such as stacks, contexts, modules or policies can be managed in a declarative way using your favorite infra-as-code tool (this rule applies also to their configuration). Yes, that’s right—<a href="https://docs.spacelift.io/vendors/terraform/terraform-provider" target="_blank" rel="noopener">Spacelift offers a Terraform provider</a> that allows you to programmatically manage the lifecycle of its own resources.
Administrative stacks get credential-less access to the subset of our <a href="https://docs.spacelift.io/integrations/api" target="_blank" rel="noopener">GraphQL API</a> that does not involve managing the actual infrastructure. For more sophisticated use cases, Spacelift allows you to generate API keys that are subject to the same access controls as normal users are, allowing you to create single-purpose tokens for restricted use by your internal scripts.

Generic IaC platforms do not provide any mechanisms to detect if your infrastructure is undergoing drift. Drift is a condition that represents the difference between the desired and the actual state of the infrastructure managed by your tool of choice &#8211; <a href="https://www.terraform.io/" target="_blank" rel="nofollow noopener">Terraform</a>, <a href="https://www.pulumi.com/" target="_blank" rel="nofollow noopener">Pulumi</a>, <a href="https://aws.amazon.com/cloudformation/" target="_blank" rel="nofollow noopener">CloudFormation,</a><a href="https://docs.spacelift.io/vendors/kubernetes"> Kubernetes</a>, or another. Drift can be caused by either or a combination of changes directly introduced by external actors &#8211; either humans or machines (scripts) or via the dependency of your resources on external data sources. In any case, drift is not good.
Spacelift has got you covered here. You can configure periodic <a href="https://docs.spacelift.io/concepts/stack/drift-detection" target="_blank" rel="noopener">drift detection </a>to notify you whenever drift happens, and take immediate action. You can even go a step further with optional automatic <a href="https://docs.spacelift.io/concepts/stack/drift-detection#to-reconcile-or-not-to-reconcile" target="_blank" rel="noopener">reconciliation</a>, ensuring your infrastructure always resembles your Terraform configuration.

General-purpose CI/CD platforms provide little to no insight into resource utilization from either a real-time or historical perspective. Which resources are over-or underutilized? 
Developers need to be able to intimately understand the material they’re working with. With regards to infra-as-code, the most important part of this story is understanding the managed resources in-depth. Both from the current perspective and through being able to put each resource in its historical context.
The resources view in Spacelift gives you greater visibility into each and every resource. With this deep insight into resources, DevOps are able to gain an understanding of the lifecycle of each resource managed by Spacelift and document it, regardless of the technology used — <a href="https://www.terraform.io/" target="_blank" rel="nofollow noopener">Terraform</a>, <a href="https://github.com/gruntwork-io/terragrunt" target="_blank" rel="nofollow noopener">Terragrunt</a>, <a href="https://spacelift.io/blog/what-is-pulumi" target="_blank" rel="noopener">Pulumi</a>, <a href="https://docs.spacelift.io/vendors/kubernetes">Kubernetes</a>, or <a href="https://aws.amazon.com/cloudformation/" target="_blank" rel="nofollow noopener">CloudFormation</a>.

Last but not least, Spacelift puts an emphasis on great user experience, offering a myriad of creature comforts. <a href="https://docs.spacelift.io/concepts/configuration/context" target="_blank" rel="noopener">Contexts</a> for example allow you to attach entire collections of configuration to individual stacks and modules. <a href="https://docs.spacelift.io/concepts/run/task" target="_blank" rel="noopener">Tasks</a> provide a powerful audited way of running one-off administrative commands on an initialized Terraform environment &#8211; subject to <a href="https://docs.spacelift.io/concepts/run/task" target="_blank" rel="noopener">their own policy constraints</a>. <a href="https://docs.spacelift.io/concepts/stack#stack-locking" target="_blank" rel="noopener">Stack locking</a> allows a single individual to take exclusive control over a stack to ensure that nobody is able to modify its state while crucial changes are being made.

Spacelift is an innovative and sophisticated SaaS product for Infrastructure as Code which helps IaC DevOps develop and deploy new infrastructures or changes quickly and with confidence.
Spacelift offers a unique set of IaC management capabilities:
<ol>
<li style="font-weight: 400;" aria-level="1">an intuitive, <a href="https://docs.spacelift.io/concepts/policy/git-push-policy" target="_blank" rel="noopener">versatile</a> and <a href="https://docs.spacelift.io/concepts/run#run-state-machine" target="_blank" rel="noopener">robust</a> workflow</li>
<li style="font-weight: 400;" aria-level="1">granular access controls on <a href="https://docs.spacelift.io/concepts/policy/login-policy" target="_blank" rel="noopener">account</a> and <a href="https://docs.spacelift.io/concepts/policy/stack-access-policy" target="_blank" rel="noopener">project</a> level that work well with an <a href="https://docs.spacelift.io/integrations/single-sign-on" target="_blank" rel="noopener">identity provider of your choice</a> (SSO)</li>
<li style="font-weight: 400;" aria-level="1">an <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy" target="_blank" rel="noopener">automated code review</a> and <a href="https://docs.spacelift.io/concepts/policy/task-run-policy" target="_blank" rel="noopener">threat detection</a> </li>
<li style="font-weight: 400;" aria-level="1">the ability to <a href="https://docs.spacelift.io/concepts/policy/trigger-policy" target="_blank" rel="noopener">declare complex workflows</a> between projects across multiple repositories</li>
<li style="font-weight: 400;" aria-level="1">a <a href="https://docs.spacelift.io/vendors/terraform/module-registry" target="_blank" rel="noopener">built-in private module registry</a> with a full CI system for modules</li>
<li style="font-weight: 400;" aria-level="1">effortless setup and customization with <a href="https://docs.spacelift.io/concepts/configuration/environment" target="_blank" rel="noopener">per-project environment management</a> and <a href="https://docs.spacelift.io/integrations/docker" target="_blank" rel="noopener">Docker integration</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/vendors/terraform/terraform-provider" target="_blank" rel="noopener">programmatic configuration</a> using Terraform</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/concepts/stack/drift-detection" target="_blank" rel="noopener">drift detection </a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/concepts/resources" target="_blank" rel="noopener">resource visualization</a></li>
</ol>
… and a multitude of additional perks like <a href="https://docs.spacelift.io/concepts/configuration/context" target="_blank" rel="noopener">contexts</a>, <a href="https://docs.spacelift.io/concepts/run/task" target="_blank" rel="noopener">tasks</a> or <a href="https://docs.spacelift.io/concepts/stack#stack-locking" target="_blank" rel="noopener">stack locking</a>.

There are many ways to manage your Infrastructure as Code. Each way is different in terms of complexity and offers a different set of features. It is important to keep in mind that choosing one way or another should be based on business and technical requirements. Most often, there is no point in implementing an in-house solution as the cost and effort of building and maintaining it may exceed its potential benefits. It is much easier and more efficient to leverage platforms such as Spacelift to provide these features for you instead.

Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation and controls to simplify and accelerate the provisioning of cloud based infrastructures

Terraform Functions, Expressions, Loops (Examples)

Terraform vs. Ansible : Key Differences and Comparison of Tools

Most often, there is no point in implementing an in-house solution as the cost and effort of building and maintaining it may exceed its potential benefits. It is much easier and more efficient to leverage ready-to-go platform to provide these features instead.

19-how_specialized_solution_can_improve_your_iac

How a Specialized Management Solution Can Improve Your IaC

Learning Terraform can be easy. The HCL configuration language is simple enough to start with basic resource creation straight away. 
But—
Once the code that defines your infrastructure grows in complexity, maintainability, and dependency, an uphill battle can ensue. And that is why working out an efficient deployment workflow is crucial for further development.
In this article you will see five approaches to managing Terraform workflows at scale, what their benefits and downsides are, and what problems they address.

A quick and easy way to start deploying resources using the Infrastructure as Code pattern is to use Terraform locally from the same machine you’re developing the code on. The only requirement is to have access to the target provider and an installed Terraform binary.
This approach makes it much easier to leverage commands related to day-to-day operations (such as state import, move, remove) or to get outputs than if doing it within the implementations where you have no direct access to the underlying provider.
You can also use all of the features of Terraform, such as remote state and state locking, to work with your teammates all-at-once. If you collaborate using a version control system and do not have continuous integration in place, but want to keep your code at a decent level of quality, there are tools to help you.
One of these is <a href="https://github.com/antonbabenko/pre-commit-terraform" target="_blank" rel="nofollow noopener">pre-commit-terraform</a> that will run a set of selected tools to lint the code you’re creating before you even commit it. Even if you have continuous integration available, it’s a handy way to avoid wasting time waiting for results from pull request status checks.
What are the drawbacks of this approach? Each action is manual and must be performed directly by the developer, which leads to several issues. 
In a situation where multiple people are working on the same codebase, you could often encounter a scenario such as this one:
Person A applies their changes to the environment and follows up with a pull request. At the same time, Person B would like to deploy their changes but is blocked as the codebase is not up-to-date with Person A’s changes. Applying the changes in this state would cause damage to the resources created in the previous deployment. When working in larger teams, this will often result in a lot of wasted time just waiting and rebasing on a codebase to apply the changes.
If you happen to write unit tests for <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a> (and you should be), you’re sure to know they need to be executed manually from your computer. Running them every time you change something in your configuration can be tedious and time-consuming. If you do not yet write tests for your modules, you’d be well advised to start looking at <a href="https://docs.spacelift.io/vendors/terraform/module-registry#tests" target="_blank" rel="noopener">Spacelift modules tests</a> or <a href="https://terratest.gruntwork.io/docs/getting-started/quick-start/" target="_blank" rel="nofollow noopener">Terratest</a>.
Privileged access to an underlying vendor is required, which can easily result in a compromised environment. It’s much harder to keep restricted access within this model, as Terraform must be allowed to create, change, and destroy resources. The same rule applies to a scenario where access to private (internal) resources is required, e.g., every developer has to set up a VPN connection to interact with them.
Moreover, every person working with the codebase needs access to the Terraform state, which creates a security concern due to how Terraform state works. Even if you’re pulling sensitive data from an external system such as Vault, it will be stored within the state file in plaintext once used in a resource. A threat actor could simply run `terraform state pull` to access all the secrets!
Keeping all these issues in mind, let’s take a look at a more automated approach.

Implementing homegrown automation means incorporating Terraform workflow into your continuous integration / continuous deployment process. A specific example would need to directly tie to a CI/CD tool you are using, so let’s try to keep the example as generic as possible.
As the applying actor is the CI/CD system, the hard requirement of privileged access for developers is gone. You can grant this type of access to the execution layer while developers keep read-only permissions. This is true for private (internal) resources, and if your code is being executed on an agent that resides within the target infrastructure with the proper permissions having been set. In a model where everything has to go through the <a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener">CI/CD pipeline</a>, visibility of changes is improved as the system logs produced during the operation execution are available in real-time. 
Linting, compliance checks, and automated unit tests can be moved to pull request status checks, and it’s up to you to configure these steps of the pipeline.
But—
If your CI/CD is executing jobs concurrently, you will end up with race conditions causing pull requests to fail non-deterministically. 
It is not difficult to imagine a situation where two operators open a pull request within several seconds of one another. The pipeline gets triggered immediately for both, and one pull request will be marked as successful, while the other will be marked as failed. This situation will definitely happen if you are using state locking (it is a best practice and you really should be using it).
The explanation is simple: the first pull request has locked the state and therefore the second one could not, so it failed. This issue can be resolved by configuring queued runs, where only one pipeline can run at a time. However, this is not so simple with some CI/CD products.
The biggest issue and concern with this approach is the effort you have to put in to build a complete pipeline. If you consider requirements such as:
<ul>
<li style="font-weight: 400;" aria-level="1">planning on pull requests</li>
<li style="font-weight: 400;" aria-level="1">linting</li>
<li style="font-weight: 400;" aria-level="1">unit tests</li>
<li style="font-weight: 400;" aria-level="1">compliance checks</li>
<li style="font-weight: 400;" aria-level="1">applying once merged</li>
<li style="font-weight: 400;" aria-level="1">periodical drift detection</li>
<li style="font-weight: 400;" aria-level="1">registry for all your versioned modules</li>
<li style="font-weight: 400;" aria-level="1">modules sharing across different parts of the organization</li>
<li style="font-weight: 400;" aria-level="1">shared variable contexts/reusable values</li>
</ul>
It might be tempting to implement solutions for those requirements by yourself. However, taking into account the time and effort required for doing it, you might want to take a look at alternatives, such as open-source tools.

One of the most popular open-source tools to use with Terraform is Atlantis, which describes itself as “Terraform pull request automation”. You can read more about it here: <a href="https://spacelift.io/blog/post/alternative-to-atlantis" target="_blank" rel="noopener">Alternative to Atlantis</a>.

Terraform Cloud is an application provided by the company behind Terraform itself—HashiCorp. It is available for free for up to 5 users. Two plans are available for more users: “Team &amp; Governance” and “Business”. As the product is created by Terraform authors themselves, any new feature that makes it to Terraform will be quickly available within Terraform Cloud as well. Neat!
<h3>Advantages of Terraform Cloud</h3>
Advanced security compliance mechanisms available through Sentinel allow you to enforce organization standards even before the code actually ships and creates resources with the chosen provider context.
Another important feature is a module registry that acts as an artifact repository for all your modules. It can be compared to JFrog Artifactory, Sonatype Nexus, or other similar software. Having every module in a separate repository and versioning allows you to control code promotion in a granular fashion or quickly roll back to if it allows you to go back to the most recent version only, then we should use “the”, but if it lets you go back to some other earlier version then we should use “a” previous version. 
Having a centralized place to store all the pieces also enables you to quickly browse through available modules, their documentation, inputs, outputs, and used resources. It is a great single pane of glass in terms of module management. However, it can only be used within the same Terraform Cloud organization and cannot be shared. This might be troublesome for operations teams that base their separation on the organization level.
There’s also the remote execution backend, which is an exclusive feature for Terraform Cloud. It allows you to work with Terraform on your local computer as you would usually do, upload your local codebase, and run a plan against it. It is a very efficient way of quickly verifying if what you are working on is actually working.
<h3>Disadvantages of Terraform Cloud</h3>
Along with so many benefits, there are some drawbacks as well. Any linting or module unit tests need to be implemented inside your own continuous integration system. This requires an additional effort the development team has to make to sustain their code quality.
Another disadvantage is that you are not able to share common values across your workspaces. Imagine a situation where you share a common set of variables like cloud region or environment name. The only way to achieve something similar to shared context is to have a dedicated workspace that will output this information as an input for all remaining workspaces, although this would be more of a workaround than an actual solution.
Compliance as code is available via Sentinel, a framework created by HashiCorp. Having a built-in solution to develop policies is nice, but operators often already use <a href="https://spacelift.io/blog/what-is-open-policy-agent-and-how-it-works" target="_blank" rel="noopener">Open Policy Agent (OPA)</a> to define these within different areas of the infrastructure. This frequently ends up in a doubled effort as you need to create a similar policy in both frameworks.
The biggest shortcoming of Terraform Cloud is in my opinion the lack of periodical drift detection runs. Once something is applied and the codebase does not get updates, you will never be notified if anything has changed in comparison to the state within the repository. Scheduling a daily “plan” to verify if everything is up-to-date would be highly beneficial to keep the infrastructure as described programmatically.

And here we are. You can <a href="https://spacelift.io/terraform-cloud-alternative" target="_blank" rel="noopener">compare Spacelift directly to Terraform Cloud</a>. They share a lot of functionality with each other in terms of the actual deployment execution, but also in the area of compliance.
While Terraform Cloud uses Sentinel (the HashiCorp approach to implementation for compliance as code), Spacelift leverages Open Policy Agent. It is beneficial to have a similar way of working with policies for companies that already incorporate OPA in their workflow (e.g., for Kubernetes). As there is no need to learn another syntax, operators can focus their efforts on providing compliance, instead of learning yet another tool.
Spacelift also allows you to automate your workflow even further. By having a built-in continuous integration for modules, you can shift your linters or unit tests really quickly into one place dedicated to Terraform. It helps you reduce the effort required to implement integration workflow by letting you utilize a solution that you are already using.
Sometimes you need to work with a Terraform state directly to import something, for example. In scenarios where you are leveraging Terraform Cloud, it is often impossible to do this without accessing the provider yourself. In contrast, Spacelift gives you the ability to run literally any command within the context of a particular codebase via tasks. This way, you do not need to grant any additional permissions to import, move, or remove resources directly.
By utilizing contexts, you can simply share anything you like across multiple stacks—whether it is a set of environment variables or a file. This is something that becomes much more valuable once your codebase complexity increases.
As mentioned in the previous paragraph, drift detection is something that becomes much more important once your infrastructure grows. With Spacelift, it is possible to <a href="https://docs.spacelift.io/concepts/stack/drift-detection#" target="_blank" rel="noopener">schedule a periodic drift detection</a> on any stack. Going even further, you can enable automated remediation if any changes are found compared to what you have in your codebase.

There are many ways of working with Terraform. Each way is different in terms of complexity, and has a different set of features. It is important to keep in mind that choosing one way or another should be based on business and technical requirements. Most times, there is no point in implementing an in-house solution as the cost and effort of building and maintaining it may often exceed its potential benefits. It is much easier and quicker to leverage platforms such as <a href="https://spacelift.io/?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7B5_ways_to_manage_terraform_at_scale%7D" target="_blank" rel="noopener">Spacelift</a> to provide these features for you instead.

How to Get Started with Terraform on Google Cloud Platform

This article goes through five approaches to managing Terraform workflows at scale: using Terraform locally, automating it in-house, using open source solution, using Terraform Cloud, and using Spacelift.

15 Kubernetes Best Practices Every Developer Should Know

Get the newsletter

Recent posts

Provision a Kubernetes Cluster with Spacelift and Terraform

Initial Availability of Kubernetes Integration in Spacelift

Running Spacelift CI/CD workers in Kubernetes using DinD

How to Provision an EKS Kubernetes Cluster with Terraform