Derek has worked in many environments ranging from an International Managed Hosting provider handling “actual metal” to Managed Service Providers to major training corporations. He has also started a Manufacturing Technology Provider utilizing Kubernetes and other cutting-edge tools. He currently works at Spacelift and his own training company morethancertified.com providing advocacy and training to all.

All posts by Derek Morgan - Infrastructure as Code Expert

Derek Morgan

Author title

Spacelift's going to AWS re:Invent in Las Vegas 🚀 🚀

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

Introducing Stack Dependencies v.2  🎉 🎉

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Product

Git has revolutionized the workflow as we know it today. Without it, deploying code can quickly become a nightmare. Developers deploy code that conflicts with each other, a change in one function breaks another function, security concerns multiply, manual processes abound, and so on. These issues can bring development to a complete standstill. Luckily, there are ways to improve this process.
Note: If you’re unfamiliar with Git, read this <a href="https://git-scm.com/docs/gittutorial" target="_blank" rel="nofollow noopener">tutorial</a> and any associated information on that page.

Tools like <a href="http://jenkins.io" target="_blank" rel="nofollow noopener">Jenkins</a>, <a href="http://circleci.com" target="_blank" rel="nofollow noopener">CircleCI</a>, <a href="http://gitlab.com" target="_blank" rel="nofollow noopener">GitLab CI/CD</a>, <a href="https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions" target="_blank" rel="nofollow noopener">GitHub Actions</a>, and many more help to keep everything under control when you are collaborating with others in a Git flow using GitOps. 
Here is a screenshot of Jenkins taken from <a href="https://en.wikipedia.org/wiki/Jenkins" target="_blank" rel="nofollow noopener">https://en.wikipedia.org/wiki/Jenkins</a>:

jenkins

Jenkins

These tools vary in complexity based on the job in question. 
For example, Jenkins is self-hosted, which can be a nightmare if your team needs to manage infrastructure better, but it’s convenient if they have to deal with compliance issues. Being self-hosted allows Jenkins to be air-gapped from the world and decreases the security footprint associated with connecting to an internet-enabled cloud service. 
GitLab is another example of a self-hosted option. GitLab also has a cloud version. Both versions are great, but the documentation can get confusing between the two. There are many other CI/CD tools out there, and choosing one that meets your security, velocity, and other criteria can be very difficult. 
<h3>CI/CD for applications and Kubernetes</h3>
One of the most important criteria when choosing a CI/CD tool is the workload you’re deploying. Most tools handle application workloads well; some specifically manage Kubernetes or container workloads, and some focus on any infrastructure as code (IaC). 
In most cases, choosing an all-in-one tool like Jenkins or GitLab is not your best choice when deploying different workloads. GitLab is fantastic for deploying application workloads. It has some great Kubernetes integrations, but unless you’re self-hosting it, it’s probably better to go with something more focused on Kubernetes, such as <a href="https://spacelift.io/blog/argocd" target="_blank" rel="noopener">ArgoCD</a>. For further information, see this article on <a href="https://spacelift.io/blog/kubernetes-ci-cd" target="_blank" rel="noopener">managing your Kubernetes deployments</a>. 
<h3>CI/CD for infrastructure</h3>
Cloud IaC deployments are similar but have some key differences. Application deployments require secret management, a build process, testing, whereas IaC deployments become even more complicated once you add cloud authentication, resource management, state management, infrastructure cost, policies, and more. 
Sure, an application deployment can cost <a href="https://www.electropages.com/blog/2021/01/how-start-received-75000-bill-2-hours-google-cloud-services" target="_blank" rel="noopener">thousands of dollars a minute</a> if something goes very wrong or leaks a secret, but an infrastructure deployment can still cost <a href="https://aws.amazon.com/message/41926/" target="_blank" rel="nofollow noopener">more</a>. One missed calculation, wrong instance size, leaked key, etc., can generate huge costs when infrastructure is involved, even if it’s serverless infrastructure. 
Luckily, several CI/CD tools are engineered explicitly for IaC. Spacelift is one of those tools, and it handles everything you need to ensure your deployments go off without a hitch.

<ul>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
<li><a href="https://spacelift.io/blog/getting-started-with-spacelift-self-hosted" target="_blank" rel="noopener">Getting Started with Spacelift Self-Hosted on AWS</a></li>
<li><a href="https://spacelift.io/terraform-cloud-alternative" target="_blank" rel="noopener">Alternative to Terraform Cloud / Enterprise</a></li>
</ul>

<h3>1. Robust access policies</h3>
The <a href="https://docs.spacelift.io/concepts/policy/" target="_blank" rel="noopener">policies</a> in Spacelift allow you to create guardrails for your engineers and ensure secure and reliable collaboration and deployments. Without these policies, engineers can deploy anything they want and potentially cause major issues. You can also create policies that manage deployment behavior. 
If you want to configure a massive monorepo and only deploy when specific files are modified, you can do it. Robust push, plan, and notification policies allow you to ensure your environment is always in check.

robust-access-policies

Robust access policies

<h3>2. SSO to your Git provider</h3>
Accessing your Git provider and repositories could not be easier, thanks to <a href="https://docs.spacelift.io/integrations/single-sign-on/" target="_blank" rel="noopener">Spacelift SSO</a>. Spacelift SSO removes the security risks of managing passwords and keys and enables a simple connection you have complete control over. Users from your Git provider are added to Spacelift, and policies can be created to manage those users and groups.

sso-to-your-git-provider

SSO to your Git provider

<h3>3. Keyless access to your cloud provider</h3>
You must control access to your cloud provider at all costs. Major cloud providers recommend assuming temporary credentials when communicating with their API, and Spacelift makes this easy. With Spacelift <a href="https://docs.spacelift.io/integrations/cloud-providers/" target="_blank" rel="noopener">cloud integrations</a>, you can configure only the access you need and assume temporary credentials that expire based on the TTL you configure.

keyless-access-to-your-cloud-provider

keyless access to your cloud provider

<h3>4. Self-hosted workers for security</h3>
Sometimes you don’t want to manage the entire platform, but regulations require specific security measures to ensure only you have access to your deployments. <a href="https://docs.spacelift.io/concepts/worker-pools" target="_blank" rel="noopener">Self-hosted worker pools</a> are perfect for this use case. The Spacelift control plane is able to orchestrate and launch the workers but has no access to anything in your environment.

worker-pools

worker pools

<h3>5. Spacelift Self-Hosted on AWS</h3>
If private workers aren’t enough to meet your security needs, you can even <a href="https://spacelift.io/blog/introducing-spacelift-self-hosted">self-host</a> the entire Spacelift platform. With Spacelift’s self-hosting ability, you can run your own Spacelift instance within your AWS environment and have complete control over all storage and traffic. This helps you achieve the highest security standards.
 You don’t have to manage a Kubernetes cluster or any complicated infrastructure. Everything is packaged into a straightforward Cloudformation script. 
<h3>6. Spaces</h3>
It can be tricky to ensure developers have access to the resources they need, and only those resources. 
Luckily, <a href="https://docs.spacelift.io/concepts/spaces/" target="_blank" rel="noopener">Spacelift Spaces</a> create a logical separation between stacks and resources and provide a visual representation of relationships and inheritance. Spaces allow your organization to achieve more complicated configurations, such as multi-tenancy.

spacelift-spaces-2

spacelift spaces

<h3>7. Blueprints</h3>
Creating a secure, homogenous environment for developers to deploy is an increasingly common practice. Platform engineering is becoming very popular, and companies are working hard to make their deployments simpler and more secure. 
<a href="https://docs.spacelift.io/concepts/blueprint/" target="_blank" rel="noopener">Spacelift Blueprints</a> facilitates platform engineering by providing a simple templating system for IaC developers to create environments for others.

spacelift-blueprints

spacelift blueprints

<h3>8. Cost estimation</h3>
Spacelift knows that predictability is a vital element of accurate cost estimation. That’s why we allow you to see from the start of the month how much you will have to pay for our services. You can also have a resource cost estimate on the stack level using our native integration with Infracost. 
You can see how easy it is to set up <a href="https://spacelift.io/blog/terraform-cost-estimation-using-infracost" target="_blank" rel="noopener">here</a>.
<h3>9. Integrations</h3>
<a href="https://spacelift.io/blog/integrating-security-tools-with-spacelift" target="_blank" rel="noopener">Custom Inputs</a> enable easy integrations with third-party tools within Spacelift. You can integrate any tool and even define different policies for these tools. You can also integrate Spacelift with monitoring, chats, and other DevOps-related tools, using <a href="https://spacelift.io/blog/introducing-notification-policies" target="_blank" rel="noopener">notification policies</a>. A ready-to-use integration with DataDog can be viewed <a href="https://docs.spacelift.io/integrations/observability/datadog" target="_blank" rel="noopener">here</a>.

IaC is designed to enable infrastructure and software development teams to collaborate, but general-purpose CI/CD tools can make things complicated. Spacelift fits easily into your existing workflows to help you manage collaboration around your IaC deployments with ease. To discover just how flexible our platform is, <a href="https://spacelift.io/schedule-demo" target="_blank" rel="noopener">book a demo</a> with one of our engineers or see how it works with a <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">free trial</a>.

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Blasting Off with Spacelift Spaces

Introducing Blueprints by Spacelift

Introducing Spacelift Cloud Integrations

How does Spacelift manage IaC CI/CD and collaboration? See how easily Spacelift fits into your existing workflows.

229-enabling-seamless-collaboration-in-iac

Enabling seamless collaboration in IaC

Enabling Seamless Collaboration in IaC

The Terraform state management features in Spacelift are extremely important to maintaining secure and reliable infrastructure deployments. Managing state yourself can be a security nightmare, lead to state corruption due to multiple engineers working simultaneously, and require difficult import operations related to infrastructure drift. 
It’s honestly just one more thing to manage in a sea of responsibilities that startups and enterprises have on their plate. Unfortunately, one of the issues with managing state remotely can be accessing that state for ad-hoc troubleshooting and cross-stack data access. 
Luckily, Spacelift has now rolled out the ability to access that remote state and manipulate it as needed! And what’s better, this feature can be toggled to ensure security when its access is not needed. Let’s take a look!

<h3>Enabling Remote State Access</h3>
Enabling remote state access is as easy as flipping a simple toggle switch or configuring it within the <a href="https://registry.terraform.io/providers/spacelift-io/spacelift/latest/docs/resources/stack#terraform_external_state_access" target="_blank" rel="nofollow noopener">Spacelift Terraform provider</a>. Remember that the stack trying to access the state of another must be an administrative stack. You can specify this in the behavior section of the stack&#8217;s setting. To enable remote state access in the Spacelift UI, head to your stack, click on “Settings,” “Backend,” and toggle “External state access”:

spacelift-terraform-provider

spacelift terraform provider

<h3>Basic State Access</h3>
Once remote state access is enabled, you will be able to use the <a href="https://developer.hashicorp.com/terraform/language/state/remote-state-data" target="_blank" rel="nofollow noopener">Terraform remote state data source</a> to access the state. Here, I have configured the data source in a main.tf file to connect to my Saturnhead Spacelift account and access the “pool_eks_stack” stack. This stack deploys an AWS EKS cluster using a modified version of the <a href="https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest" target="_blank" rel="nofollow noopener">AWS EKS Module</a> written by <a href="https://twitter.com/antonbabenko" target="_blank" rel="nofollow noopener">Anton Babenko</a>.

In this simple configuration, I can output all of the outputs configured in this module. 
If I only want to see the <code>cluster_name</code>, I can do so as follows:

Once I’ve configured these, I can run a <code>terraform plan</code> and see the output I asked for:

That’s pretty cool, but the real power comes in when you can start running Terraform commands against your state for debugging or troubleshooting purposes. I’ll add a Terraform remote backend to my main.tf file:

Once you run a <code>terraform init</code> with the configuration, you can run almost any command against the state that you wish. Of course, you’ll want to ensure you don’t make changes that will cause the code in your repository to become out of sync with the remote state. Once you have initialized the backend, you can run a <code>terraform state list</code> and see your deployed resources. 
Many useful Terraform commands can be used for troubleshooting and crafting your configuration files. Unfortunately, since the state is locked by Spacelift, you cannot run the Terraform Console against the remote state unless you pull it locally. I say “unfortunately” because you may wish to use it, but just remember that running commands that could possibly modify your remote state from the CLI is actually extremely disadvised. Hence, it’s probably best that this lock stays in place.
 You can get around this with the <code>terraform plan</code> and <code>terraform apply</code> commands by adding a simple flag, e.g. <code>terraform plan -lock=false</code>. But this can defeat the purpose of using a managed IaC deployment utility completely, so I would avoid doing this unless it’s a last resort. With that disclaimer out of the way, let’s look at the safer way to run troubleshooting commands against your state by pulling the state locally. 
<ol>
<li style="font-weight: 400;" aria-level="1">Clone the configuration directory from git that contains the code that deployed the state: <code>git clone &lt;your repo&gt;</code></li>
<li style="font-weight: 400;" aria-level="1">Pull the remote state to your local machine: <code>terraform state pull &gt; &lt;repo-directory&gt;/terraform.tfstate</code></li>
<li style="font-weight: 400;" aria-level="1">Change into the new directory: <code>cd &lt;repo-directory&gt;</code></li>
<li style="font-weight: 400;" aria-level="1">Check to ensure the <code>terraform.tfstate</code> file is in the directory, and run a <code>terraform state list</code> to ensure everything works. </li>
</ol>
After this, you can run any Terraform commands you need as if you were working with the configuration and the state locally without the risk of causing some serious state synchronization issues. 
As mentioned before, one of the most powerful tools is <code>terraform console</code>. If I wanted to test out the <a href="https://developer.hashicorp.com/terraform/language/functions/zipmap" target="_blank" rel="nofollow noopener">zipmap</a> function to see if I could construct a map of AWS AZ names to their respective zone ID, I can use <code>terraform console</code> like so:

terraform-console-2

terraform console

As you can see, I have the <code>data.aws_availability_zones.available</code> data source specified in my configuration file, I run <code>terraform console</code>, I check to see what the data source returns, and then use the <code>zipmap()</code> function to map the names and ids together. 
This is a simple example, but you can absolutely see how powerful this is when writing your Terraform configuration.

terraform-remote-state-yt-2

<ul>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Problems that Spacelift Solves&quot;}" data-sheets-userformat="{&quot;2&quot;:4224,&quot;10&quot;:2,&quot;15&quot;:&quot;Arial&quot;}">Infrastructure Problems that Spacelift Solves</a></li>
<li><a href="https://spacelift.io/blog/spacelift-pull-requests" target="_blank" rel="noopener">Working with Pull Requests on Spacelift</a></li>
</ul>

Spacelift’s new remote state access feature is one of the most powerful features we’ve released to date. Along with the incredibly powerful <a href="https://docs.spacelift.io/concepts/resources" target="_blank" rel="noopener">resource view</a>, <a href="https://docs.spacelift.io/concepts/blueprint/" target="_blank" rel="noopener">Blueprints</a>, <a href="https://docs.spacelift.io/integrations/cloud-providers/" target="_blank" rel="noopener">Cloud Integrations</a>, <a href="https://docs.spacelift.io/concepts/stack/drift-detection.html" target="_blank" rel="noopener">Drift Detection</a>, <a href="https://docs.spacelift.io/concepts/policy/" target="_blank" rel="noopener">OPA policy engine</a>, and more, Spacelift is the most powerful IaC deployment platform available, so check it out and <a href="https://spacelift.io/schedule-demo" target="_blank" rel="noopener">schedule a demo</a> today to see how we can help!

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;If you are struggling with Terraform automation and management, check out Spacelift. It helps you manage Terraform state, build more complex workflows, and adds several must-have capabilities for end-to-end infrastructure management.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">If you are struggling with Terraform automation and management, check out Spacelift. It helps you manage Terraform state, build more complex workflows, and adds several must-have capabilities for end-to-end infrastructure management.

Spacelift Module Registry - What It is and How to Use It

18 Most Useful Terraform Tools to Use in 2023

Terraform Cloud/Enterprise Pricing - Tiers Overview 2023

Spacelift has now rolled out the ability to access Terraform Remote State and manipulate it as needed.

187-remote-state-announcment

Spacelift Terraform Remote State

Terraform Remote State with Spacelift

How to use custom tools to complement Terraform

Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Terraform

By itself, Terraform is an incredibly formidable Infrastructure-as-Code tool, but as you begin to add complexity to your infrastructure, you may find that it occasionally needs a helping hand. Let’s look at some of the most popular tools used in Terraform-managed deployments today.
We will cover:
<ol>
<li><a href="https://spacelift.io/blog/terraform-tools#native-tool">Native Tools</a></li>
<li><a href="https://spacelift.io/blog/terraform-tools#ide-tool">IDE Tools</a></li>
<li><a href="https://spacelift.io/blog/terraform-tools#linting-tool">Linting Tools</a></li>
<li><a href="https://spacelift.io/blog/terraform-tools#security-tools">Security Tools</a></li>
<li><a href="https://spacelift.io/blog/terraform-tools#drift-tool">Drift Tools</a></li>
<li><a href="https://spacelift.io/blog/terraform-tools#costing-tool">Costing Tools</a></li>
<li><a href="https://spacelift.io/blog/terraform-tools#terraform-hcl-generation-tools">Terraform HCL Generation Tools</a></li>
<li><a href="https://spacelift.io/blog/terraform-tools#documentation-generation-tool">Documentation Generation Tools</a></li>
<li><a href="https://spacelift.io/blog/terraform-tools#management-tools">Management Tools</a></li>
<li><a href="https://spacelift.io/blog/terraform-tools#cicd-tools">CI/CD Tools</a></li>
</ol>

<h3>1. <a href="https://developer.hashicorp.com/terraform/cli/commands/console" target="_blank" rel="nofollow noopener">Terraform Console</a></h3>
The Terraform Console is probably one of the most overlooked tools in the arsenal. This is surprising because it’s native to Terraform and is packaged within the binary!
The <a href="https://spacelift.io/blog/terraform-console" target="_blank" rel="noopener"><code>terraform console</code></a> command provides an interactive console that can be used to test and evaluate any Terraform expressions you may wish to use in your code. Instead of endlessly experimenting, writing awkward outputs, and banging your head against the desk, the Terraform Console allows you to test out complicated expressions before you run your code. 
Let’s take a quick look at an example:

As you can see above, it’s useful for experimenting with expressions before you add them to your code. You can also evaluate variables and manipulate anything else in the state that you need. 
One thing to note is that the configuration must be able to pass a plan before the <code>terraform console</code> command will work, so make sure you comment out any issues you’re trying to solve before you enter the console to troubleshoot.

<h3>2. <a href="https://code.visualstudio.com/" target="_blank" rel="nofollow noopener">Visual Studio Code</a></h3>
I won&#8217;t be listing more than this. VS Code has just absolutely eclipsed other IDEs in the development world. There are certainly some great IDEs out there, but the integrations and plugins in VS Code are unparalleled. 
VS Code has two very popular Terraform extensions:

vs-code-terraform

vs code terraform

One is the official HashiCorp Terraform extension. It has more installations but also some pretty iffy reviews. It has seemed pretty stable whenever I’ve used it, but I typically default to the Terraform extension by Anton Kulikov. It has been solid and works very well. Your organization may have some policies around which extensions should be installed, so you should be good with either if they’re allowed.

<h3>3. <a href="https://github.com/terraform-linters/tflint" target="_blank" rel="nofollow noopener">TFLint</a></h3>
Tflint is an incredibly handy Terraform framework that lets you lint your Terraform code based on a prewritten ruleset or your own custom rules. These rules are added using “plugins.” 
Some of the most common issues you may want to uncover using <a href="https://spacelift.io/blog/what-is-tflint" target="_blank" rel="noopener">TFLint</a>, as cited from <a href="https://github.com/terraform-linters/tflint" target="_blank" rel="nofollow noopener">Github</a>, are:
<ul>
<li style="font-weight: 400;" aria-level="1">Find possible errors, such as invalid instance types, for the major cloud providers, including AWS, GCP, and Azure.</li>
<li style="font-weight: 400;" aria-level="1">Warn about deprecated syntax and unused declarations.</li>
<li style="font-weight: 400;" aria-level="1">Enforce best practices and naming conventions.</li>
</ul>
An example is below.

<h3>4. <a href="https://www.openpolicyagent.org/" target="_blank" rel="nofollow noopener">Open Policy Agent</a></h3>
Open Policy Agent isn’t specifically a Terraform framework or tool, but it can be used directly with Terraform. OPA, as it is commonly called, is written in the <a href="https://spacelift.io/blog/open-policy-agent-rego" target="_blank" rel="noopener">Rego</a> language, which is inspired by the old <a href="https://en.wikipedia.org/wiki/Datalog" target="_blank" rel="nofollow noopener">Datalog</a> language. It is also used by several tools as their policy language when analyzing Terraform code.
Tools such as Terrascan, Spacelift, Terraform Cloud, and more all support it and use it to varying extents. Spacelift, for instance, uses it for every aspect of the experience. From the time a user logs in to Spacelift to the point where they’re launching stacks, OPA Policies are there, ensuring everything goes according to plan. 
Here is an example of a policy that prevents Terraform from deploying AWS IAM access keys:

<h3>5. <a href="https://runterrascan.io/docs/" target="_blank" rel="nofollow noopener">Terrascan</a></h3>
Although the name “Terrascan” leads one to believe that it’s Terraform-specific, it actually works for many IaC providers. Terraform, Kubernetes, Ansible, Cloudformation, and more are supported by Terrascan thanks to its large set of pre-written <a href="https://runterrascan.io/docs/policies/" target="_blank" rel="nofollow noopener">policies</a>. Of course, you can write your own policies as well for custom use cases. 
<a href="https://spacelift.io/blog/what-is-terrascan" target="_blank" rel="noopener">Terrascan</a> integrates with <a href="https://runterrascan.io/docs/integrations/" target="_blank" rel="nofollow noopener">several CI/CD</a> tools and is extremely flexible when designing your pipelines. 
Here is a basic Terrascan usage example:

<h3>6. <a href="https://www.checkov.io/" target="_blank" rel="nofollow noopener">Checkov</a></h3>
<a href="https://spacelift.io/blog/what-is-checkov" target="_blank" rel="noopener">Checkov</a> is very similar to Terrascan, but it uses a <a href="https://www.checkov.io/3.Custom%20Policies/Python%20Custom%20Policies.html" target="_blank" rel="nofollow noopener">Python policy-as-code</a> framework instead of the Rego syntax found in OPA. Many engineers may find this more approachable as Python is one of the most popular programming languages today. 
Check an example of using Checkov in <a href="https://spacelift.io/blog/iac-scanning" target="_blank" rel="noopener">IaC scanning for vulnerabilities</a>.
Although Python is more approachable for many, the syntax is still a little wordy and can get a little complex, as shown from the official docs below:

The “wordiness” may be completely subjective. Python is an excellent language, and I’m sure people would happily debate Rego vs. Python for hours on end. 
Overall, the differences between Terrascan and Checkov come down to the policy language, as most other features are at parity with each other. 
<h3>7. <a href="https://github.com/aquasecurity/tfsec" target="_blank" rel="nofollow noopener">Tfsec</a></h3>
The tfsec tool is another Rego-based policy tool that is very similar to Terrascan and Checkov. One of the standout features is its multiple policy creation formats. You can use JSON, YAML, or Rego. Rego will be the most flexible and powerful, but JSON and YAML can create most policies you would need with a much more readable format. Read more in our <a href="https://spacelift.io/blog/what-is-tfsec" target="_blank" rel="noopener">What is tfsec?</a> article.
Here is an example of a check file in YAML:

<h3>8. <a href="https://driftctl.com/" target="_blank" rel="nofollow noopener">Driftctl</a></h3>
Driftctl allows you to perform one of the most crucial aspects of IaC management, which is to detect resources that have drifted from your desired state. It is capable of finding managed resources that have drifted as well as unmanaged resources that were created outside of driftctl. 
An example output is here:

<h3>9. <a href="https://www.infracost.io/" target="_blank" rel="nofollow noopener">Infracost</a></h3>
Infracost is probably one of the most popular tools used with Terraform in production deployments. It queries the pricing API of the provider and displays a cost based on what you plan to deploy. 
The great thing isn’t only that it shows you what it’s going to cost, but you can use a policy tool, such as Open Policy Agent, to write policies that can block a deployment based on the cost. CI/CD tools, such as Spacelift, can easily add tools like this into the deployment pipeline with very minimal configuration. Talk about a CFO’s dream! The community version is completely free but will require some custom tooling to get functionality like the policies mentioned above. If you choose the paid version, all of that is included. There is even a <a href="https://github.com/infracost/vscode-infracost/" target="_blank" rel="nofollow noopener">VS Code</a> extension to make things even more streamlined.
Here is an example of the truncated output:

Read more about <a href="https://spacelift.io/blog/terraform-cost-estimation-using-infracost" target="_blank" rel="noopener">InfraCost and how to estimate cloud costs with Terraform</a>.

<ul>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale</a></li>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Improve Your Infrastructure as Code using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Improve Your Infrastructure as Code using Terraform</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Automate Terraform Deployments and Infrastructure Provisioning&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
</ul>

<h3>10. <a href="https://github.com/GoogleCloudPlatform/terraformer" target="_blank" rel="nofollow noopener">Terraformer</a></h3>
Terraformer is an extremely cool product that scans your cloud environment and outputs the necessary Terraform code to redeploy it. This is definitely not the preferred route to codify your resources as, by now, most startups should have started with Terraform on day one. Unfortunately, many companies started before Terraform was the great tool it is today, so tools like Terraformer are ideal for getting things under new management.
Terraformer is great in a lot of ways, but you’ll want to be extremely careful how you import resources and that you truly understand the infrastructure that it creates and how you structure it. These types of tools rarely create perfect code at best and create an absolute mess of unreadable code at worst. I would strongly recommend reading ALL of the code that’s created and performing a thorough audit to ensure everything is in a readable and scalable structure.
<h3>11. <a href="https://github.com/JamesWoolfenden/pike" target="_blank" rel="nofollow noopener">Pike</a></h3>
Pike is an interesting tool that will analyze the resources you wish to create using Terraform and generate the necessary IAM permissions you need to complete that deployment. 
An example of this is here:

Like any security-related tool, ensure you audit the permissions afterward to ensure compliance.

<h3>12. <a href="https://github.com/terraform-docs/terraform-docs" target="_blank" rel="nofollow noopener">Terraform Docs</a></h3>
Let’s face it, writing documentation isn’t every developer’s dream job. Luckily, there’s an excellent tool to automatically generate your Terraform docs from configuration files. Terraform-docs is extremely useful and simple to use when creating documentation that shows exactly what your users need to know. It exposes variables you can add to a template file, and it will automatically generate the sections for you in an easy-to-read format. 
You can find an example of a repository that uses this <a href="https://github.com/flavius-dinu/terraform-az-aks" target="_blank" rel="nofollow noopener">here</a>.
And an example of the format is below:

<h3>13. <a href="https://tfswitch.warrensbox.com/" target="_blank" rel="nofollow noopener">TFSwitch</a></h3>
TFSwitch is a simple utility that helps to manage multiple Terraform versions. You can switch between any available Terraform version. The tool will also download the version you need if required. When working with multiple Terraform code bases, this is absolutely critical to keep pesky syntax errors and other headaches at bay if you’re dealing with code that has deprecated or newer features than your current version. 
To use TFSwitch, you just need to install it using the instructions <a href="https://tfswitch.warrensbox.com/Install/" target="_blank" rel="nofollow noopener">here</a>, and run the tfswitch command to display all of the available versions:

Once you choose the version you need, you are free to use Terraform! 
<h3>14. <a href="https://github.com/mineiros-io/terramate" target="_blank" rel="nofollow noopener">Terramate</a></h3>
Terramate is an interesting tool that can help you manage large Terraform deployments by breaking the code into “stacks.” These stacks have separate states and allow your code to be deployed with a smaller blast radius. It also has features that work with git to deploy automatically on changes. Where Terramate really shines is its ability to keep your code DRY (Do not Repeat Yourself). It is able to share variables, provider configurations, and other data between stacks and automatically generate any code necessary in pure Terraform. 
The tool is fairly complex in all of its features, so it’s probably best to read the <a href="https://blog.mineiros.io/introducing-terramate-an-orchestrator-and-code-generator-for-terraform-5e538c9ee055" target="_blank" rel="nofollow noopener">release blog</a> to let you dive in.
<h3>15. <a href="https://terragrunt.gruntwork.io/" target="_blank" rel="nofollow noopener">Terragrunt</a></h3>
Terragrunt is a very popular Terraform tool that, like Terramate, helps keep your <a href="https://spacelift.io/blog/terragrunt-with-spacelift" target="_blank" rel="noopener">Terraform code DRY</a>, manages multiple stacks, and more. It makes it easier to work with multiple cloud accounts, modules, and more with features that greatly increase the efficiency of your code. Terraform’s latest versions have honestly caught up with a lot of the features in Terragrunt, but definitely not all. 
Here is a sample configuration file.

As you can see in the comments, it makes it very easy to generate provider blocks for all modules and inject variables where they’re needed without having to pass them manually all over the place. It’s definitely an interesting project and something worth checking out if you start to juggle a lot of modules.
Check out our <a href="https://spacelift.io/blog/terragrunt-vs-terraform" target="_blank" rel="noopener">Terragrunt vs. Terraform comparison</a> to learn more.

<h3>16. <a href="https://www.runatlantis.io/" target="_blank" rel="nofollow noopener">Atlantis</a></h3>
Atlantis is a great open-source project for Terraform pull-request automation. While that sounds a little niche, it’s actually incredibly important to nail down your GitOps workflow. When dealing with multiple developers, committing code, creating a pull request, and having a platform inspect that pull request and deploy your code is critical to maintaining deployment velocity. 
Within Atlantis, you can also integrate other tools listed in this article to further enrich your pipeline and add functionality and guardrails to ensure reliable deployments. Atlantis is fairly opinionated about its pull request workflows, but if the workflow fits your GitOps strategy, it’s an excellent open-source option if you use Terraform for your IaC. 
Check out a great comparison of <a href="https://spacelift.io/blog/alternative-to-atlantis" target="_blank" rel="noopener">Spacelift and Atlantis</a>.
<h3>17. <a href="https://cloud.hashicorp.com/products/terraform" target="_blank" rel="nofollow noopener">Terraform Cloud</a></h3>
<a href="https://spacelift.io/blog/what-is-terraform-cloud" target="_blank" rel="noopener">Terraform Cloud</a> is part of the Hashicorp Cloud Platform (HCP) that, like Atlantis, manages Terraform deployments triggered by commits to your VCS. 
Terraform Cloud has excellent state management features, and the deployment pipeline can be customized with most of the tools featured in this article. Unfortunately, the ability to integrate external applications currently requires a paid plan. Policy as code with Hashicorp’s proprietary Sentinel engine or with OPA is also possible but requires a paid plan as well. 
Terraform Cloud manages your variables, secrets, resources, and more in its offering. It also allows you to script itself in Terraform, so everything can be automated. While it only supports Terraform, it does so very well. The free version is decent for a small team, but <a href="https://spacelift.io/blog/terraform-cloud-pricing" target="_blank" rel="noopener">pricing escalates once you scale</a>.
<h3>18. <a href="https://spacelift.io/" target="_blank" rel="noopener">Spacelift</a></h3>
Spacelift shares a lot of features with Atlantis, Terraform Cloud, and other “Terraform Automation and Collaboration Software” (TACOS), but keeps developers, policies, and simplicity in mind from the moment you sign in. You can sign into Spacelift using SSO from your VCS provider and <a href="https://docs.spacelift.io/getting-started" target="_blank" rel="noopener">get going right away</a>. Spacelift support Terraform as well as Cloudformation, Ansible, Pulumi, and Kubernetes, which sets it apart from other CI/CD tools mentioned in this list. 
Spacelift is built with policies in mind and allows <a href="https://docs.spacelift.io/concepts/policy/" target="_blank" rel="noopener">OPA policies</a> to be written that can manage nearly every aspect of the platform. From login, stack access, planning, deployment, and more. You can also <a href="https://docs.spacelift.io/concepts/stack/stack-settings#customizing-workflow" target="_blank" rel="noopener">integrate</a> any of the tools mentioned in the other sections of this article, such as Infracost and Terrascan, to check your code before deployment (see more examples: <a href="https://spacelift.io/blog/integrating-security-tools-with-spacelift" target="_blank" rel="noopener">Integrating Security Tools With Spacelift</a>). Writing policies based on the findings of these tools is also a really great feature that allows you to manage every aspect of your IaC strategy. 
Spacelift also manages state, secrets, <a href="https://docs.spacelift.io/integrations/cloud-providers/" target="_blank" rel="noopener">cloud credentials</a>, and more through an intuitive user interface or by scripting everything in Terraform. If you have multiple stacks that need to be run, you can chain those together using stack dependencies. This is helpful if you need to run an Ansible playbook on newly created resources. 
Spacelift has a generous free tier for a single user that covers most of the features. Like Terraform Cloud, you’ll need to upgrade to get more advanced features, such as private workers, stack scheduling, and an incredibly useful <a href="https://spacelift.io/drift-detection" target="_blank" rel="noopener">Drift Detection</a> feature.

That’s a quick run-through of several popular Terraform tools you’ll find in the wild today. New tools come out daily, and the popularity of these tools ebbs and flows frequently. Luckily, most of these tools work in very similar ways and make it very easy to move between them. 
Make sure you check them out and especially give Spacelift a go for a fully-featured <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">free trial</a>!
Note: New versions of Terraform will be placed under the BUSL license, but everything created before version 1.5.x stays open-source. <a href="https://opentofu.org/" target="_blank" rel="nofollow noopener">OpenTofu</a> is an open-source version of Terraform that will expand on Terraform’s existing concepts and offerings. It is a viable alternative to HashiCorp’s Terraform, being forked from Terraform version 1.5.6. OpenTofu retained all the features and functionalities that had made Terraform popular among developers while also introducing improvements and enhancements. OpenTofu is the future of the Terraform ecosystem, and having a truly open-source project to support all your IaC needs is the main priority.

Initialize/ plan/ apply your IaC, manage modules, state, and more. 

frame-8479

Frame 8479

Why DevOps Engineers Recommend Spacelift

How to Migrate From Terraform Cloud to Spacelift

Atlantis vs. Terraform Cloud / Terraform Enterprise - Comparison

Take a look at some of the most popular tools used in Terraform-managed deployments today.

185-terraform-custom-tools

At Spacelift, we have historically been focused on providing sophisticated tools for advanced engineers to deploy mission-critical infrastructure to the cloud. We’ve seen some pretty complex deployments and have worked hard to ensure they always go off without a hitch. 
As we’ve grown, we’ve had several engineers request to make it easier for other engineers, developers, and people who aren’t quite as involved in the infrastructure process as they are. Essentially, these engineers hope to provide a more self-service deployment mechanism to these other engineers, and Spacelift has heeded the call! Enter Blueprints by Spacelift!

Blueprints are templates for environments that let the administrators configure all defaults, guardrails, and other settings to make it easier for anyone who needs infrastructure to deploy that infrastructure. This could be developers, managers, or other administrators. Any setting available when configuring a stack can be configured as a default value or within certain parameters in Blueprints. 
Let’s take a quick look:

what-are-blueprints

what are blueprints

As you can see, there are two Blueprints. One of those Blueprints is a draft, and the other is published. 
When a Blueprint is first created, it is set as a draft. When you are ready to publish, it’s as simple as clicking a button in the UI or switching a value in the <a href="https://registry.terraform.io/providers/spacelift-io/spacelift/latest/docs/resources/blueprint" target="_blank" rel="nofollow noopener">Spacelift Terraform Provider</a>. 
Check out this short video with the Blueprints introduction:
<div style="padding: 56.25% 0 0 0; position: relative;"><iframe style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;" src="https://player.vimeo.com/video/807756671?h=609a1b2a2c" frameborder="0" allowfullscreen="allowfullscreen"></iframe></div>
<script src="https://player.vimeo.com/api/player.js"></script>

Creating a new Blueprint from the UI is a simple process. Of course, since Spacelift is an IaC-first product, you can create them using our <a href="https://registry.terraform.io/providers/spacelift-io/spacelift/latest/docs/resources/blueprint" target="_blank" rel="nofollow noopener">Terraform provider</a>. Just head to the Blueprints pane and click on “Create Blueprint,” fill out the necessary fields, and “Create Blueprint Draft”:

create-blueprint-draft

Create Blueprint Draft

Once you have created the draft, you will be presented with your new Blueprint with some uncommented fields and a whole lot of fields that you can use to customize the Blueprint for your users:

customize-the-blueprint-for-your-users

Customize the Blueprint for your users

Everything that can be customized in a stack can be customized in a Blueprint. You will also notice the robust templating system, input variables, functions, etc. All of these are well <a href="https://docs.spacelift.io/concepts/blueprint/index.html" target="_blank" rel="noopener">documented</a>. 
There is even a Command Palette that can be accessed using <code>F1</code> or right-clicking in the editor window and selecting it:

command-palette

Command Palette

In the following image, I have configured the Blueprint to hardcode the repository so that it can’t be modified, provided a list of possible Spaces to use, and added the ability to pass the chosen instance size into the code as a variable:

blueprints-ml-developer-environment

blueprints ml developer environment

<ul>
<li><a href="https://spacelift.io/blog/spacelift-module-registry" target="_blank" rel="noopener">How to Use Spacelift Module Registry</a></li>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
<li><a href="https://spacelift.io/blog/deploying-spacelift-with-spacelift" target="_blank" rel="noopener">Deploying Spacelift with Spacelift</a></li>
</ul>

Creating a stack from a Blueprint is easy! Click “Create stack” next to the Blueprint you’d like to use, fill in the necessary information, and click “Create stack” again. 
As you can see here, I have named the stack with my name, provided a list of Spaces to choose from, and can choose the size of the instance I’d like to use. 
This, of course, would be passed in as a variable to the underlying infrastructure code.

create-stack-from-blueprint

create stack from blueprint

Your stack is now ready to go with all of the defaults you configured:

blueprint-stacks

blueprint stacks

The Blueprints feature is available now for Spacelift Enterprise accounts!

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift is an alternative to using homegrown solutions on top of a generic CI solution. It allows you to automate, audit, secure, and continuously deliver your infrastructure.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Spacelift is an alternative to using homegrown solutions on top of a generic CI solution. It allows you to automate, audit, secure, and continuously deliver your infrastructure.

16 DevOps Best Practices Every Developer Should Know

Infrastructure Problems that Spacelift Solves

How Spacelift Can Improve Your Infrastructure as Code

Introducing Blueprints by Spacelift - templates for environments that let the administrators configure all defaults, guardrails, and other settings to make it easier for anyone who needs infrastructure to deploy it.

163-blueprints-announcement

Strategies to Manage Infrastructure at Scale

General | Spacelift Blog

General

Cloud-based infrastructure is amazing in how it is capable of scaling to uncanny heights in order to support millions of users. It seems effortless, in a way, but the reality is things start to get a little crazy when you’re doing anything outside of a simple autoscaling group and a load balancer. Once you start dealing with Terraform, Kubernetes, Ansible, and other <a href="https://spacelift.io/blog/infrastructure-as-code-tools" target="_blank" rel="noopener">Infrastructure as Code tools</a>, your codebase begins to swell.
A swelling codebase is usually due to a large number of engineers working on several different pieces of the puzzle. Dealing with multiple humans means there will inevitably be mistakes. Some of these mistakes, such as syntax errors or the occasional forgotten comment, are quickly mitigated and cause no harm. Other mistakes, such as a leaked security key, improper storage security setting, or an open security group, can prove to be disastrous.
Let’s take a look at some ways to manage your infrastructure at scale when things really start to take off.

I wanted to call this the “old” way, but to be honest, the non-scaleable way that I’m going to discuss is still heavily in use. In fact, the non-scaleable way is quite useful for single-developer shops and small stacks that just don’t want to deal with the added complexity and abstraction of the methods mentioned in this article.
Some of the strategies, such as using git repositories and basic security controls, are absolutely crucial at any scale if you wish for your code not to disappear into the ether after one fateful lightning strike, but many of the others just aren’t entirely necessary when starting out. 
Being pragmatic and writing readable and maintainable code is incredibly important when building, but eventually, you’ll need to start preparing for “scale”. Let’s take a look at some of the issues with the non-scaleable way of doing things.

Obviously, we’re not talking about train stations and farming infrastructure here. When infrastructure developers are running infrastructure deployments from their terminals, consistency will most likely take a major hit. 
Now, don’t get me wrong, developers coding from their terminals is perfectly fine. The idea that developers should be tied to a remote terminal managed by the company is a pipedream that only the most die-hard corporate security evangelists have. Developers need the freedom to manage their environment completely, but that freedom should be demarcated at deployment.
These terminals should have access to the repository to which the code is being committed and nowhere further. Handing out access keys to cloud environments is just asking for trouble. It seems easy when you’re deploying a few environments, but when you’re deploying to hundreds of environments or more, things begin to get pretty unwieldy. It’s not just a conversation about key rotation; you also have to consider traversal attacks and privilege escalation.
 So, keeping everything managed in a more controlled environment greatly reduces your attack surface.

We’ve established your developers are not going to leave THEIR terminals. It would be outright silly to expect developers not to be able to control their environment to maximize their own productivity. So how do we break down the silos and ensure collaboration is seamless? GitOps, of course! 
The idea of a Git repository is a single source of truth for all code and the beginning of your deployments. The core principle on which tools like <a href="https://spacelift.io/" target="_blank" rel="noopener">Spacelift</a> operate. GitOps is going to be a central theme for this article. 
By centralizing the deployment mechanisms, it’s much easier to ensure everyone is able to deploy when they need to, with the right controls and policies in place. Also, by requiring all deployed code to be checked into a repository, this removes the silos and allows visibility into what your developers are working on.

Choosing between using a typical polyrepo (multiple repositories) strategy or a monorepo (a single repository with all code) can be a difficult task. 
Facebook, for example, chose the monorepo strategy. They trust all of their engineers to see all of the code. This makes code reuse much easier and allows a more simplistic set of security policies. If you’re an employee of Facebook, you get access. It’s as simple as that. This, of course, requires a lot of trust and when you have a huge number of divisions, it definitely can lead to issues.
Since Facebook doesn’t necessarily face the same regulatory constraints as, say, a major financial institution, this is a more efficient route for them in which they thrive. For companies that do face such regulatory challenges, being able to ensure engineers in one division don’t see the code of another can be incredibly important. This also goes for companies with multiple “skunk works” style divisions where any leak can cause disastrous consequences for marketing…or the legal team.
For these companies, a polyrepo strategy is best. Managing the security of these to prevent traversal attacks and the like is the top priority for security teams, and permissions should be audited frequently. Once your repository structure is set up, your GitOps strategy can commence. Unfortunately, not everything can be managed within the repository.

State management is a huge issue (when the state is concerned). Terraform is a clear candidate for state-management issues. Other <a href="https://spacelift.io/blog/devops-automation-tools" target="_blank" rel="noopener">IaC tools</a>, such as Ansible and Kubernetes, don’t encounter state-related issues, but there are still sensitive artifacts that must be managed.
Unfortunately, state can be a huge hassle for the GitOps paradigm as you definitely don’t want to store state in your repository. Because of this, many companies use a combination of S3 and DynamoDB for their state management. Similar offerings from other cloud providers will suffice as well, but I’ll stick to the most common to keep things simple.
For those of you using Cloudformation or Arm/Bicep, congratulations, your ability to battle through cluttered syntax to avoid managing state is commendable. 
For those of you using tools such as <a href="https://spacelift.io/blog/what-is-terraform-cloud" target="_blank" rel="noopener">Terraform Cloud</a> and Spacelift, which manage Terraform state for you, you also may not find much use out of this section as your lives are much simpler.
Once you start managing workloads that could be considered scale, managing state is not the first thing you want to have to think about. Securing your state buckets, providing the right amount of access to those that need it, and managing the DynamoDB tables to maintain integrity through locks are all crucial elements to managing the state yourself. It’s not incredibly difficult, but the repercussions of a mistake are dire.
Always ensure you encrypt your state and use any features available within your IaC tool to help keep tabs on sensitive values. In Terraform, you can mark values as sensitive to ensure they don’t leak into the terminal, but these values are still in the state to be consumed by anyone with access. There’s not much else to say that the documentation doesn’t already, so just make sure you keep things secure and manage locks properly to avoid integrity issues.

A “thought-leader” blog post wouldn’t be very “thought-leadery” without some quirky new term to explain a concept. This post is no different! In this case, I’m going to discuss what I call “barbell security”.
There has been a lot of conversation around “shift left” security, which puts more of the onus of security on developers, but there still needs to be a set of controls on the other end of the spectrum in the deployed environment. These controls are deployed by a security team.  Of course, this leaves the two human components with the bulk of the weight at either end of the security barbell to constantly have to work around each other.
Using GitOps to implement security testing through the entire pipeline is critical to a scalable deployment. For anyone dealing with a deployment that’s even close to being considered “scale”, this seems like a no-brainer, but a lot of organizations have placed this on the back burner or have opted to do basic code-linting without any actual policies. 
Using a policy management platform, such as OPA, is critical to keeping your security footprint in check at every step of the deployment. By doing this, you’ll ensure you don’t have the weight on both sides of the deployment, and you distribute it more evenly throughout the process.
We’ll dive a little deeper into these processes throughout the article.

<ul>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;10 Ways to Improve Your Infrastructure as Code&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">10 Ways to Improve Your Infrastructure as Code</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale</a></li>
</ul>

“Close” only matters in Horseshoes, right? Or so the old adage goes. If you’re not familiar with the game, Horseshoes allows you to acquire points just by touching the target versus actually “ringing” it. The game of Horseshoes allows you to score points based on a larger “blast radius”. 
Well, in security, if you get your blast radius wrong, you won’t get any points. In fact, the only thing you may score is a termination. 
&nbsp;

billy-madison-wrong

Ensuring that, if something does go wrong, a minimum number of resources is affected is an incredibly important task when scaling. When dealing with thousands of resources, one fateful configuration disaster can take hours, or even days, to recover from. Some of you may remember this fateful S3 outage caused by a misconfigured “Playbook”: <a href="https://aws.amazon.com/message/41926/" target="_blank" rel="nofollow noopener">https://aws.amazon.com/message/41926/</a>
Had AWS had policies that required confirmation when a certain number of critical resources were modified, this whole incident could have been avoided. I’m certainly simplifying the incident somewhat, but you get the gist. 
There are many ways to configure these policies, but one of the most common is to use a policy-as-code tool such as <a href="https://spacelift.io/blog/what-is-open-policy-agent-and-how-it-works" target="_blank" rel="noopener">Open Policy Agent</a> (OPA) and a clever “scoring” algorithm to score your resources based on their importance. If the resources set to be modified have a score above a threshold, it can require manual intervention from management, SREs, or whoever is on the list. 
For example, 
 
EC2 instance in a 100 EC2 instance autoscaling pool: 1 point
Redundant Load Balancer in a dual LB setup: 25 points
Production Database: 100 points 
 
You could easily set a threshold using OPA’s scripting language, <a href="https://spacelift.io/blog/open-policy-agent-rego">Rego</a>, to require authorization if the total points &gt; 49. This would ensure you don’t lose two load balancers, more than half of your EC2 instances, and definitely not your production database. This example is a little contrived, but I’m sure you get the point. 
Here is another <a href="https://github.com/spacelift-io/spacelift-policies-example-library/blob/main/plan/check-blast-radius.rego" target="_blank" rel="nofollow noopener">example</a> written in Rego that illustrates the concept:

In the above example, you can see the different “blast_radii_by_type” and their definitions:
“aws_ecs_cluster”: 20
“aws_ecs_user”: 10
“aws_ecs_role”: 5
And the deny rule states that if “blast_radius_too_high” is true, then to deny the run. The current threshold is set at &gt; 100. This obviously gets much more complicated as you start working with a significant amount of infrastructure, but it’s a great starting point.

Role-Based Access Control is a very important aspect of your security posture. Kubernetes has native <a href="https://spacelift.io/blog/kubernetes-security#1-use-rbac" target="_blank" rel="noopener">RBAC</a> that is extremely useful when coupled with Namespaces. Terraform doesn’t exactly have an RBAC system, but modules are useful to help ensure standards are maintained.
 By limiting infrastructure developers’ ability to deploy arbitrary resources and attributes, you help constrain the blast radius and solidify the security posture you require. By coupling strict linting and code-scanning rules with a module-based policy, you can ensure only the right settings get deployed. This also helps with costing as you can ensure only certain resources within your budgetary constraints are available within the modules. 
An example of a module-oriented structure may be something like this:
<ol>
<li>Modules are created by the infrastructure team responsible for that module. Networking, security, compute, etc. </li>
<li><a href="https://developer.hashicorp.com/terraform/language/expressions/custom-conditions" target="_blank" rel="nofollow noopener">Condition Checks</a> are added to ensure proper constraints. </li>
<li><a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks" target="_blank" rel="nofollow noopener">Pre-commit</a> hooks lint and scan for any vulnerabilities using tools such as <a href="https://spacelift.io/blog/what-is-tfsec" target="_blank" rel="noopener">tfsec</a> or checkov. </li>
<li> Modules are committed to VCS, where they are tested or consumed by a module registry.</li>
<li> If you are using a platform such as Spacelift that has a <a href="https://spacelift.io/blog/spacelift-module-registry" target="_blank" rel="noopener">private module registry</a> with built-in testing, those tests can be run on the modules to ensure they operate correctly. </li>
<li> Policies are written to ensure deployments to break any constraints. Open Policy Agent is great for this. Terraform Cloud also offers Sentinel. </li>
<li> Engineers are given access to these modules as their roles dictate. All of the guardrails are in place, and they are unable to configure parameters outside of those boundaries without triggering policies in place. </li>
<li> Code is deployed, and final compliance tests are performed within the environment to ensure everything is still up to standards. This could be using services such as AWS Cloudtrail or GCP Cloud Audit Logs, for example.</li>
</ol>

At any scale, secret management is important. This section really doesn’t change much whether you’re a 2-person startup or a 200,000-person corporation. Protecting your secrets is a foundational concept that doesn’t require a lot of explanation.

context-and-secrets-management-meme

context and secrets management meme

Verifying that secrets never enter your repository to begin with is a more crucial element as your codebase scales. With a few thousand lines of code in a monorepo, it can be fairly straightforward to manage your secrets, but once you get into the millions of lines of code, things get rather complicated. 
Ensure you’re using code scanning tools, encryption utilities, and anything else at your disposal that can help ensure this doesn’t happen. AWS Secrets Manager, Azure Key Vault, and Hashicorp Vault are all examples of applications that can assist with this. And, of course, if you do have to have some sort of static credentials instead of recommended short-lived ones, ensure they are rotated frequently. 
In Spacelift, secret definitions and files can be managed within a <a href="https://docs.spacelift.io/concepts/configuration/context" target="_blank" rel="noopener">Context</a>, which allows you to define any secrets you have and have them automatically encrypted. 
Secrets can also be added directly to the stack’s environment, but the contexts allow these secrets to be shared between stacks. This greatly simplifies the secret management process. Authentication to the cloud providers and VCS providers is also incredibly important to manage properly. 
If your cloud is AWS, using <a href="https://spacelift.io/blog/aws-iam-roles" target="_blank" rel="noopener">AWS Roles</a>, <a href="https://spacelift.io/blog/aws-sts" target="_blank" rel="noopener">AWS STS</a>, and other temporary credentials services is crucial. Other clouds have similar services that must also be used at all costs. 
Spacelift simplifies this process with its <a href="https://spacelift.io/blog/spacelift-cloud-integrations" target="_blank" rel="noopener">Cloud Integrations</a> service that automatically assumes temporary credentials for each run. 
As far as VCS authentication goes, this should be done with an <a href="https://docs.spacelift.io/integrations/single-sign-on/" target="_blank" rel="noopener">SSO</a> service or similar whenever possible. Spacelift also automates this process and allows full access to the repositories necessary for deployment without any passing around of credentials.

The actual deployment of most resources should be an automated process. Having manual procedures for every deployment would absolutely slow the entire process down to a crawl. Requiring manual approvals should be the exception, not the rule. Using strategies outlined above, such as blast radius, OPA usage, etc., should allow you to make intelligent decisions on when to involve manual processes. 
Another step you may want to take during deployments is to test the deployment in another environment. Deploying to a test or staging environment before deploying to production is an integral part of the deployment process. Managing this pipeline, catching edge cases, and keeping it as hands-off as possible is crucial when dealing with a large number of deployments daily. 
Some organizations may deploy to a test environment first by pushing changes to a dev branch, testing fully, then run a merge with prod, which deploys again using a different context. 
Others may want to push straight to a prod branch, have the pipeline run the plan, then deploy. The difference here is that the code merge actually happens after the deployment has been made. This is sometimes referred to as a “<a href="https://spacelift.io/blog/atlantis-workflow-on-spacelift" target="_blank" rel="noopener">pre-merge apply</a>” and is commonly used within the Atlantis open-source IaC deployment tool. This strategy allows the main branch to remain clean if there are issues encountered. You definitely want to be cautious about your blast radius here. Even if the plans are usually pretty reliable, things do happen.
Check out <a href="https://spacelift.io/blog/scaling-ci-cd" target="_blank" rel="noopener">7 steps to optimize your CI/CD pipelines for scaling</a>.

Thanks to the issues mentioned above, among others, the world of infrastructure deployments can be akin to the famous shootouts of the American Wild West. Unfortunately, unlike many of the stories you hear, Wyatt Earp doesn’t go riding off into the sunset.
 On the contrary, the entire organization suffers, and technical debt begins to pile up. As bandaids are used to allow developers access to what they need to deploy emergency patches and security “workarounds” are implemented by developers to get their jobs done without having to open a ticket for every little thing they need to deploy, everything starts to collapse. 
By following the tips in this article and taking the time to really map out your processes while engaging all teams and stakeholders involved, you will be able to scale your infrastructure deployments as far as you need.

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation and controls to simplify and accelerate the provisioning of cloud based infrastructures.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation and controls to simplify and accelerate the provisioning of cloud based infrastructures.

Business Benefits of Infrastructure as Code

Infrastructure Drift Detection and How to Fix It With IaC Tools

13 Most Useful Infrastructure as Code (IaC) Tools for 2023

By following the tips and strategies in this article, you will be able to scale your infrastructure deployments as far as you need.

162-infrastructure-at-scale

10 Strategies to Build and Manage Scalable Infrastructure

Notification policies in Spacelift allow you to create custom notifications that can be routed to the notification inbox, Slack, or webhooks to give you the information you need to manage your deployments. These policies are Space aware and apply to components in the same or inherited Spaces. You can also configure filters to ensure you only get the notifications you need and cut out the rest of the noise.

what-are-notification-policies

what are notification policies

Let’s take a look at a simple example of a notification policy.
This policy will send a notification to the notification inbox with a severity level of “info” every time a tracked run finishes. This notification will create a link to the run id to allow you to access the run information and see what happened in the run. 
Also, notice the Space is the “root” Space. If you wish to allow all Spaces with inheritance enabled to be part of the notification policy, this is probably the Space in which you should save your policy.

simple-notification-policy

simple notification policy

The copyable version of the policy is here:

You can also modify this policy to work with Slack or a webhook. The parameters may be slightly different, but they are documented <a href="https://docs.spacelift.io/concepts/policy/notification-policy.html#slack-messages" target="_blank" rel="noopener">here</a> and <a href="https://docs.spacelift.io/concepts/policy/notification-policy.html#webhook-requests" target="_blank" rel="noopener">here</a>. Once you save this policy, you should start seeing notifications in your notification inbox as soon as you have tracked runs in the finished state.

modify-policy

modify policy

<ul>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
<li><a href="https://spacelift.io/blog/alternative-to-atlantis" target="_blank" rel="noopener">Alternative to Atlantis – Spacelift</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Problems that Spacelift Solves&quot;}" data-sheets-userformat="{&quot;2&quot;:4224,&quot;10&quot;:2,&quot;15&quot;:&quot;Arial&quot;}">Infrastructure Problems that Spacelift Solves</a></li>
</ul>

Let’s take a look at one more example that notifies on a tracked run failure. I’ll just paste the relevant policy code below:

As you can see, I have changed the “severity” to “ERROR”, the message to indicate the run has failed, and the <code>run.state</code> to “FAILED” instead of “FINISHED”. A pretty simple modification, and we now have notifications for all finished and failed runs.

notification-on-a-tracked-run-failure

notification on a tracked run failure

Perfect!

Check out this short video with the Notification Policy introduction: 
<iframe loading="lazy" src="https://player.vimeo.com/video/779273616?h=3d9eb9884f" width="640" height="360" frameborder="0" allowfullscreen="allowfullscreen"></iframe>

As you can see, the notification policy feature is extremely valuable and powerful when auditing and managing your infrastructure deployments. Much more information can be found in the <a href="https://docs.spacelift.io/concepts/policy/notification-policy.html" target="_blank" rel="noopener">documentation</a>.
I hope you enjoy the new feature and keep deploying!

Deploying Spacelift Stacks with Spacelift Terraform Provider

Notification policies in Spacelift allow you to create custom notifications to give you the information you need to manage your deployments.

141-notifications

Introducing Notification Policies

Introducing Spacelift Notification Policies

Until now, having stacks trigger other stacks and configuring those dependencies was possible but required trigger policies and some fine-tuning to get right. Luckily, Spacelift has now released Stack Dependencies! The Stack Dependencies feature makes it much easier to configure dependencies between stacks to deploy more modular and complex infrastructure.

If your organization has been lucky enough to have less complex infrastructure that doesn’t require stacks to trigger other stacks, consider that a good thing! Unfortunately, as infrastructure grows, so does its complexity. To manage modularity, guardrails, and other concerns, most organizations split up their infrastructure into multiple modules or logical parts.
In the simplest case, you may find an infrastructure stack and an application configuration stack where the infrastructure is deployed first, and the application is configured afterward. The infrastructure could be deployed using Terraform, Cloudformation, Pulumi, or others. The application could be configured using one of the former or even Ansible or Kubernetes stacks.
In this post, we’ll focus more on implementing the Stack Dependency feature than what’s being deployed.

stack-dependencies-intro

stack dependencies intro

As you can see in the above image, we have two stacks: Infra Stack and App Stack.
In this scenario, we will configure a simple dependency that instructs App Stack to be triggered any time Infra Stack is run. This could be triggered manually or by some other triggering event such as a push event.
To configure dependencies through the UI (the recommended way is typically to script everything in Terraform and deploy using Spacelift), click on one of the stacks that is part of the dependency chain, and choose “Add dependencies” depending on which side of the chain the chosen stack lies.

configure-dependencies

configure dependencies

Since this is the Infra Stack, we’ll want to choose the “Add dependencies” button on the “Depended on by” side.
Once we’ve done that, we can choose the App Stack:

Spacelift has now released Stack Dependencies! This feature makes it much easier to deploy more modular and complex infrastructure.

Derek Morgan

Enabling Seamless Collaboration in IaC

Get the newsletter

Recent posts

Terraform Remote State with Spacelift

18 Most Useful Terraform Tools to Use in 2023

Introducing Blueprints by Spacelift

10 Strategies to Build and Manage Scalable Infrastructure

Introducing Notification Policies

Introducing Stack Dependencies

Blasting Off with Spacelift Spaces

Announcing Spaces by Spacelift

Deploying Spacelift Stacks with Spacelift Terraform Provider

Terraform Custom Conditions – Preconditions & Postconditions

Ansible + Spacelift = Better Together