Flavius is a passionate Developer Advocate with an Infrastructure as Code mindset and expertise in DevOps & Cloud Engineering.

All posts by Flavius Dinu - DevOps Expert

Flavius Dinu

Author title

Register for the Product Momentum Webinar Aug 6 →

Learn about Spacelift's latest features and capabilities 🚀🚀

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

Transform your IaC management to scale
securely, efficiently, and productively
into the future.

Blog

What Are Terraform Modules and How to Use Them: Tutorial

Terraform Functions, Expressions, Loops (Examples)

Managing Terraform State - Best Practices & Examples

Terraform For Loop - Expression Overview with Examples

What is Terraform? What is it Used For, How it Works

44 Ansible Best Practices to Follow [Tips & Tricks]

How to Stop and Remove Docker Containers

How to Create a CI/CD Pipeline with Docker [Tutorial]

CI/CD Pipeline : Everything You Need To Know

Kubernetes CI/CD Pipelines - 7 Best Practices and Tools

How to Deploy Infrastructure in CI/CD Using Terraform (Pipeline)

Terraform with Azure DevOps CI/CD Pipelines - Tutorial

Terragrunt Tutorial - Getting Started & Examples

Pulumi Pricing – Editions Overview 2024

73 Most Useful DevOps Tools to Use in 2024

Top 10 Terraform Enterprise Alternatives for 2024

Terraform Cloud/Enterprise Pricing - Tiers Overview 2024

16 Most Useful Infrastructure as Code (IaC) Tools for 2024

Terraform Cheat Sheet - 23 Terraform CLI Commands & Examples

ebook-global-banner

ebook global banner

Product overview

How it works

Integrations

Product

Solutions

cicd

CICD

cicd-2

Infrastructure governance

This is the short description, two lines is maximum

team

Team

team-2

Collaboration

speed-2

Speed

speed-3

Developer Velocity

productivity

Productivity

productivity-2

Scaling IaC

Use cases

Financial

Technology

Industries

Request a meeting

Enjoy an action-packed day at AWS Summit London 2024, where the latest in cloud innovation comes to life.

Join us at AWS Summit  in London!

Docs

Pricing

Resources

blog

blog-2

partners

Partners

partners-2

events

Events

events-2

cheatsheets

Cheatsheets

cheatsheets-3

Ebooks & Cheatsheets

users

Users

users-2

Case studies

chat

Chat

chat-2

User reviews

388-terraform-yamlencode

terraform yamlencode

Using Terraform YAML Functions: Yamldecode & Yamlencode

About Us

Contact Us

About

cheatsheets-2

Test

See more

Test webinar name

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Nowadays, the need for robust security measures cannot be overstated. As more and more organizations move their workloads to the cloud, the risk of data breaches and unauthorized access has grown exponentially. This is where Vault comes to the rescue. 
In this post, we will explore how easy it is to configure and use Vault with Spacelift, and how you can take advantage of it inside your workflow.

Vault has become the de-facto standard for managing and securing data like API keys, tokens, and passwords. It is designed to address the challenges associated with securing secrets in a modern, dynamic environment. 
In mid-sized and big organizations, secrets are usually spread across multiple platforms and tools, making it really difficult to manage and — most important — secure them. Vault’s capabilities solve this problem of “secret sprawl.” 
It provides a comprehensive solution for managing secrets, but it requires careful configuration and management in order to harness all the features it offers. It does come with risks, and misuse could lead to exposure of sensitive data.

Some of Vault’s key features are:
<ol>
<li style="font-weight: 400;" aria-level="1">Secure Secret Storage → Gives you the ability to store secrets in a secure way. All secrets are encrypted before they are written to persistent storage, and even if an attacker gains access to the raw storage, it isn’t enough to crack them.</li>
<li aria-level="1">Dynamic Secrets → Unlike a basic secret in which you have to put data into the store yourself, dynamic secrets can be generated on-demand for some systems. For example, if your application needs access to AWS service, it can ask Vault for AWS credentials, and Vault will generate credentials granting access to that service, which it will revoke after the TTL expires.</li>
<li aria-level="1">Revocation → You can revoke not only a single instance of a secret but a tree of secrets (e.g. all secrets of a particular type)</li>
<li aria-level="1">Secret Renewal → Secrets in Vault have a lease associated with them, at the end of which, Vault will automatically revoke those secrets. These leases can be renewed via the API.</li>
<li aria-level="1">Data Encryption → Data can be encrypted and decrypted by Vault without storing it. </li>
<li aria-level="1">Integrations → Vault integrates with many services, including AWS, Azure, GCP, Spacelift, Kubernetes, and even databases.</li>
</ol>

<h3>Installing Vault on the Spacelift Runner</h3>
In this section, we will explore two ways of installing Vault on the Spacelift Runner. This is useful if you want to use Vault from the CLI in your Spacelift workflow. You can either install it:
<ul>
<li style="font-weight: 400;" aria-level="1">during a before_init hook</li>
<li style="font-weight: 400;" aria-level="1">by building a custom Docker image</li>
</ul>
To install Vault in a before_init hook, paste the following code:

It should look similar to this:

install-vault-in-a-before_init-hook

install Vault in a before_init hook

To create a <a href="https://spacelift.io/blog/dockerfile" target="_blank" rel="noopener">DockerFile</a> with Vault installed, use the following repository, or build your own image based on this DockerFile:

If you want to use the docker image, you have to build it and push it to a <a href="https://docs.spacelift.io/integrations/docker.html#allowed-registries-on-public-worker-pools" target="_blank" rel="noopener">supported registry</a>. You can also use the image I’ve already pushed in Docker hub: flaviuscdinu93/spacelift_vault:amd.
After that, add it to your stack, as in the following screenshot:

create-a-dockerfile-with-vault-installed

create a DockerFile with Vault installed

<ul>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Improve Your Infrastructure as Code using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Improve Your Infrastructure as Code using Terraform</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener">Infrastructure Problems that Spacelift Solves</a></li>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
</ul>

<h3>Dynamic Credentials via OIDC</h3>
Spacelift allows you to set up dynamic credentials for Vault via OIDC. Configuring access involves establishing a role within Vault that designates which Spacelift runs are permitted to access specific Vault secrets. This procedure can be executed using either the Vault CLI or Terraform.
In this example, we will use the cli. First, we need to authenticate to Vault. To do that from the CLI we need to:

Provide your token when asked for it. You will receive the following message if the authentication is successful:
&nbsp;
“Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run &#8220;vault login&#8221;
again. Future Vault requests will automatically use this token.”
&nbsp;
As I am using HCP vault, I also have to export a vault namespace and in my case this will be admin; the default value is root:

Now, we need to enable the JWT authentication method as shown below:

Next, you need to add the configuration for your Spacelift account as an identity provider. This will take two parameters:
<ul>
<li style="font-weight: 400;" aria-level="1">bound_issuer → the URL of your Spacelift account</li>
<li style="font-weight: 400;" aria-level="1">oidc_discovery_url → the URL of the OIDC discovery endpoint for your Spacelift account (it is usually the same as the URL of your Spacelift account)</li>
</ul>
After you get those values, run this command:

In the next step, you will create a policy used to define what Spacelift runs can access which vault secrets — something similar to this:

You will need to change the path to suit your needs. In the above example, I am giving read access to all the secrets in the secrets path.
As with all policies, you need to have a role to which this is attached. To do that, run the following command:

Above, you should only change the bound_audiences and the bound_claims parameters: 
<ul>
<li style="font-weight: 400;" aria-level="1">bound_audiences should be your spacelift account without https:// </li>
<li style="font-weight: 400;" aria-level="1">bound_claims can be a numer of things, but I believe you are going to use the spaceId parameter the most. For other options, you can check out all supported claims in Spacelift <a href="https://docs.spacelift.io/integrations/cloud-providers/oidc/" target="_blank" rel="noopener">here</a>.</li>
</ul>
<h3>Using Vault from the Terraform Provider</h3>
If the Vault instance is public, it is fairly easy to connect to it using the Terraform provider. If you have set up dynamic credentials following the example above, the only thing you need to do is set up the TERRAFORM_VAULT_AUTH_JWT environment variable to the Spacelift OIDC token either directly on the stack or in the context.
Vault offers other authentication methods, which you can find in <a href="https://registry.terraform.io/providers/hashicorp/vault/latest/docs" target="_blank" rel="nofollow noopener">Terraform’s documentation</a>.
The provider configuration should be similar to this:

To set up the TERRAFORM_VAULT_AUTH_JWT environment variable, simply leverage a before_init hook like this:

using-vault-from-the-terraform-provider

Using Vault from the Terraform Provider

<h3>Vault examples with Spacelift</h3>
In my vault account, I have created a secret that holds a vpc cidr_block in the secrets/vpc path. I want to use this vpc_cidr block in my Terraform configuration. One of the examples created leverages the vault provider,  and the other will use the CLI to get that value and save it to an environment variable.
The example code can be found <a href="https://github.com/saturnhead/blog-examples/tree/main/vault" target="_blank" rel="nofollow noopener">here</a>. It will simply create a vpc and a subnet in AWS.
The tf_vault folder will leverage the vault provider to get the value, so in this case, the only thing we need to do in a before_init hook is to export the TERRAFORM_VAULT_AUTH_JWT environment variable to use the SPACELIFT_OIDC_TOKEN, as we’ve already set up dynamic credentials via OIDC. 
For this to work for you, ensure you change the vault_address terraform variable to your vault account and set up your AWS credentials. After that, you are good to go:

vault-examples-with-spacelift

Vault examples with Spacelift

Note the values for the cidr_blocks are sensitive.
It’s a little trickier to use the second approach by leveraging the CLI. In the first approach, we didn’t even need to have Vault installed on the runner image, but now we do. We also need to export a couple of other variables to ensure that everything is working properly.
The code example can be found in the tf_env_vars folder. I will be doing everything in a before_init hook to make it easier to follow. 
After the installation of Vault as shown above, you need to:

We will export the Vault server address and Vault namespace and after that, we will generate the token based on JWT to leverage the dynamic credentials we’ve set up before. 
Finally, we will get the cidr block secret from Vault and export it as a terraform environment variable.

vault-example

vault example

In the before_init hook, the whole process is similar to what you see above. 
After running a plan, the output will be:

vault-example-output

vault example output

As we’ve saved the vpc_cidr block in an environment variable, the values in the plan will not be sensitive anymore.

In this post, we’ve explored how to create an integration between Vault and Spacelift. By leveraging dynamic credentials via OIDC and using the Terraform provider directly, you won’t need to install Vault at all on your Spacelift runner. 
On the other hand, if you just need to take some values from Vault and provide them as input variables, and you don’t want to use the Terraform provider, you will need to install Vault on the runner, but your Terraform code will be simpler.
Harnessing a secure secret storage tool like Vault and incorporating it in your Spacelift workflow streamlines secret handling, reinforces security, and makes it much easier to maintain compliant infrastructure. To find out more ways Spacelift makes managing IaC easier, <a href="https://spacelift.io/schedule-demo" target="_blank" rel="noopener">schedule a demo</a> with one of our engineers or <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">start a free trial</a>.

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift is a sophisticated SaaS product for Infrastructure as Code that helps DevOps develop and deploy new infrastructures or changes quickly and with confidence.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Spacelift is a sophisticated SaaS product for Infrastructure as Code that helps DevOps develop and deploy new infrastructures or changes quickly and with confidence.

Enabling Seamless Collaboration in IaC

Getting Started with Spacelift Self-Hosted on AWS

Infrastructure Drift Detection and How to Fix It With IaC Tools

Explore how easy it is to configure and use Vault with Spacelift, and how you can take advantage of it inside your workflow.

32-spacelift-vault

Vault Integration in Spacelift

General | Spacelift Blog

General

Derek Morgan

In the world of Continuous Integration and Continuous Deployment, <a href="http://circleci.com" target="_blank" rel="nofollow noopener">CircleCI</a> and <a href="http://jenkins.io" target="_blank" rel="nofollow noopener">Jenkins</a> are two names that people mention early in the discussion. Despite negative sentiment that can surround either of them due to their age, they’re still effective tools that can help your organization deploy faster and more reliably. Both of them have many similarities and differences. 
These can be mundane, such as the placement of certain UI elements, or crucial to your business, such as the price and available plugins. Let’s take a closer look at the two of them and discuss their similarities and differences.
We will cover:
<ol>
<li><a href="https://spacelift.io/blog/circleci-vs-jenkins#what-is-jenkins">What is Jenkins</a></li>
<li><a href="https://spacelift.io/blog/circleci-vs-jenkins#what-is-circleci">What is CircleCI</a></li>
<li><a href="https://spacelift.io/blog/circleci-vs-jenkins#jenkins-vs-circleci-comparison">Jenkins and CircleCI comparison</a></li>
<li><a href="https://spacelift.io/blog/circleci-vs-jenkins#is-circleci-better-than-jenkins">Is CircleCI better than Jenkins?</a></li>
<li><a href="https://spacelift.io/blog/circleci-vs-jenkins#what-is-the-best-alternative-to-jenkins">What is the best alternative to Jenkins?</a></li>
<li><a href="https://spacelift.io/blog/circleci-vs-jenkins#circleci-vs-jenkins-table-comparison">Table comparison</a></li>
</ol>

Jenkins is an open-source automation server that is used for the continuous integration and continuous delivery of software. It automates various tasks such as building, testing, and deploying applications. 
It is easily extendable due to its vast ecosystem of plugins, making it easy to integrate to version control systems like Git, build tools like Maven/Gradle, and deployment platforms like AWS and Docker. 
Jenkins has a distributed architecture, making it easy to distribute workloads across multiple machines, speeding up the execution of jobs. This feature is really neat when it comes to large projects that are heavily tested and deployed.
Jenkins has been around since 2011. Due to its age, it’s no surprise that the creators developed Jenkins as a self-hosted application. AWS EC2 wasn’t even available until 2008, and by then, Jenkins’ predecessor, “Hudson,” was already well-established and deployed widely. Other services, such as JFrog, that provide a managed Jenkins environment, you’ll usually find Jenkins installed on bare metal servers, VMs, cloud instances, and containers. The setup is fairly straightforward for all these methods, but the management varies in complexity based on the deployment size. 
A single Jenkins host is effortless to manage, but things can get unwieldy once you start creeping into double and triple digits. You can see the complexity of managing Jenkins at scale <a href="https://www.jenkins.io/doc/book/scaling/architecting-for-scale/" target="_blank" rel="nofollow noopener">in their docs</a>.

Unlike Jenkins, CircleCI is a Software-as-a-Service-first solution for continuous integration and delivery. They are similar in terms of capabilities. There is a self-managed version, but since the considerations involved with that are similar to the ones involved with Jenkins, we’ll stay focused on the SaaS version. 
Like any SaaS product, infrastructure, scaling, and resilience management concerns are practically non-existent. CircleCI can experience outages; they <a href="https://circleci.com/blog/an-update-on-circleci-reliability" target="_blank" rel="nofollow noopener">have promised to reduce those</a>. 
Overall, if you aren’t concerned with being responsible for the number of 9s of uptime your deployment solution has, CircleCI is definitely the more straightforward solution to manage. Of course, if there are downtime issues for Jenkins, as it is self-hosted, you will be the one that has to solve them without any help from an external team.

Let’s take a closer look at the two tools.
<h3>Configuration Files</h3>
Self-hosted vs. SaaS is a complicated debate, but it has pretty clear pros and cons. The real concerns between these solutions involve the actual usage of the products. Overall, you’ll get practitioners that love or hate both or either solution. Overall, sentiment leans towards CircleCI due to its YAML-based configuration file. Jenkins’ Groovy syntax is very powerful, but it’s also proprietary. You probably won’t find too many native methods for Groovy in your favorite programming language. Let’s take a look at each. 
With Jenkins you can configure both declarative and scripted pipeline, the difference between them being the fact that in declarative pipelines (those that start with the word pipeline), you cannot inject code directly, in the scripted ones (those that start with the word node), you can do that.
Even though, declarative pipelines may seem limiting, they are actually not, as with them, you can access environment variables, add conditional logic to the statements, and you can easily add error handling and logging.
Let’s take a look at a Jenkins declarative pipeline.
<h4>Jenkins</h4>

The content of the two files is similar despite the syntax differences. A significant benefit of using YAML is the native tools available in many programming languages if you need to generate these files dynamically. YAML will be more verbose, but Groovy can get a little complicated. Many engineers lament having to learn Groovy, but those who know it can create very powerful deployment scripts. 
Overall, either one should be accessible to a seasoned engineer and should not be a significant hindrance. 
<h3>User Interface</h3>
The two user interfaces are similar. Jenkins feels dated and very “Web 1”, even though there are attempts to make it look better (e.g., BlueOcean plugin), while CircleCI has a more modern feel. 
The functional differences are there, sometimes more subtle than others, but they definitely exist. Developer sentiment between the two will typically lean toward CircleCI unless the developer has extensive experience with Jenkins. 
Overall, except when monitoring your deployments, you probably won’t visit the UI as much as the command line.
<h4>Jenkins</h4>

jenkins-ui

Jenkins UI

circleci-ui

CircleCI UI

<h3>Plugins and Extensibility</h3>
CircleCI and Jenkins both support most of the tools you’ll use in a typical deployment. Git providers, cloud providers, <a href="https://spacelift.io/blog/container-orchestration-tools" target="_blank" rel="noopener">container orchestration systems</a>, build systems, etc. The major difference is that Jenkins supports nearly anything you can imagine, while CircleCI is more focused. 
Since Jenkins has direct access to SSH, if you want to run some complicated bash script to do some crazy hack to keep a server going, you got it! For this reason, I typically call Jenkins the “sledgehammer” of CI tools.
Unfortunately, as said in the comics, “With great power comes great responsibility.” This power with Jenkins can lead to massive security and reliability concerns. If an engineer has admin access to Jenkins, they can potentially do anything, including take down your production systems. 
This is why it is really important to ensure proper access management, as the same engineer can even destroy the whole Jenkins installation. CircleCI is also very powerful, but its guardrails are a little easier to control because you don’t have nearly the security footprint to manage. 
Since the SaaS version is separated from your production environment, building the controls necessary to prevent disaster is much easier. 
<h3>Pricing</h3>
The major pro to Jenkins is the price. The software is completely free! Being free is obviously a huge selling point to the CFOs in the room. If you aren’t managing it at scale, the CapEx and OpEx involved are surprisingly minimal. It can be run as a container on a Kubernetes cluster and is largely hands-off. 
Overall, Jenkins is a winner on price, but when a company introduces massive scale and resilience concerns, its self-hosted deployment method and cost can quickly lose their luster. 
CircleCI, on the other hand, offers four levels of pricing which can be checked out <a href="https://circleci.com/pricing/?utm_source=google&amp;utm_medium=sem&amp;utm_campaign=sem-google-dg--emea-en-brandAuth-tROAS-auth-brand&amp;utm_term=g_e-circleci%20pricing_c__rsa2_20210604&amp;utm_content=sem-google-dg--emea-en-brandAuth-tROAS-auth-brand_keyword-text_rsa-circleCIPricing_mixed-&amp;gclid=Cj0KCQjwho-lBhC_ARIsAMpgMoe8eGENOHQM-N5CX7KGZxR21fU3H2uAdOqihBEr5rq9Zsyop20lumUaAlppEALw_wcB" target="_blank" rel="nofollow noopener">here</a>. Needless to say, CircleCI has a free tier that gives you five users and up to 6000 build minutes.
<h3>Infrastructure Deployments</h3>
Jenkins and CircleCI are excellent deployment tools for application code, but if you want to deploy infrastructure, they aren’t necessarily the best option. When you need self-service environments, drift detection, Terraform state management<a href="https://docs.spacelift.io/vendors/terraform/state-management">,</a> and more, these two options don’t quite cut it. 
Sure, you can bolt together products and get them close, but you spend a lot of engineering time and risk a much larger footprint vs. a solution specifically built for this purpose. <a href="https://spacelift.io/" target="_blank" rel="noopener">Spacelift</a> is that solution.

<ul>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener">Infrastructure Problems that Spacelift Solves</a></li>
</ul>

When comparing software products that do the same thing, the answer to this question is “It depends”. 
If you are searching for a SaaS product that has an easy setup and configuration, uses a modern language (YAML) for declaring the pipeline workflows, and also offers a strong integration ecosystem, then CircleCI will be the winner for you. 
On the other hand, Jenkins will win if you want something free (pay just for the infrastructure underneath it) that is highly customizable, offers an extensive plugin ecosystem, and is 100% self-managed.
The choice between CircleCI and Jenkins will also depend on the tool familiarity, the specific needs of your project, the level of control you need, and of course, the infrastructure preferences. Whenever you are choosing a software tool, it is best to evaluate by doing trials and then deciding which one aligns better with your needs.

Right now, there are <a href="https://spacelift.io/blog/jenkins-alternatives" target="_blank" rel="noopener">many popular alternatives to Jenkins</a>, and selecting the best one would be really hard. 
Apart from CircleCI, good alternatives are GitHub Actions, GitLab CI/CD, TravisCI, and Azure Pipelines.

Below you can find a comparison table with the key features.

circleci-vs-jenkins-table-comparison

circleci vs jenkins table comparison

Jenkins and CircleCI are powerful CI/CD tools that offer distinct advantages when it comes to building your workflow. They excel at the application level, but when you need to deploy infrastructure-as-code at scale, having a tool that acts less like a sledgehammer and more like a scalpel will be the best fit. 
With Spacelift, you have all the tools necessary to deploy your IaC without the bloat of app deployment build tools. So check it out today for a <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">free trial</a> and a <a href="https://spacelift.io/schedule-demo" target="_blank" rel="noopener">custom-tailored personal demo</a>!

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

How Spacelift Can Improve Your Infrastructure as Code

Terraform with Jenkins - How to Manage Workflows

Jenkins and CircleCI are both powerful CI/CD tools with many different features - take a look at their similarities and differences.

227-circle-ci-vs-jenkins

CircleCI vs. Jenkins

CircleCI vs. Jenkins - CI/CD Tools Comparison

Infrastructure as Code (IaC) using Terraform

Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Terraform

To embrace the principles of Infrastructure as Code (IaC), Terraform has emerged as a powerful tool for managing and provisioning cloud infrastructure resources across multiple platforms. By treating infrastructure as code, Terraform aligns infrastructure management with software development best practices, driving efficiency and reliability in modern cloud environments.
In this article, we will cover:
<ol>
<li><a href="https://spacelift.io/blog/terraform-infrastructure-as-code#what-is-infrastructure-as-code-with-terraform">What is Infrastructure as Code with Terraform?</a></li>
<li><a href="https://spacelift.io/blog/terraform-infrastructure-as-code#clickops-vs-iac">ClickOps vs. IaC</a></li>
<li><a href="https://spacelift.io/blog/terraform-infrastructure-as-code#key-terraform-components">Key Terraform components</a></li>
<li><a href="https://spacelift.io/blog/terraform-infrastructure-as-code#what-is-the-structure-of-an-iac-terraform-project">What is the structure of an Iac Terraform project?</a></li>
<li><a href="https://spacelift.io/blog/terraform-infrastructure-as-code#terraform-workflow-for-infrastructure-as-">What are the benefits of using Terraform for IaC?</a></li>
<li><a href="https://spacelift.io/blog/terraform-infrastructure-as-code#what-are-the-disadvantages-of-using-terraform-for-iac">What are the disadvantages of using Terraform for IaC?</a></li>
<li><a href="https://spacelift.io/blog/terraform-infrastructure-as-code#example-using-terraform-for-kubernetes-deployment-on-azure">Example: Using Terraform for Kubernetes deployment on Azure</a></li>
<li><a href="https://spacelift.io/blog/terraform-infrastructure-as-code#enhance-your-iac-workflow-with-spacelift">Enhancing your IaC workflow with Spacelift</a></li>
</ol>

Infrastructure as Code (IaC) is the process that enables engineers to define, configure, and manage Infrastructure through code. This approach treats underlying infrastructure components, such as networks, virtual machines, storage, databases, and others, in the same way developers treat their application code. 
When it comes to managing Cloud resources, Terraform is the de-facto standard. <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> is an Infrastructure as Code tool that allows engineers to define their infrastructure resources as code, using a declarative programming language called Hashicorp Configuration Language (HCL). It is provider agnostic, meaning that it supports many Cloud providers (AWS, Azure, GCP, OCI, and many more), and the language in which you are writing the code doesn’t change from one provider to another.
By using Terraform, you are not limited to Cloud providers. You can write Terraform automations for Kubernetes, Helm, Artifactory, Aviatrix, and Spacelift, to name a few. You can even define your own provider → As long as your product exposes an API, a Terraform provider can be built on top of it.
Read more: <a href="https://spacelift.io/blog/terraform-use-cases" target="_blank" rel="noopener">Terraform Use Cases for Your Infrastructure as Code</a>

ClickOps is the term used to describe the manual management of IT infrastructure from the UI, by clicking into the portal to achieve a desired behavior (create/edit/delete resources). One may argue that this process is enough for a small architecture that needs to be deployed, but there are actually some big issues with starting like this: you cannot easily scale and replicate your configuration. 
Imagine you are creating an EC2 Instance inside AWS. By using ClickOps, you are going to create it way faster than you would normally do it through Infrastructure as Code. But what happens if you need to create 10 EC2 instances? What about 100 EC2 instances? 
Let’s suppose it takes an engineer approximately two minutes to create an EC2 through the portal. For 100 instances, this will take a little over three hours, and maybe some can be ok with that. These 100 instances will also need to reside in a network. They will require security groups, maybe some EBS storage, and other things that will again take a lot of time to configure. 
Doing this manually is very error-prone, as our attention span cannot keep up with the large number of things that we have to do. By using Terraform, you can easily define all of these components as code, validate the code, plan to see what is going to happen, and, in the end, deploy all resources in one go. Apart from that, you can easily <a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">scale</a> and replicate your configuration without spending too much time.

Terraform is stateful, tracking the state of the infrastructure by comparing the current defined IaC configuration and a state file it generates after a Terraform run. This introduces a layer of complexity because you need to manage this state file.
If organizations use a combination of IaC and ClickOps, they will introduce <a href="https://spacelift.io/blog/terraform-drift-detection" target="_blank" rel="noopener">Drift</a> and sometimes even break their infrastructure resources. So if you are getting into IaC, and you should really do, forget about ClickOps.
<h3>Providers</h3>
In essence, Terraform providers function as plugins that enable Terraform to communicate with specific infrastructure resources. These providers serve as a bridge between Terraform and the underlying infrastructure, converting Terraform configurations into relevant API calls and permitting Terraform to handle resources across numerous environments. 
Example provider:

Take a look at <a href="https://spacelift.io/blog/terraform-providers" target="_blank" rel="noopener">Terraform Providers Overview</a>.
<h3>Resources</h3>
In Terraform, resources represent the infrastructure elements that can be managed, such as virtual machines, virtual networks, DNS entries, pods, and more. 
Each resource is identified by a specific type, like &#8220;aws_instance&#8221;, or &#8220;kubernetes_pod,&#8221; and possesses a range of configurable attributes, such as instance size or type. They are the building blocks of Terraform, and every configuration that will create a piece of the infrastructure has to contain a resource.
Example resource:

<h3>Variables</h3>
<a href="https://spacelift.io/blog/how-to-use-terraform-variables" target="_blank" rel="noopener">Terraform variables</a> are similar to variables in any other programming language. They are used to organize your code better, make it easier to change its behavior, and make your configuration reusable. In Terraform, variables can have the following types: string, number, bool, list, set, map, object, and null. 
You cannot use variables to hold expression values built from resources or data sources. For that, you should use a local variable.
Example Variable:

<h3>Outputs</h3>
An output serves as a convenient method for displaying the value of a particular data source, resource, local, or variable once Terraform has completed implementing infrastructure modifications. They can also be used to expose attributes from a <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform module</a>.
Example Output:

<h3>DataSources</h3>
A data source is a Terraform object that retrieves data from an external source and can be used in resources as arguments when they are created or updated, or you can use them in locals to manipulate the data you receive.
Example Datasource:

<h3>Locals</h3>
In Terraform, a local variable assigns a name to an expression and makes it easy for the user to utilize it. Typically, in any programming language, when you have a complex expression, you don’t want to write it a million times throughout your code. You are usually defining a variable that holds it or a function that implements it. 
Local variables work similarly to that, and they can be used in conjunction with resource and datasource attributes to build complex expressions.
Example Local:

<h3>Provisioners</h3>
Terraform provisioners exist inside of a resource, and they are used to either run a local command or a remote command or to copy a file from your local environment to a remote virtual machine. 
They are considered a last resort option due to the fact they are not a part of Terraform’s declarative model. You should typically use cloud-init to run different scripts on your vm during the bootstrap phase, or if you can, use a configuration management tool like <a href="https://spacelift.io/blog/ansible-tutorial" target="_blank" rel="noopener">Ansible</a> for this.
Example Provisioner:

Learn more about <a href="https://spacelift.io/blog/terraform-provisioners" target="_blank" rel="noopener">Terraform Provisioners and why you should avoid them</a>.
Note: New versions of Terraform will be placed under the BUSL license, but everything created before version 1.5.x stays open-source. <a href="https://opentofu.org/" target="_blank" rel="nofollow noopener">OpenTofu</a> is an open-source version of Terraform that will expand on Terraform’s existing concepts and offerings. It is a viable alternative to HashiCorp’s Terraform, being forked from Terraform version 1.5.6.

<ul>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener">How to Improve Your Infrastructure as Code using Terraform</a></li>
</ul>

A basic structure of a Terraform Infrastructure as Code project, contains the following files:

<ul>
<li>In main.tf typically contains the core resource declarations and providers. This is where you would define your EC2 instance configurations and how to authenticate to the AWS provider.</li>
<li>The variables.tf file defines the input variables that allow engineers to customize the project. They are referenced in the main.tf and can easily change the behavior of a particular resource, making it easy to reuse the automations.</li>
<li>In the outputs.tf file, engineers define the outputs that are returned to the console after a terraform apply. As a best practice, they shouldn’t return all the data from a resource, only the important fields that are of interest.</li>
</ul>
Documentation is really important, so having a README.md file inside your Terraform repository that explains how to use the automation (including descriptions of variables and outputs) really helps in understanding what has been implemented. You can leverage <a href="https://terraform-docs.io/" target="_blank" rel="noopener">tfdocs</a> to easily generate the descriptions of variables and outputs.
This is just a basic structure, but it can be customized depending on the complexity of the automation and, of course, the requirements of the organizations.

The core Terraform workflow is designed to provide a consistent and repeatable process for managing your IaC. This workflow contains the following steps:
<ol>
<li style="font-weight: 400;" aria-level="1">Write — Author infrastructure as code by writing Terraform configuration files that define the desired state of your infrastructure resources across various cloud providers or on-premises environments.</li>
<li style="font-weight: 400;" aria-level="1">Init — Initialize the working directory by downloading the necessary provider plugins specified in the configuration files. This is done using the <code>terraform init</code> command.</li>
<li style="font-weight: 400;" aria-level="1">Plan — Preview the changes that Terraform will make to your infrastructure based on the current configuration files and the real infrastructure state stored in Terraform&#8217;s state file. The terraform plan command shows you a preview of the changes without actually applying them.</li>
<li style="font-weight: 400;" aria-level="1">Apply — Execute the plan and provision or modify the infrastructure resources according to the configuration files. The terraform apply command applies the changes to your infrastructure after prompting you to confirm the plan.</li>
</ol>

The benefits of managing Infrastructure as Code with Terraform include:
<h3>1. Consistency</h3>
All the resources presented above are the building blocks for creating Terraform automations. The declarative syntax that Terraform offers is easy to understand and use and enables engineers to get up to speed quickly with these concepts.
<h3>2. Efficiency</h3>
Terraform’s init/plan/apply workflow helps prevent unintended changes to your infrastructure by previewing changes before they are actually made. The initialization part also ensures the required providers are downloaded, so you don’t need to push them to your VCS repository to be able to take advantage of them. 
<h3>3. Community</h3>
Nowadays, more and more people are using Terraform. The community is large and very active, and many resources are available, like documentation and tutorials, that can be easily leveraged.
<h3>4. Repeatability</h3>
The code you build with Terraform can be packaged as a module and shared across your organization easily. There are many other features available that keep your code DRY, such as for_each, count, functions, ternary operators, loops, and <a href="https://spacelift.io/blog/terraform-dynamic-blocks" target="_blank" rel="noopener">dynamic blocks</a>. 
<h3>5. Support for multiple cloud providers</h3>
Because Terraform is cloud agnostic, you can use it to build multi-cloud automations without using multiple tools together

While Terraform offers significant advantages for IaC, acknowledging these limitations is also important:
<ul>
<li style="font-weight: 400;" aria-level="1">Learning curve — Terraform has a steep learning curve, especially for beginners or those new to IaC concepts. Understanding the declarative syntax, writing modules, and managing state files can be challenging initially.</li>
<li style="font-weight: 400;" aria-level="1">State file management — Terraform&#8217;s state file is crucial for tracking infrastructure state. However, managing it in a team setting or with remote backends requires careful consideration to avoid conflicts and potential data loss, which can lead to unintended changes or resource recreation.</li>
<li aria-level="1">Scaling challenges — Managing multiple environments and scaling can lead to complications in more complex infrastructures. As you manage more resources in Terraform, issues with one resource can potentially affect many others.</li>
</ul>

Let&#8217;s now see a practical example of configuring IaC with Terraform. 
The example code can be found <a href="https://github.com/flavius-dinu/terraform-az-free-aks" target="_blank" rel="nofollow noopener">here</a>.
We are using two resources:
<ul>
<li style="font-weight: 400;" aria-level="1">azurerm_resource_group → used to create resource groups inside of Azure</li>
<li style="font-weight: 400;" aria-level="1">azurerm_kubernetes_cluster → used to create the Kubernetes cluster inside of Azure</li>
</ul>
For this example, the cluster management will always be free, you will need to pay only for the underlying nodes of the cluster.
On both resources, we are using for_each to create the number of resources of that particular type we want.

If you look at the above resources, you will see how the link between them is created on the Kubernetes one, specifically at the location and resource_group_name parameters. This way, we ensure the resource group is created first, and we access its location and name attributed inside of the Kubernetes Cluster. 
Thus, the cluster and the resource group in which it will be created will reside in the same location.
To declare the variables, we are using map(object) types to take advantage of the full capabilities of the for_each, and we are also ensuring optional values to make the code easier to use.

To keep it simple, we are providing the values of these variables in the default block, but you can use <a href="https://spacelift.io/blog/terraform-tfvars" target="_blank" rel="noopener">terraform.tfvars</a> or a *.auto.tfvars file, or environment variables to pass these values. 
You can even use a local variable to specify the values, or if you want to take it to the next level, you can use a local variable that reads the contents of a YAML file with values by using the file and yamldecode functions.
Right now, with the default values, this looks like this:

For this configuration, we have declared two outputs: one will show a map containing name and location pairs for the resource groups, and the other will have some details related to the Kubernetes cluster in the following format name =&gt; id, fqdn.

To run this code, we have initialized the working directory using <code>terraform init</code>.

After that, we can apply the code using the <code>terraform apply</code> command.

You can see the cluster now in the Azure Portal:

terraform-iac-azure-portal

terraform iac azure portal

Terraform is really powerful, but to achieve an end-to-end secure Gitops approach, you need to use a product that can run your Terraform workflows. 
Enter Spacelift. Not only <a href="https://docs.spacelift.io/" target="_blank" rel="noopener">Spacelift</a> takes care of your Terraform workflows, but it can also help build workflows for Kubernetes, Pulumi, and CloudFormation. Spacelift is GitOps Native, and by also taking advantage of stack dependencies, you can build really sophisticated workflows.
Your workflows will most likely require policies to ensure the necessary guardrails for your infrastructure. Apart from that, taking advantage of notifications when something goes wrong is really important, so taking advantage of Spacelift’s built-in features will totally help.
Taking advantage of integrations with major cloud providers avoids using static credentials, which can be easily replicated if you are not careful with them.
<a href="https://spacelift.io/blog/integrating-security-tools-with-spacelift" target="_blank" rel="noopener">Integrating security tools</a> in your workflows can be done easily by using Custom Inputs. With this feature, not only are you integrating the tools, but you can easily run policies on it to ensure engineers are not introducing vulnerabilities with their code.
If Terraform modules make your code DRY, check out <a href="https://docs.spacelift.io/concepts/blueprint/" target="_blank" rel="noopener">Spacelift’s Blueprints feature</a>, which really takes reusability to the next level.
Let’s reuse the above example and create a stack for it in Spacelift. We will also apply a policy that ensures that people are not changing the size of the VM. I would suggest creating your own repository that holds the above code to integrate everything.
First, go to stacks and select create a new stack. Add a name for the stack, and optionally you can add labels and a description.

terraform-iac-create-new-stack-spacelift

terraform iac create new stack spacelift

How to manage Infrastructure as Code (IaC) using Terraform? Here's an overview, benefits, and different approaches.

Flavius Dinu

Highlights

Education

License and certifications

Vault Integration in Spacelift

Get the newsletter

Recent posts

CircleCI vs. Jenkins – CI/CD Tools Comparison

Managing Infrastructure as Code (IaC) With Terraform

Terraform Cloud/Enterprise Pricing – Tiers Overview 2024

How to Migrate From Terraform Cloud to Spacelift

Integrating Security Tools With Spacelift Using Custom Inputs

Why DevOps Engineers Recommend Spacelift

InfraCost – How to Estimate Cloud Costs with Terraform

16 Most Useful Infrastructure as Code (IaC) Tools for 2024

Share your data and download the guide