Serhii Vasylenko — Developer Experience Engineer

Serhii Vasylenko

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Terraform

Terraform by itself automates many tasks: it creates, changes, and versions your cloud resources. Although many teams run Terraform locally (sometimes with wrapper scripts), running Terraform in CI/CD can boost your organization&#8217;s performance and ensure consistent deployments. 
In this article, you will see a review of the different approaches to integrating Terraform into generic deployment pipelines. Let’s start with the basics.

Storing Terraform code in the same repository as the application code or maintaining a separate repository for the infrastructure? This question has no strict and clear answer, but here are some insights that may help you decide:
<ul>
<li style="font-weight: 400;">The <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> and application code coupled together represent one unit, so it&#8217;s simple to maintain by one team;</li>
<li style="font-weight: 400;">Conversely, if you have a dedicated team that manages infrastructure (e.g., platform team), a separate repository for infrastructure is more convenient because it&#8217;s a standalone project in that case.</li>
<li style="font-weight: 400;">When infrastructure code is stored with the application, sometimes you have to deal with additional rules for the pipeline to separate triggers for these code parts. But sometimes (e.g., serverless apps) changes to either part (app/infra) should trigger the deployment.</li>
</ul>
There is no right or wrong approach, but whichever you choose, remember to follow the Don’t Repeat Yourself (DRY) principle: make the infrastructure code modular by logically grouping resources into higher abstractions and reusing these modules.

Running Terraform locally generally means that all dependencies are already in-place: you have the Terraform installed and present in the user&#8217;s <code>PATH</code> and providers already stored in the <code>.terraform</code> directory. 
But, when you shift Terraform runs from your local machine to stateless pipelines, this is not the case. However, you can still have a pre-built environment — this will speed up the pipeline execution and provide control over the process.
Docker image with a Terraform binary is one of the popular solutions that address this. Once created, you can execute Terraform within a container context with configuration files mounted as a Docker volume.
You can use the official <a href="https://hub.docker.com/r/hashicorp/terraform/" target="_blank" rel="nofollow noopener">image from Hashicorp</a>, but sometimes it&#8217;s wise to maintain your own Docker images with additional tools you may need. For instance, you can bake the <code>tfsec</code> tool into the image to use for security inspection and have it ready inside the Docker container without the need to install it every single time.
Here is an example of a Dockerfile that builds an image with a custom Terraform version (you can override it as a build argument) and a <code>tfsec</code> tool. This example also shows how to verify the installed Terraform binary to make sure it&#8217;s signed by HashiCorp before we run it.

But the main functionality of Terraform is delivered by provider plugins. It takes time to download the provider: for example, the AWS provider is about 250MB, and in a large scale, with hundreds of Terraform runs per day, this makes a difference.
There are two common ways to deal with it: either use a shared cache available to your pipeline workloads or bake provider binaries into the runtime environment (i.e., Docker image).
The critical element of both approaches is the configuration of the plugin cache directory path. By default, Terraform looks for plugins and downloads them in the <code>.terraform</code> directory, which is local to the main project directory. But you can override this by leveraging the <code>TF_PLUGIN_CACHE_DIR</code> environment variable to do that.
If supported by your CI/CD tool, the shared cache can significantly reduce the operational burden because all your pipeline runtime environments can use it to get the needed provider versions.
That way, all you have to do is to maintain the provider versions in the shared cache and instruct Terraform to use it:
<ul>
<li style="font-weight: 400;">Mount the cache directory to the pipeline runtime (i.e., docker container) and specify its internal path.</li>
<li style="font-weight: 400;">Set the value of the <code>TF_PLUGIN_CACHE_DIR</code> environment variable accordingly.</li>
</ul>
On the other hand, you can bake the provider binaries into the Docker image and inject the value for the <code>TF_PLUGIN_CACHE_DIR</code> environment variable right into the Dockerfile. This approach takes more operational effort but makes the Terraform environment self-sufficient and stateless. It also allows you to set strict boundaries around permitted provider versions as a security measure.

Now, let&#8217;s review the ways to automate planning and applying of changes. Although <code>terraform apply</code> can do both, it&#8217;s sometimes useful to separate these actions. 
<h3>Initialization</h3>
CI/CD pipelines generally run in stateless environments. Thus, every subsequent run of Terraform looks like a fresh start, so the project needs to be initialized before other actions can be performed.
The usage of the <code>init</code> command in CI/CD slightly differs from its common local usage:

The <code>-input=false</code> option prevents Terraform CLI from asking for user actions (it will throw an error if the input was required).
Also, there is the <code>-no-color</code> option that prevents the usage of color codes in a shell, so the output will look much cleaner if your CI/CD logging system cannot render the terminal formatting.
Another aspect of the <a href="https://spacelift.io/blog/terraform-init" target="_blank" rel="noopener">init command</a> that is useful in CI is <code>-backend-config</code>. This option allows you to override the backend configuration in your code or define it if you prefer to use partial configuration, thus creating more uniform pipelines.
For example, here’s how you can use the same code with different roles in different environments on AWS:

However, I suggest checking in <code>.terraform.lock.hcl</code> as suggested by HashiCorp (<a href="https://www.terraform.io/docs/language/dependency-lock.html" target="_blank" rel="nofollow noopener">Dependency Lock File</a>): this way you will be able to control dependencies more thoroughly, without worrying about transferring this file between build stages.
<h3>Plan</h3>
The <code>terraform plan</code> command helps you validate the changes manually. Still, there are ways to use it in automation as well.
By default, Terraform prints the plan output in a human-friendly format, but also supports machine-readable JSON. With additional command line options, you can extend your CI experience. For example, you can use validation conditions to decide whether to apply the changes automatically; or, you can parse the plan details and integrate the summary into a Pull Request description. Let’s review a simple example that illustrates this. 
First, save the plan output to the file:

The main point here is the <code>-out</code> option — it tells Terraform to save its output into a binary plan file. We will also talk about it later in this section. 
The <code>-compact-warnings</code> option suppresses the warning-level messages produced by Terraform
Also, the <code>plan</code> command has <code>-detailed-exitcode</code> option that returns detailed exit codes when the command exits. For example, you can leverage this in a script that wraps Terraform and adds more conditional logic to its execution, because CIs will generally fail the pipeline on a command’s non-zero exit code. However, that may add complexity to the pipeline logic. So if you need to get detailed info about the plan, I suggest parsing the plan output.
Once you have a plan file, you can read it in JSON format and parse it. Here is a code snippet that illustrates this:

Another way to see the information about changes, is to run the <code>plan</code> command with <code>-json</code> option and parse its output to stdout (available starting from Terraform 1.0.5):

This technique can make your Pull Request messages more informative and improve your collaboration with teammates. (Btw. <a href="https://docs.spacelift.io/concepts/policy/git-push-policy#push-and-pull-request-events" target="_blank" rel="noopener">Spacelift</a> provides this feature out of the box).
You can write a custom script/function that sends a Pull Request comment to VCS using its API. Or, you can try the existing features of your VCS: with <a href="https://spacelift.io/blog/github-actions-terraform" target="_blank" rel="noopener">GitHub Actions</a>, you can use the <a href="https://github.com/marketplace/actions/terraform-pr-commenter" target="_blank" rel="nofollow noopener">Terraform PR Commenter</a> or similar action to achieve this; for GitLab, there is a built-in functionality that integrates plan results into the Merge Request. See more here: <a href="https://docs.gitlab.com/ee/user/infrastructure/iac/mr_integration.html" target="_blank" rel="nofollow noopener">Terraform Integration in Merge Requests</a> 
And you can find more information about the specification of the JSON output here: <a href="https://www.terraform.io/docs/internals/json-format.html" target="_blank" rel="nofollow noopener">Terraform JSON Output Format.</a>
<h3>Apply</h3>
Once the plan file is ready, and the proposed changes are expected and approved, it&#8217;s time to <code>apply</code> them.
Here is how the <code>apply</code> command may look like in automation:

The <code>plan.file</code> is the file we got from the previous plan step. 
Alternatively, you might want to omit the planning phase altogether. In that case, the following command will apply the configuration immediately, without the need for a plan:

Here, the <code>-auto-approve</code> option tells Terraform to create the plan implicitly and skip the interactive approval of that plan before applying.
Whichever way you choose, keep in mind the destructive nature of the <code>apply</code> command. Hence, the fully automated application of configuration generally works well with environments that tolerate unexpected downtimes, such as development or testing. On the other hand, plan review is recommended for production-grade environments, in which case the `apply` job is configured for a manual trigger.

<ul>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale</a></li>
<li><a href="https://spacelift.io/blog/devops-best-practices" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;16 DevOps Best Practices Every Developer Should Know&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">16 DevOps Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;5 Most Useful CI/CD Tools for DevOps in 2022&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">5 Most Useful CI/CD Tools for DevOps in 2022</a></li>
</ul>

If you run <code>init</code>, <code>plan</code>, and <code>apply</code> commands in different environments, you need to take care of some artifacts produced by Terraform:
<ul>
<li style="font-weight: 400;">The <code>.terraform</code> directory with information about <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">modules</a>, providers, and the state file (even in the case of remote state). </li>
<li style="font-weight: 400;">The <code>.terraform.lock.hcl</code> file — the dependency lock file which Terraform uses to check the integrity of provider versions used for the project. If your VCS does not track it, you&#8217;ll need to pass that file to the <code>plan</code> and <code>apply</code> commands to make them work after <code>init</code>.</li>
<li style="font-weight: 400;">The output file of the <code>plan</code> command is essential for the <code>apply</code> command, so treat it as a vital artifact. This file includes a full copy of the project configuration, the state, and variables passed to the <code>plan</code> command (if any). Therefore, mind the security precautions because sensitive information may be present there.</li>
</ul>
Make sure to pass them successively between operations: init -&gt; plan -&gt; apply.
There is one shortcut, though. You can execute the <code>init</code> and <code>plan</code> commands within the same step/stage and transfer the artifacts only once — to the <code>apply</code> execution.

Last but not least, a few words on ways to maximize the advantage of <a href="https://spacelift.io/blog/how-to-use-terraform-variables" target="_blank" rel="noopener">variables when running Terraform</a> in CI.
There are two common ways of passing values for the variables used in the configuration:
1. Using a <code>-var-file</code> option with the variable definitions file — a filename ending in <code>.tfvars</code> or <code>.tfvars.json</code>. For example:

Also, Terraform can automatically load the variables from files named exactly <code>terraform.tfvars</code> or <code>terraform.tfvars.json</code>: with that approach, you don’t need to specify the tfvar file as a command option explicitly.
2. Using environment variables with the prefix <code>TF_VAR_</code>. Implicitly, Terraform always looks for the environment variables (within its process context) with that prefix, so the same &#8220;instance_type&#8221; variables from the example above can be passed as follows:

The latter method is widely used in CI because modern CI/CD tools support the management of the environment variables for automation jobs.
Please refer to the following official documentation if you want to know more about variables: <a href="https://www.terraform.io/docs/language/values/variables.html" target="_blank" rel="nofollow noopener">Terraform Input Variables</a>.
Additionally, Terraform supports several configuration parameters in the form of environment variables. These parameters are optional; however, they can simplify the automation management and streamline its code.
<ul>
<li style="font-weight: 400;"><code>TF_INPUT</code> — when set to &#8220;false&#8221; or &#8220;0&#8221;, this tells Terraform to behave the same way as with the <code>-input=false</code> flag;</li>
<li style="font-weight: 400;"><code>TF_CLI_ARGS</code> — can contain a set of command-line options that will be passed to one or another Terraform command. Therefore, the following notation can simplify the execution of <code>apply</code> and <code>plan</code> commands by unifying their options for CI:</li>
</ul>

You can take even more advantage of this when using this variable as the environment configuration of stages or jobs in a CI/CD tool.
<ul>
<li style="font-weight: 400;"><code>TF_IN_AUTOMATION</code>  — when set to any non-empty value (e.g., &#8220;true&#8221;), Terraform stops suggesting commands run after the one you execute, hence producing less output. </li>
</ul>

Although this article provides a comprehensive overview of possible automation approaches, there is always room for some particular and unusual scenarios. Whichever use case and conditions you have, keep the following goals in mind when using Terraform in CI:
<ul>
<li style="font-weight: 400;">Ease of code management</li>
<li style="font-weight: 400;">A secure and controlled execution environment</li>
<li style="font-weight: 400;">Coherent runs of init, plan, apply phases</li>
<li style="font-weight: 400;">Leveraging of built-in Terraform capabilities </li>
</ul>
Also, there are <a href="https://spacelift.io" target="_blank" rel="noopener">CI/CD automation tools like Spacelift that solve all the issues out of the box</a>. They’re worth looking at as an alternative to a homegrown solution on top of a vanilla CI. Spacelift helps overcome common state management issues and adds several must-have features for infrastructure management. You can check it out for free, <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">by going here and creating a trial account</a>.
Was this article helpful? Do you have any questions, or would you like to share your common practices? Feel free to tell us by leaving a comment below!

Spacelift effectively manages Terraform state, more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and includes many more features.

Initialize/ plan/ apply your IaC, manage modules, state, and more. 

frame-8479

Frame 8479

How Spacelift Can Improve Your Infrastructure as Code

5 Ways to Manage Terraform at Scale - Best Practices

How to Implement GitLab CI/CD Pipeline with Terraform

Learn how to use Terraform in CI/CD. Learn different approaches to integrating Terraform into generic deployment pipelines.

Serhii Vasylenko

How to Deploy your Infrastructure in CI/CD using Terraform

Get the newsletter

No recent posts