Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

How to download and install Terraform - tutorial

Terraform

When I started learning Terraform, the first thing I struggled with was getting proper instructions on how to install Terraform on my system or any cloud server which I was using. There are many resources available online that guide you through the installation steps, and of course Terraform docs also have a very good installation guide.
In this guide you will find all the necessary steps in one place regardless of whether you are installing Terraform on Windows, Linux or macOS.

Before you start the installation process, there are some prerequisites which you will need to fulfill:
<ul>
<li style="font-weight: 400;" aria-level="1">A Windows PC if you are installing on Windows</li>
<li style="font-weight: 400;" aria-level="1">A Linux system (virtual or local) if you are installing on Linux (in this post, I am using Ubuntu 20.04 to demonstrate the steps)</li>
<li style="font-weight: 400;" aria-level="1">Some basic CLI knowledge</li>
</ul>
In this post, I will be covering the installation steps for Terraform on three different operating systems. You can follow the steps based on the system you are using. These are the high level topics I will be covering in this post: 
<ul>
<li style="font-weight: 400;" aria-level="1">Installing Terraform on Windows</li>
<li style="font-weight: 400;" aria-level="1">Installing Terraform on Linux (Ubuntu) using Package repository</li>
<li style="font-weight: 400;" aria-level="1">Installing Terraform on Linux using Zip archive file</li>
<li style="font-weight: 400;" aria-level="1">Installing Terraform on macOS</li>
</ul>

Let’s start with the basics.
<h3>Download the installation file</h3>
1. Navigate to the Terraform download page (<a href="https://www.terraform.io/downloads.html" target="_blank" rel="nofollow noopener">https://www.terraform.io/downloads.html</a>). It should list out Terraform downloads for various platforms. Navigate to the Windows section and download the respective version. For this example I am downloading V1.1.0.

image17-2

Downloading Terraform on Windows

2. It will download a zip file. Create a folder on the C drive as C:/terraform. Download the zip file in this folder. Unzip the file to extract the .exe file.

image21

Download the installation file on Windows

<h3>Update Path Environment Variable</h3>
1. Next open the Start menu and search for <a href="https://docs.microsoft.com/en-us/azure/developer/terraform/get-started-windows-bash?tabs=bash" target="_blank" rel="nofollow noopener">Environment variables</a>. Open the Environment variables settings page.

image13-3

Update Path Environment Variable

2. On the Environment variables edit page, open the Path variable as shown below:

image10-1-1

Edit environment variables

3. On the opened Path pop up, click New and add the Terraform download folder. This is the folder where the zip file was downloaded and unzipped (C:/terraform).

image4-2-2

add the Terraform download folder

4. Click OK on the above window to save the Path variable addition. If needed, restart your system to apply the variable changes.
5. Open a Command prompt and run this command to verify Terraform is installed successfully:

image20

verify if Terraform is installed successfully

<h3>Verify if the Correct Version of Terraform Is Installed</h3>
Just like below:

image5-5

correct version of Terraform

Here I am taking an example of installing on Ubuntu 20.04 to demonstrate the Linux installation. There are two paths to install on Linux. We’ll go through each path below. 
<h3>Installation Using Package Repositories on Linux (Ubuntu)</h3>
First, I will go through the steps to install Terraform using the Linux package repository. Follow these steps for Ubuntu. For other Linux versions steps are similar and will depend on the specific package repositories. Log in to the system and run these steps as root for sudo access:
1. Make sure the system is up-to-date and install these packages which are needed for further steps:

2. Next add the Hashicorp GPG key needed by the repository:

3. Then, add the official repository for HashiCorp for Linux:

4. Now that we have added the repo to the list, let&#8217;s update to add the relevant repository content:

5. Once update, run this command to install Terraform:

That should complete the installation. To verify the installation, run this and verify the output:

image12-2

verify the installation on Linux

To verify the version installed, run: <code>terraform –version</code>

<h3>Install Using Package repositories on Linux</h3>
Next, we will see the method for installing Terraform using the downloaded zip file. Log in to the system and run these commands as root:  
1. Before starting the installation, get the download link from the Terraform website. Navigate to <a href="https://www.terraform.io/downloads.html" target="_blank" rel="nofollow noopener">https://www.terraform.io/downloads.html</a> and move to the specific OS version. Here we are looking for Linux:

image3-7

download link for Linux from the Terraform site

2. Copy the link for the respective download:

image16

Copy the link for the Linux download

3. Now, log in to the Ubuntu system. Make sure Wget and Unzip are installed. If not, install the same:

4. Download the Terraform zip file by running the following command. Put the URL copied earlier in the command for download:

image8-4

wget

5. The zip file should download:

image19

The downloaded zip file

6. Unzip the zip file in the current folder:

image10-2

unzip terraform_1.1.0_linux_amd64.zip

7. The Terraform executable should get unzipped in the current folder:

image13-2

terraform executable

8. Move the Terraform executable file to /usr/bin folder:

9. That should complete the Terraform installation. Verify the installation by verifying the version:

image11-3

Verify the installation on Linux

Next, we will go through the steps to install Terraform on macOS. 
1. Log in to the system and create a folder for the Terraform executable:

2. Navigate to the new folder:

3. From the Terraform downloads website, copy the macOS Terraform zip file download link:

image5-7

copy the Mac OS terraform zip file download link

4. Download the zip file in the previously created folder:

5. Unzip the Terraform files in the current folder:

The unzipped files will be in the current folder.
6. Add Terraform to the PATH. Navigate to the home folder and execute this command to open and edit the bash profile:

Add these two lines on the opened file:

7. Execute the file for the path changes to get into effect:

8. Now that Terraform is installed, verify the Terraform version by running

This should show the Terraform version which is installed in the system.
That completes your installation of Terraform on Mac OS.
Next step? Import existing infrastructure into Terraform. See more here: <a href="https://spacelift.io/blog/importing-exisiting-infrastructure-into-terraform" target="_blank" rel="noopener">How to Import Existing Infrastructure into Terraform (Tutorial with Examples)</a>

Terraform allows you to <a href="https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Hashicorp-Terraform-Deep-Dive-with-no-Fear-Victor-Turbinsky-Texuna.pdf" target="_blank" rel="nofollow noopener">convert a whole cloud infrastructure</a> to config or code files. These files can be used to spin up the entire infrastructure quickly and easily. It is an open source tool by Hashicorp. Anyone can use it to manage a cloud infrastructure on any of the cloud providers. 
In this post, we went through the steps to install Terraform on Windows, Linux, as well as macOS. I hope this helps you with your installations. And if you’re interested in how to upgrade Terraform to the latest version, see: <a href="https://spacelift.io/blog/terraform-version-upgrade" target="_blank" rel="noopener">Upgrading Terraform to the Latest Version</a>

Spacelift effectively manages Terraform state, more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and includes many more features.

What is Terraform? A Quick Overview

How to Upgrade Terraform to the Latest Version

How to Deploy your Infrastructure in CI/CD using Terraform

Learn how to download and install Terraform on Windows, Linux, and macOS. Terraform installation made easy! See a full tutorial.

46-how_to_download_and_install_terraform

How to Download and Install Terraform on Windows, Linux & macOS

Terraform by itself automates many tasks: it creates, changes, and versions your cloud resources. Although many teams run Terraform locally (sometimes with wrapper scripts), running Terraform in CI/CD can boost your organization&#8217;s performance and ensure consistent deployments. 
In this article, you will see a review of the different approaches to integrating Terraform into generic deployment pipelines. Let’s start with the basics.

Storing Terraform code in the same repository as the application code or maintaining a separate repository for the infrastructure? This question has no strict and clear answer, but here are some insights that may help you decide:
<ul>
<li style="font-weight: 400;">The <a href="https://spacelift.io/blog/what-is-terraform">Terraform</a> and application code coupled together represent one unit, so it&#8217;s simple to maintain by one team;</li>
<li style="font-weight: 400;">Conversely, if you have a dedicated team that manages infrastructure (e.g., platform team), a separate repository for infrastructure is more convenient because it&#8217;s a standalone project in that case.</li>
<li style="font-weight: 400;">When infrastructure code is stored with the application, sometimes you have to deal with additional rules for the pipeline to separate triggers for these code parts. But sometimes (e.g., serverless apps) changes to either part (app/infra) should trigger the deployment.</li>
</ul>
There is no right or wrong approach, but whichever you choose, remember to follow the Don’t Repeat Yourself (DRY) principle: make the infrastructure code modular by logically grouping resources into higher abstractions and reusing these modules.

Running Terraform locally generally means that all dependencies are already in-place: you have the Terraform installed and present in the user&#8217;s <code>PATH</code> and providers already stored in the <code>.terraform</code> directory. 
But, when you shift Terraform runs from your local machine to stateless pipelines, this is not the case. However, you can still have a pre-built environment — this will speed up the pipeline execution and provide control over the process.
Docker image with a Terraform binary is one of the popular solutions that address this. Once created, you can execute Terraform within a container context with configuration files mounted as a Docker volume.
You can use the official <a href="https://hub.docker.com/r/hashicorp/terraform/" target="_blank" rel="nofollow noopener">image from Hashicorp</a>, but sometimes it&#8217;s wise to maintain your own Docker images with additional tools you may need. For instance, you can bake the <code>tfsec</code> tool into the image to use for security inspection and have it ready inside the Docker container without the need to install it every single time.
Here is an example of a Dockerfile that builds an image with a custom Terraform version (you can override it as a build argument) and a <code>tfsec</code> tool. This example also shows how to verify the installed Terraform binary to make sure it&#8217;s signed by HashiCorp before we run it.

But the main functionality of Terraform is delivered by provider plugins. It takes time to download the provider: for example, the AWS provider is about 250MB, and in a large scale, with hundreds of Terraform runs per day, this makes a difference.
There are two common ways to deal with it: either use a shared cache available to your pipeline workloads or bake provider binaries into the runtime environment (i.e., Docker image).
The critical element of both approaches is the configuration of the plugin cache directory path. By default, Terraform looks for plugins and downloads them in the <code>.terraform</code> directory, which is local to the main project directory. But you can override this by leveraging the <code>TF_PLUGIN_CACHE_DIR</code> environment variable to do that.
If supported by your CI/CD tool, the shared cache can significantly reduce the operational burden because all your pipeline runtime environments can use it to get the needed provider versions.
That way, all you have to do is to maintain the provider versions in the shared cache and instruct Terraform to use it:
<ul>
<li style="font-weight: 400;">Mount the cache directory to the pipeline runtime (i.e., docker container) and specify its internal path.</li>
<li style="font-weight: 400;">Set the value of the <code>TF_PLUGIN_CACHE_DIR</code> environment variable accordingly.</li>
</ul>
On the other hand, you can bake the provider binaries into the Docker image and inject the value for the <code>TF_PLUGIN_CACHE_DIR </code>environment variable right into the Dockerfile. This approach takes more operational effort but makes the Terraform environment self-sufficient and stateless. It also allows you to set strict boundaries around permitted provider versions as a security measure.

Now, let&#8217;s review the ways to automate planning and applying of changes. Although <code>terraform apply</code> can do both, it&#8217;s sometimes useful to separate these actions. 
<h3>Initialization</h3>
CI/CD pipelines generally run in stateless environments. Thus, every subsequent run of Terraform looks like a fresh start, so the project needs to be initialized before other actions can be performed.
The usage of the <code>init</code> command in CI/CD slightly differs from its common local usage:

The <code>-input=false</code> option prevents Terraform CLI from asking for user actions (it will throw an error if the input was required).
Also, there is the <code>-no-color</code> option that prevents the usage of color codes in a shell, so the output will look much cleaner if your CI/CD logging system cannot render the terminal formatting.
Another aspect of the init command that is useful in CI is <code>-backend-config</code>. This option allows you to override the backend configuration in your code or define it if you prefer to use partial configuration, thus creating more uniform pipelines.
For example, here’s how you can use the same code with different roles in different environments on AWS:

However, I suggest checking in <code>.terraform.lock.hcl</code> as suggested by HashiCorp (<a href="https://www.terraform.io/docs/language/dependency-lock.html" target="_blank" rel="nofollow noopener">Dependency Lock File</a>): this way you will be able to control dependencies more thoroughly, without worrying about transferring this file between build stages.
<h3>Plan</h3>
The <code>terraform plan</code> command helps you validate the changes manually. Still, there are ways to use it in automation as well.
By default, Terraform prints the plan output in a human-friendly format, but also supports machine-readable JSON. With additional command line options, you can extend your CI experience. For example, you can use validation conditions to decide whether to apply the changes automatically; or, you can parse the plan details and integrate the summary into a Pull Request description. Let’s review a simple example that illustrates this. 
First, save the plan output to the file:

The main point here is the <code>-out</code> option — it tells Terraform to save its output into a binary plan file. We will also talk about it later in this section. 
The <code>-compact-warnings</code> option suppresses the warning-level messages produced by Terraform
Also, the <code>plan</code> command has <code>-detailed-exitcode</code> option that returns detailed exit codes when the command exits. For example, you can leverage this in a script that wraps Terraform and adds more conditional logic to its execution, because CIs will generally fail the pipeline on a command’s non-zero exit code. However, that may add complexity to the pipeline logic. So if you need to get detailed info about the plan, I suggest parsing the plan output.
Once you have a plan file, you can read it in JSON format and parse it. Here is a code snippet that illustrates this:

Another way to see the information about changes, is to run the <code>plan</code> command with <code>-json</code> option and parse its output to stdout (available starting from Terraform 1.0.5):

This technique can make your Pull Request messages more informative and improve your collaboration with teammates. (Btw. <a href="https://docs.spacelift.io/concepts/policy/git-push-policy#push-and-pull-request-events" target="_blank" rel="noopener">Spacelift</a> provides this feature out of the box).
You can write a custom script/function that sends a Pull Request comment to VCS using its API. Or, you can try the existing features of your VCS: with GitHub Actions, you can use the <a href="https://github.com/marketplace/actions/terraform-pr-commenter" target="_blank" rel="nofollow noopener">Terraform PR Commenter</a> or similar action to achieve this; for GitLab, there is a built-in functionality that integrates plan results into the Merge Request. See more here: <a href="https://docs.gitlab.com/ee/user/infrastructure/iac/mr_integration.html" target="_blank" rel="nofollow noopener">Terraform Integration in Merge Requests</a> 
And you can find more information about the specification of the JSON output here: <a href="https://www.terraform.io/docs/internals/json-format.html" target="_blank" rel="nofollow noopener">Terraform JSON Output Format.</a>
<h3>Apply</h3>
Once the plan file is ready, and the proposed changes are expected and approved, it&#8217;s time to <code>apply</code> them.
Here is how the <code>apply</code> command may look like in automation:

The <code>plan.file</code> is the file we got from the previous plan step. 
Alternatively, you might want to omit the planning phase altogether. In that case, the following command will apply the configuration immediately, without the need for a plan:

Here, the <code>-auto-approve</code> option tells Terraform to create the plan implicitly and skip the interactive approval of that plan before applying.
Whichever way you choose, keep in mind the destructive nature of the <code>apply</code> command. Hence, the fully automated application of configuration generally works well with environments that tolerate unexpected downtimes, such as development or testing. On the other hand, plan review is recommended for production-grade environments, in which case the `apply` job is configured for a manual trigger.

If you run <code>init</code>, <code>plan</code>, and <code>apply</code> commands in different environments, you need to take care of some artifacts produced by Terraform:
<ul>
<li style="font-weight: 400;">The <code>.terraform</code> directory with information about <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work">modules</a>, providers, and the state file (even in the case of remote state). </li>
<li style="font-weight: 400;">The <code>.terraform.lock.hcl</code> file — the dependency lock file which Terraform uses to check the integrity of provider versions used for the project. If your VCS does not track it, you&#8217;ll need to pass that file to the <code>plan</code> and <code>apply</code> commands to make them work after <code>init</code>.</li>
<li style="font-weight: 400;">The output file of the <code>plan</code> command is essential for the <code>apply</code> command, so treat it as a vital artifact. This file includes a full copy of the project configuration, the state, and variables passed to the <code>plan</code> command (if any). Therefore, mind the security precautions because sensitive information may be present there.</li>
</ul>
Make sure to pass them successively between operations: init -&gt; plan -&gt; apply.
There is one shortcut, though. You can execute the <code>init</code> and <code>plan</code> commands within the same step/stage and transfer the artifacts only once — to the <code>apply</code> execution.

Last but not least, a few words on ways to maximize the advantage of <a href="https://spacelift.io/blog/how-to-use-terraform-variables" target="_blank" rel="noopener">variables when running Terraform</a> in CI.
There are two common ways of passing values for the variables used in the configuration:
1. Using a <code>-var-file</code> option with the variable definitions file — a filename ending in <code>.tfvars</code> or <code>.tfvars.json</code>. For example:

Also, Terraform can automatically load the variables from files named exactly <code>terraform.tfvars</code> or <code>terraform.tfvars.json</code>: with that approach, you don’t need to specify the tfvar file as a command option explicitly.
2. Using environment variables with the prefix <code>TF_VAR_</code>. Implicitly, Terraform always looks for the environment variables (within its process context) with that prefix, so the same &#8220;instance_type&#8221; variables from the example above can be passed as follows:

The latter method is widely used in CI because modern CI/CD tools support the management of the environment variables for automation jobs.
Please refer to the following official documentation if you want to know more about variables: <a href="https://www.terraform.io/docs/language/values/variables.html" target="_blank" rel="nofollow noopener">Terraform Input Variables</a>.
Additionally, Terraform supports several configuration parameters in the form of environment variables. These parameters are optional; however, they can simplify the automation management and streamline its code.
<ul>
<li style="font-weight: 400;"><code>TF_INPUT</code> — when set to &#8220;false&#8221; or &#8220;0&#8221;, this tells Terraform to behave the same way as with the <code>-input=false</code> flag;</li>
<li style="font-weight: 400;"><code>TF_CLI_ARGS</code> — can contain a set of command-line options that will be passed to one or another Terraform command. Therefore, the following notation can simplify the execution of <code>apply</code> and <code>plan</code> commands by unifying their options for CI:</li>
</ul>

You can take even more advantage of this when using this variable as the environment configuration of stages or jobs in a CI/CD tool.
<ul>
<li style="font-weight: 400;"><code>TF_IN_AUTOMATION</code>  — when set to any non-empty value (e.g., &#8220;true&#8221;), Terraform stops suggesting commands run after the one you execute, hence producing less output. </li>
</ul>

Although this article provides a comprehensive overview of possible automation approaches, there is always room for some particular and unusual scenarios. Whichever use case and conditions you have, keep the following goals in mind when using Terraform in CI:
<ul>
<li style="font-weight: 400;">Ease of code management</li>
<li style="font-weight: 400;">A secure and controlled execution environment</li>
<li style="font-weight: 400;">Coherent runs of init, plan, apply phases</li>
<li style="font-weight: 400;">Leveraging of built-in Terraform capabilities </li>
</ul>
Also, there are <a href="https://spacelift.io/product" target="_blank" rel="noopener">CI/CD automation tools like Spacelift that solve all the issues out of the box</a>. They’re worth looking at as an alternative to a homegrown solution on top of a vanilla CI. Spacelift helps overcome common state management issues and adds several must-have features for infrastructure management. You can check it out for free, by going <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">here</a> and creating a trial account.
Was this article helpful? Do you have any questions, or would you like to share your common practices? Feel free to tell us by leaving a comment below!

How a Specialized Management Platform Can Improve Your IaC

5 Ways to Manage Terraform at Scale

Building a Secure CI/CD Integration with Azure

Learn how to use Terraform in CI/CD. See examples of CLI options useful in CI and a full tutorial. Learn different approaches to integrating Terraform into generic deployment pipelines.

44-terraform_in_continuous_integration

Guide to using Terraform in CI

Amazon S3 Bucket is a storage service offered by AWS for storing data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, etc. The S3 stands for Simple Storage Service which can be scaled based on individual or organization needs. On top of providing a storage solution, Amazon S3 also provides comprehensive access management which can help you to set up very granular-level permissions.
This blog post is all about managing the AWS S3 bucket using Terraform. Terraform provides three S3 resources: 
<ol>
<li style="font-weight: 400;" aria-level="1">aws_s3_bucket</li>
<li style="font-weight: 400;" aria-level="1">aws_s3_bucket_object</li>
<li style="font-weight: 400;" aria-level="1">aws_s3_bucket_public_access_block</li>
</ol>
These resources are used for managing the S3 bucket, but exhibit different characteristics which we will explore in this post.
AWS S3 bucket supports versioning, replication, encryption, ACL (access control list), and bucket object policy. Here is the list of S3 tasks which we are going to complete using Terraform: 
<ol>
<li style="font-weight: 400;" aria-level="1">Setting up AWS Access Credentials (prerequisite).</li>
<li style="font-weight: 400;" aria-level="1">Using aws_s3_bucket resource to create S3 Bucket.</li>
<li style="font-weight: 400;" aria-level="1">Uploading files to S3 bucket using aws_s3_bucket_object.</li>
<li style="font-weight: 400;" aria-level="1">Managing ACL (Access Control List) using aws_s3_bucket_public_access_block.</li>
<li style="font-weight: 400;" aria-level="1">Deleting the S3 bucket using Terraform.</li>
</ol>
Let’s start.

Terraform always needs Access Key and Secret Key to work with the AWS resources. But, AWS always provides you with static plain text credentials and should not be stored, as it is contained in your Terraform file. 
There are a couple of ways to handle this problem:
<ol>
<li style="font-weight: 400;" aria-level="1">Using Spacelift AWS Integration with IAM roles. Spacelift provides AWS integration out of the box. Here is a comprehensive guide from Spacelift which can help to integrate with AWS: <a href="https://docs.spacelift.io/integrations/cloud-providers/aws" target="_blank" rel="noopener">AWS Integration Tutorial</a></li>
<li style="font-weight: 400;" aria-level="1">The second way would be generating AWS access credentials dynamically based on IAM policies and storing them into the <a href="https://www.vaultproject.io/docs/install" target="_blank" rel="nofollow noopener">Vault (Hashicorp Vault)</a>.</li>
</ol>
The above-mentioned methods will help you integrate with AWS in a more secure way. 
<a href="https://docs.spacelift.io/integrations/cloud-providers/aws#programmatic-setup" target="_blank" rel="noopener">Spacelift Programmatic Setup of IAM Role</a> —If you are using Spacelift, then here is the code snippet of Terraform which you should integrate with your existing Terraform infrastructure code base.

Hashicorp Vault programmatic Setup—If you are using the Hashicorp vault, here’s the Terraform code snippet which defines the AWS IAM role for managing an S3 Bucket.

After setting up the credentials, let’s use the Terraform aws_s3_bucket resource to create the first S3 bucket.  
The S3 Bucket name we are going to use is – spacelift-test1-s3.
Here are the names of items needed for creating the S3 bucket: 
<ol>
<li style="font-weight: 400;" aria-level="1">region—Specify the name of the region.</li>
<li style="font-weight: 400;" aria-level="1">bucket—Name the bucket i.e. – spacelift-test1-s3.</li>
<li style="font-weight: 400;" aria-level="1">acl—Access control list. We will set the S3 access as private.</li>
</ol>
Create a Terraform file named – main.tf and use the following Terraform code snippet:

Along with main.tf, let’s create version.tf for AWS and vault version.

If you are going to use Hashicorp vault instead of Spacelift, then you must also add the Hashicorp vault version.

Let’s apply the above Terraform configuration using Terraform commands:
1. $ terraform init – This is the first command we are going to run.

image6-3

terraform init

2. $ terraform plan – The second command would be to run a Terraform plan. This command will tell you how many AWS resources are going to be added, changed or destroyed.

image1-4

$ terraform plan

3. $ terraform apply – Apply the Terraform configuration using the Terraform apply command which will eventually create an S3 bucket in AWS.

image2-4

$ terraform apply

image7-3

AWS S3

In Step 2 we saw how to create an S3 bucket using the aws_s3_bucket Terraform resource.  In this step, we are going to use the same S3 bucket (spacelift-test1-s3) to upload files into. 
When we want to perform some additional operations (e.g. – upload files) on the S3 bucket then we are going to use the aws_s3_bucket_object Terraform resource.
For uploading the files to the S3 bucket we will extend the existing Terraform script from Step 2, along with the new aws_s3_bucket_object resource block.
We are going to upload the two sample text files:
<ol>
<li style="font-weight: 400;" aria-level="1">test1.txt</li>
<li style="font-weight: 400;" aria-level="1">test2.text</li>
</ol>
Here is the screenshot of my project structure for uploading files, which includes my main.tf along with test1.txt, test2.txt files.

image8-3

Terraform S3 bucket

As you can see from the project structure, I have kept my test files under the directory uploads, so I need to mention the relative path inside my Terraform file (main.tf).
Here is my Terraform file:

Here are some additional notes for the above-mentioned Terraform file – 
<ol>
<li style="font-weight: 400;" aria-level="1">for_each = fileset(“uploads/”, “*”) – For loop for iterating over the files located under upload directory.</li>
<li style="font-weight: 400;" aria-level="1">bucket =  aws_s3_bucket.spacelift-test1-s3.id – The original S3 bucket ID which we created in Step 2.</li>
<li style="font-weight: 400;" aria-level="1">Key = each.value – You have to assign a key for the name of the object, once it’s in the bucket.</li>
<li style="font-weight: 400;" aria-level="1">Source = “uploads/${each.value}” – Path of the files which will be uploaded to the S3 bucket.</li>
</ol>
<h3>How to Apply the New Changes?</h3>
Since we are working in the same main.tf file and we have added a new Terraform resource block aws_s3_bucket_object, we can start with the Terraform plan command:
1. $ terraform plan – This command will show that 2 more new resources (test1.txt, test2.txt) are going to be added to the S3 bucket. Because we have previously created an S3 bucket, this time it will only add new resources.

image3-4

aws_s3_bucket_object

2. $ terraform apply – Run the Terraform apply command and you should be able to upload the files to the S3 bucket.

image4-4

aws_s3_bucket_object test

Here is the screenshot from AWS console S3 bucket:

image9-3

Spacelift-test1-s3

There are many more things that you can do with Terraform and the S3 Bucket. Here is a guide on <a href="https://spacelift.io/blog/how-to-rename-aws-s3-bucket-in-terraform" target="_blank" rel="noopener">how to rename an AWS S3 bucket in Terraform</a> which can help you rename your S3 bucket.

Now, after uploading the files to an S3 bucket, the next Terraform resource which we are going to talk about is aws_s3_bucket_public_access_block. This resource is going to help you manage the public access associated with your S3 bucket. 
By default, the value is false, which means we are allowing public ACL (Access Control List). If you want to restrict the public ACL, you have to set the value to true.
Here, we are going to take the same example which we have taken previously for uploading the files to an S3 bucket:

You can see in the above example that we have restricted the following public ACL: 
<ol>
<li style="font-weight: 400;" aria-level="1"> block_public_acls   = true</li>
<li style="font-weight: 400;" aria-level="1"> block_public_policy = true</li>
<li style="font-weight: 400;" aria-level="1"> ignore_public_acls  = true</li>
<li style="font-weight: 400;" aria-level="1"> restrict_public_buckets = true</li>
</ol>
With the help of aws_s3_bucket_public_access_block, you can manage the public access control list onto your S3 Bucket.

In the previous steps, we have seen how to create an S3 bucket and how to upload files to the S3 bucket using terraform aws_s3_bucket, aws_s3_bucket_object resources.
In this section, you will see how we can delete the S3 bucket once we are done working with the S3.
When you are using Terraform, the deletion part is always easy. You simply need to run the universal $ <a href="https://spacelift.io/blog/how-to-destroy-terraform-resources">terraform destroy</a> command and it will delete all the resources which you have created previously.

image5-3

$ terraform destroy

As you can see in the screenshot, Terraform has deleted the resources in the reverse chronological order starting from test2.txt, test2.txt, and finally the bucket spacelift-test1-s3.

Using Terraform to create an S3 bucket is relatively simple, but it is not recommended to use Terraform for uploading thousands of files into the S3 bucket. Terraform is an infrastructure provisioning tool and should not be used for performing data-intensive tasks. This blog is a comprehensive guide to getting yourself familiar with Terraform and the S3 bucket.

If you are struggling with Terraform automation and management, check out Spacelift. It helps you manage Terraform state, build more complex workflows, and adds several must-have capabilities for end-to-end infrastructure management.

How to Provision an AWS EKS Kubernetes Cluster with Terraform

Importing Existing Infrastructure into Terraform - step by step

See how to create and manage an AWS S2 bucket using Terraform. Examples of Amazon Web Services S3 bucket in Terraform.

aws-s3-bucket

Managing S3 Bucket using Terraform

How to Create and Manage an AWS S3 Bucket Using Terraform

General | Spacelift Blog

General

Terraform is a well-known IaC tool from Hashicorp. Terragrunt is a thin wrapper of Terraform that provides extra tools for keeping your configurations DRY, working with multiple Terraform modules, and managing remote states. It is an open source project from Gruntwork, co-founded by Yevgeniy Brikman, author of Terraform up and running.
This article discusses common patterns in <a href="https://spacelift.io/blog/infrastructure-as-code">Infrastructure as Code (IaC)</a> based on Terraform and Terragrunt examples and summarizes and highlights the design principle underlying good architecture.
We will present these patterns in an abstract way. Although the examples presented in this article use Terraform and Terragrunt, the main concepts apply to other popular IaC tools and platforms like Pulumi, AWS CDK, CloudFormation, Terraform Cloud, Atlantis, Spacelift, and <a href="https://spacelift.io/terraform-cloud-alternative" target="_blank" rel="noopener">Terraform Cloud alternatives</a>.
To put it simply, “managing collections of infrastructure resources” is an abstract concept. The technologies used to implement this concept include:
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://www.terraform.io/docs/cloud/workspaces/index.html" target="_blank" rel="nofollow noopener">Terraform workspace</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html" target="_blank" rel="nofollow noopener">Cloudformation (nested) stack</a></li>
</ul>
Git branch (allows you to create two branches, one called workspace_1, the other workspace_2, then relate remote state to each of the branches, which is actually a neat trick.)

In short, to utilize IaC is to describe your infrastructure with code. Coding infrastructure in HCL (<a href="https://github.com/hashicorp/hcl" target="_blank" rel="nofollow noopener">Hashicorp Configuration Language</a>) with Terraform is just like coding with any other programming language, like Python or Java. Programming technologies such as linter, IDE, unittest, documentation, and CICD, are all applicable to IaC. Additionally, a project layout is crucial to defining a structure to organize your code, dependencies, configuration files, etc.
One common pattern of the project layout for IaC is much like the following:

image1-3

common pattern of the project layout for IaC

The basic principle is to:
<ul>
<li style="font-weight: 400;" aria-level="1">Separate the process of IaC into two phases: development and deployment.</li>
<li style="font-weight: 400;" aria-level="1">Separate the deployment environments into prod, qa, and stage, for example.</li>
<li style="font-weight: 400;" aria-level="1">Separate the cloud architecture into components according to projects, types of infrastructure, etc. Each component can be stored in a single “gigantic” repository or multiple “micro” repositories (see Pattern: Mono-repo vs Multi-repo).</li>
</ul>

The process of IaC usually consists of two phases: development and deployment (if you are interested in this topic, the <a href="https://www.weave.works/technologies/gitops/" target="_blank" rel="noopener">GitOps and weaveworks articles</a> are good sources of information).
In the development phase, one writes code to describe and configure the desired infrastructure. In the deployment phase, the written infrastructure code is applied.
The advantages of separation:
<ul>
<li style="font-weight: 400;" aria-level="1">Separation of Concern: developers who write IaC do not need to know the details of the deployment, and can work or test on a new version of the code without influencing the actual deployed infrastructure.</li>
<li style="font-weight: 400;" aria-level="1">Flexibility: a well-designed IaC codebase is modularized and parameterized. The same code snippet can be reused and deployed under different use cases in different environments with customized parameters.</li>
</ul>

Imagine you have an AWS cloud architecture for web services, including different AWS services like ECS, RDS, S3, IAM, SSM, Secret Manager, Cloudwatch, Cloudfront and so on. One option is to pack all of the components into a single Terraform remote state (main.tf). This is fine for quick and dirty prototyping, but not recommended for production.
Always cut your cloud architecture into meaningful components such as:
<ul>
<li style="font-weight: 400;" aria-level="1">Stateful components like database, filesystem, or S3.</li>
<li style="font-weight: 400;" aria-level="1">Stateless components like lambda, ECS, or API Gateway.</li>
</ul>
You can also cut it based on projects or functional areas (e.g. frontend, backend, data warehouse).
The advantages of separation:
<ol>
<li style="font-weight: 400;" aria-level="1">Avoids scanning the whole architecture if a change needs to be made only locally to one resource. One can enter the component folder which includes that resource, make changes there, and apply them. Scanning a fat cloud account can be time consuming on a large amount of cloud resources, even causing unnecessary API throttling.</li>
<li style="font-weight: 400;" aria-level="1">Avoids unnecessary touches on databases or other crucial infrastructure to prevent accidental destructive operations.</li>
<li style="font-weight: 400;" aria-level="1">Increases readability of the IaC.</li>
</ol>
Translate the above described project layout into folder structure:

image3-3

folder structure

The separation of the development and deployment process is implemented with two git repositories &#8211; env/ and modules/. Notice the .git in the folder. Here, Terragrunt.hcl is simply a configuration file for modules specifically in the language of Terragrunt. The separation of cloud architecture components is implemented with subfolders app/ , mysql/, vpc/. The separation of deployment environments is implemented with subfolders prod/ , qa/ , stage/.

One of the principles of software engineering is DRY (don’t repeat yourself). The same applies to IaC. There are at least two kinds of tricks you can use to DRY IaC:
<ul>
<li style="font-weight: 400;" aria-level="1">Modularization</li>
<li style="font-weight: 400;" aria-level="1">Configuration inheritance</li>
</ul>
<h2>1) Modularization</h2>
A <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform module</a> is a way of creating a template of a cloud pattern, parameterizing, and reusing it. Like any other artifacts, modules can be stored and versioned in the registry. The official Terraform module registry is <a href="https://registry.terraform.io/" target="_blank" rel="nofollow noopener">https://registry.Terraform.io/</a>. You can also build your own private registry.
<h2></h2>

image2-3

AWS security group

<h2>2) Configuration Inheritance</h2>
<a href="https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#include" target="_blank" rel="nofollow noopener">Configuration inheritance</a> is a feature supported by Terragrunt. The concept is similar to class inheritance in object oriented programming, except that configuration inheritance serves the purpose of configuration reusability.
In the following example (also see the project layout above), we have a configuration file at the global level (global_vars.yaml) and configuration files at the environment level (environment_specific_vars.yaml). 
In global_vars.yaml, variables like account_id, region_name and repository_url can be defined, and in environment_specific_vars, these global variables will be inherited and present, additionally defining new environment level variables like vpc_id (usually we use separate vpc for different environments), environment_name, etc.

image4-3

Configuration Inheritance

In summary, configuration inheritance increases the reusability of configuration files and reduces the difficulty in maintenance.

Last but not least, let’s review the concept of mono-repo and multi-repo. It has been pointed out by Hashicorp as a <a href="https://www.hashicorp.com/blog/terraform-mono-repo-vs-multi-repo-the-great-debate" target="_blank" rel="nofollow noopener">major architectural decision for IaC projects</a>, a view I share.
The problem is, there are two ways to organize code. In simple terms, mono-repo stores all code into one git repository (for example with multiple subfolders), whereas multi-repo separates them into multiple repositories.
Analogously, the relationship between mono-repo and multi-repo can be compared to monolith software architecture and microservice architecture. Mono-repo is intended to perceive the architecture as a whole and as evolving together. It is usually more suitable for small projects or big projects in prototyping. Multi-repo, on the other hand, facilitates collaboration but with extra overhead. Naturally, the more repos you have, the higher the maintenance cost, but also the more autonomous and self-service each repository is.

In this article, we reviewed several patterns of IaC architecture in a technology-agnostic way. The examples given are based on <a href="https://spacelift.io/blog/what-is-terraform">Terraform</a> and Terragrunt, since they are the most popular open source tools on the market at the moment. The idea, however, is to look at different technologies like pulumi, AWS CDK, cloudformation, etc., where the same patterns keep appearing in slight variations.
We rooted the principles of IaC architecture design in software engineering concepts like inheritance, DRY, project layout, parameterization, and so on. Afterall, Infrastructure as Code is coding infrastructure, and recognizing common patterns is the recommended way to learn and apply IaC.

Spacelift is a sophisticated SaaS product for Infrastructure as Code that helps DevOps develop and deploy new infrastructures or changes quickly and with confidence.

What Are Terraform Modules and How to Use Them

Multi-Cloud Infrastructure: Misconceptions, Benefits, Best Practices

See common patterns of Infrastructure as Code architecture in Terraform and Terragrunt. Learn the differences between common patterns of IaC.

terraform_and_terragrunt

Patterns of infrastructure as code

Patterns of Infrastructure as Code Architecture: Terraform & Terragrunt

How to install and upgrade Terraform to the latest version

Just like a versioning tool such as Git or SVN, which allows you to rollback or upgrade if there is a problem with the current version of your code, tfenv lets you upgrade as well as rollback versions of Terraform. 
But why do we need multiple versions of Terraform? A typical infrastructure setup always consists of development, testing, staging, and the production environment. As a rule of thumb, the production environment gets the most stable version. On the other hand, development, test, and sometimes staging environments receive newer versions of any software package. The same goes for Terraform. 
If during the testing phase there is a bug which breaks the existing infrastructure setup, tfenv can help roll back Terraform to the previous stable version. Similarly, you can upgrade to the next stable version so that your Terraform stack is up to date.
In this post, we will see how you can install Terraform, as well as how to upgrade it to a specific release version (Terraform v0.15, v0.14, v0.13, v0.12, or any other) using tfenv.

Before we go into the installation part of tfenv let’s first see what tfenv is.
tfenv is a versioning tool for Terraform, maintained by the open source community on <a href="https://github.com/tfutils/tfenv" target="_blank" rel="nofollow noopener">Github,</a> under the MIT Open source license. Keep in mind it is not an official tool by HashiCorp.
<h3>Installing tfenv on MacOS</h3>
Lets first see how we can install tfenv on MacOS. For MacOS we are going to use the brew package manager. Refer to the following command for installation: 
$ brew install tfenv
<h3>Installing tfenv on Linux</h3>
Installing on Linux is a bit more manual. Use the following installation instructions:
1: Clone the GitHub repo
$ git clone https://github.com/tfutils/tfenv.git ~/.tfenv
2: Update $PATH 
$ echo &#8216;export PATH=&#8221;$HOME/.tfenv/bin:$PATH&#8221;&#8216; &gt;&gt; ~/.bash_profile
3: Create a symlinks
$ sudo ln -s ~/.tfenv/bin/* /usr/local/bin
<h3>Installing tfenv on Windows</h3>
tfenv does not currently support Windows due to symlink issues. 
<h3>Verify tfenv installation</h3>
Whether you are on MacOS or Linux, you can verify the installation of tfenv by simply running the following command:
$ tfenv -v
It should return with the tfenv version. At the time of writing this article, the latest version of tfenv was tfenv 2.2.2

image10

installing tfenv on Linux

Now that we have seen how to install tfenv, let’s dig into the tfenv commands. The first command which we are going to see is 
 $ tfenv list-remote
The above command will list out all available versions of Terraform to date. Here’s a screenshot after running the tfenv list-remote command.

image1-2

tfenv list-remote

If you are looking for a specific version of <a href="https://spacelift.io/blog/what-is-terraform">Terraform</a> (e.g. v0.12, v0.13, v0.14, v0.15) but tfenv list-remote returns with a really long list of versions, you can run the following command that includes the version you want. In this example, I am looking for Terraform 0.12.0
$ tfenv list-remote | grep 0.12.0
&nbsp;

image11-2

$ tfenv list-remote | grep 0.12.0

You have seen in the previous steps how to install tfenv and how to list all or some specific version of Terraform. 
Let’s install a specific version (0.12.0) of Terraform using tfenv. Use the following tfenv for installation: 
$ tfenv install 0.12.0

image6-2-2

$ tfenv install 0.12.0

After installing Terraform 0.12.0, you need to run one more command to actually use that version:
$ tfenv use 0.12.0

image3-2

$ tfenv use 0.12.0

Verify the installation of Terraform by running 
$ terraform -v

image13

$ terraform -v 

Learn how to install and upgrade Terraform to the latest version. A quick step-by-step tutorial to upgrading Terraform. Terraform installation made simple.

How to Download and Install Terraform on Windows, Linux & macOS

Recent posts

How to Deploy your Infrastructure in CI/CD using Terraform

How to Create and Manage an AWS S3 Bucket Using Terraform

Common Patterns of Infrastructure as Code Architecture: Terraform & Terragrunt

How to Upgrade Terraform to the Latest Version

Terraform vs. Ansible : Key Differences

Terraform Functions, Expressions, Loops (Examples)

How to Destroy Resources from Terraform: Tutorial

How to Get Started with Terraform on Google Cloud Platform (GCP)

5 Ways to Manage Terraform at Scale

What Are Terraform Modules and How to Use Them

How to Use Terraform Variables (Locals, Input, Output, Environment)