Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

AWS: Spacelift Reduces Time Spent on Cloud Management by 90% 🚀

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

How to Manage Active Directory Objects with Azure AD Provider for Terraform

Terraform

Adam Connelly

The <a href="https://registry.terraform.io/providers/hashicorp/azuread/latest/docs" target="_blank" rel="nofollow noopener">Azure AD provider for Terraform</a> can be used to manage your Azure Active Directory resources declaratively. This allows you to do things like:
<ul>
<li style="font-weight: 400;" aria-level="1">Automatically provision users and make sure they belong to the correct groups.</li>
<li style="font-weight: 400;" aria-level="1">Manage Azure compute permissions via Azure AD groups.</li>
</ul>
In this post, you will learn what the Azure AD Terraform provider is used for, how to authenticate and grant permissions, and see examples of what you can do with it. Let’s get started.

The following example shows how to use the Azure AD provider to create a <a href="https://registry.terraform.io/providers/hashicorp/azuread/latest/docs/resources/group" target="_blank" rel="nofollow noopener">group</a> in Azure AD:

The <code>terraform</code> section at the beginning is used to specify the version of the provider that we want to use, while the <code>azuread_group</code>  resource defines our group.

The Azure AD provider allows for multiple authentication methods, which are outlined in the <a href="https://registry.terraform.io/providers/hashicorp/azuread/latest/docs#authenticating-to-azure-active-directory" target="_blank" rel="nofollow noopener">provider&#8217;s documentation</a>. To allow you to get up and running quickly, the AD provider will attempt to get your credentials via the <a href="https://docs.microsoft.com/en-us/cli/azure/install-azure-cli" target="_blank" rel="nofollow noopener">Azure CLI</a>.
While this is fine for experimentation and local testing, for non-interactive scenarios like CI you will need to use a Service Principal or a Managed Service Identity.

In order to manage your Azure AD objects, the account used by <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> needs to have the correct permissions to perform its actions. You can manage these permissions via the Roles and administrators section of Azure AD:

Roles and administrators section of Azure AD

For example, to allow a Service Principal to manage groups, you would add it to the Groups administrator role:

Azure AD Groups administrator

The Terraform provider is well documented, and will typically contain a notice at the top of each resource explaining the permissions that are required for using it.

Another option that can be used with Service Principals instead of granting an administrator role is to assign specific API permissions to them. To do this, first find the AD Application linked to your Service Principal in the App Registrations section:

Service Principal in the App Registrations section

Go to the API permissions page for the application, and click on Add a permission:

Manage Active Directory Objects

In the window that appears, choose the Azure Active Directory Graph API, and then select the relevant permission you want to add:

Azure Active Directory Graph API

Before the Service Principal can actually use the permission you just added, you need to take a final step called granting Admin Consent. You can do this by clicking on the Grant admin consent for &lt;tenant&gt; button displayed above the permissions table:

5-5-2

Azure AD Admin Consent

NOTES:
<ul>
<li style="font-weight: 400;" aria-level="1">When adding permissions to your Service Principal, you need to add Application permissions rather than Delegated permissions. This means that the Service Principal is allowed to perform the specified actions as itself, rather than on behalf of another user.</li>
<li style="font-weight: 400;" aria-level="1">The set of permissions that you can add via API permissions is quite limited. For example, to create AD groups you need to add the Directory.ReadWrite.All permission, but this will not allow your Service Principal to delete any of the groups it creates. In order to be able to delete groups, you need to grant it the Group Administrator role, so depending on your requirements there may not be any point in granting API permissions.</li>
<li style="font-weight: 400;" aria-level="1">The Azure AD Terraform provider switches to the Microsoft Graph API as of version 2.0.0, so when version 2 is released you will need to grant permissions to the Microsoft Graph API instead of to the Azure Active Directory Graph API.</li>
</ul>

<ul>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Automate Terraform Deployments and Infrastructure Provisioning&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
<li><a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Deploying Infrastructure in CI/CD with Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Deploying Infrastructure in CI/CD with Terraform</a></li>
<li><a href="https://spacelift.io/blog/drift-detection" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Drift Detection and How to Fix It With IaC Tools&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Infrastructure Drift Detection and How to Fix It With IaC Tools</a></li>
</ul>

<h3>Example 1 &#8211; Managing Users and Groups</h3>
The following example creates two users and two groups, and assigns each user to a group:

<h3>Example 2 &#8211; Creating a Service Principal and granting RBAC permissions</h3>
The following example combines the Azure AD provider with the <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs" target="_blank" rel="nofollow noopener">Azure RM provider</a>, allowing you to create a Service Principal and assign it permission to <a href="https://spacelift.io/blog/managing-azure-resources-using-spacelift" target="_blank" rel="noopener">manage certain Azure resources</a>:

In this post, we’ve covered what the Azure AD Terraform provider is used for, how to authenticate and grant the correct permissions, as well as showing a few examples of what can be done with it. Hopefully, you’ve found it useful!
If you’re interested in finding out about how you can use Spacelift to manage your Azure resources via the Terraform <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs" target="_blank" rel="nofollow noopener">Azure Provider</a>, check out our <a href="https://docs.spacelift.io/integrations/cloud-providers/azure" target="_blank" rel="noopener">Azure documentation</a>. There you can also learn how to configure the following authentication methods in Spacelift: Spacelift Managed Integration, Static Credentials and Managed Service Identities. And don’t forget that you can easily take Spacelift for a <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">free test drive</a>!

Spacelift helps manage Terraform state, build more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and many more.

Importing Existing Infrastructure into Terraform - Step by Step

5 Ways to Manage Terraform at Scale - Best Practices

How Spacelift Can Improve Your Infrastructure as Code

Learn how to manage Active Directory Objects with Azure AD Provider for Terraform. See examples of how to authenticate and grant the correct permissions.

15-azure-directories

Active Directory Objects with Azure AD Provider for Terraform

Jacob Martin

Data sources in Terraform are used to get information about resources external to Terraform, and use them to set up your Terraform resources. For example, a list of IP addresses a cloud provider exposes.

We can find excellent examples of data source usage in the <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc#example-usage" target="_blank" rel="nofollow noopener">AWS provider docs</a>:

In this scenario, we may have created our aws_vpc manually, because it also contains other manually created resources, and now we want to start adding Terraform managed resources to it.
In order to do this, we use the aws_vpc data source to find the manually created aws_vpc, and then use its properties to configure our aws_subnet.
Another fancy use case would be to use the <a href="https://registry.terraform.io/providers/integrations/github/latest/docs/data-sources/repository_pull_requests" target="_blank" rel="nofollow noopener">GitHub provider Pull Requests data source</a> to list all Pull Requests, and then provision an on-demand preview environment for each Pull Request. It would go along the lines of:

<ul>
<li><a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Deploy your Infrastructure in CI/CD using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Deploy your Infrastructure in CI/CD using Terraform</a></li>
<li><a href="https://spacelift.io/blog/drift-detection" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Drift Detection and How to Fix It&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Infrastructure Drift Detection and How to Fix It</a></li>
<li><a href="https://spacelift.io/blog/github-actions-terraform" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Managing Terraform with GitHub Actions &amp; How to Scale&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Managing Terraform with GitHub Actions &amp; How to Scale</a></li>
</ul>

By default, Terraform will refresh all data sources before creating a plan. You can also explicitly refresh all data sources by running <code>terraform refresh</code>.
Occasionally you’ll have data sources that change very often and would like to keep your resources in sync with those changes. The easiest way to achieve this is to just run Terraform every few minutes or so, do a refresh, and apply all resulting changes.
You could also use something akin to <a href="https://docs.spacelift.io/concepts/stack/drift-detection" target="_blank" rel="noopener">Spacelift’s Drift Detection</a> to automate this process and make sure it doesn’t interfere with your manual Terraform executions.

If you are struggling with Terraform automation and management, check out Spacelift. It helps you manage Terraform state, build more complex workflows, and adds several must-have capabilities for end-to-end infrastructure management.

Terraform Tutorial - Getting Started With Terraform on AWS

Data sources in Terraform are used to get information about external resources and use them to set up Terraform resources. See the example usage. 

13-terraform-data-sources

Terraform data sources

Terraform Data Sources - How They Are Utilised (Example)

<dfn>Open Policy Agent</dfn> is an open-source engine that provides a way of declaratively writing policies as code and then using those policies as part of a decision-making process. It uses a policy language called Rego, allowing you to write policies for different services using the same language.
OPA can be used for a number of purposes, including:
<ul>
<li style="font-weight: 400;" aria-level="1">Authorization of REST API endpoints.</li>
<li style="font-weight: 400;" aria-level="1">Allowing or denying Terraform changes based on compliance or safety rules.</li>
<li style="font-weight: 400;" aria-level="1">Integrating custom authorization logic into applications.</li>
<li style="font-weight: 400;" aria-level="1">Implementing Kubernetes Admission Controllers to validate API requests.</li>
</ul>
OPA was originally created by <a href="https://www.styra.com/" target="_blank" rel="noopener">Styra</a>, and is now part of the Cloud Native Computing Foundation (CNCF), alongside other CNCF technologies like Kubernetes and Prometheus.
In this article, I will give an overview of Open Policy Agent, explain why you would want to use it, as well as showcase how you can use OPA with your Spacelift account. Although OPA can serve many purposes, I’m going to focus on how it can be used alongside <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code</a>.

We can visualise how OPA works using the following diagram:

how-does-opa-work

Open Policy Agent (OPA) Example

As you can see, OPA accepts a policy, input and query, and based on that, generates a response. The input can be any valid JSON document, allowing OPA to integrate with any tool that produces JSON output.

In the following sections, I’ll go into more specific examples of using OPA that should help make things clearer but before I do, here’s a quick list of reasons why I’m interested in using OPA:
<ul>
<li style="font-weight: 400;" aria-level="1">Policies as code allow you to follow your standard development lifecycle with PRs, CI, etc., and provide you with a history of changes to your policies.</li>
<li style="font-weight: 400;" aria-level="1">OPA is designed to work with any kind of JSON input, meaning it can easily integrate with any tool that produces JSON output.</li>
<li style="font-weight: 400;" aria-level="1">Because OPA integrates with a number of different tools, it allows you to use a standard policy language across many parts of your system, rather than relying on multiple vendor-specific technologies.</li>
<li style="font-weight: 400;" aria-level="1">OPA supports unit-testing, making it easier and faster to iterate your policies with confidence that they won’t break.</li>
</ul>

Let’s try to make that a bit less theoretical by using a specific example: Terraform. Terraform can produce a plan in JSON format via the terraform show command. This means that we can define policies for our infrastructure, and use OPA to make a decision about whether a plan is safe to apply or not:

using-opa-with-terraform

using opa with terraform

For example, say we have the following Terraform definition to create an EC2 instance:

Now, say we want to ensure that every Terraform resource has a Name tag. We could enforce that by creating a file called plan.rego with the following content:

In order to use OPA to evaluate our policy, we need to take the following steps:
<ol>
<li>Generate a Terraform plan as JSON.</li>
<li>Run opa eval to verify whether that plan passes our policy or not.</li>
</ol>

<h3>Step 1 &#8211; Generate our Terraform plan as JSON</h3>
To get a JSON representation of our plan, we need to output our plan to a file, and then use the terraform show command to output that plan as JSON:

<h3>Step 2 &#8211; Run opa eval</h3>
We can then use opa eval to evaluate our plan against our policy:

As you can see, we’re using the query data.spacelift.allow because our policy is loaded as a data file, and we defined our allow rule in the spacelift namespace. You can also see that opa eval produced an empty output ({}). This means that our allow rule didn’t evaluate to true, and so produced no output.
Let’s adjust our Terraform definition to include a Name tag:

Now if we generate our plan and evaluate the policy again, we should get a slightly different output:

This tells us that the allow rule has evaluated true. We can get that in a slightly more concise way by using the pretty format option:

At this stage, you can probably imagine how you could integrate this into your <a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener">CI/CD pipeline </a>to enforce naming schemes, security rules, and various other organizational policies.

The following examples illustrate some possible use-cases for OPA with <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a>. All of the examples have been taken from the Spacelift <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy#cookbook" target="_blank" rel="noopener">plan policy documentation</a>, but have been altered to work with plain vanilla Terraform.
<h3>Example 1. Require human review when resources are deleted or updated</h3>
Adding new resources is usually a fairly safe operation, but updating or deleting existing resources can carry more risk. You could flag these for human review by using the following policy:

<h3>Example 2. Require commits to be reasonably sized</h3>
Once a PR goes over a certain size it becomes difficult to review without missing things. The same is true for Terraform plans. The following policy can be used to warn when the number of changes goes over a certain threshold:

<h3>Example 3. Blast radius</h3>
The following policy attempts to determine the risk of a particular plan by assigning different weightings to the change type (create, update, delete), along with the affected resource type (ECS cluster, EC2 instance, etc.). It takes the approach that an update or delete is more risky than a create because it affects an existing resource:

<ul>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;10 Ways to Improve Your Infrastructure as Code&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">10 Ways to Improve Your Infrastructure as Code</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
</ul>

How can we make sure that our policies work as we expect them to and that they don’t break over time as we make changes to them? You guessed it: unit testing! Luckily for us, OPA has first-class support for testing via the opa <code>test</code> command.
To create tests for our policy, all we need to do is create another Rego file with a series of rules prefixed with <code>test_</code>. Each rule starting with that prefix defines a separate test. 
Let&#8217;s go ahead and create a file called plan_test.rego, with the following contents:

As you can see, OPA makes it really easy to specify the policy input using the <code>&lt;variable&gt;</code> as <code>&lt;value&gt;</code> syntax. This allows us to create very concise tests by only including values we care about in the policy input, rather than having to use the entire plan output.
Let&#8217;s go ahead and run <code>opa test</code>:

Unsurprisingly our test passes. That&#8217;s not very exciting, so let&#8217;s add a new test to ensure our resources include an <code>Environment</code> tag:

Running opa test again shows us a failure:

We can fix this by updating our policy:

At this stage if you run the test command again it should show 2 passes:

If you want to know more about OPA testing, the <a href="https://www.openpolicyagent.org/docs/latest/policy-testing/" target="_blank" rel="nofollow noopener">official docs</a> are full of great examples and information about what you can do.

At this stage, hopefully, you’ve got a pretty good idea of what OPA is as well as how it can be useful to you. It’s not too difficult to see how you could start integrating OPA into your development process or even use it as part of production systems.
Luckily for you, at Spacelift we’ve done all the heavy lifting, allowing you to reap the benefits of using OPA for Policy-as-Code without you having to implement everything from scratch yourself. In this section, I want to showcase some of the functionality that Spacelift provides as it relates to OPA.
<h3>1) Policy Types</h3>
Spacelift allows you to use <a href="https://resources.sei.cmu.edu/asset_files/Presentation/2021_017_001_739363.pdf" target="_blank" rel="noopener">OPA policies</a> to manage various aspects of your Spacelift account, and not just during planning. For example, you can use policies to control who can log into your account, along with what they have access to. For more information, see <a href="https://docs.spacelift.io/concepts/policy/" target="_blank" rel="noopener">Policy as Code</a>.
<h3>2) PR Checks</h3>
Spacelift can automatically trigger planning runs whenever you push changes to your VCS provider. For example, here’s what you might see in GitHub after creating a PR:

pr-checks-1

Opa pr checks

If all goes well your check will succeed, but if a policy is violated, a failed check will be reported:

pr-checks-2

opa pr checks failed

You can then view the details of the failure in Spacelift:

pr-checks-3

failure details in Spacelift

<h3>3) Manual Approvals</h3>
Spacelift plan policies use a slightly different format from the one we used in our example policies earlier in this post. Not only do they allow you to specify a message to be displayed, but also have the concept of <code>deny</code> and <code>warn</code>:

The <code>deny</code> rule fails the run completely, while the <code>warn</code> rule just displays a warning in the logs.
<code>warn</code> rules take on another role when a run is going to deploy changes (vs just showing the planned changes against a PR). If any warnings are reported during a deployment, the run will wait for manual approval before applying any changes.
Let’s use the following example policy, designed to warn if a certain set of suggested tags isn’t found within our resources:

If we then attempt to add resources that don’t contain all of those tags, Spacelift will block before applying the changes, and give us the chance to manually approve or deny the run:
&nbsp;

manual-approvals

manual approvals

This allows you to build complex workflows where certain changes are completely blocked, while others are allowed as long as the changes are reviewed first.
Check out our <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy#cookbook" target="_blank" rel="noopener">plan policy cookbook</a> for more ideas about what you can do.

Spacelift provides a <a href="https://registry.terraform.io/providers/spacelift-io/spacelift/latest/docs" target="_blank" rel="nofollow noopener">Terraform provider</a> for managing your Spacelift account. This means that you can manage the policies available within your account, along with the Stacks they apply to in code.
For example, to add the policy we defined earlier to Spacelift we can use the <code>spacelift_policy</code> resource like this:

We can then attach this policy to a Spacelift Stack using the <code>spacelift_policy_attachment</code> resource:

Taking this a step further, we could create a custom module for defining Spacelift Stacks that ensured that all Stacks had a certain set of policies attached by default.

There are a number of exciting tools available that can help integrate OPA with your systems, for example <a class="c-link" tabindex="-1" href="https://www.styra.com/" target="_blank" rel="noopener" data-stringify-link="https://www.styra.com/" data-sk="tooltip_parent" data-remove-tab-index="true">Styra DAS</a>. Styra provides a way of writing policies and deploying them across your infrastructure, as well as providing tools for unit testing, monitoring policy usage and more. You can try OPA at scale for free using <a href="https://www.styra.com/das-free" target="_blank" rel="noopener">Styra DAS Free</a>.

I hope you’ve enjoyed this post, and can see the value that OPA brings to the table. If you’re interested in trying out <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">Spacelift to see what we have to offer, why not sign up for a free trial</a>? You can set up a Spacelift account in minutes, and get started on your Policy as Code journey today!

Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation and controls to simplify and accelerate the provisioning of cloud based infrastructures.

What is Open Policy Agent (OPA)? See Open Policy Agent examples and a full tutorial on when to use it.

12-opa

Open Policy Agent

Open Policy Agent: What Is OPA and How It Works (Examples)

provision an AWS EKS Kubernetes cluster with Terraform

Want to level up your Kubernetes knowledge? Explore our tutorials and step-by-step guides written by Kubernetes experts. Learn Kubernetes best practices and solutions to the most difficult problems.

Kubernetes Guides and Tutorials | Spacelift Blog

Kubernetes

AWS EKS provides managed <a href="https://spacelift.io/blog/kubernetes-cluster" target="_blank" rel="noopener">Kubernetes clusters</a> as a service. You’re on AWS and want to avoid getting into the details of setting up a Kubernetes cluster from scratch? EKS is the way to go!
In this guide, you will learn how to provision an AWS EKS Kubernetes cluster with Terraform. Let&#8217;s start with the basics.

First, locally install Terraform:

as well as the AWS CLI:

and kubectl:

If you’re on a different operating system, please find the respective installation instructions here:
<ul>
<li style="font-weight: 400;" aria-level="1">Terraform: <a href="https://learn.hashicorp.com/tutorials/terraform/install-cli" target="_blank" rel="nofollow noopener">https://learn.hashicorp.com/tutorials/terraform/install-cli</a></li>
<li style="font-weight: 400;" aria-level="1">awscli: <a href="https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html" target="_blank" rel="nofollow noopener">https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html</a></li>
<li style="font-weight: 400;" aria-level="1">kubectl: <a href="https://kubernetes.io/docs/tasks/tools/" target="_blank" rel="nofollow noopener">https://kubernetes.io/docs/tasks/tools/</a></li>
</ul>

Now, you’ll need to configure your AWS CLI with access credentials to your AWS account. You can do this by running

and providing your Access Key ID and Secret Access Key. You will also need to add the region. For the purposes of this guide, we will use us-east-2. Terraform will later use these credentials to provision your AWS resources.

You can now clone a repository which contains everything you need to set up EKS:

Inside you’ll see a few files, the main one being eks-cluster.tf:

It uses the EKS Terraform module to set up an EKS cluster with 2 worker groups (the actual nodes running your workloads): one with a single medium machine, and one with two small machines.

You can now create all of those resources using Terraform. First, run

to initialize the Terraform workspace and download any <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">modules</a> and providers which are used.
In order to do a dry run of the changes to be made, run

This will show you that 51 resources will be added, as well as their relevant details. You can then run terraform apply with the resulting plan, in order to actually provision the resources:

This may take a few minutes to finish. You might get a “timed out” error, in which case just repeat both the terraform plan and terraform apply steps.
In the end you will get a list of outputs with their respective values printed out. Make note of your cluster_name.

In order to use kubectl, which is the main tool to interact with a Kubernetes cluster, you have to give it credentials to your EKS Kubernetes cluster. You can do this by running

Make sure to replace &lt;output.cluster_name&gt; with the relevant value from your Terraform apply outputs.

<ul>
<li><a href="https://spacelift.io/blog/terraform-environments" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Managing Multiple Terraform Environments Efficiently&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Managing Multiple Terraform Environments Efficiently</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale</a></li>
</ul>

You can now view the nodes of your cluster by running

This command is so long because it displays custom columns, thanks to which we can indeed see that there are 2 smaller nodes, and 1 bigger node.
Let’s deploy an Nginx instance to see if the cluster is working correctly.

You can see the status of it by running:

And finally set up a tunnel from your computer to this pod:

If you open <a href="http://localhost:3000" target="_blank" rel="nofollow noopener">http://localhost:3000</a> in your browser, you should see the web server greet you:

welcome to ngix

In order to destroy the resources we’ve created in this session, run

This may again take up to a few minutes. You can read more about it here: <a href="https://spacelift.io/blog/how-to-destroy-terraform-resources" target="_blank" rel="noopener">How to Delete Resources from Terraform Using the Destroy Command.</a>

I hope this guide helped you on your Kubernetes journey on AWS! If you want more help managing your Terraform state file, building more complex workflows based on Terraform, and managing AWS credentials per run, instead of using a static pair on your local machine, check out <a href="https://spacelift.io" target="_blank" rel="noopener">Spacelift</a>. We’d love to have you!

Learn how to provision an AWS EKS Kubernetes cluster with Terraform. See a full step-by-step tutorial.

10-provision-kubernetes-cluster

Provisioning AWS EKS Cluster with Terraform - Tutorial

Importing existing infrastructure into terraform

Sumeet Ninawe

In this tutorial, you will learn ways to import pre-existing cloud resources before you continue to develop the IaC in Terraform. This guide will provide you with an IaC import scenario which is often faced by teams starting to adopt Terraform for their operations.

Terraform is a relatively new technology and adopting it to manage an organisation’s cloud resources might take some time and effort. The lack of human resources and the steep learning curve involved in using Terraform effectively causes teams to start using cloud infrastructure directly via their respective web consoles.
For that matter, any kind of IaC method (CloudFormation, Azure ARM templates, Pulumi, etc.) requires some training and real-time scenario handling experience. Things get especially complicated when dealing with concepts like states and remote backends. In a worst case scenario, you can lose the <a href="https://spacelift.io/blog/terraform-state" target="_blank" rel="noopener"><code>terraform.tfstate</code> file</a>. Luckily, you can use the import functionality to rebuild it.
Getting the pre-existing cloud resources under the Terraform management is facilitated by Terraform import. <code>import</code> is a Terraform CLI command which is used to read real-world infrastructure and update the state, so that future updates to the same set of infrastructure can be applied via IaC.
The import functionality helps update the state locally and it does not create the corresponding configuration automatically. However, the Terraform team is working hard to improve this function in upcoming releases.

With an understanding of why we need to import cloud resources, let us begin by importing a simple resource &#8211; EC2 instance in AWS. I am assuming the Terraform installation and configuration of AWS credentials in AWS CLI is already done locally. We will not go into the details of that in this tutorial. To import a simple resource into Terraform, follow the below step-by-step guide.

<h3>1. Prepare the EC2 Instance</h3>
Assuming the Terraform installation and configuration of AWS credentials in AWS CLI is already done locally, begin by importing a simple resource—EC2 instance in AWS. For the sake of this tutorial, we will create an EC2 resource manually to be imported. This could be an optional step if you already have a target resource to be imported.
<h4>Terraform: Create EC2 Instance in Existing VPC</h4>
Go ahead and provision an EC2 instance in your AWS account. Here are the example details of the EC2 instance thus created:
Name: MyVM 
Instance ID: i-0b9be609418aa0609 
Type: t2.micro 
VPC ID: vpc-1827ff72 
&#8230;

<h3>2. Create main.tf and Set Provider Configuration</h3>
The aim of this step is to import this EC2 instance into our Terraform configuration. In your desired path, create `main.tf` and configure the <a href="https://spacelift.io/blog/terraform-aws-provider" target="_blank" rel="noopener">AWS provider</a>. The file should look like the one below.
<h4>Importing EC2 Instance into Terraform Configuration: Example</h4>

Run <code>terraform init</code> to initialize the <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a>. Below is the output of a successful initialization.

<h3>3. Write Config for Resource To Be Imported</h3>
As discussed earlier, Terraform import does not generate the configuration files by itself. Thus, you need to create the corresponding configuration for the EC2 instance manually. This doesn’t need many arguments as we will have to add or modify them when we import the EC2 instance into our state file.
However, if you don’t mind not seeing colorful output on CLI, you can begin adding all the arguments you know. But this is not a foolproof approach, because normally the infrastructure you may have to import will not have been created by you. So it is best to skip a few arguments anyway.
In a moment we will take a look at how to adjust our configuration to reflect the exact resource. For now, append the <code>main.tf</code> file with EC2 config. For example, I have used the below config. The only reason I have included <code>ami</code> and <code>instance_type</code> attribute, is that they are the required arguments for <code>aws_instance</code> resource block.

<h3>4. Import</h3>
Think of it as if the cloud resource (EC2 instance) and its corresponding configuration were available in our files. All that’s left to do is to map the two into our state file. We do that by running the import command as follows.

A successful output should look like this:

The above command maps the <code>aws_instance.myvm</code> configuration to the EC2 instance using the ID. By mapping I mean that the state file now “knows” the existence of the EC2 instance with the given ID. The state file also contains information about each attribute of this EC2 instance, as it has fetched the same using the import command.

<h3>5. Observe State Files and Plan Output</h3>
Please notice that the directory now also contains <code>terraform.tfstate</code> file. This file was generated after the import command was successfully run. Take a moment to go through the contents of this file.
Right now our configuration does not reflect all the attributes. Any attempt to <code>plan</code>/<code>apply</code> this configuration will fail because we have not adjusted the values of its attributes. To close the gap in configuration files and state files, run <code>terraform plan</code> and observe the output.

The plan indicates that it would attempt to replace the EC2 instance. But this goes completely against our purpose. We could do it anyway by simply not caring about the existing resources, and creating new resources using configuration.
The good news is that Terraform has taken note of the existence of an EC2 instance that is associated with its state.

<h3>6. Improve Config To Avoid Replacement</h3>
At this point, it is important to understand that the <code>terraform.tfstate</code> file is a vital piece of reference for Terraform. All of its future operations are performed with consideration for this state file. You need to investigate the state file and update your configuration accordingly so that there is a minimum difference between them.
The use of the word “minimum” is intentional here. Right now, you need to focus on not replacing the given EC2 instance, but rather aligning the configuration so that the replacement can be avoided. Eventually, you would achieve a state of 0 difference.
Observe the plan output, and find all those attributes which cause the replacement. The plan output will highlight the same. In our example, the only attribute that causes replacement is the AMI ID. Closing this gap should avoid the replacement of the EC2 instance.
Change the value of <code>ami</code> from “unknown” to what is highlighted in the plan output, and run <code>terraform plan</code> again. Notice the output.

This time the plan does not indicate the replacement of the EC2 instance. If you get the same output, you are successful in partially importing our cloud resource. You are currently in a state of lowered risk—if we apply the configuration now, the resource will not be replaced, but a few attributes would change.

<ul>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
</ul>

<h3>7. Improve Config To Avoid Changes</h3>
If we want to achieve a state of 0 difference, you need to align your resource block even more. The plan output highlights the attribute changes using <code>~</code> sign. It also indicates the difference in the values. For example, it highlights the change in the instance_type value from “t2.micro” to “unknown”.
In other words, if the value of <code>instance_type</code> had been “t2.micro”, Terraform would NOT have asked for a change. Similarly, you can see there are changes to the tags highlighted as well. Let’s change the configuration accordingly so that we close these gaps. The final <code>aws_instance</code> resource block should look as follows:

Run <code>terraform plan</code> again, and observe the output.

If you have the same output, congratulations, as you have successfully imported a cloud resource into your Terraform config. It is now possible to manage this configuration via Terraform directly, without any surprises.

In this section, you will learn how to import resources into the modules. For reference, we use the AWS VPC module (version 3.2.0) that creates 29 resources. Move the <code>terraform.tfstate</code> file to another location, so that Terraform becomes unaware of the existing resources.
This poses a challenge of importing the AWS VPC module into your configuration. Modules wrap multiple AWS resources into a single package that can be reused in various projects. You should expect a large codebase and many resources to be part of the module.
The process of importing resources created using a module is very similar to what we have discussed thus far, with a little difference in running the command. Go through the <code>.tf</code> files included in the module’s source, and identify the resources to be imported.
For example, the AWS VPC module creates a VPC resource. To import this resource, run the command as below.
<h3>Terraform Import Module Example</h3>

This command imports the target VPC resource in AWS, to our module’s configuration. Running the `plan` command indicates how many resources will be created. In other words, how many resources are yet to be imported.

The above output indicates 28 more resources to be imported to (re)build a perfectly consistent <code>terraform.tfstate</code> file. One less as we have successfully imported the VPC resource. Referring to the plan output, identify the resources in AWS and repeat the process for import.

As you have seen, the process of importing simple resources is pretty straightforward. The key here is to understand how Terraform state works. However, things can get quite tedious when importing complex deployments.
In the case of complex deployments, the team should have a clear identification of resources they want to manage using Terraform. A diagram representing the entire landscape helps to plan and to get the parts of deployment under Terraform’s management.
Planning output is key. If it is well-formatted, you will build a perfectly consistent state file. Prioritize and mitigate configurations that cause replacement of target resources followed by configurations that cause changes. Import the resources as they are to satisfy the state file until there are no gaps. Any enhancement or code restructuring can be focused on and implemented once you achieve 0 gaps.

Although this article provides a comprehensive overview of possible approaches towards importing resources into Terraform, there is always room for some particular and unusual scenarios. If you have any questions or would like to share your common practices, feel free to tell us by leaving a comment below!
We encourage you also to explore <a href="https://docs.spacelift.io/" target="_blank" rel="noopener">how Spacelift makes it easy to work with Terraform</a>. If you need any help managing your Terraform infrastructure, building more complex workflows based on Terraform, and managing AWS credentials per run, instead of using a static pair on your local machine, Spacelift is a fantastic tool for this. It supports Git workflows, policy as code, programmatic configuration, context sharing, drift detection, and many more great <a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener">features right out of the box</a>. You can check it for free, by <a href="https://spacelift.io" target="_blank" rel="noopener">creating a trial account</a>.

Build more complex workflows based on Terraform using policy as code, programmatic configuration, context sharing, drift detection, resource visualization and many more.

Initialize/ plan/ apply your IaC, manage modules, state, and more. 

frame-8479

Frame 8479

How to Deploy your Infrastructure in CI/CD using Terraform

How to Automate Terraform Deployments and Infrastructure Provisioning

See a step-by-step guide on importing existing infrastructure into Terraform. Check examples and learn how to import instances into a Terraform configuration.

9-terraform-import

Occasionally you might want to rename an AWS S3 bucket you are managing with Terraform. However, names of S3 buckets are immutable, which means you can’t change them directly. If you tried, Terraform would destroy the old one and then create a new one, resulting in data loss.
To avoid this, you need to create a new bucket with the desired name, move the data over to it, make the relevant Terraform state replacements, and finally delete the old bucket.
In this post you will learn how to rename an AWS S3 bucket in <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a>. First things first &#8211; let’s say you have a bucket definition in your Terraform code:

and you want to change the name of the bucket to <code>new-name</code>.

Firstly, we’ll need to <a href="https://spacelift.io/blog/terraform-aws-s3-bucket" target="_blank" rel="noopener">create a new bucket</a>. You can do this using the AWS CLI or the <a href="https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/console-help-gsg.pdf" target="_blank" rel="nofollow noopener">AWS console</a>. Just make sure to properly replicate the old settings, especially the ACL (so that your data doesn’t accidentally become public).
Now, we can copy all the files from the old to the new bucket:

Now that we have our new bucket, we need to remove the old one from our Terraform state and <a href="https://spacelift.io/blog/importing-exisiting-infrastructure-into-terraform" target="_blank" rel="noopener">import</a> the new one in its place:

If you tried to run Terraform now, it would show you that there’s drift—and yes, there would be! We’ve just imported a bucket into a resource which still has <code>old-name</code> in the config.

That’s why we now have to finally change the name of the bucket in our Terraform config:

If you run Terraform now, you’ll see that there are no changes to be made.

<ul>
<li><a href="https://spacelift.io/blog/terraform-best-practices" target="_blank" rel="noopener">12 Best Practices for Using Terraform</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener">How to Improve Your Infrastructure as Code using Terraform</a></li>
</ul>

If you want to, you can now delete your old bucket using:

Make sure all the data has successfully been copied over to the new bucket.

If you’re using Spacelift then you can use <a href="https://docs.spacelift.io/concepts/run/task" target="_blank" rel="noopener">tasks</a> to run the above arbitrary AWS CLI commands. The default runner image already contains the AWS CLI, so no changes are necessary there.
One additional thing you can do is <a href="https://docs.spacelift.io/concepts/stack/stack-locking" target="_blank" rel="noopener">lock the Stack</a> while you’re running the migration, this way nobody will accidentally run Terraform or make any other changes while you’re moving the data.
And that’s it! If you have any questions about how to rename an AWS S3 bucket, drop me a line in the comments and I’ll get back to you straight away.

Terraform Taint, Untaint, Replace - How to Use It (Examples)

Learn how to rename an AWS S3 bucket in Terraform. See renaming process of Amazon s3 bucket in a few simple steps.

8-rename-aws-s3-bucket

How to rename aws s3 bucket

How to Rename an AWS S3 Bucket in Terraform - Step by Step

Mudassar Shafique

If you have heard of Terraform but are not exactly sure what it is or when to use it, you have landed at the right place. This post will cover it all starting off with defining Terraform.
<dfn>Terraform</dfn> is a powerful open-source Infrastructure as Code (IaC) software tool offered by HashiCorp. It facilitates provisioning and managing your infrastructure on-prem and in the cloud. As an open-source tool it can be easily extended with the help of its plugin-based architecture.

Terraform can be connected with different infrastructure hosts and achieve complex management scenarios and compliance across multiple clouds. Its configuration can be easily packaged, shared, and reused in the form of <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a>.
In this post we will go over some basic elements and features of Terraform that can help you get started. Before we move forward, however, let’s discuss Infrastructure as Code (IaC) a bit more to develop a good understanding of Terraform.

Traditionally, the process of preparing a Windows or Linux server for your lab or data center consisted of several steps involving mouse clicks, running scripts, and manually configuring different aspects such as storage and networking. Doing this at scale for hundreds or thousands of servers becomes increasingly complex and error-prone.
<a href="https://www.redhat.com/cms/managed-files/pa-terraform-and-ansible-overview-f14774wg-201811-en.pdf" target="_blank" rel="noopener">IaC</a> makes this process much easier by allowing you to define and provision your infrastructure with the help of a declarative language. With this approach, you can gain consistency, repeatability, and enhanced reliability. 
In addition to day one infrastructure setup, this approach also helps with the management of any changes that happen to your infrastructure along the way, commonly referred to as drift management. (If you’re interested in how to manage Terraform workflows at scale, see: <a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">How to Manage Terraform at Scale</a>)
Other common IaC tools include Cloud Formation Templates (CFT) for AWS, Azure Resource Manager (ARM) templates for Microsoft Azure, and Deployment Manager for Google cloud.

At a high level, Terraform can be thought of as consisting of two main parts: Terraform Core and Plugins. Core is responsible for life cycle management of infrastructure. It is the open-source binary that you download and use from the command line. 
Terraform Core takes into consideration the current state and evaluates it against your desired configuration. It then proposes a plan to add or remove infrastructure components as needed. Next, it takes care of provisioning or decommissioning any resources if you choose to apply the plan.
Terraform Plugins provide a mechanism for Terraform Core to communicate with your infrastructure host or SaaS providers. Terraform Providers and Provisioners are examples of plugins as mentioned above. Terraform Core communicates with the plugins via Remote Procedure Call (RPC).

terraform-definition

terraform definition

A typical Terraform workflow will consist of the following steps:
<h3>Step 1. Write </h3>
Declare your infrastructure resources as code in Hashicorp Configuration Language (HCL). 
<h3>Step 2. Review the Plan</h3>
Terraform will display a plan to add or remove resources based on the comparison of your declared infrastructure and the current state of any existing resources.
<h3>Step 3. Apply</h3>
Accept the planned changes to add or remove any infrastructure resources. This workflow works well for a single author as Terraform State is stored locally. As more contributors join the team, it becomes important to ensure everyone has a correct view of the current infrastructure state and changes are applied sequentially without overlaps. 
Terraform offers Enterprise and Cloud versions for teams to work together. These store the state remotely so everyone can have a consistent view. Terraform Cloud is available as a hosted service, whereas Terraform Enterprise enables you to deploy and self-host the Terraform Cloud in your private instance. Authors contribute their code to a central version control system (e.g. Github) and Terraform Cloud uses the latest code versions from the central repository.

What is Terraform? What is Terraform used for? This guide has all the answers plus Terraform definition.

How to Manage Active Directory Objects with Azure AD Provider for Terraform

Get the newsletter

Recent posts

Terraform Data Sources – How They Are Utilised

Open Policy Agent: What Is OPA and How It Works (Examples)

Provisioning AWS EKS Cluster with Terraform – Tutorial

Importing Existing Infrastructure into Terraform – Step by Step

How to Rename an AWS S3 Bucket in Terraform – Step by Step

What is Terraform? What is it Used For, How it Works (Examples)