Author title

OpenTofu is now part of the Linux Foundation 🎉

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Terraform

Sumeet Ninawe

Terraform has redefined the way we manage and provision our cloud resources. With its declarative approach, Terraform enables us to define our desired infrastructure state and effortlessly bring it to life across various cloud providers. Data sources are a fundamental concept that enables Terraform to gather information from existing resources and incorporate it into the configuration. 
In this post, we delve into their significance, mechanics, and applications, uncovering how they enhance the precision and adaptability of infrastructure management.
We will cover:
<ol>
<li><a href="https://spacelift.io/blog/terraform-data-sources-how-they-are-utilised#what-is-a-data-source-in-terraform">What is a data source in Terraform?</a></li>
<li><a href="https://spacelift.io/blog/terraform-data-sources-how-they-are-utilised#how-to-use-data-sources-in-terraform">How to use data sources in Terraform?</a></li>
<li><a href="https://spacelift.io/blog/terraform-data-sources-how-they-are-utilised#terraform-data-source-examples">Terraform data source examples</a></li>
<li><a href="https://spacelift.io/blog/terraform-data-sources-how-they-are-utilised#refreshing-data-sources">Refreshing data sources</a></li>
</ol>

Data sources in Terraform are used to get information about resources external to Terraform, and use them to set up your Terraform resources. For example, a list of IP addresses a cloud provider exposes. Data sources serve as a bridge between the current infrastructure and the desired configuration, allowing for more dynamic and context-aware provisioning.
Organizations &#8211; especially the ones who have begun IaC adoption &#8211; often provision their infrastructure via multiple processes. They typically have a pre-existing set of cloud resources and architectures deployed. When new cloud components are being developed using IaC, data sources assist in querying various attributes of the pre-existing cloud resources. This is very useful from the point of view of IaC adoption and dynamic infrastructure provisioning.
Even when all the infrastructure components are being developed using IaC, the components are independently developed based on logical separation. 
For example, a particular IaC repository may have resource declarations for database and storage, while others may have declarations for compute resources. Thus, multiple infrastructure components are provisioned using separate Terraform projects. 
In such cases as well, data sources play a crucial role in sharing the resource information, which is made available only after provisioning is complete. 
For example, compute resources may depend on database connection string so that the applications deployed on them can perform I/O operations. The database connection string, credentials, and similarly other such resource attributes are made available using data sources.
Before we move on with the use case examples, let us understand some differences between data sources and a few other aspects to avoid confusion.
<h3>What is the difference between variables and data in Terraform?</h3>
<a href="https://spacelift.io/blog/how-to-use-terraform-variables" target="_blank" rel="noopener">Terraform variables</a> and data sources serve distinct purposes within the infrastructure provisioning process. Variables are used to parameterize your Terraform configurations. They allow us to define reusable values that can be customized for different environments or deployments. Variables are typically used to make our configurations more flexible and easier to manage by centralizing values that might change based on the context.
Data sources are used to retrieve information from external systems or existing resources and incorporate that information into our configuration. Data sources provide dynamic and context-aware attributes that can be used within our resource definitions. They help us make your configurations more intelligent and adaptable by leveraging information from the real world.
<h3>What is the difference between import and data source in Terraform?</h3>
The <a href="https://spacelift.io/blog/importing-exisiting-infrastructure-into-terraform">&#8220;import&#8221; command</a> in Terraform allows us to bring existing resources under Terraform management. It helps us incorporate resources that were created outside of Terraform into our Terraform state so that we can manage them using Terraform moving forward.
A data source is used to query information from external systems or existing resources and incorporate that information into our Terraform configuration. It provides dynamic attributes that can be used to make our configurations context-aware and responsive to changes. Data sources only query for the information and do not move the resources under Terraform management.
<h3>What is the difference between resources and data sources in Terraform?</h3>
Resources in Terraform represent the infrastructure components we want to create, manage, or delete. They define the desired state of a particular resource type, such as virtual machines, databases, networks, and more. Resources are the building blocks of our infrastructure and directly interact with our cloud provider&#8217;s APIs to create or modify actual resources.
Data sources are used to query information from existing resources or external systems. They provide a way to fetch specific attributes or data that we need to incorporate into our Terraform configuration. Data sources do not create or manage resources; they retrieve information to inform the configuration of your resources.
<h3>Terraform data sources vs. locals</h3>
<a href="https://spacelift.io/blog/terraform-locals" target="_blank" rel="noopener">Local variables in Terraform</a> are used to store and reuse values within our Terraform configuration. They provide a way to keep our configuration DRY (Don&#8217;t Repeat Yourself) by defining a value in one place and referencing it in multiple locations. They are defined within a module or a single file to store values such as strings, numbers, lists, or maps, thus making our configuration more readable and maintainable.
Data sources are used to query external systems or existing resources for information that we need to incorporate into our configuration. They provide dynamic attributes that can enrich our configuration with real-world data. Data sources retrieve real-world data, such as information about existing resources, cloud provider metadata, or external system details.

To query the data from the pre-existing real-world resources in Terraform, we use a special data resource, as shown below. 
Here, the <code>provider_type</code> represents the type of the object we want to query. 
For example, for querying information about VPCs, this <code>provider_type</code> would be “aws_vpc”, as seen in the <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc" target="_blank" rel="nofollow noopener">documentation</a>.

The name is chosen by us to differentiate between various data sources being queried and when referring them elsewhere in the configuration. Within this resource block, we provide multiple configuration options that help us specify filter conditions to fetch the right data. 
For example, we may have multiple VPCs provisioned in our AWS account, but we are interested in a specific VPC. Assuming we know the ID of that VPC, we can provide the same using the “id” attribute. We can then use the data returned by this data source &#8211; information about the VPC &#8211; to configure other resources in our configuration.
The documentation in the <a href="https://registry.terraform.io/">Terraform registry</a> maintains the list of all the data sources along with configuration options per resource, which is used to fetch the information. Let us explore various use cases of using data source resource.

<ul>
<li><a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Deploy your Infrastructure in CI/CD using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Deploy your Infrastructure in CI/CD using Terraform</a></li>
<li><a href="https://spacelift.io/blog/drift-detection" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Drift Detection and How to Fix It&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Infrastructure Drift Detection and How to Fix It</a></li>
<li><a href="https://spacelift.io/blog/github-actions-terraform" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Managing Terraform with GitHub Actions &amp; How to Scale&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Managing Terraform with GitHub Actions &amp; How to Scale</a></li>
</ul>

All the examples discussed in this post are available in this <a href="https://github.com/sumeetninawe/tf-datasources" target="_blank" rel="nofollow noopener">GitHub repository</a>.
<h3>Example 1: Using data sources to access external resource attributes</h3>
Let us assume we need to fetch some details like bucket id, ARN, region, and domain name of a specific S3 bucket from our AWS account. To get the same, we use the “aws_s3_bucket” data resource and provide the name of the bucket in the “bucket” attribute, as shown below.

I have a bucket named “sumeet.life” in my AWS account. It also has the static website hosting enabled to host a basic static website. 
When this Terraform configuration with appropriate provider settings is initialized and enabled, Terraform reads the information from AWS and makes it available in the “data.aws_s3_bucket.existing_bucket” variable.
To access the desired information, let us create a few output variables like so:

As far as the attributes listed above are concerned, they are available for all the buckets, even if website hosting is not enabled. 
The <code>bucket_domain_name</code> attribute is returned by AWS by simply joining the two strings &#8211; “bucket_name” and “s3.amazonaws.com”. 
However, there are certain attributes that are only available for static site hosting enabled S3 buckets like:

If the static site hosting is not enabled for a given S3 bucket, it will simply not return the endpoint attribute. 
Running <a href="https://spacelift.io/blog/terraform-apply" target="_blank" rel="noopener"><code>terraform apply</code> command</a> returns and displays all the output variables, as shown below.

<a href="https://github.com/sumeetninawe/tf-datasources/tree/main/1" target="_blank" rel="nofollow noopener">Example 1 source</a>
<h3>Example 2: Using data from remote state files</h3>
This special data source is used to read information from remote state files. When infrastructure is managed using Terraform configuration, Terraform stores the information about managed resources in a state file remotely. The “terraform_remote_state” data source is used to query information from this state file.
Let us assume that the VPC and networking configuration is being managed in a different Terraform project, and its state file is being managed in the S3 bucket. We want to consume the information about this VPC, its subnets, and maybe <a href="https://spacelift.io/blog/terraform-security-group" target="_blank" rel="noopener">security groups in our Terraform configuration</a>. To do this, we first declare the data resource as below.

Here we specify the type of remote backend to be queried along with the configuration options. It should be noted that only the information exposed in the “output” variable can be accessed using this data source. The VPC project has to explicitly declare output variables so that they are available in our project.
Note: the source code for VPC configuration is available in <a href="https://github.com/sumeetninawe/tfremotestate-vpc" target="_blank" rel="nofollow noopener">this GitHub repository</a>.
To access the information like subnet ids, and subnet names, we declare the output variables as shown below. These variables essentially access the output variables exposed by the VPC project from their state file in the format below.

When this Terraform configuration is initialized and applied, the subnet information is printed as below.

It is important to note that the usage of this data source is discouraged due to various security concerns, one of which we will see in the next example. The main reason is that the user credentials used to access these output variables inherently have access to the complete state file, which may contain sensitive information.
Instead, it is recommended to explicitly publish the data in a separate application (like Consul) or use data sources as demonstrated in Example 1.
<a href="https://github.com/sumeetninawe/tf-datasources/tree/main/2" target="_blank" rel="nofollow noopener">Example 2 source</a>
Read also <a href="https://spacelift.io/blog/terraform-aws-vpc" target="_blank" rel="noopener">how to build AWS VPC with Terraform</a>.
<h3>Example 3: Using data sources to access sensitive data from remote state</h3>
As mentioned in the previous example, to further demonstrate why using <code>terraform_remote_state</code> is not a good way to access data &#8211; in this example we access the sensitive information exposed in Terraform state files.
Even when variables in Terraform configuration are marked as sensitive, they are not secure in transit or at rest. These sensitive values are stored and transmitted in plain text. The sensitive attribute is only used to mask these values when being printed in logs.
Let us assume that the VPC information we are trying to access has a CIDR range exposed which is considered as sensitive information. When we try to print the state information of this VPC, from the VPC project Terraform CLI, it masks the CIDR range, as shown below.

However, when we try to access the same from our project using <code>terraform_remote_state</code> data source, it is printed in plain text.

Initialize and run terraform apply.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:

Sensitive values are not completely secure and need a more holistic approach to manage the security around them.
<a href="https://github.com/sumeetninawe/tf-datasources/tree/main/1" target="_blank" rel="nofollow noopener">Example 3 source</a>
<h3>Example 4: Using data sources to access AWS secrets</h3>
Secrets Manager is a service provided by AWS to manage sensitive data. It stores the sensitive variables securely using encryption, and makes them available to various services for automation purposes.
In Terraform configurations, these secrets are accessed using data sources. 
In this example, we provision an RDS Postgres database instance and set its master username and password attributes by fetching those values from the Secrets Manager service. The data resources below fetch the secret values and make them available for the RDS configuration.

We then use these secret variables to set the username and password in the <code>aws_db_instance</code> resource as below. 
Here we are using <code>jsondecode()</code> Terraform function to decode the secret string, and use the username and password attributes to access the corresponding values.

This is a secure way to manage secrets as they are not made available in plaintext anywhere in the provisioning process. To further prove this point, let us also try to output these values using output variables as below.

Initialize and apply this Terraform configuration.
 As we can see, although we are trying to print these secret values in the output variables, they are still masked.

<a href="https://github.com/sumeetninawe/tf-datasources/tree/main/4" target="_blank" rel="nofollow noopener">Example 4 source</a>
<h3>Example 5: Creating dynamic configuration with data sources</h3>
Terraform data sources can retrieve critical details from various cloud providers, like cloud provider metadata, databases, or APIs, enabling the configuration to adapt to real-world changes in the current Terraform project. 
By seamlessly integrating these data sources, Terraform deployments become more flexible and responsive. This approach streamlines the management of resources in a dynamic manner.
In our example, we know that the VPC and networking configurations are managed in a different Terraform project. Let us assume that our Terraform configuration is expected to provision and manage compute resources. We have been tasked to create an EC2 instance in each subnet of the VPC. 
To do this, we first use a data source to get all the relevant subnets for the given VPC as shown below.

Next, we use <code>aws_instance</code> resource along with <a href="https://spacelift.io/blog/terraform-for-each" target="_blank" rel="nofollow noopener">Terraform for_each meta-argument</a> to dynamically create EC2 instances in the appropriate subnets. 
Depending on the subnets retrieved from the above data source, the corresponding number of EC2 instances will be created, and at the same time, they will be placed in each subnet.

This introduces the dynamic nature of data sources in a way that if more subnets are added to the VPC in their project, an appropriate number of EC2 instances will be added to them in the next Terraform apply run of our current project. 
Initialize and apply this Terraform configuration, and verify the same by logging on to the AWS web console.

terraform-test-data-source

terraform test data source

terraform-print-data-source-value

terraform print data source value

Check out also how to create an <a href="https://spacelift.io/blog/terraform-ec2-instance" target="_blank" rel="noopener">AWS EC2 instance using Terraform</a>.
Optionally, we can also use <code>terraform_remote_state</code> data source to query the state file of the VPC project. 
The information about the subnet is exposed by a <a href="https://spacelift.io/blog/terraform-output" target="_blank" rel="noopener">Terraform output variable </a> that provides a list of subnet IDs (<code>list(string)</code>), as shown below.

Modify the configuration for <code>aws_instance</code> to adopt this data source in the for_each meta-argument.

This would get the same result, making the code dynamic.
<a href="https://github.com/sumeetninawe/tf-datasources/tree/main/5" target="_blank" rel="nofollow noopener">Example 5 source</a>
<h3>Example 6: Managing resource dependencies with data sources</h3>
Data sources indirectly help manage resource dependencies. If the data being queried by data sources does not exist, then the resource that is dependent on the same will not be created. 
In the previous example, when we query for the subnets using <code>aws_subnets</code> data source, the VPC id provided in the filter is a real VPC &#8211; it exists. Thus it returns all the subnets (2) belonging to this VPC, and two EC2 instances are created.
However, if we provide a VPC ID that does not exist, then these resources will simply not be created. 
Modify the data source as shown below.

Try to run the <a href="https://spacelift.io/blog/terraform-plan" target="_blank" rel="noopener"><code>terraform plan</code> command</a> to the rest of the Terraform configuration and see the output. 
The output should look like this:

Data sources thus inherently manage the dependencies for creating dynamic resources dependent on the existence of the data.
<a href="https://github.com/sumeetninawe/tf-datasources/tree/main/6" target="_blank" rel="nofollow noopener">Example 6 source</a>
<h3>Example 7: Validate inputs with data sources</h3>
This is an extension of the observation made in the previous example. 
In the previous example, we used a meta-argument for_each, which works on the input received from the data source. If there is no data returned (no valid filter applied), then resource creation is simply skipped without throwing any error.
However, if we don’t want this to be ignored and instead throw an error during the plan stage, we can use a data source between the variable and EC2 instance resource block. In the example below, we have defined a variable to provide an invalid AMI value to <code>aws_instance</code> resource block.

The Terraform plan output does not throw any error.

However, it does throw an error when we proceed to apply this configuration since the given AMI does not exist. 
In certain scenarios, it might be desirable for this configuration to throw an error in the planning phase itself for various reasons. This is achieved using data sources.
Adding the data source block below validates the existence of the given AMI value in the specified region in the provider.

The terraform plan output now:

<a href="https://github.com/sumeetninawe/tf-datasources/tree/main/7" target="_blank" rel="nofollow noopener">Example 7 source</a>
<h3>Example 8: Managing configuration drift</h3>
Configuration drift refers to the gradual misalignment between the intended state of an IT infrastructure and its actual state over time. It occurs in dynamically evolving environments, where manual changes, updates, or unauthorized modifications accumulate on various resources. 
Configuration drift can lead to inconsistencies, security vulnerabilities, and operational inefficiencies as the system deviates from its expected configuration. 
This article about <a href="https://spacelift.io/blog/terraform-drift-detection" target="_blank" rel="noopener">Terraform drift detection</a> describes how to automatically detect configuration drifts and restore the infrastructure state to what was originally intended using <a href="https://spacelift.io/" target="_blank" rel="noopener">Spacelift</a>. 
Data sources also help in managing configuration drifts in their own ways. To understand it, we first have to understand what configuration drift means in this context. The two scenarios below are considered relevant sources of configuration drift in this case.
Unplanned changes
If the infrastructure is changed due to some reason (human error), then the next Terraform apply run restores it to the original state. This is a general Terraform behavior and also applies to data sources.
Planned changes
If the infrastructure depends on the <code>terraform_remote_state</code> data source of another Terraform project, and the configurations in the source Terraform project are changed in a planned manner, then the current configuration should automatically adapt to those changes. This is an example of configuration drift caused due to planned changes.
Consider the following configuration. Here, our configuration depends on:
<ol>
<li style="font-weight: 400;" aria-level="1">Terraform remote state to get the security group information</li>
<li style="font-weight: 400;" aria-level="1">AWS Subnets data source, which queries for subnets directly from AWS</li>
</ol>
It creates EC2 instances in all subnets returned from the data source query and associates all those instances with a security group returned from a remote state.

To introduce unplanned changes in the infrastructure provisioned using the above configuration, let us login to the AWS management console and manually change the Security Group for one of the instances.
Original SG:

terraform-data-source-filter

terraform data source filter

Manually drifted to:

terraform-aws-data-source

terraform aws data source

Now if we run the <code>terraform plan</code>, it highlights the change to revert this drift as shown in the output below. 
Run <code>terraform apply</code> to revert this configuration drift.

As part of planned changes, let us assume that the team responsible to manage our VPC and networking configuration decided to add one more private subnet in their configuration. As per our requirement &#8211; “An EC2 instance needs to be provisioned in each subnet”.
Since we are using <code>aws_subnet</code> data source “which queries for all the subnets which exist in a given VPC”, if we run terraform plan, it automatically detects the creation of additional subnet and provides a plan to add an EC2 instance as shown below.

This makes sure that the implementation of requirements is consistent without any drifts.
<a href="https://github.com/sumeetninawe/tf-datasources/tree/main/8" target="_blank" rel="nofollow noopener">Example 8 source</a>

By default, Terraform will refresh all data sources before creating a plan. You can also explicitly refresh all data sources by running <a href="https://spacelift.io/blog/terraform-refresh" target="_blank" rel="noopener"><code>terraform refresh</code></a>.
Occasionally you’ll have data sources that change very often and would like to keep your resources in sync with those changes. The easiest way to achieve this is to just run Terraform every few minutes or so, do a refresh, and apply all resulting changes.
You could also use something akin to <a href="https://docs.spacelift.io/concepts/stack/drift-detection" target="_blank" rel="noopener">Spacelift’s Drift Detection</a> to automate this process and make sure it doesn’t interfere with your manual Terraform executions.

In this post, we have discussed in depth about data sources in Terraform and explored various use cases to leverage the same. By integrating external information from various sources, such as cloud providers&#8217; APIs, databases, and other systems, Terraform data sources enable configurations to be dynamically shaped according to real-time requirements. This capability ensures that infrastructure remains in sync with the ever-changing landscape of modern IT environments, reducing manual intervention and minimizing configuration drift.
Then if you need any help managing your Terraform infrastructure, building more complex workflows based on Terraform, and managing AWS credentials per run, instead of using a static pair on your local machine, check out <a href="https://spacelift.io/" target="_blank" rel="noopener">Spacelift</a>. It supports policy as code, programmatic configuration, context sharing, drift detection, and many more great features right out of the box. One of the things we’re most proud of at Spacelift is the deep integration with everyone’s favorite version control system – <a href="https://docs.spacelift.io/integrations/source-control/github.html" target="_blank" rel="nofollow noopener">GitHub</a>. You can check it for free, by <a href="https://spacelift.io/" target="_blank" rel="noopener">creating a trial account</a>.

If you are struggling with Terraform automation and management, check out Spacelift. It helps you manage Terraform state, build more complex workflows, and adds several must-have capabilities for end-to-end infrastructure management.

Initialize/ plan/ apply your IaC, manage modules, state, and more. 

frame-8479

Frame 8479

5 Ways to Manage Terraform at Scale - Best Practices

How Spacelift Can Improve Your Infrastructure as Code

Terraform Tutorial - Getting Started With Terraform on AWS

Learn what is a data source in Terraform, how data sources work, and how to use them. See the examples. Data sources in remote state explained.

13-terraform-data-sources

Terraform data sources

Terraform Data Sources - How They Are Utilized (Example)

Adam Connelly

<dfn>Open Policy Agent</dfn> is an open-source engine that provides a way of declaratively writing policies as code and then using those policies as part of a decision-making process. It uses a policy language called Rego, allowing you to write policies for different services using the same language.
OPA can be used for a number of purposes, including:
<ul>
<li style="font-weight: 400;" aria-level="1">Authorization of REST API endpoints.</li>
<li style="font-weight: 400;" aria-level="1">Allowing or denying Terraform changes based on compliance or safety rules.</li>
<li style="font-weight: 400;" aria-level="1">Integrating custom authorization logic into applications.</li>
<li style="font-weight: 400;" aria-level="1">Implementing Kubernetes Admission Controllers to validate API requests.</li>
</ul>
OPA was originally created by Styra, and is now part of the Cloud Native Computing Foundation (CNCF), alongside other CNCF technologies like Kubernetes and Prometheus.
In this article, I will give an overview of Open Policy Agent, explain why you would want to use it, as well as showcase how you can use OPA with your Spacelift account. Although OPA can serve many purposes, I’m going to focus on how it can be used alongside <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code</a>.

We can visualise how OPA works using the following diagram:

how-does-opa-work

Open Policy Agent (OPA) Example

As you can see, OPA accepts a policy, input and query, and based on that, generates a response. The input can be any valid JSON document, allowing OPA to integrate with any tool that produces JSON output.

In the following sections, I’ll go into more specific examples of using OPA that should help make things clearer but before I do, here’s a quick list of reasons why I’m interested in using OPA:
<ul>
<li style="font-weight: 400;" aria-level="1">Policies as code allow you to follow your standard development lifecycle with PRs, CI, etc., and provide you with a history of changes to your policies.</li>
<li style="font-weight: 400;" aria-level="1">OPA is designed to work with any kind of JSON input, meaning it can easily integrate with any tool that produces JSON output.</li>
<li style="font-weight: 400;" aria-level="1">Because OPA integrates with a number of different tools, it allows you to use a standard policy language across many parts of your system, rather than relying on multiple vendor-specific technologies.</li>
<li style="font-weight: 400;" aria-level="1">OPA supports unit-testing, making it easier and faster to iterate your policies with confidence that they won’t break.</li>
</ul>

Let’s try to make that a bit less theoretical by using a specific example: Terraform. Terraform can produce a plan in JSON format via the terraform show command. This means that we can define policies for our infrastructure, and use OPA to make a decision about whether a plan is safe to apply or not:

using-opa-with-terraform

using opa with terraform

For example, say we have the following Terraform definition to create an EC2 instance:

Now, say we want to ensure that every Terraform resource has a Name tag. We could enforce that by creating a file called plan.rego with the following content:

In order to use OPA to evaluate our policy, we need to take the following steps:
<ol>
<li>Generate a Terraform plan as JSON.</li>
<li>Run opa eval to verify whether that plan passes our policy or not.</li>
</ol>

<h3>Step 1 &#8211; Generate our Terraform plan as JSON</h3>
To get a JSON representation of our plan, we need to output our plan to a file, and then use the terraform show command to output that plan as JSON:

<h3>Step 2 &#8211; Run opa eval</h3>
We can then use opa eval to evaluate our plan against our policy:

As you can see, we’re using the query data.spacelift.allow because our policy is loaded as a data file, and we defined our allow rule in the spacelift namespace. You can also see that opa eval produced an empty output ({}). This means that our allow rule didn’t evaluate to true, and so produced no output.
Let’s adjust our Terraform definition to include a Name tag:

Now if we generate our plan and evaluate the policy again, we should get a slightly different output:

This tells us that the allow rule has evaluated true. We can get that in a slightly more concise way by using the pretty format option:

At this stage, you can probably imagine how you could integrate this into your <a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener">CI/CD pipeline </a>to enforce naming schemes, security rules, and various other organizational policies.

The following examples illustrate some possible use-cases for OPA with <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a>. All of the examples have been taken from the Spacelift <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy#cookbook" target="_blank" rel="noopener">plan policy documentation</a>, but have been altered to work with plain vanilla Terraform.
<h3>Example 1. Require human review when resources are deleted or updated</h3>
Adding new resources is usually a fairly safe operation, but updating or deleting existing resources can carry more risk. You could flag these for human review by using the following policy:

<h3>Example 2. Require commits to be reasonably sized</h3>
Once a PR goes over a certain size it becomes difficult to review without missing things. The same is true for Terraform plans. The following policy can be used to warn when the number of changes goes over a certain threshold:

<h3>Example 3. Blast radius</h3>
The following policy attempts to determine the risk of a particular plan by assigning different weightings to the change type (create, update, delete), along with the affected resource type (ECS cluster, EC2 instance, etc.). It takes the approach that an update or delete is more risky than a create because it affects an existing resource:

<ul>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;10 Ways to Improve Your Infrastructure as Code&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">10 Ways to Improve Your Infrastructure as Code</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
</ul>

How can we make sure that our policies work as we expect them to and that they don’t break over time as we make changes to them? You guessed it: unit testing! Luckily for us, OPA has first-class support for testing via the opa <code>test</code> command.
To create tests for our policy, all we need to do is create another Rego file with a series of rules prefixed with <code>test_</code>. Each rule starting with that prefix defines a separate test. 
Let&#8217;s go ahead and create a file called plan_test.rego, with the following contents:

As you can see, OPA makes it really easy to specify the policy input using the <code>&lt;variable&gt;</code> as <code>&lt;value&gt;</code> syntax. This allows us to create very concise tests by only including values we care about in the policy input, rather than having to use the entire plan output.
Let&#8217;s go ahead and run <code>opa test</code>:

Unsurprisingly our test passes. That&#8217;s not very exciting, so let&#8217;s add a new test to ensure our resources include an <code>Environment</code> tag:

Running opa test again shows us a failure:

We can fix this by updating our policy:

At this stage if you run the test command again it should show 2 passes:

If you want to know more about OPA testing, the <a href="https://www.openpolicyagent.org/docs/latest/policy-testing/" target="_blank" rel="nofollow noopener">official docs</a> are full of great examples and information about what you can do.

At this stage, hopefully, you’ve got a pretty good idea of what OPA is as well as how it can be useful to you. It’s not too difficult to see how you could start integrating OPA into your development process or even use it as part of production systems.
Luckily for you, at Spacelift we’ve done all the heavy lifting, allowing you to reap the benefits of using OPA for Policy-as-Code without you having to implement everything from scratch yourself. In this section, I want to showcase some of the functionality that Spacelift provides as it relates to OPA.
<h3>1) Policy Types</h3>
Spacelift allows you to use <a href="https://resources.sei.cmu.edu/asset_files/Presentation/2021_017_001_739363.pdf" target="_blank" rel="noopener">OPA policies</a> to manage various aspects of your Spacelift account, and not just during planning. For example, you can use policies to control who can log into your account, along with what they have access to. For more information, see <a href="https://docs.spacelift.io/concepts/policy/" target="_blank" rel="noopener">Policy as Code</a>.
<h3>2) PR Checks</h3>
Spacelift can automatically trigger planning runs whenever you push changes to your VCS provider. For example, here’s what you might see in GitHub after creating a PR:

pr-checks-1

Opa pr checks

If all goes well your check will succeed, but if a policy is violated, a failed check will be reported:

pr-checks-2

opa pr checks failed

You can then view the details of the failure in Spacelift:

pr-checks-3

failure details in Spacelift

<h3>3) Manual Approvals</h3>
Spacelift plan policies use a slightly different format from the one we used in our example policies earlier in this post. Not only do they allow you to specify a message to be displayed, but also have the concept of <code>deny</code> and <code>warn</code>:

The <code>deny</code> rule fails the run completely, while the <code>warn</code> rule just displays a warning in the logs.
<code>warn</code> rules take on another role when a run is going to deploy changes (vs just showing the planned changes against a PR). If any warnings are reported during a deployment, the run will wait for manual approval before applying any changes.
Let’s use the following example policy, designed to warn if a certain set of suggested tags isn’t found within our resources:

If we then attempt to add resources that don’t contain all of those tags, Spacelift will block before applying the changes, and give us the chance to manually approve or deny the run:
&nbsp;

manual-approvals

manual approvals

This allows you to build complex workflows where certain changes are completely blocked, while others are allowed as long as the changes are reviewed first.
Check out our <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy#cookbook" target="_blank" rel="noopener">plan policy cookbook</a> for more ideas about what you can do.

Spacelift provides a <a href="https://registry.terraform.io/providers/spacelift-io/spacelift/latest/docs" target="_blank" rel="nofollow noopener">Terraform provider</a> for managing your Spacelift account. This means that you can manage the policies available within your account, along with the Stacks they apply to in code.
For example, to add the policy we defined earlier to Spacelift we can use the <code>spacelift_policy</code> resource like this:

We can then attach this policy to a Spacelift Stack using the <code>spacelift_policy_attachment</code> resource:

Taking this a step further, we could create a custom module for defining Spacelift Stacks that ensured that all Stacks had a certain set of policies attached by default.

There are a number of exciting tools available that can help integrate OPA with your systems, for example Styra DAS. Styra provides a way of writing policies and deploying them across your infrastructure, as well as providing tools for unit testing, monitoring policy usage and more. You can try OPA at scale for free using Styra DAS Free.

I hope you’ve enjoyed this post, and can see the value that OPA brings to the table. If you’re interested in trying out <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">Spacelift to see what we have to offer, why not sign up for a free trial</a>? You can set up a Spacelift account in minutes, and get started on your Policy as Code journey today!

Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation and controls to simplify and accelerate the provisioning of cloud based infrastructures.

Importing Existing Infrastructure into Terraform - Step by Step

What is Open Policy Agent (OPA)? See Open Policy Agent examples and a full tutorial on when to use it.

12-opa

Open Policy Agent

Open Policy Agent: What Is OPA and How It Works

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Product

In this post I want to show how easy it is to create a <a class="dt id" href="https://spacelift.io" target="_blank" rel="noopener">Spacelift</a> Stack that manages Azure resources. To keep things simple, I’m going to assume that you already know what Spacelift can do, and the basics of creating and running Stacks, but if not check out our <a class="dt id" href="https://github.com/spacelift-io/terraform-starter" target="_blank" rel="nofollow noopener">terraform-starter</a> repo that walks you through the process of setting up an account, and getting started.

In order to allow Spacelift to manage Azure resources, we need to do the following:
<ul class="">
<li id="f0f8" class="hj hk fo hl b hm jf hn ho hp jg hq hr hs jh ht hu hv ji hw hx hy jj hz ia ic jk jl jm gk" data-selectable-paragraph="">Create a Service Principal in Azure with the correct ARM permissions to manage our resources.</li>
<li id="8f2c" class="hj hk fo hl b hm jn hn ho hp jo hq hr hs jp ht hu hv jq hw hx hy jr hz ia ic jk jl jm gk" data-selectable-paragraph="">Create a Stack in Spacelift that uses the <a class="dt id" href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs" target="_blank" rel="nofollow noopener">Azure</a> provider.</li>
<li id="ba4d" class="hj hk fo hl b hm jn hn ho hp jo hq hr hs jp ht hu hv jq hw hx hy jr hz ia ic jk jl jm gk" data-selectable-paragraph="">Give our Stack the credentials for our Service Principal.</li>
</ul>
All of the information you need to configure the Azure provider can be found <a class="dt id" href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure" target="_blank" rel="nofollow noopener">here</a>. This post just explains how to combine that information with Spacelift.

To create our Service Principal, we need to use the <code class="js jt ju jv jw b">az ad sp create-for-rbac</code> command (replacing <code class="js jt ju jv jw b">&lt;subscription-id&gt;</code> with your actual subscription ID):

This will output something like the following:

The command creates a new Service Principal called <code class="js jt ju jv jw b">spacelift-sp</code>, and grants it the Contributor role on your subscription. It also outputs the <code class="js jt ju jv jw b">appId</code>, <code class="js jt ju jv jw b">password</code> and <code class="js jt ju jv jw b">tenant</code> for the Service Principal. Make a note of these because you&#8217;ll need them later.
If you would rather assign permissions yourself later, you can run the following command to just create a Service Principal with no permissions:

You should now be able to view your Service Principal in the Azure Active Directory -&gt; App registrations section of the Azure Portal:

default directory

Now that we’ve got a service principal, we need to create a Stack to actually create some Azure resources. To keep things simple all we’re going to do is add some terraform definitions that specify the version of the Azure Provider we want to use, and create an Azure Resource Group:

If you commit that change and try to run your Stack just now, you should see something similar to the following failure:

adding Azure resources

Here’s the full error message:
Error building AzureRM Client: obtain subscription() from Azure CLI: Error parsing json result from the Azure CLI: Error launching Azure CLI: exec: “az”: executable file not found in $PATH
At first glance this error is slightly confusing, but what it means is that because we didn’t provide any credentials to the Azure provider it attempted to fallback to using the <code class="js jt ju jv jw b">az</code> CLI for authentication. This works fine locally, but won&#8217;t work on Spacelift.
To solve this, we need to provide our stack with our credentials, which is explained in the next section.

<ul>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Improve Your Infrastructure as Code using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Improve Your Infrastructure as Code using Terraform</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
<li><a href="https://spacelift.io/blog/terraform-environments" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Managing Multiple Terraform Environments Efficiently&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Managing Multiple Terraform Environments Efficiently</a></li>
</ul>

If you take a look at the <a class="dt id" href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure" target="_blank" rel="nofollow noopener">Terraform documentation</a>, you’ll find that there are a number of different ways that you can authenticate <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> with Azure. What we’re going to do here is authenticate using a Service Principal with a Client Secret, and provide that configuration via <a href="https://spacelift.io/blog/how-to-use-terraform-variables" target="_blank" rel="noopener">environment variables</a>.
To do this we need to set the following environment variables, using the information returned by the Azure CLI when we created our Service Principal:
<ul class="">
<li id="1508" class="hj hk fo hl b hm jf hn ho hp jg hq hr hs jh ht hu hv ji hw hx hy jj hz ia ic jk jl jm gk" data-selectable-paragraph="">ARM_CLIENT_ID — the <code class="js jt ju jv jw b">appId</code> from the response.</li>
<li id="4e7d" class="hj hk fo hl b hm jn hn ho hp jo hq hr hs jp ht hu hv jq hw hx hy jr hz ia ic jk jl jm gk" data-selectable-paragraph="">ARM_CLIENT_SECRET — the <code class="js jt ju jv jw b">password</code> from the response.</li>
<li id="8a57" class="hj hk fo hl b hm jn hn ho hp jo hq hr hs jp ht hu hv jq hw hx hy jr hz ia ic jk jl jm gk" data-selectable-paragraph="">ARM_SUBSCRIPTION_ID — your subscription ID.</li>
<li id="7340" class="hj hk fo hl b hm jn hn ho hp jo hq hr hs jp ht hu hv jq hw hx hy jr hz ia ic jk jl jm gk" data-selectable-paragraph="">ARM_TENANT_ID — the <code class="js jt ju jv jw b">tenant</code> from the response.</li>
</ul>
You can set these environment variables directly on your Stack, but what we’re going to do here is create a <a class="dt id" href="https://docs.spacelift.io/concepts/configuration/context" target="_blank" rel="noopener">Spacelift Context</a> and attach it to our Stack. This allows us to share the same credentials with multiple Stacks if we want.
Add a new Context via the Contexts screen in Spacelift:

3-2-2

adding contexts

Once your context has been created, edit it and enter your environment variables, making sure to mark the <code class="js jt ju jv jw b">ARM_CLIENT_SECRET</code> as a secret rather than plaintext:

environment variables

Now that we’ve got a Context containing our credentials, you can attach it to your Stack. To do this, click on the Manage link next to your Stack, go to Contexts, choose your Context, and click Attach:

5-2-2

Azure stack

Now that your Context is attached, you should be able to see the it in the Environment section of your Stack:

Azure credentials

If you try to run your Azure Stack again you should get a successful plan:

running Azure Stack again

applying

If you take a look in the Azure Portal, you should now be able to see the Resource Group that was created:

Azure resource groups

Before finishing up, I wanted to just point out a few things about the Service Principal credential expiry. Although this isn’t really related to Spacelift, it’s something that you need to be aware of.
When you create your Service Principal via the Azure CLI, it automatically creates a Client Secret for you with an expiry of 1 year:

Spacelift Client Secret

What this means is that if you do absolutely nothing, your Spacelift Stacks will stop working after the credentials expire a year later. To solve this, all you need to do is generate a new Client Secret, and update the <code class="js jt ju jv jw b">ARM_CLIENT_SECRET</code> environment variable in your Context.

Hopefully, in this post, I’ve shown that although Spacelift doesn’t have an official Azure integration (yet!), it’s simple to manage Azure resources using Spacelift. Once you’ve performed a little bit of initial setup, you should be able to rapidly get new Stacks up and running, creating Azure resources.

Spacelift is a sophisticated SaaS product for Infrastructure as Code that helps DevOps develop and deploy new infrastructures or changes quickly and with confidence.

Azure Cost Management - Best Practices & Tools

It is easy to manage Azure resources using Spacelift. After a bit of initial setup, you will rapidly get new Stacks up and running, creating Azure resources.

11-managing_azure_resources_using_spacelift

Managing Azure Resources Using Spacelift

Managing Azure Resources using Spacelift

provision an AWS EKS Kubernetes cluster with Terraform

Want to level up your Kubernetes knowledge? Explore our tutorials and step-by-step guides written by Kubernetes experts. Learn Kubernetes best practices and solutions to the most difficult problems.

Kubernetes Guides and Tutorials | Spacelift Blog

Kubernetes

Jacob Martin

AWS EKS provides managed <a href="https://spacelift.io/blog/kubernetes-cluster" target="_blank" rel="noopener">Kubernetes clusters</a> as a service. You’re on AWS and want to avoid getting into the details of setting up a Kubernetes cluster from scratch? EKS is the way to go!
In this guide, you will learn how to provision an AWS EKS Kubernetes cluster with Terraform. Let&#8217;s start with the basics.

First, locally <a href="https://spacelift.io/blog/how-to-install-terraform" target="_blank" rel="noopener">install Terraform</a>:

as well as the AWS CLI:

and kubectl:

If you’re on a different operating system, please find the respective installation instructions here:
<ul>
<li style="font-weight: 400;" aria-level="1">Terraform: <a href="https://learn.hashicorp.com/tutorials/terraform/install-cli" target="_blank" rel="nofollow noopener">https://learn.hashicorp.com/tutorials/terraform/install-cli</a></li>
<li style="font-weight: 400;" aria-level="1">awscli: <a href="https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html" target="_blank" rel="nofollow noopener">https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html</a></li>
<li style="font-weight: 400;" aria-level="1">kubectl: <a href="https://kubernetes.io/docs/tasks/tools/" target="_blank" rel="nofollow noopener">https://kubernetes.io/docs/tasks/tools/</a></li>
</ul>

Now, you’ll need to configure your AWS CLI with access credentials to your AWS account. You can do this by running

and providing your Access Key ID and Secret Access Key. You will also need to add the region. For the purposes of this guide, we will use us-east-2. Terraform will later use these credentials to provision your AWS resources.

You can now clone a repository which contains everything you need to set up EKS:

Inside you’ll see a few files, the main one being eks-cluster.tf:

It uses the EKS Terraform module to set up an EKS cluster with 2 worker groups (the actual nodes running your workloads): one with a single medium machine, and one with two small machines.

You can now create all of those resources using Terraform. First, run

to initialize the <a href="https://spacelift.io/blog/terraform-workspaces" target="_blank" rel="noopener">Terraform workspace</a> and download any <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">modules</a> and providers which are used.
In order to do a dry run of the changes to be made, run

This will show you that 51 resources will be added, as well as their relevant details. You can then run terraform apply with the resulting plan, in order to actually provision the resources:

This may take a few minutes to finish. You might get a “timed out” error, in which case just repeat both the terraform plan and terraform apply steps.
In the end you will get a list of outputs with their respective values printed out. Make note of your cluster_name.

In order to use kubectl, which is the main tool to interact with a Kubernetes cluster, you have to give it credentials to your EKS Kubernetes cluster. You can do this by running

Make sure to replace &lt;output.cluster_name&gt; with the relevant value from your Terraform apply outputs.

<ul>
<li><a href="https://spacelift.io/blog/terraform-environments" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Managing Multiple Terraform Environments Efficiently&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Managing Multiple Terraform Environments Efficiently</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale</a></li>
</ul>

You can now view the nodes of your cluster by running

This command is so long because it displays custom columns, thanks to which we can indeed see that there are 2 smaller nodes, and 1 bigger node.
Let’s deploy an Nginx instance to see if the cluster is working correctly.

You can see the status of it by running:

And finally set up a tunnel from your computer to this pod:

If you open <a href="http://localhost:3000" target="_blank" rel="nofollow noopener">http://localhost:3000</a> in your browser, you should see the web server greet you:

welcome to ngix

In order to destroy the resources we’ve created in this session, run

This may again take up to a few minutes. You can read more about it here: <a href="https://spacelift.io/blog/how-to-destroy-terraform-resources" target="_blank" rel="noopener">How to Delete Resources from Terraform Using the Destroy Command.</a>

I hope this guide helped you on your Kubernetes journey on AWS! If you want more help managing your Terraform state file, building more complex workflows based on Terraform, and <a href="https://spacelift.io/blog/self-service-infrastructure-with-spacelift" target="_blank" rel="noopener">create self-service infrastructure</a>, check out <a href="https://spacelift.io" target="_blank" rel="noopener">Spacelift</a>. We’d love to have you!

Spacelift helps manage Terraform state, build more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and many more.

Learn how to provision an AWS EKS Kubernetes cluster with Terraform. See a full step-by-step tutorial.

Terraform Data Sources – How They Are Utilized

Get the newsletter

Recent posts

Terraform Data Sources – How They Are Utilized

Open Policy Agent: What Is OPA and How It Works (Examples)

Managing Azure Resources using Spacelift

Provisioning AWS EKS Cluster with Terraform – Tutorial

Importing Existing Infrastructure into Terraform – Step by Step

How to Rename an AWS S3 Bucket in Terraform – Step by Step

Alternative to Atlantis – Spacelift

What is Terraform? What is it Used For, How it Works