Adam is a backend software engineer at Spacelift. He has a background in full-stack web development, infrastructure automation, and deploying applications to Azure.

All posts by Adam Connelly - Infrastructure as Code Expert

Adam Connelly

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Running Spacelift CICD Workers in Kubernetes Using DinD

Check out the Spacelift Engineering blog to discover internal technical architecture posts written by Spacelift's very own engineers. 

Engineering | Spacelift Blog

Engineering

Want to level up your Kubernetes knowledge? Explore our tutorials and step-by-step guides written by Kubernetes experts. Learn Kubernetes best practices and solutions to the most difficult problems.

Kubernetes Guides and Tutorials | Spacelift Blog

Kubernetes

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Product

For a while now at Spacelift we’ve been hearing from users that they’d like to be able to run <a href="https://docs.spacelift.io/concepts/worker-pools" target="_blank" rel="noopener">Spacelift worker pools</a> in their Kubernetes clusters. As a first step towards that we decided to investigate whether we could run workers in Kubernetes using a Docker-in-Docker sidecar container. The rest of this post gives a rough overview of the architecture of Spacelift workers, and also explains how the sidecar container strategy works.
I’d just like to say thanks to <a href="https://medium.com/@v.m_hs" target="_blank" rel="nofollow noopener">Vadym Martsynovskyy</a> for his great article about running <a href="https://medium.com/hootsuite-engineering/building-docker-images-inside-kubernetes-42c6af855f25" target="_blank" rel="nofollow noopener">Jenkins agents in Kubernetes</a>. In that article he outlines the general approach taken here, and also describes some of the pros and cons of using Docker-in-Docker. It’s definitely well worth a read.

Before going into more detail, it’s worth explaining what a Spacelift worker is, and giving a quick overview of our architecture. At Spacelift, we provide a hybrid-SaaS model like many other CI/CD systems. What this means in practice is that the control plane that handles scheduling runs is managed by us, but the execution of runs happens in a separate process called a worker. We provide a public worker pool, but also allow you to self-host your own workers, providing complete control over your infrastructure.
Perhaps confusingly, if you follow the instructions to create a private worker pool, you’ll notice that it involves downloading something called the “launcher”. The launcher is responsible for communicating with the Spacelift mothership (control plane), and creating and destroying worker containers using Docker for each Spacelift run:

image2-5

Spacelift mothership (control plane)

As you can see, the launcher process needs access to a Docker daemon in order to create a container for each run.

We found that the launcher architecture maps very nicely to Kubernetes using a sidecar container to run the Docker daemon. For each worker in the pool, we need to create a Kubernetes Pod with two containers: one for the launcher; and another for Docker. The following shows a stripped down Deployment definition for creating a worker pool with 4 workers:

The launcher communicates with the Docker-in-Docker sidecar container via TCP, which is configured via the <code>DOCKER_HOST</code> environment variable for the launcher container. This works because the containers in a Kubernetes Pod share the same <a href="https://kubernetes.io/docs/concepts/workloads/pods/#pod-networking" target="_blank" rel="nofollow noopener">IP address and port space, and can communicate with each other via localhost</a>.
In addition, a shared volume called <code>launcher-storage</code> is mounted into each container. This is used by the launcher to store the workspaces for runs, along with other things like cached tool binaries (for example Terraform). The Docker sidecar needs access to the run workspaces in order to mount them into the worker containers, and it also uses that volume to store its image cache.
The last thing worth pointing out is that the <code>dind</code> container sets <code>securityContext.privileged</code> to <code>true</code>. This is required for Docker-in-Docker to function correctly.

The Spacelift launcher is distributed as a statically linked binary, making it simple to build an image. As you can see, the Dockerfile simply copies the launcher into an Alpine container, and then sets the startup command:

This image is rebuilt and published to our public ECR repository any time the launcher binary is updated.

We provide <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a> to make it really easy for users to deploy worker pools to AWS, Azure and GCP, and we wanted to provide a similar experience for Kubernetes. To achieve this, we created a Helm chart that makes it simple to deploy workers to Kubernetes clusters: <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">https://github.com/spacelift-io/spacelift-workerpool-k8s</a>.
Assuming you’ve already configured your worker pool in Spacelift and have access to the credentials for the pool, deploying this chart to your cluster simply requires two steps.
First, add the Spacelift Helm chart repository and update your local chart cache:

Next, install the chart:

Replace <code>&lt;worker-pool-token&gt;</code> and <code>&lt;worker-pool-private-key&gt;</code> with your own credentials, and make sure to base64-encode the private key.
If all goes well, you should be able to view the pods for your worker pool using <code>kubectl get pods</code>:

You should also be able to view the workers in your pool in Spacelift:

image1-5

Bitbucket DC Local Pool

When implementing in a production environment, you may want to use an alternative approach to providing credentials, and may want to configure a custom storage volume for the worker Pods. For more information about configuring the chart, check out the README in the <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">chart GitHub repo</a>.

If you’re interested in running Spacelift workers in Kubernetes, we’d welcome any feedback about the approach, contributions to the Helm chart, and also any issues you encounter so that we can make improvements. Also, if you aren’t already using Spacelift but are interested in trying it, you can signup and start your free evaluation of Spacelift <a href="https://spacelift.io/free-trial?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bci_cd_workers_in_kubernetes_dind%7D" target="_blank" rel="noopener">here</a>.

5 Ways to Manage Terraform at Scale - Best Practices

Building a Secure CI/CD Integration with Azure

How a Specialized Management Platform Can Improve Your IaC

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Infrastructure as Code : Best Practices, Benefits & Examples

Managing Azure Resources using Spacelift

This article is an overview of the architecture of Spacelift workers with an explanation of how the sidecar container strategy works.

49-running_spacelift_cicd_workers_in_kubernetes_using_dind-3

Running Spacelift CI/CD workers in Kubernetes using DinD

<div>
Securing your infrastructure is critical for the success of your business. Failure to take security seriously can result in major damage including fines, loss of customer confidence, or the inability to carry out crucial business functions. The growth of Infrastructure as Code tools and CI/CD systems has allowed developers to integrate infrastructure management into our typical development workflows, improving quality and delivery speed.
At the same time, in order to manage your infrastructure, the CI/CD system used needs access to sensitive credentials. At Spacelift, we aim to give our users the maximum balance between flexibility and security. Because of this, we provide multiple options for connecting your Azure subscriptions to Azure, including setting static credentials, using our fully managed integration, as well as utilizing private workers to avoid sharing credentials at all.
In this post we wanted to give you an overview of how the <a href="https://docs.spacelift.io/integrations/cloud-providers/azure#spacelift-managed-integration" target="_blank" rel="noopener">Spacelift Azure integration</a> works from a technical perspective, as well as discuss some of the issues we encountered and solved while designing and developing it. As a CI/CD system, we do quite a lot of work to integrate with other systems that our users use. Occasionally, like in the case of Azure, things get non-trivial. If you’re keen to learn more, read on, no Azure knowledge required!
</div>

Let’s start with a quick description of how our <a href="https://docs.spacelift.io/integrations/cloud-providers" target="_blank" rel="noopener">cloud integrations</a> work. The overall workflow is very simple, and looks something like this:
&nbsp;

9-2-2

cloud integrations

Breaking the requirements down, we need to be able to get management credentials for a user’s cloud provider account and pass them to <a href="https://spacelift.io/blog/how-to-use-terraform-variables" target="_blank" rel="noopener">Terraform via environment variables</a>, at which point Terraform can run an apply with access to the customer’s infrastructure.
Note: For simplicity this post uses <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> in all its examples, but this overall approach also applies to the other tools that we support, for example, <a href="https://docs.spacelift.io/vendors/pulumi" target="_blank" rel="noopener">Pulumi</a>.
The concept behind the Azure integration was to provide a similar experience to our <a href="https://docs.spacelift.io/integrations/cloud-providers/aws" target="_blank" rel="noopener">AWS</a> and <a href="https://docs.spacelift.io/integrations/cloud-providers/gcp" target="_blank" rel="noopener">GCP</a> integrations, but for our Azure customers. The following diagram shows a simplified outline of how the AWS integration works:

10-2

how the AWS integration works

AWS provides the ability to temporarily <a href="https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html" target="_blank" rel="nofollow noopener">assume a role</a> in another AWS account. This allows our users to create a role in IAM with any permissions they might want to give Spacelift. They can then set up a trust relationship for this role with our AWS account, which will allow our AWS account to assume the role. Role assumption provides us with raw AWS credentials and works seamlessly with any AWS tooling, including the Terraform AWS provider. It additionally allows us to specify the validity duration, so each run can get its own credentials which are constrained to a short period of time. 
Although Azure doesn’t have the same capability, it provides another approach called Azure Active Directory Applications, that allow service accounts to be created. Azure AD Applications are the resources that allow Spacelift to seamlessly manage access to a customer’s Azure resources.

It’s worth explaining a few pieces of terminology that are used throughout the rest of this post:
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://azure.microsoft.com/en-gb/services/active-directory/" target="_blank" rel="nofollow noopener">Azure Active Directory</a> (Azure AD) &#8211; the identity and access management component of Azure.</li>
<li style="font-weight: 400;" aria-level="1">Directory / tenant &#8211; an individual instance of Azure AD owned by a company or individual.</li>
<li style="font-weight: 400;" aria-level="1">Subscription &#8211; the container for any Azure compute resources. This roughly corresponds to an AWS Account. A subscription is linked to a single Azure AD tenant, but multiple subscriptions can be linked to the same tenant.</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-how-applications-are-added" target="_blank" rel="nofollow noopener">Azure AD Application</a> &#8211; a way of creating external integrations with Azure AD.</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/what-is-application-management" target="_blank" rel="nofollow noopener">Enterprise Application</a> &#8211; an instance of an Azure AD application that has been installed in another user’s Azure AD tenant.</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals#service-principal-object" target="_blank" rel="nofollow noopener">Service Principal</a> &#8211; a service account that is automatically created when an Azure AD application is installed. This can be used to grant permissions that allow the application to <a href="https://spacelift.io/blog/managing-azure-resources-using-spacelift" target="_blank" rel="noopener">manage Azure resources</a>.</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.microsoft.com/en-us/graph/" target="_blank" rel="nofollow noopener">Microsoft Graph API</a> &#8211; the main API for managing Azure AD resources.</li>
</ul>

We set a number of goals for the integration:
<ul>
<li style="font-weight: 400;" aria-level="1">Making it really easy for customers to manage Azure infrastructure using Spacelift.</li>
<li style="font-weight: 400;" aria-level="1">Automatic handling of credential rotation so that customers don&#8217;t have to deal with this themselves, or use very long-lived credentials to avoid it entirely.</li>
<li style="font-weight: 400;" aria-level="1">Providing a mechanism for customers to configure granular permissions in Azure for different stacks, or different types of runs (e.g. PRs vs Tracked Runs).</li>
</ul>
&nbsp;

Initially, our idea was to create a single <a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/single-and-multi-tenant-apps" target="_blank" rel="nofollow noopener">multi-tenant AD Application</a>:

multi-tenant AD Application

The idea was that we would generate an Access Token that could only be used for a specific customer directory, and pass that token to the Terraform Azure RM provider during runs. In the end, we had to revise our approach because of the following issues:
<ul>
<li style="font-weight: 400;" aria-level="1">The <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs" target="_blank" rel="nofollow noopener">Terraform Azure RM provider</a> doesn&#8217;t support authentication via an Access Token. Instead, you have to supply the underlying credentials for the account &#8211; either a Client Secret or a Client Certificate. In our case, that would have meant passing the credentials for our own multi-tenant application to Spacelift runs. Since that application would have been installed in the Azure AD tenants of any Spacelift user who had setup the integration, this could have allowed users to access other user’s Azure accounts.</li>
<li style="font-weight: 400;" aria-level="1">The integration would have been less flexible. Using a single multi-tenant AD application would have prevented customers from creating more than one Azure integration per Active Directory tenant. The ability to create multiple integrations per tenant is useful because it allows different Azure permissions to be applied to each integration.</li>
</ul>

After days of brainstorming on an alternative approach, we came up with a new architecture. We could programmatically generate a new Azure AD Application on our side for each Azure Integration created by Spacelift users. This way, having access to the credentials for an Azure AD Application would only lead to having access to a single Azure AD Tenant on a user’s side. This approach allows Client Secrets to be passed to Spacelift runs without fear of inter-user permission leakage. The final design ended up as shown in the following diagram:

generating Azure AD application

Applications are installed into a customer’s Active Directory tenant via a process called <a href="https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/grant-admin-consent" target="_blank" rel="nofollow noopener">Admin Consent</a>. After admin consent has been completed, a Service Principal is created in the user’s Azure Active Directory to which the user can grant permissions. This allows users to decide the exact level of access that Spacelift has to their resources.

The next issue we faced was related to generating credentials for a run. As described in the <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret" target="_blank" rel="nofollow noopener">provider documentation</a>, the Azure RM provider can be configured by setting certain environment variables. Initially, we took a basic approach of attempting to generate credentials during a Spacelift run. This is what we do for our AWS and GCP integrations, so we weren’t expecting major issues. The steps taken looked something like this:
<ol>
<li style="font-weight: 400;" aria-level="1">Run triggered.</li>
<li style="font-weight: 400;" aria-level="1">Generate a new Client Secret with a short expiry time.</li>
<li style="font-weight: 400;" aria-level="1">Populate the required environment variables.</li>
<li style="font-weight: 400;" aria-level="1">Execute terraform.</li>
</ol>
This seemed to work&#8230; but only some of the time.
While testing the integration, strange things were happening. As an example, the planning phase for a run would succeed, but the apply would fail with a permissions error from Azure. After investigating, we came to the conclusion that this was being caused by <a href="https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/active-directory-architecture" target="_blank" rel="nofollow noopener">eventual consistency</a> in Azure AD.
You can visualize the problem using the following diagram (note: this is just an illustration, and is not meant to be completely accurate):

generating credentials issue

In the example above, step 2 may succeed or fail depending on whether the secret has managed to replicate to the Azure AD server that its request is routed to. Initially, we attempted to test whether or not the secret was usable by making an API request, and retrying until the request succeeded using an exponential backoff. What we soon realized was that even then, subsequent requests could be routed to a different Azure AD instance, which still hasn’t received the new Client Secret, and potentially fail.
Even if it was possible to verify when the secret was fully replicated, waiting for replication to complete would have added a minimum of 30 seconds, and potentially another several minutes. Because of this, we decided to move credential generation and rotation out of the run flow, and into a scheduled task:

moving credential generation and rotation into a scheduled task

The scheduled task runs once per hour, generates secrets with an expiry of 24 hours, and attempts to generate a new secret for an integration roughly 2 hours before expiry of the old secret. This allows credential rotation while always keeping a valid secret.
When a new secret is generated, we use AWS’s <a href="https://aws.amazon.com/kms/" target="_blank" rel="nofollow noopener">Key Management Service</a> to encrypt it so that it is never stored in plaintext.
When a run is triggered, we try to find the secret for the integration with the most amount of time until expiry. We also avoid using new secrets until roughly 10 minutes after generation to avoid the eventual consistency issues caused by Azure AD&#8217;s architecture.
You can visualize the secret lifecycle using the following diagram:

the secret lifecycle

In addition, when creating a new integration we immediately generate a secret. This helps to ensure that the secret will have successfully propagated within Azure AD by the time a run is triggered.

The last major issue we faced was figuring out how to implement credential rotation for our own management account. The integration itself uses an Azure AD Service Principal to manage customer AD Applications using the Microsoft Graph API. Because we run most of our own infrastructure in AWS, we didn&#8217;t have the option of using a <a href="https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview" target="_blank" rel="nofollow noopener">Managed Service Identity</a>, meaning that we needed to handle credential rotation ourselves. In addition, our goal was to automate the process to avoid developers having to periodically perform a manual task, and to reduce the risk of forgetting to renew the credentials before expiry.
In the end, we decided to take the relatively simple approach of storing the certificate in Secrets Manager and writing a scheduled task to periodically check whether the certificate was ready to expire, similar to the approach we took for Client Secrets for the integration. If so the scheduled task generates a new certificate and uploads it to both Secrets Manager and Azure AD:

scheduled task generates a new certificate and uploads it to Secrets Manager and Azure AD

The parts of the system that need to use the client certificate for authentication periodically check for an updated certificate. As with the client secret rotation, we avoid using the new certificate for approximately 10 minutes to allow time for the certificate to propagate throughout Azure AD.
Similar to what happens with the integration client secrets, Secrets Manager uses AWS Key Management Service to encrypt the certificate at rest.

Hopefully, this post has given you a glimpse into the internals of Spacelift&#8217;s Azure integration, along with some of the problems we had to solve while implementing it. As you probably noticed, we’re willing to go to great lengths to ensure a secure and pleasant experience for our users. To find out more, take a look at our Azure integration documentation available at <a href="https://docs.spacelift.io/integrations/cloud-providers/azure" target="_blank" rel="noopener">Spacelift Documentation</a>.

How to Manage Active Directory Objects with Azure AD Provider for Terraform

Introducing the Spacelift Azure Integration!

In this post we wanted to give you an overview of how the Spacelift Azure integration works from a technical perspective, as well as discuss some of the issues we encountered and solved while designing and developing it.

2-building_a_secure_cicd_integration_with_azure

At Spacelift, we&#8217;re excited to announce our new fully-managed integration with Azure. This integration makes it even easier for customers to get up and running fast without having to manually manage Azure Service Principal credentials.

In keeping with our tagline &#8220;the most flexible management platform for Infrastructure as Code&#8221;, and to give you maximum flexibility, you have multiple choices to integrate Spacelift with Azure:
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/integrations/cloud-providers/azure#spacelift-managed-integration" target="_blank" rel="noopener">Using our Azure integration</a>.</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/integrations/cloud-providers/azure#static-credentials" target="_blank" rel="noopener">Manually configuring Azure credentials via environment variables</a>.</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/integrations/cloud-providers/azure#managed-identities" target="_blank" rel="noopener">Running a private worker, and using a Managed Identity</a>.</li>
</ul>
Spacelift’s Azure integration is aimed at customers who want a fully managed experience, new customers who want to quickly get up and running, and existing customers who are manually managing their Azure credentials via Spacelift environment variables or contexts.
We recommend that customers who desire more control for security or compliance reasons should continue using private runners.

Spacelift’s Azure integration provides the following features:
<ul>
<li style="font-weight: 400;" aria-level="1">Easily connect to your Azure subscriptions and attach them to your stacks.</li>
<li style="font-weight: 400;" aria-level="1">Automatic credential rotation.</li>
<li style="font-weight: 400;" aria-level="1">The ability to use granular permissions depending on the action being performed.</li>
</ul>

To connect to your Azure subscription, simply add a new Azure integration via your account settings, specifying your Tenant ID and optionally a default subscription ID:

Azure Integration

Install the Enterprise Application for the integration into your Azure AD tenant using the Provide Consent button:

Azure AD Provide Consent

Setup your permissions in Azure:

setting up permissions in Azure

Attach the integration to any Spacelift stacks that require it:

Attaching the integration to any Spacelift stacks

That’s it, you’re now ready for launch!

When you set up an Azure integration in Spacelift, a Service Principal is created in your directory that you can use to manage permissions. We automatically rotate the credentials for this Service Principal roughly every 24 hours while making sure that there is always a valid credential for your stacks to use.

Spacelift’s Azure integration supports granular permissions by allowing you to setup more than one integration per tenant:

Spacelift’s Azure integration

Each integration will create a Service Principal in Azure with a unique display name:

Service Principal

You can then assign the relevant permissions to each role in Azure:

assigning the relevant permissions

And finally attach them to your stack as read or write:

read or write integration

Once you’ve attached both Azure integrations to your stack, Spacelift will automatically use the correct one depending on the phase of the run being executed. This allows you to provide read-only roles to planning runs (for example PRs), while providing write access for deployments.

To find out more about the Azure integration, take a look at our documentation available at <a href="https://docs.spacelift.io/integrations/cloud-providers/azure" target="_blank" rel="noopener">Spacelift Documentation</a>. You might also be interested in the detailed overview of <a href="https://spacelift.io/blog/building-secure-cicd-integration-with-azure" target="_blank" rel="noopener">How we Built a Secure CI/CD Integration with Azure</a> and some of the issues we encountered and solved while designing and developing it.

We're excited to announce our new fully-managed integration with Azure. This integration makes it even easier for customers to get up and running fast without having to manually manage Azure Service Principal credentials.

1-azure_integration

Introducing the Spacelift Azure Integration

Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Terraform

The <a href="https://registry.terraform.io/providers/hashicorp/azuread/latest/docs" target="_blank" rel="nofollow noopener">Azure AD provider for Terraform</a> can be used to manage your Azure Active Directory resources declaratively. This allows you to do things like:
<ul>
<li style="font-weight: 400;" aria-level="1">Automatically provision users and make sure they belong to the correct groups.</li>
<li style="font-weight: 400;" aria-level="1">Manage Azure compute permissions via Azure AD groups.</li>
</ul>
In this post, you will learn what the Azure AD Terraform provider is used for, how to authenticate and grant permissions, and see examples of what you can do with it. Let’s get started.

The following example shows how to use the Azure AD provider to create a <a href="https://registry.terraform.io/providers/hashicorp/azuread/latest/docs/resources/group" target="_blank" rel="nofollow noopener">group</a> in Azure AD:

The <code>terraform</code> section at the beginning is used to specify the version of the provider that we want to use, while the <code>azuread_group</code>  resource defines our group.

The Azure AD provider allows for multiple authentication methods, which are outlined in the <a href="https://registry.terraform.io/providers/hashicorp/azuread/latest/docs#authenticating-to-azure-active-directory" target="_blank" rel="nofollow noopener">provider&#8217;s documentation</a>. To allow you to get up and running quickly, the AD provider will attempt to get your credentials via the <a href="https://docs.microsoft.com/en-us/cli/azure/install-azure-cli" target="_blank" rel="nofollow noopener">Azure CLI</a>.
While this is fine for experimentation and local testing, for non-interactive scenarios like CI you will need to use a Service Principal or a Managed Service Identity.

In order to manage your Azure AD objects, the account used by <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> needs to have the correct permissions to perform its actions. You can manage these permissions via the Roles and administrators section of Azure AD:

Roles and administrators section of Azure AD

For example, to allow a Service Principal to manage groups, you would add it to the Groups administrator role:

Azure AD Groups administrator

The Terraform provider is well documented, and will typically contain a notice at the top of each resource explaining the permissions that are required for using it.

Another option that can be used with Service Principals instead of granting an administrator role is to assign specific API permissions to them. To do this, first find the AD Application linked to your Service Principal in the App Registrations section:

Service Principal in the App Registrations section

Go to the API permissions page for the application, and click on Add a permission:

Manage Active Directory Objects

In the window that appears, choose the Azure Active Directory Graph API, and then select the relevant permission you want to add:

Azure Active Directory Graph API

Before the Service Principal can actually use the permission you just added, you need to take a final step called granting Admin Consent. You can do this by clicking on the Grant admin consent for &lt;tenant&gt; button displayed above the permissions table:

5-5-2

Azure AD Admin Consent

NOTES:
<ul>
<li style="font-weight: 400;" aria-level="1">When adding permissions to your Service Principal, you need to add Application permissions rather than Delegated permissions. This means that the Service Principal is allowed to perform the specified actions as itself, rather than on behalf of another user.</li>
<li style="font-weight: 400;" aria-level="1">The set of permissions that you can add via API permissions is quite limited. For example, to create AD groups you need to add the Directory.ReadWrite.All permission, but this will not allow your Service Principal to delete any of the groups it creates. In order to be able to delete groups, you need to grant it the Group Administrator role, so depending on your requirements there may not be any point in granting API permissions.</li>
<li style="font-weight: 400;" aria-level="1">The Azure AD Terraform provider switches to the Microsoft Graph API as of version 2.0.0, so when version 2 is released you will need to grant permissions to the Microsoft Graph API instead of to the Azure Active Directory Graph API.</li>
</ul>

<h3>Example 1 &#8211; Managing Users and Groups</h3>
The following example creates two users and two groups, and assigns each user to a group:

<h3>Example 2 &#8211; Creating a Service Principal and granting RBAC permissions</h3>
The following example combines the Azure AD provider with the <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs" target="_blank" rel="nofollow noopener">Azure RM provider</a>, allowing you to create a Service Principal and assign it permission to <a href="https://spacelift.io/blog/managing-azure-resources-using-spacelift" target="_blank" rel="noopener">manage certain Azure resources</a>:

In this post, we’ve covered what the Azure AD Terraform provider is used for, how to authenticate and grant the correct permissions, as well as showing a few examples of what can be done with it. Hopefully, you’ve found it useful!
If you’re interested in finding out about how you can use Spacelift to manage your Azure resources, check out our <a href="https://docs.spacelift.io/integrations/cloud-providers/azure" target="_blank" rel="noopener">Azure documentation</a>. Also, don’t forget that you can easily take Spacelift for a <a href="https://spacelift.io/free-trial?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bhow_to_manage_active_directory_objects_with_azure_ad_provider_for_terraform%7D" target="_blank" rel="noopener">free test drive</a>!

Spacelift helps manage Terraform state, build more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and many more.

Importing Existing Infrastructure into Terraform - Step by Step

Learn how to manage Active Directory Objects with Azure AD Provider for Terraform. See examples of how to authenticate and grant the correct permissions.

15-1

Active Directory Objects with Azure AD Provider for Terraform

<dfn>Open Policy Agent</dfn> is an open-source engine that provides a way of declaratively writing policies as code and then using those policies as part of a decision-making process. It uses a policy language called Rego, allowing you to write policies for different services using the same language.
OPA can be used for a number of purposes, including:
<ul>
<li style="font-weight: 400;" aria-level="1">Authorization of REST API endpoints.</li>
<li style="font-weight: 400;" aria-level="1">Allowing or denying Terraform changes based on compliance or safety rules.</li>
<li style="font-weight: 400;" aria-level="1">Integrating custom authorization logic into applications.</li>
<li style="font-weight: 400;" aria-level="1">Implementing Kubernetes Admission Controllers to validate API requests.</li>
</ul>
OPA was originally created by <a href="https://www.styra.com/" target="_blank" rel="noopener">Styra</a>, and is now part of the Cloud Native Computing Foundation (CNCF), alongside other CNCF technologies like Kubernetes and Prometheus.
In this article, I will give an overview of Open Policy Agent, explain why you would want to use it, as well as showcase how you can use OPA with your Spacelift account. Although OPA can serve many purposes, I’m going to focus on how it can be used alongside <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code</a>.

We can visualise how OPA works using the following diagram:

how-does-opa-work

Open Policy Agent (OPA) Example

As you can see, OPA accepts a policy, input and query, and based on that, generates a response. The input can be any valid JSON document, allowing OPA to integrate with any tool that produces JSON output.

In the following sections, I’ll go into more specific examples of using OPA that should help make things clearer but before I do, here’s a quick list of reasons why I’m interested in using OPA:
<ul>
<li style="font-weight: 400;" aria-level="1">Policies as code allow you to follow your standard development lifecycle with PRs, CI, etc., and provide you with a history of changes to your policies.</li>
<li style="font-weight: 400;" aria-level="1">OPA is designed to work with any kind of JSON input, meaning it can easily integrate with any tool that produces JSON output.</li>
<li style="font-weight: 400;" aria-level="1">Because OPA integrates with a number of different tools, it allows you to use a standard policy language across many parts of your system, rather than relying on multiple vendor-specific technologies.</li>
<li style="font-weight: 400;" aria-level="1">OPA supports unit-testing, making it easier and faster to iterate your policies with confidence that they won’t break.</li>
</ul>

Let’s try to make that a bit less theoretical by using a specific example: Terraform. Terraform can produce a plan in JSON format via the terraform show command. This means that we can define policies for our infrastructure, and use OPA to make a decision about whether a plan is safe to apply or not:

using-opa-with-terraform

using opa with terraform

For example, say we have the following Terraform definition to create an EC2 instance:

Now, say we want to ensure that every Terraform resource has a Name tag. We could enforce that by creating a file called plan.rego with the following content:

In order to use OPA to evaluate our policy, we need to take the following steps:
<ol>
<li>Generate a Terraform plan as JSON.</li>
<li>Run opa eval to verify whether that plan passes our policy or not.</li>
</ol>

<h3>Step 1 &#8211; Generate our Terraform plan as JSON</h3>
To get a JSON representation of our plan, we need to output our plan to a file, and then use the terraform show command to output that plan as JSON:

<h3>Step 2 &#8211; Run opa eval</h3>
We can then use opa eval to evaluate our plan against our policy:

As you can see, we’re using the query data.spacelift.allow because our policy is loaded as a data file, and we defined our allow rule in the spacelift namespace. You can also see that opa eval produced an empty output ({}). This means that our allow rule didn’t evaluate to true, and so produced no output.
Let’s adjust our Terraform definition to include a Name tag:

Now if we generate our plan and evaluate the policy again, we should get a slightly different output:

This tells us that the allow rule has evaluated true. We can get that in a slightly more concise way by using the pretty format option:

At this stage, you can probably imagine how you could integrate this into your <a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener">CI/CD pipeline </a>to enforce naming schemes, security rules, and various other organizational policies.

The following examples illustrate some possible use-cases for OPA with <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a>. All of the examples have been taken from the Spacelift <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy#cookbook" target="_blank" rel="noopener">plan policy documentation</a>, but have been altered to work with plain vanilla Terraform.
<h3>Example 1. Require human review when resources are deleted or updated</h3>
Adding new resources is usually a fairly safe operation, but updating or deleting existing resources can carry more risk. You could flag these for human review by using the following policy:

<h3>Example 2. Require commits to be reasonably sized</h3>
Once a PR goes over a certain size it becomes difficult to review without missing things. The same is true for Terraform plans. The following policy can be used to warn when the number of changes goes over a certain threshold:

<h3>Example 3. Blast radius</h3>
The following policy attempts to determine the risk of a particular plan by assigning different weightings to the change type (create, update, delete), along with the affected resource type (ECS cluster, EC2 instance, etc.). It takes the approach that an update or delete is more risky than a create because it affects an existing resource:

How can we make sure that our policies work as we expect them to and that they don’t break over time as we make changes to them? You guessed it: unit testing! Luckily for us, OPA has first-class support for testing via the opa <code>test</code> command.
To create tests for our policy, all we need to do is create another Rego file with a series of rules prefixed with <code>test_</code>. Each rule starting with that prefix defines a separate test. 
Let&#8217;s go ahead and create a file called plan_test.rego, with the following contents:

As you can see, OPA makes it really easy to specify the policy input using the <code>&lt;variable&gt;</code> as <code>&lt;value&gt;</code> syntax. This allows us to create very concise tests by only including values we care about in the policy input, rather than having to use the entire plan output.
Let&#8217;s go ahead and run <code>opa test</code>:

Unsurprisingly our test passes. That&#8217;s not very exciting, so let&#8217;s add a new test to ensure our resources include an <code>Environment</code> tag:

Running opa test again shows us a failure:

We can fix this by updating our policy:

At this stage if you run the test command again it should show 2 passes:

If you want to know more about OPA testing, the <a href="https://www.openpolicyagent.org/docs/latest/policy-testing/" target="_blank" rel="nofollow noopener">official docs</a> are full of great examples and information about what you can do.

At this stage, hopefully, you’ve got a pretty good idea of what OPA is as well as how it can be useful to you. It’s not too difficult to see how you could start integrating OPA into your development process or even use it as part of production systems.
Luckily for you, at Spacelift we’ve done all the heavy lifting, allowing you to reap the benefits of using OPA for Policy-as-Code without you having to implement everything from scratch yourself. In this section, I want to showcase some of the functionality that Spacelift provides as it relates to OPA.
<h3>1) Policy Types</h3>
Spacelift allows you to use <a href="https://resources.sei.cmu.edu/asset_files/Presentation/2021_017_001_739363.pdf" target="_blank" rel="noopener">OPA policies</a> to manage various aspects of your Spacelift account, and not just during planning. For example, you can use policies to control who can log into your account, along with what they have access to. For more information, see <a href="https://docs.spacelift.io/concepts/policy" target="_blank" rel="noopener">Policy as Code</a>.
<h3>2) PR Checks</h3>
Spacelift can automatically trigger planning runs whenever you push changes to your VCS provider. For example, here’s what you might see in GitHub after creating a PR:

pr-checks-1

Opa pr checks

If all goes well your check will succeed, but if a policy is violated, a failed check will be reported:

pr-checks-2

opa pr checks failed

You can then view the details of the failure in Spacelift:

pr-checks-3

failure details in Spacelift

<h3>3) Manual Approvals</h3>
Spacelift plan policies use a slightly different format from the one we used in our example policies earlier in this post. Not only do they allow you to specify a message to be displayed, but also have the concept of <code>deny</code> and <code>warn</code>:

The <code>deny</code> rule fails the run completely, while the <code>warn</code> rule just displays a warning in the logs.
<code>warn</code> rules take on another role when a run is going to deploy changes (vs just showing the planned changes against a PR). If any warnings are reported during a deployment, the run will wait for manual approval before applying any changes.
Let’s use the following example policy, designed to warn if a certain set of suggested tags isn’t found within our resources:

If we then attempt to add resources that don’t contain all of those tags, Spacelift will block before applying the changes, and give us the chance to manually approve or deny the run:
&nbsp;

manual-approvals

manual approvals

This allows you to build complex workflows where certain changes are completely blocked, while others are allowed as long as the changes are reviewed first.
Check out our <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy#cookbook" target="_blank" rel="noopener">plan policy cookbook</a> for more ideas about what you can do.

Spacelift provides a <a href="https://registry.terraform.io/providers/spacelift-io/spacelift/latest/docs" target="_blank" rel="nofollow noopener">Terraform provider</a> for managing your Spacelift account. This means that you can manage the policies available within your account, along with the Stacks they apply to in code.
For example, to add the policy we defined earlier to Spacelift we can use the <code>spacelift_policy</code> resource like this:

We can then attach this policy to a Spacelift Stack using the <code>spacelift_policy_attachment</code> resource:

Taking this a step further, we could create a custom module for defining Spacelift Stacks that ensured that all Stacks had a certain set of policies attached by default.

There are a number of exciting tools available that can help integrate OPA with your systems, for example <a class="c-link" tabindex="-1" href="https://www.styra.com/" target="_blank" rel="noopener" data-stringify-link="https://www.styra.com/" data-sk="tooltip_parent" data-remove-tab-index="true">Styra DAS</a>. Styra provides a way of writing policies and deploying them across your infrastructure, as well as providing tools for unit testing, monitoring policy usage and more. You can try OPA at scale for free using <a href="https://www.styra.com/das-free" target="_blank" rel="noopener">Styra DAS Free</a>.

I hope you’ve enjoyed this post, and can see the value that OPA brings to the table. If you’re interested in trying out <a href="https://spacelift.io/free-trial?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bwhat_is_open_policy_agent_and_how_it_works%7D" target="_blank" rel="noopener">Spacelift to see what we have to offer, why not sign up for a free trial</a>? You can set up a Spacelift account in minutes, and get started on your Policy as Code journey today!

Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation and controls to simplify and accelerate the provisioning of cloud based infrastructures.

What is Open Policy Agent (OPA)? See Open Policy Agent examples and a full tutorial on when to use it.

12-1

open policy agent

Open Policy Agent: What Is OPA and How It Works (Examples)

In this post I want to show how easy it is to create a <a class="dt id" href="https://spacelift.io/?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bmanaging_azure_resources_using_spacelift%7D" target="_blank" rel="noopener">Spacelift</a> Stack that manages Azure resources. To keep things simple, I’m going to assume that you already know what Spacelift can do, and the basics of creating and running Stacks, but if not check out our <a class="dt id" href="https://github.com/spacelift-io/terraform-starter" target="_blank" rel="nofollow noopener">terraform-starter</a> repo that walks you through the process of setting up an account, and getting started.

In order to allow Spacelift to manage Azure resources, we need to do the following:
<ul class="">
<li id="f0f8" class="hj hk fo hl b hm jf hn ho hp jg hq hr hs jh ht hu hv ji hw hx hy jj hz ia ic jk jl jm gk" data-selectable-paragraph="">Create a Service Principal in Azure with the correct ARM permissions to manage our resources.</li>
<li id="8f2c" class="hj hk fo hl b hm jn hn ho hp jo hq hr hs jp ht hu hv jq hw hx hy jr hz ia ic jk jl jm gk" data-selectable-paragraph="">Create a Stack in Spacelift that uses the <a class="dt id" href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs" target="_blank" rel="nofollow noopener">Azure</a> provider.</li>
<li id="ba4d" class="hj hk fo hl b hm jn hn ho hp jo hq hr hs jp ht hu hv jq hw hx hy jr hz ia ic jk jl jm gk" data-selectable-paragraph="">Give our Stack the credentials for our Service Principal.</li>
</ul>
All of the information you need to configure the Azure provider can be found <a class="dt id" href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure" target="_blank" rel="nofollow noopener">here</a>. This post just explains how to combine that information with Spacelift.

To create our Service Principal, we need to use the <code class="js jt ju jv jw b">az ad sp create-for-rbac</code> command (replacing <code class="js jt ju jv jw b">&lt;subscription-id&gt;</code> with your actual subscription ID):

This will output something like the following:

The command creates a new Service Principal called <code class="js jt ju jv jw b">spacelift-sp</code>, and grants it the Contributor role on your subscription. It also outputs the <code class="js jt ju jv jw b">appId</code>, <code class="js jt ju jv jw b">password</code> and <code class="js jt ju jv jw b">tenant</code> for the Service Principal. Make a note of these because you&#8217;ll need them later.
If you would rather assign permissions yourself later, you can run the following command to just create a Service Principal with no permissions:

You should now be able to view your Service Principal in the Azure Active Directory -&gt; App registrations section of the Azure Portal:

default directory

Now that we’ve got a service principal, we need to create a Stack to actually create some Azure resources. To keep things simple all we’re going to do is add some terraform definitions that specify the version of the Azure Provider we want to use, and create an Azure Resource Group:

If you commit that change and try to run your Stack just now, you should see something similar to the following failure:

adding Azure resources

Here’s the full error message:
Error building AzureRM Client: obtain subscription() from Azure CLI: Error parsing json result from the Azure CLI: Error launching Azure CLI: exec: “az”: executable file not found in $PATH
At first glance this error is slightly confusing, but what it means is that because we didn’t provide any credentials to the Azure provider it attempted to fallback to using the <code class="js jt ju jv jw b">az</code> CLI for authentication. This works fine locally, but won&#8217;t work on Spacelift.
To solve this, we need to provide our stack with our credentials, which is explained in the next section.

If you take a look at the <a class="dt id" href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure" target="_blank" rel="nofollow noopener">Terraform documentation</a>, you’ll find that there are a number of different ways that you can authenticate <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> with Azure. What we’re going to do here is authenticate using a Service Principal with a Client Secret, and provide that configuration via <a href="https://spacelift.io/blog/how-to-use-terraform-variables" target="_blank" rel="noopener">environment variables</a>.
To do this we need to set the following environment variables, using the information returned by the Azure CLI when we created our Service Principal:
<ul class="">
<li id="1508" class="hj hk fo hl b hm jf hn ho hp jg hq hr hs jh ht hu hv ji hw hx hy jj hz ia ic jk jl jm gk" data-selectable-paragraph="">ARM_CLIENT_ID — the <code class="js jt ju jv jw b">appId</code> from the response.</li>
<li id="4e7d" class="hj hk fo hl b hm jn hn ho hp jo hq hr hs jp ht hu hv jq hw hx hy jr hz ia ic jk jl jm gk" data-selectable-paragraph="">ARM_CLIENT_SECRET — the <code class="js jt ju jv jw b">password</code> from the response.</li>
<li id="8a57" class="hj hk fo hl b hm jn hn ho hp jo hq hr hs jp ht hu hv jq hw hx hy jr hz ia ic jk jl jm gk" data-selectable-paragraph="">ARM_SUBSCRIPTION_ID — your subscription ID.</li>
<li id="7340" class="hj hk fo hl b hm jn hn ho hp jo hq hr hs jp ht hu hv jq hw hx hy jr hz ia ic jk jl jm gk" data-selectable-paragraph="">ARM_TENANT_ID — the <code class="js jt ju jv jw b">tenant</code> from the response.</li>
</ul>
You can set these environment variables directly on your Stack, but what we’re going to do here is create a <a class="dt id" href="https://docs.spacelift.io/concepts/configuration/context" target="_blank" rel="noopener">Spacelift Context</a> and attach it to our Stack. This allows us to share the same credentials with multiple Stacks if we want.
Add a new Context via the Contexts screen in Spacelift:

3-2-2

adding contexts

In order to allow Spacelift to manage Azure resources, we need to create a Service Principal in Azure with the correct ARM permissions to manage our resources., create a Stack in Spacelift that uses the Azure provider and give our Stack the credentials for our Service Principal.