Check out the Spacelift Engineering blog to discover internal technical architecture posts written by Spacelift's very own engineers. 

Engineering | Spacelift Blog

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Engineering

As many others did before and regretted later, we’ve decided to move a big part of our workload from long-running processes to Lambda. That said, we haven’t regretted it a bit. It’s been a breeze so far, even if there are a few shortcomings of Lambda, which are just annoying.
In this blog post, I’ll describe the positive sides, the negatives, and the approach we’ve taken that allowed us to radically simplify our setup, as well as clarify common misconceptions you might have about AWS Lambda development.
But first of all, let’s take a look at…

Our backend is based on two main constructs:
<ul>
<li style="font-weight: 400;" aria-level="1">The server, which exposes GraphQL and REST APIs to the outside world.</li>
<li style="font-weight: 400;" aria-level="1">The drain, which consumes a bunch of SQS queues to handle asynchronous jobs. Each SQS queue transports several event types, based on a logical separation (webhooks, async jobs, cronjobs, etc.)</li>
</ul>
We have a very asynchronous architecture. The server often emits async jobs instead of doing something by itself, and the drain often emits new events when handling existing ones.
Because we’re a <a href="https://spacelift.io/blog/ci-cd-pipeline" target="_blank" rel="noopener">CI/CD system</a>, our traffic patterns are very unevenly distributed. Sometimes there are few jobs, at other times, somebody creates an avalanche of them.
Until recently, the drain was a long-running process on ECS, reading from multiple SQS queues. Autoscaling in such a situation is fairly hard. The simplest option is to have a separate process per SQS queue, with each process scaled independently. Even doing this would have resulted in paying too much for periods of low traffic, along with handling spikes in traffic too slowly. The diagram below describes that architecture visually:

image2-26

Our Architecture

Also, everything on the backend is written in Go, no exceptions.

Many people think that moving to Lambda requires huge codebase changes and has to result in having a lot of binaries, one for each Lambda Function. That’s actually not necessary. Yes, Lambda does only call the binary without any arguments (and there’s no way to change that to my knowledge &#8211; I tried), but you can provide arbitrary environment variables.
In our case, we have a single <code>spacelift</code> binary that will run the server process if called as <code>spacelift backend server</code>, but if called as plain <code>spacelift</code>, then it assumes it’s being run as a Lambda handler and will take a look into the <code>HANDLER</code> environment variable to deduce which handler to actually run. This way, we can keep using a single binary for everything, and operational simplicity ensues. The following diagram shows the new setup:

image1-27

Changing Our Codebase for Lambda

Lambda can also be quite annoying for local development. The way we solve this is by having all of our handlers &#8211; as you’ve seen above &#8211; fronted by SQS queues. This way we can run the handlers locally, and have a shim that polls the SQS queue (a copy the given developer has), translates it into the Lambda-native SQS message format, and passes that to the handler. Very similar to the way the original setup worked, but just for local development.
In practice, this means we have a <code>spacelift backend local-drain</code> command that just runs all handlers with their respective SQS queue pollers. That gets recompiled on each code change, so the feedback loop from code change to locally deployed new binary is around 5 seconds and you can easily attach a debugger to it.

On ECS, the system we moved from, <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-secrets.html" target="_blank" rel="nofollow noopener">you can just specify a list of environment variables and mark their sources as “to be taken from SecretsManager”</a>. With Lambda, that’s not possible. However, we wanted to keep dependency injection uniform across the codebase, so we wanted to have the same flow in Lambda as before.
Thus, we just pass a plain environment variable with the list of objects to get from SecretsManager. The Lambda binary will get those objects from SecretsManager, iterate over all key values in each, and set them as environment variables on the current process. This happens first-thing after starting, so the rest of our initialization process is oblivious to it. This might sound like a hack, but it’s simple, and it does the job.

Another misconception about Lambda is that Container Image support means it can take arbitrary Docker images and run them as-is. Unfortunately, that’s not true.
Container Images in Lambda basically work as a glorified file archive, nothing more, nothing less. Your binary still has to adhere to the Lambda communication protocol, Although you can still use Docker to build your image. The main difference is that thanks to the improved caching story, they let you use 10 GB images instead of just 50 MB for ZIP archives.
Another difference is that with Container Images you can’t configure Lambda layers to be used through the AWS console, as you can with ZIP archives. That said, you can easily copy layers into your Container Image:

The way your Lambda Function works is that it gets spun up, launches your binary, which can do any setup it wants and then starts polling for jobs from the Lambda runtime. Then, when the Lambda is not needed anymore (scaling down) it gets terminated.
So, you can use goroutines to do some async tasks in your Lambdas while they’re running, right? Well, no.
The Lambda Runtime will actually freeze your process between invocations. This means that if you have an async process doing work between invocations, it might be frozen right after it tries to do an IO operation and be woken up 30 seconds later, which makes the IO operation time out immediately.
In other words, you can’t safely use async processes in your Lambda binaries &#8211; at least not outside a single invocation’s timespan. Also, make sure that your database doesn’t terminate connections if it didn’t get a keepalive in 30 seconds, in case you’re connecting to it directly (which btw. was not an issue for us).

It’s not ideal.
With Lambdas, there is a concept of throttling. It happens when you try to invoke a lot of Lambdas at once, but they can’t scale fast enough. Lambda will then return a 429 error.
So, if you suddenly have lots of SQS messages, the SQS connector will scale up and send lots of invocations which might fail with the 429 error. So, the SQS connector will slow down and retry them in a second, right?
Well, no. Instead of retrying with those most recent messages a moment later, it will actually fail the message handling and put it back to SQS to be handled again after the visibility timeout passes (which might take several minutes). That is not great and can introduce additional delays in times of high traffic.
That said, we haven’t encountered this issue again since setting up high <a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html" target="_blank" rel="nofollow noopener">reserved concurrent execution numbers</a> on the Lambda functions. Based on the last few months of usage, this has been a non-issue to us.

With most systems, often “turn it off and back on again” is a quite good incident recovery mechanism. Your code gets into an unexpected state, and this way you reset it.
With Lambda, however, there’s no obvious way to invalidate all the current warm Lambda runtimes. So, if you need this, the way to go is actually to add a useless environment variable, so that you have a change, and then deploy it. Deploying will invalidate old Lambda runtime versions, and effectively  “turn it off and back on again”.

As described in the beginning, we often encounter load spikes, which required us to have many machines on standby before moving to Lambda. With Lambda, this works beautifully out of the box. Whenever we get a huge amount of events at a moment’s notice, Lambda is able to scale up to 10x the capacity, quickly handle all the events, and then scale down.
Moreover, we achieve all of this while having drastically reduced the cost of the whole setup. Lambdas for asynchronous event processing are very cost-effective. I’m not sure I’d recommend them for synchronous HTTP services, but here they definitely do shine.

Lambdas deploy instantaneously. They have perfect isolation &#8211; we don’t have to worry about one Function’s execution influencing any other Function, by starving resources or something similar. Cold starts are very short with Go, warm starts are basically instant. So as far as these considerations are concerned, we’re very happy. 
After migrating to Lambdas, they required very little upkeep, and haven’t been a source of incidents.

Overall, we’re very satisfied with our new setup. I hope the above lessons learned can be useful to somebody else considering the same switch, or planning their greenfield project on Lambda.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities s for infrastructure management.

How to Manage AWS Lambda Functions with Terraform: Tutorial

Tricking PostgresSQL into Using an Insane - but 200x Faster - Query Plan

Creating a flexible Backoffice Tool in a Technical Company using Slack

See the approach we’ve taken that allowed us to simplify our setup and clarify common misconceptions you might have about AWS Lambda development.

89-aws-lambda-land

AWS Lambda Land

Adventures in AWS Lambda Land - a Migration Gone Well

tricking Postgres into using an insane - but 200x faster - query plan

At Spacelift, we use PostgreSQL (specifically Aurora Serverless) for most of our primary database needs. Our application query patterns are mostly small transactions that touch a few rows &#8211; a typical scenario that PostgreSQL handles without breaking a sweat.
We also use Datadog as a monitoring solution. Their agent has a handy feature to create metrics based on periodically executed SQL queries, allowing us to build smart and highly contextual monitors and alerts. Occasionally, such queries might be slow and dominate the processing time of the database. After hooking up Datadog Database monitoring, we identified a lot of slow queries and moved most of them to a separate Redshift cluster. There they execute once per day on a periodic snapshot of the database.
However, a number of these queries need to happen at a more frequent interval as they can detect alertable conditions. We couldn’t move these queries to Redshift because of the staleness of the data there. Aurora Serverless also doesn’t support read replicas to offload analytical queries to them, so no luck there either. Thus, we had to optimize, add indices, optimize, and again optimize until we cut the database usage of these metric-creating queries by 1-2 orders of magnitude.
This article will show you one such optimization story, which is interesting because both the before and after query plans will look somewhat ridiculous (in terms of cost). Moreover, it shows how PostgreSQL row count estimation can go very wrong. In this case, domain knowledge will help us trick PostgreSQL into a different query plan which will be waaay faster, without adding any additional indices.

Spacelift is a CI/CD platform specializing in Infrastructure as Code. Users create Runs (executions of, i.e., Terraform) which will be executed on Worker Pools. There is a public shared Worker Pool managed by Spacelift and private Worker Pools that users can host for themselves. 
Each Run belongs to a Stack, which can be thought of as a single environment, managing a bunch of infrastructure resources. If you know Terraform or CloudFormation, a single Stack maps to a single Terraform state file or a CloudFormation root stack, respectively.
A Run can have one of many types: Tracked, Proposed, and others. That type determines what the overall workflow will look like.

The query is supposed to notify us when the public Worker Pool cannot handle demand &#8211; user Runs are waiting too long to be handled by a Worker.

That’s a big query! What’s happening here?
We’re interested in the number of Runs (as well as the age of the oldest one) which are of the Proposed or Testing type (can be scheduled right away &#8211; other Run types require exclusiveness), are in the Queued state (which means they’re not yet being processed) and don’t have a Worker attached to them (are still waiting for one).
We only want Runs on Accounts that are far from their parallelism limit to reduce the number of false positives caused by users being correctly limited. So for each Run, we count the number of active Runs on its account and check if it&#8217;s under 50% of the parallelism limit, and only then do we count this Run as properly pending.
Unfortunately, that query lasts 20 seconds and consumes 50% of our database&#8217;s processing time. That’s pretty bad.

Let’s take a look at the query plan. I’ll be using <a href="https://explain.dalibo.com" target="_blank" rel="nofollow noopener">https://explain.dalibo.com</a> to visualize query plans. To get the query plan, I’m executing it prefixed with EXPLAIN (FORMAT JSON, ANALYZE) … so we can also see information about the query’s actual execution (ANALYZE).

image5-8

visualization of query plans

What can we see here?
On the left side, we’re joining Worker Pools with Stacks. We also calculate the number of active Runs for each Account using the SubPlan. We do a three-way hash join on the (Worker Pool, Stack) tuples, the Accounts, and the SubPlan results based on the Account ID. Then we loop over all those triples and get the pending Runs for each, finally counting all the pending Runs found.
The SubPlan goes over all Stacks in an Account, and for each Stack gets the active Runs for it (the ones where worker_id is not NULL).
Let’s look at the time distribution of the plan nodes.

image4-8

time distribution of the plan nodes

We can see that the most expensive part of this query is the Bitmap Index Scan which looks for Runs whose worker_id is not NULL, so let’s take a closer look at that part of the query plan.
For each Stack in the SubPlan, we look for active Runs on it. How do we do that?

image1-8

Bitmap

We scan the runs_worker_id index for storage pages containing Runs with a non-empty worker_id and the run_stack_id index for the pages containing Runs for the current Stack. Then we AND these together &#8211; we only want pages present in both of these scans. Finally, we scan the actual Runs from the pages and only take the ones that satisfy the predicates (pages found by the index may contain both relevant and irrelevant Runs).

image2-8

bitmap index scan

In a different tab of the left Index Scan, we can see the root of the problem. We made 531358 loops over this, and we expected to scan 120 mil Runs over the whole processing of the query here, but we scanned 800 mils. That’s much more.
But later in the Heap Scan of the ANDed pages, we see a gross overestimation.

image6-4

bitmap heap scan

We expected to scan 2 mil rows but scanned 42 overall, which means no Runs were executing on most Stacks at this instant.
Overall, it’s mostly this subquery being very expensive:

Now comes the domain knowledge. Is there something we know the optimizer seems not to…? Yes!
Only a minuscule part of the Runs in the database are active at any given time. Most Stacks stay dormant most of the time. Whenever a user makes a commit, only the affected Stacks will execute Runs. There will obviously be Stacks that are constantly in use, but most won’t. Moreover, intensive users will usually use private Worker Pools. The public shared Worker Pool is predominantly utilized by smaller users, so that’s another reason for not seeing many Runs here.
Right now, we’re iterating over all existing Stacks attached to the public worker pool and getting relevant Runs for each. How about we flipped that? We know there are at most as many active Runs as there are public Workers &#8211; orders of magnitude fewer than there are Stacks. Overall active Runs (not just the ones on the public Worker Pool) are also way less numerous, at most as many as there are Workers overall (and since the number of private Workers is the main dimension we currently base our Enterprise tier pricing on, query processing time being linearly correlated with the number of Workers is OK). Thus, we could get all active Runs and filter to just those whose Stack uses the public Worker Pool.
The only problem is, we’re working with a fairly opaque query optimizer, and PostgreSQL doesn’t have any way to give hints to it &#8211; like forcing a join strategy. So we’ll have to do that differently. We’ll create a query that scans the active Runs and then uses another subquery to filter them to just the ones related to a relevant Stack. Then we’ll have to hope the query is opaque enough for Postgres not to get too clever with it.

This query does exactly what we described. Scan active Runs, filter to just those with more than 0 Stacks in the current account and connected to the public Worker Pool. There would be either 1 or 0 such Stacks for each Run.
Executing this query, we get down to 100ms instead of the original 20s. That’s a vast improvement. What does the query plan say? (apologies for the long vertical picture)

image3-8

What does the query plan say?

Exactly what we wanted! It scans active Runs, and for each of them, checks (using a primary key index lookup &#8211; fast!) whether the Stack it belongs to is relevant.
That’s it! The whole optimized query now looks like this:

Thanks for reading! The moral of this story is that query plans aren’t that hard to interpret with the right tools. Using your domain knowledge, you can rewrite your queries to achieve substantial performance boosts without adding unnecessary indices or denormalizing the data model.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Managing Azure Resources using Spacelift

Running Spacelift CI/CD workers in Kubernetes using DinD

Building a Secure CI/CD Integration with Azure

At Spacelift, we use PostgreSQL (specifically Aurora Serverless) for most of our primary database needs. This article will show you our optimization story and how PostgreSQL row count estimation can go very wrong.

50-tricking-postgres

Tricking PostgresSQL into Using 200x Faster Query Plan

Running Spacelift CICD Workers in Kubernetes Using DinD

Want to level up your Kubernetes knowledge? Explore our tutorials and step-by-step guides written by Kubernetes experts. Learn Kubernetes best practices and solutions to the most difficult problems.

Kubernetes Guides and Tutorials | Spacelift Blog

Kubernetes

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Product

For a while now at Spacelift we’ve been hearing from users that they’d like to be able to run <a href="https://docs.spacelift.io/concepts/worker-pools" target="_blank" rel="noopener">Spacelift worker pools</a> in their Kubernetes clusters. As a first step towards that we decided to investigate whether we could run workers in Kubernetes using a Docker-in-Docker sidecar container. The rest of this post gives a rough overview of the architecture of Spacelift workers, and also explains how the sidecar container strategy works.
I’d just like to say thanks to <a href="https://medium.com/@v.m_hs" target="_blank" rel="nofollow noopener">Vadym Martsynovskyy</a> for his great article about running <a href="https://medium.com/hootsuite-engineering/building-docker-images-inside-kubernetes-42c6af855f25" target="_blank" rel="nofollow noopener">Jenkins agents in Kubernetes</a>. In that article he outlines the general approach taken here, and also describes some of the pros and cons of using Docker-in-Docker. It’s definitely well worth a read.

Before going into more detail, it’s worth explaining what a Spacelift worker is, and giving a quick overview of our architecture. At Spacelift, we provide a hybrid-SaaS model like many other CI/CD systems. What this means in practice is that the control plane that handles scheduling runs is managed by us, but the execution of runs happens in a separate process called a worker. We provide a public worker pool, but also allow you to self-host your own workers, providing complete control over your infrastructure.
Perhaps confusingly, if you follow the instructions to create a private worker pool, you’ll notice that it involves downloading something called the “launcher”. The launcher is responsible for communicating with the Spacelift mothership (control plane), and creating and destroying worker containers using Docker for each Spacelift run:

image2-5

Spacelift mothership (control plane)

As you can see, the launcher process needs access to a Docker daemon in order to create a container for each run.

We found that the launcher architecture maps very nicely to Kubernetes using a sidecar container to run the Docker daemon. For each worker in the pool, we need to create a Kubernetes Pod with two containers: one for the launcher; and another for Docker. The following shows a stripped down Deployment definition for creating a worker pool with 4 workers:

The launcher communicates with the Docker-in-Docker sidecar container via TCP, which is configured via the <code>DOCKER_HOST</code> environment variable for the launcher container. This works because the containers in a Kubernetes Pod share the same <a href="https://kubernetes.io/docs/concepts/workloads/pods/#pod-networking" target="_blank" rel="nofollow noopener">IP address and port space, and can communicate with each other via localhost</a>.
In addition, a shared volume called <code>launcher-storage</code> is mounted into each container. This is used by the launcher to store the workspaces for runs, along with other things like cached tool binaries (for example Terraform). The Docker sidecar needs access to the run workspaces in order to mount them into the worker containers, and it also uses that volume to store its image cache.
The last thing worth pointing out is that the <code>dind</code> container sets <code>securityContext.privileged</code> to <code>true</code>. This is required for Docker-in-Docker to function correctly.

The Spacelift launcher is distributed as a statically linked binary, making it simple to build an image. As you can see, the Dockerfile simply copies the launcher into an Alpine container, and then sets the startup command:

This image is rebuilt and published to our public ECR repository any time the launcher binary is updated.

We provide <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a> to make it really easy for users to deploy worker pools to AWS, Azure and GCP, and we wanted to provide a similar experience for Kubernetes. To achieve this, we created a Helm chart that makes it simple to deploy workers to Kubernetes clusters: <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">https://github.com/spacelift-io/spacelift-workerpool-k8s</a>.
Assuming you’ve already configured your worker pool in Spacelift and have access to the credentials for the pool, deploying this chart to your cluster simply requires two steps.
First, add the Spacelift Helm chart repository and update your local chart cache:

Next, install the chart:

Replace <code>&lt;worker-pool-token&gt;</code> and <code>&lt;worker-pool-private-key&gt;</code> with your own credentials, and make sure to base64-encode the private key.
If all goes well, you should be able to view the pods for your worker pool using <code>kubectl get pods</code>:

You should also be able to view the workers in your pool in Spacelift:

image1-5

Bitbucket DC Local Pool

When implementing in a production environment, you may want to use an alternative approach to providing credentials, and may want to configure a custom storage volume for the worker Pods. For more information about configuring the chart, check out the README in the <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">chart GitHub repo</a>.

If you’re interested in running <a href="https://spacelift.io/blog/spacelift-worker-pools-on-kubernetes" target="_blank" rel="noopener">Spacelift workers in Kubernetes</a>, we’d welcome any feedback about the approach, contributions to the Helm chart, and also any issues you encounter so that we can make improvements. Also, if you aren’t already using Spacelift but are interested in trying it, you can signup and <a href="https://spacelift.io/free-trial?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bci_cd_workers_in_kubernetes_dind%7D" target="_blank" rel="noopener">start your free evaluation of Spacelift</a>.

5 Ways to Manage Terraform at Scale - Best Practices

Kubernetes Cheat Sheet - 15 Kubectl Commands & Objects

Infrastructure as Code : Best Practices, Benefits & Examples

How Spacelift Can Improve Your Infrastructure as Code

This article is an overview of the architecture of Spacelift workers with an explanation of how the sidecar container strategy works.

49-running_spacelift_cicd_workers_in_kubernetes_using_dind-3

Working on a SaaS product you’ll occasionally have to take administrative actions to assist a user or help them debug an issue. Maybe something is misconfigured and they need a hand, or possibly they just want another week in their trial. If you’re using something like Django, then you get a nice built-in admin UI based on your model definition, and that’s it &#8211; you have a great tool at your disposal by default. Even though you still need to carefully design the access flow, as they otherwise commonly become attack vectors.
If however you’re building a project in a custom way akin to “let’s combine many libraries and build exactly what we need” instead, then all you’ll have is the raw query language interface of your database &#8211; and that only if you’re lucky enough to have chosen a database which supports non-trivial querying.
In the long run though, as most developers know, running SQL queries on prod isn’t really a scalable or secure approach and requires significant coordination, communication and supervision in order not to accidentally corrupt your data. You also have to limit access to a few select individuals and have to manage access to sensitive data.
Moreover, you might have other dependencies than the SQL database. Then administering all the third-party services would require you to jump around various admin UIs, making everything even more complicated and error prone&#8230;

There’s a whole class of tools tailored specifically to solving this use case &#8211; WYSIWYG back office tools. Using one of these tools you can easily create dialogs with SQL queries underneath and connect them directly to your database. Depending on the tool, it might even be usable by non-technical people.
Unfortunately, these tools also bring a whole class of problems with them, mainly access management &#8211; both for people, and for the tool itself. You can use a cloud-based, managed tool, but then you have to give it arbitrary querying access to your internal production database. This usually makes sense, but is a security consideration that you have to take into account. You can also use self-hosted open source tools, but these come with their own set of trade-offs:
<ul>
<li style="font-weight: 400;" aria-level="1">You have to self-host them, costing you precious time.</li>
<li style="font-weight: 400;" aria-level="1">They need to have access to your production database, and people in your organization need to have access to them. So you either make them publicly accessible &#8211; and trust their security &#8211; or you make them internally accessible, and set up limited internal network access for everybody who needs to use them.</li>
<li style="font-weight: 400;" aria-level="1">Access-management in the tool itself.</li>
</ul>
Other than that, one common disadvantage is that they’re yet another tool you’ll have to use in your day-to-day work. They are also usually optimized for working with a SQL database, not additional third-party services.
After comparing various solutions on both sides of the cloud/open-source divide, we’ve decided we didn’t like either set of trade-offs, which put a hold on the whole initiative for a while.
Then we had an idea… What’s the most popular DevOps administration and management tool? Slack of course! I’m slightly joking, but the ChatOps trend&#8217;s significance cannot be denied. In this case, it did indeed look like it could solve all our problems.

Slack is great for a few reasons:
<ul>
<li style="font-weight: 400;" aria-level="1">It’s a textual interface &#8211; almost a terminal really &#8211; so you can transfer a lot of UX knowledge from your terminal experiences. Developers are always happy to use a text interface.</li>
<li style="font-weight: 400;" aria-level="1">You already manage access to Slack channels, so you can piggyback on top of that and only make your tool available in select channels. This way you don’t have to build additional access management for your tooling.</li>
<li style="font-weight: 400;" aria-level="1">If you design the UX well, it’s very easy to use for non-technical users.</li>
<li style="font-weight: 400;" aria-level="1">You automatically have a public audit log of all executed commands &#8211; the channel message history.</li>
<li style="font-weight: 400;" aria-level="1">You already use Slack.</li>
</ul>
It does have its disadvantages too:
<ul>
<li style="font-weight: 400;" aria-level="1">When Slack is dead (which hasn’t been a rare occurrence in recent times), you can’t access your backoffice tool. But that’s ok, as long as you keep your most critical maintenance commands available through alternative access channels.</li>
</ul>
Now for some Slack technicalities. For a project like this you can use slash commands or a slackbot. The gist of it is that slash commands are more structured, but also more limited. Slackbots on the other hand just interact with your channels like a normal user would, so the flexibility is limitless, but it requires more work on your side. In order to provide an experience that’s as user-friendly and magical as possible, we went with the slackbot approach.
And thus the Backoffice Bot was born…

slackbot1

hello Backoffice Bot

Internally, we already had an event handling system in place for application events &#8211; i.e. GitHub push notifications. We actually even have a <a href="https://docs.spacelift.io/integrations/slack" target="_blank" rel="noopener">Slack integration</a>, so there was ample opportunity for copy-paste driven development.
In practice though, it’s pretty simple. There’s an AWS API Gateway endpoint to handle Slack webhooks, it puts events on an SQS queue and then a handler which has access to all relevant production systems takes care of handling each message and potentially responding to it.

slackbot2

building the Slackbot

One more advantage of the slackbot using our existing event handling framework, is that we not only have access to our SQL database, we also have access to any third-party products we use, like various AWS offerings.
<h3>The Router Structure</h3>
All commands are registered with a message pattern, which is then converted into a regular expression with capture groups for the arguments. Here is an example code block specifying a command with its handler:

Initially we planned to make it a sophisticated <a href="https://en.wikipedia.org/wiki/Trie" target="_blank" rel="nofollow noopener">trie</a>-based router, but that’s just immensely more complex than going over a list of regular expressions and trying to match each. With this amount of traffic and number of commands there was simply no point in optimizing further.
This way of adding commands is really simple and there’s a bonus &#8211; an automatically generated <code>@Backoffice Bot help</code> message, which dynamically lists and describes all commands available in the current channel.
<h3>Approval Flow</h3>
With processes like these, oftentime you want certain commands to require approval. We do this by using Slack reactions. 
The Slackbot will first detail what it would like to do (dry-run so to say), and then ask for approval. Approval is required from the caller, as well as a predefined number of other people, which depends on the “destructiveness” of the command at hand.

slackbot3

slackbot approval flow

In order not to double-execute commands, we’re marking handled messages using a Slack reaction &#8211; this way we don’t have to use an external database. You can see this in the above picture with the Spacelift logo reaction on the root message.
<h3>Access Control</h3>
One additional requirement was that different commands are allowed to be executed by different people. In our case there are commands which are built for non-technical people, for example extending account trials; as well as commands built for technical people, for example displaying diagnostic information or cycling a worker pool’s workers.
We achieve this by limiting the available commands based on the channel. We have a #backoffice channel for non-technical people and a #backoffice-developers channel for technical people. They are private, so we can limit access levels by inviting select people to relevant channels.
The help message takes this into account, and only shows the commands available to you in the current channel.

Everybody using our new Backoffice Bot has quickly fallen in love with it. Adding new commands follows the same process as getting standard product changes into production, which is familiar to all developers out of the box. There are no additional tools, workflows, or processes to maintain. We definitely recommend this approach if you ever find yourself having similar challenges to solve.
Read more about working on our product: <a href="https://spacelift.io/blog/aws-lambda-migration" target="_blank" rel="noopener">Adventures in AWS Lambda Land – a Migration Gone Well</a>.

slackbot4

slackbot

Spacelift is an alternative to using homegrown solutions on top of a generic CI solution. It allows you to automate, audit, secure, and continuously deliver your infrastructure.

Introducing the Spacelift Azure Integration!

Everybody using our new Backoffice Bot has quickly fallen in love with it. That is why in this article, we show how to create that flexible backoffice tool using Slack and what benefits does it bring for a technical company.

26-backoffice_slacbot

flexible Backoffice Tool

<div>
Securing your infrastructure is critical for the success of your business. Failure to take security seriously can result in major damage including fines, loss of customer confidence, or the inability to carry out crucial business functions. The growth of Infrastructure as Code tools and CI/CD systems has allowed developers to integrate infrastructure management into our typical development workflows, improving quality and delivery speed.
At the same time, in order to manage your infrastructure, the CI/CD system used needs access to sensitive credentials. At Spacelift, we aim to give our users the maximum balance between flexibility and security. Because of this, we provide multiple options for connecting your Azure subscriptions to Azure, including setting static credentials, using our fully managed integration, as well as utilizing private workers to avoid sharing credentials at all.
In this post we wanted to give you an overview of how the <a href="https://docs.spacelift.io/integrations/cloud-providers/azure#spacelift-managed-integration" target="_blank" rel="noopener">Spacelift Azure integration</a> works from a technical perspective, as well as discuss some of the issues we encountered and solved while designing and developing it. As a CI/CD system, we do quite a lot of work to integrate with other systems that our users use. Occasionally, like in the case of Azure, things get non-trivial. If you’re keen to learn more, read on, no Azure knowledge required!
</div>

Let’s start with a quick description of how our <a href="https://docs.spacelift.io/integrations/cloud-providers" target="_blank" rel="noopener">cloud integrations</a> work. The overall workflow is very simple, and looks something like this:
&nbsp;

9-2-2

cloud integrations

Breaking the requirements down, we need to be able to get management credentials for a user’s cloud provider account and pass them to <a href="https://spacelift.io/blog/how-to-use-terraform-variables" target="_blank" rel="noopener">Terraform via environment variables</a>, at which point Terraform can run an apply with access to the customer’s infrastructure.
Note: For simplicity this post uses <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> in all its examples, but this overall approach also applies to the other tools that we support, for example, <a href="https://docs.spacelift.io/vendors/pulumi" target="_blank" rel="noopener">Pulumi</a>.
The concept behind the Azure integration was to provide a similar experience to our <a href="https://docs.spacelift.io/integrations/cloud-providers/aws" target="_blank" rel="noopener">AWS</a> and <a href="https://docs.spacelift.io/integrations/cloud-providers/gcp" target="_blank" rel="noopener">GCP</a> integrations, but for our Azure customers. The following diagram shows a simplified outline of how the AWS integration works:

10-2

how the AWS integration works

AWS provides the ability to temporarily <a href="https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html" target="_blank" rel="nofollow noopener">assume a role</a> in another AWS account. This allows our users to create a role in IAM with any permissions they might want to give Spacelift. They can then set up a trust relationship for this role with our AWS account, which will allow our AWS account to assume the role. Role assumption provides us with raw AWS credentials and works seamlessly with any AWS tooling, including the Terraform AWS provider. It additionally allows us to specify the validity duration, so each run can get its own credentials which are constrained to a short period of time. 
Although Azure doesn’t have the same capability, it provides another approach called Azure Active Directory Applications, that allow service accounts to be created. Azure AD Applications are the resources that allow Spacelift to seamlessly manage access to a customer’s Azure resources.

It’s worth explaining a few pieces of terminology that are used throughout the rest of this post:
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://azure.microsoft.com/en-gb/services/active-directory/" target="_blank" rel="nofollow noopener">Azure Active Directory</a> (Azure AD) &#8211; the identity and access management component of Azure.</li>
<li style="font-weight: 400;" aria-level="1">Directory / tenant &#8211; an individual instance of Azure AD owned by a company or individual.</li>
<li style="font-weight: 400;" aria-level="1">Subscription &#8211; the container for any Azure compute resources. This roughly corresponds to an AWS Account. A subscription is linked to a single Azure AD tenant, but multiple subscriptions can be linked to the same tenant.</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-how-applications-are-added" target="_blank" rel="nofollow noopener">Azure AD Application</a> &#8211; a way of creating external integrations with Azure AD.</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/what-is-application-management" target="_blank" rel="nofollow noopener">Enterprise Application</a> &#8211; an instance of an Azure AD application that has been installed in another user’s Azure AD tenant.</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals#service-principal-object" target="_blank" rel="nofollow noopener">Service Principal</a> &#8211; a service account that is automatically created when an Azure AD application is installed. This can be used to grant permissions that allow the application to <a href="https://spacelift.io/blog/managing-azure-resources-using-spacelift" target="_blank" rel="noopener">manage Azure resources</a>.</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.microsoft.com/en-us/graph/" target="_blank" rel="nofollow noopener">Microsoft Graph API</a> &#8211; the main API for managing Azure AD resources.</li>
</ul>

We set a number of goals for the integration:
<ul>
<li style="font-weight: 400;" aria-level="1">Making it really easy for customers to manage Azure infrastructure using Spacelift.</li>
<li style="font-weight: 400;" aria-level="1">Automatic handling of credential rotation so that customers don&#8217;t have to deal with this themselves, or use very long-lived credentials to avoid it entirely.</li>
<li style="font-weight: 400;" aria-level="1">Providing a mechanism for customers to configure granular permissions in Azure for different stacks, or different types of runs (e.g. PRs vs Tracked Runs).</li>
</ul>
&nbsp;

Initially, our idea was to create a single <a href="https://docs.microsoft.com/en-us/azure/active-directory/develop/single-and-multi-tenant-apps" target="_blank" rel="nofollow noopener">multi-tenant AD Application</a>:

multi-tenant AD Application

The idea was that we would generate an Access Token that could only be used for a specific customer directory, and pass that token to the Terraform Azure RM provider during runs. In the end, we had to revise our approach because of the following issues:
<ul>
<li style="font-weight: 400;" aria-level="1">The <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs" target="_blank" rel="nofollow noopener">Terraform Azure RM provider</a> doesn&#8217;t support authentication via an Access Token. Instead, you have to supply the underlying credentials for the account &#8211; either a Client Secret or a Client Certificate. In our case, that would have meant passing the credentials for our own multi-tenant application to Spacelift runs. Since that application would have been installed in the Azure AD tenants of any Spacelift user who had setup the integration, this could have allowed users to access other user’s Azure accounts.</li>
<li style="font-weight: 400;" aria-level="1">The integration would have been less flexible. Using a single multi-tenant AD application would have prevented customers from creating more than one Azure integration per Active Directory tenant. The ability to create multiple integrations per tenant is useful because it allows different Azure permissions to be applied to each integration.</li>
</ul>

After days of brainstorming on an alternative approach, we came up with a new architecture. We could programmatically generate a new Azure AD Application on our side for each Azure Integration created by Spacelift users. This way, having access to the credentials for an Azure AD Application would only lead to having access to a single Azure AD Tenant on a user’s side. This approach allows Client Secrets to be passed to Spacelift runs without fear of inter-user permission leakage. The final design ended up as shown in the following diagram:

generating Azure AD application

Applications are installed into a customer’s Active Directory tenant via a process called <a href="https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/grant-admin-consent" target="_blank" rel="nofollow noopener">Admin Consent</a>. After admin consent has been completed, a Service Principal is created in the user’s Azure Active Directory to which the user can grant permissions. This allows users to decide the exact level of access that Spacelift has to their resources.

The next issue we faced was related to generating credentials for a run. As described in the <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret" target="_blank" rel="nofollow noopener">provider documentation</a>, the Azure RM provider can be configured by setting certain environment variables. Initially, we took a basic approach of attempting to generate credentials during a Spacelift run. This is what we do for our AWS and GCP integrations, so we weren’t expecting major issues. The steps taken looked something like this:
<ol>
<li style="font-weight: 400;" aria-level="1">Run triggered.</li>
<li style="font-weight: 400;" aria-level="1">Generate a new Client Secret with a short expiry time.</li>
<li style="font-weight: 400;" aria-level="1">Populate the required environment variables.</li>
<li style="font-weight: 400;" aria-level="1">Execute terraform.</li>
</ol>
This seemed to work&#8230; but only some of the time.
While testing the integration, strange things were happening. As an example, the planning phase for a run would succeed, but the apply would fail with a permissions error from Azure. After investigating, we came to the conclusion that this was being caused by <a href="https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/active-directory-architecture" target="_blank" rel="nofollow noopener">eventual consistency</a> in Azure AD.
You can visualize the problem using the following diagram (note: this is just an illustration, and is not meant to be completely accurate):

generating credentials issue

In the example above, step 2 may succeed or fail depending on whether the secret has managed to replicate to the Azure AD server that its request is routed to. Initially, we attempted to test whether or not the secret was usable by making an API request, and retrying until the request succeeded using an exponential backoff. What we soon realized was that even then, subsequent requests could be routed to a different Azure AD instance, which still hasn’t received the new Client Secret, and potentially fail.
Even if it was possible to verify when the secret was fully replicated, waiting for replication to complete would have added a minimum of 30 seconds, and potentially another several minutes. Because of this, we decided to move credential generation and rotation out of the run flow, and into a scheduled task:

moving credential generation and rotation into a scheduled task

The scheduled task runs once per hour, generates secrets with an expiry of 24 hours, and attempts to generate a new secret for an integration roughly 2 hours before expiry of the old secret. This allows credential rotation while always keeping a valid secret.
When a new secret is generated, we use AWS’s <a href="https://aws.amazon.com/kms/" target="_blank" rel="nofollow noopener">Key Management Service</a> to encrypt it so that it is never stored in plaintext.
When a run is triggered, we try to find the secret for the integration with the most amount of time until expiry. We also avoid using new secrets until roughly 10 minutes after generation to avoid the eventual consistency issues caused by Azure AD&#8217;s architecture.
You can visualize the secret lifecycle using the following diagram:

the secret lifecycle

In addition, when creating a new integration we immediately generate a secret. This helps to ensure that the secret will have successfully propagated within Azure AD by the time a run is triggered.

The last major issue we faced was figuring out how to implement credential rotation for our own management account. The integration itself uses an Azure AD Service Principal to manage customer AD Applications using the Microsoft Graph API. Because we run most of our own infrastructure in AWS, we didn&#8217;t have the option of using a <a href="https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview" target="_blank" rel="nofollow noopener">Managed Service Identity</a>, meaning that we needed to handle credential rotation ourselves. In addition, our goal was to automate the process to avoid developers having to periodically perform a manual task, and to reduce the risk of forgetting to renew the credentials before expiry.
In the end, we decided to take the relatively simple approach of storing the certificate in Secrets Manager and writing a scheduled task to periodically check whether the certificate was ready to expire, similar to the approach we took for Client Secrets for the integration. If so the scheduled task generates a new certificate and uploads it to both Secrets Manager and Azure AD:

scheduled task generates a new certificate and uploads it to Secrets Manager and Azure AD

The parts of the system that need to use the client certificate for authentication periodically check for an updated certificate. As with the client secret rotation, we avoid using the new certificate for approximately 10 minutes to allow time for the certificate to propagate throughout Azure AD.
Similar to what happens with the integration client secrets, Secrets Manager uses AWS Key Management Service to encrypt the certificate at rest.

Hopefully, this post has given you a glimpse into the internals of Spacelift&#8217;s Azure integration, along with some of the problems we had to solve while implementing it. As you probably noticed, we’re willing to go to great lengths to ensure a secure and pleasant experience for our users. To find out more, take a look at our Azure integration documentation available at <a href="https://docs.spacelift.io/integrations/cloud-providers/azure" target="_blank" rel="noopener">Spacelift Documentation</a>.

How to Manage Active Directory Objects with Azure AD Provider for Terraform

An overview of how the Spacelift Azure integration works from a technical perspective with some of the issues solved while designing and developing it.

2-building_a_secure_cicd_integration_with_azure

The introduction of Infrastructure as Code or IaC has transformed the way you can provision and deploy high-performance cloud-based IT infrastructures.
IaC tools, such as Terraform, have been integrated into DevOps toolchains, saving DevOps IaC teams from the excessive manual effort.
While these tools undoubtedly help accelerate building IT infrastructures, their limitations can impact DevOps’ ability to optimize and improve control of their IaC processes supporting future business needs.
In this article, you’ll learn about IT infrastructure limitations that IaC DevOps teams deal with on a daily basis and how Spacelift is able to get past them.

One of the most frequent challenges while using more generic IaC tools is the non-intuitive workflow driven by its reliance on pull requests. Some solutions offer multiple workspaces, but the result can be fragile and nondeterministic. 
Since there’s no concept of mapping projects to branches or tags, anyone commenting on an approved pull request can deploy arbitrary code to production, even if the approval was meant for a short-lived experimental environment.
Spacelift does not depend on pull requests. It is mostly driven by push and tag events, so building a sophisticated Git flow is much easier. Spacelift reports the outcome of its jobs as commit status checks, allowing you to block merging the code on a failing Spacelift check.
Triggering a run can be customized using <a href="https://docs.spacelift.io/concepts/policy/git-push-policy" target="_blank" rel="noopener">Git push policies</a>. Thanks to that, Spacelift can provide the same level of comfort and security to teams using one project per repository and those using mono repo with hundreds of interdependent projects. You can read more about our approach to VCS integration <a href="https://docs.spacelift.io/integrations/source-control/github#tracked-branches" target="_blank" rel="noopener">here</a>.

The majority of generic tools don’t offer access control models but rely on comments on pull requests to drive infrastructure deployments. While it is usually fine when a single repository drives a single Terraform project, it becomes a huge liability for more complex scenarios.
Spacelift ships with a sophisticated mechanism allowing administrators to declare <a href="https://docs.spacelift.io/concepts/policy/login-policy" target="_blank" rel="noopener">who can log in</a> (and under what circumstances) and what their <a href="https://docs.spacelift.io/concepts/policy/stack-access-policy" target="_blank" rel="noopener">level of access</a> to each of the managed projects should be. Even our <a href="https://docs.spacelift.io/integrations/slack" target="_blank" rel="noopener">Slack integration</a> can be subject to policy controls, allowing an admin to grant access to a project <a href="https://docs.spacelift.io/integrations/slack#managing-access-to-stacks-with-policies" target="_blank" rel="noopener">based on Slack-specific data</a> (think team, channel, user, etc.).

One thing that’s not in scope for most IaC solutions is the way to ensure that your infrastructure is compliant with industry best practices and your company policies.
Spacelift puts policy-as-code in the center of its value proposition and builds a consistent, <a href="https://docs.spacelift.io/concepts/policy" target="_blank" rel="noopener">robust policy framework</a> on top of <a href="https://www.openpolicyagent.org/" target="_blank" rel="nofollow noopener">Open Policy Agent</a>. Apart from providing a comprehensive automated change review and ensuring compliance of your <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy" target="_blank" rel="noopener">Terraform changes</a>, Spacelift uses the same approach to allow you to declare rules around the <a href="https://docs.spacelift.io/concepts/policy/login-policy" target="_blank" rel="noopener">account</a> and <a href="https://docs.spacelift.io/concepts/policy/stack-access-policy" target="_blank" rel="noopener">project access</a>, <a href="https://docs.spacelift.io/concepts/policy/git-push-policy" target="_blank" rel="noopener">handling push notifications</a>, <a href="https://docs.spacelift.io/concepts/policy/run-initialization-policy" target="_blank" rel="noopener">starting runs</a> and <a href="https://docs.spacelift.io/concepts/policy/task-run-policy" target="_blank" rel="noopener">triggering tasks</a>, and creating <a href="https://docs.spacelift.io/concepts/policy/trigger-policy" target="_blank" rel="noopener">relationships between projects</a>.

Handling interdependencies between projects has always been Terraform’s Achilles’ heel. The usual approach to this problem is adding another layer of abstraction in the form of a Terraform wrapper like <a href="https://terragrunt.gruntwork.io/" target="_blank" rel="nofollow noopener">Terragrunt</a>. But it’s only a partial solution as it breaks the problem into smaller chunks.
<a href="https://docs.spacelift.io/concepts/policy/trigger-policy" target="_blank" rel="noopener">Spacelift’s trigger policies</a> on the other hand provide a smart, declarative automation layer on top of vanilla Terraform. They allow you to plug into state changes of individual projects and declare dependencies that should be resolved, following the changes that have just been applied. Read more <a href="https://docs.spacelift.io/concepts/policy/trigger-policy" target="_blank" rel="noopener">here</a> to discover other exciting possibilities.

Another problem to solve externally when using some of the generic tools is authoring and maintaining reusable <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a> for your organization. Terraform is flexible in allowing modules to come from <a href="https://www.terraform.io/docs/modules/sources.html" target="_blank" rel="nofollow noopener">various sources</a>, but ensuring confidential access, as well as testing and versioning, are left to you, the user.
Until now, the golden standard in that regard has been the <a href="https://www.terraform.io/docs/cloud/registry/index.html" target="_blank" rel="nofollow noopener">private module registry from HashiCorp</a>. But Spacelift offers much more. Far from being just a glorified package manager, Spacelift adds a <a href="https://docs.spacelift.io/vendors/terraform/module-registry" target="_blank" rel="noopener">full CI solution for Terraform modules</a>, out of the box and free of charge. You can thus ensure that your private modules are healthy before you distribute them to the rest of your organization.

If you manage a single or a handful of rarely changing projects, it’s likely that you just set your IaC up once and forget about it. But in a more dynamic environment, where microservices come and go, new environments proliferate and new product teams require their own Terraform workspaces. The need to configure it each and every time become a major nuisance, putting a lot of pressure on your DevOps team.
Enter Spacelift. In Spacelift, much of the configuration can be handled by the project owners themselves—you can add Terraform and/or <a href="https://docs.spacelift.io/concepts/configuration/environment" target="_blank" rel="noopener">environment variables</a> and mount files (even inject Terraform code!) programmatically or through the GUI without the need for administrative privileges or changing the central server configuration. For administrators, adding new projects requires minimal hassle since there’s no need to set up webhooks or change any YAML. And it can all be done programmatically using Terraform.

What comes as a pleasant surprise to users of generic CI tools, Spacelift entities such as stacks, contexts, modules or policies can be managed in a declarative way using your favorite infra-as-code tool (this rule applies also to their configuration). Yes, that’s right—<a href="https://docs.spacelift.io/vendors/terraform/terraform-provider" target="_blank" rel="noopener">Spacelift offers a Terraform provider</a> that allows you to programmatically manage the lifecycle of its own resources.
Administrative stacks get credential-less access to the subset of our <a href="https://docs.spacelift.io/integrations/api" target="_blank" rel="noopener">GraphQL API</a> that does not involve managing the actual infrastructure. For more sophisticated use cases, Spacelift allows you to generate API keys that are subject to the same access controls as normal users are, allowing you to create single-purpose tokens for restricted use by your internal scripts.

Generic IaC platforms do not provide any mechanisms to detect if your infrastructure is undergoing drift. Drift is a condition that represents the difference between the desired and the actual state of the infrastructure managed by your tool of choice &#8211; <a href="https://www.terraform.io/" target="_blank" rel="nofollow noopener">Terraform</a>, <a href="https://www.pulumi.com/" target="_blank" rel="nofollow noopener">Pulumi</a>, <a href="https://aws.amazon.com/cloudformation/" target="_blank" rel="nofollow noopener">CloudFormation,</a><a href="https://docs.spacelift.io/vendors/kubernetes"> Kubernetes</a>, or another. Drift can be caused by either or a combination of changes directly introduced by external actors &#8211; either humans or machines (scripts) or via the dependency of your resources on external data sources. In any case, drift is not good.
Spacelift has got you covered here. You can configure periodic <a href="https://docs.spacelift.io/concepts/stack/drift-detection" target="_blank" rel="noopener">drift detection </a>to notify you whenever drift happens, and take immediate action. You can even go a step further with optional automatic <a href="https://docs.spacelift.io/concepts/stack/drift-detection#to-reconcile-or-not-to-reconcile" target="_blank" rel="noopener">reconciliation</a>, ensuring your infrastructure always resembles your Terraform configuration.

General-purpose CI/CD platforms provide little to no insight into resource utilization from either a real-time or historical perspective. Which resources are over-or underutilized? 
Developers need to be able to intimately understand the material they’re working with. With regards to infra-as-code, the most important part of this story is understanding the managed resources in-depth. Both from the current perspective and through being able to put each resource in its historical context.
The resources view in Spacelift gives you greater visibility into each and every resource. With this deep insight into resources, DevOps are able to gain an understanding of the lifecycle of each resource managed by Spacelift and document it, regardless of the technology used — <a href="https://www.terraform.io/" target="_blank" rel="nofollow noopener">Terraform</a>, <a href="https://github.com/gruntwork-io/terragrunt" target="_blank" rel="nofollow noopener">Terragrunt</a>, <a href="https://spacelift.io/blog/what-is-pulumi" target="_blank" rel="noopener">Pulumi</a>, <a href="https://docs.spacelift.io/vendors/kubernetes">Kubernetes</a>, or <a href="https://aws.amazon.com/cloudformation/" target="_blank" rel="nofollow noopener">CloudFormation</a>.

Last but not least, Spacelift puts an emphasis on great user experience, offering a myriad of creature comforts. <a href="https://docs.spacelift.io/concepts/configuration/context" target="_blank" rel="noopener">Contexts</a> for example allow you to attach entire collections of configuration to individual stacks and modules. <a href="https://docs.spacelift.io/concepts/run/task" target="_blank" rel="noopener">Tasks</a> provide a powerful audited way of running one-off administrative commands on an initialized Terraform environment &#8211; subject to <a href="https://docs.spacelift.io/concepts/run/task" target="_blank" rel="noopener">their own policy constraints</a>. <a href="https://docs.spacelift.io/concepts/stack#stack-locking" target="_blank" rel="noopener">Stack locking</a> allows a single individual to take exclusive control over a stack to ensure that nobody is able to modify its state while crucial changes are being made.

Spacelift is an innovative and sophisticated SaaS product for Infrastructure as Code which helps IaC DevOps develop and deploy new infrastructures or changes quickly and with confidence.
Spacelift offers a unique set of IaC management capabilities:
<ol>
<li style="font-weight: 400;" aria-level="1">an intuitive, <a href="https://docs.spacelift.io/concepts/policy/git-push-policy" target="_blank" rel="noopener">versatile</a> and <a href="https://docs.spacelift.io/concepts/run#run-state-machine" target="_blank" rel="noopener">robust</a> workflow</li>
<li style="font-weight: 400;" aria-level="1">granular access controls on <a href="https://docs.spacelift.io/concepts/policy/login-policy" target="_blank" rel="noopener">account</a> and <a href="https://docs.spacelift.io/concepts/policy/stack-access-policy" target="_blank" rel="noopener">project</a> level that work well with an <a href="https://docs.spacelift.io/integrations/single-sign-on" target="_blank" rel="noopener">identity provider of your choice</a> (SSO)</li>
<li style="font-weight: 400;" aria-level="1">an <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy" target="_blank" rel="noopener">automated code review</a> and <a href="https://docs.spacelift.io/concepts/policy/task-run-policy" target="_blank" rel="noopener">threat detection</a> </li>
<li style="font-weight: 400;" aria-level="1">the ability to <a href="https://docs.spacelift.io/concepts/policy/trigger-policy" target="_blank" rel="noopener">declare complex workflows</a> between projects across multiple repositories</li>
<li style="font-weight: 400;" aria-level="1">a <a href="https://docs.spacelift.io/vendors/terraform/module-registry" target="_blank" rel="noopener">built-in private module registry</a> with a full CI system for modules</li>
<li style="font-weight: 400;" aria-level="1">effortless setup and customization with <a href="https://docs.spacelift.io/concepts/configuration/environment" target="_blank" rel="noopener">per-project environment management</a> and <a href="https://docs.spacelift.io/integrations/docker" target="_blank" rel="noopener">Docker integration</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/vendors/terraform/terraform-provider" target="_blank" rel="noopener">programmatic configuration</a> using Terraform</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/concepts/stack/drift-detection" target="_blank" rel="noopener">drift detection </a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/concepts/resources" target="_blank" rel="noopener">resource visualization</a></li>
</ol>
… and a multitude of additional perks like <a href="https://docs.spacelift.io/concepts/configuration/context" target="_blank" rel="noopener">contexts</a>, <a href="https://docs.spacelift.io/concepts/run/task" target="_blank" rel="noopener">tasks</a> or <a href="https://docs.spacelift.io/concepts/stack#stack-locking" target="_blank" rel="noopener">stack locking</a>.

There are many ways to manage your Infrastructure as Code. Each way is different in terms of complexity and offers a different set of features. It is important to keep in mind that choosing one way or another should be based on business and technical requirements. Most often, there is no point in implementing an in-house solution as the cost and effort of building and maintaining it may exceed its potential benefits. It is much easier and more efficient to leverage platforms such as Spacelift to provide these features for you instead.

Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation and controls to simplify and accelerate the provisioning of cloud based infrastructures

Terraform vs. Ansible : Key Differences and Comparison of Tools

Infrastructure Drift Detection and How to Fix It With IaC Tools

Most often, there is no point in implementing an in-house solution as the cost and effort of building and maintaining it may exceed its potential benefits. It is much easier and more efficient to leverage ready-to-go platform to provide these features instead.

how_spacelift_can_improve_your_iac

How Spacelift Can Improve Your Infrastructure as Code 

Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Terraform

Learning Terraform can be easy. The HCL configuration language is simple enough to start with basic resource creation straight away. 
But—
Once the code that defines your infrastructure grows in complexity, maintainability, and dependency, an uphill battle can ensue. And that is why working out an efficient deployment workflow is crucial for further development.
In this article you will see five approaches to managing Terraform workflows at scale, what their benefits and downsides are, and what problems they address.

A quick and easy way to start deploying resources using the Infrastructure as Code pattern is to use Terraform locally from the same machine you’re developing the code on. The only requirement is to have access to the target provider and an installed Terraform binary.
This approach makes it much easier to leverage commands related to day-to-day operations (such as state import, move, remove) or to get outputs than if doing it within the implementations where you have no direct access to the underlying provider.
You can also use all of the features of Terraform, such as remote state and state locking, to work with your teammates all-at-once. If you collaborate using a version control system and do not have continuous integration in place, but want to keep your code at a decent level of quality, there are tools to help you.
One of these is <a href="https://github.com/antonbabenko/pre-commit-terraform" target="_blank" rel="nofollow noopener">pre-commit-terraform</a> that will run a set of selected tools to lint the code you’re creating before you even commit it. Even if you have continuous integration available, it’s a handy way to avoid wasting time waiting for results from pull request status checks.
What are the drawbacks of this approach? Each action is manual and must be performed directly by the developer, which leads to several issues. 
In a situation where multiple people are working on the same codebase, you could often encounter a scenario such as this one:
Person A applies their changes to the environment and follows up with a pull request. At the same time, Person B would like to deploy their changes but is blocked as the codebase is not up-to-date with Person A’s changes. Applying the changes in this state would cause damage to the resources created in the previous deployment. When working in larger teams, this will often result in a lot of wasted time just waiting and rebasing on a codebase to apply the changes.
If you happen to write unit tests for <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a> (and you should be), you’re sure to know they need to be executed manually from your computer. Running them every time you change something in your configuration can be tedious and time-consuming. If you do not yet write tests for your modules, you’d be well advised to start looking at <a href="https://docs.spacelift.io/vendors/terraform/module-registry#tests" target="_blank" rel="noopener">Spacelift modules tests</a> or <a href="https://terratest.gruntwork.io/docs/getting-started/quick-start/" target="_blank" rel="nofollow noopener">Terratest</a>.
Privileged access to an underlying vendor is required, which can easily result in a compromised environment. It’s much harder to keep restricted access within this model, as Terraform must be allowed to create, change, and destroy resources. The same rule applies to a scenario where access to private (internal) resources is required, e.g., every developer has to set up a VPN connection to interact with them.
Moreover, every person working with the codebase needs access to the Terraform state, which creates a security concern due to how Terraform state works. Even if you’re pulling sensitive data from an external system such as Vault, it will be stored within the state file in plaintext once used in a resource. A threat actor could simply run `terraform state pull` to access all the secrets!
Keeping all these issues in mind, let’s take a look at a more automated approach.

Implementing homegrown automation means incorporating Terraform workflow into your continuous integration / continuous deployment process. A specific example would need to directly tie to a CI/CD tool you are using, so let’s try to keep the example as generic as possible.
As the applying actor is the CI/CD system, the hard requirement of privileged access for developers is gone. You can grant this type of access to the execution layer while developers keep read-only permissions. This is true for private (internal) resources, and if your code is being executed on an agent that resides within the target infrastructure with the proper permissions having been set. In a model where everything has to go through the <a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener">CI/CD pipeline</a>, visibility of changes is improved as the system logs produced during the operation execution are available in real-time. 
Linting, compliance checks, and automated unit tests can be moved to pull request status checks, and it’s up to you to configure these steps of the pipeline.
But—
If your CI/CD is executing jobs concurrently, you will end up with race conditions causing pull requests to fail non-deterministically. 
It is not difficult to imagine a situation where two operators open a pull request within several seconds of one another. The pipeline gets triggered immediately for both, and one pull request will be marked as successful, while the other will be marked as failed. This situation will definitely happen if you are using state locking (it is a best practice and you really should be using it).
The explanation is simple: the first pull request has locked the state and therefore the second one could not, so it failed. This issue can be resolved by configuring queued runs, where only one pipeline can run at a time. However, this is not so simple with some CI/CD products.
The biggest issue and concern with this approach is the effort you have to put in to build a complete pipeline. If you consider requirements such as:
<ul>
<li style="font-weight: 400;" aria-level="1">planning on pull requests</li>
<li style="font-weight: 400;" aria-level="1">linting</li>
<li style="font-weight: 400;" aria-level="1">unit tests</li>
<li style="font-weight: 400;" aria-level="1">compliance checks</li>
<li style="font-weight: 400;" aria-level="1">applying once merged</li>
<li style="font-weight: 400;" aria-level="1">periodical drift detection</li>
<li style="font-weight: 400;" aria-level="1">registry for all your versioned modules</li>
<li style="font-weight: 400;" aria-level="1">modules sharing across different parts of the organization</li>
<li style="font-weight: 400;" aria-level="1">shared variable contexts/reusable values</li>
</ul>
It might be tempting to implement solutions for those requirements by yourself. However, taking into account the time and effort required for doing it, you might want to take a look at alternatives, such as open-source tools.

One of the most popular open-source tools to use with Terraform is Atlantis, which describes itself as “Terraform pull request automation”. You can read more about it here: <a href="https://spacelift.io/blog/alternative-to-atlantis" target="_blank" rel="noopener">Alternative to Atlantis</a>.

Terraform Cloud is an application provided by the company behind Terraform itself—HashiCorp. It is available for free for up to 5 users. Two plans are available for more users: “Team &amp; Governance” and “Business”. As the product is created by Terraform authors themselves, any new feature that makes it to Terraform will be quickly available within Terraform Cloud as well. Neat!
See the comparison: <a href="https://spacelift.io/blog/atlantis-terraform" target="_blank" rel="noopener">Atlantis vs. Terraform Cloud</a>.
<h3>Advantages of Terraform Cloud</h3>
Advanced security compliance mechanisms available through Sentinel allow you to enforce organization standards even before the code actually ships and creates resources with the chosen provider context.
Another important feature is a module registry that acts as an artifact repository for all your modules. It can be compared to JFrog Artifactory, Sonatype Nexus, or other similar software. Having every module in a separate repository and versioning allows you to control code promotion in a granular fashion or quickly roll back to if it allows you to go back to the most recent version only, then we should use “the”, but if it lets you go back to some other earlier version then we should use “a” previous version. 
Having a centralized place to store all the pieces also enables you to quickly browse through available modules, their documentation, inputs, outputs, and used resources. It is a great single pane of glass in terms of module management. However, it can only be used within the same Terraform Cloud organization and cannot be shared. This might be troublesome for operations teams that base their separation on the organization level.
There’s also the remote execution backend, which is an exclusive feature for Terraform Cloud. It allows you to work with Terraform on your local computer as you would usually do, upload your local codebase, and run a plan against it. It is a very efficient way of quickly verifying if what you are working on is actually working.
<h3>Disadvantages of Terraform Cloud</h3>
Along with so many benefits, there are some drawbacks as well. Any linting or module unit tests need to be implemented inside your own continuous integration system. This requires an additional effort the development team has to make to sustain their code quality.
Another disadvantage is that you are not able to share common values across your workspaces. Imagine a situation where you share a common set of variables like cloud region or environment name. The only way to achieve something similar to shared context is to have a dedicated workspace that will output this information as an input for all remaining workspaces, although this would be more of a workaround than an actual solution.
Compliance as code is available via Sentinel, a framework created by HashiCorp. Having a built-in solution to develop policies is nice, but operators often already use <a href="https://spacelift.io/blog/what-is-open-policy-agent-and-how-it-works" target="_blank" rel="noopener">Open Policy Agent (OPA)</a> to define these within different areas of the infrastructure. This frequently ends up in a doubled effort as you need to create a similar policy in both frameworks.
The biggest shortcoming of Terraform Cloud is in my opinion the lack of periodical drift detection runs. Once something is applied and the codebase does not get updates, you will never be notified if anything has changed in comparison to the state within the repository. Scheduling a daily “plan” to verify if everything is up-to-date would be highly beneficial to keep the infrastructure as described programmatically.

And here we are. You can <a href="https://spacelift.io/terraform-cloud-alternative" target="_blank" rel="noopener">compare Spacelift directly to Terraform Cloud</a>. They share a lot of functionality with each other in terms of the actual deployment execution, but also in the area of compliance.
While Terraform Cloud uses Sentinel (the HashiCorp approach to implementation for compliance as code), Spacelift leverages Open Policy Agent. It is beneficial to have a similar way of working with policies for companies that already incorporate OPA in their workflow (e.g., for Kubernetes). As there is no need to learn another syntax, operators can focus their efforts on providing compliance, instead of learning yet another tool.
Spacelift also allows you to automate your workflow even further. By having a built-in continuous integration for modules, you can shift your linters or unit tests really quickly into one place dedicated to Terraform. It helps you reduce the effort required to implement integration workflow by letting you utilize a solution that you are already using.
Sometimes you need to work with a Terraform state directly to import something, for example. In scenarios where you are leveraging Terraform Cloud, it is often impossible to do this without accessing the provider yourself. In contrast, Spacelift gives you the ability to run literally any command within the context of a particular codebase via tasks. This way, you do not need to grant any additional permissions to import, move, or remove resources directly.
By utilizing contexts, you can simply share anything you like across multiple stacks—whether it is a set of environment variables or a file. This is something that becomes much more valuable once your codebase complexity increases.
As mentioned in the previous paragraph, drift detection is something that becomes much more important once your infrastructure grows. With Spacelift, it is possible to <a href="https://docs.spacelift.io/concepts/stack/drift-detection#" target="_blank" rel="noopener">schedule a periodic drift detection</a> on any stack. Going even further, you can enable automated remediation if any changes are found compared to what you have in your codebase.

There are many ways of working with Terraform. Each way is different in terms of complexity, and has a different set of features. It is important to keep in mind that choosing one way or another should be based on business and technical requirements. Most times, there is no point in implementing an in-house solution as the cost and effort of building and maintaining it may often exceed its potential benefits. It is much easier and quicker to leverage platforms such as <a href="https://spacelift.io/?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7B5_ways_to_manage_terraform_at_scale%7D" target="_blank" rel="noopener">Spacelift</a> to provide these features for you instead.

Spacelift helps manage Terraform state, build more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and many more.

How to Provision an EKS Kubernetes Cluster with Terraform

How to Get Started with Terraform on Google Cloud Platform

This article goes through five approaches to managing Terraform workflows at scale: using Terraform locally, automating it in-house, using open source solution, using Terraform Cloud, and using Spacelift.

Adventures in AWS Lambda Land – a Migration Gone Well

Get the newsletter

Recent posts

Tricking PostgresSQL into Using an Insane – but 200x Faster – Query Plan

Running Spacelift CI/CD workers in Kubernetes using DinD

Creating a flexible Backoffice Tool in a Technical Company using Slack

Building a Secure CI/CD Integration with Azure