Spacelift’s collaborative platform for infrastructure as code (IaC) enables DevOps teams to manage complex cloud infrastructures with ease. In line with our reputation as an authority on all things related to IaC management, we pride ourselves on sharing only the most precise, comprehensive, and actionable content related to our areas of expertise. See our <a href="https://spacelift.io/blog/editorial-guidelines" target="_blank">Editorial Policy</a>.

About the Spacelift blog

Adam is a backend software engineer at Spacelift. He has a background in full-stack web development, infrastructure automation, and deploying applications to Azure.

All posts by Adam Connelly - Infrastructure as Code Expert

Adam Connelly

Author title

Going to AWS Summit London? 🇬🇧🇬🇧

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

Announcing Enhanced VCS Integration  🎉 🎉

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Check out the Spacelift Engineering blog to discover internal technical architecture posts written by Spacelift's very own engineers. 

Engineering | Spacelift Blog

Engineering

Way back in December 2021, I wrote a post explaining how to <a href="https://spacelift.io/blog/ci-cd-workers-in-kubernetes-dind" target="_blank" rel="noopener">run Spacelift workers in Kubernetes using Docker-in-Docker.</a> We took that initial approach because it was a relatively simple way to get something working in Kubernetes, and it was a very close match to how we run workers in non-containerized environments like EC2. You could almost think of it like performing a lift and shift.
Although the Docker-in-Docker approach worked, it had some problems:
<ul>
<li style="font-weight: 400;" aria-level="1">The Docker-in-Docker daemon needs to run as a <a href="https://kubernetes.io/docs/concepts/security/pod-security-standards/#privileged" target="_blank" rel="nofollow noopener">privileged container</a>, which is a potential security issue.</li>
<li style="font-weight: 400;" aria-level="1">Each worker in the pool has an always running Pod, which uses up resources in the cluster.</li>
<li style="font-weight: 400;" aria-level="1">Scaling down is not safe because we can’t ensure that workers are idle before Kubernetes removes their Pods.</li>
<li style="font-weight: 400;" aria-level="1">It’s not easy to make full use of Kubernetes functionality; for example, using the cluster credentials to pull a private runner image.</li>
</ul>
We were aware at the time that the Docker-in-Docker approach was only ever meant to be a temporary measure, and that eventually we would need to create a more Kubernetes-native approach that solved these problems. We also realized that by integrating more fully with Kubernetes, we could do some really exciting things like provide the ability to view status information about Spacelift workers and runs from within the Kubernetes cluster itself (more on that later!).

Before going any further, it’s worth giving a quick overview of the architecture of Spacelift workers. The following diagram shows the basic architecture of a Spacelift launcher:

basic-architecture-of-a-spacelift-launcher

basic architecture of a Spacelift launcher

As you can see, there’s a “Worker Instance” that runs a “Launcher” process. This process is responsible for communicating with Spacelift (via MQTT and HTTPS, in case you’re interested), and for creating Docker containers when scheduling messages are received from the Spacelift Mothership.
The Docker-in-Docker approach basically mimics this by running Docker as a side-car container (the DinD containers in the following diagram):

dind-containers-diagram

DinD containers diagram

As I mentioned at the start of this post, in the Docker-in-Docker approach used above, a separate Pod is created for each Spacelift Worker in the pool, meaning that if you have a pool size of ten, you’ll have ten Pods sitting in your cluster using up resources, even when all of the Workers are idle.

The approach we’ve taken to solving these problems is to create a <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/" target="_blank" rel="nofollow noopener">Kubernetes operator</a>. The operator pattern allows you to extend Kubernetes by introducing new types of resources into the Kubernetes API. You then build a controller that knows how to reconcile those custom resources, providing additional functionality.
In our case, we’ve defined two new types of resource:
<ul>
<li style="font-weight: 400;" aria-level="1">A WorkerPool resource that represents a set of Spacelift workers.</li>
<li style="font-weight: 400;" aria-level="1">A Worker resource, representing an individual worker.</li>
</ul>
When using the operator, you only define WorkerPool resources. The operator handles creating the Workers based on your WorkerPool definition and also handles all of the communication with the Spacelift Mothership; for example, registering workers and handling scheduling messages.
The following diagram provides a high-level overview of the WorkerPool operator:

high-level-overview-of-the-workerpool-operator

high-level overview of the WorkerPool operator

It’s worth noting here that the WorkerPool and Worker resources are only entries in the Kubernetes API. They don’t consume any compute resources themselves, and no Pods exist for workers, except when a run is in progress. The Run Pods in the diagram above are created as a result of Spacelift runs being assigned to a worker in the pool. They only exist for the lifetime of the run, and then they are removed.
The “Spacelift Controller” shown in the diagram is the part responsible for managing the workers and communicating with Spacelift. It runs as one or more Pods in your cluster and reacts to Kubernetes API events as well as messages from Spacelift. When a new run is scheduled onto a worker that the controller manages, it uses the Kubernetes API to create a Pod for that run, based on the configuration information contained in the WorkerPool resource for the worker.
This design means that we no longer need Docker-in-Docker (normal Kubernetes Pods are created for each Spacelift run), we avoid Pods running in the cluster for workers when no runs are in progress, and we can handle scaling gracefully, making sure we don’t remove a worker while a run is in progress. Also, by making virtually all of the Pod configuration options available via the WorkerPool resource, we allow customers to take advantage of standard Kubernetes features like service accounts, node affinity, custom volumes, etc.

<ul>
<li><a href="https://spacelift.io/blog/how-to-maintain-operations-around-kubernetes-cluster" target="_blank" rel="noopener">How to Maintain Operations Around Kubernetes Cluster</a></li>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Love and Recommend Spacelift</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-as-code-with-generic-ci-cd">Why Generic CI/CD Tools Will Not Deliver Successful IaC</a></li>
</ul>

<a href="https://docs.spacelift.io/concepts/worker-pools#installation" target="_blank" rel="noopener">Getting started</a> with the operator is simple. We provide Kubernetes manifests that can be used with <code>kubectl apply</code>, as well as a <a href="https://github.com/spacelift-io/spacelift-helm-charts/tree/main/spacelift-workerpool-controller" target="_blank" rel="nofollow noopener">Helm chart</a> that can be used to install the Custom Resource Definitions (CRDs) and controller Deployment. For example, here’s how to install the operator using kubectl:

Once the operator is installed, you need to create a <a href="https://spacelift.io/blog/kubernetes-secrets" target="_blank" rel="noopener">Kubernetes Secret</a> to store your worker pool credentials. These credentials can be created during the <a href="https://docs.spacelift.io/concepts/worker-pools#setting-up">standard worker pool setup process</a> on Spacelift.

And finally create your pool:

For the full details of how to install the operator and configure your WorkerPools, please see the documentation here: <a href="https://docs.spacelift.io/concepts/worker-pools#installation" target="_blank" rel="noopener">https://docs.spacelift.io/concepts/worker-pools#installation</a>.

<h3>WorkerPools and workers</h3>
If we take a look at the cluster (you can use <code>kubectl get workerpools</code>), we can see our new pool:

kubectl-get-workerpools

kubectl get workerpools

As well as the name of the pool, you can see a desired pool size, and an actual pool size. The desired pool size corresponds to the <code>poolSize</code> field in your <code>WorkerPool</code> definition, and defines how many workers you want to have in your pool. The actual pool size indicates the number of <code>Worker</code> resources that actually exist, connected to the pool. The operator is responsible for creating these workers and adding or removing workers as the <code>poolSize</code> is updated.
If we take a look at the workers in the pool (<code>kubectl get workers</code>), we can see that there are indeed two workers:

workers-in-the-pool

workers in the pool

The name of each worker is made up of two items: the name of the <code>WorkerPool</code> resource and a unique ID for that worker. Registered indicates whether that worker has been registered with the Spacelift backend, and the status can be either “IDLE” or “BUSY”, indicating whether the worker is ready to process a run, or whether a run is already in progress on that worker.
Describing the worker shows a number of labels and annotations containing information about that worker and the pool it’s connected to:

describing-the-worker

Describing the worker

In the screenshot above, the workers.spacelift.io/ulid label contains the unique ID for that worker, and the workers.spacelift.io/workerpool label contains the unique ID for the pool it belongs to. If we take a look in Spacelift, we can see both workers connected to our pool:

local-kind-pool

local kind pool

You can also see that we automatically include some information about each worker in the “Metadata tags” column. In this case, we can see a <code>k8s-native</code> tag indicating that this worker is from a native Kubernetes pool, a <code>k8s-namespace</code> tag telling us the Kubernetes namespace that the worker is in, as well as the <code>k8s-name</code> tag that contains the name of the Worker resource in Kubernetes.
<h3>Run scheduling</h3>
When a run is assigned to a worker in Spacelift, the Spacelift backend sends a message to that worker via MQTT with all the information required to process that run. In the case of the Kubernetes operator, the controller component receives that scheduling message, creates a new <code>Pod</code> via the Kubernetes API to process the run, and then marks the associated <code>Worker</code> resource in Kubernetes as busy.
When this happens, if you query for the workers in Kubernetes, you will see the updated status, as well as the ID of the run assigned to the worker:

query-for-the-workers-in-kubernetes

query for the workers in Kubernetes

If you take a look at your Pods, you will also see a Pod that’s been created for that run:

look-at-your-pods

look at your Pods

Once the run completes successfully, the Pod is deleted, and the worker status goes back to IDLE again:

worker-idle-status

worker idle status

<h3>Run Pods</h3>
The definition of the <code>Pod</code> created for each run is based on information in the <code>WorkerPool</code> resource. If you create the most basic <code>WorkerPool</code> definition possible, the Pods created will look something like this:

The <code>Pod</code> spec above has been simplified a bit to reduce some of the noise, but at its core are three separate containers used to process the Spacelift run:
<ul>
<li style="font-weight: 400;" aria-level="1">An init container called “init” that performs one-off tasks like downloading the source code for the run, performing role assumption, etc.</li>
<li style="font-weight: 400;" aria-level="1">The “worker” container that actually executes the run.</li>
<li style="font-weight: 400;" aria-level="1">A container called “launcher-grpc” that provides certain services to the worker container that would normally be provided by the launcher process.</li>
</ul>
The <code>init</code> and <code>launcher-grpc</code> containers both use the <code>public.ecr.aws/spacelift/launcher</code> container image. The version of the image to use is sent from the Spacelift backend as part of the run scheduling message, ensuring that the image used is compatible with the current version of the Spacelift backend.
The <code>worker</code> container uses the <a href="https://docs.spacelift.io/concepts/stack/stack-settings.html#runner-image" target="_blank" rel="noopener">runner image</a> defined in your Spacelift stack, falling back to a default image if none is specified (for example, <code>public.ecr.aws/spacelift/runner-terraform:latest</code> for Terraform stacks). The image to use is sent through from the Spacelift backend based on your stack settings.
<h3>Run Pod customization</h3>
When designing the operator, we wanted to keep things as flexible as possible and allow users to make use of as much standard Kubernetes functionality as possible. Because of that, we’ve taken the approach of exposing any Pod configuration options as long as there isn’t a good technical reason to avoid doing so.
For example, say you wanted to use a custom service account, you could define a WorkerPool like this:

The full set of configuration options can be found in our documentation: <a href="https://docs.spacelift.io/concepts/worker-pools#configuration" target="_blank" rel="noopener">https://docs.spacelift.io/concepts/worker-pools#configuration</a>.
<h3>Scaling WorkerPools</h3>
The size of your <code>WorkerPool</code> defines how many <code>Worker</code> resources are created. These <code>workers</code> are effectively virtual, and tell the Spacelift backend that there are resources available to schedule runs. This means the number of workers in your pool defines the maximum concurrent Spacelift runs you can execute at any given time (since each worker can only be assigned a single run at any point in time).
If you need to scale your pool out to allow you to process more concurrent runs, or if you have more workers than you currently need and want to scale down, you have two options: You can edit the <code>poolSize</code> in your <code>WorkerPool</code> resource, or you can use the <code>kubectl scale</code> command. There’s no difference between the two approaches, so feel free to use whichever option makes sense for your use case, but here’s an example of the <code>kubectl scale</code> command:

One of the major advantages of the operator over the previous Docker-in-Docker approach is when scaling down. Because the Docker-in-Docker approach used <code>Deployments</code> or <code>StatefulSets</code>, there wasn’t a great way to hook into the scale-down process to ensure that a <code>Pod</code> processing a run was not deleted while a run was in progress.
The operator solves this problem. When you attempt to scale down a <code>WorkerPool</code>, the operator only scales down idle workers. If there are currently no idle workers available to scale down, the operator waits until the run for a worker completes, and only then removes that worker from the pool.

Hopefully, this post has been interesting and has provided some insight into the reasons we created the Spacelift worker Kubernetes operator, as well as shedding some light on some of the internals of how the operator works.
I’m really proud of what we’ve managed to achieve so far, and I&#8217;m excited to see customers using the operator to run their pools. If you decide to use the operator, please feel free to reach out to tell me and the rest of the team about your experience and to give us any feedback we can use to improve. 
I want to give a huge thanks to everyone at Spacelift who’s contributed to this project. It’s been a multi-team effort, getting tons of help and input from colleagues from our customer success team, solutions engineering, data, and product teams, amongst others. I also want to thank our amazing customers who gave us feedback and participated in early testing of the operator.
And in particular, I want to give a huge shout out to my partner in crime on this project, Elie. It’s been amazing working with such a talented engineer who has such a great mix of pragmatism and attention to detail. There’s no way this project would have been as successful without him!

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities s for infrastructure management.

How to Use Amazon CodeWhisperer (AI Code Generator)

Terraform Cloud Agents vs. Spacelift Workers

What is a Kubernetes Operator? Definition & Examples

See the reasons we created the Spacelift worker Kubernetes operator, and some of the internals of how the operator works.

333-spacelift-workerpool-operator

spacelift workerpool operator

Spacelift WorkerPool Operator: Technical Overview

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Product

The Spacelift module registry provides a private Terraform module registry that is fully compatible with Terraform. The module registry allows you to define reusable <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a> and share them with various different audiences. This includes sharing modules with multiple Stacks in your Spacelift account but also includes sharing modules with other accounts.
The module registry is available to all Spacelift accounts regardless of tier, although certain functionality, like the ability to share modules with other Spacelift accounts, is only available on the Enterprise tier.

Using the module registry involves the following steps:
<ol>
<li style="font-weight: 400;" aria-level="1">Declaring the module source.</li>
<li style="font-weight: 400;" aria-level="1">Adding the module to your Spacelift account.</li>
<li style="font-weight: 400;" aria-level="1">Deploying a version of that module.</li>
<li style="font-weight: 400;" aria-level="1">Consuming that version.</li>
</ol>
<h3>1. Declaring the module source</h3>
Creating a module is simple, and if you already have an existing Terraform module, you&#8217;re almost ready to go. The only additional steps you need to take are defining a config.yml file that describes the module as well as pointing Spacelift at your module source.
The basic folder structure of a Spacelift module looks like this:

As you can see, as well as any Terraform definitions your module requires, there&#8217;s also a .spacelift folder. This folder contains a single file: config.yml. This config file defines certain pieces of information about your module, including its version, along with any test cases.
A minimum example of a config.yml file looks like this:

The folder structure above also includes an examples folder. This folder is not required, and also does not need to be called examples. It would typically contain one or more test cases that can double as usage examples for your module.
<h3>2. Adding the module to your Spacelift account</h3>
There are two main ways to add a module to your Spacelift account: using the UI, or using the <a href="https://spacelift.io/blog/terraform-providers" target="_blank" rel="noopener">Terraform provider</a>. The following steps will show how to add the module via the UI, but if you prefer to manage your account via Terraform, take a look at the <a href="https://registry.terraform.io/providers/spacelift-io/spacelift/latest/docs/resources/module" target="_blank" rel="nofollow noopener">spacelift_module</a> resource for more information.
The first step is to navigate to the modules section of the Spacelift UI and click the Add module button:

spacelift-module-registry-add-a-module

spacelift module registry - add a module

Next, choose the repository containing your module along with your main branch.
For now, we&#8217;ll assume that your repo only contains a single module, so the project root can be left empty:

spacelift-module-registry-create-a-module

spacelift module registry - create a module

For simple use-cases, you should be able to use the default behavior settings:

spacelift-module-registry-default-behavior-settings

spacelift module registry - default behavior settings

Finally, fill out the description settings:

spacelift-module-registry-description-settings

spacelift module registry description settings

For simple situations where your repository contains a single module and is named using <a href="https://developer.hashicorp.com/terraform/registry/modules/publish#requirements" target="_blank" rel="nofollow noopener">HashiCorp&#8217;s module repository naming scheme</a> of <code>terraform-&lt;PROVIDER&gt;-&lt;NAME&gt;</code>, Spacelift will automatically infer the Name and Provider for you, meaning that you don&#8217;t need to enter anything on this screen.
For example, the repository used in the screenshots above was named <code>terraform-blog-8ball</code>, giving it a name of 8ball and a provider of blog.
Congratulations &#8211; you now have a module:

spacelift-module-registry-module

spacelift module registry - module

<h3>3. Publishing a version</h3>
Before you can actually use your module, you need to publish a version. To publish the initial version of your module, click on the Trigger button. Once the version has been published, it should look something like this:

spacelift-module-registry-trigger-button

spacelift module registry - Trigger button

Further versions can be published simply by pushing changes to your Git repository. The only thing to be aware of is that new versions will only be published if you update the <code>module_version</code> in your config.yml file. Another approach to module versioning using git tags is described later in this post.
<h3>4. Consuming that version</h3>
OK great &#8211; we&#8217;ve got a version! Now what?
Now that you have a version published, you need to consume that version from a stack. Consuming a Spacelift module is exactly the same as consuming a module from the Terraform registry. The only difference is you need to specify a different source:

In fact, if you click the Show Instructions button in the Spacelift UI, it will give you a usage example:

spacelift-module-registry-show-instructions-button

spacelift module registry Show Instructions button

NOTE: in order to be able to access your module, Terraform needs to be authenticated with the Spacelift module registry. This happens automatically for you when running Terraform inside a Spacelift stack.
If you want to be able to use a module from outside Spacelift, for example, if you want to test a module out locally, there are a few more steps involved. A few approaches are outlined later in this post.

<ul>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener">How to Improve Your Infrastructure as Code using Terraform</a></li>
</ul>

One of the pieces of functionality that the Spacelift registry provides that isn&#8217;t part of the standard Terraform module registry is testing functionality. Each module can contain one or more test cases.
Test cases serve two purposes: they provide you with confidence that any changes you make haven&#8217;t broken your module, and they serve as usage examples for any consumers.
<h3>Defining a test case</h3>
Each test case is like a mini stack that uses your module, making sure that it can be applied and then destroyed correctly. To add a test case, create a Terraform file that references your module:

And then reference that test case in your config.yml file:

Note, when referencing the module, you should use a relative reference (for example source = &#8220;../../&#8221;) and avoid specifying a version constraint. This ensures that your tests are run against the current version of your module code rather than a previously published version.
Now that your test case has been defined commit your code, push and create a pull request. After opening your PR, you should be able to see it along with any test cases defined in the PRs tab for your module:

spacelift-module-registry-prs-tab

spacelift module registry PRs tab

If you click on a test case you can view the Spacelift run, showing the full plan, apply and destroy lifecycle:

spacelift-module-registry-spacelift-run

spacelift module registry Spacelift run

Note: test cases will only be triggered if the version number of your module is increased. Please remember to do this, otherwise, your changes will be ignored!
<h3>Extending test cases via hooks</h3>
Although the module registry gives you the ability to run one or more test cases, it doesn&#8217;t provide a particular test framework out of the box. However, you can extend your test cases using <a href="https://docs.spacelift.io/concepts/stack/stack-settings.html#customizing-workflow" target="_blank" rel="noopener">Workflow Customization</a> (a.k.a. hooks) to run any additional checks you need.
Let&#8217;s pretend that we want to extend the magic 8-ball module to allow a list of custom outcomes to be supplied. The first step would be to add a new variable to our module to support this:

At this point we can define our new test case by extending our config.yml file to look like this:

As you can see, a new test case has been defined named Custom outcomes. It defines an environment variable <code>TF_VAR_custom_outcomes</code> to allow us to define the set of outcomes via the config.yml file, and defines some <code>after_apply</code> hooks that run the additional checks.
In this simple example, all we want to do is get the value of a <a href="https://spacelift.io/blog/terraform-output" target="_blank" rel="noopener">Terraform output</a> (<code>terraform output -raw outcome</code>), and then check whether it exists in our list of expected outcomes (the echo and <code>grep</code> commands). The reason that this works is that an exit code of 0 from our custom hook will allow the test case to pass, whereas a non-zero exit code causes a failure.
Now that our config.yml file has been updated, it&#8217;s time to create the Terraform definition for the test case. It should look something like the following:

If we try to run this test case now, it should result in a failure since we haven&#8217;t actually implemented the changes. You can see that in the following screenshot:

spacelift-module-registry-custom-outcomes

spacelift module registry custom outcomes

We could then add an implementation like the following to our module to get the test case to pass:

Obviously this is a very simple example, but it should be possible to use the same approach to integrate with more sophisticated tools.

Spacelift modules support <a href="https://docs.spacelift.io/concepts/configuration/environment" target="_blank" rel="noopener">environment variables</a>, <a href="https://docs.spacelift.io/concepts/configuration/context" target="_blank" rel="noopener">context attachments</a> and <a href="https://docs.spacelift.io/integrations/cloud-providers/" target="_blank" rel="noopener">Cloud credential integrations</a>, just like a stack.
The difference between module and stack environments is that with stacks, the environment is made available to each run, whereas with a module, the environment is made available to each test case that runs.
<h3>Cloud Credentials</h3>
Since each module test-case goes through a full plan-apply-destroy lifecycle, it&#8217;s important to understand that Terraform resources will be created by test case executions even though they will be destroyed as part of the test run. If your module creates resources in a cloud provider, we would recommend that you use a completely separate account dedicated for testing to avoid accidentally impacting other environments.
On a related note, if you have multiple test-cases defined for your module, be aware that they could execute simultaneously, leading to conflicts with resource names. For example, if you created a module that contained the following S3 bucket definition, it could fail if two module tests attempted to create the resource at the same time:

Instead, it would make more sense to use a <code>bucket_prefix</code>, or to allow the bucket name to be parameterized via a <a href="https://spacelift.io/blog/how-to-use-terraform-variables" target="_blank" rel="noopener">Terraform variable</a>:

<h3>Overriding test case environment</h3>
The environment for test cases can be overridden on a case-by-case basis via the config.yml file. The config file allows a set of default environment variables to be specified and then allows that environment to be overridden by each test:

Please note, Terraform variables don&#8217;t automatically propagate through to the underlying module used by each test case. This means that for the example above to work, each test case would need to contain Terraform code similar to the following:

The module registry integrates with your VCS system to provide feedback while working on changes to modules in PRs, and also to connect published versions back to their commits. After publishing a new version, you should see a status against your commit that looks something like this:

spacelift-module-registry-integration-with-vcs-systems

spacelift module registry Integration with VCS Systems

Clicking on details shows more information about the version that was published:

spacelift-module-registry-integration-with-vcs-systems-details

spacelift module registry Integration with VCS Systems details

If your module has test cases associated with it, and you push a PR, the result of the checks will be displayed:

spacelift-module-registry-add-simple-example-of-using-hooks-for-testing

spacelift module registry add simple example of using hooks for testing

Spacelift also supports using a single Git repo to store multiple modules. For example, you might have a structure like this:

In this example, we&#8217;re defining four modules: vpc; eks; sqs; and s3. The contents of the eks, sqs and s3 modules have been omitted for simplicity, but each would also contain a config.yml file along with any required Terraform definitions.
To support this, simply set the project root of each module. Here&#8217;s what the vpc module&#8217;s settings might look like:

spacelift-module-registry-vpc-module-settings

spacelift module registry vpc module settings

<h3>Manually via the config.yml file</h3>
The simplest way to version your modules is to manually update the <code>module_version</code> property in your config.yml file:

A typical workflow might look like this:
<ol>
<li> Make any required changes to your module.</li>
<li> Edit config.yml and increase module_version as required.</li>
<li> Create a pull request and make sure any tests pass.</li>
<li> Once your PR has been approved, merge it to publish the new version.</li>
</ol>
<h3>Automatically via tags</h3>
If you don&#8217;t want to have to manually edit your config.yml file each time you make a change, another option is to use a tag based approach. To do this, create a new push policy with the following contents:

This policy tells Spacelift to trigger version publishing whenever a new tag in the format <code>vx.y.z</code> is pushed to your repo and also tells Spacelift to use the tag as the version number. Because Terraform module versions don&#8217;t support the <code>v</code> prefix, the policy removes that from the start of the tag.
To enable the policy, attach it to your module:

spacelift-module-registry-policy

spacelift module registry policy

Now that your policy is in place, pushes to your main branch will not trigger new versions to be published. Instead a new version will be published when a new tag is pushed to your repository.

Spacelift keeps track of the stacks that consume particular modules. This means that you can automatically trigger any consumers when a new version of the module is published. To do this, you need to do two things:
<ol>
<li style="font-weight: 400;" aria-level="1">Make sure that any Terraform version constraints allow new versions to be used.</li>
<li style="font-weight: 400;" aria-level="1">Attach a trigger policy to your module telling it to trigger any consumers.</li>
</ol>
<h3>Version constraints</h3>
When referencing a Terraform module, you can specify an optional version constraint. This defines the set of versions that your stack is able to use. For example, if you published version 2.0.0 of a module, but had the following module reference, the new version would not be picked up automatically:

<h3>Trigger policy</h3>
Trigger policies for modules accept the list of consumers of the current latest version of the module as an input to the policy. This allows you to write a simple trigger policy like the following to automatically re-trigger any consumers:

Once you have attached this trigger policy to your module and released a new version of the module, you should see runs triggered on any stacks that are currently using the latest version of the module. It should look something like this:

spaceift-module-registry-trigger-policy

spaceift module registry trigger policy

NOTE: Please be aware that the list of stacks provided to the trigger policy only includes those that are using the latest version of the module at the time that the new version is triggered. This means that if the latest version of a module was 0.0.5, but a stack is currently only using version 0.0.4 of your module, it will not automatically be included in the list of consumers. To fix this, you will need to manually trigger your stack once to make sure it&#8217;s using the latest version.
For more information, see our <a href="https://docs.spacelift.io/concepts/policy/trigger-policy#module-updates" target="_blank" rel="noopener">trigger policy</a> documentation.

Spacelift modules can optionally be used by other Spacelift accounts than the one the module is defined in. This can be useful, for example, if you have multiple Spacelift accounts or if you have clients who also use Spacelift who use your custom Terraform modules.
The list of accounts that can access a module can be configured via the sharing section of the module settings:

sharing-modules-with-other-accounts

Sharing Modules with Other Accounts

For more information, see our <a href="https://docs.spacelift.io/vendors/terraform/module-registry#sharing-modules" target="_blank" rel="noopener">documentation</a>. Also, please note that module sharing is only available to customers on the Enterprise tier.

<h3>Using modules locally</h3>
If you want to use Spacelift modules while running Terraform locally, you need to tell Terraform how to authenticate with the Spacelift registry. The easiest way to do this is via the terraform login command. To do this, run the following command and then follow the instructions to login to your Spacelift account via your web browser:

Once authentication has completed successfully, you should see a message similar to the following in your console:

spacelift-module-registry-using-modules-locally

spacelift module registry using modules locally

Now that you are authenticated, you can add a reference to a Spacelift module and run terraform commands locally as normal.
<h3>Using modules in an automated environment</h3>
If you need to use a module in an automated environment outside of Spacelift, the best bet is to use an API key. Please see our <a href="https://docs.spacelift.io/integrations/api#spacelift-api-key-token" target="_blank" rel="noopener">documentation</a> for more information on generating and using API keys.

For more information about the Spacelift module registry, please see the documentation available at <a href="https://docs.spacelift.io/vendors/terraform/module-registry" target="_blank" rel="noopener">docs.spacelift.io</a>.
And see <a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">why DevOps Engineers recommend Spacelift</a>.

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift effectively manages Terraform state, more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and includes many more features.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Spacelift effectively manages Terraform state, more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and includes many more features.

Monitoring Your Spacelift Account via Prometheus

Introduction to Open Policy Agent (OPA) Rego Language

5 Ways to Manage Terraform at Scale - Best Practices

See how to use the Spacelift module registry - a private Terraform module registry that is fully compatible with Terraform.

147-spacelift-module-registry

Spacelift Module Registry

Spacelift Module Registry - What It is and How to Use It

At Spacelift, we&#8217;re excited to announce our new Cloud Integrations section and our update to support account-level AWS integrations. Public Cloud integration is nothing new in Spacelift, but we wanted to simplify the implementation and organization. Cloud Integrations brings feature parity to AWS and Azure integrations across your Spacelift account.

We’ve created a new Cloud Integrations section in the Spacelift console. This section allows you to manage AWS and Azure integrations from a single location. Azure integrations were previously in the settings section and have been moved from there.

spacelift-cloud-integration-aws

spacelift cloud integration - aws

In addition, we’ve added the ability to attach more than one AWS integration to your stack:

spacelift-cloud-integration-more-than-one-aws-integration

spacelift cloud integration - more than one AWS integration

A stack can have at most two AWS integrations attached, with only one read and one write integration (see “<a href="#read-vs-write-attachments">Read vs. Write Attachments</a>” for more info). An integration can also be attached as both read and write for simple use-cases.
Finally, we’ve also altered the external ID format used when assuming roles. More details are available later in this post.

<ul>
<li><a href="https://spacelift.io/blog/spacelift-in-github-marketplace" target="_blank" rel="noopener">Spacelift in the GitHub Marketplace</a></li>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Problems that Spacelift Solves&quot;}" data-sheets-userformat="{&quot;2&quot;:4224,&quot;10&quot;:2,&quot;15&quot;:&quot;Arial&quot;}">Infrastructure Problems that Spacelift Solves</a></li>
</ul>

Don’t worry, your existing AWS integrations will continue to work, and the existing <code>spacelift_aws_role</code> resource can still be used. The only caveat to this is that you will only be able to add new integrations to stacks via the front end using the new Cloud Integrations approach.

Migrating from legacy integrations is easy, with slightly different steps depending on whether you manage your integrations via Terraform or the Spacelift UI. In the simplest case, you just need to create a new integration, delete your old integration, and attach a new one, but you may also need to adjust the Trust Relationships for your <a href="https://spacelift.io/blog/aws-iam-roles" target="_blank" rel="noopener">IAM roles</a> depending on exactly how they are configured.
For more information on migrating, take a look at our <a href="https://docs.spacelift.io/integrations/cloud-providers/aws#migrating-from-legacy-integrations" target="_blank" rel="noopener">documentation</a>.

An integration can be attached to a stack as either read or write. This tells Spacelift which run phases to use each integration during. For simple use-cases, you can attach a single integration as both read and write, but if you want more control, you can create a read-only role and a write role, and attach both to your stack. An example use-case for this is to make sure that a proposed run doesn’t have write permissions to avoid infrastructure changes being made via a pull request.
Read attachments are used during the Initialize and Plan phases of runs. Write attachments are used during the Apply phase of a run, the Perform phase of a task, and the Destroy phase of a stack destructor.

When choosing to assume the role at Spacelift’s end rather than on your private workers, we automatically generate an External ID based on the stack the integration is attached to. Previously we used the following external ID format:

As part of the changes we’ve made, we’ve adjusted our external ID format to the following:

The change means that you can use the “read” or “write” suffix to ensure that a read phase of a run can only assume a read-only role.

We really hope that these changes will be useful and will provide extra flexibility for those users who need it while making simple use-cases easier through reusing integrations. To find out more about Cloud Integrations, review the <a href="https://docs.spacelift.io/integrations/cloud-providers/" target="_blank" rel="noopener">documentation</a>.

Spacelift is a sophisticated SaaS product for Infrastructure as Code that helps DevOps develop and deploy new infrastructures or changes quickly and with confidence.

Spacelift in the GitHub Marketplace

How Spacelift Can Improve Your Infrastructure as Code

We are excited to announce our new Cloud Integrations section and our update to support account-level AWS integrations.

111-spacelift-cloud-integration

Introducing Cloud Integrations

Introducing Spacelift Cloud Integrations

Monitoring your Spacelift account via Prometheus

For a while now, we have received feedback from customers that they would like to be able to connect their Spacelift account to external monitoring systems. Today we are excited to announce the <a href="https://github.com/spacelift-io/prometheus-exporter" target="_blank" rel="nofollow noopener">Prometheus Exporter for Spacelift</a>! We also have plans to add support for &lt;insert monitoring system X here&gt;, so if you’re using another system, we should have you covered soon.

The Prometheus exporter allows you to monitor various metrics about your Spacelift account over time. You can then use tools like Grafana to visualize those changes and Alertmanager to take actions based on account metrics. Several metrics are available, and you can find the complete list of available metrics <a href="https://github.com/spacelift-io/prometheus-exporter#available-metrics" target="_blank" rel="nofollow noopener">here</a>. Below are a few examples of the information the exporter currently provides:
<ul>
<li style="font-weight: 400;" aria-level="1">The number of runs pending and currently executing in both public and private worker pools.</li>
<li style="font-weight: 400;" aria-level="1">The number of workers in a pool.</li>
<li style="font-weight: 400;" aria-level="1">Usage information, including the number of public and private worker minutes used during the current billing period.</li>
</ul>
Once you have that information, it opens a number of possibilities, including visualizing information about your account via Grafana dashboards, alerting on events like a lack of private workers, as well as using that information to autoscale worker pools via the Horizontal Pod Autoscaler.
To give you a taste, here’s an example Grafana dashboard showing some of the information available from the exporter:

grafana-dashboard

Grafana dashboard

The Prometheus exporter is an adaptor between Prometheus and the Spacelift GraphQL API. Whenever Prometheus asks for the current metrics, the exporter makes a GraphQL request and converts it into the metrics format Prometheus expects.
The following diagram gives an overview of this process:

prometheus-exporter-diagram

Prometheus exporter diagram

Ok, great, but how do I use the Prometheus Exporter?
To help with setup and deployment, we are going to guide you through the following:
<ol>
<li style="font-weight: 400;" aria-level="1">Deploy a Prometheus stack to a Kubernetes cluster.</li>
<li style="font-weight: 400;" aria-level="1">Deploy the exporter.</li>
<li style="font-weight: 400;" aria-level="1">Configure  Prometheus to monitor it.</li>
</ol>
The <a href="https://github.com/spacelift-io/prometheus-exporter#quick-start" target="_blank" rel="nofollow noopener">Quick Start</a> section in the exporter repo outlines several options for deploying the exporter. Review the best deployment option that makes sense, depending on your setup and requirements.
If you already have a Prometheus stack setup and have plenty of experience, feel free to skip to the “Installing the Prometheus Exporter” section.
We also assume that you already have a Kubernetes cluster provisioned (hopefully <a href="https://docs.spacelift.io/vendors/kubernetes/" target="_blank" rel="noopener">via Spacelift!</a>) and available to complete the following steps. If not, a local installation like <a href="https://minikube.sigs.k8s.io/docs/start/" target="_blank" rel="nofollow noopener">Minikube</a> will work fine for illustration purposes.
<h3>Installing kube-prometheus-stack</h3>
The first step is to install the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" target="_blank" rel="nofollow noopener">kube-prometheus-stack Helm chart</a>. You can do that with the following commands:

That will install a Prometheus, Grafana, and Alertmanager stack into your cluster&#8217;s namespace called <code>monitoring</code>. Run the <code>kubectl get pods -n monitoring</code> command to see the installed components.

<h3>Getting API Credentials</h3>
The Prometheus exporter authenticates to the Spacelift GraphQL API using an API key. Follow the <a href="https://docs.spacelift.io/integrations/api#spacelift-api-key-greater-than-token" target="_blank" rel="noopener">guide</a><a href="https://docs.spacelift.io/integrations/api#spacelift-api-key-greater-than-token" target="_blank" rel="noopener"> to create a new API key</a> required by the explorer. After you create your key, please note the API Key ID and API Key Secret &#8211; you’ll need both when configuring the exporter.
<h3>Installing the Prometheus Exporter</h3>
The exporter is available via a Docker image published to the <code>public.ecr.aws/spacelift/promex</code> container registry. To deploy the exporter to Kubernetes, we need to create the following resources:
<ul>
<li style="font-weight: 400;" aria-level="1">A Deployment &#8211; to run the exporter container.</li>
<li style="font-weight: 400;" aria-level="1">A Service &#8211; to allow Prometheus to scrape the exporter.</li>
<li style="font-weight: 400;" aria-level="1">A ServiceMonitor &#8211; to let Prometheus know that it needs to scrape the exporter.</li>
</ul>
The following is an example Deployment definition for running the exporter. Make sure to replace the <code>&lt;account name&gt;</code>, <code>&lt;API Key&gt;</code> and <code>&lt;API Secret&gt;</code> placeholders with the correct values:

Note: The example above defines the API key and secret as normal environment variables. We would recommend that you use <a href="https://spacelift.io/blog/kubernetes-secrets" target="_blank" rel="noopener">Kubernetes Secrets</a> for anything other than testing purposes.
Next, create a Service to expose the exporter:

And finally create your ServiceMonitor:

The ServiceMonitor definition above tells Kubernetes to scrape any services that contain an <code>app</code> label. The granularity of the metrics can be increased or decreased depending on requirements and organizational standards. The above example is configured for 15-second intervals.
<h3>Viewing your Metrics</h3>
Once you have your Prometheus stack up and running and have deployed the Spacelift exporter, you can use port-forwarding to access each component. First, let’s port-forward the exporter to port 8080 locally using the following command:

Assuming all is well, you should be able to see the raw metrics output by accessing <a href="http://localhost:8080/metrics" target="_blank" rel="nofollow noopener">http://localhost:8080/metrics</a>:

We are excited to announce the Prometheus Exporter for Spacelift! See how it allows you to monitor various metrics about your Spacelift account over time.

Adam Connelly

About the Spacelift blog

Spacelift WorkerPool Operator: Technical Overview

Get the newsletter

Recent posts

Spacelift Module Registry – What It is and How to Use It

Introducing Spacelift Cloud Integrations

Monitoring Your Spacelift Account via Prometheus

Running Spacelift CI/CD workers in Kubernetes using DinD

Building a Secure CI/CD Integration with Azure

Introducing the Spacelift Azure Integration!

Managing Active Directory with Azure AD Terraform Provider

What is OPA? Open Policy Agent Examples & Tutorial

Managing Azure Resources using Spacelift