Adam is a backend software engineer at Spacelift. He has a background in full-stack web development, infrastructure automation, and deploying applications to Azure.

All posts by Adam Connelly - Infrastructure as Code Expert

Adam Connelly

Author title

OpenTofu is now part of the Linux Foundation 🎉

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Product

The Spacelift module registry provides a private Terraform module registry that is fully compatible with Terraform. The module registry allows you to define reusable <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a> and share them with various different audiences. This includes sharing modules with multiple Stacks in your Spacelift account but also includes sharing modules with other accounts.
The module registry is available to all Spacelift accounts regardless of tier, although certain functionality, like the ability to share modules with other Spacelift accounts, is only available on the Enterprise tier.

Using the module registry involves the following steps:
<ol>
<li style="font-weight: 400;" aria-level="1">Declaring the module source.</li>
<li style="font-weight: 400;" aria-level="1">Adding the module to your Spacelift account.</li>
<li style="font-weight: 400;" aria-level="1">Deploying a version of that module.</li>
<li style="font-weight: 400;" aria-level="1">Consuming that version.</li>
</ol>
<h3>1. Declaring the module source</h3>
Creating a module is simple, and if you already have an existing Terraform module, you&#8217;re almost ready to go. The only additional steps you need to take are defining a config.yml file that describes the module as well as pointing Spacelift at your module source.
The basic folder structure of a Spacelift module looks like this:

As you can see, as well as any Terraform definitions your module requires, there&#8217;s also a .spacelift folder. This folder contains a single file: config.yml. This config file defines certain pieces of information about your module, including its version, along with any test cases.
A minimum example of a config.yml file looks like this:

The folder structure above also includes an examples folder. This folder is not required, and also does not need to be called examples. It would typically contain one or more test cases that can double as usage examples for your module.
<h3>2. Adding the module to your Spacelift account</h3>
There are two main ways to add a module to your Spacelift account: using the UI, or using the <a href="https://spacelift.io/blog/terraform-providers" target="_blank" rel="noopener">Terraform provider</a>. The following steps will show how to add the module via the UI, but if you prefer to manage your account via Terraform, take a look at the <a href="https://registry.terraform.io/providers/spacelift-io/spacelift/latest/docs/resources/module" target="_blank" rel="nofollow noopener">spacelift_module</a> resource for more information.
The first step is to navigate to the modules section of the Spacelift UI and click the Add module button:

spacelift-module-registry-add-a-module

spacelift module registry - add a module

Next, choose the repository containing your module along with your main branch.
For now, we&#8217;ll assume that your repo only contains a single module, so the project root can be left empty:

spacelift-module-registry-create-a-module

spacelift module registry - create a module

For simple use-cases, you should be able to use the default behavior settings:

spacelift-module-registry-default-behavior-settings

spacelift module registry - default behavior settings

Finally, fill out the description settings:

spacelift-module-registry-description-settings

spacelift module registry description settings

For simple situations where your repository contains a single module and is named using <a href="https://developer.hashicorp.com/terraform/registry/modules/publish#requirements" target="_blank" rel="nofollow noopener">HashiCorp&#8217;s module repository naming scheme</a> of <code>terraform-&lt;PROVIDER&gt;-&lt;NAME&gt;</code>, Spacelift will automatically infer the Name and Provider for you, meaning that you don&#8217;t need to enter anything on this screen.
For example, the repository used in the screenshots above was named <code>terraform-blog-8ball</code>, giving it a name of 8ball and a provider of blog.
Congratulations &#8211; you now have a module:

spacelift-module-registry-module

spacelift module registry - module

<h3>3. Publishing a version</h3>
Before you can actually use your module, you need to publish a version. To publish the initial version of your module, click on the Trigger button. Once the version has been published, it should look something like this:

spacelift-module-registry-trigger-button

spacelift module registry - Trigger button

Further versions can be published simply by pushing changes to your Git repository. The only thing to be aware of is that new versions will only be published if you update the <code>module_version</code> in your config.yml file. Another approach to module versioning using git tags is described later in this post.
<h3>4. Consuming that version</h3>
OK great &#8211; we&#8217;ve got a version! Now what?
Now that you have a version published, you need to consume that version from a stack. Consuming a Spacelift module is exactly the same as consuming a module from the Terraform registry. The only difference is you need to specify a different source:

In fact, if you click the Show Instructions button in the Spacelift UI, it will give you a usage example:

spacelift-module-registry-show-instructions-button

spacelift module registry Show Instructions button

NOTE: in order to be able to access your module, Terraform needs to be authenticated with the Spacelift module registry. This happens automatically for you when running Terraform inside a Spacelift stack.
If you want to be able to use a module from outside Spacelift, for example, if you want to test a module out locally, there are a few more steps involved. A few approaches are outlined later in this post.

<ul>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener">How to Improve Your Infrastructure as Code using Terraform</a></li>
</ul>

One of the pieces of functionality that the Spacelift registry provides that isn&#8217;t part of the standard Terraform module registry is testing functionality. Each module can contain one or more test cases.
Test cases serve two purposes: they provide you with confidence that any changes you make haven&#8217;t broken your module, and they serve as usage examples for any consumers.
<h3>Defining a test case</h3>
Each test case is like a mini stack that uses your module, making sure that it can be applied and then destroyed correctly. To add a test case, create a Terraform file that references your module:

And then reference that test case in your config.yml file:

Note, when referencing the module, you should use a relative reference (for example source = &#8220;../../&#8221;) and avoid specifying a version constraint. This ensures that your tests are run against the current version of your module code rather than a previously published version.
Now that your test case has been defined commit your code, push and create a pull request. After opening your PR, you should be able to see it along with any test cases defined in the PRs tab for your module:

spacelift-module-registry-prs-tab

spacelift module registry PRs tab

If you click on a test case you can view the Spacelift run, showing the full plan, apply and destroy lifecycle:

spacelift-module-registry-spacelift-run

spacelift module registry Spacelift run

Note: test cases will only be triggered if the version number of your module is increased. Please remember to do this, otherwise, your changes will be ignored!
<h3>Extending test cases via hooks</h3>
Although the module registry gives you the ability to run one or more test cases, it doesn&#8217;t provide a particular test framework out of the box. However, you can extend your test cases using <a href="https://docs.spacelift.io/concepts/stack/stack-settings.html#customizing-workflow" target="_blank" rel="noopener">Workflow Customization</a> (a.k.a. hooks) to run any additional checks you need.
Let&#8217;s pretend that we want to extend the magic 8-ball module to allow a list of custom outcomes to be supplied. The first step would be to add a new variable to our module to support this:

At this point we can define our new test case by extending our config.yml file to look like this:

As you can see, a new test case has been defined named Custom outcomes. It defines an environment variable <code>TF_VAR_custom_outcomes</code> to allow us to define the set of outcomes via the config.yml file, and defines some <code>after_apply</code> hooks that run the additional checks.
In this simple example, all we want to do is get the value of a <a href="https://spacelift.io/blog/terraform-output" target="_blank" rel="noopener">Terraform output</a> (<code>terraform output -raw outcome</code>), and then check whether it exists in our list of expected outcomes (the echo and <code>grep</code> commands). The reason that this works is that an exit code of 0 from our custom hook will allow the test case to pass, whereas a non-zero exit code causes a failure.
Now that our config.yml file has been updated, it&#8217;s time to create the Terraform definition for the test case. It should look something like the following:

If we try to run this test case now, it should result in a failure since we haven&#8217;t actually implemented the changes. You can see that in the following screenshot:

spacelift-module-registry-custom-outcomes

spacelift module registry custom outcomes

We could then add an implementation like the following to our module to get the test case to pass:

Obviously this is a very simple example, but it should be possible to use the same approach to integrate with more sophisticated tools.

Spacelift modules support <a href="https://docs.spacelift.io/concepts/configuration/environment" target="_blank" rel="noopener">environment variables</a>, <a href="https://docs.spacelift.io/concepts/configuration/context" target="_blank" rel="noopener">context attachments</a> and <a href="https://docs.spacelift.io/integrations/cloud-providers/" target="_blank" rel="noopener">Cloud credential integrations</a>, just like a stack.
The difference between module and stack environments is that with stacks, the environment is made available to each run, whereas with a module, the environment is made available to each test case that runs.
<h3>Cloud Credentials</h3>
Since each module test-case goes through a full plan-apply-destroy lifecycle, it&#8217;s important to understand that Terraform resources will be created by test case executions even though they will be destroyed as part of the test run. If your module creates resources in a cloud provider, we would recommend that you use a completely separate account dedicated for testing to avoid accidentally impacting other environments.
On a related note, if you have multiple test-cases defined for your module, be aware that they could execute simultaneously, leading to conflicts with resource names. For example, if you created a module that contained the following S3 bucket definition, it could fail if two module tests attempted to create the resource at the same time:

Instead, it would make more sense to use a <code>bucket_prefix</code>, or to allow the bucket name to be parameterized via a <a href="https://spacelift.io/blog/how-to-use-terraform-variables" target="_blank" rel="noopener">Terraform variable</a>:

<h3>Overriding test case environment</h3>
The environment for test cases can be overridden on a case-by-case basis via the config.yml file. The config file allows a set of default environment variables to be specified and then allows that environment to be overridden by each test:

Please note, Terraform variables don&#8217;t automatically propagate through to the underlying module used by each test case. This means that for the example above to work, each test case would need to contain Terraform code similar to the following:

The module registry integrates with your VCS system to provide feedback while working on changes to modules in PRs, and also to connect published versions back to their commits. After publishing a new version, you should see a status against your commit that looks something like this:

spacelift-module-registry-integration-with-vcs-systems

spacelift module registry Integration with VCS Systems

Clicking on details shows more information about the version that was published:

spacelift-module-registry-integration-with-vcs-systems-details

spacelift module registry Integration with VCS Systems details

If your module has test cases associated with it, and you push a PR, the result of the checks will be displayed:

spacelift-module-registry-add-simple-example-of-using-hooks-for-testing

spacelift module registry add simple example of using hooks for testing

Spacelift also supports using a single Git repo to store multiple modules. For example, you might have a structure like this:

In this example, we&#8217;re defining four modules: vpc; eks; sqs; and s3. The contents of the eks, sqs and s3 modules have been omitted for simplicity, but each would also contain a config.yml file along with any required Terraform definitions.
To support this, simply set the project root of each module. Here&#8217;s what the vpc module&#8217;s settings might look like:

spacelift-module-registry-vpc-module-settings

spacelift module registry vpc module settings

<h3>Manually via the config.yml file</h3>
The simplest way to version your modules is to manually update the <code>module_version</code> property in your config.yml file:

A typical workflow might look like this:
<ol>
<li> Make any required changes to your module.</li>
<li> Edit config.yml and increase module_version as required.</li>
<li> Create a pull request and make sure any tests pass.</li>
<li> Once your PR has been approved, merge it to publish the new version.</li>
</ol>
<h3>Automatically via tags</h3>
If you don&#8217;t want to have to manually edit your config.yml file each time you make a change, another option is to use a tag based approach. To do this, create a new push policy with the following contents:

This policy tells Spacelift to trigger version publishing whenever a new tag in the format <code>vx.y.z</code> is pushed to your repo and also tells Spacelift to use the tag as the version number. Because Terraform module versions don&#8217;t support the <code>v</code> prefix, the policy removes that from the start of the tag.
To enable the policy, attach it to your module:

spacelift-module-registry-policy

spacelift module registry policy

Now that your policy is in place, pushes to your main branch will not trigger new versions to be published. Instead a new version will be published when a new tag is pushed to your repository.

Spacelift keeps track of the stacks that consume particular modules. This means that you can automatically trigger any consumers when a new version of the module is published. To do this, you need to do two things:
<ol>
<li style="font-weight: 400;" aria-level="1">Make sure that any Terraform version constraints allow new versions to be used.</li>
<li style="font-weight: 400;" aria-level="1">Attach a trigger policy to your module telling it to trigger any consumers.</li>
</ol>
<h3>Version constraints</h3>
When referencing a Terraform module, you can specify an optional version constraint. This defines the set of versions that your stack is able to use. For example, if you published version 2.0.0 of a module, but had the following module reference, the new version would not be picked up automatically:

<h3>Trigger policy</h3>
Trigger policies for modules accept the list of consumers of the current latest version of the module as an input to the policy. This allows you to write a simple trigger policy like the following to automatically re-trigger any consumers:

Once you have attached this trigger policy to your module and released a new version of the module, you should see runs triggered on any stacks that are currently using the latest version of the module. It should look something like this:

spaceift-module-registry-trigger-policy

spaceift module registry trigger policy

NOTE: Please be aware that the list of stacks provided to the trigger policy only includes those that are using the latest version of the module at the time that the new version is triggered. This means that if the latest version of a module was 0.0.5, but a stack is currently only using version 0.0.4 of your module, it will not automatically be included in the list of consumers. To fix this, you will need to manually trigger your stack once to make sure it&#8217;s using the latest version.
For more information, see our <a href="https://docs.spacelift.io/concepts/policy/trigger-policy#module-updates" target="_blank" rel="noopener">trigger policy</a> documentation.

Spacelift modules can optionally be used by other Spacelift accounts than the one the module is defined in. This can be useful, for example, if you have multiple Spacelift accounts or if you have clients who also use Spacelift who use your custom Terraform modules.
The list of accounts that can access a module can be configured via the sharing section of the module settings:

sharing-modules-with-other-accounts

Sharing Modules with Other Accounts

For more information, see our <a href="https://docs.spacelift.io/vendors/terraform/module-registry#sharing-modules" target="_blank" rel="noopener">documentation</a>. Also, please note that module sharing is only available to customers on the Enterprise tier.

<h3>Using modules locally</h3>
If you want to use Spacelift modules while running Terraform locally, you need to tell Terraform how to authenticate with the Spacelift registry. The easiest way to do this is via the terraform login command. To do this, run the following command and then follow the instructions to login to your Spacelift account via your web browser:

Once authentication has completed successfully, you should see a message similar to the following in your console:

spacelift-module-registry-using-modules-locally

spacelift module registry using modules locally

Now that you are authenticated, you can add a reference to a Spacelift module and run terraform commands locally as normal.
<h3>Using modules in an automated environment</h3>
If you need to use a module in an automated environment outside of Spacelift, the best bet is to use an API key. Please see our <a href="https://docs.spacelift.io/integrations/api#spacelift-api-key-token" target="_blank" rel="noopener">documentation</a> for more information on generating and using API keys.

For more information about the Spacelift module registry, please see the documentation available at <a href="https://docs.spacelift.io/vendors/terraform/module-registry" target="_blank" rel="noopener">docs.spacelift.io</a>.
And see <a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">why DevOps Engineers recommend Spacelift</a>.

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift effectively manages Terraform state, more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and includes many more features.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Spacelift effectively manages Terraform state, more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and includes many more features.

Monitoring Your Spacelift Account via Prometheus

Introduction to Open Policy Agent (OPA) Rego Language

5 Ways to Manage Terraform at Scale - Best Practices

See how to use the Spacelift module registry - a private Terraform module registry that is fully compatible with Terraform.

147-spacelift-module-registry

Spacelift Module Registry

Spacelift Module Registry - What It is and How to Use It

At Spacelift, we&#8217;re excited to announce our new Cloud Integrations section and our update to support account-level AWS integrations. Public Cloud integration is nothing new in Spacelift, but we wanted to simplify the implementation and organization. Cloud Integrations brings feature parity to AWS and Azure integrations across your Spacelift account.

We’ve created a new Cloud Integrations section in the Spacelift console. This section allows you to manage AWS and Azure integrations from a single location. Azure integrations were previously in the settings section and have been moved from there.

spacelift-cloud-integration-aws

spacelift cloud integration - aws

In addition, we’ve added the ability to attach more than one AWS integration to your stack:

spacelift-cloud-integration-more-than-one-aws-integration

spacelift cloud integration - more than one AWS integration

A stack can have at most two AWS integrations attached, with only one read and one write integration (see “<a href="#read-vs-write-attachments">Read vs. Write Attachments</a>” for more info). An integration can also be attached as both read and write for simple use-cases.
Finally, we’ve also altered the external ID format used when assuming roles. More details are available later in this post.

<ul>
<li><a href="https://spacelift.io/blog/spacelift-in-github-marketplace" target="_blank" rel="noopener">Spacelift in the GitHub Marketplace</a></li>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Problems that Spacelift Solves&quot;}" data-sheets-userformat="{&quot;2&quot;:4224,&quot;10&quot;:2,&quot;15&quot;:&quot;Arial&quot;}">Infrastructure Problems that Spacelift Solves</a></li>
</ul>

Don’t worry, your existing AWS integrations will continue to work, and the existing <code>spacelift_aws_role</code> resource can still be used. The only caveat to this is that you will only be able to add new integrations to stacks via the front end using the new Cloud Integrations approach.

Migrating from legacy integrations is easy, with slightly different steps depending on whether you manage your integrations via Terraform or the Spacelift UI. In the simplest case, you just need to create a new integration, delete your old integration, and attach a new one, but you may also need to adjust the Trust Relationships for your <a href="https://spacelift.io/blog/aws-iam-roles" target="_blank" rel="noopener">IAM roles</a> depending on exactly how they are configured.
For more information on migrating, take a look at our <a href="https://docs.spacelift.io/integrations/cloud-providers/aws#migrating-from-legacy-integrations" target="_blank" rel="noopener">documentation</a>.

An integration can be attached to a stack as either read or write. This tells Spacelift which run phases to use each integration during. For simple use-cases, you can attach a single integration as both read and write, but if you want more control, you can create a read-only role and a write role, and attach both to your stack. An example use-case for this is to make sure that a proposed run doesn’t have write permissions to avoid infrastructure changes being made via a pull request.
Read attachments are used during the Initialize and Plan phases of runs. Write attachments are used during the Apply phase of a run, the Perform phase of a task, and the Destroy phase of a stack destructor.

When choosing to assume the role at Spacelift’s end rather than on your private workers, we automatically generate an External ID based on the stack the integration is attached to. Previously we used the following external ID format:

As part of the changes we’ve made, we’ve adjusted our external ID format to the following:

The change means that you can use the “read” or “write” suffix to ensure that a read phase of a run can only assume a read-only role.

We really hope that these changes will be useful and will provide extra flexibility for those users who need it while making simple use-cases easier through reusing integrations. To find out more about Cloud Integrations, review the <a href="https://docs.spacelift.io/integrations/cloud-providers/" target="_blank" rel="noopener">documentation</a>.

Spacelift is a sophisticated SaaS product for Infrastructure as Code that helps DevOps develop and deploy new infrastructures or changes quickly and with confidence.

Spacelift in the GitHub Marketplace

How Spacelift Can Improve Your Infrastructure as Code

We are excited to announce our new Cloud Integrations section and our update to support account-level AWS integrations.

111-spacelift-cloud-integration

Introducing Cloud Integrations

Introducing Spacelift Cloud Integrations

Monitoring your Spacelift account via Prometheus

For a while now, we have received feedback from customers that they would like to be able to connect their Spacelift account to external monitoring systems. Today we are excited to announce the <a href="https://github.com/spacelift-io/prometheus-exporter" target="_blank" rel="nofollow noopener">Prometheus Exporter for Spacelift</a>! We also have plans to add support for &lt;insert monitoring system X here&gt;, so if you’re using another system, we should have you covered soon.

The Prometheus exporter allows you to monitor various metrics about your Spacelift account over time. You can then use tools like Grafana to visualize those changes and Alertmanager to take actions based on account metrics. Several metrics are available, and you can find the complete list of available metrics <a href="https://github.com/spacelift-io/prometheus-exporter#available-metrics" target="_blank" rel="nofollow noopener">here</a>. Below are a few examples of the information the exporter currently provides:
<ul>
<li style="font-weight: 400;" aria-level="1">The number of runs pending and currently executing in both public and private worker pools.</li>
<li style="font-weight: 400;" aria-level="1">The number of workers in a pool.</li>
<li style="font-weight: 400;" aria-level="1">Usage information, including the number of public and private worker minutes used during the current billing period.</li>
</ul>
Once you have that information, it opens a number of possibilities, including visualizing information about your account via Grafana dashboards, alerting on events like a lack of private workers, as well as using that information to autoscale worker pools via the Horizontal Pod Autoscaler.
To give you a taste, here’s an example Grafana dashboard showing some of the information available from the exporter:

grafana-dashboard

Grafana dashboard

The Prometheus exporter is an adaptor between Prometheus and the Spacelift GraphQL API. Whenever Prometheus asks for the current metrics, the exporter makes a GraphQL request and converts it into the metrics format Prometheus expects.
The following diagram gives an overview of this process:

prometheus-exporter-diagram

Prometheus exporter diagram

Ok, great, but how do I use the Prometheus Exporter?
To help with setup and deployment, we are going to guide you through the following:
<ol>
<li style="font-weight: 400;" aria-level="1">Deploy a Prometheus stack to a Kubernetes cluster.</li>
<li style="font-weight: 400;" aria-level="1">Deploy the exporter.</li>
<li style="font-weight: 400;" aria-level="1">Configure  Prometheus to monitor it.</li>
</ol>
The <a href="https://github.com/spacelift-io/prometheus-exporter#quick-start" target="_blank" rel="nofollow noopener">Quick Start</a> section in the exporter repo outlines several options for deploying the exporter. Review the best deployment option that makes sense, depending on your setup and requirements.
If you already have a Prometheus stack setup and have plenty of experience, feel free to skip to the “Installing the Prometheus Exporter” section.
We also assume that you already have a Kubernetes cluster provisioned (hopefully <a href="https://docs.spacelift.io/vendors/kubernetes/" target="_blank" rel="noopener">via Spacelift!</a>) and available to complete the following steps. If not, a local installation like <a href="https://minikube.sigs.k8s.io/docs/start/" target="_blank" rel="nofollow noopener">Minikube</a> will work fine for illustration purposes.
<h3>Installing kube-prometheus-stack</h3>
The first step is to install the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" target="_blank" rel="nofollow noopener">kube-prometheus-stack Helm chart</a>. You can do that with the following commands:

That will install a Prometheus, Grafana, and Alertmanager stack into your cluster&#8217;s namespace called <code>monitoring</code>. Run the <code>kubectl get pods -n monitoring</code> command to see the installed components.

<h3>Getting API Credentials</h3>
The Prometheus exporter authenticates to the Spacelift GraphQL API using an API key. Follow the <a href="https://docs.spacelift.io/integrations/api#spacelift-api-key-greater-than-token" target="_blank" rel="noopener">guide</a><a href="https://docs.spacelift.io/integrations/api#spacelift-api-key-greater-than-token" target="_blank" rel="noopener"> to create a new API key</a> required by the explorer. After you create your key, please note the API Key ID and API Key Secret &#8211; you’ll need both when configuring the exporter.
<h3>Installing the Prometheus Exporter</h3>
The exporter is available via a Docker image published to the <code>public.ecr.aws/spacelift/promex</code> container registry. To deploy the exporter to Kubernetes, we need to create the following resources:
<ul>
<li style="font-weight: 400;" aria-level="1">A Deployment &#8211; to run the exporter container.</li>
<li style="font-weight: 400;" aria-level="1">A Service &#8211; to allow Prometheus to scrape the exporter.</li>
<li style="font-weight: 400;" aria-level="1">A ServiceMonitor &#8211; to let Prometheus know that it needs to scrape the exporter.</li>
</ul>
The following is an example Deployment definition for running the exporter. Make sure to replace the <code>&lt;account name&gt;</code>, <code>&lt;API Key&gt;</code> and <code>&lt;API Secret&gt;</code> placeholders with the correct values:

Note: The example above defines the API key and secret as normal environment variables. We would recommend that you use <a href="https://spacelift.io/blog/kubernetes-secrets" target="_blank" rel="noopener">Kubernetes Secrets</a> for anything other than testing purposes.
Next, create a Service to expose the exporter:

And finally create your ServiceMonitor:

The ServiceMonitor definition above tells Kubernetes to scrape any services that contain an <code>app</code> label. The granularity of the metrics can be increased or decreased depending on requirements and organizational standards. The above example is configured for 15-second intervals.
<h3>Viewing your Metrics</h3>
Once you have your Prometheus stack up and running and have deployed the Spacelift exporter, you can use port-forwarding to access each component. First, let’s port-forward the exporter to port 8080 locally using the following command:

Assuming all is well, you should be able to see the raw metrics output by accessing <a href="http://localhost:8080/metrics" target="_blank" rel="nofollow noopener">http://localhost:8080/metrics</a>:

localhost8080-metrics

localhost8080 metrics

You can also port-forward to your Grafana instance to view the metrics in Grafana:

You can then quickly discover the available metrics via the Grafana Explore view:

grafana-explore-view

Grafana Explore view

<ul>
<li><a href="https://spacelift.io/blog/devops-best-practices" target="_blank" rel="noopener">16 DevOps Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Automate Terraform Deployments and Infrastructure Provisioning&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
</ul>

One of the things that I think is amazing about the Prometheus stack is the ability to use PromQL queries not just for monitoring but for defining alerts. For example, if we want to trigger an alert whenever our worker pool has no available workers for a specific time period, we can use a query like this:

Similarly, if we want to alert when the number of queued runs for a pool gets too high, we can use a query like this:

NOTE: This section is intended to outline some of the possibilities that exist when using the Prometheus exporter and should NOT be used as a guide for a production-ready autoscaling solution. For example, it does not consider properly draining workers before scaling them down to avoid in-progress runs being terminated.
Since we have metrics related to queued runs, we can use them to autoscale private workers. We can use the <code>spacelift_worker_pool_runs_pending</code> metric, which tells us how many runs for a given worker pool are waiting to be scheduled, to detect when we need to add more workers to our pool. Similarly, we can use the <code>spacelift_worker_pool_workers</code> and <code>spacelift_worker_pool_workers_busy</code> metrics to decide when to scale down.
We need to take both sets of metrics into account to avoid scaling down just because there are no queued runs. In this situation, we might still have runs in progress that need workers.
For this to work, we need the following components available:
<ol>
<li style="font-weight: 400;" aria-level="1">A working Prometheus stack with the Spacelift Prometheus Exporter running.</li>
<li style="font-weight: 400;" aria-level="1">A Spacelift <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">worker pool</a>.</li>
<li style="font-weight: 400;" aria-level="1">A <a href="https://github.com/kubernetes-sigs/prometheus-adapter" target="_blank" rel="nofollow noopener">prometheus-adapter</a> installation.</li>
<li style="font-weight: 400;" aria-level="1">A Horizontal Pod Autoscaler resource to tell Kubernetes how to scale our worker pool.</li>
</ol>
For the sake of simplicity, I’m going to assume that you already have steps 1 and 2 covered, and let&#8217;s assume that you’ve deployed the Spacelift worker pool chart with the default settings. I’ll also assume that you have a standard installation of the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" target="_blank" rel="nofollow noopener">kube-prometheus-stack</a> chart.
The first thing we need to do is set up a Prometheus-adapter installation. The Prometheus-adapter works like a bridge between the Kubernetes metrics API and your Prometheus installation. It allows you to use Prometheus metrics to make autoscaling decisions within your cluster. We can visualize it something like this:

kubernetes-cluster-2

Kubernetes cluster

The Prometheus-adapter uses a configuration format to map between Prometheus queries and Kubernetes metrics. In our case, we can use something like the following to generate the two metrics we need:

After deploying the adapter, you can query your metrics via the Kubernetes API, using the following commands (assuming you’ve deployed everything to a namespace called <code>monitoring</code>):

Finally, we can create a Horizontal Pod Autoscaler definition to scale our worker pool Deployment based on those metrics automatically:

Notice the metrics are grouped by the <code>worker_pool_id</code>; we can use this to target an individual worker pool when creating our HPA!
That’s it &#8211; bask in your autoscaling glory:

public-workers

worker pool

We are excited to learn how our customers use the exporter and the problems they solve with it! Feel free to provide feedback and comments via Issues in the <a href="https://github.com/spacelift-io/prometheus-exporter" target="_blank" rel="nofollow noopener">Prometheus Exporter Github repository</a>.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities s for infrastructure management.

Alternative to Atlantis – Spacelift

Spacelift Worker Pools on Kubernetes with Spacelift

Working with Pull Requests on Spacelift

We are excited to announce the Prometheus Exporter for Spacelift! See how it allows you to monitor various metrics about your Spacelift account over time.

98-monitoring_spacellift_with_prometheus

Running Spacelift CICD Workers in Kubernetes Using DinD

Check out the Spacelift Engineering blog to discover internal technical architecture posts written by Spacelift's very own engineers. 

Engineering | Spacelift Blog

Engineering

Want to level up your Kubernetes knowledge? Explore our tutorials and step-by-step guides written by Kubernetes experts. Learn Kubernetes best practices and solutions to the most difficult problems.

Kubernetes Guides and Tutorials | Spacelift Blog

Kubernetes

For a while now at Spacelift we’ve been hearing from users that they’d like to be able to run <a href="https://docs.spacelift.io/concepts/worker-pools" target="_blank" rel="noopener">Spacelift worker pools</a> in their Kubernetes clusters. As a first step towards that we decided to investigate whether we could run workers in Kubernetes using a Docker-in-Docker sidecar container. The rest of this post gives a rough overview of the architecture of Spacelift workers, and also explains how the sidecar container strategy works.
I’d just like to say thanks to <a href="https://medium.com/@v.m_hs" target="_blank" rel="nofollow noopener">Vadym Martsynovskyy</a> for his great article about running <a href="https://medium.com/hootsuite-engineering/building-docker-images-inside-kubernetes-42c6af855f25" target="_blank" rel="nofollow noopener">Jenkins agents in Kubernetes</a>. In that article he outlines the general approach taken here, and also describes some of the pros and cons of using Docker-in-Docker. It’s definitely well worth a read.

Before going into more detail, it’s worth explaining what a Spacelift worker is, and giving a quick overview of our architecture. At Spacelift, we provide a hybrid-SaaS model like many other CI/CD systems. What this means in practice is that the control plane that handles scheduling runs is managed by us, but the execution of runs happens in a separate process called a worker. We provide a public worker pool, but also allow you to self-host your own workers, providing complete control over your infrastructure.
Perhaps confusingly, if you follow the instructions to create a private worker pool, you’ll notice that it involves downloading something called the “launcher”. The launcher is responsible for communicating with the Spacelift mothership (control plane), and creating and destroying worker containers using Docker for each Spacelift run:

image2-5

Spacelift mothership (control plane)

As you can see, the launcher process needs access to a Docker daemon in order to create a container for each run.

<ul>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
<li><a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Deploy your Infrastructure in CI/CD using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Deploy your Infrastructure in CI/CD using Terraform</a></li>
<li><a href="https://spacelift.io/blog/devops-best-practices" target="_blank" rel="noopener">16 DevOps Best Practices to Follow</a></li>
</ul>

We found that the launcher architecture maps very nicely to Kubernetes using a sidecar container to run the Docker daemon. For each worker in the pool, we need to create a Kubernetes Pod with two containers: one for the launcher; and another for Docker. The following shows a stripped down Deployment definition for creating a worker pool with 4 workers:

The launcher communicates with the Docker-in-Docker sidecar container via TCP, which is configured via the <code>DOCKER_HOST</code> environment variable for the launcher container. This works because the containers in a Kubernetes Pod share the same <a href="https://kubernetes.io/docs/concepts/workloads/pods/#pod-networking" target="_blank" rel="nofollow noopener">IP address and port space, and can communicate with each other via localhost</a>.
In addition, a shared volume called <code>launcher-storage</code> is mounted into each container. This is used by the launcher to store the workspaces for runs, along with other things like cached tool binaries (for example Terraform). The Docker sidecar needs access to the run workspaces in order to mount them into the worker containers, and it also uses that volume to store its image cache.
The last thing worth pointing out is that the <code>dind</code> container sets <code>securityContext.privileged</code> to <code>true</code>. This is required for Docker-in-Docker to function correctly.

The Spacelift launcher is distributed as a statically linked binary, making it simple to build an image. As you can see, the <a href="https://spacelift.io/blog/dockerfile" target="_blank" rel="noopener">Dockerfile</a> simply copies the launcher into an Alpine container, and then sets the startup command:

This image is rebuilt and published to our public ECR repository any time the launcher binary is updated.

We provide <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform modules</a> to make it really easy for users to deploy worker pools to AWS, Azure and GCP, and we wanted to provide a similar experience for Kubernetes. To achieve this, we created a Helm chart that makes it simple to deploy workers to Kubernetes clusters: <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">https://github.com/spacelift-io/spacelift-workerpool-k8s</a>.
Assuming you’ve already configured your worker pool in Spacelift and have access to the credentials for the pool, deploying this chart to your cluster simply requires two steps.
First, add the Spacelift Helm chart repository and update your local chart cache:

Next, install the chart:

Replace <code>&lt;worker-pool-token&gt;</code> and <code>&lt;worker-pool-private-key&gt;</code> with your own credentials, and make sure to base64-encode the private key.
If all goes well, you should be able to view the pods for your worker pool using <code>kubectl get pods</code>:

You should also be able to view the workers in your pool in Spacelift:

image1-5

Bitbucket DC Local Pool

When implementing in a production environment, you may want to use an alternative approach to providing credentials, and may want to configure a custom storage volume for the worker Pods. For more information about configuring the chart, check out the README in the <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">chart GitHub repo</a>.

If you’re interested in running <a href="https://spacelift.io/blog/spacelift-worker-pools-on-kubernetes" target="_blank" rel="noopener">Spacelift workers in Kubernetes</a>, we’d welcome any feedback about the approach, contributions to the Helm chart, and also any issues you encounter so that we can make improvements. Also, if you aren’t already using Spacelift but are interested in trying it, you can signup and <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">start your free evaluation of Spacelift</a>.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Kubernetes Sidecar Container - Best Practices and Examples

Managing Azure Resources using Spacelift

This article is an overview of the architecture of Spacelift workers with an explanation of how the sidecar container strategy works.

49-running_spacelift_cicd_workers_in_kubernetes_using_dind-3

Running Spacelift CI/CD workers in Kubernetes using DinD

Building a Secure CI/CD Integration with Azure

Adam Connelly

Spacelift Module Registry – What It is and How to Use It

Get the newsletter

Recent posts

Introducing Spacelift Cloud Integrations

Monitoring Your Spacelift Account via Prometheus

Running Spacelift CI/CD workers in Kubernetes using DinD

Building a Secure CI/CD Integration with Azure

Introducing the Spacelift Azure Integration!

How to Manage Active Directory Objects with Azure AD Provider for Terraform

Open Policy Agent: What Is OPA and How It Works (Examples)

Managing Azure Resources using Spacelift