How to Set Up a Spacelift Worker Pool on EC2

Initiating the configuration of a worker pool within Spacelift might initially appear to be a complex task. However, it is remarkably straightforward and accessible. 

This post is designed to serve as a comprehensive guide that streamlines and expedites the setup process. It aims to provide clear, step-by-step instructions, ensuring that you can approach this task with confidence.

What we will cover:

1. Prerequisites
2. Generating the Worker pool
3. Running the worker automation module

Before creating the worker pool we need to:

• Generate a worker private key
• Generate a Spacelift API key
• Set up a cloud Integration to AWS in your Spacelift account
• Get Spacelift’s mothership IPS
• Set up a network

1. Generate a worker private key

To generate a worker private key, simply go to your terminal and run the following command:

This command will generate two files: spacelift.key and spacelift.csr. Remember to store them securely, as they will be essential when you are creating the worker pool and initiating workers within it.

2. Generate a Spacelift API key

In your account, click in the bottom left corner on your username and select Organization settings.

Then, under access, select API keys and click on Create API key:

Add the required details for the API key, such as the name, space, and role, and then click on Create. Ensure you save the generated file in a safe place, as you will need it to use it when we run an automation.

The API secret will be in this file and we will need it later:

3. Create an AWS Cloud Integration

The process for creating an AWS Cloud Integration is easy to follow and can be found here. 

If you don’t want to use dynamic credentials, you can pass static ones as environment variables when we create the stack that does the automation.

4. Get Spacelift’s mothership IPS

We will need the Spacelift’s mothership IPS to whitelist connections to them in the security groups our automation will use.

There is an automation available that you can easily run to get the IPS:

For that, go to your Spacelift account and create a stack, add a name, and select a space for it, in the VCS screen select Raw Git and provide the following URL: https://github.com/saturnhead/mothership_ips/ 

Select either OpenTofu or Terraform and click on Create & Continue. In the Define behavior tab, check the Administrative option:

Now skip to the summary, confirm, and trigger a run. After the plan finishes successfully, you will see the ips. 

5. Network setup

We advise you to use at least two public subnets and two security groups. The subnets should have a route to an internet gateway with 0.0.0.0/0 destination.

The security groups should permit the following traffic:

• Ingress: 443 for all Spacelift Mothership IPs
• Egress: 443 for 0.0.0.0/0

In your Spacelift account, go to Worker Pools and click on Create Worker Pool.

Add a name to your worker pool, upload the spacelift.csr file, and select a space in which you want to use your worker pool. You have the option to add a description and label. After you click on Create, a worker pool config will download automatically.

The module that we will use can be found here. Based on this module, we’ve created a repository that can be leveraged for automating the worker creation. You can fork this repository and use it as it is, or you can customize it to your liking.

Go to your Spacelift account, select Stacks, and then click on Create Stack.

Add a name, select a space for your stack, and click on Next. In the next screen, select your VCS provider, the forked repository, and the branch. If you have made changes and modified the path to the OpenTofu/Terraform scripts, ensure you set up the project root accordingly.

Next, select the vendor (OpenTofu or Terraform will both work), and click on Create & continue:

After this, you can go directly to the “Attach cloud screen” and attach the cloud integration you have created:

Click on Attach, Skip to Summary, and Confirm. You will be redirected to the newly created stack.

Before running the stack, head to the environment and fill in the following env vars:

• TF_VAR_spacelift_api_key_endpoint – this will be your Spacelift account (e.g https://saturnhead.app.spacelift.io)
• TF_VAR_spacelift_api_key_id – the API key id, can be easily taken from your Organization Settings – API keys and by clicking on the copy button (ensure you click on it, as they Key ID is longer that what is shown in the app for security reasons)

TF_VAR_spacelift_api_key_secret – the secret of the API key id. This can be taken from the file that downloaded automatically when you created the API key:

• TF_VAR_worker_pool_private_key – the private key from the worker pool base encoded, which you can get from running: cat spacelift.key | base64 -b 0 | pbcopy
• TF_VAR_worker_pool_id –  the id of the worker pool that you can copy from the worker pool page

• TF_VAR_worker_pool_config – the content of the config file that was automatically downloaded when we created the worker pool. The file should be in this format – worker-pool-wp_id.config
• TF_VAR_worker_pool_security_groups – the security groups used by the worker pool (these security groups should have traffic whitelisted from the Spacelift mothership). Provide at least two.
• TF_VAR_worker_pool_subnets – the subnets used by the worker pool; provide at least two.

After you’ve populated all of these env vars, you are ready to run the code:

Go to tracked runs and click on Trigger. The plan will show 15 resources to add:

Next, confirm the run and wait for it to apply. In the end, you should see that everything was created successfully:

Now, if you go back to your Worker Pools, you should see the worker pool has one worker:

Based on demand, the worker pool will increase/decrease the number of workers and if you haven’t changed the OpenTofu code, you will have a minimum of one and a maximum of five.

In this post, we’ve seen how to set up private workers in Spacelift using EC2. The overall process is simple, but you need to be careful to pass the correct environment variables in the stack that runs the automation.

If you want to learn more about Spacelift, you can book a demo with one of our engineers, and if you don’t have an account, you can create one here.