Spacelift Private Workers vs. Public Workers

When you are building a workflow, you need computing power behind the scenes to handle the different tasks you require. This is where workers or runners come into play, as they will be the ones that handle all the tasks your workflows go through.

In Spacelift, a worker is an entity that processes a single run at a time, meaning that the number of workers you can use will be equal to the number of the maximum concurrency that you can obtain.

In SaaS, public workers are managed by your SaaS provider, while private workers are managed by the customer. By default, Spacelift uses a public worker pool, which is hosted and operated by Spacelift.

The main benefits of using public workers are:

• Zero maintenance – everything will be handled by Spacelift (patching, updates, etc.)
• No setup is required; you can use them out of the box 
• Predictable performance
• Built-in security measures

This makes the process very convenient, but to adhere to all of your security requirements, you can leverage private workers that are hosted in your environments.

This doesn’t mean, however, that Spacelift’s public worker pool is not secure, but private workers are meant to offer you the flexibility you need.

Private workers offer you complete control over where your workflow runs, and based on how you configure them, they can give you direct access to private networks, enhanced security and isolation, compliance with strict policies, support for the air-gapped environment, and more.

From a network perspective, private workers excel at providing better isolation and security. You can implement precise network configurations, establish custom firewall rules, and define specific security groups and network access control lists that align with your organization’s security requirements. 

They are key for environments where compliance is a top priority. For industries such as finance, health, and government, where operating in air-gapped environments is a must, taking advantage of private workers ensures there are no external threats and vulnerabilities are limited.

If you are thinking about performance, private workers are also a great option because you can allocate how much computing power and memory you want. In this way, you ensure that you can handle intensive tasks better than in a shared environment. At the same time, with private workers, you can ensure that high-priority operations can execute reliably without any performance bottlenecks.

Spacelift allows you to set priorities inside your running queue, ensuring that critical tasks receive immediate attention while less urgent ones are queued. 

With private workers, you can enable better debugging and troubleshooting capabilities because you have full access to the worker environment. This allows you to pinpoint issues, monitor different metrics, and optimize configurations accordingly. 

One of the best aspects of leveraging private workers is related to scalability. You can dynamically adjust the number of workers on demand, spinning up additional workers to handle your high-load periods and spinning down the workers when the load is low. By leveraging this elasticity, you ensure that you’re not overprovisioning while maintaining the ability to handle peak workloads.

Private workers will require internal maintenance, meaning that patching, updates and all the security responsibilities fall on your team. This can also introduce operational overhead, but the good news is that their creation can be easily automated, simplifying at the same time, all the maintenance operations.

To take advantage of private workers, you will need to choose what kind of workers you want to use, between Docker-based workers and Kubernetes workers. If you want to see a walkthrough of how to set up Docker-based workers, check out this example.

Note: Private workers are only available on the Starter+ plan and above.

To see how Spacelift workers compare to Terraform Cloud Agents, take a look here.

All Spacelift features can leverage private workers, and the majority of them also work on public workers. Let’s now take a look at some of the features that only work on private workers.

Schedules

Spacelift supports several scheduling options:

1. Drift detection and remediation
2. Arbitrary tasks
3. Stack deletion 
4. Runs

1. Drift detection and remediation

Infrastructure drift refers to changes made to your infrastructure outside of your code base, which usually result in a difference between the desired state of your infrastructure and its actual state.

Check this video to learn more about our drift detection feature. Alternatively, if you want to learn more about the top causes of drift and the hidden impacts, check the links to the respective articles.

2. Arbitrary tasks

Spacelift gives you the flexibility to run arbitrary tasks against your stacks. These tasks can also be scheduled, and you can select between having them run recurrently or on a specific time frame.

3. Stack deletion

Having the option to delete your stacks on a schedule can be really helpful, especially in cases in which you leverage ephemeral environments. When you create a schedule for stack deletion, you can select whether or not to keep the underlying resources.

4. Runs

You can also schedule runs, either recurrently or on a schedule, and you even have the option to attach a custom runtime configuration:

VCS Agent Pools

If you need to host your VCS system in a way that’s only accessible internally, this is where our VCS Agent Pools feature comes into play.

This feature is supported only when you use private workers, and these workers must be able to reach your VCS instance.

Approval policy

To ensure your guardrails are in place, you can require certain stacks to only run on private workers.

This is where approval policies shine. In this way, if someone makes a mistake related to a stack and tries to use a public worker pool for the workflow, the run will be automatically denied.

Blueprints

Blueprints let you self-serve infrastructure, ensuring that your templates are pre-approved by your DevOps and Platform Teams before they can be used by the entire organization.

When configuring a blueprint, you have the option to force all the stacks that will be created from it to leverage a private worker pool by adding the worker_pool field inside the yaml template:

Private workers offer many benefits, such as complete control over where your workflow executes, give you better security isolation, and superior resource management capabilities. They are very powerful in regulated industries, offering great support for air-gapped environments.

Their flexibility extends to computational and scalability features, and they also offer better debugging and troubleshooting capabilities.

Spacelift offers a one-stop shop for provisioning, configuring, and governing your infrastructure. By pairing it with private workers you can get all the benefits mentioned above.

If you want to learn more about Spacelift, book a demo with one of our engineers.