Sean is the Head of Developer Relations at Spacelift. Sean believes every IT organization can and should develop and practice the DevOps way of life!

All posts by Sean O'Dell - Infrastructure as Code Expert

Sean O'Dell

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Product

We are pleased to announce the availability of the Spacelift as a GitHub App in the GitHub Marketplace. The <a href="https://github.com/marketplace" target="_blank" rel="nofollow noopener">GitHub Marketplace</a> is the official location for all apps and tools that integrate with GitHub.

spacelift-in-the-github-marketplace

the GitHub Marketplace

Spacelift is a sophisticated and compliant infrastructure delivery platform for Terraform (and Terragrunt), Pulumi, CloudFormation, Ansible, and Kubernetes, with a strong dependency on git and associated version control systems. The Spacelift platform provides deep integration with <a href="https://docs.spacelift.io/integrations/source-control/" target="_blank" rel="noopener">VCS Providers</a> and Public <a href="https://docs.spacelift.io/integrations/cloud-providers/" target="_blank" rel="noopener">Cloud Providers</a>, including Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). 
 
GitHub support in Spacelift is nothing new, and GitHub was a supported VCS Provider from the beginning, but we wanted to make it a bit more official. To learn more about our integration with Github, review the <a href="https://docs.spacelift.io/integrations/source-control/github" target="_blank" rel="noopener">documentation</a>. Historically, there was a step to <a href="https://docs.spacelift.io/integrations/source-control/github" target="_blank" rel="noopener">Create a Github App</a>, which allows for a verified and more straightforward process. Below are a few of the examples where Spacelift + GitHub are better together.

The stack or module creation and editing forms will show a dropdown from which you can choose the VCS provider to use.

create-new-stack-github-spacelift

create new stack - Github Spacelift

<ul>
<li><a href="https://spacelift.io/blog/spacelift-worker-pools-on-kubernetes" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift Worker Pools on Kubernetes with Spacelift&quot;}" data-sheets-userformat="{&quot;2&quot;:4224,&quot;10&quot;:2,&quot;15&quot;:&quot;Arial&quot;}">Spacelift Worker Pools on Kubernetes with Spacelift</a></li>
<li><a href="https://spacelift.io/blog/spacelift-module-registry" target="_blank" rel="noopener">How to Use Spacelift Module Registry</a></li>
<li><a href="https://spacelift.io/blog/spacelift-spaces-in-action" target="_blank" rel="noopener">Spacelift Spaces in Action</a></li>
</ul>

Suppose you use GitHub as your identity provider (the default) upon login. In that case, Spacelift uses GitHub API to determine organization and team membership levels. It persists in the session token, which is valid for one hour. Based on that, you can set up <a href="https://docs.spacelift.io/concepts/policy/login-policy" target="_blank" rel="noopener">login policies</a> to determine who can log in to your Spacelift account and <a href="https://docs.spacelift.io/concepts/policy/stack-access-policy" target="_blank" rel="noopener">stack access policies</a> that can grant an appropriate level of access to individual <a href="https://docs.spacelift.io/concepts/stack/" target="_blank" rel="noopener">Stacks</a>.

A commit status notification is triggered for <a href="https://docs.spacelift.io/concepts/run/proposed" target="_blank" rel="noopener">proposed runs</a> to provide feedback on the proposed changes to your stack. The screenshot below is an example of the preview command (e.g., <code>terraform plan</code> for Terraform).

commit-status-notifications

Commit Status Notifications

With the integration, you can use commit statuses to protect your branches tracked by Spacelift stacks by ensuring that proposed runs succeed before merging their Pull Requests.

using-spacelift-checks-to-protect-branches

Using Spacelift checks to protect branches

Spacelift also has a PRs tab that lists all the active <a href="https://docs.spacelift.io/integrations/source-control/github#pull-requests" target="_blank" rel="noopener">Pull Requests</a> against your tracked branch. Each of the entries shows the current status of the change as determined by Spacelift and a link to the most recent Run responsible for determining that status.

pull-requests

Pull Requests

If you want to see the power of Spacelift and GitHub in complete form, check out the <a href="https://docs.spacelift.io/integrations/source-control/github" target="_blank" rel="noopener">documentation</a> or sign up for a <a href="https://spacelift.io/free-trial?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bspacelift_in_github_marketplace%7D" target="_blank" rel="noopener">free trial</a> today.
Spacelift is committed to building the best integrations with VCS Providers, and GitHub is a top priority for our customers and team. If you have any questions or want some additional guidance, feel free to reach out to us on <a href="https://twitter.com/spaceliftio" target="_blank" rel="nofollow noopener">Twitter</a>.

Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure.  It helps overcome common state management issues and adds several must-have features for infrastructure management.

Which Cloud Deployment Model Is Right for Your Business?

Alternative to Atlantis – Spacelift

5 Ways to Manage Terraform at Scale - Best Practices

We are announcing the availability of the Spacelift as a GitHub App in the GitHub Marketplace. See how Spacelift + GitHub work better together.

95-spacelift-in-the-github-marketplace

Spacelift in the GitHub Marketplace

Spacelift Worker Pools on Kubernetes with Spacelift

You are probably a bit confused by the description of the subtitle; stay with me! If you have not seen the latest news, Spacelift supports Kubernetes! Check out the announcement blog and documentation.
This blog aims to demonstrate the power of Spacelift by providing a view of how Spacelift uses Spacelift to provision Kubernetes clusters, a private worker pool, and Kubernetes deployments all within the same organization and structure.
To summarize, we are looking at the following objectives for Spacelift:
<ul>
<li style="font-weight: 400;">Provision Kubernetes clusters
<ul>
<li style="font-weight: 400;">Application</li>
<li style="font-weight: 400;">Worker (pool)</li>
</ul>
</li>
<li style="font-weight: 400;">Setup Private Worker Pool</li>
<li style="font-weight: 400;">Provision Spacelift Worker Pool</li>
<li style="font-weight: 400;">Deploy Test Kubernetes Deployment</li>
</ul>
All of this via the most influential management platform for IaC!

image5-13

Kubernetes clusters

Spacelift supports externally deployed and provision Kubernetes clusters, but what is the fun in that! If you bring your own Kubernetes clusters, you can skip this step.
As mentioned previously, we need two Kubernetes clusters for this scenario. One will be utilized as the Worker pool cluster and the other for a test deployment of Nginx.
Due to the length of this blog, I opted to separate this specific step to keep things brief and clean. You can follow the steps in the blog titled <a href="https://spacelift.io/blog/provision-kubernetes-cluster-terraform-spacelift" target="_blank" rel="noopener">Provision a Kubernetes Cluster with Spacelift and Terraform</a>.

<h3>Certificate Signing Request</h3>
To ensure that we have no access to your private key, you will need to generate it on your end and use it to create a certificate signing request to give to Spacelift. We will create a certificate for you so that workers can use it to authenticate with the Spacelift backend. The following command will generate the key and CSR:

You should now store the spacelift.key file in a secure place. You’ll need it later when launching workers in your worker pool.
<h3>Private Key Encode</h3>
You need to encode the entire private key using base-64, making it a single line of text. The most straightforward approach is to just run something like <code>cat spacelift.key | base64 -w 0</code> in your command line. For Mac users, the command will be something like <code>cat spacelift.key | base64 -b 0</code>.
Save the response in a safe place. You will need it shortly.
<h3>Add Worker Pool</h3>
From the Spacelift UI, navigate to the Worker Pool page by clicking the icon on the navigation bar. Choose the add worker pool button.

image3-17

Add worker pool

<ul>
<li><a href="https://spacelift.io/blog/spacelift-module-registry" target="_blank" rel="noopener">How to Use Spacelift Module Registry</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Problems that Spacelift Solves&quot;}" data-sheets-userformat="{&quot;2&quot;:4224,&quot;10&quot;:2,&quot;15&quot;:&quot;Arial&quot;}">Infrastructure Problems that Spacelift Solves</a></li>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
</ul>

<h3>Create new worker pool</h3>
<ul>
<li style="font-weight: 400;">Name: Name of your worker pool</li>
<li style="font-weight: 400;">Your certificate: Upload the generated CSR that was created a few steps prior</li>
<li style="font-weight: 400;">Labels: Optionally add labels</li>
<li style="font-weight: 400;">Description: Optionally add a description</li>
</ul>
Name and Certificate are required.

image9-5

Create new worker pool

Now you can submit the spacelift.csr file in the worker pool creation form. In response, you will receive a Spacelift token and be taken to the newly created Worker Pool page.

image14

Spacelift token

Now that we have the Spacelift token, we can proceed with deploying the Spacelift Worker pool.

Support for worker pools on Kubernetes is nothing new, and you can check out the <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">repository</a> for additional details.
<h3>Repository Creation</h3>
Start by creating a new repository for your Helm stack. This repository only needs to contain a single item &#8211; a kustomization.yaml.

The kustomization file is used to tell kubectl where to find the file containing the output of the helm template command and prevents kubectl from attempting to apply every YAML file in your repository.
<h3>Create Worker Cluster Stack</h3>
In Spacelift, click the Add Stack button to create a Stack.
<h4>Integrate VCS</h4>
<ul>
<li style="font-weight: 400;">Select the repository you created in the initial step, as seen in the picture.</li>
</ul>

image8-8

Integrate VCS

<h4>Configure Backend</h4>
For this example, select the Kubernetes backend.

image10-5

Configure Backend

Type in spacelift-worker into the Namespace field. This namespace will be utilized for all deployments associated with this Stack.
<h4>Define Behavior</h4>
Select the arrow next to Show Advanced Options to expose the advanced configuration options. 
The following options should be reviewed and validated.
<ul>
<li style="font-weight: 400;">Worker Pool: Depending on the maturity of your environment. By default, the none &#8211; use shared option will work.</li>
<li style="font-weight: 400;">Runner Image: Spacelift does not provide an image that includes helm, which is required in this step. You must provide your own image.</li>
<li style="font-weight: 400;">Customize workflow: During the initialization phase, you must specify the following commands:
<ul>
<li style="font-weight: 400;"><code>aws eks update-kubeconfig --region $region-name --name $cluster_name</code></li>
<li style="font-weight: 400;"><code>helm repo add spacelift https://downloads.spacelift.io/helm</code></li>
<li style="font-weight: 400;"><code>helm repo update</code></li>
<li style="font-weight: 400;"><code>helm template spacelift-worker-pool spacelift/spacelift-worker --set "replicaCount=$SPACELIFT_WORKER_REPLICAS" --set "credentials.token=$SPACELIFT_TOKEN" --set "credentials.privateKey=$SPACELIFT_WORKER_POOL_PRIVATE_KEY" &gt; spacelift-worker-pool.yaml</code></li>
</ul>
</li>
</ul>
We will provide the necessary environment variables at a later stage.

image12-5

Define Behavior

<h4>Name Stack</h4>
Provide the name of your Stack. Labels and Description are not required but recommended.

image7-8

Name Stack

<h3>Environment Variables</h3>
To simplify the deployment and improve security, I have chosen to set up variables within the Stack. Each variable is used at different times throughout the deployment process.
<ul>
<li style="font-weight: 400;">SPACELIFT_WORKER_REPLICAS
<ul>
<li style="font-weight: 400;">Defines the number of replicas deployed for use in the worker pool</li>
</ul>
</li>
<li style="font-weight: 400;">SPACELIFT_TOKEN
<ul>
<li style="font-weight: 400;">This specific variable was provided earlier in the CSR phase and should be associated accordingly.</li>
</ul>
</li>
<li style="font-weight: 400;">SPACELIFT_WORKER_POOL_PRIVATE_KEY
<ul>
<li style="font-weight: 400;">This private key was created earlier during the CSR phase and should be added appropriately.</li>
</ul>
</li>
</ul>
<h3>Cloud Integration</h3>
Navigate to the Settings, Integrations page and select the dropdown arrow to access the following selection screen:

image13-4

image13

The following links will help you set up the necessary integration with your Cloud Provider of choice.
<ul>
<li style="font-weight: 400;"><a href="https://docs.spacelift.io/integrations/cloud-providers/aws" target="_blank" rel="noopener">AWS</a></li>
<li style="font-weight: 400;"><a href="https://docs.spacelift.io/integrations/cloud-providers/azure" target="_blank" rel="noopener">Azure</a></li>
<li style="font-weight: 400;"><a href="https://docs.spacelift.io/integrations/cloud-providers/gcp" target="_blank" rel="noopener">GCP</a></li>
</ul>
To simplify things, I decided to utilize an AWS role that already exists with the necessary permissions to provide the resources needed to be associated with this Terraform example.

image11-6

<h3>Trigger a Run</h3>
To Trigger a Run, select Trigger on the right side of the Stacks view.

image6-8

Trigger a Run

<h3>Trigger Run Status</h3>
<h4>Planning</h4>
After you manually trigger the Run in the Stack view, Spacelift will deploy a runner image, initialize the Cloud Provider, and run a Terraform Plan based upon the contents of the terraform code.
After a successful planning phase, you can check the log to see the planned changes.

image4-16

Trigger Run Status

To confirm the Triggered run, click the CONFIRM button at the top of the screen.
<h4>Finished Deployment</h4>
The following screen highlights the Finished Run and output from a successful deployment to your Kubernetes cluster.

image2-19

Finished Deployment

Get started on your journey by signing up for a <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">Free Trial</a> and taking it for a spin around the world! Sign-up is as easy as authenticating with the provider of your choice.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities s for infrastructure management.

Running Spacelift CI/CD workers in Kubernetes using DinD

Achieving the Atlantis Pull Request Automation with Spacelift

How Spacelift Can Improve Your Infrastructure as Code

See how Spacelift uses Spacelift to provision Kubernetes clusters, a private worker pool, and Kubernetes deployments within the same organization.

72-spacelift-worker-pools-on-kubernetes-with-spacelift

Provision a Kubernetes Cluster with Spacelift and Terraform

Want to level up your Kubernetes knowledge? Explore our tutorials and step-by-step guides written by Kubernetes experts. Learn Kubernetes best practices and solutions to the most difficult problems.

Kubernetes Guides and Tutorials | Spacelift Blog

Kubernetes

By now, you have probably heard about the initial availability of Kubernetes integration in Spacelift. You can <a href="https://spacelift.io/blog/spacelift-kubernetes-integration-beta" target="_blank" rel="noopener">read more about the announcement</a> and review the documentation.
Spacelift, like many of our customers, utilizes Kubernetes in a variety of ways. Before the integration, we operated Kubernetes or Public worker pools, testing and hosting the Spacelift platform. Now that the Kubernetes integration is initially available, we thought it would be good to provide a simple way for you to provision a Kubernetes cluster on AWS utilizing Terraform with Spacelift.
This blog will guide you through the process of deploying a Terraform based deployment of an EKS cluster with Spacelift. Below is an example of the cloned Terraform repository.

image2-18

devrel-k8s-worker-cluster

Start by cloning the repository found on <a href="https://github.com/hashicorp/learn-terraform-provision-eks-cluster" target="_blank" rel="nofollow noopener">GitHub</a> and commit to your respective VCS provider. We made a few changes, but that was only to help with the naming and possible confusion in our dev environment. Otherwise, utilizing the repository as-is will work.

In Spacelift, click the Add Stack button to create a Stack.
<h3>Integrate VCS</h3>
<ul>
<li style="font-weight: 400;" aria-level="1">Select the repository you created in the initial step, as seen in the picture.</li>
</ul>

image11-5

<h3>Configure Backend</h3>
For this example, I will keep the default selections.

image7-7

<h3>Define Behavior</h3>
If you need to modify the configuration, select the arrow next to Show Advanced Options. You can find out more about these options by reading the <a href="https://docs.spacelift.io/concepts/stack/creating-a-stack#define-behavior" target="_blank" rel="noopener">documentation</a>.

image4-15

<h3>Name Stack</h3>
Name the stack and provide any labels or descriptions.

image8-7

<ul>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
<li><a href="https://spacelift.io/blog/kubernetes-best-practices" target="_blank" rel="noopener">15 Kubernetes Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;10 Ways to Improve Your Infrastructure as Code&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">10 Ways to Improve Your Infrastructure as Code</a></li>
</ul>

To provision the EKS cluster, this Stack requires integration with AWS. 
Navigate to the Settings, Integrations page and select the dropdown arrow to access the following selection screen:

image1-18

image1

The following links will help you set up the necessary integration with your Cloud Provider of choice.
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/integrations/cloud-providers/aws" target="_blank" rel="noopener">AWS</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/integrations/cloud-providers/azure" target="_blank" rel="noopener">Azure</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.spacelift.io/integrations/cloud-providers/gcp" target="_blank" rel="noopener">GCP</a></li>
</ul>
To simplify things, I decided to utilize an AWS role that already exists with the necessary permissions to provide the essential resources associated with this Terraform example.

image9-4

To Trigger a Run, select Trigger on the right side of the Stacks view.

image3-16

<h3>Trigger Run Status</h3>
<h4>Planning</h4>
After you manually trigger the Run in the Stack view, Spacelift will deploy a runner image, initialize the Cloud Provider, and run a Terraform Plan based upon the contents of the Terraform code.
After a successful planning phase, you can check the log to see the planned changes.

image5-12

Trigger Run Status - Planning

image6-7

Trigger Run Status - Finished Deployment

To review the outputs associated with this EKS cluster deployment, navigate to the outputs page. You can find all pertinent information about this cluster.

image12-4

Outputs

You have completed the deployment of an EKS cluster via Terraform with Spacelift!

Spacelift effectively manages Terraform state, more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and includes many more features.

How to Deploy your Infrastructure in CI/CD using Terraform

Managing Azure Resources using Spacelift

Now that the Kubernetes integration is initially available learn how to complete the deployment of an EKS cluster via Terraform with Spacelift!

70-kubernetes-cluster-with-terraform-and-spacelift

Initial Availability of Kubernetes Integration in Spacelift

At its most basic form, <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code (IaC)</a> is the provisioning and lifecycle management of infrastructure through code instead of manual processes. Over time, the code part has evolved, and each language is either imperative or declarative. The IaC tooling landscape has matured over the years. We have plenty of options to choose from; open-source and commercial. Over the past several years, a majority of the infrastructure related developments is summed up in a single word or ecosystem… Kubernetes!
Kubernetes is nothing new to Spacelift. The platform utilizes Kubernetes for infrastructure, our Public workers run on Kubernetes, and Spacelift support’s our customer&#8217;s deployment of <a href="https://github.com/spacelift-io/spacelift-workerpool-k8s" target="_blank" rel="nofollow noopener">Private workers</a> on Kubernetes via <a href="https://helm.sh/" target="_blank" rel="nofollow noopener">Helm</a>.

image1-17

Kubernetes (Beta)

Kustomize drives Kubernetes support in Spacelift with native support in kubectl. The screenshot below highlights the ability to run kubectl from a Task within a given Stack.

image4-14

Tasks

The team is very excited to see what our customers and community decide to do with the newest integration. The following screenshot is a sample of the output from Spacelift during a Kubernetes Planning phase.

image2-17

Kubernetes Planning phase

<ul>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Problems that Spacelift Solves&quot;}" data-sheets-userformat="{&quot;2&quot;:4224,&quot;10&quot;:2,&quot;15&quot;:&quot;Arial&quot;}">Infrastructure Problems that Spacelift Solves</a></li>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
<li><a href="https://spacelift.io/blog/spacelift-worker-pools-on-kubernetes" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift Worker Pools on Kubernetes with Spacelift&quot;}" data-sheets-userformat="{&quot;2&quot;:4224,&quot;10&quot;:2,&quot;15&quot;:&quot;Arial&quot;}">Spacelift Worker Pools on Kubernetes with Spacelift</a></li>
</ul>

Get started on your journey by signing up for a <a href="https://spacelift.io" target="_blank" rel="noopener">Free Trial</a> and taking it for a spin worldwide! Sign-up is as easy as authenticating with the provider of your choice.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Kubernetes is nothing new to Spacelift, but the team is very excited to see what our customers and community decide to do with the newest integration.

Sean O'Dell

Spacelift in the GitHub Marketplace

Get the newsletter

Recent posts

Spacelift Worker Pools on Kubernetes with Spacelift

Provision a Kubernetes Cluster with Spacelift and Terraform

Initial Availability of Kubernetes Integration in Spacelift