Elevating IaC Workflows with Spacelift Stacks and Dependencies 🛠️

Register for the July 23 demo →

Product

How to Migrate From Terraform Cloud to Spacelift

170.migrate from terraform cloud

When it comes to changing products, the biggest obstacle is often migration. Yes, the product looks amazing, but once you start thinking about all the work involved migrating from the old one to the new one, you abandon the idea. A seamless migration experience would stop you missing out. Think of how Apple enables painless device upgrades: You simply keep your old phone next to your new one for less than an hour, and everything migrates. 

Terraform Cloud (TFC) recently switched to a RUM (Resources Under Management) pricing model. But this model is unsustainable for many customers, as it will be tough to predict what their bill will look like at the end of the month. You can read more about the changes here.

Migrating from Terraform Cloud is easier than enduring their new pricing model, so at Spacelift, we developed a way to speed up the migration process: We call it the Spacelift Migration Kit

What we will cover:

  1. Why migrate from Terraform Cloud?
  2. Overview of Spacelift features
  3. Getting started with Spacelift Migration Kit
  4. How to migrate from Terraform Cloud
  5. Known limitations
  6. Migrate from Terraform Cloud to Spacelift – Video

Why migrate from Terraform Cloud?

There are several reasons why you would want to migrate from Terraform Cloud:

1. Pricing

Terraform Cloud’s RUM pricing is unpredictable and will cost you a lot, and even if you sign a yearly deal for several resources, you will still end up paying for things that TFC doesn’t manage.

2. Terraform only-management

With TFC, you can only manage Terraform and, to some extent, Terragrunt. You can’t customize the binary, nor can you use other tools such as Pulumi, CloudFormation, or OpenTofu. This means that you will need other products to manage other parts of your infrastructure.

3. No dependencies workflow

With TFC, you can’t create dependencies between your workspaces, nor can you natively share outputs between them, making environment promotion impossible to implement.

4. Limits for policies and run tasks in lower tiers

In the lower pricing tiers, you can create 1 policy set with 5 policies in it, and only one of these policies can be mandatory. Similarly, you can integrate only 1 run task for 10 workspaces. This is highly limiting and you won’t be able to do the job with this kind of limits, forcing you to switch to the next tier.

5. Don’t have full control over your workflow or over the tools you are integrating

With TFC you can’t fully control what happens between all runner phases, and you are limited to integrating only the third-party tools that are supported by run tasks.

6. No-code provisioning is limited to modules

No-code provisioning is a really helpful feature for achieving self-service infrastructure. However, TFC limits the self-service aspect only to your modules which makes the whole aspect kind of limiting. Usually, in your automations, you are using multiple modules in a single configuration, which translates to you workaround this issue by creating a “module” of modules, which in Terraform isn’t a best practice.

This will also force you to push the “module” of modules to your registry, making things even more complicated and confusing, especially for new people in your organization.

7. HashiCorp support

Many people are complaining about the level of support they receive from HashiCorp whenever they encounter issues. You can read more about their support issues in this Reddit thread.

8. HashiCorp license changes

The latest license change to BSL and also the rumors around HashiCorp selling, bring a lot of uncertainty in TFC as well.

Overview of Spacelift features

Spacelift is an infrastructure automation management platform for Terraform, OpenTofu, Terragrunt, Pulumi, CloudFormation, Kubernetes, and Ansible. The platform’s pricing is predictable and based on concurrency, allowing you to correctly appreciate how your bill would look at the end of the month.

Being a multi-infrastructure automation platform allows you to integrate multiple tools into the same workflow by creating dependencies and sharing outputs between the configuration stacks, making it easy to achieve environment promotion and making each configuration item small to easily isolate where the issues are actually coming from.

Spacelift uses OPA for policy as code and lets you define policies to control various aspects of the platform, from what resources you can create, what parameters those resources may have, how many approvals you need for a run, where to send notifications, what happens when a PR is open an more.

For self-service infrastructure, Spacelift offers Blueprints, which can configure every aspect of your stack, starting from what repo to use, which integrations to attach (cloud/policies/contexts) to defining input variables that will be available in a form for people to add when they are running the blueprint.

In addition, Spacelift offers its own Kubernetes Operator that can be easily used to define Spacelift resources, which, in the end, will provision and configure your infrastructure.

Getting started with Spacelift Migration Kit

To migrate from Terraform Cloud and move your workspaces in bulk, take the following steps:

  1. Install Python3.10 or newer and Poetry
  2. Clone the Spacelift Migration Kit.
  3. Install other dependencies and the spacemk command in a Python virtual environment
  4. Create your config.yml
  5. [Optional] Audit your current setup
  6. Export the configuration
  7. Generate the Terraform/OpenTofu configuration
  8. Create a repository based on the generated code
  9. Create a stack based on the repository
  10. State Migration
  11. [Optional] Customization
  12. Useful information

Let’s cover these steps in detail.

How to migrate from Terraform Cloud

1. Install prerequisites

We need to ensure that a version newer than python3.10 is installed and also that poetry is installed.

Installing poetry on MacOS using pipx

brew install pipx
pipx install poetry
pipx ensurepath

To install it on any other operating system, check out this tutorial.

2. Clone the Spacelift migration kit

Cloning the repository is a fairly simple process. Simply go to Spacelift Migration Kit, and select either the ssh or http links and run git clone.

git clone git@github.com:spacelift-io/spacelift-migration-kit.git

3. Install spacemk

Go to the Spacelift migration kit directory, and run the following:

poetry install            
Creating virtualenv spacemk-liMdtQjr-py3.12 in /Users/flaviuscdinu/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file

Now after you have installed spacemk, activate the virtual environment:

poetry shell

4. Create your config.yml

Inside the repository, there is a config.yml.example file, containing the values you need to configure for the migration:

# These are all the possible properties for the configuration files.
# The values included in this file are the default values for the property they are associated with.
exporter:
 name: # Valid values: terraform
 settings:
   # Specific to the Terraform exporter (exporter.name: terraform)
   api_endpoint: https://app.terraform.io
   api_token:
   include:
     workspaces: ^example-.*$


generator:
 extra_vars:
   foo: bar # "{{ extra_vars.foo }}" in a template will be replaced by "bar"


github:
 api_token:
 endpoint: https://api.github.com


spacelift:
 api:
   api_key_endpoint: https://<ACCOUNT NAME>>.app.spacelift.io/graphql
   api_key_id:
   api_key_secret:

Rename this file, or create a new one that should be called “config.yml”.

In this file, you must add the login details to your Terraform Cloud account and your Spacelift account. You can also use environment variables inside this file, to avoid using credentials inside the file (these variables should be prefixed with the “$” sign).

Here is an example that uses environment variables:

# These are all the possible properties for the configuration files.
# The values included in this file are the default values for the property they are associated with.
exporter:
 name: terraform
 settings:
   # Specific to the Terraform exporter (exporter.name: terraform)
   api_endpoint: https://app.terraform.io
   api_token: $TFC_TOKEN


generator:
 extra_vars:
   foo: bar # "{{ extra_vars.foo }}" in a template will be replaced by "bar"


github:
 api_token: $GITHUB_TOKEN
 endpoint: https://api.github.com


spacelift:
 api:
   api_key_endpoint: https://saturnhead.app.spacelift.io/graphql
   api_key_id: $SPACELIFT_API_KEY_ID
   api_key_secret: $SPACELIFT_API_KEY_SECRET

You will need to export these values inside of your terminal (e.g.):

export TFC_TOKEN=tfctoken

Based on the values above, we will include all workspaces in the migration.

If you don’t already have a Spacelift API key, you can go to Settings → API keys and generate one:

tfc migration create spacelift api

After you click Add Key, a file will be downloaded that contains the API key secret. The API Key ID can be found in the previous screen.

To create an API key for Terraform Cloud check this tutorial, and to create an GitHub token, check this tutorial.

5. [Optional] Audit your current setup

To audit your current setup, you can simply run the spacemk audit command:

Agent Pools: 1 (including 1 with warnings)
Modules: 1
Organizations: 1
Policies: 1 (including 1 with warnings)
Policy Sets: 1
Projects: 4
Providers: 0
Tasks: 0
Teams: 601
Variable Set Variables: 0
Variable Sets: 0
Workspace Variables: 0
Workspaces: 1

This will give you all the information about your Terraform Cloud account, and everything that can be exported.

6. Export your configuration

To export your configuration, simply run spacemk export. This command will generate a json file in the tmp directory, called data.json.

In this file, you will see information about everything that will be migrated and how they are mapping out to Spacelift resources.

{
 "context_variables": [],
 "contexts": [],
 "modules": [
   {
     "_migration_id": "aks",
     "_relationships": {
       "space": {
         "_migration_id": "saturnhead",
         "_source_id": "saturnhead",
         "name": "saturnhead",
         "requires_terraform_workflow_tool": false
       }
     },
     "_source_id": "...",
     "name": "aks",
     "status": "setup_complete",
     "terraform_provider": "az",
     "vcs": {
       "branch": "main",
       "namespace": "flavius-dinu",
       "provider": "github_custom",
       "repository": "terraform-az-aks"
     },
     "visibility": "private"
   }
 ],
 "spaces": [
   {
     "_migration_id": "saturnhead",
     "_relationships": {},
     "_source_id": "saturnhead",
     "name": "saturnhead",
     "requires_terraform_workflow_tool": false
   },
   {
     "_migration_id": "default_project",
     "_relationships": {},
     "_source_id": "...",
     "name": "Default Project",
     "requires_terraform_workflow_tool": false
   },
   {
     "_migration_id": "sdsd",
     "_relationships": {},
     "_source_id": "...",
     "name": "sdsd",
     "requires_terraform_workflow_tool": false
   }
 ],
 "stack_variables": [],
 "stacks": [
   {
     "_migration_id": "blog_examples",
     "_relationships": {
       "space": {
         "_migration_id": "default_project",
         "_source_id": "...",
         "name": "Default Project",
         "requires_terraform_workflow_tool": false
       }
     },
     "_source_id": "...",
     "autodeploy": false,
     "description": null,
     "has_secret_variables_with_invalid_name": false,
     "has_variables_with_invalid_name": false,
     "name": "blog-examples",
     "slug": "blog-examples",
     "terraform": {
       "version": "1.5.5",
       "workflow_tool": "TERRAFORM_FOSS"
     },
     "vcs": {
       "branch": "main",
       "namespace": "flavius-dinu",
       "project_root": "migration",
       "provider": "github_custom",
       "repository": "blog-examples"
     }
   }
 ]
}

This file can be reviewed and modified to accommodate your needs.

7. Generate the Terraform/OpenTofu configuration

To generate the Terraform/OpenTofu configuration, you will need to use the spacemk generate command.

terraform {
 required_providers {
   spacelift = {
     source  = "spacelift-io/spacelift"
     version = "~> 1.0"
   }
 }
}


resource "spacelift_space" "saturnhead" {
 inherit_entities = true
 name             = "saturnhead"
 parent_space_id  = "root"
}


resource "spacelift_stack" "default_project_blog_examples" {
 branch            = "main"
 manage_state      = false
 name              = "blog-examples"
 project_root      = "migration"
 repository        = "blog-examples"
 space_id          = spacelift_space.default_project.id
 terraform_version = "1.5.5"


 github_enterprise {
   namespace = "flavius-dinu"
 }
}




resource "spacelift_module" "saturnhead_aks" {
 branch             = "main"
 name               = "aks"
 repository         = "terraform-az-aks"
 space_id           = spacelift_space.saturnhead.id
 terraform_provider = "az"


 github_enterprise {
   namespace = "flavius-dinu"
 }
}

This will generate the Spacelift configuration required for the migration. Again, the code can be modified to accommodate your needs.

If you are not using the default VCS integration, you will also need to specify the id property in the vcs block (in my case, I will add an id to the github_enterprise block):

github_enterprise {
   namespace = "flavius-dinu"
   id        = "flavius-dinu"
 }

8. Create a repository based on this code

In your VCS, simply create a repository with the exported code. Ensure the vcs provider you are using is configured in Spacelift, and the code repositories you’ve exported from TFC are available in this vcs provider.

To configure a new VCS provider inside your Spacelift provider, follow the steps from this tutorial for your VCS.

Create the repo with the exported code, this is how it looks like for my GitHub configuration:

tfc migrate kit repository

9. Create a stack based on the repository

Go to your Spacelift account, select Stacks, and the Create Stack option:

create import tfc stack

Add a name to your stack, select a space, and add optional labels and description.

connect to source code tfc migration

In the next step, select the repository containing your migration code as shown above, and specify the branch you want to use.

choose vendor tfc migration kit

Next, you can choose your vendor, it can be either Terraform, or OpenTofu, so I will accept the default Terraform FOSS workflow.

spacelift stack settings

Ensure that the Administrative option is toggled on in the define behavior tab, and accept all other defaults to create the stack.

After the stack is created, trigger a run:

spacelift trigger run stack

After reaching the unconfirmed state, you will see all the resources that it will create, in my case, there are 5 spaces, 1 stack, and 1 module.

spacelift resources tfc migrate

You can easily see that all resources have been migrated successfully.

10. State Migration

If we go to our stack and select resources, we will see that there are no resources in the state:

spacelift stack resources stack

In TFC, however, we have a couple of resources created:

tfc resources created

In our local environment, when we’ve done the export, the state files were also downloaded:

tfc migration state files

To upload our state files, we must run the following command: spacemk import-state-files-to-spacelift.

Note: if you have changed the name of the stack in your terraform configuration, you will also need to modify the data.json file, to reflect the stack name, when importing the state.

After running the command, if we go back to our stack, we can see the state has been migrated successfully:

successfull state migration

11. [Optional] Customization

While the Migration Kit does a great job when it comes to migrating to Spacelift, we understand that every migration is different and you may need to customize the functionality of the automation. To do that, you can easily implement your own commands and even your own exporter. More details about this can be found here.

Key points

This automation makes it easier to migrate from one vendor to Spacelift by getting the existing configuration and state and mapping it to Spacelift resources. We know that Terraform Cloud’s new pricing model is causing problems, so we really wanted to build something that will make it quicker for you to migrate to a product that offers more for less.

To learn more about the advantages of migrating from Terraform Cloud to Spacelift, check out our Terraform Cloud alternative page.

This is not a one-size-fits-all solution. You will probably need to tweak it to adapt it to your requirements. Nonetheless, it saves considerable time and makes the overall process leaner.

Terraform Management Made Easy

Spacelift effectively manages Terraform state and more complex workflows, and it supports policy as code, programmatic configuration, context sharing, drift detection, and resource visualization, as well as many other features.

Start free trial

How can Spacelift stacks & dependencies elevate your IaC workflows?

Don’t miss our July 23 webinar.

Register for the webinar