When it comes to changing products, the biggest obstacle is often migration. Yes, the product looks amazing, but once you start thinking about all the work involved migrating from the old one to the new one, you abandon the idea. A seamless migration experience would stop you missing out. Think of how Apple enables painless device upgrades: You simply keep your old phone next to your new one for less than an hour, and everything migrates.
Terraform Cloud (TFC) recently switched to a RUM (Resources Under Management) pricing model. But this model is unsustainable for many customers, as it will be tough to predict what their bill will look like at the end of the month. You can read more about the changes here.
Migrating from Terraform Cloud is easier than enduring their new pricing model, so at Spacelift, we developed a way to speed up the migration process: We call it the Spacelift Migration Kit.
What we will cover:
There are several reasons why you would want to migrate from Terraform Cloud:
1. Pricing
Terraform Cloud’s RUM pricing is unpredictable and will cost you a lot, and even if you sign a yearly deal for several resources, you will still end up paying for things that TFC doesn’t manage.
2. Terraform only-management
With TFC, you can only manage Terraform and, to some extent, Terragrunt. You can’t customize the binary, nor can you use other tools such as Pulumi, CloudFormation, or OpenTofu. This means that you will need other products to manage other parts of your infrastructure.
3. No dependencies workflow
With TFC, you can’t create dependencies between your workspaces, nor can you natively share outputs between them, making environment promotion impossible to implement.
4. Limits for policies and run tasks in lower tiers
In the lower pricing tiers, you can create 1 policy set with 5 policies in it, and only one of these policies can be mandatory. Similarly, you can integrate only 1 run task for 10 workspaces. This is highly limiting and you won’t be able to do the job with this kind of limits, forcing you to switch to the next tier.
5. Don’t have full control over your workflow or over the tools you are integrating
With TFC you can’t fully control what happens between all runner phases, and you are limited to integrating only the third-party tools that are supported by run tasks.
6. No-code provisioning is limited to modules
No-code provisioning is a really helpful feature for achieving self-service infrastructure. However, TFC limits the self-service aspect only to your modules which makes the whole aspect kind of limiting. Usually, in your automations, you are using multiple modules in a single configuration, which translates to you workaround this issue by creating a “module” of modules, which in Terraform isn’t a best practice.
This will also force you to push the “module” of modules to your registry, making things even more complicated and confusing, especially for new people in your organization.
7. HashiCorp support
Many people are complaining about the level of support they receive from HashiCorp whenever they encounter issues. You can read more about their support issues in this Reddit thread.
8. HashiCorp license changes
The latest license change to BSL and also the rumors around HashiCorp selling, bring a lot of uncertainty in TFC as well.
Spacelift is an infrastructure automation management platform for Terraform, OpenTofu, Terragrunt, Pulumi, CloudFormation, Kubernetes, and Ansible. The platform’s pricing is predictable and based on concurrency, allowing you to correctly appreciate how your bill would look at the end of the month.
Being a multi-infrastructure automation platform allows you to integrate multiple tools into the same workflow by creating dependencies and sharing outputs between the configuration stacks, making it easy to achieve environment promotion and making each configuration item small to easily isolate where the issues are actually coming from.
Spacelift uses OPA for policy as code and lets you define policies to control various aspects of the platform, from what resources you can create, what parameters those resources may have, how many approvals you need for a run, where to send notifications, what happens when a PR is open an more.
For self-service infrastructure, Spacelift offers Blueprints, which can configure every aspect of your stack, starting from what repo to use, which integrations to attach (cloud/policies/contexts) to defining input variables that will be available in a form for people to add when they are running the blueprint.
In addition, Spacelift offers its own Kubernetes Operator that can be easily used to define Spacelift resources, which, in the end, will provision and configure your infrastructure.
To migrate from Terraform Cloud and move your workspaces in bulk, take the following steps:
- Install Python3.10 or newer and Poetry
- Clone the Spacelift Migration Kit.
- Install other dependencies and the spacemk command in a Python virtual environment
- Create your config.yml
- [Optional] Audit your current setup
- Export the configuration
- Generate the Terraform/OpenTofu configuration
- Create a repository based on the generated code
- Create a stack based on the repository
- State Migration
- [Optional] Customization
- Useful information
Let’s cover these steps in detail.
We need to ensure that a version newer than python3.10 is installed and also that poetry is installed.
Installing poetry on MacOS using pipx
brew install pipx
pipx install poetry
pipx ensurepath
To install it on any other operating system, check out this tutorial.
Cloning the repository is a fairly simple process. Simply go to Spacelift Migration Kit, and select either the ssh or http links and run git clone.
git clone git@github.com:spacelift-io/spacelift-migration-kit.gitGo to the Spacelift migration kit directory, and run the following:
poetry install
Creating virtualenv spacemk-liMdtQjr-py3.12 in /Users/flaviuscdinu/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock fileNow after you have installed spacemk, activate the virtual environment:
poetry shellInside the repository, there is a config.yml.example file, containing the values you need to configure for the migration:
# These are all the possible properties for the configuration files.
# The values included in this file are the default values for the property they are associated with.
exporter:
name: # Valid values: terraform
settings:
# Specific to the Terraform exporter (exporter.name: terraform)
api_endpoint: https://app.terraform.io
api_token:
include:
workspaces: ^example-.*$
generator:
extra_vars:
foo: bar # "{{ extra_vars.foo }}" in a template will be replaced by "bar"
github:
api_token:
endpoint: https://api.github.com
spacelift:
api:
api_key_endpoint: https://<ACCOUNT NAME>>.app.spacelift.io/graphql
api_key_id:
api_key_secret:Rename this file, or create a new one that should be called “config.yml”.
In this file, you must add the login details to your Terraform Cloud account and your Spacelift account. You can also use environment variables inside this file, to avoid using credentials inside the file (these variables should be prefixed with the “$” sign).
Here is an example that uses environment variables:
# These are all the possible properties for the configuration files.
# The values included in this file are the default values for the property they are associated with.
exporter:
name: terraform
settings:
# Specific to the Terraform exporter (exporter.name: terraform)
api_endpoint: https://app.terraform.io
api_token: $TFC_TOKEN
generator:
extra_vars:
foo: bar # "{{ extra_vars.foo }}" in a template will be replaced by "bar"
github:
api_token: $GITHUB_TOKEN
endpoint: https://api.github.com
spacelift:
api:
api_key_endpoint: https://saturnhead.app.spacelift.io/graphql
api_key_id: $SPACELIFT_API_KEY_ID
api_key_secret: $SPACELIFT_API_KEY_SECRETYou will need to export these values inside of your terminal (e.g.):
export TFC_TOKEN=tfctokenBased on the values above, we will include all workspaces in the migration.
If you don’t already have a Spacelift API key, you can go to Settings → API keys and generate one:
After you click Add Key, a file will be downloaded that contains the API key secret. The API Key ID can be found in the previous screen.
To create an API key for Terraform Cloud check this tutorial, and to create an GitHub token, check this tutorial.
To audit your current setup, you can simply run the spacemk audit command:
Agent Pools: 1 (including 1 with warnings)
Modules: 1
Organizations: 1
Policies: 1 (including 1 with warnings)
Policy Sets: 1
Projects: 4
Providers: 0
Tasks: 0
Teams: 601
Variable Set Variables: 0
Variable Sets: 0
Workspace Variables: 0
Workspaces: 1This will give you all the information about your Terraform Cloud account, and everything that can be exported.
To export your configuration, simply run spacemk export. This command will generate a json file in the tmp directory, called data.json.
In this file, you will see information about everything that will be migrated and how they are mapping out to Spacelift resources.
{
"context_variables": [],
"contexts": [],
"modules": [
{
"_migration_id": "aks",
"_relationships": {
"space": {
"_migration_id": "saturnhead",
"_source_id": "saturnhead",
"name": "saturnhead",
"requires_terraform_workflow_tool": false
}
},
"_source_id": "...",
"name": "aks",
"status": "setup_complete",
"terraform_provider": "az",
"vcs": {
"branch": "main",
"namespace": "flavius-dinu",
"provider": "github_custom",
"repository": "terraform-az-aks"
},
"visibility": "private"
}
],
"spaces": [
{
"_migration_id": "saturnhead",
"_relationships": {},
"_source_id": "saturnhead",
"name": "saturnhead",
"requires_terraform_workflow_tool": false
},
{
"_migration_id": "default_project",
"_relationships": {},
"_source_id": "...",
"name": "Default Project",
"requires_terraform_workflow_tool": false
},
…
{
"_migration_id": "sdsd",
"_relationships": {},
"_source_id": "...",
"name": "sdsd",
"requires_terraform_workflow_tool": false
}
],
"stack_variables": [],
"stacks": [
{
"_migration_id": "blog_examples",
"_relationships": {
"space": {
"_migration_id": "default_project",
"_source_id": "...",
"name": "Default Project",
"requires_terraform_workflow_tool": false
}
},
"_source_id": "...",
"autodeploy": false,
"description": null,
"has_secret_variables_with_invalid_name": false,
"has_variables_with_invalid_name": false,
"name": "blog-examples",
"slug": "blog-examples",
"terraform": {
"version": "1.5.5",
"workflow_tool": "TERRAFORM_FOSS"
},
"vcs": {
"branch": "main",
"namespace": "flavius-dinu",
"project_root": "migration",
"provider": "github_custom",
"repository": "blog-examples"
}
}
]
}This file can be reviewed and modified to accommodate your needs.
To generate the Terraform/OpenTofu configuration, you will need to use the spacemk generate command.
terraform {
required_providers {
spacelift = {
source = "spacelift-io/spacelift"
version = "~> 1.0"
}
}
}
resource "spacelift_space" "saturnhead" {
inherit_entities = true
name = "saturnhead"
parent_space_id = "root"
}
resource "spacelift_stack" "default_project_blog_examples" {
branch = "main"
manage_state = false
name = "blog-examples"
project_root = "migration"
repository = "blog-examples"
space_id = spacelift_space.default_project.id
terraform_version = "1.5.5"
github_enterprise {
namespace = "flavius-dinu"
}
}
…
resource "spacelift_module" "saturnhead_aks" {
branch = "main"
name = "aks"
repository = "terraform-az-aks"
space_id = spacelift_space.saturnhead.id
terraform_provider = "az"
github_enterprise {
namespace = "flavius-dinu"
}
}This will generate the Spacelift configuration required for the migration. Again, the code can be modified to accommodate your needs.
If you are not using the default VCS integration, you will also need to specify the id property in the vcs block (in my case, I will add an id to the github_enterprise block):
github_enterprise {
namespace = "flavius-dinu"
id = "flavius-dinu"
}In your VCS, simply create a repository with the exported code. Ensure the vcs provider you are using is configured in Spacelift, and the code repositories you’ve exported from TFC are available in this vcs provider.
To configure a new VCS provider inside your Spacelift provider, follow the steps from this tutorial for your VCS.
Create the repo with the exported code, this is how it looks like for my GitHub configuration:
Go to your Spacelift account, select Stacks, and the Create Stack option:
Add a name to your stack, select a space, and add optional labels and description.
In the next step, select the repository containing your migration code as shown above, and specify the branch you want to use.
Next, you can choose your vendor, it can be either Terraform, or OpenTofu, so I will accept the default Terraform FOSS workflow.
Ensure that the Administrative option is toggled on in the define behavior tab, and accept all other defaults to create the stack.
After the stack is created, trigger a run:
After reaching the unconfirmed state, you will see all the resources that it will create, in my case, there are 5 spaces, 1 stack, and 1 module.
You can easily see that all resources have been migrated successfully.
If we go to our stack and select resources, we will see that there are no resources in the state:
In TFC, however, we have a couple of resources created:
In our local environment, when we’ve done the export, the state files were also downloaded:
To upload our state files, we must run the following command: spacemk import-state-files-to-spacelift.
Note: if you have changed the name of the stack in your terraform configuration, you will also need to modify the data.json file, to reflect the stack name, when importing the state.
After running the command, if we go back to our stack, we can see the state has been migrated successfully:
While the Migration Kit does a great job when it comes to migrating to Spacelift, we understand that every migration is different and you may need to customize the functionality of the automation. To do that, you can easily implement your own commands and even your own exporter. More details about this can be found here.
This automation makes it easier to migrate from one vendor to Spacelift by getting the existing configuration and state and mapping it to Spacelift resources. We know that Terraform Cloud’s new pricing model is causing problems, so we really wanted to build something that will make it quicker for you to migrate to a product that offers more for less.
To learn more about the advantages of migrating from Terraform Cloud to Spacelift, check out our Terraform Cloud alternative page.
This is not a one-size-fits-all solution. You will probably need to tweak it to adapt it to your requirements. Nonetheless, it saves considerable time and makes the overall process leaner.
Terraform Management Made Easy
Spacelift effectively manages Terraform state and more complex workflows, and it supports policy as code, programmatic configuration, context sharing, drift detection, and resource visualization, as well as many other features.
