When it comes to changing products, the biggest obstacle is often migration. Yes, the product looks amazing, but once you start thinking about all the work involved migrating from the old one to the new one, you abandon the idea. A seamless migration experience would stop you missing out. Think of how Apple enables painless device upgrades: You simply keep your old phone next to your new one for less than an hour, and everything migrates.
Terraform Cloud (TFC) recently switched to a RUM (Resources Under Management) pricing model. But this model is unsustainable for many customers, as it will be tough to predict what their bill will look like at the end of the month. You can read more about the changes here.
Migrating from Terraform Cloud is easier than enduring their new pricing model, so at Spacelift, we developed a way to speed up the migration process: We call it the Spacelift Migration Kit.
To migrate from Terraform Cloud and move your workspaces in bulk, take the following steps:
- Clone the Spacelift Migration Kit.
- Log in to both Spacelift and Terraform Cloud using the `terraform login` command.
- Modify the exporter.tfc.example.tfvars and make changes to the values related to your Terraform Cloud account. Copy this file to exporters/tfc and rename this file to terraform.tfvars.
- Run terraform init and terraform apply in the exporters/tfc directory to export the tfc configuration → this will generate an out directory with a file called data.json.
- Go to the generator directory and run init and apply using the data.json as a variable file.
- Review and edit (if necessary) the generated code and commit it to a repository.
- Modify the manager-stack.example.tfvars file to reflect your environment, change its name to terraform.tfvars and copy it to the manager-stack directory (add here the name and the branch of the created repository).
- Run terraform init and apply. You will get the manager stack in your Spacelift account and once this runs, all of your Terraform Cloud workspaces will be migrated.
Let’s cover these steps in detail.
Cloning the repository is a fairly simple process. Simply go to Spacelift Migration Kit, and select either the ssh or http links and run git clone.
You will need to log in to Spacelift from your CLI. This step will be valid for every migration:
$ terraform login spacelift.ioAdd “yes” to the prompt and your browser will open:
Make sure you clone the Spacelift Migration Kit GitHub Repository.
To start, you will also need to log in to Terraform Cloud. To do that, you will need to run a similar command to the one we’ve done for Spacelift:
$ terraform loginAgain, say “yes” to the prompt, generate your API token, and you are good to go.
Exporting Terraform Cloud Resources
Open exporter.tfc.example.tfvars and modify it with values related to your Terraform Cloud account:
# Terraform Cloud/Enterprise organization name
tfc_organization = "your_tfc_account"
# Export Terraform state to files?
export_state = true
# Terraform Cloud/Enterprise does not return the VCS provider name so we use the value below instead.
vcs_provider = "vcs-provider"
# The name of the entity containing the repository.
# The value should be empty for GitHub.com, the user/organization for GitHub (custom application),
# the project for Bitbucket, and the namespace for Gitlab.
vcs_namespace = "vcs_namespace"
# When the branch for the stack is the repository's default branch,
# the value is empty so we use the value provided below instead
vcs_default_branch = "main"After you have completed the changes, you can copy the file to the exporters/tfc directory, and rename it to terraform.tfvars to avoid having to pass the variable file to the Terraform commands that you are going to use.
Go to the exports/tfc directory and run the following commands:
$ terraform init
$ terraform apply -auto-approve
The apply command will generate an out directory containing:
.
├── data.json
└── state-files
├── stack1.tfstate
└── stackn.tfstate
In the state-files directory, you will have all the state files downloaded for all of your workspaces. In the data.json file, you will have the Terraform Cloud Workspaces information exported as Spacelift Stacks.
Once they are exported, the state files can be imported into Spacelift or any other backend supported by Terraform.
Generate Spacelift Configuration
When you have done this, go to the generator directory. The generator.tftpl file is the template file that will generate the configuration for the Spacelift Stacks and Environment Variables. You don’t need to make any changes here, but you have the option to easily customize it to add other parameters to those resources. Just make sure you are building a valid configuration because otherwise, you will receive errors.
To generate the configuration per se, all you have to do is:
$ terraform init
$ terraform apply -auto-approve -var-file=../out/data.jsonThis will generate a main.tf file in the out directory, containing information related to the Stacks:
.
├── data.json
├── main.tf
└── state-files
├──stack1.tfstate
└── stackn.tfstate
Mapping resources from a vendor to Spacelift resources is an inherently complex task that does not always produce a precise or definitive outcome. This is primarily due to the presence of certain discrepancies or limitations in the functionality of the resources, as well as potential challenges that may arise during the mapping process.
In light of this, it is imperative to exercise due diligence with the generated Terraform code to ensure that it aligns with your expectations and requirements. If the code fails to meet these standards, you may need to either attempt the mapping process again with an alternative configuration or make necessary modifications to the Terraform code itself.
After you finish the changes related to Terraform code, you should create a VCS repository containing the main.tf file from the out directory. Your Spacelift account should have access to the VCS provider in which the Terraform Cloud provider repositories were initially created.
Creating the Stacks
To create the stacks, you will need to copy the manager-stack.example.tfvars file to the manager-stack directory and rename it terraform.tfvars to avoid having to pass it to the Terraform commands.
You will need to make changes to this file to reflect your Spacelift environment:
# Name of the manager stack
stack_name = "Spacelift Manager"
# Description for the stack
stack_description = "Spacelift resources manager"
# Name of the repository to associate with the stack
repository = "spacelift"
# Name of the branch to associate with the stack
branch = "main"
# Path to the folder containing the Terraform code, in case of a monorepo
project_root = ""
# Import the Terraform state for the managed stacks into Spacelift?
import_state = false
# Spacelift API endpoint
spacelift_api_key_endpoint = "https://example.app.spacelift.io/"
# Spacelift API key ID - Alternatively, you could pass that value via the SPACELIFT_API_KEY_ID env var
spacelift_api_key_id = ""
# Spacelift API key secret - Alternatively, you could pass that value via the SPACELIFT_API_KEY_SECRET env var
spacelift_api_key_secret = ""If you don’t already have a Spacelift API key, you can go to Settings → API keys and generate one:
After you click Add Key, a file will be downloaded that contains the API key secret. The API Key ID can be found in the previous screen.
The keys can be passed directly to the tfvars file, which is not recommended, or you can load them from environment variables.
Now that you have completed the changes, all you have to do is go to the manager-stack directory and run the following:
$ terraform init
$ terraform apply -auto-approve
After the apply finishes, you can go to the Spacelift console and see the management stack in action. This stack will be responsible for the lifecycle of all the other stacks that are migrated from Terraform Cloud Workspaces. After this stack finishes running, all of your Terraform Cloud Workspaces will be migrated.
Post-Migration Cleanup
Before you can use Spacelift to manage your infrastructure, you may need to make changes to the Terraform code for your infrastructure, depending on how the Terraform state is managed.
If the Terraform state is managed by Spacelift, perform the following actions. Otherwise, you can skip this section:
- Remove any backend/cloud block from the Terraform code that manages your infrastructure to avoid a conflict with Spacelift’s backend.
- Delete the import_state_file arguments from the main.tf file in the Terraform code repository you have created.
- After the manager stack has successfully run, the mounted Terraform state files are not needed anymore and can be deleted by setting the import_state argument to false in the terraform.tfvars file from the manager-stack directory and running terraform apply -auto-approve
Starting Fresh
You can delete all the changes that you’ve made by running:
$ rm -rf ./out ./{exporters/tfc,generator,manager-stack}/.terraform ./{exporters/tfc,generator,manager-stack}/.terraform.lock.hcl ./{exporters/tfc,generator,manager-stack}/terraform.tfstate ./{exporters/tfc,generator,manager-stack}/terraform.tfstate.backup ./{exporters/tfc,generator,manager-stack}/terraform.tfvarsThis will help you start a new migration.
Right now, this migration kit works only for Terraform Cloud and Terraform Enterprise.
For Terraform Cloud and Enterprise, the following limitations have been identified:
- The variable sets are not exposed, so they cannot be listed and exported.
- The name of the Version Control System (VCS) provider for a stack is not returned, so it has to be set in the exporter configuration file.
- When the branch for the stack is the repository default branch, the value is empty. You can set the value for the default branch in the exporter configuration file or edit the generated Terraform code.
This automation makes it easier to migrate from one vendor to Spacelift by getting the existing configuration and state and mapping it to Spacelift resources. We know that Terraform Cloud’s new pricing model is causing problems, so we really wanted to build something that will make it quicker for you to migrate to a product that offers more for less.
To learn more about the advantages of migrating from Terraform Cloud to Spacelift, check out our Terraform Cloud alternative page.
This is not a one-size-fits-all solution. You will probably need to tweak it to adapt it to your requirements. Nonetheless, it saves considerable time and makes the overall process leaner.
Terraform Management Made Easy
Spacelift effectively manages Terraform state and more complex workflows, and it supports policy as code, programmatic configuration, context sharing, drift detection, and resource visualization, as well as many other features.
