When it comes to changing products, the biggest obstacle is often migration. Yes, the product looks amazing, but once you start thinking about all the work involved migrating from the old one to the new one, you abandon the idea. A seamless migration experience would stop you missing out. Think of how Apple enables painless device upgrades: You simply keep your old phone next to your new one for less than an hour, and everything migrates.
Terraform Cloud (TFC) recently switched to a RUM (Resources Under Management) pricing model. But this model is unsustainable for many customers, as it will be tough to predict what their bill will look like at the end of the month. You can read more about the changes here.
Migrating from Terraform Cloud is easier than enduring their new pricing model, so at Spacelift, we developed a way to speed up the migration process: We call it the Spacelift Migration Kit.
What we will cover:
There are several reasons why you would want to migrate from Terraform Cloud:
1. Pricing
Terraform Cloud’s RUM pricing is unpredictable and will cost you a lot, and even if you sign a yearly deal for several resources, you will still end up paying for things that TFC doesn’t manage.
2. Terraform only-management
With TFC, you can only manage Terraform and, to some extent, Terragrunt. You can’t customize the binary, nor can you use other tools such as Pulumi, CloudFormation, or OpenTofu. This means that you will need other products to manage other parts of your infrastructure.
3. No dependencies workflow
With TFC, you can’t create dependencies between your workspaces, nor can you natively share outputs between them, making environment promotion impossible to implement.
4. Limits for policies and run tasks in lower tiers
In the lower pricing tiers, you can create 1 policy set with 5 policies in it, and only one of these policies can be mandatory. Similarly, you can integrate only 1 run task for 10 workspaces. This is highly limiting and you won’t be able to do the job with this kind of limits, forcing you to switch to the next tier.
5. Don’t have full control over your workflow or over the tools you are integrating
With TFC you can’t fully control what happens between all runner phases, and you are limited to integrating only the third-party tools that are supported by run tasks.
6. No-code provisioning is limited to modules
No-code provisioning is a really helpful feature for achieving self-service infrastructure. However, TFC limits the self-service aspect only to your modules which makes the whole aspect kind of limiting. Usually, in your automations, you are using multiple modules in a single configuration, which translates to you workaround this issue by creating a “module” of modules, which in Terraform isn’t a best practice.
This will also force you to push the “module” of modules to your registry, making things even more complicated and confusing, especially for new people in your organization.
7. HashiCorp support
Many people are complaining about the level of support they receive from HashiCorp whenever they encounter issues. You can read more about their support issues in this Reddit thread.
8. HashiCorp license changes
The latest license change to BSL and also the rumors around HashiCorp selling, bring a lot of uncertainty in TFC as well.
Spacelift is an infrastructure automation management platform for Terraform, OpenTofu, Terragrunt, Pulumi, CloudFormation, Kubernetes, and Ansible. The platform’s pricing is predictable and based on concurrency, allowing you to correctly appreciate how your bill would look at the end of the month.
Being a multi-infrastructure automation platform allows you to integrate multiple tools into the same workflow by creating dependencies and sharing outputs between the configuration stacks, making it easy to achieve environment promotion and making each configuration item small to easily isolate where the issues are actually coming from.
Spacelift uses OPA for policy as code and lets you define policies to control various aspects of the platform, from what resources you can create, what parameters those resources may have, how many approvals you need for a run, where to send notifications, what happens when a PR is open an more.
For self-service infrastructure, Spacelift offers Blueprints, which can configure every aspect of your stack, starting from what repo to use, which integrations to attach (cloud/policies/contexts) to defining input variables that will be available in a form for people to add when they are running the blueprint.
In addition, Spacelift offers its own Kubernetes Operator that can be easily used to define Spacelift resources, which, in the end, will provision and configure your infrastructure.
To migrate from Terraform Cloud and move your workspaces in bulk, take the following steps:
- Clone the Spacelift Migration Kit.
- Log in to both Spacelift and Terraform Cloud using the
terraform logincommand. - Modify the exporter.tfc.example.tfvars and make changes to the values related to your Terraform Cloud account. Copy this file to exporters/tfc and rename this file to terraform.tfvars.
- Run
terraform initandterraform applyin the exporters/tfc directory to export the tfc configuration → this will generate an out directory with a file called data.json. - Go to the generator directory and run
initandapplyusing the data.json as a variable file. - Review and edit (if necessary) the generated code and commit it to a repository.
- Modify the manager-stack.example.tfvars file to reflect your environment, change its name to terraform.tfvars and copy it to the manager-stack directory (add here the name and the branch of the created repository).
- Run
terraform initandapply. You will get the manager stack in your Spacelift account and once this runs, all of your Terraform Cloud workspaces will be migrated.
Let’s cover these steps in detail.
Cloning the repository is a fairly simple process. Simply go to Spacelift Migration Kit, and select either the ssh or http links and run git clone.
1. Log in to Spacelift
You will need to log in to Spacelift from your CLI. This step will be valid for every migration:
$ terraform login spacelift.ioAdd “yes” to the prompt and your browser will open:
Make sure you clone the Spacelift Migration Kit GitHub Repository.
2. Log in to TFC
To start, you will also need to log in to Terraform Cloud. To do that, you will need to run a similar command to the one we’ve done for Spacelift:
$ terraform loginAgain, say “yes” to the prompt, generate your API token, and you are good to go.
3. Export Terraform Cloud resources
Open exporter.tfc.example.tfvars and modify it with values related to your Terraform Cloud account:
# Terraform Cloud/Enterprise organization name
tfc_organization = "your_tfc_account"
# Export Terraform state to files?
export_state = true
# Terraform Cloud/Enterprise does not return the VCS provider name so we use the value below instead.
vcs_provider = "vcs-provider"
# The name of the entity containing the repository.
# The value should be empty for GitHub.com, the user/organization for GitHub (custom application),
# the project for Bitbucket, and the namespace for Gitlab.
vcs_namespace = "vcs_namespace"
# When the branch for the stack is the repository's default branch,
# the value is empty so we use the value provided below instead
vcs_default_branch = "main"After you have completed the changes, you can copy the file to the exporters/tfc directory, and rename it to terraform.tfvars to avoid having to pass the variable file to the Terraform commands that you are going to use.
Go to the exports/tfc directory and run the following commands:
$ terraform init
$ terraform apply -auto-approve
The apply command will generate an out directory containing:
.
├── data.json
└── state-files
├── stack1.tfstate
└── stackn.tfstate
In the state-files directory, you will have all the state files downloaded for all of your workspaces. In the data.json file, you will have the Terraform Cloud Workspaces information exported as Spacelift Stacks.
Once they are exported, the state files can be imported into Spacelift or any other backend supported by Terraform.
4. Generate Spacelift configuration
When you have done this, go to the generator directory. The generator.tftpl file is the template file that will generate the configuration for the Spacelift Stacks and Environment Variables. You don’t need to make any changes here, but you have the option to easily customize it to add other parameters to those resources. Just make sure you are building a valid configuration because otherwise, you will receive errors.
To generate the configuration per se, all you have to do is:
$ terraform init
$ terraform apply -auto-approve -var-file=../out/data.jsonThis will generate a main.tf file in the out directory, containing information related to the Stacks:
.
├── data.json
├── main.tf
└── state-files
├──stack1.tfstate
└── stackn.tfstate
Mapping resources from a vendor to Spacelift resources is an inherently complex task that does not always produce a precise or definitive outcome. This is primarily due to the presence of certain discrepancies or limitations in the functionality of the resources, as well as potential challenges that may arise during the mapping process.
In light of this, it is imperative to exercise due diligence with the generated Terraform code to ensure that it aligns with your expectations and requirements. If the code fails to meet these standards, you may need to either attempt the mapping process again with an alternative configuration or make necessary modifications to the Terraform code itself.
After you finish the changes related to Terraform code, you should create a VCS repository containing the main.tf file from the out directory. Your Spacelift account should have access to the VCS provider in which the Terraform Cloud provider repositories were initially created.
5. Create the stacks
To create the stacks, you will need to copy the manager-stack.example.tfvars file to the manager-stack directory and rename it terraform.tfvars to avoid having to pass it to the Terraform commands.
You will need to make changes to this file to reflect your Spacelift environment:
# Name of the manager stack
stack_name = "Spacelift Manager"
# Description for the stack
stack_description = "Spacelift resources manager"
# Name of the repository to associate with the stack
repository = "spacelift"
# Name of the branch to associate with the stack
branch = "main"
# Path to the folder containing the Terraform code, in case of a monorepo
project_root = ""
# Import the Terraform state for the managed stacks into Spacelift?
import_state = false
# Spacelift API endpoint
spacelift_api_key_endpoint = "https://example.app.spacelift.io/"
# Spacelift API key ID - Alternatively, you could pass that value via the SPACELIFT_API_KEY_ID env var
spacelift_api_key_id = ""
# Spacelift API key secret - Alternatively, you could pass that value via the SPACELIFT_API_KEY_SECRET env var
spacelift_api_key_secret = ""If you don’t already have a Spacelift API key, you can go to Settings → API keys and generate one:
After you click Add Key, a file will be downloaded that contains the API key secret. The API Key ID can be found in the previous screen.
The keys can be passed directly to the tfvars file, which is not recommended, or you can load them from environment variables.
Now that you have completed the changes, all you have to do is go to the manager-stack directory and run the following:
$ terraform init
$ terraform apply -auto-approve
After the apply finishes, you can go to the Spacelift console and see the management stack in action. This stack will be responsible for the lifecycle of all the other stacks that are migrated from Terraform Cloud Workspaces. After this stack finishes running, all of your Terraform Cloud Workspaces will be migrated.
6. Post-migration cleanup
Before you can use Spacelift to manage your infrastructure, you may need to make changes to the Terraform code for your infrastructure, depending on how the Terraform state is managed.
If the Terraform state is managed by Spacelift, perform the following actions. Otherwise, you can skip this section:
- Remove any backend/cloud block from the Terraform code that manages your infrastructure to avoid a conflict with Spacelift’s backend.
- Delete the import_state_file arguments from the main.tf file in the Terraform code repository you have created.
- After the manager stack has successfully run, the mounted Terraform state files are not needed anymore and can be deleted by setting the import_state argument to false in the terraform.tfvars file from the manager-stack directory and running terraform apply -auto-approve
Starting fresh
You can delete all the changes that you’ve made by running:
$ rm -rf ./out ./{exporters/tfc,generator,manager-stack}/.terraform ./{exporters/tfc,generator,manager-stack}/.terraform.lock.hcl ./{exporters/tfc,generator,manager-stack}/terraform.tfstate ./{exporters/tfc,generator,manager-stack}/terraform.tfstate.backup ./{exporters/tfc,generator,manager-stack}/terraform.tfvarsThis will help you start a new migration.
Right now, this migration kit works only for Terraform Cloud and Terraform Enterprise.
For Terraform Cloud and Enterprise, the following limitations have been identified:
- The variable sets are not exposed, so they cannot be listed and exported.
- The name of the Version Control System (VCS) provider for a stack is not returned, so it has to be set in the exporter configuration file.
- When the branch for the stack is the repository default branch, the value is empty. You can set the value for the default branch in the exporter configuration file or edit the generated Terraform code.
This automation makes it easier to migrate from one vendor to Spacelift by getting the existing configuration and state and mapping it to Spacelift resources. We know that Terraform Cloud’s new pricing model is causing problems, so we really wanted to build something that will make it quicker for you to migrate to a product that offers more for less.
To learn more about the advantages of migrating from Terraform Cloud to Spacelift, check out our Terraform Cloud alternative page.
This is not a one-size-fits-all solution. You will probably need to tweak it to adapt it to your requirements. Nonetheless, it saves considerable time and makes the overall process leaner.
Terraform Management Made Easy
Spacelift effectively manages Terraform state and more complex workflows, and it supports policy as code, programmatic configuration, context sharing, drift detection, and resource visualization, as well as many other features.
