Going to AWS Summit London? 🇬🇧🇬🇧

Meet us there →

Product

How to Migrate From Terraform Cloud to Spacelift

170.migrate from terraform cloud

When it comes to changing products, the biggest obstacle is often migration. Yes, the product looks amazing, but once you start thinking about all the work involved migrating from the old one to the new one, you abandon the idea. A seamless migration experience would stop you missing out. Think of how Apple enables painless device upgrades: You simply keep your old phone next to your new one for less than an hour, and everything migrates. 

Terraform Cloud (TFC) recently switched to a RUM (Resources Under Management) pricing model. But this model is unsustainable for many customers, as it will be tough to predict what their bill will look like at the end of the month. You can read more about the changes here.

Migrating from Terraform Cloud is easier than enduring their new pricing model, so at Spacelift, we developed a way to speed up the migration process: We call it the Spacelift Migration Kit

What we will cover:

  1. Why migrate from Terraform Cloud?
  2. Overview of Spacelift features
  3. Getting started with Spacelift Migration Kit
  4. How to migrate from Terraform Cloud
  5. Known limitations
  6. Migrate from Terraform Cloud to Spacelift – Video

Why migrate from Terraform Cloud?

There are several reasons why you would want to migrate from Terraform Cloud:

1. Pricing

Terraform Cloud’s RUM pricing is unpredictable and will cost you a lot, and even if you sign a yearly deal for several resources, you will still end up paying for things that TFC doesn’t manage.

2. Terraform only-management

With TFC, you can only manage Terraform and, to some extent, Terragrunt. You can’t customize the binary, nor can you use other tools such as Pulumi, CloudFormation, or OpenTofu. This means that you will need other products to manage other parts of your infrastructure.

3. No dependencies workflow

With TFC, you can’t create dependencies between your workspaces, nor can you natively share outputs between them, making environment promotion impossible to implement.

4. Limits for policies and run tasks in lower tiers

In the lower pricing tiers, you can create 1 policy set with 5 policies in it, and only one of these policies can be mandatory. Similarly, you can integrate only 1 run task for 10 workspaces. This is highly limiting and you won’t be able to do the job with this kind of limits, forcing you to switch to the next tier.

5. Don’t have full control over your workflow or over the tools you are integrating

With TFC you can’t fully control what happens between all runner phases, and you are limited to integrating only the third-party tools that are supported by run tasks.

6. No-code provisioning is limited to modules

No-code provisioning is a really helpful feature for achieving self-service infrastructure. However, TFC limits the self-service aspect only to your modules which makes the whole aspect kind of limiting. Usually, in your automations, you are using multiple modules in a single configuration, which translates to you workaround this issue by creating a “module” of modules, which in Terraform isn’t a best practice.

This will also force you to push the “module” of modules to your registry, making things even more complicated and confusing, especially for new people in your organization.

7. HashiCorp support

Many people are complaining about the level of support they receive from HashiCorp whenever they encounter issues. You can read more about their support issues in this Reddit thread.

8. HashiCorp license changes

The latest license change to BSL and also the rumors around HashiCorp selling, bring a lot of uncertainty in TFC as well.

Overview of Spacelift features

Spacelift is an infrastructure automation management platform for Terraform, OpenTofu, Terragrunt, Pulumi, CloudFormation, Kubernetes, and Ansible. The platform’s pricing is predictable and based on concurrency, allowing you to correctly appreciate how your bill would look at the end of the month.

Being a multi-infrastructure automation platform allows you to integrate multiple tools into the same workflow by creating dependencies and sharing outputs between the configuration stacks, making it easy to achieve environment promotion and making each configuration item small to easily isolate where the issues are actually coming from.

Spacelift uses OPA for policy as code and lets you define policies to control various aspects of the platform, from what resources you can create, what parameters those resources may have, how many approvals you need for a run, where to send notifications, what happens when a PR is open an more.

For self-service infrastructure, Spacelift offers Blueprints, which can configure every aspect of your stack, starting from what repo to use, which integrations to attach (cloud/policies/contexts) to defining input variables that will be available in a form for people to add when they are running the blueprint.

In addition, Spacelift offers its own Kubernetes Operator that can be easily used to define Spacelift resources, which, in the end, will provision and configure your infrastructure.

Getting started with Spacelift Migration Kit

To migrate from Terraform Cloud and move your workspaces in bulk, take the following steps:

  1. Clone the Spacelift Migration Kit.
  2. Log in to both Spacelift and Terraform Cloud using the terraform login command.
  3. Modify the exporter.tfc.example.tfvars and make changes to the values related to your Terraform Cloud account. Copy this file to exporters/tfc and rename this file to terraform.tfvars.
  4. Run terraform init and terraform apply in the exporters/tfc directory to export the tfc configuration → this will generate an out directory with a file called data.json.
  5. Go to the generator directory and run init and apply using the data.json as a variable file.
  6. Review and edit (if necessary) the generated code and commit it to a repository.
  7. Modify the manager-stack.example.tfvars file to reflect your environment, change its name to terraform.tfvars and copy it to the manager-stack directory (add here the name and the branch of the created repository).
  8. Run terraform init and apply. You will get the manager stack in your Spacelift account and once this runs, all of your Terraform Cloud workspaces will be migrated.

Let’s cover these steps in detail. 

Cloning the repository is a fairly simple process. Simply go to Spacelift Migration Kit, and select either the ssh or http links and run git clone.

How to migrate from Terraform Cloud

1. Log in to Spacelift

You will need to log in to Spacelift from your CLI. This step will be valid for every migration:

$ terraform login spacelift.io

Add “yes” to the prompt and your browser will open:

spacelift migration tool log in to spacelift

Make sure you clone the Spacelift Migration Kit GitHub Repository.

2. Log in to TFC

To start, you will also need to log in to Terraform Cloud. To do that, you will need to run a similar command to the one we’ve done for Spacelift:

$ terraform login

Again, say “yes” to the prompt, generate your API token, and you are good to go.

spacelift migration tool api token

3. Export Terraform Cloud resources

Open exporter.tfc.example.tfvars and modify it with values related to your Terraform Cloud account:

# Terraform Cloud/Enterprise organization name
tfc_organization = "your_tfc_account"

# Export Terraform state to files?
export_state = true

# Terraform Cloud/Enterprise does not return the VCS provider name so we use the value below instead.
vcs_provider = "vcs-provider"

# The name of the entity containing the repository.
# The value should be empty for GitHub.com, the user/organization for GitHub (custom application),
# the project for Bitbucket, and the namespace for Gitlab.
vcs_namespace = "vcs_namespace"

# When the branch for the stack is the repository's default branch,
# the value is empty so we use the value provided below instead
vcs_default_branch = "main"

After you have completed the changes, you can copy the file to the exporters/tfc directory, and rename it to terraform.tfvars to avoid having to pass the variable file to the Terraform commands that you are going to use.

Go to the exports/tfc directory and run the following commands:

$ terraform init
$ terraform apply -auto-approve

The apply command will generate an out directory containing:

 .

├── data.json

└── state-files

    ├── stack1.tfstate

    └── stackn.tfstate

In the state-files directory, you will have all the state files downloaded for all of your workspaces. In the data.json file, you will have the Terraform Cloud Workspaces information exported as Spacelift Stacks. 

Once they are exported, the state files can be imported into Spacelift or any other backend supported by Terraform. 

4. Generate Spacelift configuration

When you have done this, go to the generator directory. The generator.tftpl file is the template file that will generate the configuration for the Spacelift Stacks and Environment Variables. You don’t need to make any changes here, but you have the option to easily customize it to add other parameters to those resources. Just make sure you are building a valid configuration because otherwise, you will receive errors.

To generate the configuration per se, all you have to do is:

$ terraform init
$ terraform apply -auto-approve -var-file=../out/data.json

This will generate a main.tf file in the out directory, containing information related to the Stacks:

.

├── data.json

├── main.tf

└── state-files

    ├──stack1.tfstate

    └── stackn.tfstate

Mapping resources from a vendor to Spacelift resources is an inherently complex task that does not always produce a precise or definitive outcome. This is primarily due to the presence of certain discrepancies or limitations in the functionality of the resources, as well as potential challenges that may arise during the mapping process. 

In light of this, it is imperative to exercise due diligence with the generated Terraform code to ensure that it aligns with your expectations and requirements. If the code fails to meet these standards, you may need to either attempt the mapping process again with an alternative configuration or make necessary modifications to the Terraform code itself.

After you finish the changes related to Terraform code, you should create a VCS repository containing the main.tf file from the out directory. Your Spacelift account should have access to the VCS provider in which the Terraform Cloud provider repositories were initially created.

5. Create the stacks

To create the stacks, you will need to copy the manager-stack.example.tfvars file to the manager-stack directory and rename it terraform.tfvars to avoid having to pass it to the Terraform commands.

You will need to make changes to this file to reflect your Spacelift environment:

# Name of the manager stack
stack_name = "Spacelift Manager"

# Description for the stack
stack_description = "Spacelift resources manager"

# Name of the repository to associate with the stack
repository = "spacelift"

# Name of the branch to associate with the stack
branch = "main"

# Path to the folder containing the Terraform code, in case of a monorepo
project_root = ""

# Import the Terraform state for the managed stacks into Spacelift?
import_state = false

# Spacelift API endpoint
spacelift_api_key_endpoint = "https://example.app.spacelift.io/"

# Spacelift API key ID - Alternatively, you could pass that value via the SPACELIFT_API_KEY_ID env var
spacelift_api_key_id = ""

# Spacelift API key secret - Alternatively, you could pass that value via the SPACELIFT_API_KEY_SECRET env var
spacelift_api_key_secret = ""

If you don’t already have a Spacelift API key, you can go to Settings → API keys and generate one:

spacelift migration tool account settings

After you click Add Key, a file will be downloaded that contains the API key secret. The API Key ID can be found in the previous screen.

The keys can be passed directly to the tfvars file, which is not recommended, or you can load them from environment variables.

Now that you have completed the changes, all you have to do is go to the manager-stack directory and run the following:

$ terraform init
$ terraform apply -auto-approve

After the apply finishes, you can go to the Spacelift console and see the management stack in action. This stack will be responsible for the lifecycle of all the other stacks that are migrated from Terraform Cloud Workspaces. After this stack finishes running, all of your Terraform Cloud Workspaces will be migrated.

6. Post-migration cleanup

Before you can use Spacelift to manage your infrastructure, you may need to make changes to the Terraform code for your infrastructure, depending on how the Terraform state is managed.

If the Terraform state is managed by Spacelift, perform the following actions. Otherwise, you can skip this section:

  • Remove any backend/cloud block from the Terraform code that manages your infrastructure to avoid a conflict with Spacelift’s backend.
  • Delete the import_state_file arguments from the main.tf file in the Terraform code repository you have created.
  • After the manager stack has successfully run, the mounted Terraform state files are not needed anymore and can be deleted by setting the import_state argument to false in the terraform.tfvars file from the manager-stack directory and running terraform apply -auto-approve

Starting fresh

You can delete all the changes that you’ve made by running:

$ rm -rf ./out ./{exporters/tfc,generator,manager-stack}/.terraform ./{exporters/tfc,generator,manager-stack}/.terraform.lock.hcl ./{exporters/tfc,generator,manager-stack}/terraform.tfstate ./{exporters/tfc,generator,manager-stack}/terraform.tfstate.backup ./{exporters/tfc,generator,manager-stack}/terraform.tfvars

This will help you start a new migration.

Known limitations

Right now, this migration kit works only for Terraform Cloud and Terraform Enterprise. 

For Terraform Cloud and Enterprise, the following limitations have been identified:

  • The variable sets are not exposed, so they cannot be listed and exported.
  • The name of the Version Control System (VCS) provider for a stack is not returned, so it has to be set in the exporter configuration file.
  • When the branch for the stack is the repository default branch, the value is empty. You can set the value for the default branch in the exporter configuration file or edit the generated Terraform code.

Migrate from Terraform Cloud to Spacelift - Video

Thumbnail image for video on migrating from terraform cloud to spacelift

Key points

This automation makes it easier to migrate from one vendor to Spacelift by getting the existing configuration and state and mapping it to Spacelift resources. We know that Terraform Cloud’s new pricing model is causing problems, so we really wanted to build something that will make it quicker for you to migrate to a product that offers more for less.

To learn more about the advantages of migrating from Terraform Cloud to Spacelift, check out our Terraform Cloud alternative page.

This is not a one-size-fits-all solution. You will probably need to tweak it to adapt it to your requirements. Nonetheless, it saves considerable time and makes the overall process leaner.

Terraform Management Made Easy

Spacelift effectively manages Terraform state and more complex workflows, and it supports policy as code, programmatic configuration, context sharing, drift detection, and resource visualization, as well as many other features.

Start free trial