Multi-Cloud Provisioning and Management with Terraform

Multi-cloud provisioning is the practice of provisioning cloud resources to multiple cloud providers. This can be from the same cloud infrastructure project (e.g., Terraform configuration) or separate projects.

This blog post focuses primarily on use cases where multiple cloud providers (e.g., AWS, Azure, Google Cloud) are involved. However, multi-provider provisioning can also be discussed in a more general sense.

Apart from AWS, Azure, and Google Cloud, multiple cloud platforms could mean your VCS system (e.g., GitHub, GitLab), your private on-premises data centers, your identity-management solutions (e.g., Microsoft Entra ID), Kubernetes clusters, etc.

The workflow for targeting multiple providers using Terraform is the same, regardless of which providers you are working with.

Terraform is a provider-agnostic tool for infrastructure as code. This is in contrast to provider-specific tools such as AWS CloudFormation, Azure Bicep, or Google Cloud Deployment Manager.

Terraform achieves this through the separation of concerns into two main components:

• Terraform core: The Terraform binary you download and install to run Terraform commands
• Terraform providers: Individual binaries that are plugins to the Terraform core. Each provider knows how to work with a target API (e.g., a cloud provider like AWS)

Without this separation, the Terraform binary would be huge.

This separation of concern allows you to use a single workflow (through Terraform core) to target multiple providers, even though the details of each provider’s work are different.

Working on a Terraform configuration for multi-cloud deployments is not that different from a Terraform configuration targeting a single cloud provider. However, some details are worth discussing.

In the following example, we want to provision cloud infrastructure across the three large cloud providers:

• Amazon Web Services (AWS)
• Microsoft Azure
• Google Cloud (GCP)

To keep the example at a reasonable level of complexity, we will provision a blob storage solution in each target cloud. The following figure illustrates this:

One thing we notice right away from the figure above is that we need to be aware of the differences between the three cloud providers. Each provider has their own resource hierarchies and target scopes.

• For AWS, you target an AWS account as the provisioning scope. The blob storage solution on AWS is S3. You create one or more S3 buckets and store blobs in them.
• For Azure, you target an Azure subscription as the provisioning scope. All resources on Azure must be placed in a resource group, so you will need to provision this as well. The blob storage solution on Azure is a storage account. In the storage account, you create one or more containers where you can store blobs.
• For Google Cloud, you target a Google Cloud project as the provisioning scope. The blob storage solution on GCP is Google Cloud Storage (GCS). You create one or more GCS buckets to store blobs.

An important point to solidify here is that even though Terraform is cloud agnostic by itself, each provider works differently. There is no simple way to migrate from one provider (e.g., AWS) to another provider (e.g., GCP).

Step 1: Specify required Terraform providers

A Terraform configuration can use any number of providers. As usual, you specify what providers you intend to use in your Terraform configuration.

This is done in the required_providers block inside a terraform block:

The providers we have specified here are:

• The aws provider to target Amazon Web Services. The latest version of this provider, as of this writing is 5.98.0.
• The azurerm provider to target Microsoft Azure. The “rm” part of the name is short for resource manager. The latest version of this provider, as of this writing, is 4.30.0.
• The google provider to target GCP. The latest version of this provider, as of this writing, is 6.36.1.

Step 2: Configure the Terraform providers

Multi-cloud provisioning requires the Terraform configuration to configure settings for all the target providers. This is similar to configuring a single provider; you just need to understand the difference between how your different providers are configured.

Just as each provider’s target scope and resource hierarchy differ, so do the authentication requirements and provider configurations.

In the following code samples, we will see explicit configuration of these providers.

You should not explicitly provide credentials like this in your code. The following example illustrates how each provider’s configuration is different. You can also configure provider settings using environment variables. Other means of configuring the providers are based on a preconfigured OIDC trust relationship between your IaC platform and the target provider.

The Azure provider requires you to configure the Azure subscription ID along with some information, depending on how you want to authenticate. 

In the following example, we authenticate as an Azure service principal with a client ID and client secret together with an Azure tenant ID to identify in which tenant the service principal resides:

The AWS provider requires an access key ID and a secret access key along with what region you will be targeting (note that in the coming version 6 of the AWS provider you will no longer need to configure the provider with an explicit region, and you will be able to easily provision resources across multiple regions with a single provider instance):

Finally, the Google Cloud provider is configured with the Google Cloud project ID that you will be targeting (this is not a requirement; you could configure it for each resource individually) along with a credential file:

The credential file contains further details on what identity is authenticating to Google Cloud. See the Google Cloud documentation for details.

In the code samples above, we have configured a single instance of each provider. However, it is also possible to configure multiple provider instances if your Terraform configuration requires you to target multiple scopes (e.g., multiple Azure subscriptions or multiple AWS accounts).

Each provider offers many other configuration options. See the documentation for each provider to learn more about the possibilities they offer:

• Azure provider
• AWS provider
• Google Cloud provider

Step 3: Decide on a remote state storage option

Even if you write a Terraform configuration that targets multiple cloud providers, you still need to decide on a single location to store your state file. 

There is often one cloud provider that you would designate as your primary cloud provider. This may be the cloud provider you initially started with, or the one your organization has decided to focus on. This could also mean this is the provider where you will store your Terraform state files.

If you are using an infrastructure-as-code provisioning platform (e.g., Spacelift or HCP Terraform), state storage will be managed for you. Other third-party options exist for state file storage.

To avoid adding more providers in this example, we will use Amazon S3 as our state storage.

Your Terraform configuration should include a backend block inside a terraform block similar to the following:

You might wonder why I can’t store the state in all of the configured cloud providers for redundancy. For a given Terraform configuration, you can only store the state file in one location. This is a Terraform requirement. You could set up replication to create backups of your state file in other locations, but only a single state file will be used.

If you need to separate your Terraform state in different locations, you need to use multiple Terraform configurations, where you configure each Terraform project to store the state in a specific location.

Step 4: Write a multi-cloud Terraform configuration

Now all the initial configuration is in place, and we are ready to write the Terraform configuration for the resources we need.

When working with multi-cloud provisioning, you do not have to follow any specific code structure. Follow the same principles you have in place, e.g., group related resources in specific Terraform files and extract common patterns into modules.

In this simple example, it’s advisable to work with three main Terraform files:

• aws.tf for all AWS resources.
• azure.tf for all Azure resources.
• google.tf for all GCP resources.

Assuming you follow this structure, add the following code in aws.tf to configure the S3 bucket:

The only argument provided to the S3 bucket is bucket. This argument specifies that the bucket name should be simply spacelift.

Add the following code to azure.tf to configure the resource group, storage account, and container resources:

The Azure resources are named following Microsoft recommendations. Resource groups should start with an rg- prefix, and storage accounts should have the st prefix. Note that storage account names can only include lowercase letters and numbers. Container names have no recommended standard.

Finally, add the following code to google.tf to configure the GCS bucket:

This bucket will be named spacelift and placed in the EU multi-region (this is not a single region;, it is a collection of European regions).

The names of the main resources (buckets and storage accounts) for all three cloud providers’ blog storage solutions must be globally unique. In addition, each resource and its configuration differ.

This is one of the main challenges with multi-cloud provisioning: You must understand resources across all target clouds, how resources are configured, whether there are any specific restrictions you need to take into account, what recommended practices to follow, etc. Finding engineers who are skilled in all your target cloud providers can also be difficult.

Step 5: Provision your multi-cloud infrastructure

Using Terraform for multi-cloud provisioning involves following the exact same steps as targeting a single cloud provider.

As always, the first step to provision infrastructure with Terraform is to initialize the Terraform configuration. Notice how all the required providers are installed one by one to the .terraform directory inside of your working directory:

Run through a plan operation to verify that the expected number of resources will be created (the output is truncated for brevity):

Five resources will be created: one on AWS, three on Azure, and one on Google Cloud. This is what we expect, so we can move on to the apply operation (the output is truncated for brevity):

The output shows us how Terraform is interacting with all three cloud providers in parallel.

All resources were successfully created. Notice the surprising result that buckets named spacelift did not already exist on either AWS or Google Cloud.

Multi-cloud provisioning is not the first step an organization would take when starting to use the cloud. However, it will become necessary to use multiple cloud providers at some point. 

Multi-cloud provisioning is worth pursuing for both technical and business use cases and reasons.

Technical use cases

Here are some technical use cases or reasons why you would use multi-cloud provisioning:

• Each cloud provider has their strengths compared to other cloud providers.
  • Traditionally, Google Cloud has offered excellent services for big data, machine learning, and AI. Google Cloud was also early in offering a fully managed offering for Kubernetes.
  • Azure offers great integration with Entra ID (formerly known as Azure AD) and the rest of the Microsoft product sphere. Many organizations use Entra ID or on-premises AD and can benefit from the tight integration that this offers with Azure.
  • AWS is the largest cloud provider and is a great choice for workloads that require scale.
• If your workloads are latency-sensitive, you might need to make an optimal decision about where in the world you run them. Each cloud provider has data centers in most global regions, but you may need to host a specific workload in a country where only one of the providers has a data center.
• An important factor to consider is your organization’s engineering skills. You may have many engineers who are fluent in both AWS and Azure, so utilizing both platforms would give you the combined benefits of both.
• Many organizations worry about vendor lock-in, which is the cloud equivalent of putting all your eggs in one basket. Multi-cloud provisioning avoids reliance on the fate of a single cloud provider.

Business use cases

Here are some business use cases for multi-cloud provisioning:

• If you utilize more than one of the big cloud service providers, you can start comparing the cost of running a workload in either cloud. You could even move workloads between cloud providers if one provider is cheaper than another.
• Each cloud provider may fulfill different types of regulatory compliance frameworks that may affect where you run specific workloads. Regulatory compliance reasons usually outweigh business considerations.
• Another use case for multi-cloud provisioning is keeping the lights on when a cloud provider has platform issues. You could run your main workload in one cloud provider and have a backup site waiting to take over in case the main site stops answering your health checks. This is a requirement for business-critical applications.
• If your customers compete directly with one of the large cloud providers, demands from these customers may dictate where you run workloads related to this customer. You may need to provision these workloads to a specific cloud provider while leaving your other workloads on different cloud providers.
• If your organization acquires another organization that uses a different cloud provider from yours, you just bought yourself into a multi-cloud environment. You will need to manage both cloud environments, possibly using Terraform for a unified experience.

Before we discuss the best practices for multi-cloud provisioning with Terraform, you should determine whether you need it. If you are a start-up in the early stages of developing a new product, you probably should not use multi-cloud yet. Managing multi-cloud infrastructure is hard and should not be taken on lightly.

If you do decide to use multi-cloud, keep the following best practices in mind:

• Use a robust secrets management solution and authentication mechanisms to keep your cloud environments secure. Targeting multiple clouds from the same Terraform configuration requires Terraform to have credentials for all target clouds. This makes it even more important to handle these credentials carefully, due to the potential blast radius of having them leaked. 
• Organize your code to fit your needs and skills. Just because you target multiple clouds does not mean you must provision infrastructure to all clouds from all Terraform configurations. If your teams are skilled at working in different clouds, it makes sense to split the code accordingly.
• Establish a robust FinOps organization. If you already feel that keeping track of your costs in one cloud is challenging, this problem will multiply when you use multiple clouds. You must have a working FinOps organization that keeps track of all costs across all target clouds.
• Utilize each cloud provider’s strengths. Deploy your data analytics workloads where you can process data in the best way or possibly the cheapest way. Provision your Kubernetes workloads where cluster management comes with the least overhead. Use each provider’s unique service offerings to benefit all target providers.

Terraform is really powerful, but to achieve an end-to-end secure Gitops approach, you need to use a product that can run your Terraform workflows. 

Spacelift takes cloud automation and orchestration to the next level. It is a platform designed to manage infrastructure-as-code tools such as OpenTofu, Terraform, CloudFormation, Kubernetes, Pulumi, Ansible, and Terragrunt, allowing teams to use their favorite tools without compromising functionality or efficiency.

Spacelift provides a unified interface for deploying, managing, and controlling cloud resources across various providers. Still, it is API-first, so whatever you can do in the interface, you could do via the API, the CLI it offers, or even the OpenTofu/Terraform provider.

The platform enhances collaboration among DevOps teams, streamlines workflow management, and enforces governance across all infrastructure deployments. Spacelift’s dashboard provides visibility into the state of your infrastructure, enabling real-time monitoring and decision-making, and it can also detect and remediate drift.

You can leverage your favorite VCS (GitHub/GitLab/Bitbucket/Azure DevOps), and executing multi-IaC workflows is a question of simply implementing dependencies and sharing outputs between your configurations.

With Spacelift, you get:

• Policies (based on Open Policy Agent) to control what kind of resources engineers can create, what parameters they can have, how many approvals you need for a run, what kind of task you execute, what happens when a pull request is open, and where to send your notifications
• Stack dependencies to build multi-infrastructure automation workflows with dependencies, having the ability to build a workflow that, for example, generates your EC2 instances using Terraform and combines it with Ansible to configure them
• Self-service infrastructure via Blueprints enabling your developers to do what matters – developing application code while not sacrificing control
• Integrations with any third-party tools, so you can integrate with your favorite third-party tools and even build policies for them. For example, see how to integrate security tools in your workflows using Custom Inputs.
• Creature comforts such as contexts (reusable containers for your environment variables, files, and hooks), and the ability to run arbitrary code
• Drift detection and optional remediation

If you want to learn more about Spacelift, create a free account today or book a demo with one of our engineers.

Using Terraform multi-cloud provisioning is not that different from using Terraform for a single cloud provider. Multi-cloud provisioning brings challenges and benefits, but it is not a strategy you should undertake without a business or technical use case.

In this blog post, we learned about using Terraform for multi-cloud provisioning and saw an example of what it takes to write a Terraform configuration targeting the three large cloud service providers: AWS, Azure, and GCP.

A few key takeaways from this blog post are:

• The biggest challenge when building Terraform configurations for multi-cloud architecture is understanding the differences between the providers you target. Each provider has its own target scopes, resource hierarchies, configuration options, and authentication mechanisms.
• A Terraform workflow for multi-cloud deployments follows similar steps to a workflow targeting a single cloud provider:
  • Configure the required providers (in the example, we used AWS, Azure, and GCP).
  • Configure each provider with the required arguments and authentication details.
  • Structure your Terraform configuration code following the same principles as usual, based on how resources are related.
  • Run through the Terraform core workflow using terraform init, terraform plan, and terraform apply.
• Technical use cases and requirements for provisioning a multi-cloud architecture include utilizing the strengths of service offerings from different providers and optimizing latency-sensitive workloads by utilizing cloud provider availability worldwide.
• Business use cases and requirements that lead to multi-cloud provisioning include guaranteeing the availability of a workload (meeting an SLA) and optimizing your cloud spend.

Note: New versions of Terraform are placed under the BUSL license, but everything created before version 1.5.x stays open-source. OpenTofu is an open-source version of Terraform that expands on Terraform’s existing concepts and offerings. It is a viable alternative to HashiCorp’s Terraform, being forked from Terraform version 1.5.6.