Terraform on GCP : Google Cloud Tutorial

Using Terraform on Google Cloud (GCP) provides a robust and efficient way to manage your cloud infrastructure. With Terraform’s declarative configuration language, you can define your GCP resources as code, allowing for version control, collaboration, and automated provisioning. Whether you’re deploying virtual machines, managing Kubernetes clusters, or setting up networking components, Terraform simplifies the process, making it an invaluable tool for any GCP user.

In this post, we’ll look at:

1. What is the Google Cloud Provider?
2. How does Terraform create Google Cloud resources in the background?
3. How to integrate Terraform with GCP?

Google Cloud infrastructure consists of physical assets, like computers and HDDs, and virtual resources, such as VMs or cloud functions contained in data centers around the world. Each of their locations is in a region. Regions are available in America, Australia, Europe, Africa, Asia, the Middle East, and North and South America. Each region is a group of zones isolated from each other. Each zone is identified by a name that combines the letter identifier with the region’s name. For example, zone a in the East Asia region is named asia-east1-a.

What’s neat about GCP is their focus on AI and machine learning tools, which are integrated into many of their services. So, you could easily train a machine learning model using TensorFlow and then deploy it at scale using GCP’s infrastructure. It’s all about scalability and performance, and they have a global network of data centers to make sure your applications run smoothly worldwide. Think of it as Google’s powerhouse for all things cloud computing.

As with any other cloud provider, you write Terraform configuration files that specify what resources you need, like virtual machines, databases, or storage buckets, and how you want them configured. Terraform has a GCP provider and many others, so you can interact with all of Google’s cloud services directly through Terraform. You can also create terraform modules of the GCP resources to reuse the same code and avoid redundancy.

In the backend, Terraform uses APIs to call GCP resources. For example, by writing Terraform code, you could set up a complete network environment with VPCs, subnets, and firewall rules. This is super handy because it makes your infrastructure reproducible and version-controlled. It also gives you the desired state configuration, which would have been impossible with their cloud console or shell scripts.

Terraform & GCP integration

The diagram above shows that:

1.  The DevOps/Infra Engineer creates the manifest files in Terraform for the GCP resources.
2. Terraform CLI reads these files, sets up the Google Cloud provider and the required provisioners, and updates or creates the Terraform state file.
3. Finally, it sends a request to the Google Cloud API to create the infrastructure in GCP according to the configuration(.tf files) and updates the state accordingly.

What is Terraform Google Cloud provider?

The Terraform Google Cloud provider is a plugin that enables Terraform to interact with GCP services. It translates your Terraform configurations into API calls to GCP, allowing Terraform to provision resources in GCP and manage your infrastructure-as-code (IaC).

When setting up your infrastructure on Google Cloud using Terraform, you’ll start by configuring the GCP provider. This is the entry point of your configuration files since without authentication, GCP won’t know if you can access it. Here’s how you typically do it:

Now, let’s break down the above configuration:

1. The project is your GCP project ID, which is where your resources live.
2. The region specifies which geographical region these resources are located in.
3. The credentials point to your service account key file, which Terraform uses to authenticate with GCP.

How does Terraform create Google Cloud resources?

Creating resources in GCP with Terraform is a straightforward process that involves several steps happening behind the scenes:

1. Manifest files (.tf) — You define your desired infrastructure in .tf files describing the resources and their configurations. This is where you lay down the “what” of your infrastructure.
2. Initialization — Terraform uses the necessary plugins (Terraform providers) to interact with GCP. It also sets it up if the code mentions a remote backend configuration, such as Google Cloud Storage. 
3. Plan — Terraform reads the current state from the .tfstate file to understand the existing infrastructure. It compares the desired state (from your .tf files) with the current state (from the .tfstate file). This difference between the two states tells Terraform what needs to be created, updated, or deleted and generates an execution plan.
4. Apply — Terraform executes the plan using the Google Cloud provider plugin to send requests to GCP’s REST APIs. Terraform updates the state file to reflect the new actual state as each resource is created or modified.
5. State management — Terraform maintains a state file (.tfstate) to track the current state of your infrastructure. The state file contains the IDs of the resources created in GCP, which Terraform needs to know which resources to manage.

Now that we’ve outlined the theoretical steps, let’s dive into a practical example to create a Google Cloud Storage. To integrate Terraform with GCP and start creating resources, the first step is to create a Service Account in GCP with the required permissions.

Prerequisites

Before we begin working on GCP, ensure that the Terraform CLI is installed. If you haven’t done this yet, follow this Terraform installation guide to install it. You will also need a Google Cloud account.

1. Create a Service Account in GCP

In the Service Accounts section of GCP, create a new Service Account and assign a role to it with the required permissions:

On the right-most side of your Service account, click on the ‘Actions’ menu and select ‘Manage keys’:

In the ‘ADD KEY’ dropdown, select ‘Create new key’ and select the JSON in the next pop-up as shown in the below image:

2. Configure the Google provider

Now, create a new Terraform project on your local machine and add a main.tf Terraform configuration file and the below Google provider plugin:

3. Define your Google Storage Bucket

Next, add the configuration for the Google Cloud Storage Bucket you want to create in your main.tf. Provide the bucket a unique name in the name parameter and pass its location in the location parameter. Add output variable bucket_name to verify the resource creation. Here’s a simple example:

4. Initialize and apply your configuration

Apply the following commands to execute the above code:

1. terraform init
2. terraform plan
3. terraform apply

Run terraform destroy if you want to delete the created resources using terraform apply command.

Finally, verify the creation of the resources in the Google Cloud console.

By now, you must have experienced Terraform’s power and flexibility when managing infrastructure on GCP. It allows you to define resources declaratively and manage them efficiently. However, managing complex Terraform configurations locally across multiple environments can be quite challenging.

When using Terraform locally, you quickly run into several issues that can make managing infrastructure a real headache. For starters, it’s tough to maintain consistency across different team members’ environments. What works on one engineer’s laptop might not work on another’s due to version mismatches or local config differences. There’s also the challenge of tracking and reviewing changes. Without a centralized CI/CD pipeline, it’s easy for infrastructure updates to slip through without proper review, leading to potential outages or misconfigurations. Plus, managing state files locally is a nightmare. You risk state file corruption or loss, which can wreak havoc on your infrastructure.

When managing infrastructure with Terraform, it’s crucial to implement robust CI/CD pipelines to handle code changes efficiently. In real-world scenarios, running Terraform commands locally is not practical. Instead, using a governed pipeline with CI/CD ensures consistent, secure, and automated deployments. This is where Spacelift comes in, enhancing your Terraform workflows by integrating with your source code repository and automating the entire process.

Spacelift is a sophisticated CI/CD platform for infrastructure-as-code, supporting Terraform, Ansible, OpenTofu, Terragrunt, Pulumi, AWS CloudFormation, AWS CDK, Kubernetes, and more. It is designed and implemented by long-time DevOps engineers based on previous experience with large-scale installations – dozens of teams, hundreds of people, and tens of thousands of cloud assets.

Also, Spacelift is super easy to get started with, you can fully managing your cloud infrastructure within less than a minutes, with no prerequisites. It integrates well with the large organizations in the industry – notably GitHub and AWS. 

The diagram above shows that:

1. The DevOps/Infra Engineer creates the manifest files in Terraform for the GCP resources and pushes the code to an SCM, in this case, GitHub.
2. Spacelift identifies the change in the repository on code push and pulls the manifest files.
3. Spacelift runs the complete CI/CD pipeline on the files, applies all the environment variables, and provides all the pipeline logs.
4. Finally, Spacelift deploys all the changes to the infrastructure in the cloud provider, which is GCP in this case.

Now that you understand Spacelift, its benefits, and how it works, let’s set up a Spacelift pipeline for your infrastructure deployment. This process will involve integrating your Terraform code into Spacelift to efficiently automate and manage your infrastructure.

The first step involves integrating a Version Control System (VCS) like GitHub with Spacelift. This integration automates the detection of code changes and triggers the CI/CD pipeline whenever updates occur, ensuring seamless and consistent deployments.

1. Set up GitHub as the source code in Spacelift

Login to the Spacelift platform, and in the Source code section, select ‘GitHub’ in the ‘Set up integration’ dropdown as shown in the below image:

Set VCS via the wizard and give the app a unique name in GitHub. After getting redirected back to the Spacelift platform, provide an integration name, select the integration type as ‘Default’ and click ‘Set up’:

The integration should now be visible in the Source code. Click on ‘Install the app’ in the integration:

Install the app in your GitHub account and select the repositories that have to be integrated with Spacelift:

This integration ensures that Spacelift automatically detects every change pushed to your GitHub repository, triggering the appropriate CI/CD pipelines for infrastructure deployment.

After setting up GitHub as the source code, the next step is to create a new stack in Spacelift. A stack in Spacelift represents a unit of infrastructure code and its state, allowing you to manage and deploy your infrastructure seamlessly.

2. Creating a new stack in Spacelift

Return to the Spacelift platform, and in the Stacks section, click on ‘Create stack’:

In the next panel, add a unique name to the stack and click ‘Continue’:

Set up the configuration of your repository and click on ‘Continue’:

Set up the remaining stack configurations according to your requirements. This setup is crucial because it defines the infrastructure deployment process and ensures that all changes are managed consistently and securely.

The next step is setting up the environment variables in Spacelift. Environment variables are key for securely storing sensitive data like API keys, credentials, and other config settings needed for your deployment. This way, you avoid hard-coding sensitive info directly into your infrastructure files, which boosts security and flexibility.

Environment variables also help define different environments, such as production, pre-production, or development. For example, you might use environment variables to scale resources based on the environment: spin up multiple high-powered EC2 instances for production but just one low-powered instance for development. This approach makes your infrastructure more dynamic and adaptable to different needs.

3. Setting up environment variables in Spacelift

In the ‘Environment’ tab of the stack, click edit to add a new variable. Set the variable name to GOOGLE_CREDENTIALS and paste the content of the GCP Credentials JSON you downloaded. Remember to keep the variable secret.

Now, trigger the pipeline if it does not happen automatically. In its run history, you can also check the Google authentication, output values and all the other CI/CD operational logs of the pipeline as shown in the below image.

Now, all you need to do is sit back and relax as Spacelift effortlessly handles the CI/CD for your infrastructure. After the pipeline is finished, just verify the resources in your GCP account to ensure everything is deployed correctly.

Once your infrastructure is up and running, maintaining its integrity over time is crucial. This is where drift detection comes into play. In the infrastructure-as-code (IaC) world, drifts are an inevitable headache. A drift occurs when there’s a discrepancy between the desired state of your infrastructure (as defined in your IaC tool) and the actual state of your resources. This mismatch can be caused by manual changes, scripts, or even automated processes that bypass your version-controlled configurations.

Identifying drifts in GCP using Spacelift’s drift detection

Suppose you’re using Terraform to manage your GCP infrastructure. Your Terraform scripts define that a VM instance should have specific network settings. However, during an on-call firefighting session, a DevOps engineer manually changes the VM’s subnet to fix a connectivity issue quickly. Now, the actual state of the VM differs from the configuration declared in Terraform. This discrepancy is what we call a drift.

Identifying and correcting drifts involves a lot of manual checks and balances, which is time-consuming and error-prone. This is where Spacelift’s drift detection capabilities come into play.

Spacelift’s drift detection

Spacelift provides a powerful drift detection feature that continuously monitors your infrastructure to detect any deviations from the desired state defined in your IaC tool. Here’s how it works and why it’s invaluable:

1. Continuous monitoring — Spacelift automatically scans your GCP infrastructure to compare the actual state of resources with your Terraform configurations. This continuous monitoring ensures that any unauthorized changes are promptly detected.
2. Alerting — When Spacelift detects a drift, it can send alerts to your preferred communication channels (like Slack or email), allowing your team to take immediate action. This proactive approach helps mitigate potential issues before they escalate into larger problems.
3. Automated correction — Spacelift can automatically attempt to correct the drift by reapplying the desired state depending on your setup. This feature is very useful in dynamic environments where manual intervention is impractical.

In this guide, we’ve explored how to effectively use Terraform with Google Cloud Platform (GCP) and integrate Spacelift to streamline your CI/CD workflows. Here’s a recap of what we covered:

1. We introduced Google Cloud Platform (GCP) and highlighted its extensive range of cloud services.
2. We discussed Terraform as a powerful infrastructure-as-code tool for defining and managing GCP resources.
3. We walked through the process of setting up Terraform with GCP, including configuring the provider and deploying resources like Google Cloud Storage.
4. We covered how Spacelift enhances Terraform workflows by integrating seamlessly with version control systems like GitHub, automating the CI/CD process, and managing complex infrastructure configurations.
5. We explored Spacelift’s drift detection to identify discrepancies between your Terraform configurations and actual GCP infrastructure, ensuring consistency and compliance.

By combining Terraform with GCP and utilizing Spacelift for your CI/CD pipeline, you can achieve a robust and scalable infrastructure management setup. This approach not only simplifies deployment but also enhances control and visibility over your cloud resources.

Frequently Asked Questions

Q: What is Terraform in GCP?

Terraform is an Infrastructure-as-Code (IaC) tool developed by HashiCorp that allows users to define and provision infrastructure using a high-level configuration language called HCL (HashiCorp Configuration Language). When used with Google Cloud Platform (GCP), Terraform enables users to manage and automate the deployment of various GCP resources, such as virtual machines, storage buckets, networking components, and more, through declarative configuration files.

Q: How does Terraform authenticate to GCP?

Terraform authenticates to Google Cloud Platform (GCP) using a service account key file. This file contains credentials that Terraform uses to authenticate to GCP.

Example Terraform configuration file:

Q: What is Terraform used for?

Terraform automates the provisioning, management, and scaling of infrastructure resources across multiple cloud providers by defining Infrastructure-as-Code.

Q: Is Terraform a cloud platform?

No, Terraform is not a cloud platform. Using Terraform, users can automate cloud resources’ creation, modification, and destruction, ensuring that infrastructure is defined and managed consistently and predictably. However, Terraform itself is not a cloud platform; instead, it interacts with cloud platforms to perform its tasks.