Managing Terraform with GitHub Actions & How to Scale

In this blog post, we will explore managing infrastructure with Terraform and GitHub actions. Terraform has become the standard nowadays in managing Infrastructure as Code, while GitHub Actions is a continuous integration and delivery (CI/CD) platform integrated into GitHub. 

Let’s see how we can combine them to orchestrate our infrastructure workflows and what are the benefits and caveats of this approach. If you aren’t familiar with Terraform, look at the content-rich Terraform section of Spacelift’s blog. 

Terraform is an open-source Infrastructure-as-Code software tool that allows us to safely and predictably manage infrastructure at scale. It has been widely adopted by organizations and IT professionals over the years and is recognized as one of the most influential tools in the space. Its cloud-agnostic characteristics, infrastructure as code principles, modularity concepts, and automation capabilities make it a powerful tool that facilitates infrastructure management in any environment. 

Terraform keeps and manages an internal state of its managed infrastructure and is used to create plans, track changes, and enable safe modifications in live environments. One of the reasons that led to Terraform’s success has been the intuitive and easy-to-understand workflow. 

The core Terraform workflow consists of three concrete stages. First, we generate the Infrastructure as Code configuration files representing our environment’s desired state. Next, we check the output of the generated plan based on our newly edited manifests. After carefully reviewing the changes, we apply the plan to provision the infrastructure changes and resources.

GitHub Actions is a modern CI/CD tool integrated natively on GitHub. It provides the possibility to quickly automate build, test, deployment, and other custom workflows on GitHub without needing additional external tools. 

Its focus is to provide an easy and seamless way to automate every software workflow right from GitHub while providing and abstracting all the necessary infrastructure pieces. The tool’s architecture heavily depends on events that trigger further actions combined to generate custom user-defined workflows.

GitHub Actions has evolved into a mature tool in the CI/CD ecosystem over the years and provides many customization options and a robust workflow engine if your team is already using GitHub. There are options for customizing runners or bringing your own VMs, support for workflows across different environments, operating systems, versions, and programming languages.

A great benefit of using GitHub actions is the open-source community-powered workflows available on GitHub to get you up and running quickly.

Check out the official GitHub Action docs and the Learn GitHub Actions section for more information and examples.

To create an automated infrastructure management pipeline with GitOps principles, we can combine GitHub, GitHub Actions, and Terraform. The first step would be to store our Terraform code on GitHub. Then, we configure a dedicated GitHub Actions workflow based on our needs that handles infrastructure changes by updating the Terraform configuration files. 

Let’s check an example of such a setup. We will set up a GitHub repository with a simple Terraform file that deploys an EC2 instance on AWS. You can find the demo contents here.

This post won’t discuss how Terraform code repositories should be structured, as this is a separate topic. Also, for this simplistic Terraform example, I ignore many best practices, such as using variables, modules, separating environments, etc., to focus on the GitHub Actions workflow.

Check out these articles if you are looking for inspiration for a production-grade Terraform setup or configuring Terraform in automation.

Our Terraform file main.tf contains provider configuration, backend state configuration with an S3 bucket, and a simple ec2 instance. Nice and simple for our demo’s needs. 

main.tf

Let’s move into the more interesting part: the GitHub Actions workflow definition. To define workflows to run on GitHub Actions runners based on events, create a YAML file inside the .github/workflows directory of the repository. 

For our example, we define a .github/workflows/terraform.yaml file.

.github/workflows/terraform.yml 

Let’s inspect the above GitHub Actions workflow definition file from top to bottom. 

First, we define the triggers. This is achieved by setting different options for the on keyword. Here, we want to trigger the pipeline if there is a push or pull_request targeting the main branch and any changes on the path terraform/**.

Next, we define global environment variables that can be used at any pipeline step with the keyword env. These are the AWS credentials to deploy (not a best practice for production, instead, use short-lived credentials and assume an AWS role), the Terraform logs verbosity, and the bucket name that contains our Terraform state for this setup.

Notice that we use the secrets functionality of GitHub repositories to store and fetch these credentials, and we don’t commit them to the repository. 

If you wish to follow along, you can create these repository secrets from the Settings of your code repository by selecting New repository secret. 

Continuing with decomposing the GitHub Actions file, we use the keyword jobs to define our single job terraform with the keyword jobs. We set the type of machine to run the job on with runs-on and the default shell and working-directory with defaults.

A job is composed of a sequence of steps. Each step is a task that runs in its own process in the GitHub Actions runner environment.

For our terraform job, we set up steps to checkout the repository to the runner and set up a specific version of Terraform. Then, we perform Terraform-related tasks as separate steps. We initialize our Terraform environment according to the backend configuration, check for correct formatting, and if the configuration is valid.

In case of a Pull Request, we generate a plan of the incoming changes and add the output of this plan to the Pull Request as a comment. This way, we ensure our engineers have visibility of the effects of their changes without leaving GitHub.

Finally, if we are happy with the changes and the Pull Request has been approved, we merge the changes to the `main` branch, which triggers the terraform apply command to perform the infrastructure changes.

Our example includes only a few of the many existing options and features for configuring workflows with GitHub Actions. Look at the official Workflow Syntax docs for more options and ideas of what is possible.

Setting a custom CI/CD pipeline with GitHub Actions that suits your needs to manage Terraform’s lifecycle and infrastructure provisioning is an excellent method for enforcing best practices, applying infrastructure changes predictably and safely, and eliminating the need for human intervention.

Although setting up pipelines with a CI/CD tool like GitHub Actions for managing infrastructure with Terraform is a great and one of the most common approaches, this method requires substantial engineering efforts and a lot of maintenance. 

Firstly, running Terraform in CI/CD pipelines requires teams to take into account considerations for working with distributed systems with stateless components. Teams must take care of the state separately and consider using mechanics like version control and state locking to avoid disasters and race conditions.

Scaling and securing infrastructure management with Terraform and CI/CD pipelines in big organizations with multiple environments and teams isn’t trivial. Custom guardrails, policies, and controls must be integrated and set up in workflows. Handling dependencies and passing values between Terraform environments and workspaces require manual effort and possible extra tools. Gaining visibility on configuration drift requires additional tooling and processes.

To support an organization’s growing infrastructure needs, teams must invest time in continuously improving such setups and assume the overhead of maintaining custom-made solutions in the long term.

Read more about managing Terraform at scale in this article: 5 ways to manage Terraform at scale.

Compared to building a custom and production-grade infrastructure management pipeline with a CI/CD tool like GitHub Actions, adopting a collaborative infrastructure delivery tool like Spacelift feels a bit like cheating. 

Many of the custom tools and features your team would need to build and integrate into a CI/CD pipeline already exist within the ecosystem of Spacelift, making the whole infrastructure delivery journey a lot easier and smoother. It provides a flexible and robust workflow and a native GitOps experience. Configuration drift is detected and, if desired, reconciled automatically. Spacelift runners are Docker containers that allow any type of customizability. 

Security, guardrails, and policies are a vital part of Spacelift’s offering for governing infrastructure changes and ensuring compliance. Spacelift’s built-in functionality for developing custom modules allows teams to adopt testing early into each module’s development lifecycle. Dependencies between projects and deployments can be handled with Trigger Policies.

This blog post delved into the intricacies of managing infrastructure with Terraform and GitHub Actions. We quickly looked into both tools’ main functionalities and concepts and went over a demo of combining them to build a pipeline that follows infrastructure as code principles. Lastly, we discussed nuances and considerations for setting up a CI/CD pipeline for Terraform and considered a more robust alternative solution, Spacelift.