Terraform

Managing Terraform State – Best Practices & Examples

How to manage Terraform state

In this article, we will look at how to manage Terraform State. First, we will cover what Terraform state is and why it is required before looking at some best practices for storing, organizing, and isolating your state files. We will then move on to look at referencing the remote state utilizing a data source, and finally, how to use the terraform state command to manipulate the contents of the state file.

What is Terraform State?

Terraform logs information about the resources it has created in a state file. This enables Terraform to know which resources are under its control and when to update and destroy them. The terraform state file, by default, is named terraform.tfstate and is held in the same directory where Terraform is run. It is created after running terraform apply. The actual content of this file is a JSON formatted mapping of the resources defined in the configuration and those that exist in your infrastructure. When Terraform is run, it can then use this mapping to compare infrastructure to the code and make any adjustments as necessary.

Storing State Files

State files, by default, are stored in the local directory where Terraform is run. If you are using Terraform to test or for a personal project, this is fine (so long as your state file is secure and backed up!). However, when working on Terraform projects in a team, this becomes a problem as multiple people will need to access the state file.

Also, when using automation and CI/CD pipelines to run Terraform, the state file needs to be accessible, and permission given to the service principal running the pipeline to be able to access the storage account container that holds the state file. This makes shared storage a perfect candidate to hold the state file. Permissions can be granted as needed. Azure Storage accounts or Amazon S3 buckets are an ideal choice. You can also use a tool such as Spacelift to manage your state for you.

terraform {
  backend "azurerm" {
    resource_group_name  = "terraform-rg"
    storage_account_name = "terraformsa"
    container_name       = "terraformstate"
    key                  = "terraform.tfstate"
  }
}

Isolating and Organising State Files

State files should be isolated to reduce the “blast radius”. Usually, projects are structured in a single folder and use a single state file for all resources. This immediately introduces risk, as a mistake in the configuration could change the state file and cause unwanted consequences to all your resources.

A better way is to use multiple state files for parts of your infrastructure. Logically separating resources from each other and giving them their own state file in the backend means that changes to one resource will not affect the other. Different state files for different environments are also a good idea.

terraformstate
--development
  --webapp.tfstate
  --sqldb.tfstate
  --vnet.tfstate
--UAT
  --webapp.tfstate
  --sqldb.tfstate
  --vnet.tfstate
--production
  --webapp.tfstate
  --sqldb.tfstate
  --vnet.tfstate

The backend configuration for the Azure Web App in the development environment:

terraform {
  backend "azurerm" {
    resource_group_name  = "terraform-rg"
    storage_account_name = "terraformsa"
    container_name       = "terraformstate"
    key                  = "development\webapp.tfstate"
  }
}
$ terraform workspace new development
Created and switched to workspace "development"! You're now on a new, empty workspace. Workspaces isolate their state, so if you run "terraform plan" Terraform will not see any existing state for this configuration.

Terraform Remote State

terraform_remote_state is a data source that can be used to fetch details from the remote state file directly. This is useful when you need to reference the outputs of configurations that are stored in different state files. When an output block is defined in your configuration, the contents are included in the state file. These details can then be referenced elsewhere in your project.

output "sqldb_id" {
  value       = azurerm_sql_database.example.id
  description = "Database ID"
}

We can now reference the resulting SQL database ID in the Web App configuration code by configuring a data block using terraform_remote_state:

data "terraform_remote_state" "dev_sqldb" {   
  backend = "azurerm"
   
  config = {     
    storage_account_name = "terraformsa"     
    container_name       = "terraformstate"     
    key                  = "development/sqldb.tfstate"   
  }
}

And then reference the name of the output where necessary in the configuration:

data.terraform_remote_state.dev_sqldb.outputs.sqldb_id

Manipulating the State File

It is sometimes necessary to directly interact with the state file, either to check its contents, remove items when they have been imported incorrectly or no longer exist in the real infrastructure, or import items that already exist to bring them under Terraform management.

To import existing items into the state file that have been created by other methods in the infrastructure to bring them under Terraform control, the terraform import command can be used. Each resource on the Terraform docs pages has an import section detailing how to use the command for a particular resource. For example, on the azurerm_sql_database docs page, it states that SQL Databases can be imported using the resource id and gives an example.

terraform import azurerm_sql_database.database1 /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myresourcegroup/providers/Microsoft.Sql/servers/myserver/databases/database1

The corresponding resource block azurerm_sql_database.database1 needs to be written in the configuration file prior to using this command, with the settings matching the real infrastructure. Importing resources can be very time-consuming, so avoiding any manual creation of resources from the get-go is normally a wise idea!

To learn more, see our step-by-step tutorial: Importing Existing Infrastructure into Terraform.

Key Points

Understanding aspects of Terraform state management and best practices is key to becoming proficient with Terraform. State files should be stored remotely as a general rule and isolated and organized in such a way that separate state files exist for logical groups of resources and environments in order to reduce the “blast radius” should any mistakes occur.

The terraform_remote_state data source can be used to reference outputs from state files. Finally the terraform state and terraform import commands can be used to manipulate the contents of the state file.

Cheers!

Manage Terraform Better and Faster

If you are struggling with Terraform automation and management, check out Spacelift. It helps you manage Terraform state, build more complex workflows, and adds several must-have capabilities for end-to-end infrastructure management.

Start free trial