In this article, we will look at how to manage Terraform State. First, we will cover what Terraform state is and why it is required before looking at some best practices for storing, organizing, and isolating your state files. We will then move on to look at referencing the remote state utilizing a data source, and finally, how to use the terraform state command to manipulate the contents of the state file.
Terraform logs information about the resources it has created in a state file. This enables Terraform to know which resources are under its control and when to update and destroy them. The terraform state file, by default, is named terraform.tfstate and is held in the same directory where Terraform is run. It is created after running terraform apply.
The actual content of this file is a JSON formatted mapping of the resources defined in the configuration and those that exist in your infrastructure. When Terraform is run, it can then use this mapping to compare infrastructure to the code and make any adjustments as necessary.
Read more about the elements of Terraform architecture.
State files, by default, are stored in the local directory where Terraform is run. If you are using Terraform to test or for a personal project, this is fine (so long as your state file is secure and backed up!). However, when working on Terraform projects in a team, this becomes a problem as multiple people will need to access the state file.
Also, when using automation and CI/CD pipelines to run Terraform, the state file needs to be accessible, and permission given to the service principal running the pipeline to be able to access the storage account container that holds the state file. This makes shared storage a perfect candidate to hold the state file. Permissions can be granted as needed. Azure Storage accounts or Amazon S3 buckets are an ideal choice. You can also use a tool such as Spacelift to manage your state for you.
You should store your state files remotely, not on your local machine! The location of the remote state file can then be referenced using a backendblock in the terraform block (which is usually in the main.tf file).
The example below shows a configuration using a storage account in Azure:
terraform {
backend "azurerm" {
resource_group_name = "terraform-rg"
storage_account_name = "terraformsa"
container_name = "terraformstate"
key = "terraform.tfstate"
}
}It is not a good idea to store the state file in source control. This is because Terraform state files contain all data in plain text, which may contain secrets. Storing secrets anywhere other than a secure location is never a good idea and definitely should not be put into source control. To avoid this, for example, in Azure, Azure Key Vault can be referenced.
Secondly, most version control systems do not allow the locking of files, which may cause issues when multiple people attempt to access the file at the same time. Lastly, storing state files in version control can introduce human error, as the latest state file would need to be pulled each time Terraform is run. This could result in an old state file being used and unexpected changes to the infrastructure.
Using a remote backend instead solves these challenges. Locking is natively supported; the state file will be automatically loaded, and encryption can be used to secure the state file on disk and in transit. Remote backend storage such as Azure Storage or Amazon S3 is designed to be highly available. They also support versioning so you can roll back to a previous state file should it get corrupted. Lastly, it is inexpensive to use. Usually, state files are very small and can normally fit into the free tier.
State files should be isolated to reduce the “blast radius”. Usually, projects are structured in a single folder and use a single state file for all resources. This immediately introduces risk, as a mistake in the configuration could change the state file and cause unwanted consequences to all your resources.
A better way is to use multiple state files for parts of your infrastructure. Logically separating resources from each other and giving them their own state file in the backend means that changes to one resource will not affect the other. Different state files for different environments are also a good idea.
Check our guide on How to Manage Multiple Terraform Environments Efficiently.
For example, consider we have an Azure SQL Database, Azure VNET, and Azure Web App in our project. The infrastructure is the same across three environments: development, UAT (user acceptance testing), and production. Each of the three resources would have its own state file for each environment, providing a strong separation. Taking it one stage further, you may wish to use a completely separate storage account for each environment in different subscriptions.
Assuming we are holding all state files in a single storage account, the container folder structure for the remote backend could end up looking something like the example below:
terraformstate
--development
--webapp.tfstate
--sqldb.tfstate
--vnet.tfstate
--UAT
--webapp.tfstate
--sqldb.tfstate
--vnet.tfstate
--production
--webapp.tfstate
--sqldb.tfstate
--vnet.tfstate
The backend configuration for the Azure Web App in the development environment:
terraform {
backend "azurerm" {
resource_group_name = "terraform-rg"
storage_account_name = "terraformsa"
container_name = "terraformstate"
key = "development\webapp.tfstate"
}
}Making a change to the Azure Web App in development will have no effect on the Azure Web App in production.
Isolation can also be achieved using Terraform workspaces. These are less useful in production environments but are good for a quick way to test isolated environments. Using workspaces to manage actual environments is discouraged. Terraform has one workspace by default (called default!). State files are isolated to each workspace.
You can create a new workspace using the terraform workspace new command.
$ terraform workspace new development
Created and switched to workspace "development"! You're now on a new, empty workspace. Workspaces isolate their state, so if you run "terraform plan" Terraform will not see any existing state for this configuration.Useful command quick reference when working with workspaces:
terraform workspace list — list your workspaces.
terraform workspace new [workspace name] — create a new workspace.
terraform workspace select [workspace name] — switch to the specified workspace.
terraform_remote_state is a data source that can be used to fetch details from the remote state file directly. This is useful when you need to reference the outputs of configurations that are stored in different state files. When an output block is defined in your configuration, the contents are included in the state file. These details can then be referenced elsewhere in your project.
terraform_remote_state Data Source Example
For example, again, consider we have an Azure SQL Database and Azure Web App. The configuration files for these should be set up to use different state files. When creating our database that the web app needs to connect to, we add an output block to expose the resulting ID of the database after it has been created:
output "sqldb_id" {
value = azurerm_sql_database.example.id
description = "Database ID"
}We can now reference the resulting SQL database ID in the Web App configuration code by configuring a data block using terraform_remote_state:
data "terraform_remote_state" "dev_sqldb" {
backend = "azurerm"
config = {
storage_account_name = "terraformsa"
container_name = "terraformstate"
key = "development/sqldb.tfstate"
}
}And then reference the name of the output where necessary in the configuration:
data.terraform_remote_state.dev_sqldb.outputs.sqldb_idIt is sometimes necessary to directly interact with the state file, either to check its contents, remove items when they have been imported incorrectly or no longer exist in the real infrastructure, or import items that already exist to bring them under Terraform management.
The terraform state command can be used to perform advanced state management. All state management commands that modify the state create a timestamped backup of the state prior to making modifications.
Useful terraform state commands:
terraform state list— List the contents of the state file.terraform state rm— Remove an item from the state file.terraform state show— Show a resource in the state file.
To import existing items into the state file that have been created by other methods in the infrastructure to bring them under Terraform control, the terraform import command can be used. Each resource on the Terraform docs pages has an import section detailing how to use the command for a particular resource. For example, on the azurerm_sql_database docs page, it states that SQL Databases can be imported using the resource id and gives an example.
terraform import azurerm_sql_database.database1 /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myresourcegroup/providers/Microsoft.Sql/servers/myserver/databases/database1The corresponding resource block azurerm_sql_database.database1 needs to be written in the configuration file prior to using this command, with the settings matching the real infrastructure. Importing resources can be very time-consuming, so avoiding any manual creation of resources from the get-go is normally a wise idea!
To learn more, see our step-by-step tutorial: Importing Existing Infrastructure into Terraform.
Understanding aspects of Terraform state management and best practices is key to becoming proficient with Terraform. State files should be stored remotely as a general rule and isolated and organized in such a way that separate state files exist for logical groups of resources and environments in order to reduce the “blast radius” should any mistakes occur.
The terraform_remote_state data source can be used to reference outputs from state files. Finally the terraform state and terraform import commands can be used to manipulate the contents of the state file.
For more information on Terraform state, check out the Terraform docs.
Cheers!
Manage Terraform Better and Faster
If you are struggling with Terraform automation and management, check out Spacelift. It helps you manage Terraform state, build more complex workflows, and adds several must-have capabilities for end-to-end infrastructure management.
Terraform State Cheatsheet
Grab our ultimate cheat sheet PDF and keep your IaC safe while managing State.
