Modern infrastructure management revolves around “secrets” to ensure high environmental security. While most organizations are adopting Infrastructure as Code, managing secrets becomes challenging for technical teams.
In this post, we will explore multiple ways of managing the secrets with Terraform code.
We will cover:
Secrets protect sensitive information about the organization’s infrastructure and operations. This includes system passwords, encryption keys, APIs, service certificates, and other forms of confidential data. Secrets secure such information by preventing unauthorized access, data breaches, or critical security incidents.
Secrets are used in various phases of Terraform provisioning for activities like:
- Secure access to services provided by cloud platforms such as AWS, Azure, and GCP.
- Secure access to active databases that contain sensitive data, such as customer information, financial records, etc.
- Setting up authentication through API keys, OAuth tokens, and SSL certificates to allow the user access to applications.
- Setting up access to network components such as routers, switches, and firewalls.
Terraform makes use of secrets to automate infrastructure provisioning activities which are similar to the ones listed above. It should be noted that the secrets are also stored/managed in state files referenced by Terraform for its workflow.
Secrets are used to set up authentication and authorization to cloud resources. If secrets (e.g., access keys) are compromised, then the cloud resources are exposed to potential security breaches. Terraform developers must understand the nature of secrets to take adequate measures to protect them.
Access to every cloud provider platform varies. For example, both AWS and GCP provide a range of cloud services and resources to build and deploy applications in the cloud. To interact with AWS/GCP cloud platforms, we use Terraform code and provide credentials to authenticate access to the cloud APIs. For AWS, these credentials are in the form of access keys and secrets. For GCP, these credentials are included in service account key files.
Below is an example of configuring AWS credentials in Terraform code. We have used the variables aws_access_key and aws_secret_key to pass the AWS access key and secret values.
provider "aws" {
access_key = var.aws_access_key
secret_key = var.aws_secret_key
region = var.aws_region
}Similarly, the provider block for GCP looks like below. The variable gcp_credentials_file stores the value of the GCP service account key file path.
provider "google" {
credentials = file(var.gcp_credentials_file)
project = var.gcp_project_id
region = var.gcp_region
}Terraform config files and state files become vulnerable if they contain access keys and secrets in plain text. Such situations should be avoided at all costs. Here is an overview of why it’s recommended to avoid storing secrets in Terraform config and state files:
- Anyone with access to the version control system can access that secret.
- Every host associated with the version control system stores a copy of the secret. Anyone using the host server can easily find and misuse the secrets.
- Every running software gets access to read the secret written in plain text.
- It is difficult to audit who has access to the secrets and who uses them.
In short, secrets stored in plain text in the config and state files are highly discouraged due to their potential security risks.
Instead, users should consider using alternative methods to store and manage secrets in the config and state files:
- Use input variables to store secrets and then reference them in configuration files. The values are passed during runtime. However, this is not the best way to manage secrets.
- Use Terraform’s built-in capability to mask the values of any resource or variable. Marking input variables as “sensitive” redacts the secrets being output on any console.
- Use environment variables to store secret values. However, this means anyone who has access to the host, will also have access to these environment variables.
- Use external data sources to fetch secrets from external sources at runtime. For example, integrating with Vault/Secret manager applications help to securely fetch secrets during runtime.
But also, when using automation and CI/CD pipelines to run Terraform, you can use a tool such as Spacelift to manage your state for you. The Terraform state management features in Spacelift are extremely important to maintaining secure and reliable infrastructure deployments. Spacelift has now rolled out the ability to access that remote state and manipulate it as needed. You can read more about the external state access in the documentation.
By default, the terraform.tfstate file gets auto-generated in plain text after a terraform command runs. We use the remote backend to collaborate on Terraform projects along with securely storing the Terraform state file. It is possible to securely store sensitive data like access keys and secrets, along with other state information, ensuring that the sensitive data is not exposed to unauthorized parties.
Some security features to look out for in a remote backend are:
- Encryption: built-in encryption feature helps provide an additional layer of security.
- Version control: to enable tracking of changes to infrastructure and roll-back capability.
- Access control: to align with the principle of least privilege and controlling access to the information stored in state files at a very granular level.
- Backup and recovery: in case of infrastructure failure, it should be possible to quickly recover and reinstate the state files to minimize the impact on infrastructure provisioning.
Following are the general steps to configure a secure remote backend in Terraform:
- Choose a secure backend provider.
- Create a backend configuration with the details of the backend provider and storage location.
- Initialize the backend. This also migrates any pre-existing state information to the remote backend.
- Create and apply the Terraform code.
- Store the Terraform state in the remote backend.
This helps improve the security of the information managed in state files and ensures the integrity of the infrastructure.
In Terraform, we use environment variables to store and configure variables that are needed for our configuration to provision desired infrastructure components. We specifically use environment variables with the prefix TF_VAR_<variable_name> to define them as Terraform variables. During runtime, the values of the input variable (variable_name) are referenced from the environment with a corresponding variable with TF_VAR_ prefix.
Learn more in our Terraform environment variables introduction.
Let us take an example of creating an RDS database instance that needs a username and password attributes to be set. Additionally, we also need a couple of variables for Terraform to access the AWS platform – access key, secret key, and region.
We begin by creating variables for these secrets.
#Define variables for secrets
variable "username" {
type = string
}
variable "password" {
type = string
}
#Define variables for AWS access key, secret key, and region
variable "aws_access_key" {}
variable "aws_secret_key" {}
variable "aws_region" {}Next, we set the corresponding environment variables in the format – TF_VAR_variable_name, where “variable_name” is the name of the input variables we defined in the previous step.
To do the same, run the below commands in the terminal with appropriate values.
export TF_VAR_aws_access_key=<access_key_value>
export TF_VAR_aws_secret_key=<secret_key_value>
export TF_VAR_aws_region=<region>
export TF_VAR_username=<username_value>
export TF_VAR_password=<password_value>Next, we will define the aws_db_instance resource in Terraform configuration that uses these secrets.
Also, note that we have defined the provider block with corresponding input variables.
provider "aws" {
access_key = var.aws_access_key
secret_key = var.aws_secret_key
region = var.aws_region
}
# Create an AWS DB instance resource that requires secrets
resource "aws_db_instance" "mydb" {
allocated_storage = 10
db_name = "mydb"
engine = "mysql"
engine_version = "5.7"
instance_class = "db.t3.micro"
username = var.username
password = var.password
parameter_group_name = "mydb.mysql5.7"
skip_final_snapshot = true
}Here we used environment variables to store AWS keys and accessed them using TF_VAR_ variables and similar behavior for the database username and password. The above Terraform code picks up the secrets automatically. This technique does not store secrets in plain text in the code.
Terraform hosts are the virtual or physical machines that serve as the deployment targets for infrastructure code implementation. They are used to provision and manage resources on cloud providers and on-premises infrastructure, ensuring consistency and reproducibility of the infrastructure.
Securing the Terraform host is crucial because:
- As the Terraform CLI is installed in the host, it must be protected against unauthorized access.
- Stop unwanted Terraform code execution that can impact infrastructure security.
- Protect Terraform config and state files against data leakage.
- Ensure compliance requirements are met, such as HIPAA and PCI-DSS.
- Protect against insider threats who have access to the infrastructure.
Setting and securing the Terraform host manually is complex and time-consuming. Additional costs may be involved with securing the host, such as investing in security software, hiring security professionals, addressing newly identified vulnerabilities, etc. Any failure to secure the host leads to data loss, security breaches, and compliance violations.
File encryption is an effective technique to store and manage access keys and secrets. Terraform users can use this technique to encrypt sensitive information stored in the config and state files. This technique relies on:
- Encrypting the secrets.
- Storing the cipher text in a file.
- Checking that file into the version control.
The most common solution is to store the keys in a key service provided by any cloud provider like Azure Key Vault, AWS KMS, and GCP KMS.Â
These key services solve the “kick the can down the road” problem by relying on human memory: in this case, our ability to memorize a password that gives us access to our cloud provider (or perhaps we store that password in a password manager and memorize the password to that instead).
Example – AWS KMS
AWS KMS is the key management service from Amazon that encrypts sensitive data stored in terraform config and state files. To implement AWS KMS in the RDS database example discussed previously, create a file with credentials as content in key-value format.
For the sake of this example, we have given it the name creds.yml.
username: username
password: passwordCreate a KMS key (default symmetric) in AWS, and use the below command to create an encrypted file to store and check in to VCS, the credentials from creds.yml.
aws kms encrypt \
--key-id <alias>OR<arn> \
--region eu-central-1 \
--plaintext fileb://creds.yml \
--output text \
--query CiphertextBlob > creds.yml.encryptedOnce the creds.yml.encrypted is checked-in to VCS and cloned on another system, the credentials are fetched using the data store for aws_kms_secrets as shown below. Use a local variable to decrypt the value from cipher text stored in creds.yml.encrypted file.
data "aws_kms_secrets" "creds" {
secret {
name = "dbexample"
payload = file("${path.module}/creds.yml.encrypted")
}
}
locals {
db_creds = yamldecode(data.aws_kms_secrets.creds.plaintext["dbexample"])
}Once the secrets credentials are decrypted, use the same to set the database resource credentials as shown in the below configuration.
resource "aws_db_instance" "mydb" {
allocated_storage = 10
db_name = "mydb"
engine = "mysql"
engine_version = "5.7"
instance_class = "db.t3.micro"
username = local.db_creds.username
password = local.db_creds.password
parameter_group_name = "mydb.mysql5.7"
skip_final_snapshot = true
}Although this is a very secure way of managing sensitive information, the process of encrypting files has certain drawbacks:
- Every time the credentials are updated, we have to encrypt the file locally and check the same into VCS.
- We need to take added precautions while re-encrypting the file.Â
- Using file encryption adds steps in setting up and managing the encryption keys and files.
Secret stores are secure solutions to store and manage secrets and keys like the AWS Secrets Manager. It is a dedicated database designed specifically to store secrets securely and stop unauthorized access.
Here are a few popular, secret stores:
- AWS Secrets Manager
- HashiCorp Vault
- AWS Param Store
- GCP Secret Manager
Example – AWS Secrets Manager
Let us consider an example using AWS Secrets Manager. Create a secret to store our database credentials in the AWS Secrets Manager.
In our case, we have named this secret as dbcreds. It is of the type “others” and stores 2 key-value pairs namely: db_username and db_password, as shown below.
Note that we have used the same encryption key used in the previous example to encrypt these secrets in Secrets Manager.
To read and use these secrets in our Terraform configuration create below data sources.
The aws_secretsmanager_secret fetches the secrets data, but we cannot use the same to read secret values. To do the same, we have created another data source named aws_secretsmanager_secret_version to read the values using jsondecode() function.
data "aws_secretsmanager_secret" "dbcreds" {
name = "dbcreds"
}
data "aws_secretsmanager_secret_version" "secret_credentials" {
secret_id = data.aws_secretsmanager_secret.dbcreds.id
}Back into our database resource configuration, change the username and password values as shown below.
# Create an AWS DB instance resource that requires secrets
resource "aws_db_instance" "mydb" {
allocated_storage = 10
db_name = "mydb"
engine = "mysql"
engine_version = "5.7"
instance_class = "db.t3.micro"
username = jsondecode(data.aws_secretsmanager_secret_version.secret_credentials.secret_string)["db_username"]
password = jsondecode(data.aws_secretsmanager_secret_version.secret_credentials.secret_string)["db_password"]
parameter_group_name = "mydb.mysql5.7"
skip_final_snapshot = true
}One of the advantages of using secrets_manager data source in Terraform is that it automatically marks these values being read as sensitive. Thus the CLI output or state files do not expose the secrets.
The plan output below confirms the same.
...
+ multi_az = (known after apply)
+ name = (known after apply)
+ nchar_character_set_name = (known after apply)
+ network_type = (known after apply)
+ option_group_name = (known after apply)
+ parameter_group_name = "mydb.mysql5.7"
+ password = (sensitive value)
+ performance_insights_enabled = false
+ performance_insights_kms_key_id = (known after apply)
+ performance_insights_retention_period = (known after apply)
+ port = (known after apply)
+ publicly_accessible = false
+ replica_mode = (known after apply)
+ replicas = (known after apply)
+ resource_id = (known after apply)
+ skip_final_snapshot = true
+ snapshot_identifier = (known after apply)
+ status = (known after apply)
+ storage_throughput = (known after apply)
+ storage_type = (known after apply)
+ tags_all = (known after apply)
+ timezone = (known after apply)
+ username = (sensitive value)
+ vpc_security_group_ids = (known after apply)
}
Plan: 1 to add, 0 to change, 0 to destroy.As seen in the previous example, Terraform automatically masks the values which are inherently secret. Terraform “knows” which data sources or attributes are secrets as these modules and resources are defined by them.Â
However, in the case of custom resources and attribute values, where input variables are used to provide sensitive information during runtime, we need to let Terraform know about the sensitivity of those variables.
Consider the very first example where we used TF_VAR_ environment variables to set values for AWS Access and Secret keys.
The variables are defined below.
#Define variables for AWS access key, secret key, and region
variable "aws_access_key" {}
variable "aws_secret_key" {}
variable "aws_region" {}The corresponding TF_VAR_ variables were set as environment variables. In this case, Terraform does not “know” about these variables holding secret values.
If we define an output variable to output the values of these keys, it will NOT mask the same.Â
output "accesskey_value" {
value = var.aws_access_key
}
output "secret_value" {
value = var.aws_secret_key
}
This is also true when these values are stored in state files. To prevent this from happening, declare an additional attribute named “sensitive = true” to mask these values.
Updated variables as shown below.
variable "aws_access_key" {
sensitive = true
}
variable "aws_secret_key" {
sensitive = true
}
output "accesskey_value" {
value = var.aws_access_key
sensitive = true
}
output "secret_value" {
value = var.aws_secret_key
sensitive = true
}Output of the Terraform plan command after marking these variables as sensitive:
If we are using the right technique, then managing secrets in Terraform code is not at all complex. We have discussed the below approaches in this blog post.
- Not storing secrets in plain text in the config and state files
- Using a secure remote backed
- Using environment variables
- Encrypting files with KMS, PGP, or SOPS
- Using secret stores like Key Vault, and AWS Secrets Manager
- Masking variables while displaying through CLI
The above techniques are some of the ways to ensure that sensitive data is stored securely and protected from any unauthorized access. There are several methods to store and manage sensitive data. However, choose a technique that fits with the nature of the secrets and take adequate measures to protect them.
Note: New versions of Terraform will be placed under the BUSL license, but everything created before version 1.5.x stays open-source. OpenTofu is an open-source version of Terraform that will expand on Terraform’s existing concepts and offerings. It is a viable alternative to HashiCorp’s Terraform, being forked from Terraform version 1.5.6. OpenTofu retained all the features and functionalities that had made Terraform popular among developers while also introducing improvements and enhancements. OpenTofu is the future of the Terraform ecosystem, and having a truly open-source project to support all your IaC needs is the main priority.
Terraform Management Made Easy
Spacelift effectively manages Terraform state, more complex workflows, supports policy as code, programmatic configuration, context sharing, drift detection, resource visualization and includes many more features.
