![How to Use Terraform For_Each Meta-Argument [Examples]](/_next/image?url=https%3A%2F%2Fsecure.gravatar.com%2Favatar%2F524354b1760c2985f5da5f72c3168080%3Fs%3D48%26d%3Dmm%26r%3Dg&w=96&q=75)
There are several occasions where we have to create multiple resources with similar configurations. It does not make sense to just copy and paste the Terraform resource blocks with minor tweaks in each block. Doing this only affects the readability and unnecessarily lengthens the IaC configuration files.
In this post, we will explore Terraform’s for_each meta arguments and address the challenge described above. Additionally, we will also take a look at other advantages and use cases along with appropriate examples.
We will cover:
- What is Terraform for each
- How to use for each in Terraform
- Example 1: Using for_each with set of strings
- Example 2: Using for_each with map
- Example 3: Using for_each with modules
- Example 4: Using for_each with list of objects
- Example 5: Using for_each with Terraform data sources
- For each vs. count
- Conditional creation with for_each
- Resource chaining
- Terraform for each indexing issues
- Benefits of using for_each meta-argument
Terraform for_each is a meta argument that helps in creating multiple instances of a defined resource. It also provides us with the flexibility of dynamically setting the attributes of each resource instance created, depending on the type of variables being used to create real-world replicas.
for_each primarily works with a set of strings (set(string)) and map of strings (map(string)). The provided string values are used to set instance specific attributes.
For example, when creating multiple subnets, we can specify different CIDR ranges for each subnet created using the same resource block.
When for_each meta-argument is used in a resource block, a special object each is automatically available to refer to each instance created by the for_each. each object is used to refer to the values provided in set, and key and value pairs provided in a map type variable.
A general syntax of using for_each meta argument is expressed below.
resource "<resource type>" "<resource name>" {
for_each = var.instances
// Other attributes
tags = {
Name = each.<value/key>
}
}The <resource type> is the type of Terraform resources, for example “aws_vpc”.
<resource name> is a user defined name of this resource, which is used to reference elsewhere in the Terraform configuration. for_each attribute is assigned a variable value in the form of “var.instances”. “var.instances” may be of list or map type.
Depending on the length of this set or map, the number of resources of “<resource type>” will be created.
Finally, the each object is used to assign a Name tag for each resource instance created by this. If the “var.instances” is of set type, then “each.value” is the only available property. Else in case of map, it is possible to retrieve both key and value by using each.key and each.value.
There are advanced use cases where it is also possible to use map of objects (map(object)) with for_each meta-argument. We will cover this later in this post.
Note: The terminal outputs presented in this blog post only display the “terraform plan” command output to prove the usage of for_each. The readers are encouraged to run the “apply” commands by themselves.
Let’s assume we have to create a specific number of EC2 instances in AWS.
The exact number of instances to be created depends on the input provided. If the input is an “array of strings” – in terms of Terraform, if the input is set(string) – then the Terraform configuration for it is as shown below.
variable "instance_set" {
type = set(string)
default = ["Instance A", "Instance B"]
}
resource "aws_instance" "by_set" {
for_each = var.instance_set
ami = "ami-0b08bfc6ff7069aff"
instance_type = "t2.micro"
tags = {
Name = each.value
}
}Here, we have declared an input variable of type set(string). The default value of this variable is a couple of strings. As far as the length of this set is concerned, it is 2, and we would expect our resource block for “aws_instance” to create two EC2 instances.
The very first attribute of aws_instance resource block is for_each which is assigned the “instance_set” variable value of type set(string). The ami and instance_type attributes are hardcoded since they are not relevant for the topic of this blog post.
Further, the Name tag uses the each object, to refer to the value of each string included in the instance_set variable.
When this Terraform configuration is executed, it creates two EC2 instances named “Instance A” and “Instance B”.
It can be verified with the plan command output below.
+ tenancy = (known after apply)
+ user_data = (known after apply)
+ user_data_base64 = (known after apply)
+ user_data_replace_on_change = false
+ vpc_security_group_ids = (known after apply)
}
Plan: 2 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.Terraform for_each with a list of strings
For occasions where instead of set, a list(string) is used – use the Terraform’s inbuilt function toset() to perform the type conversion.
The config changes for using list(string) are simple and shown below.
variable "instance_set" {
type = list(string)
default = ["Instance A", "Instance B"]
}
resource "aws_instance" "by_set" {
for_each = toset(var.instance_set)
ami = "ami-0b08bfc6ff7069aff"
instance_type = "t2.micro"
tags = {
Name = each.value
}
}The map type provides a “set” of key value pairs, offering more customization options for more complex recurring resources to be provisioned.
In this case, the number of instances to be created is equal to the length of the map object. However, along with the each.value, we can also leverage the string value stored in each.key. This is a next step where instead of one, there are two attributes to be set dynamically using the same Terraform resource block.
variable "instance_map" {
type = map(string)
default = {
"inst_a" = "Instance A",
"inst_b" = "Instance B"
}
}The instance_map variable above has a default value of two key-value pairs.
We use this variable in the for_each meta-argument to create two EC2 instances, as shown in the configuration block below.
resource "aws_instance" "by_map" {
for_each = var.instance_map
ami = "ami-0b08bfc6ff7069aff"
instance_type = "t2.micro"
tags = {
Name = each.value
ID = each.key
}
}The for_each meta-argument in the above code snippet is responsible for creating two EC2 instances.
Additionally, the Name and ID tag refer to the value and key strings set in the default value of the instance_map variable.
The plan command output confirms the same.
.
.
.
}
+ tags_all = {
+ "ID" = "inst_b"
+ "Name" = "Instance B"
}
+ tenancy = (known after apply)
+ user_data = (known after apply)
+ user_data_base64 = (known after apply)
+ user_data_replace_on_change = false
+ vpc_security_group_ids = (known after apply)
}
Plan: 2 to add, 0 to change, 0 to destroy.Terraform modules explicitly expose some input variables that are required to be supplied while using them. Modules help us package the IaC to provision a predefined set of infrastructure. Sometimes it is desired to provision the resources included in modules multiple times. This is where for_each meta-argument is also used with modules.
In the example below, we use a module from Terraform registry to create Security Groups in AWS.
This module is responsible for creating a security group and the ingress rules as a basic requirement for the Terraform security group. If we have to create multiple security groups with different ingress CIDR blocks and different names to identify them with, then we can follow the approach given below.
- Define a variable of type
map(string), where the key in the key-value pair represents the name of the security group, and the value represents the CIDR in string format. - Include the module resource block and initialize the same.
- Use
for_eachto create multiple security groups and substitutenameandingress_cidr_blocksinput with appropriate values using each object.
See the Terraform code below.
variable "sg_map" {
type = map(string)
default = {
"SG 1" = "10.10.1.0/24",
"SG 2" = "10.10.2.0/24"
}
}
module "web_server_sg" {
for_each = var.sg_map
source = "terraform-aws-modules/security-group/aws//modules/http-80"
name = each.key
vpc_id = aws_vpc.example_vpc.id
ingress_cidr_blocks = [each.value]
}When we run the terraform plan command, it proposes to create two security groups along with the ingress blocks for each of them, as shown below.
# module.web_server_sg["SG 1"].module.sg.aws_security_group.this_name_prefix[0] will be created
+ resource "aws_security_group" "this_name_prefix" {
+ arn = (known after apply)
+ description = "Security Group managed by Terraform"
+ egress = (known after apply)
+ id = (known after apply)
+ ingress = (known after apply)
+ name = (known after apply)
+ name_prefix = "SG 1-"
+ owner_id = (known after apply)
+ revoke_rules_on_delete = false
+ tags = {
+ "Name" = "SG 1"
}
+ tags_all = {
+ "Name" = "SG 1"
}
+ vpc_id = (known after apply)
+ timeouts {
+ create = "10m"
+ delete = "15m"
}
}
.
.
.
# module.web_server_sg["SG 1"].module.sg.aws_security_group_rule.ingress_rules[0] will be created
+ resource "aws_security_group_rule" "ingress_rules" {
+ cidr_blocks = [
+ "10.10.1.0/24",
]
+ description = "HTTP"
+ from_port = 80
+ id = (known after apply)
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_group_id = (known after apply)
+ self = false
+ source_security_group_id = (known after apply)
+ to_port = 80
+ type = "ingress"
}
.
.
.
# module.web_server_sg["SG 2"].module.sg.aws_security_group.this_name_prefix[0] will be created
+ resource "aws_security_group" "this_name_prefix" {
+ arn = (known after apply)
+ description = "Security Group managed by Terraform"
+ egress = (known after apply)
+ id = (known after apply)
+ ingress = (known after apply)
+ name = (known after apply)
+ name_prefix = "SG 2-"
+ owner_id = (known after apply)
+ revoke_rules_on_delete = false
+ tags = {
+ "Name" = "SG 2"
}
+ tags_all = {
+ "Name" = "SG 2"
}
+ vpc_id = (known after apply)
+ timeouts {
+ create = "10m"
+ delete = "15m"
}
}
.
.
.
# module.web_server_sg["SG 2"].module.sg.aws_security_group_rule.ingress_rules[0] will be created
+ resource "aws_security_group_rule" "ingress_rules" {
+ cidr_blocks = [
+ "10.10.2.0/24",
]
+ description = "HTTP"
+ from_port = 80
+ id = (known after apply)
+ ipv6_cidr_blocks = []
+ prefix_list_ids = []
+ protocol = "tcp"
+ security_group_id = (known after apply)
+ self = false
+ source_security_group_id = (known after apply)
+ to_port = 80
+ type = "ingress"
}
.
.
.
Plan: 9 to add, 0 to change, 0 to destroy.As a Terraform developer intending to use for_each for creating multiple resource instances, there will come a point where using map of strings (map(string)) might prove to be limiting. Especially in cases where the each object is expected to return more than two values (key and value) for more than two attributes to be tweaked in any resource.
In these cases it is possible to use map(object), where each object may have multiple properties. However, for the sake of this example we will work with list(object).
Understanding how to make for_each work with a list(object) is important when the input from a 3rd party application is not in our control.
Note that, it is much better to use map(object) instead of list(object) as it is a much cleaner way to use for_each meta-argument since it is easier to retrieve the key and value strings. We will cover an example on map(object) in the “Chaining” section further in the post.
The for_each meta-argument accepts either a set or map(string) type. Thus to make it work with list(object) it needs to be used along with a for loop which returns a string value. The for loop is used with for_each to return a specific property of the object – which evaluates to a string value.
The code below, declares a variable of type (list(object)) to create two EC2 instances.
variable "instance_object" {
type = list(object({
name = string
enabled = bool
instance_type = string
env = string
}))
default = [
{
name = "instance A"
enabled = true
instance_type = "t2.micro"
env = "dev"
},
{
name = "instance B"
enabled = false
instance_type = "t2.micro"
env = "prod"
},
]
}The instance object holds some attributes:
name– To assign a name tag to the EC2 instanceenabled– a boolean value that decides whether to provision the EC2 instance or not. More on this in the next sectioninstance_type– AWS instance typeenv– To assign environment tag to the EC2 instance
As seen here, the number of EC2 instances created from this object will differ with respect to the multiple attributes provided in the default values.
The code below makes use of for_each meta-argument to create two EC2 instances as provided in the default value of the above object.
resource "aws_instance" "by_object" {
for_each = { for inst in var.instance_object : inst.name => inst }
ami = "ami-0b08bfc6ff7069aff"
instance_type = each.value.instance_type
tags = {
Name = each.key
Env = each.value.env
}
}Here we have used the instance_type, name, and env properties provided by the instance_object variable to set the appropriate attributes of aws_instance resource using “each.value.<property>” syntax.
Note that the Terraform for loop used to set for_each meta-argument selects the name property from the instance_object input variable. Since there are two objects defined as default value, two EC2 instances will be created.
This also results in Terraform setting the “key” of the “each” object as the name property. This each.key is used in setting the Name tag. Optionally, it is also valid to use “each.value.name” to achieve the same result.
This is apparent from the plan output, as shown below.
.
.
.
+ security_groups = (known after apply)
+ source_dest_check = true
+ subnet_id = (known after apply)
+ tags = {
+ "Env" = "prod"
+ "Name" = "instance B"
}
+ tags_all = {
+ "Env" = "prod"
+ "Name" = "instance B"
}
+ tenancy = (known after apply)
+ user_data = (known after apply)
+ user_data_base64 = (known after apply)
+ user_data_replace_on_change = false
+ vpc_security_group_ids = (known after apply)
}
Plan: 2 to add, 0 to change, 0 to destroy.Assuming we want to use information related to multiple resources of a specific kind in AWS, which are provisioned and managed via a different route – perhaps a different Terraform repo? – in such cases for_each can be used.
For example, let us assume there are several EC2 instances provisioned in one of the AWS regions, and we want to use the information related to them in our Terraform configuration. The variable of type list(string) holds the list of EC2 instance ids.
variable "instance_ids" {
description = "List of EC2 instance IDs"
type = list(string)
default = ["i-1abcd234", "i-2efgh345", "i-3ijkl456", "i-4mnop567", "i-5qrst678"]
}To access the information related to these EC2 instances, we use the aws_instance Terraform data source as shown below. Here we use for_each meta-argument to read the values of each instance mentioned in the list above.
variable "instance_ids" {
description = "List of EC2 instance IDs"
type = list(string)
default = ["i-1abcd234", "i-2efgh345", "i-3ijkl456", "i-4mnop567", "i-5qrst678"]
}To access the information related to these EC2 instances, we use the aws_instance data source as shown below.
Here we use for_each meta-argument to read the values of each instance mentioned in the list above.
data "aws_instance" "ec2_instances" {
for_each = toset(var.instance_ids)
instance_id = each.value
}Later, we can use this Terraform data source to configure more resources based on these EC2 instances.
In the code below, we use this data source to print several attributes of these instances via an output variable.
output "instance_info" {
value = {
for instance_id, instance in data.aws_instance.ec2_instances :
instance_id => {
id = instance.id
public_ip = instance.public_ip
private_ip = instance.private_ip
instance_type = instance.instance_type
# more attributes as needed
}
}
}The count meta-argument also helps in creating multiple instances of a given Terraform resource.
However, there are a couple of major differences.
- Fixed vs. dynamic: A
countmeta-argument accepts an integer value and simply creates those many numbers of resources. Terraform does not retain the IDs of the instances thus created. When thecountis modified, all the resources are re-created.On the other hand, usingfor_eachprovides the flexibility of creating as many resources which are specified in the input. It does not recreate the existing instances if the input value is changed. - Resource instance attribute customization: An apparent limitation of the
countmeta-argument is that it is not possible to tweak attributes per resource instance being created.For example, when subnets are created usingcount, all of them would be assigned the same CIDR range. In this case,for_eachmeta-argument is used to provide that kind of variation.
Read more about the Terraform count meta-argument.
Continuing with the same example from the Using for_each with list of objects section, we created a property named “enabled” in our list(object) variable instance_object. When we have to conditionally create these instances, we can simply toggle the value of this attribute.
This value is then interpreted using the Terraform if condition included in the for loop as shown below.
resource "aws_instance" "by_object" {
for_each = { for inst in var.instance_object : inst.name => inst if inst.enabled }
ami = "ami-0b08bfc6ff7069aff"
instance_type = each.value.instance_type
tags = {
Name = each.key
Env = each.value.env
}
}Since we have “enabled” instance A only to be created, running the terraform plan command should propose creating a single instance as shown in the terminal output below.
.
.
.
+ public_dns = (known after apply)
+ public_ip = (known after apply)
+ secondary_private_ips = (known after apply)
+ security_groups = (known after apply)
+ source_dest_check = true
+ subnet_id = (known after apply)
+ tags = {
+ "Env" = "dev"
+ "Name" = "instance A"
}
+ tags_all = {
+ "Env" = "dev"
+ "Name" = "instance A"
}
+ tenancy = (known after apply)
+ user_data = (known after apply)
+ user_data_base64 = (known after apply)
+ user_data_replace_on_change = false
+ vpc_security_group_ids = (known after apply)
}
Plan: 1 to add, 0 to change, 0 to destroy.Read more about Terraform conditional expressions.
We have discussed and implemented the for_each meta-argument in multiple ways till now. In this section, we will see how other resources, which further depend on these individual instances created by a single resource block using for_each, are configured.
Let us understand this with an example. The hierarchy involved in implementing a VPC, then the subnets as part of VPC, and then the EC2 instances to be placed in those subnets, is understood.
As depicted in the image above, multiple subnets are created using for_each meta-arguments, and they all belong to the same VPC. Now using Terraform IaC, we would want to place an EC2 instance in each of these subnets.
To create multiple EC2 instances, we again use for_each meta-argument. This time, instead of supplying any input variable, we “chain” the other resource block (subnet) which was created using for_each meta-argument. This is where we use the each object to chain the EC2 instance resource with subnets.
The each object here represents all the attributes available as output in the subnet resource. The way to retrieve a specific attribute of a subnet is by referring to the attribute on each.value.<attribute>.
We will see the same in action below.
variable "subnet_object" {
type = map(object({
name = string,
cidr = string
}))
default = {
"subnet_a" = {
name = "Subnet A",
cidr = "10.10.1.0/24"
},
"subnet_b" = {
name = "Subnet B",
cidr = "10.10.2.0/24"
},
"subnet_c" = {
name = "Subnet C",
cidr = "10.10.3.0/24"
}
}
}Next, we create the VPC and three subnets using for_each meta-argument as we would normally do.
We have used the input variable subnet_object to define subnet attributes like CIDR block and Name tag.
# Create VPC
resource "aws_vpc" "example_vpc" {
cidr_block = "10.10.0.0/16"
enable_dns_hostnames = true
enable_dns_support = true
tags = {
Name = "Example VPC"
}
}
# Create public subnets
resource "aws_subnet" "example_subnets" {
for_each = var.subnet_object
vpc_id = aws_vpc.example_vpc.id
cidr_block = each.value.cidr
tags = {
Name = each.value.name
}
}Finally, we create EC2 instances and place them in the subnets provisioned by aws_subnet resource. The for_each meta-argument takes the aws_subnet resource block as input.
Note how the “each” subnet is associated with “each” EC2 instance that will be created.
resource "aws_instance" "by_chaining" {
for_each = aws_subnet.example_subnets
ami = "ami-0b08bfc6ff7069aff"
instance_type = "t2.micro"
subnet_id = each.value.id
tags = {
Name = "${each.value.id} instance"
}
}The number of EC2 instances to be created depends on the number of subnets being created. And the number of subnets being created depend on the input variable.
Applying this Terraform configuration should create seven resources in total, and as per the diagram above – one VPC, three Subnets, and three EC2 instances.
The plan command output looks like below.
.
.
.
+ ipv6_cidr_block_network_border_group = (known after apply)
+ main_route_table_id = (known after apply)
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "Example VPC"
}
+ tags_all = {
+ "Name" = "Example VPC"
}
}
Plan: 7 to add, 0 to change, 0 to destroy.Read more about how to build AWS VPC using Terraform.
Terraform’s for_each meta-argument is a powerful tool for dynamically managing resources, allowing us to iterate over a map or set of values.
However, it’s important to be cautious of potential indexing issues that can arise when using for_each. Unlike the traditional count approach, where resources are indexed numerically, for_each relies on unique keys. This can lead to challenges in scenarios where precise indexing is required. Instances created using for_each are referenced by their keys, making it less straightforward to perform sequential operations or handle specific ordering requirements.
Additionally, modifications to the map or set used with for_each can trigger resource re-creation, which might lead to unexpected changes in the infrastructure.
While for_each offers flexibility and scalability, it’s crucial to be mindful of these indexing nuances and to design configurations that accommodate the unique characteristics of this approach.
The Terraform for_each meta-argument offers several benefits:
- Dynamic Instance Creation: Instead of manually repeating resource or module blocks, we can use
for_eachto dynamically create instances based on the elements of a map or set. This makes our configuration more concise and maintainable, especially when dealing with a variable number of similar resources. - Enhanced Readability: Instead of duplicating resource blocks with minor differences, we can maintain a single resource block with
for_eachmeta-argument, making our configuration more readable and reducing the potential for errors due to copy-pasting. - Consistency: When creating similar instances,
for_eachhelps maintain consistency in the configuration. This is particularly important in environments where standardization is crucial. - Simplified Data Source Iteration: In addition to resources and modules,
for_eachcan also be used with data sources to dynamically iterate over data for various use cases, such as creating dynamic security groups based on an external source of IP addresses. - Managing Dynamic Workloads: For scenarios where our infrastructure’s size and shape may vary based on factors like the number of users or projects,
for_eachprovides a more scalable and adaptable approach compared to hardcoding resource counts.
In this post, we have explored various ways of using for_each meta-arguments to create multiple resource instances using the same resource block. Using input variables like set(string) and maps, we were able to tweak the differences in each resource created. We also explored advanced topics like using conditional creation and chaining to refer to the parent resource attributes created using for_each, in the resources which depend on them. Implementing for_each over count meta-argument improves the readability and makes the Terraform code more manageable.
We encourage you also to explore how Spacelift makes it easy to work with Terraform.
If you need any help managing your Terraform infrastructure, building more complex workflows based on Terraform, and managing AWS credentials per run, instead of using a static pair on your local machine, Spacelift is a fantastic tool for this. It supports Git workflows, policy as code, programmatic configuration, context sharing, drift detection, and many more great features right out of the box.
You can check it for free, by creating a trial account.
Note: New versions of Terraform will be placed under the BUSL license, but everything created before version 1.5.x stays open-source. OpenTofu is an open-source version of Terraform that will expand on Terraform’s existing concepts and offerings. It is a viable alternative to HashiCorp’s Terraform, being forked from Terraform version 1.5.6. OpenTofu retained all the features and functionalities that had made Terraform popular among developers while also introducing improvements and enhancements. OpenTofu works with your existing Terraform state file, so you won’t have any issues when you are migrating to it.
Automate Terraform Deployments with Spacelift
Automate your infrastructure provisioning, and build more complex workflows based on Terraform using policy as code, programmatic configuration, context sharing, drift detection, resource visualization, and many more.
