Importing Existing Infrastructure into Terraform - step by step

Importing Existing Infrastructure into Terraform – step by step

Importing Existing Infrastructure into Terraform – step by step

In this tutorial, we explore various ways to import pre-existing cloud resources before we continue to develop the IaC in Terraform. This scenario is often faced by teams that are starting to adopt Terraform for their operations.

Why import?

Terraform, however useful, is not a mandate for teams to use the same right from day one. The lack of human resources and considering the learning curve involved in using Terraform effectively causes teams to start using cloud infrastructure directly via their respective web consoles.

For that matter, any kind of IaC method (CloudFormation, Azure ARM templates, Pulumi, etc.) requires some training and real-time scenario handling experience. Things get especially complicated when dealing with concepts like states and remote backends.

Also, Terraform is a relatively new technology. Organizations have embarked on the journey of cloud adoption much before. Thus, this is an expected scenario when trying to adopt Terraform to manage their cloud resources.

Worst case – if you manage to lose the terraform.tfstate file. You can use the import functionality to rebuild the same in this extreme scenario. Although it is going to be a tedious process, it is possible.

Getting the pre-existing cloud resources under the Terraform management is facilitated by Terraform import. import is a Terraform CLI command which is used to read real-world infrastructure and update the state, so that future updates to the same set of infrastructure can be applied via IaC.

At this moment, the import functionality helps update the state locally and it does not create the corresponding configuration automatically. However, it is believed that the team is working hard to improve this function in upcoming releases.

Simple import

With an understanding of why we need to import cloud resources, let us begin by importing a simple resource – EC2 instance in AWS. I am assuming the Terraform installation and configuration of AWS credentials in AWS CLI is already done locally. We will not go into the details of that in this tutorial.

Step 1 - Preparation

For this tutorial, let us create an EC2 resource manually to be imported. This could be an optional step if you already have a target resource to be imported.

Go ahead and provision an EC2 instance in your AWS account. Let us note some of the details of the EC2 instance thus created. In my case they are as below:

Name: MyVM
Instance ID: i-0b9be609418aa0609
Type: t2.micro
VPC ID: vpc-1827ff72

Step 2 - Create main.tf, and set provider configuration

We aim to import this EC2 instance into our Terraform configuration. In your desired path, create `main.tf` and configure the AWS provider. The file should look like below.

// Provider configuration
terraform {
 required_providers {
   aws = {
     source  = "hashicorp/aws"
     version = "~> 3.0"
   }
 }
}
 
provider "aws" {
 region = "eu-central-1"
}

Run terraform init to initialize the Terraform modules. Below is the output of successful initialization.

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.0"...
- Installing hashicorp/aws v3.51.0...
- Installed hashicorp/aws v3.51.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Step 3 - Write config for resource to be imported

As discussed earlier, Terraform import does not generate the configuration files by itself. Thus, we need to create the corresponding configuration for EC2 instance manually. This doesn’t have to be with many arguments as we will have to add or modify them when we import the EC2 instance into our state file.

However, if you don’t like to see colorful output on CLI, you can already add all the arguments you know. But this is not a foolproof approach, because normally the infrastructure you may have to import will not be created by you. So you are ought to skip a few arguments anyway.

We will take a look at how to adjust our configuration to reflect the exact resource in just a moment. For now, append the main.tf file with EC2 config. For example, I have used the below config. The only reason I have included ami and instance_type attribute, is that they are the required arguments for aws_instance resource block.

resource "aws_instance" "myvm" {
 ami           = "unknown"
 instance_type = "unknown"
}

Step 4 - Import!

Think of it as if the cloud resource (EC2 instance) and its corresponding configuration are available in our files. All that is left to do is to map the two into our state file. We do that by running the import command as below.

terraform import aws_instance.myvm <Instance ID>

A successful output should look like below:

aws_instance.myvm: Importing from ID "i-0b9be609418aa0609"...
aws_instance.myvm: Import prepared!
  Prepared aws_instance for import
aws_instance.myvm: Refreshing state... [id=i-0b9be609418aa0609]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

The above command maps the aws_instance.myvm configuration to the EC2 instance using the ID. By mapping, we mean that the state file now “knows” the existence of EC2 instance with the given ID. The state file also contains information about each attribute of this EC2 instance, as it has fetched the same using the import command.

Step 5 - Observe state files and plan output

Please observe, that the directory now also contains terraform.tfstate file. This file was generated after the import command was successfully run. Take a moment to go through the contents of this file.

At this moment our configuration does not reflect all the attributes. Any attempt, to plan/apply this configuration will fail because we have not adjusted the values of its attributes. To close the gap in configuration files and state files, run terraform plan and observe the output.

.
.
.
          } -> (known after apply)
          ~ throughput            = 0 -> (known after apply)
          ~ volume_id             = "vol-0fa93084426be508a" -> (known after apply)
          ~ volume_size           = 8 -> (known after apply)
          ~ volume_type           = "gp2" -> (known after apply)
        }

      - timeouts {}
    }

Plan: 1 to add, 0 to change, 1 to destroy.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply"
now.

The plan indicates that it would attempt to replace the EC2 instance. But this is completely against our purpose. We could do it anyway by simply not caring about the existing resources, and create new resources using configuration.

However, the good news is that Terraform has taken note of the existence of an EC2 instance, that is associated with its state. 

Step 6 - Improve config to avoid replacement

At this point, it is important to understand that the terraform.tfstate file is an important piece of reference for Terraform. All of its future operations are performed concerning this state file. Of course, the fault is with our configuration. We need to learn from the state file and update our configuration accordingly so that there is a minimum difference between them.

The use of the word “minimum” is intentional here. Right now, we need to focus on not replacing the given EC2 instance, but rather align the configuration so that, the replacement can be avoided. Eventually, we would achieve a state of 0 difference.

Observe the plan output, and find all those attributes which cause the replacement. The plan output is friendly enough to highlight the same. In our given example, the only attribute that causes replacement is the AMI ID. Closing this gap should avoid the replacement of the EC2 instance.

Change the value of ami from “unknown” to what is highlighted in the plan output, and run terraform plan again. Observe the output this time.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_instance.myvm will be updated in-place
  ~ resource "aws_instance" "myvm" {
        id                                   = "i-0b9be609418aa0609"
      ~ instance_type                        = "t2.micro" -> "unknown"
      ~ tags                                 = {
          - "Name" = "MyVM" -> null
        }
      ~ tags_all                             = {
          - "Name" = "MyVM"
        } -> (known after apply)
        # (27 unchanged attributes hidden)






        # (6 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

This time the plan does not indicate the replacement of EC2 instance. If you got the same output, we are successful in partially importing our cloud resource. We are currently in a state of lowered risk – if we apply the configuration now, the resource will not be replaced, but a few attributes would change.

Step 7 - Improve config to avoid changes

It indicates that we need to align our resource block more if we want to achieve a state of 0 difference. The plan output highlights the attribute changes using ~ sign. It also indicates the difference in the values. For example, it has highlighted the change in the instance_type value from “t2.micro” to “unknown”.

To look at it the other way around, had the value of  instance_type been “t2.micro”, Terraform would NOT have asked for a change. Similarly, we can see there are changes to the tags highlighted as well. Let us change the configuration accordingly so that we close these gaps. The final aws_instance resource block should look like below:

resource "aws_instance" "myvm" {
 ami           = "ami-00f22f6155d6d92c5"
 instance_type = "t2.micro"
 
 tags = {
     "Name": "MyVM"
 }
}

Run terraform plan again, and observe the output.

aws_instance.myvm: Refreshing state... [id=i-0b9be609418aa0609]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

If you have the same output, then congratulations, you have successfully imported a cloud resource into your Terraform config. It is now possible to manage this configuration via Terraform directly, without any surprises.

Importing modules

In this section, we understand how to import resources into the modules. For reference, we use AWS VPC module (version 3.2.0) that creates 29 resources. Move the terraform.tfstate file to another location. By doing this, Terraform becomes unaware of the existing resources.

This poses a challenge of importing AWS VPC module into our configuration. Modules wrap multiple AWS resources into a single package that can be reused in various projects. It is okay to expect a large codebase and many resources to be part of the module.

The process of importing resources created using a module is very similar to what we have discussed until now, with a little difference in running the command. Go through the .tf files included in the module’s source, and identify the resources to be imported.

For example, AWS VPC module creates a VPC resource. To import this resource, run the command as below.

terraform import module.vpc.aws_vpc.this <VPC ID>

The above command imports the target VPC resource in AWS, to our module’s configuration. Running `plan` command indicates how many resources will be created. In other words, how many resources are yet to be imported.

      + "Name"        = "my-vpc"
          + "Terraform"   = "true"
        }
      + tags_all        = {
          + "Environment" = "dev"
          + "Name"        = "my-vpc"
          + "Terraform"   = "true"
        }
      + vpc_id          = "vpc-0127895db175d45ff"
    }

Plan: 28 to add, 0 to change, 0 to destroy.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply"
now.

The output above indicates 28 more resources to be imported to (re)build a perfectly consistent terraform.tfstate file. This is one less, as we successfully imported the VPC resource.

Referring to the plan output, identify the resources in AWS and repeat the process for import.

Approach towards complex imports

As we have seen, the process to import simple resources is pretty straightforward. The key here is to understand how Terraform state works. However, things can get tedious when importing complex deployments.

In the case of complex deployments, the team should have a clear identification of resources they want to manage using Terraform. A diagram representing the entire landscape helps to plan and get the parts of deployment under Terraform’s management.

Plan output is the key. The best hint is given by the plan output. It is well-formatted, and with the help of the same, we can make sure to build a perfectly consistent state file.

Prioritize and mitigate configurations that cause replacement of target resource followed by configurations that cause changes.

Import the resources as they are to satisfy the state file until there are no gaps. Any enhancement, code restructuring can be focused on later when we achieve 0 gaps.

Share this post

twitter logo

Comments