Jean-Marc is a Solutions Engineer at Spacelift. His experience ranges from software development to cloud infrastructure architecture and management.

All posts by Jean-Marc Fontaine - Infrastructure as Code Expert

Jean-Marc Fontaine

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

General | Spacelift Blog

General

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Product

So your team learned about <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code (IaC)</a>? They got all excited about this way of managing infrastructure. Soon enough, you had multiple stacks described in definition files. This source code was versioned in your favorite Version Control System such as GitHub, and integrated with your favorite CI/CD platform.
Finally, the infrastructure was built from those definition files, and it was time to celebrate the end of this journey, right? Or was it the end of the journey?
What the definition files describe is the desired state of the infrastructure. The way you want it to be set up. The responsibility of your IaC tool is to turn that desired state into reality. We call the current reality of your infrastructure the actual state. Right after you ran your IaC tool, both are identical but unfortunately, they might not stay in sync for long.
Any difference between the desired state and the actual state is called drift.

It might be surprising at first that infrastructure drifts. After all, you are managing it via code, right? Unfortunately, there are different ways your infrastructure can be modified outside of your IaC process.
<h3>Manual changes</h3>
Most of the drift is usually caused by manual changes performed by individuals.
Some of those reasons are understandable. For example, during an incident, an engineer might need to increase the number of resources to handle an elevated load or make up for resources being down. During those times of high stress when a lot is at stake, manual mitigation changes are perfectly acceptable. The goal is to get to a better place as soon as possible, and the regular IaC process might take too long. Especially when you need to respond to fast-changing conditions.
This becomes a problem if, after the fire has been put out, the changes are not reverted or backported to the IaC definition files.
There are also bad reasons for manual changes. Those cannot be justified, even temporarily, and often stem from poor education on best IaC practices, loose access permissions, and a lack of proper communication regarding the infrastructure management process.
<h3>Overlapping/conflicting IaC code</h3>
In some cases, humans are not directly at fault. Resources may end up being managed by multiple sets of IaC definition files. Applying some definition files might revert changes made by other definition files.
This can happen when the IaC practices evolve over time. For example, it is not uncommon to switch to a different IaC tool after a few years because the team realized that, in hindsight, they had not picked the best tool for their use cases.
Another reason for this is overlapping boundaries between stack definitions which can happen easily when the infrastructure is extensive and managed by many different teams over the course of a long period of time.

IaC is all about improving the governance of your infrastructure by defining it as code which allows you to leverage a wealth of practices and tools that have been available to developers for a long time such as code versioning, code reviews, static analysis, automated tests, etc.
Letting drift occur undermines that effort and provides a false sense of governance.

As we have seen earlier, the primary source of drift is manual changes. Those are often linked to loose access control practices. The Principle of least privilege recommends granting only the necessary permissions. The fewer people can manually modify the infrastructure, the better. Admin-level access is typically limited to senior infrastructure engineers and SREs.
Because you will always need to have some people with the permissions to perform manual changes to your infrastructure, you need to make sure that they are aware of the process to either revert or backport those changes in due time.
That being said, even with the most trained engineers and the best intentions, drift will happen. It is inevitable, so you need to make sure that you can easily and quickly detect it and possibly revert it.

As we said earlier, drift is the difference between the desired state of the infrastructure as defined in the IaC source code and the actual state of the infrastructure.
Said differently, drift is getting a non-empty list of proposed changes when running the plan command for your IaC tool.
Here is how to display the proposed changes with the main IaC tools.
<h3>Terraform</h3>
Run the <code>terraform plan</code> command to get the list of proposed changes. 

In the screenshot below, we can see that the maximum number of servers in the Auto Scaling Group was set to 5 outside of Terraform which is drift.

image4-19

Auto Scaling Group outside Terraform

<h3>CloudFormation</h3>
CloudFormation has a built-in drift detection feature that can be used either via the AWS Console or via the AWS CLI command.
CloudFormation’s drift detection must be triggered manually. There is no built-in automation to make it run on a schedule. Also, <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-supported-resources.html" target="_blank" rel="nofollow noopener">not all resource types can detect drift at this time</a>.
Drift detection checks can be run via the AWS Console:

image14-2

Drift detection checks via the AWS Console

image6-11

AWS Console Overview

image1-25

AWS Console Drifts

image10-7

AWS Console Drift details

Or with on the command line with AWS CLI:

image8-10

Drift detection on the command line with AWS CLI

image9-7

describe-steck-drift-detection-status

image2-24

describe-stack-resource-drifts

<ul>
<li><a href="https://spacelift.io/blog/devops-best-practices" target="_blank" rel="noopener">16 DevOps Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Automate Terraform Deployments and Infrastructure Provisioning&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
</ul>

<h3>Pulumi</h3>
Run the <code>pulumi preview --refresh --stack &lt;STACK NAME&gt;</code> command to get the list of proposed changes.
The screenshot below shows that the tags and user data of the AWS EC2 instance have been modified manually.

image11-8

pulumi preview --refresh --stack <STACK NAME>

<h3>Spacelift</h3>
Drift can occur at any time. As a result, drift detection must be run on a regular schedule to catch it as quickly as possible which is not practical when running those commands on one’s laptop.
A better approach would be to use a tool such as Spacelift that can check for drift automatically on a schedule that you set.

image12-7

Spacelift drift detection

The view that shows all the resources for a stack uses an eye-catching icon for resources that have drifted so that they can be easily spotted.

image3-21

resources that have drifted

Another benefit of using Spacelift is that the drift detection management experience is consistent across the supported IaC tools. Under the hood, different commands will be run but for the most part, the workflow and the screens will be identical.

You probably want to get rid of most drift which we will explain in a bit, but there might be manual changes that should make their way to the definition files.
For example, you had to increase the number of resources during an elevated load episode, but realistically, this is not a one-off but the new normal. Then, you should not eliminate the drift but update the IaC definition files to reflect your new expectations.

Since drift is having a non-empty list of proposed changes when the definition files have not changed, fixing that drift is applying the proposed changes. That will restore the infrastructure to its desired state.
Here is how to remove the drift and get back to the desired infrastructure state with the main IaC tools.
<h3>Terraform</h3>
Run the <code>terraform apply</code> command to revert the external changes and remove the drift.

image15-2

terraform apply

<h3>AWS CloudFormation</h3>
CloudFormation can revert drift in some cases only.
For example, if a resource is missing it will be recreated but if a property of a resource was modified it might not be detected by CloudFormation and as a result, it won&#8217;t be fixed automatically.
If CloudFormation cannot automatically fix the detected drift, you can use the information provided to manually revert the unexpected changes.
<h3>Pulumi</h3>
Run the <code>pulumi up --stack &lt;STACK NAME&gt;</code> command to revert the external changes and remove the drift.

image5-17

pulumi up --stack <STACK NAME>

image13-6

Spacelift Edit Drift Detection

When drift is detected, Spacelift can optionally revert the changes found by following the same workflow that is used for regular IaC code changes, enforcing all the configured guardrails such as <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy" target="_blank" rel="noopener">automated validation of the plan</a> and <a href="https://docs.spacelift.io/concepts/policy/approval-policy" target="_blank" rel="noopener">approval workflow</a>.

image7-10

Spacelift Drift Detected

Like incidents, drift is inevitable and part of the life of any infrastructure. It must be taken into account when defining the processes and selecting your tools so that you do not get caught off guard and stay on top of things regarding your infrastructure governance.

Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation and controls to simplify and accelerate the provisioning of cloud based infrastructures.

How Spacelift Can Improve Your Infrastructure as Code

5 Ways to Manage Terraform at Scale - Best Practices

Terraform Drift - How to Detect and Manage It

What is drift and why does it happen? See how Terraform, Cloudformation, Pulumi, or Spacelift, can help you with drift detection and remediation.

Jean-Marc Fontaine

Infrastructure Drift Detection and How to Fix It With IaC Tools

Get the newsletter

No recent posts