What Is Terraform Observability and How to Implement It?

Terraform is a software product like any other. You use it to provision infrastructure that other applications run on. Like any other application or system that you run you want to make sure that Terraform works as intended and that it does what it is supposed to do. If not, you want to know why.

In this blog post, we will explore what Terraform observability is, how it works, and why you should care about it. Note that everything covered in this blog post is also applicable to OpenTofu.

What we’ll cover:

1. What is Terraform observability?
2. How to implement Terraform observability
3. Best practices for Terraform observability

There are two ways of looking at Terraform observability:

1. Observability of the Terraform process itself. This involves a number of components, e.g.:
   • the behavior of the Terraform binary,
   • the source code repository where you store your Terraform code,
   • the environment where Terraform executes,
   • the Terraform state backend,
   • the infrastructure that Terraform provisions,
2. Using Terraform to provision the required infrastructure to enable observability of other systems and applications.

Both of these are important in modern infrastructure management.

The second category is broad, and it involves instrumenting the various infrastructure components that you provision, configuring log, metrics, and trace collection, setting up dashboards, and providing developers with access to these observability tools.

The exact details of how you do this are highly dependent on your context, and it is outside the scope of this blog post to cover this. In the rest of this blog post, we will focus on Terraform observability, where the Terraform environment itself is the center of attention.

What is the difference between Terraform monitoring and Terraform observability?

Observability is sometimes confused with monitoring. These are related, but different, concepts.

Monitoring is the process of collecting, aggregating, visualizing and alerting on data describing a process or system. This data comes in the form of metrics, logs, traces, etc.

A few examples of metrics that are of interest in a Terraform environment are:

• The execution time in seconds it takes to run plan and apply operations.
• The number of resources for which drift has been detected.
• The number of planned changes in a plan operation.
• The number of parallel Terraform operations taking place on your automation platform.
• The current CPU and memory utilization on the self-managed Terraform execution agents.

Metrics by themselves do not necessarily say if something is good or bad; they just report numerical values of something you are monitoring. For instance, if your apply operation takes 300 seconds to complete, is that good or bad? It’s difficult to say from this single value, but by monitoring this metric over time you can start to understand what range of values are expected.

Logs are text-based messages collected from various components of your infrastructure. In a typical Terraform environment you would collect logs from the Terraform binary, your worker machines, your VCS repository, and any other system that is involved in the process.

This leads us to observability. Observability is the ability to understand the inner workings of a system as a whole using the metrics, logs, traces, and other signals the system emits. Observability allows us to understand why something occurred when we are diagnosing a problem.

The three pillars of observability applied to Terraform

Observability is usually split into three signal types: metrics, logs, and traces. All three apply to Terraform, but not equally well.

• Metrics come from two places: terraform plan -json, which emits a change_summary with counts of resources to add, change, remove, and import, and your automation platform, which knows queue depth, worker saturation, and run durations. Both are covered below.
• Logs come from two subsystems, Terraform core and the provider plugins, controlled separately with TF_LOG_CORE and TF_LOG_PROVIDER. They are what tell you why a run behaved the way it did.
• Traces are the weakest pillar, and it is worth saying so plainly. Terraform CLI has an OTLP exporter behind OTEL_TRACES_EXPORTER=otlp, but HashiCorp marks it in the source as experimental and not a committed interface, so it can change or disappear between releases.

Terragrunt has proper support through TG_TELEMETRY_TRACE_EXPORTER, and the HCP Terraform agent exports through TFC_AGENT_OTLP_ADDRESS. For everything else, instrument the pipeline that calls Terraform rather than Terraform itself.

Why does Terraform observability matter at scale?

Terraform observability involves observing and understanding the behavior of your broader Terraform environment. This involves a number of related components:

• The version-control system platform where you host your Terraform code.
• The runner you use to run Terraform operations (e.g. managed by a third-party platform, self-managed VM or container on a Kubernetes cluster, etc).
• The Terraform binary process itself that runs on the runner.
• The Terraform state backend.
• The infrastructure that Terraform provisions. This includes drift-detection which is an ongoing concern during the full lifetime of your infrastructure.

Understanding how Terraform behaves in your environment means understanding the full environment with all related systems, from source code to a running workload. This is the goal of Terraform observability.

In the previous section, we learned about a few different components that are typically of interest in Terraform observability.

Most of these are dependent on the specific automation platform you use to run Terraform. This could be a completely self-hosted environment with virtual machines running on-premises, all the way up to a fully-managed environment with GitHub Enterprise Cloud and an automation platform such as Spacelift or HCP Terraform.

In the following sections, we will go through details of how to implement Terraform observability, without diving deep into the technical details of how this is done, since this will vary widely between environments.

1. Collect state backend logs and metrics

There are many available options for Terraform state storage. Arguably, the most popular options are:

• Using a fully-managed state backend on the automation platform you are using (e.g., Spacelift or HCP Terraform).
• A managed object storage service from a cloud provider (e.g., AWS S3, Azure blob storage, or Google Cloud Storage)

A fully-managed state backend generally gives you insights in terms of audit logs, run history with change diffs, and more. These are fully abstracted solutions that do not require any explicit configuration. Since those are handled for you, this section will focus on the managed object storage services.

If your organization provisions infrastructure on AWS, chances are you are using an AWS S3 Terraform backend. The underlying AWS S3 service offers a few metrics and logs that are important for your Terraform observability:

• Access logs tell you what entity (e.g., IAM user or role) accessed a given object in your S3 bucket, when it happened, and what actions were performed on the object.
• CloudTrail data events report similar activity for S3 objects.
• CloudWatch metrics are published that report the number of objects in a bucket, the total size of all stored objects, the number and type of requests that take place, request latency, the number of different errors appearing, and more.

Check which metrics and logs are enabled by default, and enable any non-default logs if you think they can help you with your Terraform observability.

2. Implement resource drift detection

A resource drift is when the actual state of a resource does not correspond to what is recorded in your Terraform state file for that same resource. This occurs when someone or something modifies the resource outside of your Terraform workflow.

Detecting resource drift is important if you intend for Terraform to be the source of truth for your infrastructure. If you used Terraform to provision a resource, then it is very likely that your intention is to provide future updates to the resource via Terraform and not through different tools or processes.

Automation platforms such as Spacelift and HCP Terraform often include some level of drift detection for you. But if you run Terraform in your own environment you can easily set up drift detection on your own. The general idea is to run a plan operation at regular intervals:

$ terraform plan -refresh-only -detailed-exitcode

You must add the -refresh-only flag to not report planned changes coming from new code pushes to the repository. Also, add the -detailed-exitcode flag to the command so that Terraform exits with an exit code of 2 if there are planned changes (i.e. drift).

An example of what this looks like in a GitHub Actions workflow is this:

name: Resource drift detection

on:
  workflow_dispatch: # allows manual triggers
  schedule:
    - cron: "0 6 * * *" # run daily at 06:00 UTC

jobs:
  drift-detection:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: hashicorp/setup-terraform@v4

      # authenticate to your cloud, e.g. AWS, Azure, GCP, …
      # steps omitted for brevity …

      - run: terraform init -input=false
      - id: plan
        run: terraform plan -refresh-only -detailed-exitcode -input=false
        continue-on-error: true
      - name: Fail on plan error
        if: steps.plan.outputs.exitcode == '1'
        run: |
          echo "::error::Terraform plan failed"
          exit 1
      - name: Report drift
        if: steps.plan.outputs.exitcode == '2'
        run: |
          echo "::warning::Drift detected"
          exit 1

You can easily implement a similar approach to drift detection on other platforms.

What should you do when drift is detected? If the drift is unexpected and the drifted configuration is not desired:

1. Remediate the detected drift by running an apply operation.
2. Determine the cause of the drift and implement measures to avoid it from occurring again.

If the detected drift comes from a manual configuration that should be persisted to the state file you can instead run terraform apply -refresh-only and update your Terraform configuration to match.

3. Collect logs from Terraform operations

If you run Terraform plan and apply operations with its default behavior, you will get a subset of all the output information that Terraform produces. In a happy-path scenario, this is usually more than enough information.

However, sometimes you need additional logs to understand what Terraform is doing when you are triaging an issue or trying to understand an error you encounter.

To achieve this you must set an explicit value of the environment variable named TF_LOG. You can set this to one of TRACE, DEBUG, INFO, WARN, or ERROR. The TRACE level is the most verbose level, and the ERROR level is the least verbose. An example of setting this flag for a given plan operation looks like this:

$ TF_LOG=TRACE terraform plan

You can also set it for all future Terraform commands in the current shell session:

$ export TF_LOG=TRACE
$ terraform plan
…

There is also a special value of JSON that you can set. This is the same as setting TF_LOG to TRACE, but produces logs in JSON format that are easier to parse for automation purposes.

The logs you get when running Terraform come from two sources: the Terraform core and the Terraform providers that you use in your configuration. You can handle the verbosity of each type of logs by setting TF_LOG_CORE and TF_LOG_PROVIDER to separate values:

$ export TF_LOG_CORE=TRACE
$ export TF_LOG_PROVIDER=ERROR
$ terraform plan
…

If you manage your own Terraform runners (e.g. a virtual machine or a container) you can persist the logs to a file by setting the TF_LOG_PATH environment variable to a file path:

$ export TF_LOG_PATH=/var/logs/terraform.log

Note that you must set an explicit value for TF_LOG (or TF_LOG_CORE and TF_LOG_PROVIDER) together with TF_LOG_PATH.

You may not want to have verbose logging enabled at all times, because the amount of logs can quickly get out of hand. However, you should implement an easy way to enable debug logging when required.

An important detail to notice is that TRACE and DEBUG logs may contain sensitive data from your resources, data sources and other parts of your Terraform configuration. For this reason you should only use these log levels when the situation demands it.

4. Collect metrics from Terraform operations

In the previous section, we saw how to collect logs from your Terraform operations. You could also extract metrics.

For instance, to get the number of planned new resources you can run (using the jq utility on Mac/Linux):

$ terraform plan -json | \
      jq -r 'select(.type == "change_summary") | .changes.add'

This will output the number of resources that will be added. You can replace “add” by “change”, “import” or “remove” to get those metrics as well.

Another metric you can produce is the number of resources contained in your state file:

$ terraform show -json | \
      jq '[.values.root_module | .. | .resources? // empty | .[] | select(.mode == "managed")] | length'

The syntax of this statement is out of scope of this blog post. The basic idea is that you must take care to only count managed resources, not data sources or anything else.

Push these metrics to your metrics collection system.

5. Collect logs and metrics from your automation platform

The automation platform you use to run Terraform usually has a lot to say about the health of your environment. If you use self-hosted virtual machines or Kubernetes clusters to run Terraform, you will have access to a wealth of metrics and logs describing how the underlying machines behave.

Apart from understanding the health of individual machines, you should have a collective understanding of how many workers are available, how many are busy, what the typical idle time for a worker is between runs, and more.

For managed automation platforms, you can start by reading the documentation to learn about what metrics and logs you can extract from the platform.

6. Audit provider and API activity

You interact a lot with your Terraform providers with every Terraform operation you perform. You should include monitoring of your provider platform as part of your Terraform environment. An important part of this is audit logs that describe who did what and when it happened.

The big cloud providers offer managed services that collect audit logs (e.g. CloudTrail on AWS and activity logs on Microsoft Azure).

Avoid treating audit logs as a checkbox for an audit, make sure you know how to find relevant information within them.

7. Build a Terraform dashboard

The previous sections have gone through different types of data you should collect to understand your Terraform environment. Data is no good if you do not have a way to observe it. A good entrypoint to observability is a dashboard.

Fill your dashboard with important numbers split into different panels and charts. The exact design of your dashboard depends a lot on what your Terraform environment looks like.

If you only have a few Terraform configurations (or even a single one) you could include a lot of detailed data, including logs. For a large environment, you should create sensible summaries for your full environment without adding too detailed information about any single Terraform operation.

Keep the following best practices in mind when working with Terraform observability.

Track Terraform as a workload

Track Terraform as any other workload in your environment. Monitor metrics around plan and apply durations, time spent in queues waiting for an available agent/worker, the number of resource changes applied, how many resources are managed by Terraform, success and failure rates, and more.

With an understanding of the baseline for these metrics, you can more easily determine when one or more metrics are outside of expected values. These can be clues to help you understand other observed behaviors in the Terraform environment.

Alert on actionable conditions

Terraform operations fail. The reasons for failure can be almost anything; a few examples include:

• A typo in a resource attribute.
• An expired OIDC token because a Terraform apply operation took too long to complete.
• An eventual consistency issue where a newly created resource is not yet visible to the provider’s API, causing the creation of a dependent resource to fail.

Many types of errors can be caught at an early stage using testing with the Terraform test framework and the terraform test command, code formatting with terraform fmt and validation with terraform validate. There are also many third-party tools available that help you in this regard.

When you alert on failing Terraform operations, you want the alerts to be actionable. For instance, alert on:

• A state file that is stuck in a locked state.
• A Terraform run has been stuck in a queue for longer than usual.
• You are being rate-limited by a provider API.
• A failed apply operation. In this case you have a partially applied configuration, which can be a cause of concern.

An alert should require manual intervention. If possible, you can automate the action instead and avoid alerting a human.

Monitor critical dependencies

There are two major dependencies for any Terraform configuration:

1. A state file and state backend.
2. One or more providers.

Each Terraform plan and apply operation will interact with the state file in your selected state backend. If you use a managed service such as AWS S3 for state storage you should monitor the health of the AWS region where your bucket lives, and monitor the bucket itself and each interaction that Terraform has with it.

If you see many failing Terraform operations at the same time as your primary AWS region experiences issues, you could quickly get an idea of what is going on and what to do about it.

Another important aspect to monitor is state locking. The exact mechanism of state locking varies and could include additional resources (e.g. DynamoDB for legacy state locking for AWS S3).

Likewise, the providers you use interact with the target platforms you are provisioning infrastructure on. These provider platforms can experience issues that have a negative impact on your Terraform environment. Another common issue to monitor is provider API rate limits.

Track resource drift

You should explicitly track resource drift for every resource that Terraform is managing. If you don’t, you lose much of the benefits of managing infrastructure with Terraform. Drift can also lead to unexpected behavior during your next apply operation.

If you track resource drift and alert your platform engineers when it is introduced in your environment, you have a better chance of handling it in a controlled fashion rather than encountering it in a panic during an outage.

Enable debug logging on demand

It is easy to set a value of the TF_LOG environment variable locally when running Terraform, but do you know how to do it in your current Terraform automation platform?

A good idea is to implement an easy way to set the TF_LOG environment variable when required, even in production workflows. This could be a simple true/false flag input to your automation pipeline.

Keep track of your audit trail

Operational logs explain why something happened, including provider versions, API calls, command execution order, and more. However, you should also keep track of your audit logs. These are the logs that say when something happened and who performed the action.

Make sure you can answer these types of questions using your audit logs:

• When was the AWS S3 bucket named “production-data” first provisioned, and when has it been updated since? If you do not use AWS S3 resources, replace this example by an object storage solution in your cloud environment of choice.
• When was the last time someone modified the Azure storage account named “stproduction” outside of Terraform?
• Who has updated the state file today?

Spacelift connects to and orchestrates the infrastructure tooling your teams already use: infrastructure as code, version control systems, observability tools, governance and compliance tooling, and cloud providers.

Spacelift runs CI/CD workflows for OpenTofu, Terraform, Terragrunt, Pulumi, AWS CloudFormation, AWS CDK, Kubernetes, and Ansible. It also supports observability integrations with Prometheus and Datadog, so run outcomes, timings, and account metrics land in the dashboards your team already watches.

With Spacelift, you get:

• Drift detection and remediation: Ensure the reliability of your infrastructure by detecting and remediating drift on a schedule you define. Drift runs are marked with a driftDetection field in webhooks and a drift_detection field in plan and trigger policy input, so you can require human approval before anything reconciles.
• Audit trail: Every operation that changes a Spacelift resource is logged. Read the built-in log from your account settings, and forward the same events to a webhook endpoint to keep them in your own system beyond the 30-day retention window.
• Policy as code with Open Policy Agent (OPA): Control approvals, restrict the resources teams can create, validate configuration parameters, and define how runs behave when pull requests are opened or merged. Guardrails travel with every deployment instead of living in a script someone forgot to update.
• Multi-IaC workflows: Orchestrate OpenTofu, Terraform, Terragrunt, Kubernetes, Ansible, Pulumi, CloudFormation, and other tools from one control plane. Model dependencies between workflows and share outputs across them, so you stop wiring tools together by hand.
• Governed self-service infrastructure: Use Templates to give developers a curated catalog they deploy from by filling out a form. Inputs are validated before anything runs, and publishing a version pins it to the current commit SHA for deterministic results. Blueprints remain available when you just need an independent, editable stack.
• Spacelift Intelligence: AI capabilities are built into the platform: Infra Assistant, Intent, and the Spacelift MCP server. They run on the same policy engine and audit trail as everything else, with a dedicated Intent policy governing which resource operations Intent and Infra Assistant Build mode can perform.
• Integrations with third-party tools: Connect Spacelift to the tools your teams already use, and extend governance across them. For example, you can integrate security tools into your workflows using Custom Inputs.

To learn more about Spacelift, create a free account or book a demo with one of our engineers.

Terraform observability comes in two types:

1. Observability of the Terraform execution environment (source code repository, runner VM or container, the Terraform binary, state backend, and the provisioned infrastructure, including drift detection).
2. Observability infrastructure provisioned by Terraform in order to enable observability of some other application or system.

In this blog post, we covered the first type of Terraform observability.

Enabling observability for your Terraform environment means doing one or more of the following things:

• Collect logs and metrics from your state backend.
• Implement resource drift detection.
• Collect logs and metrics from Terraform operations.
• Collect logs and metrics from your automation platform.
• Audit provider and API activity.
• Build a Terraform dashboard (or many).

Together, these provide insights into your Terraform environment and can help you understand why something happens.

Keep the following best practices in mind when working with Terraform observability:

• Track Terraform as a workload similar to how you handle observability for user-facing applications.
• Alert on actionable conditions. An alert should require manual intervention. Automate it if possible and remove the alert.
• Monitor your critical dependencies, including provider platforms and state backends.
• Track resource drift to make sure Terraform is aware and that your state matches the reality.
• Enable debug logging on demand to help triage issues when they occur. Make sure you know how to do this also for your production workloads.
• Keep track of your audit logs. Audit logs answer “when?” and “who?” questions, which can be a helpful addition in your Terraform observability.