Terraform Private Registry: Setup, Publishing & Best Practices

Terraform configurations consist of resources, data sources, variables, outputs, local values, and more. We often group resources and data sources together into reusable modules. A module is not reusable if it is not easily available for other Terraform consumers to discover and use.

Terraform has a large ecosystem of publicly available providers. Some of the most prominent are for the hyperscalers AWS, Microsoft Azure, and Google Cloud. If your organization has an internal product or API that you want to interact with using your normal Terraform workflows, you can write your own providers. To make this provider available for your internal Terraform consumers, you need to publish it somewhere.

To solve both of these issues, we can use a Terraform private registry. In a private registry, you publish internal modules and providers intended for use within your organization.

This blog post is a guide to the Terraform private registry. In the following sections, we will explore what a private registry is, the options available for hosting or using one, and best practices to follow when working with one.

1. What is a Terraform private registry?
2. How a private registry works
3. How to publish to Terraform private registries
4. Best practices for Terraform private registry

A Terraform private registry is a secure, organization-specific version of Terraform’s public registry. It serves as a central hub for sharing, versioning, and governing Terraform modules and providers within your company, rather than relying on the open, public Terraform Registry.

Private registries also provide:

• Access control: Only authorized users or teams can see or use the modules and providers.
• Version management: You can publish multiple versions and control which ones are approved for production.
• Auditability: You can track who published which version and when.
• Consistency: Everyone uses the same, vetted Terraform components, reducing drift and errors.

Most Terraform configurations that grow beyond a few resources and data sources should use Terraform modules. Terraform modules group logically connected pieces of infrastructure into a reusable package.

A module can be local, existing only as a subdirectory in the same Git repository as the root module where it is used. Using a local module means nobody else can reuse the same module without copying and pasting the same code into their own root module. This approach does not scale.

The solution is to publish modules to a central location accessible by other Terraform consumers. This central location is the Terraform private registry.

The Terraform private registry contains metadata of your internal modules and providers. The private registry does not necessarily store the source code of the modules and providers themselves; it only stores metadata for where the modules and providers are available for download and which versions of these exist.

Module source code is stored in a version control system (VCS). For a Terraform private registry, you should use your private VCS (e.g., GitHub Enterprise). Likewise, provider binaries published to your private registry should be stored in a private object or file storage solution.

What makes a registry private is where you host the registry and how you provide access to it. A private registry should only be accessible to Terraform consumers in either your organization or a subset of your organization. The private registry should be secured with strong authentication and authorization mechanisms. You can also lock down network access to the private registry.

Do you need a private registry when you can directly reference module source from your private VCS? The drawback with this approach is that it does not support version constraints. This makes it much more difficult to manage the use of these types of modules. Versioning modules in a private registry is a better solution.

The most popular alternative to using a Terraform private registry is to use the public registry.

Terraform public registry

An official public registry is available at registry.terraform.io.

The public registry is available to all Terraform configurations by default and does not require any specific credentials. If you do not specify a hostname from whichto download modules and providers, they will be downloaded through the public registry.

To illustrate this, consider the following two Terraform configurations. The first Terraform configuration specified two providers in the required_providers block, and it includes two modules (one from each of the provider ecosystems):

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }

    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

module "aws_networking" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "6.4.0"
  # other arguments omitted for brevity
}

module "azure_networking" {
  source  = "Azure/vnet/azurerm"
  version = "5.0.1"
  # other arguments omitted for brevity
}

Both of the providers (aws and azurerm), as well as both of the modules (terraform-aws-modules/vpc/aws and Azure/vnet/azurerm), are downloaded from the public registry during terraform init.

The previous Terraform configuration is equivalent to the next configuration, where the full URLs for the providers and modules are explicitly spelled out:

terraform {
  required_providers {
    aws = {
      source  = "registry.terraform.io/hashicorp/aws"
      version = "~> 6.0"
    }

    azurerm = {
      source  = "registry.terraform.io/hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

module "aws_networking" {
  source  = "registry.terraform.io/terraform-aws-modules/vpc/aws"
  version = "6.4.0"
  # other arguments omitted for brevity
}

module "azure_networking" {
  source  = "registry.terraform.io/Azure/vnet/azurerm"
  version = "5.0.1"
  # other arguments omitted for brevity
}

The Terraform public registry is a great option when your goal is to share a module or provider with the whole world — for instance, when you write open-source modules and providers.

There are three main options for how to set up a Terraform private registry:

1. Build and host the private registry yourself.
2. Use a managed private registry on your infrastructure automation platform (e.g., Spacelift or HCP Terraform).
3. Use a third-party implementation of a private registry.

In the following sections, we discuss these options.

1. Self-hosted private registry

If you run Terraform on your own private infrastructure, you likely also want to set up your own Terraform private registry. This is especially true if your Terraform environment is locked down without full outbound internet access or if your VCS system is not publicly accessible.

The full details of how to implement and host your own private registry are beyond the scope of this blog post; however, we will discuss some details below.

To build your own Terraform private registry for modules and providers, you must implement the module registry protocol and the provider registry protocol. If your registry implements both of these protocols, it will work seamlessly with the Terraform CLI.

Modules

Modules published to a private registry use the following naming convention:

<hostname>/<namespace>/<name>/<system>

The different parts of the name are explained below:

• <hostname> is the address where your private registry is located (e.g. registry.myorganization.com).
• <namespace> is typically the name of the organization publishing the module. For a private registry, the namespace could be your organization name, the name of a business unit, or the name of an internal team.
• <name> is a descriptive name that explains what the module does or what type of infrastructure abstraction it provides (e.g., vpc, eks-cluster, iam-policy, website, etc).
• <system> is the name of the main Terraform provider used in the module (e.g., aws or azurerm).

To fulfill the module registry protocol, your registry must implement the following two API endpoints:

• GET <hostname>/<namespace>/<name>/<system>/versions to list all available versions of a given module identified by <namespace>/<name>/<system>.
• GET <hostname>/<namespace>/<name>/<system>/<version>/download to download a specific version of a given module identified by <namespace>/<name>/<system>/<version>.

The Terraform CLI uses both of these endpoints when you run terraform init.

Providers

Providers published to a private registry should use the following naming convention:

<hostname>/<namespace>/<type>

The different parts of the name are explained below:

• <hostname> is the address where your private registry is hosted (e.g., registry.myorganization.com).
• <namespace> is the name of the organization publishing the provider. For a private registry, this could be the name of your organization, a business unit, or an internal team within your organization.
• <type> is the name of the API, product, or system that the provider is for.

Your Terraform private registry must implement the following two API endpoints to fulfill the provider registry protocol:

• GET <hostname>/<namespace>/<type>/versions to list the available versions of a given provider identified by <namespace>/<type>.
• GET <hostname>/<namespace>/<type>/<version>/download/<os>/<arch> to get the download URL and associated metadata for a specific build (<os>/<arch>) of a given provider version identified by <namespace>/<type>/<version>.

In addition to the above endpoints for modules and providers, you can implement other endpoints to simplify the usage of your Terraform private registry. If you plan to add a web interface to your private registry, you can implement search endpoints for both modules and providers to simplify how your users discover what is available in the registry.

You can host your private registry in any way you like as long as the required endpoints are accessible from where you run the Terraform CLI.

2. Spacelift private module registry

Your Spacelift account includes a private module registry ready to be used for your private Terraform modules.

Structure your module VCS repository according to the following example:

.
├── .spacelift
│   └── config.yaml
├── README.md
├── main.tf # (or possibly split into other .tf files)
├── outputs.tf
└── variables.tf

An important part is the config.yaml file located in a directory named .spacelift. In this file, you configure details about your module, including test cases that should be run when the module is updated. Read the documentation for examples of what you can configure in this file.

The module should also include a README.md file with a detailed description of its purpose and functionality.

Apart from config.yaml and README.md, you should follow the normal design guidelines for how to split up a module in multiple Terraform files (e.g., variables.tf, outputs.tf, etc). For small modules, you can use a main.tf file containing all resources, but for larger modules, you should split up the content of main.tf into smaller files (e.g., networking.tf, compute.tf, etc).

Once you have the repository structure in place, go to your Spacelift module registry and follow the steps to publish your module, making it available to your Terraform consumers (see the documentation for the full details).

To use a module from your Spacelift module registry, specify the hostname spacelift.io together with the <namespace>/<name>/<system> string, following the naming convention for Terraform modules.

Here is an example of referencing a private AWS VPC module located in your Spacelift module registry:

module "network" {
  source  = "spacelift.io/my-organization/vpc/aws"
  version = "1.0.0"

  cidr_block = "10.0.0.0/18"
  # other arguments omitted for brevity
}

Apart from a module registry, Spacelift also offers a provider registry. If you are building providers for your custom application or even extending the functionality of an existing provider and need a place to host and easily version it, Spacelift can help.

3. HCP Terraform (Terraform Cloud) private registry

Your organization on HCP Terraform comes with a private registry. You can use the private registry to publish modules for internal use, as well as publish custom providers.

To publish a module to the private registry on HCP Terraform, create a Git repository following the basic structure for a Terraform module:

.
├── README.md
├── main.tf # (or possibly split into other .tf files)
├── outputs.tf
└── variables.tf

Next, go to the private registry of your HCP Terraform organization, select the publish module button, and follow the steps.

To use a published module from your HCP Terraform module registry, specify the hostname app.terraform.io together with the <namespace>/<name>/<system> string, following the naming convention for Terraform modules.

Here is an example of referencing a private AWS VPC module located in your HCP Terraform module registry: 

module "network" {
  source  = "app.terraform.io/my-organization/vpc/aws"
  version = "1.0.0"

  cidr_block = "10.0.0.0/18"
}

4. Other third-party options

There are many third-party open-source alternatives to a Terraform private registry. Some options include Tapir, nrkno/terraform-registry, and terrareg. These options are outside the scope of this blog post.

Before selecting an open-source alternative, ensure the project is actively maintained. Many alternatives have not been updated for many years.

Keep the following best practices in mind when you work with a Terraform private registry for your organization, whether you are hosting your own private registry or using a hosted solution (e.g., the Spacelift module registry):

1. Minimize the use of the public registry

Reasons to use a private registry include avoiding reliance on third-party modules and reducing the risk of introducing vulnerabilities into your environment through Terraform configurations.

Therefore, you should minimize the use of the public registry for critical production infrastructure. Build your own modules and publish them in your private registry, or bring modules from the public registry into your own private registry in a controlled fashion.

For governance, you can use policy as code to validate that only your own private registry is used for all your Terraform configurations.

The downside to this approach is that your organization must manage all Terraform modules; however, this is unavoidable in certain contexts and industries.

2. Secure access to your private registry

The Terraform modules and providers you publish to your private registry should not contain any secret values, but the information within them could still be sensitive in nature. An attacker with access to your Terraform registry will understand how your cloud infrastructure is configured and may identify weaknesses to exploit.

If you are using a managed private registry (e.g., Spacelift’s), you start with a good security baseline. However, if you are self-hosting your private registry, you must take care to properly secure it.

Securing your Terraform private registry follows the same best practices as securing other critical parts of your infrastructure. Implement robust authentication and authorization mechanisms, as well as effective network security.

3. Follow proper versioning of modules and providers

Publishing modules and providers to a private registry enables versioning. As we learned earlier, a Terraform private registry is a collection of metadata for modules and providers specifying which versions exist and where they can be downloaded from.

Versioning modules and providers allows you to use version constraints in your Terraform configurations. This is not possible for modules fetched directly from a source code repository or for internal modules packaged together with root modules.

Using a private registry is no reason to take shortcuts in how you manage Terraform modules and providers.

4. Use an established naming convention for modules

Follow the recommended naming conventions for modules and providers that you publish to your private registry. It can be tempting to invent your own naming conventions, but doing this could lead to incompatibilities with the Terraform CLI during terraform init. 

If your registry implements the module registry protocol and the provider registry protocol, you are good to go.

5. Test your Terraform modules

A Terraform private registry contains modules intended for sharing with other Terraform users. This means the code you publish to the private registry should be tested to ensure it works as intended. You should handle modules in a private registry with the same care as modules in the public registry.

You can use the Terraform test framework to write tests for your modules. As part of the release pipeline for new module versions, run through the test suite and ensure each test passes.

If you use the Spacelift private module registry, you can define tests for your module in the config.yaml file. These tests differ from the Terraform test framework, but you could combine them.

6. Use a private registry for Terraform providers

The primary use case for a Terraform private registry is usually to publish internal private modules. To use your private registry to its full potential, you should also publish custom Terraform providers.

A custom Terraform provider is a provider built for specific internal use cases, such as targeting internal APIs. These providers clearly do not belong in the public registry.

For network-constrained environments, you could also republish providers from the public registry in your private registry. This gives you more control over which versions of these providers your Terraform consumers can use. This requires additional work from your platform engineers, but greatly decreases the risk of inadvertently introducing security vulnerabilities from new provider versions.

A Terraform private registry is a metadata service for Terraform modules and providers that you publish for internal use within your organization. A private registry works in the same way as the public registry as long as it implements the module registry protocol and the provider registry protocol.

There are three options for a Terraform private registry:

• Build and host the registry yourself.
• Use a managed registry (e.g., the Spacelift private registry).
• Use a third-party implementation.

The key differences when using a private registry from your Terraform configurations are:

• In general, you must authenticate to the private registry, using an authentication mechanism you have implemented or the authentication mechanism available for a managed private registry (e.g., Spacelift module registry).
• You need to provide the hostname when referencing the source of modules and providers. This is not required when using the public registry.
• You may not always have direct access to your private registry. One situation where you would not have direct access is if it is hosted in a secure network partition in your environment. Terraform must be able to reach the repository during terraform init.

To elevate your Terraform management, create a free Spacelift account today or book a demo with one of our engineers.

Note: New versions of Terraform are placed under the BUSL license, but everything created before version 1.5.x stays open-source. OpenTofu is an open-source version of Terraform that expands on Terraform’s existing concepts and offerings. It is a viable alternative to HashiCorp’s Terraform, being forked from Terraform version 1.5.6.