Using Terraform Registry – Providers, Modules & Publishing

Terraform Registry is the one of the most important parts of any Terraform workflow. Simply put – without Terraform Registry, it is not possible to develop and use any configuration without accessing it.

This is not a limitation. Rather, there are many critical aspects associated with Terraform registry, which we will explore in this post.

Whenever we think of provisioning infrastructure using Terraform, we think of components like the configuration files, VCS, the Terraform host, the remote backend, etc. A typical Terraform workflow is represented in the diagram below.

However, we almost never mention Terraform Registry in this interpretation.

The existence of the registry is always assumed, and it is true. Unlike the remote backends, and VCS repositories, there are no manual steps involved in configuring the registry in our project (unless the registry is private) and yet the projects cannot run without the registry.

Terraform is built using a modular approach. When Terraform is installed for the first time on any system, the core, that is responsible for running all the core commands offered by Terraform is installed. However, this does not mean that any config created to provision infrastructure on any cloud provider platform will work. The cloud provider modules still need to be installed for any configuration to work.

Referring to the diagram below, if the Terraform configuration contains IaC for AWS, Azure, and GCP, then corresponding provider modules will be installed before these configurations are provisioned using the CLI or any CI/CD automation.

These modules are installed during the initialization (init) phase of any project. A freshly written or cloned Terraform configuration always needs to be initialized. The terraform init first identifies the provider modules and versions being used to make this config work, and then also downloads and installs them from the Terraform registry.

This is a one time activity. Every subsequent execution of Terraform CLI commands – as far as new providers are not being introduced in the config – will not require initialization. If the new providers are introduced, Terraform forces reinitialization, which installs the additional provider modules.

If we remember any provider configuration we have written, we always mention the required_providers block within the terraform block. Conventionally, we find this block in the provider.tf config file.

In the example below, we can see that the rest of the configuration depends on only one AWS provider. We also specify the version of the corresponding AWS provider, which instructs Terraform to download appropriate source code.

Till now, we have known that the Terraform Registry is a repository of all the cloud provider modules. However, the Terraform Registry is also used for other purposes, as discussed in this section.

Providers

As discussed earlier, the Terraform Registry is a repository of cloud provider modules. We are able to find the providers for all the major cloud providers, as well as any other cloud providers who wish to support their infrastructure provisioning using Terraform.

The providers are classified into three different tiers.

1. Official – Providers that are developed, maintained and owned by Hashicorp. Examples: AWS, Azure, GCP, Kubernetes, etc.
2. Partner – Providers that are developed, maintained, owned, and published by other organizations on the Terraform Registry. These provider modules have undergone a thorough onboarding process and are actively supported by Hashicorp.
3. Community – These providers are developed, maintained, owned, and contributed to the Terraform registry by individual contributors. The contributions done may be public or private.

One of the purposes of Terraform Registry is to provide documentation and support to developers who are using these provider modules in their projects. Thus, every provider documentation contains in-depth information regarding how that module and the related resources are supposed to be used. This includes the required and non-mandatory parameters.

As mentioned earlier, publishing your own Terraform provider on the Terraform registry for others to use is possible. We will shed some light on this later in this blog post.

Modules

We use modules in Terraform configurations mainly to follow the DRY principle – avoid writing the same configuration multiple times or promoting reuse of Terraform configurations. Modules are also a way to share the infrastructure designs using Terraform registry.

In any Terraform project, the configuration files included in the subdirectories are not processed by the Terraform execution workflow. The sub-directories are treated as individual modules and are supposed to be imported into one of the config files in the project root directory.

The Terraform registry also acts as the repository of reusable modules. It is possible to publicly or privately publish a module.

Similar to provider configurations, modules are also declared in Terraform configuration by specifying its source and version.

In the example below, we are trying to use the AWS VPC module with version 3.19.0. When we use this module in such a way, it readily helps us provision a VPC with default values (CIDR ranges, etc.). Thus, by reusing the AWS VPC config, we are able to save quite some development bandwidth.

When we declare a “module” block, Terraform understands the same and downloads the corresponding module source code (Terraform configuration) in a sub directory of the project root. Please keep in mind that the modules may, in turn, contain submodules.

Along with hosting these modules to support initializations in Terraform workflows, Terraform also provides documentation around these modules. There is a specific way to pass and retrieve input and output variables when working with modules. This information is made available in the documentation.

Organizations may choose to standardize their infrastructure provisioning process with appropriate security controls ingrained in the approved modules.

We have discussed using the already published providers and modules in our Terraform configurations. The discussion was mainly geared from a developer’s perspective.

It should also be noted that, just like consuming these modules, it is also possible to publish them to Terraform registry so that others – within the team or globally – are able to use these providers and modules.

We need to have a Terraform Cloud account to publish any type of resource to the Terraform registry. Additionally, there are standards, specific procedures, and conventions in place that enable us to do this.

Publishing a provider

Organizations wishing to extend their products’ support using Terraform are required to create and publish a provider plugin on the Terraform registry. By doing this, the plugin is made available for developers all over the internet to start building products on the organization’s platform.

Typically, the organizations which offer cloud infrastructure services, orchestration, containerization, CI/CD, monitoring, automation, etc., require these plugins to extend the accessibility of their products via IaC.

Terraform provider plugins are developed using Terraform’s Go SDK. The SDK is used to interpret the given Terraform configuration, and perform appropriate provisioning actions (CRUD operations), via the API interface exposed by the organizations.

Once the plugin is published, it can be used by the developers in the same way they do it for other providers. Each provider block specifies the required_plugin block, which gives information about the right provider plugin to download. 

Developing a provider requires us to follow the Terraform Plugin Framework. This framework is a package of provider servers, various abstractions like providers and resources, data sources, schemas, etc. The process of developing a provider also involves embedding bits of documentation throughout the development process, which are then rendered appropriately on the registry when published.

Publishing modules

Creating custom Terraform modules and publishing them on the Terraform registry is also possible. As discussed before, modules contain reusable Terraform configuration files, which reduce the repetition of development efforts. Modules are used to encapsulate infrastructure provisioning standard practices and policies.

The entire discussion around accessing the provider plugins, modules, Sentinel policies, and run tasks till now have assumed the fact that these resources are available publicly on the Terraform registry. It should be noted that to successfully develop a Terraform configuration using these resources, these resources have to be public. Otherwise, if the access to the respective provider plugin is lost, the workflow cannot progress.

However, organizations often have a need to protect the proprietary providers, modules, policies from public access. Irrespective of the reasons, the Terraform Cloud does make it possible to publish resources privately. This way, organizations can have a control over which teams can access these resources internally.

The Terraform private registry is different from the public registry. This private registry is made available to the organizations who have Terraform Cloud subscription. Private resources are generally published and accessed from this private registry.

The way to use the privately published modules differs slightly as compared to public modules when using them in the configuration. This is attributed to the Terraform Cloud API, which explains how the source attribute needs to be specified with reference to the organization.

To go a step further in privacy, if we want to take full control, then Terraform Enterprise offers a self-hosted installation of Terraform Cloud that also includes a private registry. Although the registry is private, it is still possible to make the publicly available plugins and modules available on the private registry as well.

For those who use Spacelift to manage infrastructure using Terraform IaC, Spacelift provides a Module Registry to all the accounts belonging to all the tiers. This inhouse registry is better integrated with Stacks – the way infrastructure is managed in Spacelift.

Publishing or creating a module in Spacelift is similar to the way Stacks are created by pointing to the tracked branch of the VCP where module configurations exist. Spacelift’s module registry goes beyond the traditional approach offered by Terraform registry with features like:

1. Testing – it is possible to set up automated test cases for the modules being published. Tests are performed by executing the plan-apply-destroy cycle with a given set of inputs in the form of variables. Multiple ordered test cases offer greater code coverage.
2. Environment and context – modules have their environment and contexts. However, it is also possible to pass values from the environment of the associated stacks in module’s test cases.
3. Versioning – updating modules in Spacelift’s Module Registry follows semantic versioning scheme. New releases are well aligned with the common VCS, where it is completely flexible to follow the branching strategy of our choice.
4. Single repo, multiple modules – By setting the correct project_root directory, it is possible to manage multiple modules using a single remote VCS repository.
5. Automatic triggering of module consumers – since the module registry is inbuilt, Spacelift keeps track of new releases/updates on the modules. If a module being used is updated, Spacelift automatically triggers the run for the stack consuming it.

Spacelift’s Module Registry has much more to offer. This blog explains the in depth usage of Module Registry.

Terraform registry is an integral part of any Terraform workflow, given the plugin-oriented design. In this post, we have seen its importance and have taken a look at various types of resources which are available to be accessed. 

The Terraform registry is well integrated with Terraform Cloud. Apart from the publicly available provider plugins and modules, it is also possible to publish proprietary modules privately. Terraform Enterprise takes the privacy to the next step of self-hosting the Terraform Cloud on premise.

The core of the communication between various entities like the Terraform host, Cloud, registry, etc., is based on the Registry API. Any installation that is capable of understanding the registry API is able to bi-directionally communicate with other Terraform hosts. This is an important piece that requires understanding especially in the case of Enterprise installation.