In the last two years teams have accelerated their dependency on the public cloud. Whether you are scaling an already existing product or building a new product that is native to the cloud, chances are you’ve felt that there has got to be a better way to build cloud infrastructure. There is. Enter Pulumi.
Pulumi is an infrastructure-as-code platform for full-stack developers and cloud engineers who are interested in using a general-purpose programming language for their cloud resources. Pulumi isn’t just one thing. It’s an ecosystem of components that all work together. At a high-level these are the components that you are most likely to hear a lot about:
- Provider SDKs
- CLI
- Service backend
SDKs are installable packages your program consumes using the respective language’s package registry that you are more than likely already familiar with and probably already work with. Today, Pulumi supports four highly popular programming languages.
- Go
- JavaScript/TypeScript
- Python
- .NET Platform (C#, F#, VB)
What that means is, you can create cloud resources using general-purpose code; think constructs such as variables, loops, functions, classes (well, at least in OOP), strong-type support, code reuse, and everything else you can do in those programming languages. It also gives you the option to maintain a homogeneous codebase, if that’s important to your team. For example, a team working on a C# .NET app may be interested in implementing their cloud infrastructure also in C# .NET. Or how about a team working on a React-based app that is strong in JS and the Node.js ecosystem wanting to use JS for their cloud infrastructure too?
Let’s examine each of those components in detail.
Provider SDKs
There are over 75 providers with new ones being added regularly. And you can contribute new ones too. All of the SDKs are available in all of the supported languages. Extension libraries are typically not supported in every one of the supported languages since they are typically manually authored and in a language based on community demand. For example, the official Pulumi EKS is a popular Pulumi component library that is supported in JS/TS and Python. But wait! The EKS library is also a great example of how you can author and use multi-language components in the Pulumi ecosystem. More on multi-language components later.
CLI
The CLI is one of the ways users can run previews and updates on their infrastructure programs. The CLI supports several powerful commands to mutate the state of your cloud resources. There is another powerful way to automate execution of your Pulumi apps.
Service Backend
The Pulumi Service backend is the default state storage service for all Pulumi projects. What this means is, every time you update your cloud resources using Pulumi, a snapshot of the state of your resources known as the checkpoint is stored by the service. The checkpoint allows you to confidently make changes to your infrastructure.
As mentioned before several components work together in the Pulumi ecosystem to mutate cloud resources. Let’s examine this with an example. Say, we want to create a Bucket using DigitalOcean Spaces. We’ll use JS as an example for demonstration purposes.
const foobar = new digitalocean.SpacesBucket("foobar", {
region: "nyc3",
});
// Export the bucket name as an output. This is a familiar concept for JS developers.
// A similar construct exists in all of the supported languages so you can export
// outputs regardless of the language you choose.
module.exports = {
bucketName: foobar.name
};Note that I’ve skipped the part about bootstrapping a new project, just like with creating any other new app Pulumi requires some minimal project setup too. This can be done easily with pulumi new which lets you pick a template to start from.
In order to create the Bucket in a DigitalOcean account, we’ll need an account and an access key for Spaces so that Pulumi can authenticate on our behalf to mutate resources. We’ll assume that a token is created and configured correctly. Now that we have authentication with DigitalOcean squared-away we can run pulumi preview from the terminal (or an IDEs integrated terminal.)
The Pulumi CLI “processed” the index.js file and generated a graph of the resources that needed to be created (or deleted, updated…or do nothing if nothing has changed.) The graph, in this case, is simple. We just have a single bucket that we want to create. Behind-the-scenes, the CLI uses this graph to calculate the set of CRUD operations using the DigitalOcean API.
Running pulumi up -y will auto-approve Pulumi to go ahead and create the bucket.
Using a general-purpose programming language means all language features are also implicitly Pulumi’s features too. Aside from that, there is a rich feature set to help with an ever-evolving cloud infrastructure landscape.
ComponentResources
ComponentResources are resources that encapsulate multiple child resources. In simple terms, think of them as a class that you can create that lets you represent several related resources as a single unit. For example, you might be interested in creating a component that handles setting up an S3 bucket with replication in a secondary region for disaster recovery scenarios. You could create a component that accepts as input the bucket name but behind-the-scenes creates two buckets – one in the primary region and the other in the failover region. Oh, but wait! There is already a component for that.
StackReferences
As your infrastructure evolves, you may be interested in creating multiple stacks with certain stacks needing to consume the outputs exported by other “upstream” stacks. This is easy to do with StackReferences. A quick example:
const docker = require(“@pulumi/docker”);
const other = new pulumi.StackReference("acmecorp/infra/database");
const dbEndpoint = other.getOutput("dbEndpoint");
// Build a container image for an API service that uses the DB endpoint.
const api = new docker.Image(“api”, {
build: {
context: “”,
env: {
“DB_ENDPOINT”: dbEndpoint
}
}
});Dynamic Providers
Although it’s limited to specific languages, it can serve as sort of an escape-hatch for when you need to mimic the CRUD lifecycle of a resource but there is no official provider for that resource. A great example of dynamic providers would be to answer the question “how do I configure a newly provisioned VM?”
Automation API
When I talked about the CLI, I said it is one of the ways to run Pulumi apps on users’ machines as well as in CI/CD pipelines. Automation API allows users to programmatically invoke the CLI operations. This unlocks a whole another set of possibilities. Pulumi’s customers have been creative with this feature and have fully exploited it to enable advanced deployment scenarios for their organizations.
Alternatives vary based on which aspects of Pulumi you would like to compare.
Other General-Purpose Programming Language Tools
There’s AWS CDK. It’s similar to Pulumi on the surface. However, AWS CDK is limited to… well, AWS. It is also fundamentally different in how it works behind-the-scenes. As a user provisioning resources on AWS you may not need to worry about the implementation details of the technology ever for your use case.
AWS CDK vs. Pulumi:
| CDK | Pulumi |
| Supports popular languages
|
Supports popular languages
|
| Transpiles to CloudFormation template
|
Doesn’t transpile to any proprietary format
|
| Cannot natively provision non-AWS resources
|
Can use any of the 70+ providers as well author your own for edge-cases using Dynamic Providers or contribute your own
|
| Can create custom encapsulations using CdkContructs that create other AWS resources or Constructs
|
Can create custom encapsulations using ComponentResources that can
|
| Has a registry showcasing community-supported as well as AWS-supported Constructs in the Construct Hub
|
Has a registry showcasing community-supported as well as Pulumi-supported components and packages in the Pulumi Registry
|
Other Infrastructure-as-Code Tools
IaC isn’t restricted to just general-purpose programming language-based tools. In fact, one of the most common IaC tool that Pulumi is often compared with is Terraform. HashiCorp’s Terraform has existed in the IaC space well before Pulumi was created. If you look deeper Pulumi extends (the official word is “bridge”, however) many of Terraform’s providers as well as has native providers that are direct implementations from Open API specs. They are conceptually similar in that both tools want you to think about everything in the cloud as a “resource”. The resource model is what drives the core of your infrastructure state. Every resource has a CRUD lifecycle.
As much as they are similar in concept, they are also very different. The fundamental difference is the choice that you, as a developer, have when it comes to choosing the language that suits you and your team best. There are other differences too, of course.
Terraform vs. Pulumi:
| Terraform | Pulumi |
| Uses proprietary but open-source language HCL which stands for HashiCorp Configuration Language
|
Supports popular open-source, general-purpose programming languages
|
| HCL is a type of domain-specific language
|
None of the supported languages are DSLs
|
| By definition, DSLs are limited in their language feature set
|
Out-of-box support for language-level features that are not custom-made by Pulumi
|
| Has a registry showcasing community-supported as well as Terraform-supported Constructs
|
Has a registry showcasing community-supported as well as Pulumi-supported components and packages
|
| No escape-hatch like functionality to create self-managed resources in cases where a provider doesn’t support a feature
|
Dynamic Provider feature to model the CRUD lifecycle of custom one-off resources
|
Check out a detailed comparison of Pulumi vs. Terraform.
OpenTofu
OpenTofu is an open-source version of Terraform that will expand on Terraform’s existing concepts and offerings. It is a viable alternative to Terraform, being forked from Terraform version 1.5.6, created as a community-driven initiative supported by many of the most recognized companies and projects in the Terraform ecosystem in response to Hashicorp’s BSL license change.
Others
Some of the biggest cloud providers have their own proprietary tools for implementing cloud infrastructure.
- AWS CloudFormation – A JSON/YAML-based authoring experience for provisioning AWS resources. AWS CDK which lets you author infrastructure for AWS resources transpiles to a CloudFormation template and then deploys the infrastructure using CloudFormation.
- Azure Bicep – Microsoft Azure’s version of a DSL IaC tool that is meant to replace ARM templates.
- GCP Deployment Manager – a deployment service whose primary configuration language is YAML. In some cases documentation is scarce and feature support is very limited.
Let’s examine a few concerns I have heard developers express when it comes to adopting a fundamentally new way to implement cloud infrastructure with a tool like Pulumi.
Deciding On A Language
Picking a language in a new tool can feel like a pretty drastic one-way decision. Fortunately, Pulumi interoperates with the supported languages. Delving into how multi-language components work perhaps deserves its own post some day, so I’ll leave that as a task for you to read 🙂
If I might dare say that being stuck with a general-purpose programming language is still better than having to learn a cloud provider’s DSL (templates or otherwise.) Learning a new language or technology has a cost. That cost is especially high when members of your team may not spend a lot of time with infrastructure on a daily basis.
DSL Is Easier/Quicker
One of the strengths for any DSL is its succinctness. It’s one of the reasons why DSLs are great for all types of configuration management. I think this is a matter of perspective. Developer skills are highly varied and everyone has their sweet spot of languages they are comfortable with. I mean, why do we need anything other than JS, am I right? Just kidding. On a more serious note, many developers are indeed more comfortable using DSL-based technologies such as Terraform for their cloud automation needs. While many others want something that is familiar and can be supported by any developer with language familiarity.
State Storage
One of the nice features of Pulumi is that the Pulumi Service stores the state for your infrastructure, by default. Right from the moment you run pulumi stack init <stackName>. But if things change and you would like to manage the state on your own, Pulumi does support self-managed backends too. That means, you could export your state and store it on your own inside one of the supported self-managed backends such as AWS S3, Azure Storage or GCS.
New Technology/Never Heard Of It
Pulumi has been in service since 2018. A number of well-known orgs use Pulumi. During my time at Pulumi I had come across several teams that have built complex cloud infrastructure installations spanning several cloud providers. But not all use cases are complex and yours doesn’t have to be to use Pulumi. Many of them start simple but grow organically and don’t realize that the growth was facilitated by Pulumi.
See also: Pulumi Pricing – Editions Overview.
Also by reading this post this changes for you today 🙂
Instead of asking why, you should ask yourself why not Pulumi? Even if you know the language well, cloud infrastructure is complex (want to create a private subnet with a NAT gateway? Wait, what’s a NAT and why do I need a private subnet? Exactly.) Pulumi helps reduce some of that complexity by removing the cognitive load that comes from familiarizing yourself with a DSL or a template system and in some cases actually helps address the complexity associated with cloud infrastructure with packages like the Crosswalk for AWS or the Pulumi Cloud framework, but does not eliminate it. You’ll discover that as your cloud infrastructure needs evolve, Pulumi’s capabilities scale well with that.
Once you’ve got your Pulumi stack configured, you might be wondering, well now what? It is pretty typical to wire-up a CI/CD pipeline to run the same Pulumi commands that you ran on your machine. Pulumi integrates really well with many CI/CD services.
This is where Spacelift excels. Spacelift is the most flexible management platform for Infrastructure as Code. Spacelift provides excellent capabilities to automate running your Pulumi stacks. That means you don’t have to configure your own CI pipelines. Especially if you have several stacks and want to connect them together. And it does not stop there.
Spacelift allows you to configure your workflows with OPA policies. For example, apply ACLs on stacks or detect changes to upstream stacks and trigger dependent downstream stacks.
Drift usually occurs when people manually modify resources, outside of Pulumi. Spacelift can detect it automatically, and optionally remediate it.
Hopefully, this post has given you many reasons to try out Pulumi. There are plenty of resources out there to get you started but we would recommend starting with the Pulumi and Spacelift documentations.
Continuous Integration and Deployment for your IaC
Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure. It helps overcome common state management issues and adds several must-have features for infrastructure management.
