General | Spacelift Blog

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

AWS: Spacelift Reduces Time Spent on Cloud Management by 90% 🚀

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

General

Learn more about new features, updates, and Spacelift product announcements.

Product Features and Updates | Spacelift Blog

Product

Jean-Marc Fontaine

So your team learned about <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code (IaC)</a>? They got all excited about this way of managing infrastructure. Soon enough, you had multiple stacks described in definition files. This source code was versioned in your favorite Version Control System such as GitHub, and integrated with your favorite CI/CD platform.
Finally, the infrastructure was built from those definition files, and it was time to celebrate the end of this journey, right? Or was it the end of the journey?
What the definition files describe is the desired state of the infrastructure. The way you want it to be set up. The responsibility of your IaC tool is to turn that desired state into reality. We call the current reality of your infrastructure the actual state. Right after you ran your IaC tool, both are identical but unfortunately, they might not stay in sync for long.
Any difference between the desired state and the actual state is called drift.

It might be surprising at first that infrastructure drifts. After all, you are managing it via code, right? Unfortunately, there are different ways your infrastructure can be modified outside of your IaC process.
<h3>Manual changes</h3>
Most of the drift is usually caused by manual changes performed by individuals.
Some of those reasons are understandable. For example, during an incident, an engineer might need to increase the number of resources to handle an elevated load or make up for resources being down. During those times of high stress when a lot is at stake, manual mitigation changes are perfectly acceptable. The goal is to get to a better place as soon as possible, and the regular IaC process might take too long. Especially when you need to respond to fast-changing conditions.
This becomes a problem if, after the fire has been put out, the changes are not reverted or backported to the IaC definition files.
There are also bad reasons for manual changes. Those cannot be justified, even temporarily, and often stem from poor education on best IaC practices, loose access permissions, and a lack of proper communication regarding the infrastructure management process.
<h3>Overlapping/conflicting IaC code</h3>
In some cases, humans are not directly at fault. Resources may end up being managed by multiple sets of IaC definition files. Applying some definition files might revert changes made by other definition files.
This can happen when the IaC practices evolve over time. For example, it is not uncommon to switch to a different IaC tool after a few years because the team realized that, in hindsight, they had not picked the best tool for their use cases.
Another reason for this is overlapping boundaries between stack definitions which can happen easily when the infrastructure is extensive and managed by many different teams over the course of a long period of time.

IaC is all about improving the governance of your infrastructure by defining it as code which allows you to leverage a wealth of practices and tools that have been available to developers for a long time such as code versioning, code reviews, static analysis, automated tests, etc.
Letting drift occur undermines that effort and provides a false sense of governance.

As we have seen earlier, the primary source of drift is manual changes. Those are often linked to loose access control practices. The Principle of least privilege recommends granting only the necessary permissions. The fewer people can manually modify the infrastructure, the better. Admin-level access is typically limited to senior infrastructure engineers and SREs.
Because you will always need to have some people with the permissions to perform manual changes to your infrastructure, you need to make sure that they are aware of the process to either revert or backport those changes in due time.
That being said, even with the most trained engineers and the best intentions, drift will happen. It is inevitable, so you need to make sure that you can easily and quickly detect it and possibly revert it.

As we said earlier, drift is the difference between the desired state of the infrastructure as defined in the IaC source code and the actual state of the infrastructure.
Said differently, drift is getting a non-empty list of proposed changes when running the plan command for your IaC tool.
Here is how to display the proposed changes with the main IaC tools.
<h3>Terraform</h3>
Run the <code>terraform plan</code> command to get the list of proposed changes. 

In the screenshot below, we can see that the maximum number of servers in the Auto Scaling Group was set to 5 outside of Terraform which is drift.

image4-19

Auto Scaling Group outside Terraform

<h3>CloudFormation</h3>
CloudFormation has a built-in drift detection feature that can be used either via the AWS Console or via the AWS CLI command.
CloudFormation’s drift detection must be triggered manually. There is no built-in automation to make it run on a schedule. Also, <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-supported-resources.html" target="_blank" rel="nofollow noopener">not all resource types can detect drift at this time</a>.
Drift detection checks can be run via the AWS Console:

image14-2

Drift detection checks via the AWS Console

image6-11

AWS Console Overview

image1-25

AWS Console Drifts

image10-7

AWS Console Drift details

Or with on the command line with AWS CLI:

image8-10

Drift detection on the command line with AWS CLI

image9-7

describe-steck-drift-detection-status

image2-24

describe-stack-resource-drifts

<ul>
<li><a href="https://spacelift.io/blog/devops-best-practices" target="_blank" rel="noopener">16 DevOps Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Automate Terraform Deployments and Infrastructure Provisioning&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
</ul>

<h3>Pulumi</h3>
Run the <code>pulumi preview --refresh --stack &lt;STACK NAME&gt;</code> command to get the list of proposed changes.
The screenshot below shows that the tags and user data of the AWS EC2 instance have been modified manually.

image11-8

pulumi preview --refresh --stack <STACK NAME>

<h3>Spacelift</h3>
Drift can occur at any time. As a result, drift detection must be run on a regular schedule to catch it as quickly as possible which is not practical when running those commands on one’s laptop.
A better approach would be to use a tool such as Spacelift that can check for drift automatically on a schedule that you set.

image12-7

Spacelift drift detection

The view that shows all the resources for a stack uses an eye-catching icon for resources that have drifted so that they can be easily spotted.

image3-21

resources that have drifted

Another benefit of using Spacelift is that the drift detection management experience is consistent across the supported IaC tools. Under the hood, different commands will be run but for the most part, the workflow and the screens will be identical.

You probably want to get rid of most drift which we will explain in a bit, but there might be manual changes that should make their way to the definition files.
For example, you had to increase the number of resources during an elevated load episode, but realistically, this is not a one-off but the new normal. Then, you should not eliminate the drift but update the IaC definition files to reflect your new expectations.

Since drift is having a non-empty list of proposed changes when the definition files have not changed, fixing that drift is applying the proposed changes. That will restore the infrastructure to its desired state.
Here is how to remove the drift and get back to the desired infrastructure state with the main IaC tools.
<h3>Terraform</h3>
Run the <code>terraform apply</code> command to revert the external changes and remove the drift.

image15-2

terraform apply

<h3>AWS CloudFormation</h3>
CloudFormation can revert drift in some cases only.
For example, if a resource is missing it will be recreated but if a property of a resource was modified it might not be detected by CloudFormation and as a result, it won&#8217;t be fixed automatically.
If CloudFormation cannot automatically fix the detected drift, you can use the information provided to manually revert the unexpected changes.
<h3>Pulumi</h3>
Run the <code>pulumi up --stack &lt;STACK NAME&gt;</code> command to revert the external changes and remove the drift.

image5-17

pulumi up --stack <STACK NAME>

image13-6

Spacelift Edit Drift Detection

When drift is detected, Spacelift can optionally revert the changes found by following the same workflow that is used for regular IaC code changes, enforcing all the configured guardrails such as <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy" target="_blank" rel="noopener">automated validation of the plan</a> and <a href="https://docs.spacelift.io/concepts/policy/approval-policy" target="_blank" rel="noopener">approval workflow</a>.

image7-10

Spacelift Drift Detected

Like incidents, drift is inevitable and part of the life of any infrastructure. It must be taken into account when defining the processes and selecting your tools so that you do not get caught off guard and stay on top of things regarding your infrastructure governance.

Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation and controls to simplify and accelerate the provisioning of cloud based infrastructures.

How Spacelift Can Improve Your Infrastructure as Code

Infrastructure Problems that Spacelift Solves

5 Ways to Manage Terraform at Scale - Best Practices

What is drift and why does it happen? See how Terraform, Cloudformation, Pulumi, or Spacelift, can help you with drift detection and remediation.

36-drift-detection

Drift detection with IaC tools

Infrastructure Drift Detection | How to Fix It With IaC Tool

Jack Roper

In this post we will delve into the basics of CI/CD pipelines, outlining the benefits of using them. We will then look at each stage in the pipeline, and what makes up a good pipeline, along with some examples relating to Terraform.

A CI/CD pipeline is used to automate software or <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">infrastructure-as-code</a> delivery, from source code to production. It can be thought of as a series of steps that needs to be taken for code to be released.
CI stands for Continuous Integration, and CD stands for Continuous Delivery or Deployment. ‘Pipeline’ refers to the automation of the delivery workflow, consisting of build, test, delivery, and deployment stages. Although each of these steps can be executed manually, automating and orchestrating the stages maximizes the benefits of using CI/CD pipelines by minimizing human error and bringing consistency to each release.
CI/CD pipelines are commonly used by application delivery teams and operations teams (<a href="https://spacelift.io/blog/who-is-devops-engineer" target="_blank" rel="noopener">DevOps</a>). They are the backbone of adopting a DevOps methodology.
CI/CD pipelines are themselves commonly configured in code, sometimes referred to as ‘pipelines-as-code’.
Typically a build server (or build agent) is used to enable CI/CD runs. Typically these take the form of self-hosted virtual machines in the cloud which can be fully configured but also have to be maintained, or virtual machines provided as part of the platform you are using, which are typically less flexible in terms of adding software and plugins to them.
Containers can also be used to enable consistent build environments, further removing the reliance on having to maintain a build server. Each step of the CI/CD pipeline can be run in its own container, allowing each step to be run inside a fully customized container. This also enables pipelines to make full use of all the benefits containerization orchestration affords, such as resilience and scaling where required.
<h3 id="aa7b" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">Continuous Integration (CI)</h3>
CI covers the build and test stages of the pipeline. Each change in code should trigger an automated build and test, allowing the developer of the code to get quick feedback.
<h3 id="03cb" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">Continuous Delivery / Deployment (CD)</h3>
The CD part of a CI/CD pipeline refers to Delivery and Deployment (CI/CDD anyone?!). CD takes place after the code successfully passes the testing stage of the pipeline. Continuous delivery refers to the automatic release to a repository after the CI stage. Continuous deployment refers to the automatic deployment of the artifact that has been delivered.

Starting with writing the source code, and ending up in production, these phases make up the development workflow and form the lifecycle of the CI/CD pipelines. Pipeline runs are usually triggered automatically by a change in the code, but can also be run on a schedule, run manually by a user, or triggered after another pipeline has run.
The four parts of the CI/CD pipeline are:
<ul>
<li>Build stage</li>
<li>Test stage</li>
<li>Deliver stage</li>
<li>Deploy stage</li>
</ul>
<h3 id="47f4" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">1) Build</h3>
The build stage involves the code being written. This is typically done by multiple people in a team, and for larger projects, multiple teams. Code is held in a version control system (VCS), and a Git-based workflow is commonly used to control code being added to the repository (also referred to as GitOps). When used in a pipeline, tools to regulate developer environments and standardize them are particularly useful in order to eliminate any differences between different authors&#8217; code. This will usually take the form of a docker container when using cloud-native software.
<h3 id="f401" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">2) Test</h3>
As greater resiliency is required and more varied infrastructure is introduced, testing code leads to greater confidence that the code will perform as expected. Testing of your code can be automated. In general, this is normally a repetitive, complex and sometimes tedious process to perform manually. A mistake that a lot of teams make is to skip the test stage or underuse it, as the benefits of properly testing code before delivery and deployment can be huge, contributing to a high-quality product.
Testing can be broken up into multiple types, using a combination of these with a combination of different tools will give the highest code coverage and result in a higher quality product. Where lots of different tests are used, these can be parallelized to reduce the pipeline run time.
Smoke testing — Perform quick sanity checks on the code.
Integration Testing — Integration tests are intended to test all the code, or system as a whole. They validate that newly introduced code does not break the existing code.
Unit Testing — Unit tests are intended to test a particular function, several functions together, or part of the code. Smaller parts of the infrastructure can be isolated, and tests can be run in parallel to shorten the feedback cycle.
Compliance Testing — Compliance testing is used to ensure the configuration follows the policies you’ve defined for the project.
End-to-end Testing (E2E) — E2E tests validate everything works together before deploying to production. It is the complete test of the whole process.
<h3 id="5674" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">3) Deliver</h3>
After the code has been tested, the code is packaged up as an artifact and committed to a repository.
<h3 id="3451" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">4) Deploy</h3>
The deploy stage allows the orchestration of the artifact release. Usually, teams will deploy to multiple environments, including environments for internal use such as development and staging, and Production for end-user consumption. Using this model, teams can automatically deploy to a staging environment when a pipeline is triggered. Once the staging environment has been reviewed and approved, the code can be merged into the main branch, and then automatically deployed to Production.
Below you can find a diagram of the CI/CD pipeline.

cicd-pipeline-diagram

cicd pipeline diagram

<ul>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
<li><a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Deploy your Infrastructure in CI/CD using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Deploy your Infrastructure in CI/CD using Terraform</a></li>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;10 Ways to Improve Your Infrastructure as Code&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">10 Ways to Improve Your Infrastructure as Code</a></li>
</ul>

Adopting the use of CI/CD pipelines brings many benefits, including:
<ul class="">
<li id="c6f7" class="mn mo ip kg b kh ki kk kl kn mp kr mq kv mr kz ms mt mu mv gt" data-selectable-paragraph="">Reduced costs — Reducing the time it takes the app or infrastructure to get coded and deployed means less human resources are taken up.</li>
<li id="1cbd" class="mn mo ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Reduced time to deployment — Time to deploy is reduced through automation. The entire process, from coding to deployment, is streamlined, reducing the total process length and making it more efficient. Deployments can be made more often. The rate of release can give companies a significant competitive business advantage over competitors. Deployments can also be rolled back easily, should there be a problem. Developers of the code can remain focused.</li>
<li id="623d" class="mn mo ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Enabling continuous feedback — Enables teams to improve their code and workflow based on the feedback received. The testing stage highlights problems or faults and immediately gives feedback to allow code to be improved. Detecting errors earlier in the coding process means that these are easier and quicker to fix. Notifications can be set up to alert team members to failures or successes at each stage.</li>
<li id="f7aa" class="mn mo ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Improving team collaboration — The team has visibility into problems, can view feedback, and make changes where appropriate.</li>
<li id="2c90" class="mn mo ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Audit trails — Each stage of the pipeline generates logs and can enable accountability and traceability.</li>
</ul>

A good pipeline should be reliable, accurate, and fast.
<h3 id="6cd4" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">Reliability</h3>
A pipeline should run reliably each time, without any errors, or unexpected intermittent errors. Broken pipelines can cause frustration and wasted time. When a pipeline is broken or throwing errors, it is recommended that the owner fix it as soon as possible before continuing work on the product. This approach should save other users from the same issues and aid the overall team. It is for this reason that many organizations have a specified ‘DevOps’ team, that will own the pipelines and monitor their success.
A reliable pipeline will produce the same result, given the same input.
Self-hosted build agents can often be the cause of unreliable pipeline runs, due to the required maintenance required on these (underlying infrastructure, patching, package management, security, etc). However, they are sometimes necessary as they offer greater flexibility when compared to platform-provided build agents.
<h3 id="2a9a" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">Accuracy</h3>
Ironing out errors in repetitive tasks is where CI/CD pipelines can really shine to improve the overall quality of a product.
<h3 id="5862" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">Speed</h3>
Optimizing a pipeline to run as quickly as possible ensures that the developer of the code can get feedback on the success or failure of the pipeline run quickly, making them more efficient, reducing the chance of distractions and ‘context switching’ (switching to another task/meeting/browsing the web).
Fast pipeline runs can also enable deployments more often. For example, if a pipeline takes 1 hour to run, and the working day is 8 hours long, then an absolute maximum of 8 deployments can be made per day. Reduce the pipeline run time down to 30 minutes, now 16 deployments can be made.
A common problem with pipeline runs is that they end up in a queued state, waiting for a build agent to process a previous run before the next can start. For this reason, multiple agents should be provisioned so different pipelines can be run in parallel. A serverless model or container orchestration solution is particularly useful here in order to scale the build agent capacity dynamically when demand is high.
Good pipelines are also quick to create. They should be written in code and kept in a VCS alongside the product code where appropriate. Base templates can be written by providing a common structure, which other templates can reference, speeding up the time required to create a new pipeline.
See our <a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">list of the most useful CI/CD tools</a> that define the state of today’s software automation.
And take a look at how <a href="https://spacelift.io/ci-cd-for-infrastructure" target="_blank" rel="noopener">Spacelift</a> provides a platform to manage IaC in an automated, easy, controlled, and secure manner. It goes above and beyond the support that is offered by the plain backend system. Spacelift is a sophisticated CI/CD platform for IaC that enables Gitops on infrastructure provisioning workflows. Check it out for free and create a <a href="https://spacelift.io/free-trial" target="_blank" rel="noopener">trial account</a>.

<h3 id="fbe2" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">1) Build</h3>
Write and store the Terraform Code in VCS.
<h3 id="7082" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">2) Test</h3>
Testing tools used with Terraform CLI-driven pipeline runs include Tfsec and Checkov. Many infrastructure-as-code-specific CI/CD platforms enable policy as code such as Terraform Cloud, or better still, Spacelift, which uses the open-source <a href="https://spacelift.io/blog/what-is-open-policy-agent-and-how-it-works" target="_blank" rel="noopener">Open Policy Agent (OPA)</a>, meaning this testing language could be used across multiple projects using different tooling.
<h3 id="9836" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">3) Deliver</h3>
Prepare Terraform Execution Environment (could include things like getting secrets from a key vault, or setting environment variables).
<a href="https://spacelift.io/blog/terraform-init" target="_blank" rel="noopener">Terraform init</a>
Terraform plan (copy output file to a repository, ready to be used by the apply command in the deploy stage.
<h3 id="7d0b" class="nb lr ip bn ls nc nd ne lw nf ng nh ma kn ni nj mc kr nk nl me kv nm nn mg no gt">4) Deploy</h3>
<a href="https://spacelift.io/blog/terraform-apply" target="_blank" rel="noopener">Terraform apply</a>
See also best practices and techniques for managing <a href="https://spacelift.io/blog/kubernetes-ci-cd" target="_blank" rel="noopener">CI/CD pipelines with Kubernetes</a>.

<h3>1) Build</h3>
Developers write Kubernetes configuration files and push them to a Git repository in a VCS. YAML files could be added to define other components of the pipeline, such as ConfigMaps for configuration data or Secrets for sensitive information.
<h3>2) Test</h3>
The code is built, and tests are run on it. Kube-bench is a security tool that checks Kubernetes configurations against best practices and industry standards, including CIS benchmarks. Helm unit testing is a testing tool for Helm charts.
<h3>3) Deliver</h3>
Once the tests pass, the CI server creates a Docker container image of the application and pushes it to a container registry such as Docker Hub or Amazon ECR.
<h3>4) Deploy</h3>
Once the container image is available in the registry, Kubernetes can be used to deploy the application. The docker image can be tagged as appropriate.
For more details, check out &#8211; <a href="https://spacelift.io/blog/kubernetes-ci-cd" target="_blank" rel="noopener">Kubernetes CI/CD Pipelines – 7 Best Practices and Tools</a>.

<h3>1) Build</h3>
Developers write .NET code and push it to a Git repository in a VCS. The code is built when it is pushed, triggered by CI using a build tool such as MSBuild or dotnet CLI.
<h3>2) Test</h3>
The code is built, and tests are run on it. Unit testing can be done using a framework such as NUnit or xUnit.
<h3>3) Deliver</h3>
Once the tests pass, the CI server generates an artifact, such as a NuGet package or a self-contained executable that can be deployed.
<h3>4) Deploy</h3>
The artifact is deployed.

CI/CD pipelines are an important concept for the modern engineer to understand fully, especially in the current cloud-native world. CI/CD pipelines enhance the software delivery process by automating key stages such as building, testing, delivery and deployment. Adopting a pipeline-based workflow helps you ship more quickly by passing all code through a consistent set of steps.

Spacelift is an alternative to using homegrown solutions on top of a generic  CI solution. It allows you to automate, audit, secure, and continuously deliver your infrastructure.

How to Deploy your Infrastructure in CI/CD using Terraform

What makes a good CI/CD pipeline? Take a closer look at the pipeline stages and some examples relating to Terraform.

74-cicd-pipeline

CI/CD Pipeline

CI/CD Pipeline : Everything You Need To Know with Examples

Praneet Loke

In the last two years teams have accelerated their dependency on the public cloud. Whether you are scaling an already existing product or building a new product that is native to the cloud, chances are you’ve felt that there has got to be a better way to build cloud infrastructure. There is. Enter Pulumi.

Pulumi is an infrastructure-as-code platform for full-stack developers and cloud engineers who are interested in using a general-purpose programming language for their cloud resources. Pulumi isn’t just one thing. It’s an ecosystem of components that all work together. At a high-level these are the components that you are most likely to hear a lot about:
<ul>
<li style="font-weight: 400;">Provider SDKs</li>
<li style="font-weight: 400;">CLI</li>
<li style="font-weight: 400;">Service backend</li>
</ul>
SDKs are installable packages your program consumes using the respective language’s package registry that you are more than likely already familiar with and probably already work with. Today, Pulumi supports four highly <a href="https://insights.stackoverflow.com/survey/2021#most-popular-technologies-language" target="_blank" rel="noopener">popular programming languages</a>.
<ul>
<li style="font-weight: 400;"><a href="https://www.pulumi.com/docs/intro/languages/go/" target="_blank" rel="nofollow noopener">Go</a></li>
<li style="font-weight: 400;"><a href="https://www.pulumi.com/docs/intro/languages/javascript/" target="_blank" rel="nofollow noopener">JavaScript/TypeScript</a></li>
<li style="font-weight: 400;"><a href="https://www.pulumi.com/docs/intro/languages/python/" target="_blank" rel="nofollow noopener">Python</a></li>
<li style="font-weight: 400;"><a href="https://www.pulumi.com/docs/intro/languages/dotnet/" target="_blank" rel="nofollow noopener">.NET Platform</a> (C#, F#, VB)</li>
</ul>
What that means is, you can create cloud resources using general-purpose code; think constructs such as variables, loops, functions, classes (well, at least in OOP), strong-type support, code reuse, and everything else you can do in those programming languages. It also gives you the option to maintain a homogeneous codebase, if that’s important to your team. For example, a team working on a C# .NET app may be interested in implementing their cloud infrastructure also in C# .NET. Or how about a team working on a React-based app that is strong in JS and the Node.js ecosystem wanting to use JS for their cloud infrastructure too?
Let’s examine each of those components in detail.
<h3>Provider SDKs</h3>
There are over <a href="https://www.pulumi.com/registry" target="_blank" rel="nofollow noopener">75 providers</a> with new ones being added regularly. And you can contribute new ones too. All of the SDKs are available in all of the supported languages. Extension libraries are typically not supported in every one of the supported languages since they are typically manually authored and in a language based on community demand. For example, the official Pulumi EKS is a popular <a href="https://www.pulumi.com/docs/intro/concepts/resources/components/" target="_blank" rel="nofollow noopener">Pulumi component</a> library that is supported in JS/TS and Python. But wait! The EKS library is also a great example of how you can <a href="https://www.pulumi.com/blog/create-eks-clusters-in-your-favorite-language/" target="_blank" rel="nofollow noopener">author and use multi-language components</a> in the Pulumi ecosystem. More on multi-language components later.
<h3>CLI</h3>
The CLI is one of the ways users can run previews and updates on their infrastructure programs. The CLI supports several powerful commands to mutate the state of your cloud resources. There is another powerful way to automate execution of your Pulumi apps.
<h3>Service Backend</h3>
The Pulumi Service backend is the default state storage service for all Pulumi projects. What this means is, every time you update your cloud resources using Pulumi, a snapshot of the state of your resources known as the checkpoint is stored by the service. The checkpoint allows you to confidently make changes to your infrastructure.

As mentioned before several components work together in the Pulumi ecosystem to mutate cloud resources. Let’s examine this with an example. Say, we want to create a <a href="https://www.pulumi.com/registry/packages/digitalocean/api-docs/spacesbucket/#example-usage" target="_blank" rel="nofollow noopener">Bucket</a> using DigitalOcean Spaces. We’ll use JS as an example for demonstration purposes.

Note that I’ve skipped the part about bootstrapping a new project, just like with creating any other new app Pulumi requires some minimal project setup too. This can be done easily with <code>pulumi new</code> which lets you pick a template to start from.
In order to create the Bucket in a DigitalOcean account, we’ll need an account and <a href="https://www.pulumi.com/registry/packages/digitalocean/installation-configuration/#spaces-api-credentials" target="_blank" rel="nofollow noopener">an access key for Spaces</a> so that Pulumi can authenticate on our behalf to mutate resources. We’ll assume that a token is created and configured correctly. Now that we have authentication with DigitalOcean squared-away we can run <code>pulumi preview</code> from the terminal (or an IDEs integrated terminal.)

image2-12

pulumi preview

The Pulumi CLI “processed” the <code>index.js</code> file and generated a graph of the resources that needed to be created (or deleted, updated…or do nothing if nothing has changed.) The graph, in this case, is simple. We just have a single bucket that we want to create. Behind-the-scenes, the CLI uses this graph to calculate the set of CRUD operations using the DigitalOcean API.
Running <code>pulumi up -y</code> will auto-approve Pulumi to go ahead and create the bucket.

image1-12

pulumi up -y

<div class="styles_wrapper__Bzgki">
<div class="styles_cardWrapper__ofReA">
<div class="styles_textWrapper___qfEA">
<div class="bodyTextWrapper WysiwigContent_wysiwigContent__duACp">
<ul>
<li><a href="https://spacelift.io/blog/devops-best-practices" target="_blank" rel="noopener">16 DevOps Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/drift-detection" target="_blank" rel="noopener">Fixing Infrastructure Drift with IaC Tools</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
</ul>
</div>
</div>
</div>
</div>

Using a general-purpose programming language means all language features are also implicitly Pulumi’s features too. Aside from that, there is a rich feature set to help with an ever-evolving cloud infrastructure landscape.
<h3>ComponentResources</h3>
<a href="https://www.pulumi.com/docs/intro/concepts/resources/components/" target="_blank" rel="nofollow noopener">ComponentResources</a> are resources that encapsulate multiple child resources. In simple terms, think of them as a class that you can create that lets you represent several related resources as a single unit. For example, you might be interested in creating a component that handles setting up an S3 bucket with replication in a secondary region for disaster recovery scenarios. You could create a component that accepts as input the bucket name but behind-the-scenes creates two buckets – one in the primary region and the other in the failover region. Oh, but wait! There is already a <a href="https://www.pulumi.com/registry/packages/aws-s3-replicated-bucket/" target="_blank" rel="nofollow noopener">component</a> for that.
<h3>StackReferences</h3>
As your infrastructure evolves, you may be interested in creating multiple stacks with certain stacks needing to consume the outputs exported by other “upstream” stacks. This is easy to do with <a href="https://www.pulumi.com/docs/intro/concepts/stack/#stackreferences" target="_blank" rel="nofollow noopener">StackReferences</a>. A quick example:

<h3>Dynamic Providers</h3>
Although it’s limited to specific languages, it can serve as sort of an escape-hatch for when you need to mimic the CRUD lifecycle of a resource but there is no official provider for that resource. A great example of dynamic providers would be to answer the question “<a href="https://github.com/pulumi/examples/tree/master/aws-ts-ec2-provisioners" target="_blank" rel="nofollow noopener">how do I configure a newly provisioned VM?</a>” 
<h3>Automation API</h3>
When I talked about the CLI, I said it is one of the ways to run Pulumi apps on users’ machines as well as in CI/CD pipelines. <a href="https://www.pulumi.com/docs/guides/automation-api/" target="_blank" rel="nofollow noopener">Automation API</a> allows users to programmatically invoke the CLI operations. This unlocks a whole another set of possibilities. Pulumi’s customers have been creative with this feature and have fully exploited it to enable advanced deployment scenarios for their organizations.

Alternatives vary based on which aspects of Pulumi you would like to compare.
<h3>Other General-Purpose Programming Language Tools</h3>
There’s <a href="https://aws.amazon.com/cdk/" target="_blank" rel="nofollow noopener">AWS CDK</a>. It’s similar to Pulumi on the surface. However, AWS CDK is limited to… well, AWS. It is also fundamentally different in how it works behind-the-scenes. As a user provisioning resources on AWS you may not need to worry about the implementation details of the technology ever for your use case.
<h4>AWS CDK vs. Pulumi:</h4>
<table>
<tbody>
<tr>
<td>CDK</td>
<td>Pulumi</td>
</tr>
<tr>
<td>Supports popular languages
&nbsp;</td>
<td>Supports popular languages
&nbsp;</td>
</tr>
<tr>
<td>Transpiles to CloudFormation template
&nbsp;</td>
<td>Doesn’t transpile to any proprietary format
&nbsp;</td>
</tr>
<tr>
<td>Cannot natively provision non-AWS resources
&nbsp;</td>
<td>Can use any of the 70+ providers as well author your own for edge-cases using Dynamic Providers or contribute your own
&nbsp;</td>
</tr>
<tr>
<td>Can create custom encapsulations using CdkContructs that create other AWS resources or Constructs
&nbsp;</td>
<td>Can create custom encapsulations using ComponentResources that can
&nbsp;</td>
</tr>
<tr>
<td>Has a registry showcasing community-supported as well as AWS-supported Constructs in the <a href="https://constructs.dev/" target="_blank" rel="nofollow noopener">Construct Hub</a>
&nbsp;</td>
<td>Has a registry showcasing community-supported as well as Pulumi-supported components and packages in the <a href="https://registry.pulumi.com" target="_blank" rel="nofollow noopener">Pulumi Registry</a>
&nbsp;</td>
</tr>
</tbody>
</table>
&nbsp;

<h3>Other Infrastructure-as-Code Tools</h3>
IaC isn&#8217;t restricted to just general-purpose programming language-based tools. In fact, one of the most common IaC tool that Pulumi is often compared with is Terraform. HashiCorp&#8217;s Terraform has existed in the IaC space well before Pulumi was created. If you look deeper Pulumi extends (the official word is &#8220;bridge&#8221;, however) many of Terraform&#8217;s providers as well as has native providers that are direct implementations from Open API specs. They are conceptually similar in that both tools want you to think about everything in the cloud as a &#8220;resource&#8221;. The resource model is what drives the core of your infrastructure state. Every resource has a CRUD lifecycle.
As much as they are similar in concept, they are also very different. The fundamental difference is the choice that you, as a developer, have when it comes to choosing the language that suits you and your team best. There are other differences too, of course.
<h4>Terraform vs. Pulumi:</h4>
<table>
<tbody>
<tr>
<td>Terraform</td>
<td>Pulumi</td>
</tr>
<tr>
<td>Uses proprietary but open-source language HCL which stands for HashiCorp Configuration Language
&nbsp;</td>
<td>Supports popular open-source, general-purpose programming languages
&nbsp;</td>
</tr>
<tr>
<td>HCL is a type of domain-specific language
&nbsp;</td>
<td>None of the supported languages are DSLs
&nbsp;</td>
</tr>
<tr>
<td>By definition, DSLs are limited in their language feature set
&nbsp;</td>
<td>Out-of-box support for language-level features that are not custom-made by Pulumi
&nbsp;</td>
</tr>
<tr>
<td>Has a registry showcasing community-supported as well as Terraform-supported Constructs
&nbsp;</td>
<td>Has a registry showcasing community-supported as well as Pulumi-supported components and packages
&nbsp;</td>
</tr>
<tr>
<td>No escape-hatch like functionality to create self-managed resources in cases where a provider doesn&#8217;t support a feature
&nbsp;</td>
<td>Dynamic Provider feature to model the CRUD lifecycle of custom one-off resources
&nbsp;</td>
</tr>
</tbody>
</table>
&nbsp;

<h3>Others</h3>
Some of the biggest cloud providers have their own proprietary tools for implementing cloud infrastructure.
<ul>
<li style="font-weight: 400;"><a href="https://aws.amazon.com/cloudformation/features/?pg=ln&amp;sec=hs" target="_blank" rel="nofollow noopener">AWS CloudFormation</a> &#8211; A JSON/YAML-based authoring experience for provisioning AWS resources. AWS CDK which lets you author infrastructure for AWS resources transpiles to a CloudFormation template and then deploys the infrastructure using CloudFormation.</li>
<li style="font-weight: 400;"><a href="https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview?tabs=bicep" target="_blank" rel="nofollow noopener">Azure Bicep</a> &#8211; Microsoft Azure’s version of a DSL IaC tool that is meant to replace ARM templates.</li>
<li><a href="https://cloud.google.com/deployment-manager/docs/#docs" target="_blank" rel="nofollow noopener">GCP Deployment Manager</a> &#8211; a deployment service whose primary configuration language is YAML. In some cases documentation is scarce and feature support is very limited.</li>
</ul>

Let’s examine a few concerns I have heard developers express when it comes to adopting a fundamentally new way to implement cloud infrastructure with a tool like Pulumi.
<h3>Deciding On A Language</h3>
Picking a language in a new tool can feel like a pretty drastic one-way decision. Fortunately, Pulumi interoperates with the supported languages. Delving into how <a href="https://www.pulumi.com/blog/pulumiup-pulumi-packages-multi-language-components/" target="_blank" rel="nofollow noopener">multi-language components</a> work perhaps deserves its own post some day, so I’ll leave that as a task for you to read 🙂
If I might dare say that being stuck with a general-purpose programming language is still better than having to learn a cloud provider’s DSL (templates or otherwise.) Learning a new language or technology has a cost. That cost is especially high when members of your team may not spend a lot of time with infrastructure on a daily basis.
<h3>DSL Is Easier/Quicker</h3>
One of the strengths for any DSL is its succinctness. It’s one of the reasons why DSLs are great for all types of configuration management. I think this is a matter of perspective. Developer skills are highly varied and everyone has their sweet spot of languages they are comfortable with. I mean, why do we need anything other than JS, am I right? Just kidding. On a more serious note, many developers are indeed more comfortable using DSL-based technologies such as Terraform for their cloud automation needs. While many others want something that is familiar and can be supported by any developer with language familiarity.
<h3>State Storage</h3>
One of the nice features of Pulumi is that the Pulumi Service stores the state for your infrastructure, by default. Right from the moment you run <code>pulumi stack init &lt;stackName&gt;</code>. But if things change and you would like to manage the state on your own, Pulumi does support self-managed backends too. That means, you could export your state and store it on your own inside one of the supported <a href="https://www.pulumi.com/docs/intro/concepts/state/#logging-in" target="_blank" rel="nofollow noopener">self-managed backends</a> such as AWS S3, Azure Storage or GCS.
<h3>New Technology/Never Heard Of It</h3>
Pulumi has been in service <a href="https://www.pulumi.com/blog/introducing-pulumi-a-cloud-development-platform/" target="_blank" rel="nofollow noopener">since 2018</a>. A number of <a href="https://www.pulumi.com/case-studies/" target="_blank" rel="nofollow noopener">well-known orgs</a> use Pulumi. During my time at Pulumi I had come across several teams that have built complex cloud infrastructure installations spanning several cloud providers. But not all use cases are complex and yours doesn’t have to be to use Pulumi. Many of them start simple but grow organically and don’t realize that the growth was facilitated by Pulumi.
Also by reading this post this changes for you today 🙂

Instead of asking why, you should ask yourself why not Pulumi? Even if you know the language well, cloud infrastructure is complex (want to create a private subnet with a NAT gateway? Wait, what’s a NAT and why do I need a private subnet? Exactly.) Pulumi helps reduce some of that complexity by removing the cognitive load that comes from familiarizing yourself with a DSL or a template system and in some cases actually helps address the complexity associated with cloud infrastructure with packages like the <a href="https://www.pulumi.com/docs/guides/crosswalk/aws/" target="_blank" rel="nofollow noopener">Crosswalk for AWS</a> or the <a href="https://www.pulumi.com/docs/tutorials/cloudfx/" target="_blank" rel="nofollow noopener">Pulumi Cloud</a> framework, but does not eliminate it. You’ll discover that as your cloud infrastructure needs evolve, Pulumi’s capabilities scale well with that.

Once you’ve got your Pulumi stack configured, you might be wondering, well now what? It is pretty typical to wire-up a CI/CD pipeline to run the same Pulumi commands that you ran on your machine. Pulumi integrates really well with many CI/CD services.
This is where Spacelift excels. <a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener">Spacelift</a> is the most flexible management platform for Infrastructure as Code. Spacelift provides excellent capabilities to automate running your Pulumi stacks. That means you don’t have to configure your own CI pipelines. Especially if you have several stacks and want to connect them together. And it does not stop there.
Spacelift allows you to configure your workflows with <a href="https://spacelift.io/blog/what-is-open-policy-agent-and-how-it-works" target="_blank" rel="noopener">OPA</a> policies. For example, <a href="https://docs.spacelift.io/concepts/policy/trigger-policy" target="_blank" rel="noopener">apply ACLs</a> on stacks or detect changes to upstream stacks and trigger dependent downstream stacks.
Drift usually occurs when people manually modify resources, outside of Pulumi. Spacelift can detect it automatically, and optionally remediate it.
Hopefully, this post has given you many reasons to try out Pulumi. There are plenty of resources out there to get you started but we would recommend starting with the <a href="https://www.pulumi.com/docs/get-started/" target="_blank" rel="nofollow noopener">Pulumi</a> and <a href="https://docs.spacelift.io/vendors/pulumi/" target="_blank" rel="noopener">Spacelift</a> documentations.

Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure. It helps overcome common state management issues and adds several must-have features for infrastructure management.

In this article you'll learn what Pulumi is, how it works and why it can be helpful for your Infrastructure as Code (IaC).

61-pulumi-2

What is Pulumi

What is Pulumi? Key Concepts and Features Overview

Rahul Wagh

DevOps is a set of practices where development, IT operations, quality, and security are glued to Continuous Integration/Continuous Delivery (CI/CD) to deliver a reliable product to end customers. DevOps culture facilitates the collaboration of the Core Development and Operation team, allowing companies to reduce organization silos, expect and handle failure as part of the process, implement gradual changes, leverage automation tools, and at last, measure everything.
Whereas Site Reliability Engineering (SRE) is responsible for implementing the product developed by the Core Development team. The key objective of the SREs is to implement and automate DevOps practices to reduce the level of incidents and improve reliability and scalability. SREs are also responsible for sending swift and constant feedback to the Development team based on performance metrics – availability, latency, efficiency, capacity, and incident.

While DevOps is all about what aspect of the matters, SRE talks about the how part of it all. Nevertheless, there are a few other differences between the two. 
<ol>
<li style="font-weight: 400;" aria-level="1">Implementing New Features – DevOps is responsible for implementing the new features request to a product, whereas SREs ensure those new changes don’t increase the overall failure rates in production.</li>
<li style="font-weight: 400;" aria-level="1">Process Flow – A DevOps team has a perspective of the development environment to put changes from development to production. On the other hand, SREs have a perspective of production, so they can make suggestions to the development team to limit the failure rates despite the new changes.</li>
<li style="font-weight: 400;" aria-level="1">Focus – DevOps’s primary focus is on continuity and speed of product development, whereas SRE’s main focus is on the system’s reliability, scalability, and availability.</li>
<li style="font-weight: 400;" aria-level="1">Team Structure – A typical DevOps team consists of professionals with dedicated roles and responsibilities such as – Product Owner, Team Lead, Cloud Architect, Software Developer, QA Engineer, Release Manager, System Administrator. In contrast, SREs have a team of engineers with operational and development skills set.</li>
</ol>

Although there is some overlap in the job roles of SREs and DevOps, there is wide segregation of functions:
<table>
<tbody>
<tr>
<td></td>
<td>DevOps</td>
<td>SREs</td>
</tr>
<tr>
<td>
Role
&nbsp;</td>
<td>The main role of the DevOps team is to solve the development problem, build solutions to cater to business requirements.
&nbsp;</td>
<td>SREs’ main role is to deal with operational problems, for example – production failures, infrastructure issues (disk, memory), security, monitoring.
&nbsp;</td>
</tr>
<tr>
<td>Focus
&nbsp;</td>
<td>Focus on product development with Continuous Integration/ Continuous Delivery.
&nbsp;</td>
<td>SREs put more focus on resilience, scaling, reliability, uptime, and robustness.
&nbsp;</td>
</tr>
<tr>
<td>Tools
&nbsp;</td>
<td>In the DevOps role, the most widely used tools are – Integrated Development Environment (IDEs) for development purposes, <a href="https://spacelift.io/blog/terraform-jenkins" target="_blank" rel="noopener">Jenkins</a> for Continuous Integration and Development, JIRA for change management, Splunk for log monitoring, SVN, GitHub.
&nbsp;</td>
<td>In the SRE role, the most widely used tools are Prometheus and Grafana for collecting and visualizing the different metrics (CPU usage, memory, disk space, etc.), incident alert tools (OP5, PageDuty, xMatters, etc.), Ansible, Puppet, or Chef, Kubernetes and Docker for container orchestration, cloud platform AWS, GCP, Azure, JIRA, SVN, GitHub.
&nbsp;</td>
</tr>
<tr>
<td>Bug reporting
&nbsp;</td>
<td>DevOps team is responsible for debugging the code in case of any bug reported in the end product.
&nbsp;</td>
<td>The SRE team is responsible for reporting the bug to the Core development team and does not get involved in debugging unless it is a production outage. SRE team is also responsible for debugging and fixing the infrastructure issues.
&nbsp;</td>
</tr>
<tr>
<td>Measurement metrics
&nbsp;</td>
<td>Typical measurement metrics for the DevOps role are Deployment Frequency and the Deployment Failure rate.
&nbsp;</td>
<td>Typical measurement metrics for the SRE role are Error Budgets, SLOs (Service Level Objective), SLIs (Service Level Indicator), SLAs (Service Level Agreement).
&nbsp;</td>
</tr>
<tr>
<td>Incident handling
&nbsp;</td>
<td>DevOps teams work on the incident feedback to mitigate the issue.</td>
<td>Conductsthe Post-Incident reviews to identify the root cause and document the findings to provide feedback to the core development team.</td>
</tr>
</tbody>
</table>

<ul>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
<li><a href="https://spacelift.io/blog/devops-best-practices" target="_blank" rel="noopener">16 DevOps Best Practices for 2023</a></li>
<li><a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Deploy your Infrastructure in CI/CD using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Deploy your Infrastructure in CI/CD using Terraform</a></li>
</ul>

Implementing the DevOps practices can reduce the friction between Development and Operations teams. It can also help you deliver the end product reliably along with other challenges and problems that the DevOps teams can solve.
<h3>1. Reduced Cost of Development and Maintenance</h3>
A DevOps team always works towards CI/CD, putting more effort into automated testing rather than manual testing and improving release management by automating it all.
As for the traditional Software Development Life Cycle, there is always toil on the effort (development, testing, release), which increases the overall cost for the product development and production maintenance. Putting DevOps practice in execution can significantly reduce the delivery time, development, and maintenance costs.
<h3>2. Shorter Release Cycle</h3>
One of the most effective changes a DevOps team brings is to deliver faster with a shorter release cycle. The reason why the DevOps team advocates a shorter release cycle is that it is easy to manage and roll back to the stable version in case there are any issues. 
As opposed to the traditional release cycles, where the focus is on getting everything delivered in one release, which increases the risk of failure in production and is much harder to roll back. If DevOps practices are followed strictly, the organization will always have a proper release version system with release versions and minimal manual interventions with the release artifacts.
Here are the gains of a shorter release cycle:
<ol>
<li style="font-weight: 400;" aria-level="1">Deliver the new change request more frequently;</li>
<li style="font-weight: 400;" aria-level="1">Pushing the upgrades (bug fixes, security patches, version upgrades) to production is much easier.</li>
</ol>
<h3>3. Automated and Continuous Testing</h3>
In contrast to the traditional development cycle, where the testing team has to wait for the delivery of the product in the test environment to begin the testing DevOps, testing is injected from the beginning of the development lifecycle.
DevOps facilitates continuous and automated testing with the help of the CI/CD tool (Jenkins) and version control (Git, BitBucket). Adequate coverage of functional, nonfunctional, and interaction tests running in the <a href="https://spacelift.io/blog/ci-cd-pipeline" target="_blank" rel="noopener">pipelines</a> can significantly improve the testing automation aspects of the project.
To learn more about DevOps, see our article on <a href="https://spacelift.io/blog/who-is-devops-engineer" target="_blank" rel="noopener">Who is DevOps?</a>

<h3>1. Reduced Mean Time to Recovery (MTTR)</h3>
SRE team is responsible for keeping the production up and running. In the event of a bug or production failure, SRE teams can roll back to the previous stable version of a product so that Mean Time to Recovery (MTTR) is reduced.
<h3>2. Reduced Mean Time to Detect (MTTD)</h3>
The other problem that the SRE team is trying to solve is to reduce the Mean Time to Detect(MTTD) using the <a href="https://spacelift.io/blog/kubernetes-deployment-strategies#canary-deployments" target="_blank" rel="noopener">Canary Rollouts</a> so that the new release is made available to a small group of users before doing full rollouts. Canary rollouts help the SRE team find the issues in the early stage with a limited number of affected users.
<h3>3. Automated Everything</h3>
Automation is one of the biggest challenges the SRE team has to face. It is often observed that rollouts and supporting tasks are carried out manually, leading to inconsistency and increasing the probability of human error. 
A good practice for managing the infrastructure is to use <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code (IaC)</a> with the help of Terraform, Pulumi, and the automation tools such as Ansible, Puppet, Chef. SRE team can leverage those tools to solve the problem of automation.
<h3>4. Automated Functional and Non Functional Testing in Production</h3>
The Core Development team can automate functional and non-functional testing in the test and stage environments but not in production.
Reliability engineers can help implement automation testing on Production environments without affecting the end-user.
<h3>5. On-Calls and Incident Documentation</h3>
Often reliability engineers have to take the on-call duties for managing unforeseen incidents, but they also have to prepare the documentation of the incidents and the troubleshooting steps so that it can help others perform the on-call duties. 
The SRE team can build up a valuable knowledge base on incidents to improve the incident troubleshooting time.
<h3>6. Shared Knowledge</h3>
Gaining exposure and building the knowledge base of the product development ecosystem (dev, test, stage, prod) is always beneficial for reliability engineers to foresee the issues in the production environment. 
But the main problem arises when the knowledge base is outdated, automation playbooks have irrelevant comments. Regular knowledge base updates by SREs in collaboration with DevOps can fill the knowledge gap between the teams.

When we talk about the tools of DevOps and SRE, it is often observed that most of the tools are being used commonly by both DevOps and SREs. 
<h3>Common Tools</h3>
<h4>Planning</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Jira Software</li>
<li style="font-weight: 400;" aria-level="1">Confluence</li>
<li style="font-weight: 400;" aria-level="1">Slack</li>
<li style="font-weight: 400;" aria-level="1">Microsoft Teams</li>
</ul>
<h4>Configuration Management Tools</h4>
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-pulumi" target="_blank" rel="noopener">Pulumi</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/ansible-tutorial" target="_blank" rel="noopener">Ansible</a></li>
<li style="font-weight: 400;" aria-level="1">Puppet</li>
<li style="font-weight: 400;" aria-level="1">Chef</li>
</ul>
<h4>Version Management</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">GitHub</li>
<li style="font-weight: 400;" aria-level="1">BitBucket</li>
<li style="font-weight: 400;" aria-level="1">GitLab</li>
</ul>
<h4>Log Monitoring</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Splunk</li>
</ul>
<h3>DevOps Tools</h3>
<h4>Continuous Integration Continuous Delivery</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Jenkins</li>
<li style="font-weight: 400;" aria-level="1">AWS CodePipeline</li>
</ul>
<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Added: See our list of the most useful CI/CD tools that define the state of today’s software automation.&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">See our <a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">list of the most useful CI/CD tools</a> that define the state of today’s software automation. And take a look at <a href="https://spacelift.io/ci-cd-for-infrastructure" target="_blank" rel="noopener">Spacelift</a>, which is a sophisticated CI/CD platform for IaC that enables Gitops on infrastructure provisioning workflows. It provides a platform to manage IaC in an automated, easy, controlled, and secure manner. Spacelift goes above and beyond the support that is offered by the plain backend system.
<h4>Integrated Development Environment</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Intellij </li>
<li style="font-weight: 400;" aria-level="1">Visual Studio</li>
<li style="font-weight: 400;" aria-level="1">Sublime</li>
</ul>
<h4>Automated and Security Testing</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Jmeter</li>
<li style="font-weight: 400;" aria-level="1">Robot Framework</li>
<li style="font-weight: 400;" aria-level="1">Burp</li>
<li style="font-weight: 400;" aria-level="1">Wireshark</li>
</ul>
<h3>SRE Tools</h3>
<h4>Monitoring</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Kibana</li>
<li style="font-weight: 400;" aria-level="1">Prometheus</li>
<li style="font-weight: 400;" aria-level="1">Grafana</li>
<li style="font-weight: 400;" aria-level="1">New Relic</li>
<li style="font-weight: 400;" aria-level="1">Istio</li>
</ul>
<h4>Incident Reporting System</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">PagerDuty</li>
<li style="font-weight: 400;" aria-level="1">OP5</li>
<li style="font-weight: 400;" aria-level="1">Opsgenie</li>
<li style="font-weight: 400;" aria-level="1">VictorOps</li>
</ul>

While the two share some core values, the focus of their work is different &#8211; the application lifecycle through DevOps and operations lifecycle management through SRE. Nevertheless, they both connect the Development and Operation teams while sharing similar responsibilities. And they are both working towards the same goal &#8211; enhancing the release cycle and achieving better product reliability.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

How to Become a DevOps Engineer in Six Months

Business Benefits of Infrastructure as Code

Which Cloud Deployment Model Is Right for Your Business?

What does DevOps do? And what about SRE? See our article and find out the differences between DevOps and SRE. 

56-sre-vs-devops

SRE vs DevOps

SRE vs. DevOps: What’s the Difference Between Them?

how to manage infrastructure as code at scale

Peter Christensen

Scalability and IaC are not always a match made in heaven. There are several touchy areas within the development environment, and associated tooling must be seriously considered to ensure effective “management” of IaC to assure scalability. 
This blog post will touch on a few critical areas that can negatively impact IaC when provisioning and deploying complex, large-scale infrastructures. It will also show how to manage IaC at scale without workarounds, a manual, or a partially automated development process.

The first area to understand is IaC code platforms. <a title="Terraform is the de facto standard and most widely used IaC coding platform" href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform is the de facto standard and most widely used IaC coding platform</a> by far; several alternatives, such as Pulumi, Cloudformation, and Ansible, are available. Each of them has its relative strengths and weaknesses.
While we are huge fans of Terraform, we are also pragmatic about its shortcomings, limiting the ability of a developer to manage IaC at scale cleanly. Below, we highlight some of these weaknesses that should be considered:
<ul>
<li style="font-weight: 400;" aria-level="1">Resource management: Running the terraform state mv command every time there is a changed resource identifier is painful and time-consuming. It is further complicated by a large codebase, as it is challenging to maintain clean code and evolve over time</li>
<li style="font-weight: 400;" aria-level="1">TF State: Terraform tracks the current state of managed resources using plan and apply. State is very fragile and can be impacted, for example, by a stale state file or by forgetting to run “terraform remote config”</li>
<li style="font-weight: 400;" aria-level="1">Local machines: Each action is manual and must be performed directly by the developer, leading to multiple people working on the same codebase. Developer A applies their changes to the environment and initiates a pull request. At the same time, developer B would like to deploy their changes but is blocked as the codebase is not up-to-date with developer A’s changes. Applying the changes in this state would cause damage to the resources created in the previous deployment. Working in larger teams will often result in wasted time just waiting and rebasing the codebase to apply changes.</li>
<li style="font-weight: 400;" aria-level="1">Security: Devs working with the codebase need access to the Terraform state, which creates a security concern due to how Terraform state works. Pulling sensitive data from an external system such as Vault will be stored within the state file in plaintext once used in a resource, so a threat actor could run `terraform state pull` to access all the secrets</li>
<li style="font-weight: 400;" aria-level="1">CLI Interface: Typos and other types of errors are common, resulting in misconfigurations or failed plans and applies</li>
<li style="font-weight: 400;" aria-level="1">Infrastructure Fine-Tuning: It is difficult to interface with subsets of the infrastructure to make small changes dynamically. Terraform works well for bulk deployments but lacks the delicate touch necessary for fine-tuning infrastructures</li>
<li style="font-weight: 400;" aria-level="1">Rollback: Terraform does not support rollback to a previous state due to limitations of Terraform plan, which means that the only option is to roll forward</li>
</ul>
New to IaC? See: <a title="What Is Infrastructure as Code? Examples, Best Practices &amp; Tools" href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">What Is Infrastructure as Code? Examples, Best Practices &amp; Tools</a>

The short answer is yes! IaC development can be significantly enhanced using <a title="CI/CD processes" href="https://upcommons.upc.edu/bitstream/handle/2117/174561/TFG-definitiu.pdf" target="_blank" rel="nofollow noopener">CI/CD processes</a> and workflows like applications or tool development. Using CI/CD to provide workflows for infrastructure provisioning and deployment offers the following benefits: 
Consistency:
<ul>
<li style="font-weight: 400;" aria-level="1">Code is controlled &#8211; well documented and optimized.</li>
<li style="font-weight: 400;" aria-level="1">Code is modular &#8211; reused across multiple projects.</li>
</ul>
Testing:
<ul>
<li style="font-weight: 400;" aria-level="1">Testing is integrated &#8211; cannot be skipped.</li>
<li style="font-weight: 400;" aria-level="1">It is far simpler to test code in small bites &#8211; find issues faster not hunting for a needle in a haystack &#8211; 100 loc vs. 10K loc.</li>
</ul>
Automation:
<ul>
<li style="font-weight: 400;" aria-level="1">Routing of change approvals &#8211; single or multiple sign-offs.</li>
<li style="font-weight: 400;" aria-level="1">Establishes a set of controllable mechanisms &#8211; gates for events, etc.</li>
</ul>
See how to deploy your infrastructure in CI/CD here: <a title="How to Deploy Your Infrastructure in CI/CD Using Terraform" href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener">How to Deploy Your Infrastructure in CI/CD Using Terraform</a>

The below diagram provides an example of a Fintech application and support for infra implemented in AWS. This solution is used by financial institutions, banks, or specialized lenders to manage their Asset Based Lending portfolio and deliver their services to their clients.

image1-9

Fintech ABL Application and Infrastructure

In this example infrastructure, each of the infra resources designated with the TF logo has Terraform providers, or HCL code is available to configure the infrastructure.
Provisioning and deploying for new clients is complex as it has to meet their individual compliance requirements and banking and regulatory rules, regulations, and standards. And what about a foreign client bank? For this, banks must meet completely different banking and regulatory rules and compliance standards. 
And of course, overarching everything are stringent cybersecurity requirements, which are necessary in many cases as the application has to be able to securely interface with external core banking systems for daily updates on loan status with each customer.
Imagine duplicating this infrastructure each time a new client is onboarded using a non-IaC optimized CI/CD solution. 
<h3>IaC and CI/CD</h3>
As mentioned, a CI/CD solution is strongly recommended for provisioning and deploying IaC based infrastructures. One of the most widely used CI/CD solutions is Jenkins. 
However, <a href="https://spacelift.io/blog/terraform-jenkins" target="_blank" rel="noopener">Jenkins</a> and other generic CI/CD solutions were designed and optimized for application-centric workflows, and have limitations that will prevent scalability when implemented in support of IaC-specific development. Some of these issues include:
<ul>
<li style="font-weight: 400;" aria-level="1">IaC is a stack-based technology, and generic CI/CD tools do not understand stacks.</li>
<li style="font-weight: 400;" aria-level="1">No unified toolset or framework for policy management requiring manual checks.</li>
<li style="font-weight: 400;" aria-level="1">Lack of built-in synchronization, resulting in manual updates and synchronization.</li>
<li style="font-weight: 400;" aria-level="1">Inability to provide an automated workflow across multiple repositories and stacks.</li>
<li style="font-weight: 400;" aria-level="1">Cannot visualize or provide cost tracking of resources, <a href="https://spacelift.io/blog/drift-detection" target="_blank" rel="noopener">drift detection</a>, and remediation.</li>
<li style="font-weight: 400;" aria-level="1">In most cases, SSO and SAML are plugins—just another component to configure, manage, and update</li>
<li style="font-weight: 400;" aria-level="1">Conflicting pull requests where one developer is already working on a piece of code and another developer is also making changes, possibly resulting in code discrepancies.</li>
</ul>
Generic CI/CD solutions check all the boxes to support implementing a well-defined development process, adding end-to-end workflows allowing DevOps teams to deliver better applications faster, but not for IaC. 
<h3>Accountability and IaC</h3>
What do we mean by accountability? It is analogous to a business having a set of accounting books that represent the financial condition of the company as it stands at the present moment, but also its financial history. 
Without this history, it is impossible to trace back and establish accountability, to understand when financial decisions and changes were made, actions taken, and by whom and for what purpose.
Accountability is also applicable to your infrastructure as it enables you to understand the present state and how you arrived at this point. 
Infrastructure accountability provides checks and balances to ensure that you can account for all infrastructure-related events. From the first line of code to production, you need to know and understand every event that has or will affect the infrastructure to ensure that process integrity exists and, by extension, so does proof of compliance, both internally and externally. Some of the accountability questions that you have to be able to answer are:
<ul>
<li style="font-weight: 400;" aria-level="1">Code History:
<ul>
<li style="font-weight: 400;" aria-level="2">What is the history of the code?</li>
<li style="font-weight: 400;" aria-level="2">Who made changes to the code and infrastructure.</li>
<li style="font-weight: 400;" aria-level="2">Do you know if access to your repo is secure? </li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Change Implementation:
<ul>
<li style="font-weight: 400;" aria-level="2">Has a change been fully implemented?</li>
<li style="font-weight: 400;" aria-level="2">Can you validate and prove that it was deployed?</li>
<li style="font-weight: 400;" aria-level="2">Are paper-based policies and regulatory compliance requirements being met? Can you prove it?</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Change Approval Process:
<ul>
<li style="font-weight: 400;" aria-level="2">What is the change approval process?</li>
<li style="font-weight: 400;" aria-level="2">Are all changes approved, no matter how small or innocuous they seem?</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Auditability:
<ul>
<li style="font-weight: 400;" aria-level="2">Are you prepared for an audit?</li>
<li style="font-weight: 400;" aria-level="2">Do you have the supporting materials necessary to satisfy security and compliance audits?</li>
</ul>
</li>
</ul>
What types of information do you need to answer the above questions?
<ul>
<li style="font-weight: 400;" aria-level="1">Provenance: The ability to trace every piece of code back to inception, along with any changes made.</li>
<li style="font-weight: 400;" aria-level="1">Compliance: Provide detailed information needed to ensure compliance with policies and procedures, and the ability to provide complete documentation for an audit.</li>
<li style="font-weight: 400;" aria-level="1">Access: Who has access, what do they have access to, what are they allowed to do, when are they allowed to do it, why are they doing it.</li>
<li>Visibility: Understand the state of the infrastructure and resources at any point in time.</li>
</ul>

<ul>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;5 Ways to Manage Terraform at Scale – Best Practices&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">5 Ways to Manage Terraform at Scale – Best Practices</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
<li><a href="https://spacelift.io/blog/drift-detection" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Drift Detection and How to Fix It With IaC Tools&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Infrastructure Drift Detection and How to Fix It With IaC Tools</a></li>
</ul>

The complex infrastructure example above gives an overview of a current ABL Fintech solution. This diagram represents the security that is provisioned and deployed on top of the infrastructure for the application.

image3-9

Fintech ABL Security Overlay

In many cases, individual clients require VPCs for their implementation, with multiple layers of protection, as they may have different security requirements or their specific vendor solutions in place. It is especially relevant if financial institutions have product offerings beyond ABL that have strict regulatory or industry compliance requirements based on their geographical location.
The Terraform logo highlights these security-specific resources supported in Terraform, and can be defined and provisioned.
The ability to control the following security aspects for access, source code, changes, and deployments, as well as validation thereof, is not achievable without embedded policy guardrails in place. They are essential to achieving secure infrastructure provisioning and deployments. Without policies in place, the probability of a bank passing an internal or external audit is very low.

Now that we’ve covered code platforms, CI/CD solutions, and accountability to help you understand the significant issues that stand in your way of IaC scalability, let’s take a look at what can help you to manage IaC at scale without workarounds, a manual, or a partially automated development process: 
<h3>Automate</h3>
Automation covers the end-to-end IaC workflow from repo to production:
<ul>
<li style="font-weight: 400;" aria-level="1">Integrated with source code repos such as Github to trigger IaC workflows when committing new or revised code.</li>
<li style="font-weight: 400;" aria-level="1">Creates a library of pre-configured infrastructure environments, enabling sharing across teams and projects.</li>
</ul>
<h3>Use Policy as Code</h3>
Policy as Code provides immediately enforceable guardrails:
<ul>
<li style="font-weight: 400;" aria-level="1">Declare rules around infrastructure, access to code and stacks, workflow, state changes, set triggers. </li>
<li style="font-weight: 400;" aria-level="1">Access controls for the 4Ws.</li>
<li style="font-weight: 400;" aria-level="1">Multi-layer change approvals and applies after approval.</li>
<li style="font-weight: 400;" aria-level="1">Audit support, SSO (SAML2.0) fully integrated. </li>
</ul>
<h3>Collaborate</h3>
Adaptable to existing workflows with no force-fitting or changes required, or fully customized to meet goals:
<ul>
<li style="font-weight: 400;" aria-level="1">Coordination of all runs with the ability to review, comment, and iterate across teams.</li>
<li style="font-weight: 400;" aria-level="1">Prevent conflicting changes and reconcile the differences automatically.</li>
</ul>
<h3>Introduce Resource Management</h3>
Understand resources and their state:
<ul>
<li style="font-weight: 400;" aria-level="1">Plan out changes.</li>
<li style="font-weight: 400;" aria-level="1">Drift detection and remediation.</li>
<li style="font-weight: 400;" aria-level="1">Cost of infrastructure.</li>
</ul>
In addition to all of the above, here are some additional vital capabilities that an ideal, purpose-built orchestration and management solution for IaC should also provide, at a minimum! Starting from the left of the diagram:
<ul>
<li style="font-weight: 400;" aria-level="1">Creation of pre-built library of infra components.</li>
<li style="font-weight: 400;" aria-level="1">Support of mono or multiple repos with version control.</li>
<li style="font-weight: 400;" aria-level="1">Ability to support multiple projects simultaneously.</li>
<li style="font-weight: 400;" aria-level="1">Integration and application of policy as code in concert with IaC configurations, ensuring that compliance and security initiatives are met.</li>
</ul>

image2-9

image2

Learn how to manage Terraform at scale here: <a title="5 Ways to Manage Terraform at Scale (Examples)" href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale (Examples)</a>

You can take positive actions to overcome the scalability issues of IaC code platforms, generic CI/CD solutions, and accountability to prevent your infrastructure from inhibiting your future business initiatives and growth.
You can be realistic and find an IaC orchestration and management solution that enables your development process to scale. 
And finally, you ought to be aware that proactive management and control over compliance and audit objectives are necessary and can be covered through Policy as Code, integrated into your IaC development, security, provisioning, and deployment workflows.
If you are concerned about achieving IaC scalability, <a title="try a free trial of Spacelift!" href="https://spacelift.io/free-trial" target="_blank" rel="noopener">try a free trial of Spacelift!</a> You can overcome code platform issues and enable accountability with a CI/CD solution built from the ground up for IaC. Control the process, visualize resources and create workflows that work, using Spacelift as your IaC orchestration and management solution.

Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure.  It helps overcome common state management issues and adds several must-have features for infrastructure management.

Infrastructure as Code : Best Practices, Benefits & Examples

Managing infrastructure as code at scale tutorial. See how to manage IaC at scale. Examples and diagrams included. 

45-how-to-manage-infrastructure-as-code-at-scale

How to Manage Infrastructure as Code at Scale (Examples)

Peter is Head of Product Marketing at Spacelift. He brings over 20 experience in product marketing, having launched a
number of products ranging from complex network protocols to developer tools.

Stuart Burns

The term DevOps is used as a massive catchall for several different tech-job scenarios and skills. But essentially it boils down to building useful tools for your internal customers.
DevOps as a career can be rewarding in many ways. First, there’s the <a href="https://www.glassdoor.com/Salaries/devops-engineer-salary-SRCH_KO0,15.htm" target="_blank" rel="noopener">average salary of $105,000</a> to <a href="https://www.ziprecruiter.com/Salaries/What-Is-the-Average-Devops-Engineer-Salary-by-State" target="_blank" rel="noopener">$114,000</a>, spiking to over $160,000 at the top end. Then there’s the swift job growth, estimated at over 20% per decade, with over <a href="https://www.linkedin.com/jobs/devops-jobs" target="_blank" rel="nofollow noopener">118,000 openings listed on LinkedIn</a> right now.
Plus, DevOps beats the daily grind of “same old, same old,” thanks to its fast-paced, system-support setup. In this article, we share how to become a DevOps engineer — the systems administrator who just happens to be a coder that develops internal tools as well. You’ll learn about the mindset, tools, and skills you’ll need to master, and also how to get a DevOps job.
Before you start though, is DevOps the right choice for you? See our guide: <a href="https://spacelift.io/blog/who-is-devops-engineer" target="_blank" rel="noopener">Is it Worth it to Become a DevOps Engineer?</a>

DevOps is a mix of development (Dev) and operations (Ops). Beyond that, it’s all the connected tools and disciplines of both of those business areas that let an organization deliver services and software applications at high speed, so they can better serve their customers.
The Microsoft-approved definition of DevOps is “the union of people, processes and technology to continually provide value to customers.” Both definitions are open to interpretation, but DevOps centers around writing code that pulls together customers (internal and external), and business processes.
Sometimes, DevOps engineering means just “being that go-to employee” who can quickly and efficiently write code to address an engineering issue. In other words, in some organizations, DevOps is the indispensable IT employee who knows how to write effective code.

The question of how to become a DevOps engineer has a relatively straightforward answer. With that said, you’ll need to bring a few things to the table. First and most important to the DevOps career path is a passion for learning, knowledge, and logic.

DevOps engineers need to be able to read between the lines in their customers’ requirements. They also have to produce software and services that meet those requirements in a usable, testable form. Since development doesn’t happen in a vacuum, you’ll also need leadership and management skills, along with a cool head under pressure.

When most DevOps hiring managers look for a new employee, they’re more concerned with mindset than with tools. If you’ve got a tech background, you’re willing to learn, and you’re an engineer at heart, you’ve already got the basics of a DevOps career.
DevOps engineers are curious, constantly improving their skillsets, and focused on lifelong learning. So while you can build the core skillset in a few months, your main driver should be on learning, with a goal of providing massive value to your next employer.
Learn to understand systems and processes, and you have the right mindset. That mindset will help you learn how to start a career in DevOps, and more important, how to be a good DevOps engineer.

It takes about six months to become a DevOps engineer, assuming you have some basic Linux admin and networking skills, and that you apply the DevOps engineer learning path outlined below. With that said, that career won’t just happen overnight. The length of time required depends on several factors, including your mindset, your current skill level, and your career position.
With that caveat, there’s no shortage of free tools and resources you can use to help you on your journey. Some professional DevOps engineering sites even offer free or vastly reduced exams to help you grow and prove your worth.  Let’s dig into how to become a DevOps engineer, starting with the tools and skills.

Let&#8217;s take a look at the DevOps Roadmap. To become a DevOps engineer, at the bare minimum you’ll need basic Linux admin and networking skills, plus some scripting fundamentals, along with the following DevOps skills:
<h3>1. Intermediate to advanced Linux skills</h3>
In DevOps, you’re not installing a server once and then logging in every now and then to perform a few admin tasks. You need to understand how to create highly customized Linux images from the ground up, both for VM and container use cases — unless you plan to become a Windows Server DevOps engineer.
<h3>2. Intermediate networking skills</h3>
In DevOps there’s no “network team.” All network resources are software-defined. In other words, networks are part of infrastructure as code. At a bare minimum, you’ll need a solid grasp on the OSI model, IPV4, subnetting, static and stateful firewalling, and DNS. These skills are usually included in advanced cloud certifications.
<h3>3. A commitment to at least one cloud</h3>
Clouds aren’t merely managed data centers. In order for you to automate workloads in a given cloud (AWS, Azure, GCP, etc.), you need a firm grasp of their specific semantics. You’ll need to know what resources are available, how they’re organized, and what properties they have. 
<h3>4. Infrastructure automation</h3>
Once you understand the resources (and their properties) applicable to a cloud, you’re ready to automate their creation using tools such as <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> and <a href="https://spacelift.io/blog/ansible-tutorial" target="_blank" rel="noopener">Ansible</a>.
<h3>5. SDLC, CI/CD pipelines, and scripting</h3>
In DevOps, we deliver infrastructure in a similar way to applications. So — you’ll need to be acquainted with the fundamentals of the software development life cycle (SDLC). This includes versioning strategies using source control code management systems like Git, and CI/CD pipelines such as Jenkins, and CircleCI. Advanced automation tasks may prove difficult through shell scripts alone. You’ll often require more powerful scripting using the likes of Python, Perl, or Ruby. 
<h3>6. Container technology</h3>
For legacy workloads you may automate the creation of a VM image. But for new applications you’ll be working with containers. As such, you need to know how to build your own Docker images (Linux skills required!) and deploy them using Kubernetes. FaaS technology like AWS Lambda also uses container technology behind the scenes.
<h3>7. Observability technology</h3>
While all clouds have monitoring dashboards and standard “telemetry” hooks, most large employers use third-party (both commercial and open source) monitoring tools such as Prometheus, DynaTrace, Datadog, or the ELK stack.
Beyond that skills list, tool-building in DevOps requires a fundamental understanding of logic, its application, and how to apply it in a computer-recognizable form. While that may sound a tad scary for the uninitiated, there are several good books that cover programming fundamentals without using any specific language. Here are a couple of my favorites:
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://www.amazon.com/_/dp/1449355730?tag=oreilly20-20" target="_blank" rel="nofollow noopener">Learning Python</a> (O’Reilly)</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://www.amazon.com/Learn-Windows-PowerShell-Month-Lunches/dp/1617291080" target="_blank" rel="nofollow noopener">Learn Windows PowerShell in a month of lunches</a> (Jones, Hicks — A personal favorite as I love PowerShell)</li>
</ul>
Now let’s dig into the nuts and bolts of how to become a DevOps engineer — starting with education.

<ul>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
<li><a href="https://spacelift.io/blog/devops-best-practices" target="_blank" rel="noopener">16 DevOps Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/terraform-vs-kubernetes" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;What is the difference between Terraform and Kubernetes?&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">What is the difference between Terraform and Kubernetes?</a></li>
</ul>

One of the great things about DevOps is that it’s about what you can do, not what qualifications you have. Some of the best DevOps engineers in the field are self-taught, with little in the way of formal higher education. The biggest requirement is motivation and an interest in DevOps engineering.
With that said, you’ll have a much easier time both learning DevOps skills and getting a company to hire you if you have a bachelor’s degree in software development, IT, or a related field.

Start your DevOps engineer roadmap by looking through the skills list above. If you already have some of those skills — great. If not, be honest about the time you’ll need to spend to learn them. But don’t stress about getting everything perfect before you start. If you wait for mastery, you’ll never get a DevOps job.
Start by learning a few of the easy-to-learn skills. If you’re already employed in a non-DevOps job, start working on some DevOps projects now, to build mastery and proof you have the skills. Then make the switch to a full-time DevOps career.
You may even find that your own company has DevOps openings you could move into. Keep a keen eye on internal and external vacancies alike.

Let’s take a deeper look now at how to become a DevOps engineer — the DevOps roadmap and how to build the skills. We’ll share the reasons each of these tools is important, and how long it’ll take to learn each one. We’ll also point you to some good online classes and certifications.
You can learn most of these skills on the job — but a word of caution. In the sink-or-swim world of DevOps career growth, different companies have different requirements. There’s no one-size-fits-all approach.

<h3>Foundation knowledge: 4 months</h3>
We’ve put a plus-sign after each of the time frames below, because while you can learn the basics quickly, mastery can take much longer.

devops-foundation-knowledge-580-x-746-px

DevOps Foundation Knowledge

<h4>1. Intermediate to Advanced Linux and Networking: 1 month+</h4>
Linux is the OS and server platform of choice for DevOps engineers in companies of any size. Linux’s open-source nature, small operational footprint, and support from the Likes of Redhat and Ubuntu make it the go-to not only for DevOps, but for tool building in general.  One of the best things about Linux is that you can download it and start using it today.
If you feel that your Linux skills are rusty, you can get started with the free course offered by <a href="https://www.udemy.com/course/learn-linux-in-5-days/" target="_blank" rel="noopener">Udemy</a>. 
In terms of networking, you’ll get the necessary skills if you do an intermediate cloud certification, such as <a href="https://aws.amazon.com/certification/certified-solutions-architect-associate/" target="_blank" rel="noopener">AWS Certified Solution Architect</a>, but it helps if you take a specialized course such as <a href="https://www.coursera.org/learn/computer-networking?specialization=google-it-support" target="_blank" rel="noopener">The Bits and Bytes of Computer Networking</a> on Coursera.
<h4>2. Advanced Scripting: 2 months+</h4>
First of all, you’ll always need shell (e.g., bash) scripting skills, because this is the default for Linux and most tools. 
For “advanced” scripting use cases, there are quite a few languages out there, but Python is a good start if you don’t know what scripting language to pick.
You can <a href="https://www.coursera.org/articles/how-long-does-it-take-to-learn-python-tips-for-learning#:~:text=In%20general%2C%20it%20takes%20around,can%20take%20months%20or%20years." target="_blank" rel="noopener">master Python in as little as two months</a> with online tutorials from <a href="https://www.learnpython.org/" target="_blank" rel="noopener">LearnPython.org</a>. However, you’ll find that many employers also use other languages such as Perl and Ruby as well, so be ready to learn those, if need be.
<h4>3. Cloud Training and Certification: 1 month+</h4>
AWS is the 600lb gorilla in terms of agile cloud providers, and AWS and Linux go together like Strawberries and cream. You’ll need to be fluent in AWS before you can call yourself part of the DevOps community.
The beauty of AWS and cloud development in general is that you only pay for what you use. That model makes cloud computing ideal for DevOps testing. You can set up an environment quickly, use it for what you need, then pull it down again.
It’s easy to start using AWS, since there’s a 12-month free version available to anyone who signs up. You can learn professional-grade skill in AWS in <a href="https://openupthecloud.com/how-long-to-learn-aws/" target="_blank" rel="noopener">as little as one month</a>, though mastery can take years of continual on-the-job use. Get your <a href="https://aws.amazon.com/certification/" target="_blank" rel="noopener">AWS certification here</a>.
<h4>4. Google Cloud: 1 month+</h4>
Azure offers similar employment opportunities to AWS, but what about GCP?
The Google Cloud Platform (GCP) is smaller than AWS and Azure but it excels particularly in data mining and artificial intelligence (and other deep learning technologies). Google’s DevOps-related offerings are becoming increasingly popular with large companies.
In the banking industry for example, the Google AI/ML tools are creating new ways of doing business, plus adding fraud detection and usage-pattern tracking. This saves huge amounts of time trying to develop similar tools in-house.
Similarly, other large companies are using Google’s ML tools to bring massive data sets down to size, drawing business-driving insights from previously unmanageable seas of data.
Want to know more about how to become a DevOps engineer with Google Cloud? You can get your <a href="https://cloud.google.com/blog/topics/training-certifications/get-google-cloud-certified-in-3-months#:~:text=Get%20Google%20Cloud%20Certified%20in%203%20months,-Rochana%20Golani" target="_blank" rel="noopener">Google Cloud certification here in three months</a>, though you can <a href="https://www.coursera.org/specializations/developing-apps-gcp" target="_blank" rel="noopener">learn to develop applications with Google Cloud in as little as one month</a>.

<h3>Skills: 1 month</h3>
It doesn’t take long to learn the DevOps skills you’ll need to succeed in your new career. All these tools are free to use and experiment with. They just require a little time and effort on your part. Let’s look at how long it takes to learn the basic DevOps tools like Terraform, Git, Docker, Jenkins, ECS, and ELK Stack.

devops-skills-580-x-563-px

DevOps Skills

<h4>1. Configure: Terraform (and Ansible) — 1 week+</h4>
Configuration management is at the heart of fast software development. Poorly configured tools waste time, while well-configured tools save it.
As its name implies, Terraform has one purpose in life — to create infrastructure as code in an automated way that speeds up your entire process.
Ansible concerns itself with server-desired state configuration, ensuring that servers are configured to specs. These two technologies are cornerstones of DevOps. Both may seem complex at first, but they’re all based around configuration files written in YAML. 
Terraform takes about a week to learn the basics. Take a look at a <a href="https://courses.morethancertified.com/p/mtc-terraform" target="_blank" rel="noopener">Terraform course from </a><a href="https://courses.morethancertified.com/p/mtc-terraform" target="_blank" rel="noopener">More than Certified</a>. You can also learn more about Ansible in <a href="https://www.exitcertified.com/it-training/topics/devops" target="_blank" rel="noopener">DevOps courses from ExitCertified</a>.
<h4>2. Version control: Git and GitHub (GitLab) — 20 minutes+</h4>
Version control is key to any DevOps endeavour. It lets DevOps Engineers and their team members create and review code faster, without wasting time sharing endless files and iterations.
Git is a standalone product that by default is used on local machines and networks. This is different from GitHub, which facilitates version control in the cloud, with the overhead managed by GitHub itself. In the world of <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">infrastructure as code</a>, version control with products like Git and GitLab are essential.
<a href="https://about.gitlab.com/" target="_blank" rel="nofollow noopener">GitLab</a> is a complete open-source DevOps platform. It helps users deliver software faster, with collaboration and security all rolled into one. Looking to learn more about how to become a DevOps engineer with Git? You can <a href="https://www.udacity.com/course/version-control-with-git--ud123" target="_blank" rel="noopener">learn the basics of Git</a> in minutes if you’re already a programmer. 
<h4>3. Package: Docker (Lambda) — 3 days+</h4>
Packaging is where build management meets release management. It’s where your code and infrastructure come together for deployment.
Without Docker there would be no DevOps. Docker essentially allows DevOps to run code in small isolated containers. That way, building services and replacing services becomes simpler than updating everything in one go (which is very non-DevOps).
Amazon’s Lambda is an alternative to Docker that many companies use instead. Though it’s best to know both tools, Docker is an excellent starting point. You can learn Docker in just a few days. More than Certified offers a solid <a href="https://courses.morethancertified.com/p/mtc-docker" target="_blank" rel="noopener">beginner’s course online for DevOps</a>. 
<h4>4. Deploy: Jenkins (CodeDeploy) — 2 days+</h4>
During deployment, you’ll take your code from version control to users of your application. Automation is a key component of this step, and Jenkins is the central way to automate.
Jenkins allows automation for all manner of tasks, including running build tests and making decisions based on whether code passes or fails the build process. You can also use Jenkins for more mundane purposes, like centralized management of scripts and executing commands via SSH (and other authentication pathways).
It’s a tool to automate those frequent and boring tasks that computers can do better than even the best DevOps engineer could. Some companies choose CodeDeploy over Jenkins, making it another useful DevOps tool to learn.
You can learn to use Jenkins in just a few days. <a href="https://courses.morethancertified.com/p/devops-in-the-cloud" target="_blank" rel="noopener">More than Certified offers a great online class</a> that bundles Terraform, Ansible and Jenkins. And see our <a href="https://spacelift.io/blog/terraform-jenkins" target="_blank" rel="noopener">How to Manage Terraform With Jenkins – Step by Step Tutorial</a>.
<h4>5. Run: ECS (Kubernetes) — 1 day</h4>
Kubernetes is DevOps bread and butter. It starts with Docker and adds extra functionality and tools. For instance, it lets the administrator ensure that several copies of a container image are running. That way, if a single VM or host is lost, the service is still available.
ECS and Kubernetes perform valuable services like this in the background. They deliver several automated DevOps tools that allow useful additions to manage containers, and their availability. They also add important items such as introducing role-based access control and more centralized auditing and management functionality.
See <a href="https://developer.ibm.com/series/kubernetes-learning-path/" target="_blank" rel="nofollow noopener">IBM’s Kubernetes learning path and guide</a> for a 13-hour course.
<h4>6. Monitor: ELK Stack (Prometheus) — 2 days</h4>
Once your new application is up and running, you’ll need a real-time view of its status, infrastructure, and services. To this end, DevOps engineers love ELK.
ELK provides all the base components for effective log management and search functionality. It’s Elasticsearch, Logstash, and Kibana — three open source applications offered by the <a href="https://www.elastic.co/what-is/elk-stack" target="_blank" rel="noopener">Elastic</a> company.
ELK takes data from multiple sources and lets you visualize it by using useful charts and graphs. Its rival platform, Prometheus, is just as important for a DevOps engineer to understand. You can learn to use the ELK Stack in just a few days with <a href="https://www.udemy.com/course/elasticsearch-logstash-kibana-learn-elasticsearch-search-server/" target="_blank" rel="noopener">Udemy’s 4-star online class</a>.
Those are the basics of how to become a DevOps engineer. In fact, if you want to learn how to become a DevOps engineer exclusively on one platform, More than Certified has <a href="https://courses.morethancertified.com/p/devops-triple-bundle" target="_blank" rel="noopener">an entire curriculum of core DevOps classes. </a>
Now let’s look into why Git matters so much, and how to get a DevOps job.

One more word on GitHub as a shortcut to starting a career in DevOps. GitHub is essentially the CV of the DevOps world. Any DevOps hiring person will check out your GitHub profile as a very first step and point of contact. Yet it’s easy to learn GitHub and other DevOps tools while you create your virtual CV at the same time.

Knowing how to become a DevOps engineer doesn’t stop with skills. The next step in your DevOps engineer career path is getting the job. That sounds daunting, but if you’ve got software development experience, the skills above, and a few DevOps achievements for your resume, you’re well on your way to getting hired.
Here’s how to get into DevOps.
<h3>Rewrite your DevOps resume</h3>
The first step in getting a DevOps job is to rewrite your resume. A DevOps resume doesn’t need to show years of experience. Start with a reverse-chronological format. Then in your work history, education, and projects section, list achievements, including:
<ul>
<li style="font-weight: 400;" aria-level="1">Development tasks from past jobs</li>
<li style="font-weight: 400;" aria-level="1">Side projects in development, IT, Agile, scripting, or automation</li>
<li style="font-weight: 400;" aria-level="1">Volunteer work in coding or distribution</li>
<li style="font-weight: 400;" aria-level="1">Projects in Terraform or other DevOps tools</li>
<li style="font-weight: 400;" aria-level="1">Linux/Unix projects</li>
<li style="font-weight: 400;" aria-level="1">Scripting in Python or Ruby</li>
<li style="font-weight: 400;" aria-level="1">Tasks completed with AWS, Jenkins, Maven, etc.</li>
</ul>
Start each resume bullet point with an action verb like developed, wrote, created, built, deployed, etc. And use numbers to show how many projects, deployments, scripts, tests, containers, and how many customers, team members, etc.
The more you show DevOps achievements in your history, with measurable details, the higher your chance of getting hired. Knowing how to become a DevOps cloud engineer is all about showing your projects and accomplishments.
<h3>Apply to lots of DevOps jobs</h3>
Finding a DevOps job is a numbers game. If you apply to three jobs, you won’t hear back from any. If you apply to 50, you’ll get a few responses and maybe an interview. Plan to hear back from about one in every 30 applications, and get interviewed by one in every 100.
In other words? You’ll need to apply to a lot of DevOps jobs. Probably something like 300 in a month to get one job (about 14 every weekday). But — you can vastly boost your chance of getting hired if you lean on networking. The easiest way? Start connecting with DevOps engineers on LinkedIn.
Then — don’t ask them for a job. Just ask if you can chat with them about their cool career. About 20% will be glad to share their success. You’ll learn tons about how to start a career in DevOps. And surprise surprise — some will even introduce you to their contacts.
<h3>Shun the unicorns</h3>
Don’t worry about being front-and-center in a DevOps job at Google, Amazon, or another giant company. Everybody clamors to get hired at those firms, creating stifling competition. Don’t be afraid to pursue a DevOps job at a less glamorous firm. Once you’ve got a little experience under your belt, then you can go unicorn hunting.
<h3>Consider working your way up</h3>
Is DevOps a good career for freshers? Not really. Don’t count on landing an entry-level DevOps job. DevOps is, by nature, an advanced position that requires highly skilled candidates. But — don’t let that discourage you. One of the best DevOps career paths is to start as a software developer or IT specialist in a company that also hires DevOps engineers.
If your current employer doesn’t hire DevOps pros, consider switching to one that does. Don’t stay entry-level for long. Three to six months is plenty. Once you’ve logged that time, commit to applying internally to DevOps positions in your new company. During your entry-level tenure, work to build accomplishments that look good on a DevOps resume.

Those are the basics of how to become a DevOps engineer. Though becoming a DevOps engineer takes persistence and passion, it’s not rocket science. Anyone with the drive (and a little time) can follow the DevOps career path, learn the necessary skills in five months, and get a DevOps job in one month. With the right skillset and job search strategy, you can be in your DevOps dream job very soon.
In the meantime, go ahead and learn how a platform like <a href="https://spacelift.io/" target="_blank" rel="noopener">Spacelift</a> can help you and your organization fully manage cloud resources within minutes.
Spacelift is a CI/CD platform for infrastructure-as-code that supports tools like Terraform, Pulumi, Kubernetes, and more. For example, it enables <a href="https://docs.spacelift.io/concepts/policy/" target="_blank" rel="noopener">policy-as-code</a>, which lets you define policies and rules that govern your infrastructure automatically. You can even invite your security and compliance teams to collaborate on and approve certain workflows and policies for parts that require a more manual approach. Get a head start with <a href="https://docs.spacelift.io/" target="_blank" rel="noopener">Spacelift’s documentation</a>.

What Are Terraform Modules and How to Use Them: Tutorial

Though becoming a DevOps engineer takes persistence and passion, it’s not rocket science. See this article if you are interested in a career in DevOps.

31-how-to-become-devops

How to Become a DevOps Engineer in 6 Months: Comprehensive Guide

Is it Worth it to Become a DevOps Engineer?

Phil Vuollet

There’s no doubt about it, working as a DevOps engineer — an engineer that enables DevOps culture — is challenging, cutting-edge, and financially rewarding. DevOps engineering is a relatively new career, with Larry-Page-level opportunities for those with the right technical skills.
It’s also highly relevant in the workplace, and it’s ideal if you’re hungry for both technological and interpersonal challenges. DevOps is an intelligent job transition if you’ve been doing software development, networking, or operations.
DevOps engineers do a lot of automation, monitoring, testing, configuring, networking, and <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code</a> (IaC). That means you’ll need to bring a variety of skills and talent to the table. Let’s face it, it’s a highly technical role that’s usually best left to experienced engineers.
There’s some opportunity in the DevOps field for entry-level engineers, but most of the jobs require experience. So, is it worth it to become a DevOps engineer? And — do you have what it takes? After reading this article, you’ll be better equipped to find out!
Interested in a career in DevOps? See our article on <a href="https://spacelift.io/blog/how-to-become-devops-engineer" target="_blank" rel="noopener">How to Become a DevOps Engineer</a>.

DevOps Engineers use their skills to deliver software quickly, continuously, and reliably. The goal of DevOps is to shorten the software delivery and feedback cycles. DevOps pros typically orchestrate multiple components in a cloud environment. They’re the ones who make sure the system keeps running smoothly, day after day.
DevOps professionals also keep things up to date with the latest changes and security updates. They automate testing, scanning, and deploying. If there’d been DevOps on the Death Star, the rebels never would have blown it up.
<h3>Where did DevOps come from?</h3>
DevOps evolved as a cultural phenomenon. Recently, companies have created specialist DevOps roles to help create and support their culture. There are so many specialized tools and skills in the IT side of every business that supporting DevOps is increasingly a full-time job.
While DevOps has certainly changed how we develop software, it has also fundamentally changed how software is deployed. This is largely thanks to the general availability and lower cost of <a href="https://spacelift.io/blog/cloud-deployment-models" target="_blank" rel="noopener">cloud infrastructure</a>, containerized environments, and the evolution of tools that enable automation.

Working in DevOps isn’t a bad way to spend your time these days. As a field, DevOps is experiencing job growth. Furthermore, the jobs in DevOps pay more than the average tech or computer science job. There’s some level of upward and horizontal mobility too.
Let’s dive a little deeper into each of the three reasons DevOps is a good career.
<h3>1) Massive DevOps job growth</h3>
To say that jobs in the DevOps field are growing would be an understatement. In fact, the Bureau of Labor Statistics (BLS) <a href="https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm#tab-1" target="_blank" rel="noopener">shows a projected growth rate</a> this decade in the software field at 22%, compared to an 8% median growth rate for all jobs.
While DevOps jobs are only part of that growth, it’s still substantial. Let’s assume, for example, that DevOps jobs growth is only 5%. With 1.8 million jobs in the field now, that 5% growth means 90,000 new DevOps jobs. And given the popularity of DevOps with employers, 5% is almost certainly a lowball estimate.
<h3>2) High salary for DevOps jobs</h3>
DevOps jobs pay very well, like many jobs in IT. If you’re into being paid comfortably and you have the right qualifications, you’re probably already looking at salaries for DevOps jobs.
<a href="https://builtin.com/salaries/dev-engineer/devops-engineer" target="_blank" rel="noopener">Builtin</a> puts the average base salary for a DevOps Engineer in the U.S. at $125 — and upwards in the $300k range. <a href="https://www.glassdoor.com/Salaries/devops-engineer-salary-SRCH_KO0,15.htm" target="_blank" rel="noopener">Glassdoor</a> pegs it closer to $105k. These are just two examples of average pay. The range is fairly wide, but still, the pay is quite nice for a 9-5 job.
Of course, there’s always some variability in the market depending on location, experience, and the company you end up working for.

avarage-devops-salary-in-the-usa-in-different-industries-2

Avarage DevOps salary in the USA in different industries

<h3>3) Upward mobility</h3>
There’s some upward mobility for DevOps Engineers depending on the company. You might find a career path from junior engineer all the way up to DevOps architect. If you have the right stuff, you may even end up managing the department.
It’s also possible to transition to similar roles like site reliability engineer (SRE). Of course, the higher you go the fewer positions are available. (The corporate pyramid leaves less room near the top.)
DevOps offers plenty of horizontal mobility too, since DevOps engineering skills usually translate between industries. Plus, it doesn’t make too much of a difference whether the company makes widgets or life insurance. The work in DevOps is generally the same.

average-devops-salary-2

Average DevOps salary

<ul>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
<li><a href="https://spacelift.io/blog/devops-best-practices" target="_blank" rel="noopener">16 DevOps Best Practices to Follow</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Automating Infrastructure Deployments Using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Automating Infrastructure Deployments Using Terraform</a></li>
</ul>

Depending on the company and your specific DevOps role, you may end up working on any number of issues. Your role may relate to automation, infrastructure, tooling, quality, or monitoring (to name a subset). The automation aspect includes code deployment, environment and infrastructure configuration, and even automatic quality checks.
The goal of all this automation is to speed up delivery without sacrificing quality or security. Much of the work is setting up monitoring and alerting for the automation and production systems. Monitoring and altering are important aspects of the feedback loops that power continuous improvement.
Read more about DevOps and security: <a href="https://spacelift.io/blog/what-is-devsecops" target="_blank" rel="noopener">DevSecOps: Making Security Central To Your DevOps Pipeline</a>

There are a few key downsides of a career in DevOps, including a high-pressure atmosphere, lack of recognition, and frustrations from butting heads with other technical teams. You may also feel like a jackrabbit in a field of turtles as you champion cutting-edge advancements while the rest of your organization slogs behind.
Most of the work in DevOps happens behind the scenes. In other words, your achievements won’t be put on display. You may get zero recognition outside your branch in the org chart. In less collaborative environments, DevOps Engineers might face frustrations with other technical teams. See our <a href="https://spacelift.io/blog/are-it-jobs-stressful" target="_blank" rel="noopener">report on stress in IT</a>.
<h3>Out on the bleeding edge</h3>
There may be situations where the goals of DevOps run counter to the goals of other groups that are slow to change or adopt complementary practices. The tools can sometimes be frustrating to work with too. And, in the real world, any number of things can go wrong at any given time.
Part of your job in DevOps is to make sure systems are running well. If something goes wrong, you may have to work outside normal hours to resolve the problem. You may feel intense pressure from extremely high expectations, especially during challenging times.

By now, you should have the answer to the key question: is DevOps a good career? It is.
Still, you might not know if DevOps is right for you. So — if you’re still on the fence, here are a few questions (and answers) to help make up your mind:
<h3>Are you interested in automation?</h3>
If you’re interested in automation, that’s one point for team DevOps. If you prefer coordinated manual deployments, maybe a DevOps environment won’t be the best fit for you. DevOps teams do a lot of automation.
<h3>Do you like the DevOps framework?</h3>
The DevOps framework is a set of behaviors that seeks close collaboration between software development and operations. If using communication, processes, and tools to tighten that loop is appealing, you’d do well in a DevOps environment.
<h3>Do you like being in charge of infrastructure?</h3>
DevOps Engineers play a Linus-Torvalds-size role in establishing and maintaining the infrastructure that runs applications. Most often, the actual infrastructure (the hardware) is provided by a cloud vendor. The work of DevOps Engineers is to configure the rented infrastructure for efficient operations of applications. It’s interesting work if you like fine tuning and solving data flow problems.

Do you have the right skills to be a DevOps engineer? The often-quoted acronym CAMS paints a picture of the spirit of DevOps as: Culture, Automation, Measurement, and Sharing.
That acronym was coined by John Willis and Damon Edwards in 2010. As Willis <a href="https://itrevolution.com/devops-culture-part-1/" target="_blank" rel="noopener">pointed out</a>, Jez Humble later added an ‘L’ for Lean. CALMS pretty much sums up the skills you need for a DevOps career.

top-8-devops-skills-580-x-435-px

top 8 devops skills

<h3>1) Interpersonal</h3>
Patrick DuBois, co-originator of DevOps, still says all problems are people problems. In Jan 2021, he still fights the good fight against organizational silos <a href="https://twitter.com/patrickdebois/status/1350116781982162946" target="_blank" rel="nofollow noopener">as mentioned in this tweet</a>. And since DevOps is a culture of sharing, it helps to have good interpersonal skills.
Part of the job is reaching out with empathy and helping others achieve their goals. And, you know — things will go wrong from time to time. When they do, your interpersonal skills will go a long way toward helping you come out on top.
<h3>2) A programming language</h3>
If you’re thinking about a career in DevOps, you should know how to code, at least at a basic level. Much of the work involves writing shell scripts or using a coding language such as Python to automate tasks. You may need to write Python scripts to call various APIs, or use shell scripts to manipulate files.
Some DevOps roles require knowledge of VB-Script and Windows PowerShell. Either way, you should have a foundation of coding, so you can learn the nuts and bolts of your environment.
<h3>3) Automated build/test</h3>
One of the core principles of DevOps is automation. Automating builds is a human-genome-sized step toward consistency, reliability, and traceability. Once that’s done, it’s time to automate the tests for that build. Sometimes this demands a big shift in thinking and a whole lot of work.
Still, it’s a big win when you can get to automated build and test. DevOps culture centers around this kind of win. You may write tests using Selenium, Postman, the Robot Framework, or other types of automated testing tools. You’ll have to incorporate these tools into the automated release process.
See our list of <a href="https://spacelift.io/blog/devops-tools" rel="">+70 useful DevOps tools</a>.
<h3>4) Containers</h3>
In all likelihood, you’ll use containers in your career as a DevOps Engineer. Many organizations are either using or switching to containers (mainly Docker) for several types of workloads. A container is an isolated runtime environment that includes most of an operating system. It runs on a server or VM. You can launch containers quickly, making them ideal for running tasks fast — a key focus in DevOps work.
<h3>5) Configuration</h3>
Configuration is another task DevOps Engineers spend lots of time on. Actually, most of the work in automating is configuring the systems that perform the work. Even the infamous IaC (infrastructure as code) is really about configuring the infrastructure and environment using a specification language.
DevOps cultures prefer using IaC because IaC tracks changes. That lets DevOps pros measure the effects (remember: CALMS). There’s nothing worse than tracking down a problem without knowing a configuration changed! In DevOps, you want to monitor events like Tony Stark.
<h3>6) Version control</h3>
Along with one of the tools you’ll use frequently as a DevOps engineer — the Linux-based operating system — you’ll most likely use another of Linus Torvald’s inventions — git! Or you might use another version control system like subversion or TFS. They all do the same general thing: keep track of changes to the codebase.
For automated delivery pipelines, you’ll keep the configuration under version control. You’ll store IaC files in VCS as well. And often, a VCS is hosted on a platform that runs a pipeline and can be configured to kick off automation. The VCS is also the heart of a change-management system in a DevOps world.
<h3>7) Package</h3>
Developers from different groups or projects often need the same types of features. They might need logging, authentication, or any kind of functionality plugged into a software development environment. DevOps culture embraces sharing (the “S” in CALMS), so its denizens will often “package” functionality.
DevOps engineers might even dive in and create these packages to share. Or, they may just manage package repositories both public and private. Note that package repositories are another possible destination for delivery.
<h3>8) Networking</h3>
Ahhh, networking! If a problem isn’t a code error, it’s probably something in the network. Networking is another critical skill in a DevOps role. It could be enough to know the IP protocol, but you’ll often need an understanding of certificates and routing, too.
Even in a cloud environment, it’s important to know how networks and subnetworks operate. And even containers have their own type of network configuration to consider.
Take a look at our list of <a href="https://spacelift.io/blog/devops-books" target="_blank" rel="noopener">books for DevOps, Site Reliability, and Cloud Engineers &amp; Architects</a> to advance your career to the next level.

You still wondering if DevOps is right for you? See our article and find out whether it is worth it to become a DevOps engineer in 2023.

Infrastructure Drift Detection and How to Fix It With IaC Tools

Get the newsletter

Recent posts

CI/CD Pipeline : Everything You Need To Know with Examples

What is Pulumi? Key Concepts and Features Overview

SRE vs. DevOps: What’s the Difference Between Them?

How to Manage Infrastructure as Code at Scale (Examples)

How to Become a DevOps Engineer in Six Months

Who is DevOps? Is it Worth it to Become a DevOps Engineer?

Infrastructure as Code : Best Practices, Benefits & Examples

Common Patterns of Infrastructure as Code Architecture: Terraform & Terragrunt

What is Multi-Cloud Infrastructure? Benefits, Best Practices

Spacelift raises $15m in Series B funding from Insight Partners

Business Benefits of Infrastructure as Code