General | Spacelift Blog

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

General

In this post, I will run through some best practices to use when using Kubernetes (K8s).
As the most popular container orchestration system, K8s is the de-facto standard for the modern cloud engineer to get to grips with. K8s is a notoriously complex system to use and maintain, so getting a good grasp of what you should and should not be doing, and knowing what is possible will get your deployment off to a solid start.
These recommendations cover common issues within 3 broad categories, application development, governance, and cluster configuration.
<h3 id="b48f" class="lq lr ip bn ls lt lu lv lw lx ly lz ma kn mb mc md kr me mf mg kv mh mi mj mk gt">Best practices contents</h3>
<ol class="">
<li id="d51a" class="ml mm ip kg b kh mn kk mo kn mp kr mq kv mr kz ms mt mu mv gt" data-selectable-paragraph="">Use namespaces</li>
<li id="10ef" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Use readiness and liveness probes</li>
<li id="39ca" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Use resource requests and limits</li>
<li id="b033" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Deploy your Pods as part of a Deployment, DaemonSet, ReplicaSet or StatefulSet across nodes.</li>
<li id="257b" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Use multiple nodes</li>
<li id="3469" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Use Role-based access control (RBAC)</li>
<li id="3bde" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Host your Kubernetes cluster externally (use a cloud service)</li>
<li id="8020" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Upgrade your Kubernetes version</li>
<li id="aceb" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Monitor your cluster resources and audit policy logs</li>
<li id="1c77" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Use a version control system</li>
<li id="5eae" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Use a Git-based workflow (GitOps)</li>
<li id="00e8" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Reduce the size of your containers</li>
<li id="5459" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Organize your objects with labels</li>
<li id="8a71" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Use network policies</li>
<li id="eecf" class="ml mm ip kg b kh mw kk mx kn my kr mz kv na kz ms mt mu mv gt" data-selectable-paragraph="">Use a firewall</li>
</ol>

Namespaces in K8s are important to utilize in order to organize your objects, create logical partitions within your cluster, and for security purposes. By default, there are 3 namespaces in a K8s cluster, <code class="gs nl nm nn no b">default</code>, <code class="gs nl nm nn no b">kube-public</code> and <code class="gs nl nm nn no b">kube-system</code>.
RBAC can be used to control access to particular namespaces in order to limit the access of a group to control the blast-radius of any mistakes that might occur, for example, a group of developers may only have access to a namespace called <code class="gs nl nm nn no b">dev</code> , and have no access to the <code class="gs nl nm nn no b">production</code> namespace. The ability to limit different teams to different namespaces can be valuable to avoid duplicated work or resource conflict.
LimitRange objects can also be configured against namespaces to define the standard size for a container deployed in the namespace. ResourceQuotas can also be used to limit the total resource consumption of all containers inside a Namespace. Network policies can be used against namespaces to limit traffic between pods.

Readiness and Liveness probes are essentially types of health checks. These are another very important concept to utilize in K8s.
Readiness probes ensure that requests to a pod are only directed to it when the pod is ready to serve requests. If it is not ready, then requests are directed elsewhere. It is important to define the readiness probe for each container, as there are no default values set for these in K8s. For example, if a pod takes 20 seconds to start and the readiness probe was missing, then any traffic directed to that pod during the startup time would cause a failure. Readiness probes should be independent and not take into account any dependencies on other services, such as a backend database or caching service.
Liveness probes test if the application is running in order to mark it as healthy. For example, a particular path of a web app could be tested to ensure it is responding. If not, the pod will not be marked as healthy and the probe failure will cause the <code class="gs nl nm nn no b">kubelet</code>to launch a new pod, which will then be tested again. This type of probe is used as a recovery mechanism in case the process becomes unresponsive.

Where it is appropriate, autoscaling can be employed to dynamically adjust the number of pods (horizontal pod autoscaler), the amount of resources consumed by the pods (vertical autoscaler), or the number of nodes in the cluster (cluster autoscaler), depending on the demand for the resources.
The horizontal pod autoscaler can also scale a replication controller, replica set, or stateful set based on CPU demand.
Using scaling also brings some challenges, such as not storing persistent data in the container&#8217;s local filesystem, as this would prevent horizontal autoscaling. Instead, a PersistentVolume could be used.
The cluster autoscaler is useful when highly variable workloads exist on the cluster that may require different amounts of resources at different times based on demand. Removing unused nodes automatically is also a great way to save money!

Resource requests and limits (minimum and maximum amount of resources that can be used in a container) should be set to avoid a container starting without the required resources assigned, or the cluster running out of available resources.
Without limits, pods can utilize more resources than required, causing the total available resources to be reduced which may cause a problem with other applications on the cluster. Nodes may crash, and new pods may not be able to be placed corrected by the scheduler.
Without requests, if the application cannot be assigned enough resources, it may fail when attempting to start or perform erratically.
Resource requests and limits define the amount of CPU and Memory available in millicores and mebibytes. Note that if your process goes over the memory limit, the process is terminated, so it may not always be appropriate to set this in all cases. If your container goes over the CPU limit, the process is throttled.

A single pod should never be run individually. To improve fault tolerance, instead, they should always be part of a Deployment, DaemonSet, ReplicaSet or StatefulSet. The pods can then be deployed across nodes using anti-affinity rules in your deployments to avoid all pods being run on a single node, which may cause downtime if it was to go down.

Running K8s on a single node is not a good idea if you want to build in fault tolerance. Multiple nodes should be employed in your cluster so workloads can be spread between them.

Using RBAC in your K8s cluster is essential to properly secure your system. Users, Groups, and Service accounts can be assigned permissions to perform permitted actions on a particular namespace (a Role), or to the entire cluster (ClusterRole). Each role can have multiple permissions. To tie the defined roles to the users, groups, or service accounts, RoleBinding or ClusterRoleBinding objects are used.
RBAC roles should be set up to grant using the principle of least privilege, i.e. only permissions that are required are granted. For example, the admins group may have access to all resources, and your operators group may be able to deploy, but not be able to read secrets.

Hosting a K8s cluster on your own hardware can be a complex undertaking. Cloud services offer K8s clusters as platform as a service (PaaS), such as AKS (Azure Kubernetes Service) on Azure, or EKS (Amazon Elastic Kubernetes Service) on Amazon Web Services. Taking advantage of this means the underlying infrastructure will be managed by your cloud provider, and tasks around scaling your cluster, such as adding and removing nodes can be much more easily achieved, leaving your engineers to the management of what is running on the K8s cluster itself.

As well as introducing new features, new K8s versions also include vulnerability and security fixes, which make it important to run an up-to-date version of K8s on your cluster. Support for older versions will likely not be as good as newer ones.
Migrating to a new version should be treated with caution however as certain features can be depreciated, as well as new ones added. Also, the apps running on your cluster should be checked that they are compatible with the newer targeted version before upgrading.

Monitoring the components in the K8s control plane is important to keep resource consumption under control. The control plane is the core of K8s, these components keep the system running and so are vital to correct K8s operations. Kubernetes API, kubelet, etcd, controller-manager, kube-proxy and kube-dns make up the control plane.
Control plane components can output metrics in a format that can be used by Prometheus, the most common K8s monitoring tool.
Automated monitoring tools should be used rather than manually managing alerts.
Audit logging in K8s can be turned on whilst starting the kube-apiserver to enable deeper investigation using the tools of your choice. The audit.log will detail all requests made to the K8s API and should be inspected regularly for any issues that might be a problem on the cluster. The Kubernetes cluster default policies are defined in the audit-policy.yaml file and can be amended as required.
A log aggregation tool such as Azure Monitor can be used to send logs to a log analytics workspace from AKS for future interrogation using Kusto queries. On AWS Cloudwatch can be used. Third-party tools also provide deeper monitoring functionality such as Dynatrace and Datadog.
Finally, a defined retention period should be in place for the logs, around 30–45 days is common.

K8s configuration files should be controlled in a version control system (VCS). This enables a raft of benefits, including increased security, enabling an audit trail of changes, and will increase the stability of the cluster. Approval gates should be put in place for any changes made so the team can peer-review the changes before they are committed to the main branch.

Successful deployments of K8s require thought on the workflow processes used by your team. Using a git-based workflow enables automation through the use of CI/CD (Continuous Integration / Continuous Delivery) pipelines, which will increase application deployment efficiency and speed. CI/CD will also provide an audit trail of deployments. Git should be the single source of truth for all automation and will enable unified management of the K8s cluster. You can also consider using a dedicated infrastructure delivery platform such as <a class="au lp" href="https://spacelift.io/" target="_blank" rel="noopener">Spacelift</a>, which recently introduced Kubernetes support.

Smaller image sizes will help speed up your builds and deployments and reduce the amount of resources the containers consumed on your K8s cluster. Uneccesery packages should be removed where possible, and small OS distribution images such as Alpine should be favored. Smaller images can be pulled faster than larger images, and consume less storage space.
Following this approach will also provide security benefits as there will be fewer potential vectors of attack for malicious actors.

K8s labels are key-value pairs that are attached to objects in order to organize your cluster resources. Labels should be meaningful metadata that provide a mechanism to track how different components in the K8s system interact.
Recommended labels for pods in the <a class="au lp" href="https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/" target="_blank" rel="noopener ugc nofollow">official K8s documentation</a> include <code class="gs nl nm nn no b">name, instance, version, component, part-of,</code> and <code class="gs nl nm nn no b">managed-by</code>.
Labels can also be used in a similar way to using tags in a cloud environment on resources in order to track things related to the business, such as object <code class="gs nl nm nn no b">ownership</code>, and the <code class="gs nl nm nn no b">environment</code>an object should belong to.
Also recommended is to use labels to detail security requirements, including <code class="gs nl nm nn no b">confidentiality</code> and <code class="gs nl nm nn no b">compliance</code>.

Network policies should be employed to restrict traffic between objects in the K8s cluster. By default, all containers can talk to each other in the network, something that presents a security risk if malicious actors gain access to a container, allowing them to traverse objects in the cluster. Network policies can control traffic at the IP and port level, similar to the concept of security groups in cloud platforms to restrict access to resources. Typically, all traffic should be denied by default, then allow rules should be put in place to allow required traffic.

As well as using network policies to restrict internal traffic on your K8s cluster, you should also put a firewall in front of your K8s cluster in order to restrict requests to the API server from the outside world. IP addresses should be whitelisted and open ports restricted.

Following the best practices listed in this article when designing, running, and maintaining your Kubernetes cluster will put you on the path to success on your modern application journey!

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities s for infrastructure management.

How to Provision an EKS Kubernetes Cluster with Terraform

Infrastructure as Code : Best Practices, Benefits & Examples

Initial Availability of Kubernetes Integration in Spacelift

Follow the best practices for Kubernetes listed in this article for painless and efficient maintenance of your K8s cluster.

73-kubernetes_best_practices_1

Kubernetes Best Practices

15 Kubernetes Best Practices Every Developer Should Know

In the last two years teams have accelerated their dependency on the public cloud. Whether you are scaling an already existing product or building a new product that is native to the cloud, chances are you’ve felt that there has got to be a better way to build cloud infrastructure. There is. Enter Pulumi.

Pulumi is an infrastructure-as-code platform for full-stack developers and cloud engineers who are interested in using a general-purpose programming language for their cloud resources. Pulumi isn’t just one thing. It’s an ecosystem of components that all work together. At a high-level these are the components that you are most likely to hear a lot about:
<ul>
<li style="font-weight: 400;">Provider SDKs</li>
<li style="font-weight: 400;">CLI</li>
<li style="font-weight: 400;">Service backend</li>
</ul>
SDKs are installable packages your program consumes using the respective language’s package registry that you are more than likely already familiar with and probably already work with. Today, Pulumi supports four highly <a href="https://insights.stackoverflow.com/survey/2021#most-popular-technologies-language" target="_blank" rel="noopener">popular programming languages</a>.
<ul>
<li style="font-weight: 400;"><a href="https://www.pulumi.com/docs/intro/languages/go/" target="_blank" rel="nofollow noopener">Go</a></li>
<li style="font-weight: 400;"><a href="https://www.pulumi.com/docs/intro/languages/javascript/" target="_blank" rel="nofollow noopener">JavaScript/TypeScript</a></li>
<li style="font-weight: 400;"><a href="https://www.pulumi.com/docs/intro/languages/python/" target="_blank" rel="nofollow noopener">Python</a></li>
<li style="font-weight: 400;"><a href="https://www.pulumi.com/docs/intro/languages/dotnet/" target="_blank" rel="nofollow noopener">.NET Platform</a> (C#, F#, VB)</li>
</ul>
What that means is, you can create cloud resources using general-purpose code; think constructs such as variables, loops, functions, classes (well, at least in OOP), strong-type support, code reuse, and everything else you can do in those programming languages. It also gives you the option to maintain a homogeneous codebase, if that’s important to your team. For example, a team working on a C# .NET app may be interested in implementing their cloud infrastructure also in C# .NET. Or how about a team working on a React-based app that is strong in JS and the Node.js ecosystem wanting to use JS for their cloud infrastructure too?
Let’s examine each of those components in detail.
<h3>Provider SDKs</h3>
There are over <a href="https://www.pulumi.com/registry" target="_blank" rel="nofollow noopener">75 providers</a> with new ones being added regularly. And you can contribute new ones too. All of the SDKs are available in all of the supported languages. Extension libraries are typically not supported in every one of the supported languages since they are typically manually authored and in a language based on community demand. For example, the official Pulumi EKS is a popular <a href="https://www.pulumi.com/docs/intro/concepts/resources/components/" target="_blank" rel="nofollow noopener">Pulumi component</a> library that is supported in JS/TS and Python. But wait! The EKS library is also a great example of how you can <a href="https://www.pulumi.com/blog/create-eks-clusters-in-your-favorite-language/" target="_blank" rel="nofollow noopener">author and use multi-language components</a> in the Pulumi ecosystem. More on multi-language components later.
<h3>CLI</h3>
The CLI is one of the ways users can run previews and updates on their infrastructure programs. The CLI supports several powerful commands to mutate the state of your cloud resources. There is another powerful way to automate execution of your Pulumi apps.
<h3>Service Backend</h3>
The Pulumi Service backend is the default state storage service for all Pulumi projects. What this means is, every time you update your cloud resources using Pulumi, a snapshot of the state of your resources known as the checkpoint is stored by the service. The checkpoint allows you to confidently make changes to your infrastructure.

As mentioned before several components work together in the Pulumi ecosystem to mutate cloud resources. Let’s examine this with an example. Say, we want to create a <a href="https://www.pulumi.com/registry/packages/digitalocean/api-docs/spacesbucket/#example-usage" target="_blank" rel="nofollow noopener">Bucket</a> using DigitalOcean Spaces. We’ll use JS as an example for demonstration purposes.

Note that I’ve skipped the part about bootstrapping a new project, just like with creating any other new app Pulumi requires some minimal project setup too. This can be done easily with <code>pulumi new</code> which lets you pick a template to start from.
In order to create the Bucket in a DigitalOcean account, we’ll need an account and <a href="https://www.pulumi.com/registry/packages/digitalocean/installation-configuration/#spaces-api-credentials" target="_blank" rel="nofollow noopener">an access key for Spaces</a> so that Pulumi can authenticate on our behalf to mutate resources. We’ll assume that a token is created and configured correctly. Now that we have authentication with DigitalOcean squared-away we can run <code>pulumi preview</code> from the terminal (or an IDEs integrated terminal.)

image2-12

pulumi preview

The Pulumi CLI “processed” the <code>index.js</code> file and generated a graph of the resources that needed to be created (or deleted, updated…or do nothing if nothing has changed.) The graph, in this case, is simple. We just have a single bucket that we want to create. Behind-the-scenes, the CLI uses this graph to calculate the set of CRUD operations using the DigitalOcean API.
Running <code>pulumi up -y</code> will auto-approve Pulumi to go ahead and create the bucket.

image1-12

pulumi up -y

Using a general-purpose programming language means all language features are also implicitly Pulumi’s features too. Aside from that, there is a rich feature set to help with an ever-evolving cloud infrastructure landscape.
<h3>ComponentResources</h3>
<a href="https://www.pulumi.com/docs/intro/concepts/resources/components/" target="_blank" rel="nofollow noopener">ComponentResources</a> are resources that encapsulate multiple child resources. In simple terms, think of them as a class that you can create that lets you represent several related resources as a single unit. For example, you might be interested in creating a component that handles setting up an S3 bucket with replication in a secondary region for disaster recovery scenarios. You could create a component that accepts as input the bucket name but behind-the-scenes creates two buckets – one in the primary region and the other in the failover region. Oh, but wait! There is already a <a href="https://www.pulumi.com/registry/packages/aws-s3-replicated-bucket/" target="_blank" rel="nofollow noopener">component</a> for that.
<h3>StackReferences</h3>
As your infrastructure evolves, you may be interested in creating multiple stacks with certain stacks needing to consume the outputs exported by other “upstream” stacks. This is easy to do with <a href="https://www.pulumi.com/docs/intro/concepts/stack/#stackreferences" target="_blank" rel="nofollow noopener">StackReferences</a>. A quick example:

<h3>Dynamic Providers</h3>
Although it’s limited to specific languages, it can serve as sort of an escape-hatch for when you need to mimic the CRUD lifecycle of a resource but there is no official provider for that resource. A great example of dynamic providers would be to answer the question “<a href="https://github.com/pulumi/examples/tree/master/aws-ts-ec2-provisioners" target="_blank" rel="nofollow noopener">how do I configure a newly provisioned VM?</a>” 
<h3>Automation API</h3>
When I talked about the CLI, I said it is one of the ways to run Pulumi apps on users’ machines as well as in CI/CD pipelines. <a href="https://www.pulumi.com/docs/guides/automation-api/" target="_blank" rel="nofollow noopener">Automation API</a> allows users to programmatically invoke the CLI operations. This unlocks a whole another set of possibilities. Pulumi’s customers have been creative with this feature and have fully exploited it to enable advanced deployment scenarios for their organizations.

Alternatives vary based on which aspects of Pulumi you would like to compare.
<h3>Other General-Purpose Programming Language Tools</h3>
There’s <a href="https://aws.amazon.com/cdk/" target="_blank" rel="nofollow noopener">AWS CDK</a>. It’s similar to Pulumi on the surface. However, AWS CDK is limited to… well, AWS. It is also fundamentally different in how it works behind-the-scenes. As a user provisioning resources on AWS you may not need to worry about the implementation details of the technology ever for your use case.
<h4>AWS CDK vs. Pulumi:</h4>
<table>
<tbody>
<tr>
<td>CDK</td>
<td>Pulumi</td>
</tr>
<tr>
<td>Supports popular languages
&nbsp;</td>
<td>Supports popular languages
&nbsp;</td>
</tr>
<tr>
<td>Transpiles to CloudFormation template
&nbsp;</td>
<td>Doesn’t transpile to any proprietary format
&nbsp;</td>
</tr>
<tr>
<td>Cannot natively provision non-AWS resources
&nbsp;</td>
<td>Can use any of the 70+ providers as well author your own for edge-cases using Dynamic Providers or contribute your own
&nbsp;</td>
</tr>
<tr>
<td>Can create custom encapsulations using CdkContructs that create other AWS resources or Constructs
&nbsp;</td>
<td>Can create custom encapsulations using ComponentResources that can
&nbsp;</td>
</tr>
<tr>
<td>Has a registry showcasing community-supported as well as AWS-supported Constructs in the <a href="https://constructs.dev/" target="_blank" rel="nofollow noopener">Construct Hub</a>
&nbsp;</td>
<td>Has a registry showcasing community-supported as well as Pulumi-supported components and packages in the <a href="https://registry.pulumi.com" target="_blank" rel="nofollow noopener">Pulumi Registry</a>
&nbsp;</td>
</tr>
</tbody>
</table>
&nbsp;

<h3>Other Infrastructure-as-Code Tools</h3>
IaC isn&#8217;t restricted to just general-purpose programming language-based tools. In fact, one of the most common IaC tool that Pulumi is often compared with is Terraform. HashiCorp&#8217;s Terraform has existed in the IaC space well before Pulumi was created. If you look deeper Pulumi extends (the official word is &#8220;bridge&#8221;, however) many of Terraform&#8217;s providers as well as has native providers that are direct implementations from Open API specs. They are conceptually similar in that both tools want you to think about everything in the cloud as a &#8220;resource&#8221;. The resource model is what drives the core of your infrastructure state. Every resource has a CRUD lifecycle.
As much as they are similar in concept, they are also very different. The fundamental difference is the choice that you, as a developer, have when it comes to choosing the language that suits you and your team best. There are other differences too, of course.
<h4>Terraform vs. Pulumi:</h4>
<table>
<tbody>
<tr>
<td>Terraform</td>
<td>Pulumi</td>
</tr>
<tr>
<td>Uses proprietary but open-source language HCL which stands for HashiCorp Configuration Language
&nbsp;</td>
<td>Supports popular open-source, general-purpose programming languages
&nbsp;</td>
</tr>
<tr>
<td>HCL is a type of domain-specific language
&nbsp;</td>
<td>None of the supported languages are DSLs
&nbsp;</td>
</tr>
<tr>
<td>By definition, DSLs are limited in their language feature set
&nbsp;</td>
<td>Out-of-box support for language-level features that are not custom-made by Pulumi
&nbsp;</td>
</tr>
<tr>
<td>Has a registry showcasing community-supported as well as Terraform-supported Constructs
&nbsp;</td>
<td>Has a registry showcasing community-supported as well as Pulumi-supported components and packages
&nbsp;</td>
</tr>
<tr>
<td>No escape-hatch like functionality to create self-managed resources in cases where a provider doesn&#8217;t support a feature
&nbsp;</td>
<td>Dynamic Provider feature to model the CRUD lifecycle of custom one-off resources
&nbsp;</td>
</tr>
</tbody>
</table>
&nbsp;

<h3>Others</h3>
Some of the biggest cloud providers have their own proprietary tools for implementing cloud infrastructure.
<ul>
<li style="font-weight: 400;"><a href="https://aws.amazon.com/cloudformation/features/?pg=ln&amp;sec=hs" target="_blank" rel="nofollow noopener">AWS CloudFormation</a> &#8211; A JSON/YAML-based authoring experience for provisioning AWS resources. AWS CDK which lets you author infrastructure for AWS resources transpiles to a CloudFormation template and then deploys the infrastructure using CloudFormation.</li>
<li style="font-weight: 400;"><a href="https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview?tabs=bicep" target="_blank" rel="nofollow noopener">Azure Bicep</a> &#8211; Microsoft Azure’s version of a DSL IaC tool that is meant to replace ARM templates.</li>
<li><a href="https://cloud.google.com/deployment-manager/docs/#docs" target="_blank" rel="nofollow noopener">GCP Deployment Manager</a> &#8211; a deployment service whose primary configuration language is YAML. In some cases documentation is scarce and feature support is very limited.</li>
</ul>

Let’s examine a few concerns I have heard developers express when it comes to adopting a fundamentally new way to implement cloud infrastructure with a tool like Pulumi.
<h3>Deciding On A Language</h3>
Picking a language in a new tool can feel like a pretty drastic one-way decision. Fortunately, Pulumi interoperates with the supported languages. Delving into how <a href="https://www.pulumi.com/blog/pulumiup-pulumi-packages-multi-language-components/" target="_blank" rel="nofollow noopener">multi-language components</a> work perhaps deserves its own post some day, so I’ll leave that as a task for you to read 🙂
If I might dare say that being stuck with a general-purpose programming language is still better than having to learn a cloud provider’s DSL (templates or otherwise.) Learning a new language or technology has a cost. That cost is especially high when members of your team may not spend a lot of time with infrastructure on a daily basis.
<h3>DSL Is Easier/Quicker</h3>
One of the strengths for any DSL is its succinctness. It’s one of the reasons why DSLs are great for all types of configuration management. I think this is a matter of perspective. Developer skills are highly varied and everyone has their sweet spot of languages they are comfortable with. I mean, why do we need anything other than JS, am I right? Just kidding. On a more serious note, many developers are indeed more comfortable using DSL-based technologies such as Terraform for their cloud automation needs. While many others want something that is familiar and can be supported by any developer with language familiarity.
<h3>State Storage</h3>
One of the nice features of Pulumi is that the Pulumi Service stores the state for your infrastructure, by default. Right from the moment you run <code>pulumi stack init &lt;stackName&gt;</code>. But if things change and you would like to manage the state on your own, Pulumi does support self-managed backends too. That means, you could export your state and store it on your own inside one of the supported <a href="https://www.pulumi.com/docs/intro/concepts/state/#logging-in" target="_blank" rel="nofollow noopener">self-managed backends</a> such as AWS S3, Azure Storage or GCS.
<h3>New Technology/Never Heard Of It</h3>
Pulumi has been in service <a href="https://www.pulumi.com/blog/introducing-pulumi-a-cloud-development-platform/" target="_blank" rel="nofollow noopener">since 2018</a>. A number of <a href="https://www.pulumi.com/case-studies/" target="_blank" rel="nofollow noopener">well-known orgs</a> use Pulumi. During my time at Pulumi I had come across several teams that have built complex cloud infrastructure installations spanning several cloud providers. But not all use cases are complex and yours doesn’t have to be to use Pulumi. Many of them start simple but grow organically and don’t realize that the growth was facilitated by Pulumi.
Also by reading this post this changes for you today 🙂

Instead of asking why, you should ask yourself why not Pulumi? Even if you know the language well, cloud infrastructure is complex (want to create a private subnet with a NAT gateway? Wait, what’s a NAT and why do I need a private subnet? Exactly.) Pulumi helps reduce some of that complexity by removing the cognitive load that comes from familiarizing yourself with a DSL or a template system and in some cases actually helps address the complexity associated with cloud infrastructure with packages like the <a href="https://www.pulumi.com/docs/guides/crosswalk/aws/" target="_blank" rel="nofollow noopener">Crosswalk for AWS</a> or the <a href="https://www.pulumi.com/docs/tutorials/cloudfx/" target="_blank" rel="nofollow noopener">Pulumi Cloud</a> framework, but does not eliminate it. You’ll discover that as your cloud infrastructure needs evolve, Pulumi’s capabilities scale well with that.

Once you’ve got your Pulumi stack configured, you might be wondering, well now what? It is pretty typical to wire-up a CI/CD pipeline to run the same Pulumi commands that you ran on your machine. Pulumi integrates really well with many CI/CD services.
This is where Spacelift excels. <a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener">Spacelift</a> is the most flexible management platform for Infrastructure as Code. Spacelift provides excellent capabilities to automate running your Pulumi stacks. That means you don’t have to configure your own CI pipelines. Especially if you have several stacks and want to connect them together. And it does not stop there.
Spacelift allows you to configure your workflows with <a href="https://spacelift.io/blog/what-is-open-policy-agent-and-how-it-works" target="_blank" rel="noopener">OPA</a> policies. For example, <a href="https://docs.spacelift.io/concepts/policy/trigger-policy" target="_blank" rel="noopener">apply ACLs</a> on stacks or detect changes to upstream stacks and trigger dependent downstream stacks.
Drift usually occurs when people manually modify resources, outside of Pulumi. Spacelift can detect it automatically, and optionally remediate it.
Hopefully, this post has given you many reasons to try out Pulumi. There are plenty of resources out there to get you started but we would recommend starting with the <a href="https://www.pulumi.com/docs/get-started/" target="_blank" rel="nofollow noopener">Pulumi</a> and <a href="https://docs.spacelift.io/vendors/pulumi" target="_blank" rel="noopener">Spacelift</a> documentations.

Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure. It helps overcome common state management issues and adds several must-have features for infrastructure management.

Infrastructure Problems that Spacelift Solves

5 Ways to Manage Terraform at Scale - Best Practices

How a Specialized Management Platform Can Improve Your IaC

In this article you'll learn what Pulumi is, how it works and why it can be helpful for your Infrastructure as Code (IaC).

61-pulumi

What is Pulumi?

What is Pulumi? Key Concepts and Features Overview

DevOps is a set of practices where development, IT operations, quality, and security are glued to Continuous Integration/Continuous Delivery (CI/CD) to deliver a reliable product to end customers. DevOps culture facilitates the collaboration of the Core Development and Operation team, allowing companies to reduce organization silos, expect and handle failure as part of the process, implement gradual changes, leverage automation tools, and at last, measure everything.
Whereas Site Reliability Engineering (SRE) is responsible for implementing the product developed by the Core Development team. The key objective of the SREs is to implement and automate DevOps practices to reduce the level of incidents and improve reliability and scalability. SREs are also responsible for sending swift and constant feedback to the Development team based on performance metrics – availability, latency, efficiency, capacity, and incident.

While DevOps is all about what aspect of the matters, SRE talks about the how part of it all. Nevertheless, there are a few other differences between the two. 
<ol>
<li style="font-weight: 400;" aria-level="1">Implementing New Features – DevOps is responsible for implementing the new features request to a product, whereas SREs ensure those new changes don’t increase the overall failure rates in production.</li>
<li style="font-weight: 400;" aria-level="1">Process Flow – A DevOps team has a perspective of the development environment to put changes from development to production. On the other hand, SREs have a perspective of production, so they can make suggestions to the development team to limit the failure rates despite the new changes.</li>
<li style="font-weight: 400;" aria-level="1">Focus – DevOps’s primary focus is on continuity and speed of product development, whereas SRE’s main focus is on the system’s reliability, scalability, and availability.</li>
<li style="font-weight: 400;" aria-level="1">Team Structure – A typical DevOps team consists of professionals with dedicated roles and responsibilities such as – Product Owner, Team Lead, Cloud Architect, Software Developer, QA Engineer, Release Manager, System Administrator. In contrast, SREs have a team of engineers with operational and development skills set.</li>
</ol>

Although there is some overlap in the job roles of SREs and DevOps, there is wide segregation of functions:
<table>
<tbody>
<tr>
<td></td>
<td>DevOps</td>
<td>SREs</td>
</tr>
<tr>
<td>
Role
&nbsp;</td>
<td>The main role of the DevOps team is to solve the development problem, build solutions to cater to business requirements.
&nbsp;</td>
<td>SREs’ main role is to deal with operational problems, for example – production failures, infrastructure issues (disk, memory), security, monitoring.
&nbsp;</td>
</tr>
<tr>
<td>Focus
&nbsp;</td>
<td>Focus on product development with Continuous Integration/ Continuous Delivery.
&nbsp;</td>
<td>SREs put more focus on resilience, scaling, reliability, uptime, and robustness.
&nbsp;</td>
</tr>
<tr>
<td>Tools
&nbsp;</td>
<td>In the DevOps role, the most widely used tools are – Integrated Development Environment (IDEs) for development purposes, Jenkins for Continuous Integration and Development, JIRA for change management, Splunk for log monitoring, SVN, GitHub.
&nbsp;</td>
<td>In the SRE role, the most widely used tools are Prometheus and Grafana for collecting and visualizing the different metrics (CPU usage, memory, disk space, etc.), incident alert tools (OP5, PageDuty, xMatters, etc.), Ansible, Puppet, or Chef, Kubernetes and Docker for container orchestration, cloud platform AWS, GCP, Azure, JIRA, SVN, GitHub.
&nbsp;</td>
</tr>
<tr>
<td>Bug reporting
&nbsp;</td>
<td>DevOps team is responsible for debugging the code in case of any bug reported in the end product.
&nbsp;</td>
<td>The SRE team is responsible for reporting the bug to the Core development team and does not get involved in debugging unless it is a production outage. SRE team is also responsible for debugging and fixing the infrastructure issues.
&nbsp;</td>
</tr>
<tr>
<td>Measurement metrics
&nbsp;</td>
<td>Typical measurement metrics for the DevOps role are Deployment Frequency and the Deployment Failure rate.
&nbsp;</td>
<td>Typical measurement metrics for the SRE role are Error Budgets, SLOs (Service Level Objective), SLIs (Service Level Indicator), SLAs (Service Level Agreement).
&nbsp;</td>
</tr>
<tr>
<td>Incident handling
&nbsp;</td>
<td>DevOps teams work on the incident feedback to mitigate the issue.</td>
<td>Conductsthe Post-Incident reviews to identify the root cause and document the findings to provide feedback to the core development team.</td>
</tr>
</tbody>
</table>

Implementing the DevOps practices can reduce the friction between Development and Operations teams. It can also help you deliver the end product reliably along with other challenges and problems that the DevOps teams can solve.
<h3>1. Reduced Cost of Development and Maintenance</h3>
A DevOps team always works towards CI/CD, putting more effort into automated testing rather than manual testing and improving release management by automating it all.
As for the traditional Software Development Life Cycle, there is always toil on the effort (development, testing, release), which increases the overall cost for the product development and production maintenance. Putting DevOps practice in execution can significantly reduce the delivery time, development, and maintenance costs.
<h3>2. Shorter Release Cycle</h3>
One of the most effective changes a DevOps team brings is to deliver faster with a shorter release cycle. The reason why the DevOps team advocates a shorter release cycle is that it is easy to manage and roll back to the stable version in case there are any issues. 
As opposed to the traditional release cycles, where the focus is on getting everything delivered in one release, which increases the risk of failure in production and is much harder to roll back. If DevOps practices are followed strictly, the organization will always have a proper release version system with release versions and minimal manual interventions with the release artifacts.
Here are the gains of a shorter release cycle:
<ol>
<li style="font-weight: 400;" aria-level="1">Deliver the new change request more frequently;</li>
<li style="font-weight: 400;" aria-level="1">Pushing the upgrades (bug fixes, security patches, version upgrades) to production is much easier.</li>
</ol>
<h3>3. Automated and Continuous Testing</h3>
In contrast to the traditional development cycle, where the testing team has to wait for the delivery of the product in the test environment to begin the testing DevOps, testing is injected from the beginning of the development lifecycle.
DevOps facilitates continuous and automated testing with the help of the CI/CD tool (Jenkins) and version control (Git, BitBucket). Adequate coverage of functional, nonfunctional, and interaction tests running in the pipelines can significantly improve the testing automation aspects of the project.
To learn more about DevOps, see our article on <a href="https://spacelift.io/blog/who-is-devops-engineer" target="_blank" rel="noopener">Who is DevOps?</a>

<h3>1. Reduced Mean Time to Recovery (MTTR)</h3>
SRE team is responsible for keeping the production up and running. In the event of a bug or production failure, SRE teams can roll back to the previous stable version of a product so that Mean Time to Recovery (MTTR) is reduced.
<h3>2. Reduced Mean Time to Detect (MTTD)</h3>
The other problem that the SRE team is trying to solve is to reduce the Mean Time to Detect(MTTD) using the Canary Rollouts so that the new release is made available to a small group of users before doing full rollouts. Canary rollouts help the SRE team find the issues in the early stage with a limited number of affected users.
<h3>3. Automated Everything</h3>
Automation is one of the biggest challenges the SRE team has to face. It is often observed that rollouts and supporting tasks are carried out manually, leading to inconsistency and increasing the probability of human error. 
A good practice for managing the infrastructure is to use <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code (IaC)</a> with the help of Terraform, Pulumi, and the automation tools such as Ansible, Puppet, Chef. SRE team can leverage those tools to solve the problem of automation.
<h3>4. Automated Functional and Non Functional Testing in Production</h3>
The Core Development team can automate functional and non-functional testing in the test and stage environments but not in production.
Reliability engineers can help implement automation testing on Production environments without affecting the end-user.
<h3>5. On-Calls and Incident Documentation</h3>
Often reliability engineers have to take the on-call duties for managing unforeseen incidents, but they also have to prepare the documentation of the incidents and the troubleshooting steps so that it can help others perform the on-call duties. 
The SRE team can build up a valuable knowledge base on incidents to improve the incident troubleshooting time.
<h3>6. Shared Knowledge</h3>
Gaining exposure and building the knowledge base of the product development ecosystem (dev, test, stage, prod) is always beneficial for reliability engineers to foresee the issues in the production environment. 
But the main problem arises when the knowledge base is outdated, automation playbooks have irrelevant comments. Regular knowledge base updates by SREs in collaboration with DevOps can fill the knowledge gap between the teams.

When we talk about the tools of DevOps and SRE, it is often observed that most of the tools are being used commonly by both DevOps and SREs. 
<h3>Common Tools</h3>
<h4>Planning</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Jira Software</li>
<li style="font-weight: 400;" aria-level="1">Confluence</li>
<li style="font-weight: 400;" aria-level="1">Slack</li>
<li style="font-weight: 400;" aria-level="1">Microsoft Teams</li>
</ul>
<h4>Configuration Management Tools</h4>
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/what-is-pulumi" target="_blank" rel="noopener">Pulumi</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://spacelift.io/blog/ansible-tutorial" target="_blank" rel="noopener">Ansible</a></li>
<li style="font-weight: 400;" aria-level="1">Puppet</li>
<li style="font-weight: 400;" aria-level="1">Chef</li>
</ul>
<h4>Version Management</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">GitHub</li>
<li style="font-weight: 400;" aria-level="1">BitBucket</li>
<li style="font-weight: 400;" aria-level="1">GitLab</li>
</ul>
<h4>Log Monitoring</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Splunk</li>
</ul>
<h3>DevOps Tools</h3>
<h4>Continuous Integration Continuous Delivery</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Jenkins</li>
<li style="font-weight: 400;" aria-level="1">AWS CodePipeline</li>
</ul>
<h4>Integrated Development Environment</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Intellij </li>
<li style="font-weight: 400;" aria-level="1">Visual Studio</li>
<li style="font-weight: 400;" aria-level="1">Sublime</li>
</ul>
<h4>Automated and Security Testing</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Jmeter</li>
<li style="font-weight: 400;" aria-level="1">Robot Framework</li>
<li style="font-weight: 400;" aria-level="1">Burp</li>
<li style="font-weight: 400;" aria-level="1">Wireshark</li>
</ul>
<h3>SRE Tools</h3>
<h4>Monitoring</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">Kibana</li>
<li style="font-weight: 400;" aria-level="1">Prometheus</li>
<li style="font-weight: 400;" aria-level="1">Grafana</li>
<li style="font-weight: 400;" aria-level="1">New Relic</li>
<li style="font-weight: 400;" aria-level="1">Istio</li>
</ul>
<h4>Incident Reporting System</h4>
<ul>
<li style="font-weight: 400;" aria-level="1">PagerDuty</li>
<li style="font-weight: 400;" aria-level="1">OP5</li>
<li style="font-weight: 400;" aria-level="1">Opsgenie</li>
<li style="font-weight: 400;" aria-level="1">VictorOps</li>
</ul>

While the two share some core values, the focus of their work is different &#8211; the application lifecycle through DevOps and operations lifecycle management through SRE. Nevertheless, they both connect the Development and Operation teams while sharing similar responsibilities. And they are both working towards the same goal &#8211; enhancing the release cycle and achieving better product reliability.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Who is DevOps? Is it Worth it to Become a DevOps Engineer?

How to Become a DevOps Engineer in Six Months

Business Benefits of Infrastructure as Code

What does DevOps do? And what about SRE? See our article and find out the differences between DevOps and SRE. 

56-sre_vs_devops

SRE vs DevOps

SRE vs. DevOps: What’s the Difference Between Them?

45.how_to_manage_infrastructure_as_code_at_scale

Scalability and IaC are not always a match made in heaven. There are several touchy areas within the development environment, and associated tooling must be seriously considered to ensure effective “management” of IaC to assure scalability. 
This blog post will touch on a few critical areas that can negatively impact IaC when provisioning and deploying complex, large-scale infrastructures. It will also show how to manage IaC at scale without workarounds, a manual, or a partially automated development process.

The first area to understand is IaC code platforms. <a title="Terraform is the de facto standard and most widely used IaC coding platform" href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform is the de facto standard and most widely used IaC coding platform</a> by far; several alternatives, such as Pulumi, Cloudformation, and Ansible, are available. Each of them has its relative strengths and weaknesses.
While we are huge fans of Terraform, we are also pragmatic about its shortcomings, limiting the ability of a developer to manage IaC at scale cleanly. Below, we highlight some of these weaknesses that should be considered:
<ul>
<li style="font-weight: 400;" aria-level="1">Resource management: Running the terraform state mv command every time there is a changed resource identifier is painful and time-consuming. It is further complicated by a large codebase, as it is challenging to maintain clean code and evolve over time</li>
<li style="font-weight: 400;" aria-level="1">TF State: Terraform tracks the current state of managed resources using plan and apply. State is very fragile and can be impacted, for example, by a stale state file or by forgetting to run “terraform remote config”</li>
<li style="font-weight: 400;" aria-level="1">Local machines: Each action is manual and must be performed directly by the developer, leading to multiple people working on the same codebase. Developer A applies their changes to the environment and initiates a pull request. At the same time, developer B would like to deploy their changes but is blocked as the codebase is not up-to-date with developer A’s changes. Applying the changes in this state would cause damage to the resources created in the previous deployment. Working in larger teams will often result in wasted time just waiting and rebasing the codebase to apply changes.</li>
<li style="font-weight: 400;" aria-level="1">Security: Devs working with the codebase need access to the Terraform state, which creates a security concern due to how Terraform state works. Pulling sensitive data from an external system such as Vault will be stored within the state file in plaintext once used in a resource, so a threat actor could run `terraform state pull` to access all the secrets</li>
<li style="font-weight: 400;" aria-level="1">CLI Interface: Typos and other types of errors are common, resulting in misconfigurations or failed plans and applies</li>
<li style="font-weight: 400;" aria-level="1">Infrastructure Fine-Tuning: It is difficult to interface with subsets of the infrastructure to make small changes dynamically. Terraform works well for bulk deployments but lacks the delicate touch necessary for fine-tuning infrastructures</li>
<li style="font-weight: 400;" aria-level="1">Rollback: Terraform does not support rollback to a previous state due to limitations of Terraform plan, which means that the only option is to roll forward</li>
</ul>
New to IaC? See: <a title="What Is Infrastructure as Code? Examples, Best Practices &amp; Tools" href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">What Is Infrastructure as Code? Examples, Best Practices &amp; Tools</a>

The short answer is yes! IaC development can be significantly enhanced using <a title="CI/CD processes" href="https://upcommons.upc.edu/bitstream/handle/2117/174561/TFG-definitiu.pdf" target="_blank" rel="nofollow noopener">CI/CD processes</a> and workflows like applications or tool development. Using CI/CD to provide workflows for infrastructure provisioning and deployment offers the following benefits: 
Consistency:
<ul>
<li style="font-weight: 400;" aria-level="1">Code is controlled &#8211; well documented and optimized.</li>
<li style="font-weight: 400;" aria-level="1">Code is modular &#8211; reused across multiple projects.</li>
</ul>
Testing:
<ul>
<li style="font-weight: 400;" aria-level="1">Testing is integrated &#8211; cannot be skipped.</li>
<li style="font-weight: 400;" aria-level="1">It is far simpler to test code in small bites &#8211; find issues faster not hunting for a needle in a haystack &#8211; 100 loc vs. 10K loc.</li>
</ul>
Automation:
<ul>
<li style="font-weight: 400;" aria-level="1">Routing of change approvals &#8211; single or multiple sign-offs.</li>
<li style="font-weight: 400;" aria-level="1">Establishes a set of controllable mechanisms &#8211; gates for events, etc.</li>
</ul>
See how to deploy your infrastructure in CI/CD here: <a title="How to Deploy Your Infrastructure in CI/CD Using Terraform" href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener">How to Deploy Your Infrastructure in CI/CD Using Terraform</a>

The below diagram provides an example of a Fintech application and support for infra implemented in AWS. This solution is used by financial institutions, banks, or specialized lenders to manage their Asset Based Lending portfolio and deliver their services to their clients.

image1-9

Fintech ABL Application and Infrastructure

In this example infrastructure, each of the infra resources designated with the TF logo has Terraform providers, or HCL code is available to configure the infrastructure.
Provisioning and deploying for new clients is complex as it has to meet their individual compliance requirements and banking and regulatory rules, regulations, and standards. And what about a foreign client bank? For this, banks must meet completely different banking and regulatory rules and compliance standards. 
And of course, overarching everything are stringent cybersecurity requirements, which are necessary in many cases as the application has to be able to securely interface with external core banking systems for daily updates on loan status with each customer.
Imagine duplicating this infrastructure each time a new client is onboarded using a non-IaC optimized CI/CD solution. 
<h3>IaC and CI/CD</h3>
As mentioned, a CI/CD solution is strongly recommended for provisioning and deploying IaC based infrastructures. One of the most widely used CI/CD solutions is Jenkins. 
However, Jenkins and other generic CI/CD solutions were designed and optimized for application-centric workflows, and have limitations that will prevent scalability when implemented in support of IaC-specific development. Some of these issues include:
<ul>
<li style="font-weight: 400;" aria-level="1">IaC is a stack-based technology, and generic CI/CD tools do not understand stacks.</li>
<li style="font-weight: 400;" aria-level="1">No unified toolset or framework for policy management requiring manual checks.</li>
<li style="font-weight: 400;" aria-level="1">Lack of built-in synchronization, resulting in manual updates and synchronization.</li>
<li style="font-weight: 400;" aria-level="1">Inability to provide an automated workflow across multiple repositories and stacks.</li>
<li style="font-weight: 400;" aria-level="1">Cannot visualize or provide cost tracking of resources, drift detection, and remediation.</li>
<li style="font-weight: 400;" aria-level="1">In most cases, SSO and SAML are plugins—just another component to configure, manage, and update</li>
<li style="font-weight: 400;" aria-level="1">Conflicting pull requests where one developer is already working on a piece of code and another developer is also making changes, possibly resulting in code discrepancies.</li>
</ul>
Generic CI/CD solutions check all the boxes to support implementing a well-defined development process, adding end-to-end workflows allowing DevOps teams to deliver better applications faster, but not for IaC. 
<h3>Accountability and IaC</h3>
What do we mean by accountability? It is analogous to a business having a set of accounting books that represent the financial condition of the company as it stands at the present moment, but also its financial history. 
Without this history, it is impossible to trace back and establish accountability, to understand when financial decisions and changes were made, actions taken, and by whom and for what purpose.
Accountability is also applicable to your infrastructure as it enables you to understand the present state and how you arrived at this point. 
Infrastructure accountability provides checks and balances to ensure that you can account for all infrastructure-related events. From the first line of code to production, you need to know and understand every event that has or will affect the infrastructure to ensure that process integrity exists and, by extension, so does proof of compliance, both internally and externally. Some of the accountability questions that you have to be able to answer are:
<ul>
<li style="font-weight: 400;" aria-level="1">Code History:
<ul>
<li style="font-weight: 400;" aria-level="2">What is the history of the code?</li>
<li style="font-weight: 400;" aria-level="2">Who made changes to the code and infrastructure.</li>
<li style="font-weight: 400;" aria-level="2">Do you know if access to your repo is secure? </li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Change Implementation:
<ul>
<li style="font-weight: 400;" aria-level="2">Has a change been fully implemented?</li>
<li style="font-weight: 400;" aria-level="2">Can you validate and prove that it was deployed?</li>
<li style="font-weight: 400;" aria-level="2">Are paper-based policies and regulatory compliance requirements being met? Can you prove it?</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Change Approval Process:
<ul>
<li style="font-weight: 400;" aria-level="2">What is the change approval process?</li>
<li style="font-weight: 400;" aria-level="2">Are all changes approved, no matter how small or innocuous they seem?</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Auditability:
<ul>
<li style="font-weight: 400;" aria-level="2">Are you prepared for an audit?</li>
<li style="font-weight: 400;" aria-level="2">Do you have the supporting materials necessary to satisfy security and compliance audits?</li>
</ul>
</li>
</ul>
What types of information do you need to answer the above questions?
<ul>
<li style="font-weight: 400;" aria-level="1">Provenance: The ability to trace every piece of code back to inception, along with any changes made.</li>
<li style="font-weight: 400;" aria-level="1">Compliance: Provide detailed information needed to ensure compliance with policies and procedures, and the ability to provide complete documentation for an audit.</li>
<li style="font-weight: 400;" aria-level="1">Access: Who has access, what do they have access to, what are they allowed to do, when are they allowed to do it, why are they doing it.</li>
<li>Visibility: Understand the state of the infrastructure and resources at any point in time.</li>
</ul>

The complex infrastructure example above gives an overview of a current ABL Fintech solution. This diagram represents the security that is provisioned and deployed on top of the infrastructure for the application.

image3-9

Fintech ABL Security Overlay

In many cases, individual clients require VPCs for their implementation, with multiple layers of protection, as they may have different security requirements or their specific vendor solutions in place. It is especially relevant if financial institutions have product offerings beyond ABL that have strict regulatory or industry compliance requirements based on their geographical location.
The Terraform logo highlights these security-specific resources supported in Terraform, and can be defined and provisioned.
The ability to control the following security aspects for access, source code, changes, and deployments, as well as validation thereof, is not achievable without embedded policy guardrails in place. They are essential to achieving secure infrastructure provisioning and deployments. Without policies in place, the probability of a bank passing an internal or external audit is very low.

Now that we’ve covered code platforms, CI/CD solutions, and accountability to help you understand the significant issues that stand in your way of IaC scalability, let’s take a look at what can help you to manage IaC at scale without workarounds, a manual, or a partially automated development process: 
<h3>Automate</h3>
Automation covers the end-to-end IaC workflow from repo to production:
<ul>
<li style="font-weight: 400;" aria-level="1">Integrated with source code repos such as Github to trigger IaC workflows when committing new or revised code.</li>
<li style="font-weight: 400;" aria-level="1">Creates a library of pre-configured infrastructure environments, enabling sharing across teams and projects.</li>
</ul>
<h3>Use Policy as Code</h3>
Policy as Code provides immediately enforceable guardrails:
<ul>
<li style="font-weight: 400;" aria-level="1">Declare rules around infrastructure, access to code and stacks, workflow, state changes, set triggers. </li>
<li style="font-weight: 400;" aria-level="1">Access controls for the 4Ws.</li>
<li style="font-weight: 400;" aria-level="1">Multi-layer change approvals and applies after approval.</li>
<li style="font-weight: 400;" aria-level="1">Audit support, SSO (SAML2.0) fully integrated. </li>
</ul>
<h3>Collaborate</h3>
Adaptable to existing workflows with no force-fitting or changes required, or fully customized to meet goals:
<ul>
<li style="font-weight: 400;" aria-level="1">Coordination of all runs with the ability to review, comment, and iterate across teams.</li>
<li style="font-weight: 400;" aria-level="1">Prevent conflicting changes and reconcile the differences automatically.</li>
</ul>
<h3>Introduce Resource Management</h3>
Understand resources and their state:
<ul>
<li style="font-weight: 400;" aria-level="1">Plan out changes.</li>
<li style="font-weight: 400;" aria-level="1">Drift detection and remediation.</li>
<li style="font-weight: 400;" aria-level="1">Cost of infrastructure.</li>
</ul>
In addition to all of the above, here are some additional vital capabilities that an ideal, purpose-built orchestration and management solution for IaC should also provide, at a minimum! Starting from the left of the diagram:
<ul>
<li style="font-weight: 400;" aria-level="1">Creation of pre-built library of infra components.</li>
<li style="font-weight: 400;" aria-level="1">Support of mono or multiple repos with version control.</li>
<li style="font-weight: 400;" aria-level="1">Ability to support multiple projects simultaneously.</li>
<li style="font-weight: 400;" aria-level="1">Integration and application of policy as code in concert with IaC configurations, ensuring that compliance and security initiatives are met.</li>
</ul>

image2-9

image2

Learn how to manage Terraform at scale here: <a title="5 Ways to Manage Terraform at Scale (Examples)" href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale (Examples)</a>

You can take positive actions to overcome the scalability issues of IaC code platforms, generic CI/CD solutions, and accountability to prevent your infrastructure from inhibiting your future business initiatives and growth.
You can be realistic and find an IaC orchestration and management solution that enables your development process to scale. 
And finally, you ought to be aware that proactive management and control over compliance and audit objectives are necessary and can be covered through Policy as Code, integrated into your IaC development, security, provisioning, and deployment workflows.
If you are concerned about achieving IaC scalability, <a title="try a free trial of Spacelift!" href="https://spacelift.io/free-trial?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Bscaling_infrastructure_as_code%7D" target="_blank" rel="noopener">try a free trial of Spacelift!</a> You can overcome code platform issues and enable accountability with a CI/CD solution built from the ground up for IaC. Control the process, visualize resources and create workflows that work, using Spacelift as your IaC orchestration and management solution.

Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure.  It helps overcome common state management issues and adds several must-have features for infrastructure management.

Managing infrastructure as code at scale tutorial. See how to manage IaC at scale. Examples and diagrams included. 

45-how_to_manage_infrastructure_as_code_at_scale

How to Manage Infrastructure as Code at Scale (Examples)

The term DevOps is used as a massive catchall for several different tech-job scenarios and skills. But essentially it boils down to building useful tools for your internal customers.
DevOps as a career can be rewarding in many ways. First, there’s the <a href="https://www.glassdoor.com/Salaries/devops-engineer-salary-SRCH_KO0,15.htm" target="_blank" rel="noopener">average salary of $105,000</a> to <a href="https://www.ziprecruiter.com/Salaries/What-Is-the-Average-Devops-Engineer-Salary-by-State" target="_blank" rel="noopener">$114,000</a>, spiking to over $160,000 at the top end. Then there’s the swift job growth, estimated at over 20% per decade, with over <a href="https://www.linkedin.com/jobs/devops-jobs" target="_blank" rel="nofollow noopener">118,000 openings listed on LinkedIn</a> right now.
Plus, DevOps beats the daily grind of “same old, same old,” thanks to its fast-paced, system-support setup. In this article, we share how to become a DevOps engineer — the systems administrator who just happens to be a coder that develops internal tools as well. You’ll learn about the mindset, tools, and skills you’ll need to master, and also how to get a DevOps job.
Before you start though, is DevOps the right choice for you? See our guide: <a href="https://spacelift.io/blog/who-is-devops-engineer" target="_blank" rel="noopener">Is it Worth it to Become a DevOps Engineer?</a>

DevOps is a mix of development (Dev) and operations (Ops). Beyond that, it’s all the connected tools and disciplines of both of those business areas that let an organization deliver services and software applications at high speed, so they can better serve their customers.
The Microsoft-approved definition of DevOps is “the union of people, processes and technology to continually provide value to customers.” Both definitions are open to interpretation, but DevOps centers around writing code that pulls together customers (internal and external), and business processes.
Sometimes, DevOps engineering means just “being that go-to employee” who can quickly and efficiently write code to address an engineering issue. In other words, in some organizations, DevOps is the indispensable IT employee who knows how to write effective code.

The question of how to become a DevOps engineer has a relatively straightforward answer. With that said, you’ll need to bring a few things to the table. First and most important to the DevOps career path is a passion for learning, knowledge, and logic.

DevOps engineers need to be able to read between the lines in their customers’ requirements. They also have to produce software and services that meet those requirements in a usable, testable form. Since development doesn’t happen in a vacuum, you’ll also need leadership and management skills, along with a cool head under pressure.

When most DevOps hiring managers look for a new employee, they’re more concerned with mindset than with tools. If you’ve got a tech background, you’re willing to learn, and you’re an engineer at heart, you’ve already got the basics of a DevOps career.
DevOps engineers are curious, constantly improving their skillsets, and focused on lifelong learning. So while you can build the core skillset in a few months, your main driver should be on learning, with a goal of providing massive value to your next employer.
Learn to understand systems and processes, and you have the right mindset. That mindset will help you learn how to start a career in DevOps, and more important, how to be a good DevOps engineer.

It takes about six months to become a DevOps engineer, assuming you have some basic Linux admin and networking skills, and that you apply the DevOps engineer learning path outlined below. With that said, that career won’t just happen overnight. The length of time required depends on several factors, including your mindset, your current skill level, and your career position.
With that caveat, there’s no shortage of free tools and resources you can use to help you on your journey. Some professional DevOps engineering sites even offer free or vastly reduced exams to help you grow and prove your worth.  Let’s dig into how to become a DevOps engineer, starting with the tools and skills.

To become a DevOps engineer, at the bare minimum you’ll need basic Linux admin and networking skills, plus some scripting fundamentals, along with the following DevOps skills:
<h3>1. Intermediate to advanced Linux skills</h3>
In DevOps, you’re not installing a server once and then logging in every now and then to perform a few admin tasks. You need to understand how to create highly customized Linux images from the ground up, both for VM and container use cases — unless you plan to become a Windows Server DevOps engineer.
<h3>2. Intermediate networking skills</h3>
In DevOps there’s no “network team.” All network resources are software-defined. In other words, networks are part of infrastructure as code. At a bare minimum, you’ll need a solid grasp on the OSI model, IPV4, subnetting, static and stateful firewalling, and DNS. These skills are usually included in advanced cloud certifications.
<h3>3. A commitment to at least one cloud</h3>
Clouds aren’t merely managed data centers. In order for you to automate workloads in a given cloud (AWS, Azure, GCP, etc.), you need a firm grasp of their specific semantics. You’ll need to know what resources are available, how they’re organized, and what properties they have. 
<h3>4. Infrastructure automation</h3>
Once you understand the resources (and their properties) applicable to a cloud, you’re ready to automate their creation using tools such as <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> and Ansible.
<h3>5. SDLC, CI/CD pipelines, and scripting</h3>
In DevOps, we deliver infrastructure in a similar way to applications. So — you’ll need to be acquainted with the fundamentals of the software development life cycle (SDLC). This includes versioning strategies using source control code management systems like Git, and CI/CD pipelines such as Jenkins, and CircleCI. Advanced automation tasks may prove difficult through shell scripts alone. You’ll often require more powerful scripting using the likes of Python, Perl, or Ruby. 
<h3>6. Container technology</h3>
For legacy workloads you may automate the creation of a VM image. But for new applications you’ll be working with containers. As such, you need to know how to build your own Docker images (Linux skills required!) and deploy them using Kubernetes. FaaS technology like AWS Lambda also uses container technology behind the scenes.
<h3>7. Observability technology</h3>
While all clouds have monitoring dashboards and standard “telemetry” hooks, most large employers use third party (both commercial and open source) monitoring tools such as Prometheus, DynaTrace, Datadog, or the ELK stack.
Beyond that skills list, tool-building in DevOps requires a fundamental understanding of logic, its application, and how to apply it in a computer-recognizable form. While that may sound a tad scary for the uninitiated, there are several good books that cover programming fundamentals without using any specific language. Here are a couple of my favorites:
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://www.amazon.com/_/dp/1449355730?tag=oreilly20-20" target="_blank" rel="nofollow noopener">Learning Python</a> (O’Reilly)</li>
<li style="font-weight: 400;" aria-level="1"><a href="https://www.amazon.com/Learn-Windows-PowerShell-Month-Lunches/dp/1617291080" target="_blank" rel="nofollow noopener">Learn Windows PowerShell in a month of lunches</a> (Jones, Hicks — A personal favourite as I love PowerShell)</li>
</ul>
Now let’s dig into the nuts and bolts of how to become a DevOps engineer — starting with education.

One of the great things about DevOps is that it’s about what you can do, not what qualifications you have. Some of the best DevOps engineers in the field are self-taught, with little in the way of formal higher education. The biggest requirement is motivation and an interest in DevOps engineering.
With that said, you’ll have a much easier time both learning DevOps skills and getting a company to hire you if you have a bachelor’s degree in software development, IT, or a related field.

Start your DevOps engineer roadmap by looking through the skills list above. If you already have some of those skills — great. If not, be honest about the time you’ll need to spend to learn them. But don’t stress about getting everything perfect before you start. If you wait for mastery, you’ll never get a DevOps job.
Start by learning a few of the easy-to-learn skills. If you’re already employed in a non-DevOps job, start working on some DevOps projects now, to build mastery and proof you have the skills. Then make the switch to a full-time DevOps career.
You may even find that your own company has DevOps openings you could move into. Keep a keen eye on internal and external vacancies alike.

Let’s take a deeper look now at how to become a DevOps engineer — the DevOps career path and how to build the skills. We’ll share the reasons each of these tools is important, and how long it’ll take to learn each one. We’ll also point you to some good online classes and certifications.
You can learn most of these skills on the job — but a word of caution. In the sink-or-swim world of DevOps career growth, different companies have different requirements. There’s no one-size-fits-all approach.

<h3>Foundation knowledge: 4 months</h3>
We’ve put a plus-sign after each of the time frames below, because while you can learn the basics quickly, mastery can take much longer.

devops-foundation-knowledge-580-x-746-px

DevOps Foundation Knowledge

<h4>1. Intermediate to Advanced Linux and Networking: 1 month+</h4>
Linux is the OS and server platform of choice for DevOps engineers in companies of any size. Linux’s open-source nature, small operational footprint, and support from the Likes of Redhat and Ubuntu make it the go-to not only for DevOps, but for tool building in general.  One of the best things about Linux is that you can download it and start using it today.
If you feel that your Linux skills are rusty, you can get started with the free course offered by <a href="https://www.udemy.com/course/learn-linux-in-5-days/" target="_blank" rel="noopener">Udemy</a>. In fact, if you want to learn how to become a DevOps engineer exclusively from Udemy, they have an entire <a href="https://www.udemy.com/topic/devops/" target="_blank" rel="noopener">curriculum of core DevOps classes</a>.
In terms of networking, you’ll get the necessary skills if you do an intermediate cloud certification, such as <a href="https://aws.amazon.com/certification/certified-solutions-architect-associate/" target="_blank" rel="noopener">AWS Certified Solution Architect</a>, but it helps if you take a specialized course such as <a href="https://www.coursera.org/learn/computer-networking?specialization=google-it-support" target="_blank" rel="noopener">The Bits and Bytes of Computer Networking</a> on Coursera.
<h4>2. Advanced Scripting: 2 months+</h4>
First of all, you’ll always need shell (e.g., bash) scripting skills, because this is the default for Linux and most tools. 
For “advanced” scripting use cases, there are quite a few languages out there, but Python is a good start if you don’t know what scripting language to pick.
You can <a href="https://www.coursera.org/articles/how-long-does-it-take-to-learn-python-tips-for-learning#:~:text=In%20general%2C%20it%20takes%20around,can%20take%20months%20or%20years." target="_blank" rel="noopener">master Python in as little as two months</a> with online tutorials from <a href="https://www.learnpython.org/" target="_blank" rel="noopener">LearnPython.org</a>. However, you’ll find that many employers also use other languages such as Perl and Ruby as well, so be ready to learn those, if need be.
<h4>3. Cloud Training and Certification: 1 month+</h4>
AWS is the 600lb gorilla in terms of agile cloud providers, and AWS and Linux go together like Strawberries and cream. You’ll need to be fluent in AWS before you can call yourself part of the DevOps community.
The beauty of AWS and cloud development in general is that you only pay for what you use. That model makes cloud computing ideal for DevOps testing. You can set up an environment quickly, use it for what you need, then pull it down again.
It’s easy to start using AWS, since there’s a 12-month free version available to anyone who signs up. You can learn professional-grade skill in AWS in <a href="https://openupthecloud.com/how-long-to-learn-aws/" target="_blank" rel="noopener">as little as one month</a>, though mastery can take years of continual on-the-job use. Get your <a href="https://aws.amazon.com/certification/" target="_blank" rel="noopener">AWS certification here</a>.
<h4>4. Google Cloud: 1 month+</h4>
Azure offers similar employment opportunities to AWS, but what about GCP?
The Google Cloud Platform (GCP) is smaller than AWS and Azure but it excels particularly in data mining and artificial intelligence (and other deep learning technologies). Google’s DevOps-related offerings are becoming increasingly popular with large companies.
In the banking industry for example, the Google AI/ML tools are creating new ways of doing business, plus adding fraud detection and usage-pattern tracking. This saves huge amounts of time trying to develop similar tools in-house.
Similarly, other large companies are using Google’s ML tools to bring massive data sets down to size, drawing business-driving insights from previously unmanageable seas of data.
Want to know more about how to become a DevOps engineer with Google Cloud? You can get your <a href="https://cloud.google.com/blog/topics/training-certifications/get-google-cloud-certified-in-3-months#:~:text=Get%20Google%20Cloud%20Certified%20in%203%20months,-Rochana%20Golani" target="_blank" rel="noopener">Google Cloud certification here in three months</a>, though you can <a href="https://www.coursera.org/specializations/developing-apps-gcp" target="_blank" rel="noopener">learn to develop applications with Google Cloud in as little as one month</a>.

<h3>Skills: 1 month</h3>
It doesn’t take long to learn the DevOps skills you’ll need to succeed in your new career. All these tools are free to use and experiment with. They just require a little time and effort on your part. Let’s look at how long it takes to learn the basic DevOps tools like Terraform, Git, Docker, Jenkins, ECS, and ELK Stack.

devops-skills-580-x-563-px

DevOps Skills

<h4>1. Configure: Terraform (and Ansible) — 1 week+</h4>
Configuration management is at the heart of fast software development. Poorly configured tools waste time, while well-configured tools save it.
As its name implies, Terraform has one purpose in life — to create infrastructure as code in an automated way that speeds up your entire process.
Ansible concerns itself with server-desired state configuration, ensuring that servers are configured to specs. These two technologies are cornerstones of DevOps. Both may seem complex at first, but they’re all based around configuration files written in YAML. 
Terraform takes about a week to learn the basics. <a href="https://www.udemy.com/course/devops-deployment-automation-terraform-aws-docker/" target="_blank" rel="noopener">Udemy offers a great online class</a> that bundles AWS, Terraform, and Docker. You can also learn more about Ansible in <a href="https://www.exitcertified.com/it-training/topics/devops" target="_blank" rel="noopener">DevOps courses from ExitCertified</a>.
<h4>2. Version control: Git and GitHub (GitLab) — 20 minutes+</h4>
Version control is key to any DevOps endeavour. It lets DevOps Engineers and their team members create and review code faster, without wasting time sharing endless files and iterations.
Git is a standalone product that by default is used on local machines and networks. This is different from GitHub, which facilitates version control in the cloud, with the overhead managed by GitHub itself. In the world of <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">infrastructure as code</a>, version control with products like Git and GitLab are essential.
<a href="https://about.gitlab.com/" target="_blank" rel="nofollow noopener">GitLab</a> is a complete open-source DevOps platform. It helps users deliver software faster, with collaboration and security all rolled into one. Looking to learn more about how to become a DevOps engineer with Git? You can <a href="https://www.udacity.com/course/version-control-with-git--ud123" target="_blank" rel="noopener">learn the basics of Git</a> in minutes if you’re already a programmer. 
<h4>3. Package: Docker (Lambda) — 3 days+</h4>
Packaging is where build management meets release management. It’s where your code and infrastructure come together for deployment.
Without Docker there would be no DevOps. Docker essentially allows DevOps to run code in small isolated containers. That way, building services and replacing services becomes simpler than updating everything in one go (which is very non-DevOps).
Amazon’s Lambda is an alternative to Docker that many companies use instead. Though it’s best to know both tools, Docker is an excellent starting point. You can learn Docker in just a few days. Udemy offers a solid <a href="https://www.udemy.com/course/learn-docker/" target="_blank" rel="noopener">beginner’s course online for DevOps</a>. They also have a <a href="https://www.udemy.com/course/docker-and-kubernetes-the-complete-guide/" target="_blank" rel="noopener">Docker class bundled with Kubernetes</a>.
<h4>4. Deploy: Jenkins (CodeDeploy) — 2 days+</h4>
During deployment, you’ll take your code from version control to users of your application. Automation is a key component of this step, and Jenkins is the central way to automate.
Jenkins allows automation for all manner of tasks, including running build tests and making decisions based on whether code passes or fails the build process. You can also use Jenkins for more mundane purposes, like centralized management of scripts and executing commands via SSH (and other authentication pathways).
It’s a tool to automate those frequent and boring tasks that computers can do better than even the best DevOps engineer could. Some companies choose CodeDeploy over Jenkins, making it another useful DevOps tool to learn.
You can learn to use Jenkins in just a few days. Udemy offers a great <a href="https://www.udemy.com/course/jenkins-from-zero-to-hero/" target="_blank" rel="noopener">Jenkins class online</a> for DevOps engineers.
<h4>5. Run: ECS (Kubernetes) — 1 day</h4>
Kubernetes is DevOps bread and butter. It starts with Docker and adds extra functionality and tools. For instance, it lets the administrator ensure that several copies of a container image are running. That way, if a single VM or host is lost, the service is still available.
ECS and Kubernetes perform valuable services like this in the background. They deliver several automated DevOps tools that allow useful additions to manage containers, and their availability. They also add important items such as introducing role-based access control and more centralized auditing and management functionality.
See <a href="https://developer.ibm.com/series/kubernetes-learning-path/" target="_blank" rel="nofollow noopener">IBM’s Kubernetes learning path and guide</a> for a 13-hour course.
<h4>6. Monitor: ELK Stack (Prometheus) — 2 days</h4>
Once your new application is up and running, you’ll need a real-time view of its status, infrastructure, and services. To this end, DevOps engineers love ELK.
ELK provides all the base components for effective log management and search functionality. It’s Elasticsearch, Logstash, and Kibana — three open source applications offered by the <a href="https://www.elastic.co/what-is/elk-stack" target="_blank" rel="noopener">Elastic</a> company.
ELK takes data from multiple sources, and lets you visualize it by using useful charts and graphs. Its rival platform, Prometheus, is just as important for a DevOps engineer to understand. You can learn to use the ELK Stack in just a few days with <a href="https://www.udemy.com/course/elasticsearch-logstash-kibana-learn-elasticsearch-search-server/" target="_blank" rel="noopener">Udemy’s 4-star online class</a>.
Those are the basics of how to become a DevOps engineer. Now let’s look into why Git matters so much, and how to get a DevOps job.

One more word on GitHub as a shortcut to starting a career in DevOps. GitHub is essentially the CV of the DevOps world. Any DevOps hiring person will check out your GitHub profile as a very first step and point of contact. Yet it’s easy to learn GitHub and other DevOps tools while you create your virtual CV at the same time.

Knowing how to become a DevOps engineer doesn’t stop with skills. The next step in your DevOps engineer career path is getting the job. That sounds daunting, but if you’ve got software development experience, the skills above, and a few DevOps achievements for your resume, you’re well on your way to getting hired.
Here’s how to get into DevOps.
<h3>Rewrite your DevOps resume</h3>
The first step in getting a DevOps job is to rewrite your resume. A DevOps resume doesn’t need to show years of experience. As this excellent <a href="https://zety.com/blog/devops-resume-example" target="_blank" rel="noopener">DevOps resume guide</a> shows, start with a reverse-chronological format. Then in your work history, education, and projects section, list achievements, including:
<ul>
<li style="font-weight: 400;" aria-level="1">Development tasks from past jobs</li>
<li style="font-weight: 400;" aria-level="1">Side projects in development, IT, Agile, scripting, or automation</li>
<li style="font-weight: 400;" aria-level="1">Volunteer work in coding or distribution</li>
<li style="font-weight: 400;" aria-level="1">Projects in Terraform or other DevOps tools</li>
<li style="font-weight: 400;" aria-level="1">Linux/Unix projects</li>
<li style="font-weight: 400;" aria-level="1">Scripting in Python or Ruby</li>
<li style="font-weight: 400;" aria-level="1">Tasks completed with AWS, Jenkins, Maven, etc.</li>
</ul>
Start each resume bullet point with an action verb like developed, wrote, created, built, deployed, etc. And use numbers to show how many projects, deployments, scripts, tests, containers, and how many customers, team members, etc.
The more you show DevOps achievements in your history, with measurable details, the higher your chance of getting hired. Knowing how to become a DevOps cloud engineer is all about showing your projects and accomplishments.
<h3>Apply to lots of DevOps jobs</h3>
Finding a DevOps job is a numbers game. If you apply to three jobs, you won’t hear back from any. If you apply to 50, you’ll get a few responses and maybe an interview. Plan to hear back from about one in every 30 applications, and get interviewed by one in every 100.
In other words? You’ll need to apply to a lot of DevOps jobs. Probably something like 300 in a month to get one job (about 14 every weekday). But — you can vastly boost your chance of getting hired if you lean on networking. The easiest way? Start connecting with DevOps engineers on LinkedIn.
Then — don’t ask them for a job. Just ask if you can chat with them about their cool career. About 20% will be glad to share their success. You’ll learn tons about how to start a career in DevOps. And surprise surprise — some will even introduce you to their contacts.
<h3>Shun the unicorns</h3>
Don’t worry about being front-and-center in a DevOps job at Google, Amazon, or another giant company. Everybody clamors to get hired at those firms, creating stifling competition. Don’t be afraid to pursue a DevOps job at a less glamorous firm. Once you’ve got a little experience under your belt, then you can go unicorn hunting.
<h3>Consider working your way up</h3>
Is DevOps a good career for freshers? Not really. Don’t count on landing an entry-level DevOps job. DevOps is, by nature, an advanced position that requires highly skilled candidates. But — don’t let that discourage you. One of the best DevOps career paths is to start as a software developer or IT specialist in a company that also hires DevOps engineers.
If your current employer doesn’t hire DevOps pros, consider switching to one that does. Don’t stay entry-level for long. Three to six months is plenty. Once you’ve logged that time, commit to applying internally to DevOps positions in your new company. During your entry-level tenure, work to build accomplishments that look good on a DevOps resume.

Those are the basics of how to become a DevOps engineer. Though becoming a DevOps engineer takes persistence and passion, it’s not rocket science. Anyone with the drive (and a little time) can follow the DevOps career path, learn the necessary skills in five months, and get a DevOps job in one month. With the right skillset and job search strategy, you can be in your DevOps dream job very soon.

What Are Terraform Modules and How to Use Them: Tutorial

Though becoming a DevOps engineer takes persistence and passion, it’s not rocket science. See this article if you are interested in a career in DevOps.

how-to-become-devops-engineer-2

How to Become a DevOps Engineer in 6 Months: Comprehensive Guide

Is it Worth it to Become a DevOps Engineer?

There’s no doubt about it, working as a DevOps engineer — an engineer that enables DevOps culture — is challenging, cutting-edge, and financially rewarding. DevOps engineering is a relatively new career, with Larry-Page-level opportunities for those with the right technical skills.
It’s also highly relevant in the workplace, and it’s ideal if you’re hungry for both technological and interpersonal challenges. DevOps is an intelligent job transition if you’ve been doing software development, networking, or operations.
DevOps engineers do a lot of automation, monitoring, testing, configuring, networking, and <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">infrastructure as code</a> (IaC). That means you’ll need to bring a variety of skills and talent to the table. Let’s face it, it’s a highly technical role that’s usually best left to experienced engineers.
There’s some opportunity in the DevOps field for entry-level engineers, but most of the jobs require experience. So, is it worth it to become a DevOps engineer? And — do you have what it takes? After reading this article, you’ll be better equipped to find out!
Interested in a career in DevOps? See our article on <a href="https://spacelift.io/blog/how-to-become-devops-engineer" target="_blank" rel="noopener">How to Become a DevOps Engineer</a>.

DevOps Engineers use their skills to deliver software quickly, continuously, and reliably. The goal of DevOps is to shorten the software delivery and feedback cycles. DevOps pros typically orchestrate multiple components in a cloud environment. They’re the ones who make sure the system keeps running smoothly, day after day.
DevOps professionals also keep things up to date with the latest changes and security updates. They automate testing, scanning, and deploying. If there’d been DevOps on the Death Star, the rebels never would have blown it up.
<h3>Where did DevOps come from?</h3>
DevOps evolved as a cultural phenomenon. Recently, companies have created specialist DevOps roles to help create and support their culture. There are so many specialized tools and skills in the IT side of every business that supporting DevOps is increasingly a full-time job.
While DevOps has certainly changed how we develop software, it has also fundamentally changed how software is deployed. This is largely thanks to the general availability and lower cost of cloud infrastructure, containerized environments, and the evolution of tools that enable automation.

Working in DevOps isn’t a bad way to spend your time these days. As a field, DevOps is experiencing job growth. Furthermore, the jobs in DevOps pay more than the average tech or computer science job. There’s some level of upward and horizontal mobility too.
Let’s dive a little deeper into each of the three reasons DevOps is a good career.
<h3>1) Massive DevOps job growth</h3>
To say that jobs in the DevOps field are growing would be an understatement. In fact, the Bureau of Labor Statistics (BLS) <a href="https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm#tab-1" target="_blank" rel="noopener">shows a projected growth rate</a> this decade in the software field at 22%, compared to an 8% median growth rate for all jobs.
While DevOps jobs are only part of that growth, it’s still substantial. Let’s assume, for example, that DevOps jobs growth is only 5%. With 1.8 million jobs in the field now, that 5% growth means 90,000 new DevOps jobs. And given the popularity of DevOps with employers, 5% is almost certainly a lowball estimate.
<h3>2) High salary for DevOps jobs</h3>
DevOps jobs pay very well, like many jobs in IT. If you’re into being paid comfortably and you have the right qualifications, you’re probably already looking at salaries for DevOps jobs.
<a href="https://builtin.com/salaries/dev-engineer/devops-engineer" target="_blank" rel="noopener">Builtin</a> puts the average base salary for a DevOps Engineer in the U.S. at $125 — and upwards in the $300k range. <a href="https://www.glassdoor.com/Salaries/devops-engineer-salary-SRCH_KO0,15.htm" target="_blank" rel="noopener">Glassdoor</a> pegs it closer to $105k. These are just two examples of average pay. The range is fairly wide, but still, the pay is quite nice for a 9-5 job.
Of course, there’s always some variability in the market depending on location, experience, and the company you end up working for.

avarage-devops-salary-in-the-usa-in-different-industries-2

Avarage DevOps salary in the USA in different industries

<h3>3) Upward mobility</h3>
There’s some upward mobility for DevOps Engineers depending on the company. You might find a career path from junior engineer all the way up to DevOps architect. If you have the right stuff, you may even end up managing the department.
It’s also possible to transition to similar roles like site reliability engineer (SRE). Of course, the higher you go the fewer positions are available. (The corporate pyramid leaves less room near the top.)
DevOps offers plenty of horizontal mobility too, since DevOps engineering skills usually translate between industries. Plus, it doesn’t make too much of a difference whether the company makes widgets or life insurance. The work in DevOps is generally the same.

average-devops-salary-2

Average DevOps salary

Depending on the company and your specific DevOps role, you may end up working on any number of issues. Your role may relate to automation, infrastructure, tooling, quality, or monitoring (to name a subset). The automation aspect includes code deployment, environment and infrastructure configuration, and even automatic quality checks.
The goal of all this automation is to speed up delivery without sacrificing quality or security. Much of the work is setting up monitoring and alerting for the automation and production systems. Monitoring and altering are important aspects of the feedback loops that power continuous improvement.

There are a few key downsides of a career in DevOps, including a high-pressure atmosphere, lack of recognition, and frustrations from butting heads with other technical teams. You may also feel like a jackrabbit in a field of turtles as you champion cutting-edge advancements while the rest of your organization slogs behind.
Most of the work in DevOps happens behind the scenes. In other words, your achievements won’t be put on display. You may get zero recognition outside your branch in the org chart. In less collaborative environments, DevOps Engineers might face frustrations with other technical teams.
<h3>Out on the bleeding edge</h3>
There may be situations where the goals of DevOps run counter to the goals of other groups that are slow to change or adopt complementary practices. The tools can sometimes be frustrating to work with too. And, in the real world, any number of things can go wrong at any given time.
Part of your job in DevOps is to make sure systems are running well. If something goes wrong, you may have to work outside normal hours to resolve the problem. You may feel intense pressure from extremely high expectations, especially during challenging times.

By now, you should have the answer to the key question: is DevOps a good career? It is.
Still, you might not know if DevOps is right for you. So — if you’re still on the fence, here are a few questions (and answers) to help make up your mind:
<h3>Are you interested in automation?</h3>
If you’re interested in automation, that’s one point for team DevOps. If you prefer coordinated manual deployments, maybe a DevOps environment won’t be the best fit for you. DevOps teams do a lot of automation.
<h3>Do you like the DevOps framework?</h3>
The DevOps framework is a set of behaviors that seeks close collaboration between software development and operations. If using communication, processes, and tools to tighten that loop is appealing, you’d do well in a DevOps environment.
<h3>Do you like being in charge of infrastructure?</h3>
DevOps Engineers play a Linus-Torvalds-size role in establishing and maintaining the infrastructure that runs applications. Most often, the actual infrastructure (the hardware) is provided by a cloud vendor. The work of DevOps Engineers is to configure the rented infrastructure for efficient operations of applications. It’s interesting work if you like fine tuning and solving data flow problems.

Do you have the right skills to be a DevOps engineer? The often-quoted acronym CAMS paints a picture of the spirit of DevOps as: Culture, Automation, Measurement, and Sharing.
That acronym was coined by John Willis and Damon Edwards in 2010. As Willis <a href="https://itrevolution.com/devops-culture-part-1/" target="_blank" rel="noopener">pointed out</a>, Jez Humble later added an ‘L’ for Lean. CALMS pretty much sums up the skills you need for a DevOps career.

top-8-devops-skills-580-x-435-px

top 8 devops skills

<h3>1) Interpersonal</h3>
Patrick DuBois, co-originator of DevOps, still says all problems are people problems. In Jan 2021, he still fights the good fight against organizational silos <a href="https://twitter.com/patrickdebois/status/1350116781982162946" target="_blank" rel="nofollow noopener">as mentioned in this tweet</a>. And since DevOps is a culture of sharing, it helps to have good interpersonal skills.
Part of the job is reaching out with empathy and helping others achieve their goals. And, you know — things will go wrong from time to time. When they do, your interpersonal skills will go a long way toward helping you come out on top.
<h3>2) A programming language</h3>
If you’re thinking about a career in DevOps, you should know how to code, at least at a basic level. Much of the work involves writing shell scripts or using a coding language such as Python to automate tasks. You may need to write Python scripts to call various APIs, or use shell scripts to manipulate files.
Some DevOps roles require knowledge of VB-Script and Windows PowerShell. Either way, you should have a foundation of coding, so you can learn the nuts and bolts of your environment.
<h3>3) Automated build/test</h3>
One of the core principles of DevOps is automation. Automating builds is a human-genome-sized step toward consistency, reliability, and traceability. Once that’s done, it’s time to automate the tests for that build. Sometimes this demands a big shift in thinking and a whole lot of work.
Still, it’s a big win when you can get to automated build and test. DevOps culture centers around this kind of win. You may write tests using Selenium, Postman, the Robot Framework, or other types of automated testing tools. You’ll have to incorporate these tools into the automated release process.
<h3>4) Containers</h3>
In all likelihood, you’ll use containers in your career as a DevOps Engineer. Many organizations are either using or switching to containers (mainly Docker) for several types of workloads. A container is an isolated runtime environment that includes most of an operating system. It runs on a server or VM. You can launch containers quickly, making them ideal for running tasks fast — a key focus in DevOps work.
<h3>5) Configuration</h3>
Configuration is another task DevOps Engineers spend lots of time on. Actually, most of the work in automating is configuring the systems that perform the work. Even the infamous IaC (infrastructure as code) is really about configuring the infrastructure and environment using a specification language.
DevOps cultures prefer using IaC because IaC tracks changes. That lets DevOps pros measure the effects (remember: CALMS). There’s nothing worse than tracking down a problem without knowing a configuration changed! In DevOps, you want to monitor events like Tony Stark.
<h3>6) Version control</h3>
Along with one of the tools you’ll use frequently as a DevOps engineer — the Linux-based operating system — you’ll most likely use another of Linus Torvald’s inventions — git! Or you might use another version control system like subversion or TFS. They all do the same general thing: keep track of changes to the codebase.
For automated delivery pipelines, you’ll keep the configuration under version control. You’ll store IaC files in VCS as well. And often, a VCS is hosted on a platform that runs a pipeline and can be configured to kick off automation. The VCS is also the heart of a change-management system in a DevOps world.
<h3>7) Package</h3>
Developers from different groups or projects often need the same types of features. They might need logging, authentication, or any kind of functionality plugged into a software development environment. DevOps culture embraces sharing (the “S” in CALMS), so its denizens will often “package” functionality.
DevOps engineers might even dive in and create these packages to share. Or, they may just manage package repositories both public and private. Note that package repositories are another possible destination for delivery.
<h3>8) Networking</h3>
Ahhh, networking! If a problem isn’t a code error, it’s probably something in the network. Networking is another critical skill in a DevOps role. It could be enough to know the IP protocol, but you’ll often need an understanding of certificates and routing, too.
Even in a cloud environment, it’s important to know how networks and subnetworks operate. And even containers have their own type of network configuration to consider.

DevOps is a great career in 2022 and beyond. If you’re at all interested in the crossover between development and operations, it could be a Ken-Thompson-level field for you. With six-figure salaries, swift job growth, and plenty of upward mobility, the future for DevOps is blindingly bright. If you’ve felt strongly that DevOps is a good career to get into, then definitely give it a shot.

What is Terraform? What is it Used For, How it Works (Examples)

You still wondering if DevOps is right for you? See our article and find out whether it is worth it to become a DevOps engineer in 2022.

41-why_is_it_worth_to_become_devops_engineer

41.why_is_it_worth_to_become_devops_engineer

What Is Infrastructure as Code? Examples, Best Practices & Tools

Infrastructure-as-code (IaC) is a very important concept to understand in the DevOps world today. It has become almost ubiquitous across the industry and is absolutely key to modern engineering roles.
In this post, I will explain what IaC is, what its benefits are, some of the tooling options available, and point out some best practices along the way. I will then look at some code examples for each tool to compare and contrast.

Put in the simplest terms, using code, you define the infrastructure that needs to be deployed in a descriptive model. Similar to code for applications, the code for the infrastructure becomes part of the project and is stored inside your version control system (or VCS).
For example, consider that you have developed a web application. That web application needs to be hosted on infrastructure somewhere to be consumed. Using IaC you could define things like where the infrastructure is deployed to, such as a public cloud provider like Microsoft Azure, Amazon AWS, or Google Cloud, and what type of service your web application will run on, such as an Azure web app, or an AWS S3 Storage account. Further to this, you can then define the settings required for the web app, which might include things like how much server compute power is required (in terms of CPU and memory), how the networking is secured, and how the domain name for your app will be exposed, to name a few among many other considerations.

IaC solves many common problems with provisioning Infrastructure.
<ul class="">
<li id="597c" class="mp mq io kf b kg kh kj kk km mr kq ms ku mt ky mu mv mw mx gj" data-selectable-paragraph="">New environments or infrastructure can be provisioned easily from your IaC configuration code. Infrastructure deployments with IaC are repeatable.</li>
<li id="2ed8" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">Manually configured environments are difficult to scale. With environments provisioned using IaC, they can be deployed and scaled rapidly.</li>
<li id="1fa7" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">If you want to make changes to the existing infrastructure that has been deployed with IaC, this can be done in code, and the changes will be tracked.</li>
<li id="e257" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">When IaC is used with a declarative tool (it describes the state you want your environment to look like), you can detect and correct environment drift. If a part of the infrastructure is modified manually outside of the code, it can be brought back in line with the desired state on the next run.</li>
<li id="42dd" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">Changes can be applied multiple times without changing the result beyond the initial application. This is known as idempotence.</li>
<li id="e195" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">Avoid manual configuration of environments which can typically introduce mistakes due to human error. With IaC, these can be avoided.</li>
<li id="449b" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">IaC is a means to achieve consistency across environments and infrastructure. The code can be reused.</li>
<li id="7a68" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">Infrastructure costs are lowered as the time to deploy and effort to manage, administer and maintain environments decrease.</li>
<li id="3dd7" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">IaC can be used in Continuous Integration / Continuous Deployment (or CI/CD) pipelines. The main benefit of doing this is to automate your Infrastructure deployments.</li>
<li id="b7bc" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">DevOps teams can test applications in production-like environments early in the development cycle.</li>
<li id="9dd2" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">With your Infrastructure configuration code held in your version control system alongside your application source code, commonly in the same repository. Now everything can be held together.</li>
<li id="7002" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">Productivity will increase due to a combination of all the benefits of using IaC.</li>
<li id="f688" class="mp mq io kf b kg my kj mz km na kq nb ku nc ky mu mv mw mx gj" data-selectable-paragraph="">As the code is held in your version control system, it gains all the benefits of the VCS. More on that in the next section.</li>
</ul>

The adoption of IaC is not without its challenges.
Typically, traditional infrastructure or operations teams within organizations may not be familiar with version control systems, use of Git, or be comfortable using tools for code editing such as visual studio code.
Further to this, there is certainly a learning curve when it comes to the adoption of new technology within an organization. Training will be required and time will be needed to develop the appropriate skills.
Skills in IaC and DevOps are currently highly sought after in the industry and so therefore it may be difficult to hire staff with these skills.
The journey to managing your environments using IaC will typically start with a small deployment of new resources to your chosen cloud platform, and from there once adoption within the organization grows, more of your infrastructure can start to be deployed and managed in code. Eventually, when your organization is mature and comfortable with the principles and operation of your chosen IaC systems and tooling, existing resources can be brought under IaC control.

Without diving too deeply into the benefits of using a VCS in this article, we will summarise them here as storing your IaC in a VCS automatically gives you a raft of additional benefits.
In summary, a VCS enables developers and organizations to work more efficiently in improving the quality of their product while recording and evaluating a detailed history of their improvements that will lead to a successful end product. In short, using a VCS enables governance, versioning, and increases collaboration.
1. Efficiency
As configuration files are amended incrementally with changes, testing becomes easier as rollback to previous versions is possible. New features can be built up over time.
2. Tracking &amp; Versioning
Track who made the change, and in which version of the file. Comments can be added to each code change (or commit).
3. Collaboration
Multiple people can work on the configuration files at the same time using branching. Changes can be merged together using pull requests.
4. Governance &amp; Compliance
Tracking changes automatically gives you a powerful audit trail, enabling risk management.
5. Management
A management overview becomes possible through being aware of the author of the configuration, how long it took to make changes, the timeline and its impact.
6. Reduces Duplication
Multiple and out-of-date configuration files are reduced.
7. Backup
Users will typically clone the repository their configuration code is held in and work on it locally. The code then exists locally and in the VCS. The VCS itself is also backed up.

There are 2 approaches to the templates used when writing IaC configuration files. Different tools use different approaches, which are listed in the next section.
Declarative — you define the desired state of the final solution.
The tool or automation platform determines how the goal is reached, the step-by-step executions are handled by the tool and hidden from the user. Declarative tools are the most popular and most dominant in the IaC space. They are most useful when changes or updates need to be made to your solution.
Declarative tools are idempotent because you are defining the required state of the solution. Idempotency refers to a process that can be executed multiple times with the same result.
Imperative — you define the steps to execute in order to reach the desired solution.
An imperative approach allows you to build up multiple layers of commands to reach the end goal. Imperative tools give you more control over how the goal is reached. These tools are most useful when you need to deploy and not update or change the solution in the future.
Imperative approaches may not lead to idempotency, the end goal may be different depending on the starting point, as a series of steps form the process. Consider a process with 10 steps, for example, starting from step 1 would lead to a different result compared to starting with step 6.

So you’ve decided to pursue IaC due to the myriad of benefits it enables. Now you need to decide which tool to use! This will depend on a number of factors, including your engineering capability, and current use of cloud platforms.
You should also consider the use of a Continuous Integration / Continuous Delivery (CI/CD) platform and the tools that it supports before deciding. The most flexible platforms such as <a class="au lo" href="https://spacelift.io/?utm_source=blog&amp;utm_medium=text&amp;utm_id=blogpost&amp;utm_content=%7Binfrastructure_as_code_tools%7D" target="_blank" rel="noopener">Spacelift</a> support multiple tools, including Terraform, Pulumi, and CloudFormation.
First, we will outline 2 cloud-agnostic options, Terraform and Pulumi, before moving on to the Microsoft Azure cloud-specific ARM templates and Bicep. Lastly, we will briefly discuss other options for AWS, CloudFormation, and CDK, as well as other tools such as Ansible, Chef, Salt, and Puppet.
<h3>Terraform</h3>

iac1

Terraform logo

<a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> takes the declarative approach. It uses its own language called Hashicorp Configuration Language (HCL) which is considered by most as very easy to pick up and is very human-readable. HCL is based on Go.
Terraform can be used with almost any cloud provider, as well as on-premise infrastructure through its provider mechanism, of which there are thousands available. It is for this reason that many choose Terraform over other options, if you want to use the same language and formatting to configure infrastructure across multiple cloud platforms and infrastructure, it is a natural choice.
In my opinion, even if you only want to use it for one cloud platform (e.g. Microsoft Azure), it is still the best option, due mainly to the maturity of the language and its supporting ecosystem.
Terraform maintains a state file. The state file describes the existing state of the infrastructure and allows Terraform to query, build, maintain and change the infrastructure as defined in your configuration files. Maintaining the state file introduces challenges around its security, but many solutions exist to the problem.
Dependency mapping between resources is done in the background automatically by Terraform and is largely hidden from the user, but can be controlled if required. Dependency mapping refers to the order resources are created in. Consider the creation of a virtual machine, first, you must have a VNET, subnet, disk, and key vault for disk encryption in place before it can be created successfully. Terraform makes sure resources are created in the correct order. If these need to be manually manipulated, the <code class="ng nh ni nj nk b">depends_on</code> met-argument can be utilized in the configuration to achieve this.
Terraform has a raft of great features, including allowing you to deploy to different environments using workspaces, and the ability to run plans to show what will change without making any alterations to name a couple.
Terraform does have a slight delay in terms of the features it supports. For example, when a new feature or service is released in Azure, this capability must be added to the <code class="ng nh ni nj nk b">azurerm</code> provider by the team that maintains it before the feature will be able to be referenced using Terraform. However, the delay is usually very small and almost all features that are available to a user in the Azure portal and available through Terraform configuration language. If your team lives on the cutting edge and utilizes lots of preview services, this might be an issue. This is not always the case for all providers, as when it comes to the <code class="ng nh ni nj nk b">aws</code> Terraform provider, some new features have been implemented before the AWS native CloudFormation!
<h3>Pulumi</h3>

iac2

Pulumi logo

<a href="https://spacelift.io/blog/what-is-pulumi" target="_blank" rel="noopener">Pulumi</a> is another declarative IaC tool. Instead of having its own language, it allows teams to use an existing programming language, something which can be highly desirable if your team already has strong preferences or skills. Pulumi supports TypeScript, JavaScript, Python, Go, and C#.
Like Terraform, Pulumi also supports multiple clouds, infrastructure, and providers.
Note that any existing Terraform or ARM configuration files you have can be converted into Pulumi files.
Pulumi is harder for coding novices to get to grips with compared to Terraform or Bicep. If no prior coding skills exist within your team, it will take time to learn.
As Pulumi configuration files are coded with the programming language of your choice, this means you can use the same testing tools as you use for your main code, another major advantage.
<h3 id="e9c7" class="pw-post-body-paragraph kd ke io kf b kg kh jp ki kj kk js kl km kn ko kp kq kr ks kt ku kv kw kx ky ih gj">Microsoft Azure ARM Templates</h3>
ARM templates are the native templating option for all resources in Azure. ARM templates are written in JSON format.
Getting started using ARM templates is easy as ARM templates can be downloaded for any existing resource straight from the Azure Portal and can be modified as needed. These templates can allow you to redeploy resources easily. Because ARM templates are the native option in Azure, they are well documented and supported by Microsoft. They can also be used for any resource in Azure from the day of release, an advantage over Terraform.
However, when writing ARM templates from scratch there is a lot of ‘boilerplate’ code that is required, something that tools like Terraform and Bicep extract away from the user, making the code easier to both read and write. JSON language is picky about formatting and can be a chore to learn. Because of the additional code required for ARM templates compared to other options, the configuration files can get very large, complex, and hard to follow, and any syntax errors can become tricky to troubleshoot. There is no concept of ‘state’ when using ARM templates, so any changes that are applied can be breaking. In addition to these disadvantages, there is no plan, or ‘dry-run’ option available when using ARM templates, making it difficult to confirm that what you are deploying will result in the desired outcome.
<h3>Microsoft Azure Bicep</h3>

iac3

Azure Bicep logo

Azure Bicep is a declarative abstraction language used to simplify .json ARM templates that are used to provision infrastructure in Azure. Bicep is a Domain-specific language (DSL), so if you want to use the same language across multiple clouds, this isn’t the option for you.
Bicep aims to take away some of the unnecessary .json boilerplating, leaving simpler, cleaner code, similar to Hashicorps HCL. You can use modules, something not really available with ARM templates, allowing you to simplify complex deployments. Bicep can query the state directly from Azure, so there is no need to manage a state file, which is a great advantage over Terraform.
Bicep is still in its relatively early days, having come to prominence in the last couple of years, and so the community around it is not as large as that of Terraform. However, Bicep is fully supported by Azure Support teams meaning you can raise a ticket through the Azure portal for Bicep-related issues, similar to how you would raise a ticket for any Azure resource problems you might encounter. In theory, anything you can do with ARM templates you can do with Azure Bicep (and more) although some limitations still exist during to the ongoing development of the tool.
Bicep is part of the Azure CLI. Note that you can convert existing ARM templates into bicep files. As an example running the command below in cloud shell will convert your .json ARM template into a .bicep file:

<h3 id="1b64" class="pw-post-body-paragraph kd ke io kf b kg kh jp ki kj kk js kl km kn ko kp kq kr ks kt ku kv kw kx ky ih gj">Amazon Web Services (AWS)</h3>
<a href="https://spacelift.io/blog/terraform-vs-cloudformation" target="_blank" rel="noopener">CloudFormation</a> is the native AWS IaC tool and takes the declarative approach. CloudFormation templates are written in JSON or YAML format. CloudFormation is a managed AWS service and will check the infrastructure it has provisioned to detect if it is maintaining the described state, no state management is required which is a benefit over Terraform. Similar to modules in Terraform, AWS uses a concept called ‘nested stacks’, which enables templates to be called from other templates. Note that interestingly it is possible to provision resources on other clouds or non-AWS resources using CloudFormation, as the custom resources feature enables this. However, this involves additional templating and is certainly not as easy as using Terraform. CloudFormation support is included in AWS support plans.
Similar to Pulumi but not cloud-agnostic, the AWS Cloud Development Kit (AWS CDK) is an open-source software development framework to define your cloud application resources using familiar programming languages. CDK would be generally considered as taking an imperative approach.
<h3 id="b150" class="pw-post-body-paragraph kd ke io kf b kg kh jp ki kj kk js kl km kn ko kp kq kr ks kt ku kv kw kx ky ih gj">Ansible, Salt, Chef, and Puppet</h3>
These are generally considered as configuration management as code tools, but can also be used to provision and manage infrastructure.
Puppet takes the declarative approach. Chef takes the imperative approach. <a href="https://spacelift.io/blog/ansible-vs-terraform" target="_blank" rel="noopener">Ansible</a> and Salt are mostly declarative but offer some support for imperative commands.

In this section we will compare the code required to create a Storage account in Azure, using Terraform, Pulumi, Bicep and ARM templates. All four examples will create the same result in Azure, a storage account with a container called ‘logs’.
<h3 id="830d" class="pw-post-body-paragraph kd ke io kf b kg kh jp ki kj kk js kl km kn ko kp kq kr ks kt ku kv kw kx ky ih gj">Terraform</h3>
Notice how easy to read the HCL language is, making it arguably the most accessible to the beginner of the four compared options.

<h3 id="7418" class="pw-post-body-paragraph kd ke io kf b kg kh jp ki kj kk js kl km kn ko kp kq kr ks kt ku kv kw kx ky ih gj">Pulumi</h3>
The Pulumi example can be coded in many different programming languages, here I have shown the Python equivalent as it will be a commonly used and understood language across infrastructure teams. In Python, it actually comes in slightly shorter than the terraform equivalent but is harder to read for the beginner.

Existing Terraform <a href="https://www.pulumi.com/tf2pulumi/" target="_blank" rel="nofollow noopener">can be converted</a> to Pulumi files using tf2pulumi, a great resource that lets you view the code side-by-side and convert it to Typescript, Python, Go, or C#.
<h3 id="8c19" class="pw-post-body-paragraph kd ke io kf b kg kh jp ki kj kk js kl km kn ko kp kq kr ks kt ku kv kw kx ky ih gj">ARM Templates</h3>
The ARM template example is much more lengthy and verbose, highlighting the issues with ARM templates that Bicep aims to resolve.

[parameters('storageAccountName')]

[format('{0}/default/{1}', parameters('storageAccountName'), parameters('containerName'))]

<h3 id="2d17" class="pw-post-body-paragraph kd ke io kf b kg kh jp ki kj kk js kl km kn ko kp kq kr ks kt ku kv kw kx ky ih gj">Bicep</h3>
The Bicep example is much more concise are easier to read than the ARM equivalent, although not quite as short as the Terraform.

The Bicep playground is a great resource that allows you to <a href="https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/compare-template-syntax/?WT.mc_id=modinfra-57552-apedward" target="_blank" rel="nofollow noopener">directly compare</a> ARM templates and Bicep configuration files. Use the ‘Sample Template’ button to select a template to compare side-by-side. You will notice that Bicep files are always shorter, remove the ARM boilerplating, and as a result are easier to use.

Automating your Infrastructure using IaC can pay dividends, enabling your team to deliver more and be more agile, saving you time and money.
Selecting the correct IaC tooling for your team can be a daunting task as there are plenty of options available. Comparing the pros and cons of each, along with an evaluation of your requirements and available skill sets in your team, should give you a good starting point.
If you are just starting out with IaC, I would personally recommend looking at Terraform first as it is a highly sought-after skill in the industry today, and is the prevalent IaC tool. There are lots of learning materials available on the web and the community support is great.

Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation and controls to simplify and accelerate the provisioning of cloud based infrastructures.

What is Multi-Cloud Infrastructure? Benefits, Best Practices

Infrastructure as Code (IaC) isn’t a new notion, but it still causes confusion. In this article, you’ll learn what IaC is, its benefits, best practices and code examples.

what-is-infrastructure-as-code

Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Terraform

Terraform is a well-known IaC tool from Hashicorp. Terragrunt is a thin wrapper of Terraform that provides extra tools for keeping your configurations DRY, working with multiple Terraform modules, and managing remote states. It is an open source project from Gruntwork, co-founded by Yevgeniy Brikman, author of Terraform up and running.
This article discusses common patterns in <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code (IaC)</a> based on Terraform and Terragrunt examples and summarizes and highlights the design principle underlying good architecture.
We will present these patterns in an abstract way. Although the examples presented in this article use Terraform and Terragrunt, the main concepts apply to other popular IaC tools and platforms like Pulumi, AWS CDK, CloudFormation, Terraform Cloud, Atlantis, Spacelift, and <a href="https://spacelift.io/terraform-cloud-alternative" target="_blank" rel="noopener">Terraform Cloud alternatives</a>.
To put it simply, “managing collections of infrastructure resources” is an abstract concept. The technologies used to implement this concept include:
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://www.terraform.io/docs/cloud/workspaces/index.html" target="_blank" rel="nofollow noopener">Terraform workspace</a></li>
<li style="font-weight: 400;" aria-level="1"><a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html" target="_blank" rel="nofollow noopener">Cloudformation (nested) stack</a></li>
</ul>
Git branch (allows you to create two branches, one called workspace_1, the other workspace_2, then relate remote state to each of the branches, which is actually a neat trick.)

In short, to utilize IaC is to describe your infrastructure with code. Coding infrastructure in HCL (<a href="https://github.com/hashicorp/hcl" target="_blank" rel="nofollow noopener">Hashicorp Configuration Language</a>) with Terraform is just like coding with any other programming language, like Python or Java. Programming technologies such as linter, IDE, unittest, documentation, and CICD, are all applicable to IaC. Additionally, a project layout is crucial to defining a structure to organize your code, dependencies, configuration files, etc.
One common pattern of the project layout for IaC is much like the following:

image1-3

common pattern of the project layout for IaC

The basic principle is to:
<ul>
<li style="font-weight: 400;" aria-level="1">Separate the process of IaC into two phases: development and deployment.</li>
<li style="font-weight: 400;" aria-level="1">Separate the deployment environments into prod, qa, and stage, for example.</li>
<li style="font-weight: 400;" aria-level="1">Separate the cloud architecture into components according to projects, types of infrastructure, etc. Each component can be stored in a single “gigantic” repository or multiple “micro” repositories (see Pattern: Mono-repo vs Multi-repo).</li>
</ul>

The process of IaC usually consists of two phases: development and deployment (if you are interested in this topic, the <a href="https://www.weave.works/technologies/gitops/" target="_blank" rel="noopener">GitOps and weaveworks articles</a> are good sources of information).
In the development phase, one writes code to describe and configure the desired infrastructure. In the deployment phase, the written infrastructure code is applied.
The advantages of separation:
<ul>
<li style="font-weight: 400;" aria-level="1">Separation of Concern: developers who write IaC do not need to know the details of the deployment, and can work or test on a new version of the code without influencing the actual deployed infrastructure.</li>
<li style="font-weight: 400;" aria-level="1">Flexibility: a well-designed IaC codebase is modularized and parameterized. The same code snippet can be reused and deployed under different use cases in different environments with customized parameters.</li>
</ul>

Imagine you have an AWS cloud architecture for web services, including different AWS services like ECS, RDS, S3, IAM, SSM, Secret Manager, Cloudwatch, Cloudfront and so on. One option is to pack all of the components into a single Terraform remote state (main.tf). This is fine for quick and dirty prototyping, but not recommended for production.
Always cut your cloud architecture into meaningful components such as:
<ul>
<li style="font-weight: 400;" aria-level="1">Stateful components like database, filesystem, or S3.</li>
<li style="font-weight: 400;" aria-level="1">Stateless components like lambda, ECS, or API Gateway.</li>
</ul>
You can also cut it based on projects or functional areas (e.g. frontend, backend, data warehouse).
The advantages of separation:
<ol>
<li style="font-weight: 400;" aria-level="1">Avoids scanning the whole architecture if a change needs to be made only locally to one resource. One can enter the component folder which includes that resource, make changes there, and apply them. Scanning a fat cloud account can be time consuming on a large amount of cloud resources, even causing unnecessary API throttling.</li>
<li style="font-weight: 400;" aria-level="1">Avoids unnecessary touches on databases or other crucial infrastructure to prevent accidental destructive operations.</li>
<li style="font-weight: 400;" aria-level="1">Increases readability of the IaC.</li>
</ol>
Translate the above described project layout into folder structure:

image3-3

folder structure

The separation of the development and deployment process is implemented with two git repositories &#8211; env/ and modules/. Notice the .git in the folder. Here, Terragrunt.hcl is simply a configuration file for modules specifically in the language of Terragrunt. The separation of cloud architecture components is implemented with subfolders app/ , mysql/, vpc/. The separation of deployment environments is implemented with subfolders prod/ , qa/ , stage/.

One of the principles of software engineering is DRY (don’t repeat yourself). The same applies to IaC. There are at least two kinds of tricks you can use to DRY IaC:
<ul>
<li style="font-weight: 400;" aria-level="1">Modularization</li>
<li style="font-weight: 400;" aria-level="1">Configuration inheritance</li>
</ul>
<h2>1) Modularization</h2>
A <a href="https://spacelift.io/blog/what-are-terraform-modules-and-how-do-they-work" target="_blank" rel="noopener">Terraform module</a> is a way of creating a template of a cloud pattern, parameterizing, and reusing it. Like any other artifacts, modules can be stored and versioned in the registry. The official Terraform module registry is <a href="https://registry.terraform.io/" target="_blank" rel="nofollow noopener">https://registry.Terraform.io/</a>. You can also build your own private registry.
<h2></h2>

image2-3

AWS security group

<h2>2) Configuration Inheritance</h2>
<a href="https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#include" target="_blank" rel="nofollow noopener">Configuration inheritance</a> is a feature supported by Terragrunt. The concept is similar to class inheritance in object oriented programming, except that configuration inheritance serves the purpose of configuration reusability.
In the following example (also see the project layout above), we have a configuration file at the global level (global_vars.yaml) and configuration files at the environment level (environment_specific_vars.yaml). 
In global_vars.yaml, variables like account_id, region_name and repository_url can be defined, and in environment_specific_vars, these global variables will be inherited and present, additionally defining new environment level variables like vpc_id (usually we use separate vpc for different environments), environment_name, etc.

image4-3

Configuration Inheritance

In summary, configuration inheritance increases the reusability of configuration files and reduces the difficulty in maintenance.

Last but not least, let’s review the concept of mono-repo and multi-repo. It has been pointed out by Hashicorp as a <a href="https://www.hashicorp.com/blog/terraform-mono-repo-vs-multi-repo-the-great-debate" target="_blank" rel="nofollow noopener">major architectural decision for IaC projects</a>, a view I share.
The problem is, there are two ways to organize code. In simple terms, mono-repo stores all code into one git repository (for example with multiple subfolders), whereas multi-repo separates them into multiple repositories.
Analogously, the relationship between mono-repo and multi-repo can be compared to monolith software architecture and microservice architecture. Mono-repo is intended to perceive the architecture as a whole and as evolving together. It is usually more suitable for small projects or big projects in prototyping. Multi-repo, on the other hand, facilitates collaboration but with extra overhead. Naturally, the more repos you have, the higher the maintenance cost, but also the more autonomous and self-service each repository is.

In this article, we reviewed several patterns of IaC architecture in a technology-agnostic way. The examples given are based on <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> and Terragrunt, since they are the most popular open source tools on the market at the moment. The idea, however, is to look at different technologies like <a href="https://spacelift.io/blog/what-is-pulumi">Pulumi</a>, AWS CDK, Cloudformation, etc., where the same patterns keep appearing in slight variations.
We rooted the principles of IaC architecture design in software engineering concepts like inheritance, DRY, project layout, parameterization, and so on. Afterall, Infrastructure as Code is coding infrastructure, and recognizing common patterns is the recommended way to learn and apply IaC.

Spacelift is a sophisticated SaaS product for Infrastructure as Code that helps DevOps develop and deploy new infrastructures or changes quickly and with confidence.

See common patterns of Infrastructure as Code architecture in Terraform and Terragrunt. Learn the differences between common patterns of IaC.