All posts by Peter Christensen

Peter Christensen

Author title

OpenTofu is now part of the Linux Foundation 🎉

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

how to manage infrastructure as code at scale

General | Spacelift Blog

General

Scalability and IaC are not always a match made in heaven. There are several touchy areas within the development environment, and associated tooling must be seriously considered to ensure effective “management” of IaC to assure scalability. 
This blog post will touch on a few critical areas that can negatively impact IaC when provisioning and deploying complex, large-scale infrastructures. It will also show how to manage IaC at scale without workarounds, a manual, or a partially automated development process.

The first area to understand is IaC code platforms. <a title="Terraform is the de facto standard and most widely used IaC coding platform" href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform is the de facto standard and most widely used IaC coding platform</a> by far; several alternatives, such as <a href="https://spacelift.io/blog/what-is-pulumi" target="_blank" rel="noopener">Pulumi</a>, Cloudformation, and Ansible, are available. Each of them has its relative strengths and weaknesses.
While we are huge fans of Terraform, we are also pragmatic about its shortcomings, limiting the ability of a developer to manage IaC at scale cleanly. Below, we highlight some of these weaknesses that should be considered:
<ul>
<li style="font-weight: 400;" aria-level="1">Resource management: Running the terraform state mv command every time there is a changed resource identifier is painful and time-consuming. It is further complicated by a large codebase, as it is challenging to maintain clean code and evolve over time</li>
<li style="font-weight: 400;" aria-level="1">TF State: Terraform tracks the current state of managed resources using plan and apply. State is very fragile and can be impacted, for example, by a stale state file or by forgetting to run “terraform remote config”</li>
<li style="font-weight: 400;" aria-level="1">Local machines: Each action is manual and must be performed directly by the developer, leading to multiple people working on the same codebase. Developer A applies their changes to the environment and initiates a pull request. At the same time, developer B would like to deploy their changes but is blocked as the codebase is not up-to-date with developer A’s changes. Applying the changes in this state would cause damage to the resources created in the previous deployment. Working in larger teams will often result in wasted time just waiting and rebasing the codebase to apply changes.</li>
<li style="font-weight: 400;" aria-level="1">Security: Devs working with the codebase need access to the Terraform state, which creates a security concern due to how Terraform state works. Pulling sensitive data from an external system such as Vault will be stored within the state file in plaintext once used in a resource, so a threat actor could run `terraform state pull` to access all the secrets</li>
<li style="font-weight: 400;" aria-level="1">CLI Interface: Typos and other types of errors are common, resulting in misconfigurations or failed plans and applies</li>
<li style="font-weight: 400;" aria-level="1">Infrastructure Fine-Tuning: It is difficult to interface with subsets of the infrastructure to make small changes dynamically. Terraform works well for bulk deployments but lacks the delicate touch necessary for fine-tuning infrastructures</li>
<li style="font-weight: 400;" aria-level="1">Rollback: Terraform does not support rollback to a previous state due to limitations of Terraform plan, which means that the only option is to roll forward</li>
</ul>
New to IaC? See: <a title="What Is Infrastructure as Code? Examples, Best Practices &amp; Tools" href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">What Is Infrastructure as Code? Examples, Best Practices &amp; Tools</a>

The short answer is yes! IaC development can be significantly enhanced using <a title="CI/CD processes" href="https://upcommons.upc.edu/bitstream/handle/2117/174561/TFG-definitiu.pdf" target="_blank" rel="nofollow noopener">CI/CD processes</a> and workflows like applications or tool development. Using CI/CD to provide workflows for infrastructure provisioning and deployment offers the following benefits: 
Consistency:
<ul>
<li style="font-weight: 400;" aria-level="1">Code is controlled &#8211; well documented and optimized.</li>
<li style="font-weight: 400;" aria-level="1">Code is modular &#8211; reused across multiple projects.</li>
</ul>
Testing:
<ul>
<li style="font-weight: 400;" aria-level="1">Testing is integrated &#8211; cannot be skipped.</li>
<li style="font-weight: 400;" aria-level="1">It is far simpler to test code in small bites &#8211; find issues faster not hunting for a needle in a haystack &#8211; 100 loc vs. 10K loc.</li>
</ul>
Automation:
<ul>
<li style="font-weight: 400;" aria-level="1">Routing of change approvals &#8211; single or multiple sign-offs.</li>
<li style="font-weight: 400;" aria-level="1">Establishes a set of controllable mechanisms &#8211; gates for events, etc.</li>
</ul>
See how to deploy your infrastructure in CI/CD here: <a title="How to Deploy Your Infrastructure in CI/CD Using Terraform" href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener">How to Deploy Your Infrastructure in CI/CD Using Terraform</a>

The below diagram provides an example of a Fintech application and support for infra implemented in AWS. This solution is used by financial institutions, banks, or specialized lenders to manage their Asset Based Lending portfolio and deliver their services to their clients.

image1-9

Fintech ABL Application and Infrastructure

In this example infrastructure, each of the infra resources designated with the TF logo has Terraform providers, or HCL code is available to configure the infrastructure.
Provisioning and deploying for new clients is complex as it has to meet their individual compliance requirements and banking and regulatory rules, regulations, and standards. And what about a foreign client bank? For this, banks must meet completely different banking and regulatory rules and compliance standards. 
And of course, overarching everything are stringent cybersecurity requirements, which are necessary in many cases as the application has to be able to securely interface with external core banking systems for daily updates on loan status with each customer.
Imagine duplicating this infrastructure each time a new client is onboarded using a non-IaC optimized CI/CD solution. 
<h3>IaC and CI/CD</h3>
As mentioned, a CI/CD solution is strongly recommended for provisioning and deploying IaC based infrastructures. One of the most widely used CI/CD solutions is Jenkins. 
However, <a href="https://spacelift.io/blog/terraform-jenkins" target="_blank" rel="noopener">Jenkins</a> and other generic CI/CD solutions were designed and optimized for application-centric workflows, and have limitations that will prevent scalability when implemented in support of IaC-specific development. Some of these issues include:
<ul>
<li style="font-weight: 400;" aria-level="1">IaC is a stack-based technology, and generic CI/CD tools do not understand stacks.</li>
<li style="font-weight: 400;" aria-level="1">No unified toolset or framework for policy management requiring manual checks.</li>
<li style="font-weight: 400;" aria-level="1">Lack of built-in synchronization, resulting in manual updates and synchronization.</li>
<li style="font-weight: 400;" aria-level="1">Inability to provide an automated workflow across multiple repositories and stacks.</li>
<li style="font-weight: 400;" aria-level="1">Cannot visualize or provide cost tracking of resources, <a href="https://spacelift.io/blog/drift-detection" target="_blank" rel="noopener">drift detection</a>, and remediation.</li>
<li style="font-weight: 400;" aria-level="1">In most cases, SSO and SAML are plugins—just another component to configure, manage, and update</li>
<li style="font-weight: 400;" aria-level="1">Conflicting pull requests where one developer is already working on a piece of code and another developer is also making changes, possibly resulting in code discrepancies.</li>
</ul>
Generic CI/CD solutions check all the boxes to support implementing a well-defined development process, adding end-to-end workflows allowing DevOps teams to deliver better applications faster, but not for IaC. 
<h3>Accountability and IaC</h3>
What do we mean by accountability? It is analogous to a business having a set of accounting books that represent the financial condition of the company as it stands at the present moment, but also its financial history. 
Without this history, it is impossible to trace back and establish accountability, to understand when financial decisions and changes were made, actions taken, and by whom and for what purpose.
Accountability is also applicable to your infrastructure as it enables you to understand the present state and how you arrived at this point. 
Infrastructure accountability provides checks and balances to ensure that you can account for all infrastructure-related events. From the first line of code to production, you need to know and understand every event that has or will affect the infrastructure to ensure that process integrity exists and, by extension, so does proof of compliance, both internally and externally. Some of the accountability questions that you have to be able to answer are:
<ul>
<li style="font-weight: 400;" aria-level="1">Code History:
<ul>
<li style="font-weight: 400;" aria-level="2">What is the history of the code?</li>
<li style="font-weight: 400;" aria-level="2">Who made changes to the code and infrastructure.</li>
<li style="font-weight: 400;" aria-level="2">Do you know if access to your repo is secure? </li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Change Implementation:
<ul>
<li style="font-weight: 400;" aria-level="2">Has a change been fully implemented?</li>
<li style="font-weight: 400;" aria-level="2">Can you validate and prove that it was deployed?</li>
<li style="font-weight: 400;" aria-level="2">Are paper-based policies and regulatory compliance requirements being met? Can you prove it?</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Change Approval Process:
<ul>
<li style="font-weight: 400;" aria-level="2">What is the change approval process?</li>
<li style="font-weight: 400;" aria-level="2">Are all changes approved, no matter how small or innocuous they seem?</li>
</ul>
</li>
<li style="font-weight: 400;" aria-level="1">Auditability:
<ul>
<li style="font-weight: 400;" aria-level="2">Are you prepared for an audit?</li>
<li style="font-weight: 400;" aria-level="2">Do you have the supporting materials necessary to satisfy security and compliance audits?</li>
</ul>
</li>
</ul>
What types of information do you need to answer the above questions?
<ul>
<li style="font-weight: 400;" aria-level="1">Provenance: The ability to trace every piece of code back to inception, along with any changes made.</li>
<li style="font-weight: 400;" aria-level="1">Compliance: Provide detailed information needed to ensure compliance with policies and procedures, and the ability to provide complete documentation for an audit.</li>
<li style="font-weight: 400;" aria-level="1">Access: Who has access, what do they have access to, what are they allowed to do, when are they allowed to do it, why are they doing it.</li>
<li>Visibility: Understand the state of the infrastructure and resources at any point in time.</li>
</ul>

<ul>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;5 Ways to Manage Terraform at Scale – Best Practices&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">5 Ways to Manage Terraform at Scale – Best Practices</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
<li><a href="https://spacelift.io/blog/drift-detection" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Drift Detection and How to Fix It With IaC Tools&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Infrastructure Drift Detection and How to Fix It With IaC Tools</a></li>
</ul>

The complex infrastructure example above gives an overview of a current ABL Fintech solution. This diagram represents the security that is provisioned and deployed on top of the infrastructure for the application.

image3-9

Fintech ABL Security Overlay

In many cases, individual clients require VPCs for their implementation, with multiple layers of protection, as they may have different security requirements or their specific vendor solutions in place. It is especially relevant if financial institutions have product offerings beyond ABL that have strict regulatory or industry compliance requirements based on their geographical location.
The Terraform logo highlights these security-specific resources supported in Terraform, and can be defined and provisioned.
The ability to control the following security aspects for access, source code, changes, and deployments, as well as validation thereof, is not achievable without embedded policy guardrails in place. They are essential to achieving secure infrastructure provisioning and deployments. Without policies in place, the probability of a bank passing an internal or external audit is very low.

Now that we’ve covered code platforms, CI/CD solutions, and accountability to help you understand the significant issues that stand in your way of IaC scalability, let’s take a look at what can help you to manage IaC at scale without workarounds, a manual, or a partially automated development process: 
<h3>Automate</h3>
Automation covers the end-to-end IaC workflow from repo to production:
<ul>
<li style="font-weight: 400;" aria-level="1">Integrated with source code repos such as Github to trigger IaC workflows when committing new or revised code.</li>
<li style="font-weight: 400;" aria-level="1">Creates a library of pre-configured infrastructure environments, enabling sharing across teams and projects.</li>
</ul>
<h3>Use Policy as Code</h3>
Policy as Code provides immediately enforceable guardrails:
<ul>
<li style="font-weight: 400;" aria-level="1">Declare rules around infrastructure, access to code and stacks, workflow, state changes, set triggers. </li>
<li style="font-weight: 400;" aria-level="1">Access controls for the 4Ws.</li>
<li style="font-weight: 400;" aria-level="1">Multi-layer change approvals and applies after approval.</li>
<li style="font-weight: 400;" aria-level="1">Audit support, SSO (SAML2.0) fully integrated. </li>
</ul>
<h3>Collaborate</h3>
Adaptable to existing workflows with no force-fitting or changes required, or fully customized to meet goals:
<ul>
<li style="font-weight: 400;" aria-level="1">Coordination of all runs with the ability to review, comment, and iterate across teams.</li>
<li style="font-weight: 400;" aria-level="1">Prevent conflicting changes and reconcile the differences automatically.</li>
</ul>
<h3>Introduce Resource Management</h3>
Understand resources and their state:
<ul>
<li style="font-weight: 400;" aria-level="1">Plan out changes.</li>
<li style="font-weight: 400;" aria-level="1">Drift detection and remediation.</li>
<li style="font-weight: 400;" aria-level="1">Cost of infrastructure.</li>
</ul>
In addition to all of the above, here are some additional vital capabilities that an ideal, purpose-built orchestration and management solution for IaC should also provide, at a minimum! Starting from the left of the diagram:
<ul>
<li style="font-weight: 400;" aria-level="1">Creation of pre-built library of infra components.</li>
<li style="font-weight: 400;" aria-level="1">Support of mono or multiple repos with version control.</li>
<li style="font-weight: 400;" aria-level="1">Ability to support multiple projects simultaneously.</li>
<li style="font-weight: 400;" aria-level="1">Integration and application of policy as code in concert with IaC configurations, ensuring that compliance and security initiatives are met.</li>
</ul>

image2-9

image2

Learn how to manage Terraform at scale here: <a title="5 Ways to Manage Terraform at Scale (Examples)" href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale (Examples)</a>

You can take positive actions to overcome the scalability issues of IaC code platforms, generic CI/CD solutions, and accountability to prevent your infrastructure from inhibiting your future business initiatives and growth.
You can be realistic and find an IaC orchestration and management solution that enables your development process to scale. 
And finally, you ought to be aware that proactive management and control over compliance and audit objectives are necessary and can be covered through Policy as Code, integrated into your IaC development, security, provisioning, and deployment workflows.
If you are concerned about achieving IaC scalability, <a title="try a free trial of Spacelift!" href="https://spacelift.io/free-trial" target="_blank" rel="noopener">try a free trial of Spacelift!</a> You can overcome code platform issues and enable accountability with a CI/CD solution built from the ground up for IaC. Control the process, visualize resources and create workflows that work, using Spacelift as your IaC orchestration and management solution.

Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure.  It helps overcome common state management issues and adds several must-have features for infrastructure management.

5 Ways to Manage Terraform at Scale - Best Practices

Business Benefits of Infrastructure as Code

Infrastructure as Code : Best Practices, Benefits & Examples

Managing infrastructure as code at scale tutorial. See how to manage IaC at scale. Examples and diagrams included. 

45-how-to-manage-infrastructure-as-code-at-scale

How to Manage Infrastructure as Code at Scale (Examples)

Peter is Head of Product Marketing at Spacelift. He brings over 20 experience in product marketing, having launched a
number of products ranging from complex network protocols to developer tools.

Warsaw and Redwood City, June 15, 2021: Poland and US-based startup <a href="https://spacelift.io" target="_blank" rel="noopener">Spacelift</a> has raised $15m in Series B funding led by Insight Partners to scale its management platform for Infrastructure as Code in the US market. Existing investors Blossom Capital, Inovo Venture Partners, and Hoxton Ventures also participated in the round. 
This new investment will allow Spacelift to quadruple its engineering team to accelerate product development and to build out its commercial team in the United States. The company is headquartered in Redwood City, California, USA, and has an office in Warsaw, Poland. 
Over the past few years, managing cloud infrastructure has become significantly more complex. The migration of infrastructures to the cloud has created an explosion in DevOps, but few overarching tools exist to help teams of developers safely manage infrastructure as code at scale. Spacelift has created a plug-and-play platform that allows IaC DevOps teams to: define standardized infrastructures; understand their resources; build policies directly into their workflows and code; synchronize code across multiple projects and/or repositories; automatically detect and remediate resource drift, track lifecycle and cost of every managed resource; and take advantage of advanced workflow delegations and autonomous project teams. 
“With the growth of cloud-native development and DevOps practices, infrastructure as code has become the standard way to provision infrastructure, and Spacelift is addressing an acute need around infrastructure as code management, automation, and security and compliance,” said Josh Zelman, Vice President at Insight Partners. “Spacelift enables developers and DevOps teams to more easily and effectively manage infrastructure across all platforms. In speaking with DevOps teams within the Insight portfolio, we continuously heard about this need and the strength of the Spacelift platform. Insight is thrilled to partner with Spacelift and support its ScaleUp efforts in its next phase of growth.”
Spacelift is the only IaC management solution that extends IaC DevOps to encompass the complete set of IaC frameworks including Terraform, Pulumi and CloudFormation, with Ansible coming soon. Offering a consolidated approach to IaC is beneficial to engineering teams in large corporations that have been acquiring smaller companies that bring different infrastructures and to smaller but diverse organizations aiming to empower application developers. 
Marcin Wyszynski, co-founder and CEO at Spacelift, said: “Since we&#8217;ve founded Spacelift, we&#8217;ve seen a Cambrian explosion of new tools designed for teams to make the most of their cloud &#8211; ranging from cost management to audit to proactive security. Hence our vision of providing the most flexible and extensible platform for infrastructure and policy as-code is now more relevant than ever before. By using Spacelift, our customers get the whole cloud management story in one convenient package.”
Pawel Hytry, co-founder and COO at Spacelift, said: “Over the past few years, IT organizations relied on generic CI/CD tools, open-source components and custom-made solutions to manage their infrastructure. With increasing infrastructure complexity and development of infrastructure as code, this approach proves unscalable and error-prone. Spacelift addresses the issues of scaling and infrastructure complexity by providing a robust, <a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener">specialized management platform for infrastructure as code</a>. We are eager to partner with Insight Partners and benefit from their experience in the developer tools space.” 
About Spacelift: 
Spacelift is building a platform to help DevOps teams manage their infrastructure-as-code performance for maximum optimization of scaling businesses. Founded in 2020, the management platform allows teams to manage complex cloud infrastructures simultaneously, providing visibility and collaboration. Spacelift has raised a total of $22.6M in funding from Insight Partners, Blossom Capital, Hoxton Ventures, Inovo Venture Partners, and the founders of Supercell and Yelp.
About Insight Partners:
Insight Partners is a leading global venture capital and private equity firm investing in high-growth technology and software ScaleUp companies that are driving transformative change in their industries. Founded in 1995, Insight Partners has invested in more than 400 companies worldwide and has raised through a series of funds more than $30 billion in capital commitments.

<ul>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Problems that Spacelift Solves&quot;}" data-sheets-userformat="{&quot;2&quot;:4224,&quot;10&quot;:2,&quot;15&quot;:&quot;Arial&quot;}">Infrastructure Problems that Spacelift Solves</a></li>
<li><a href="https://spacelift.io/blog/alternative-to-atlantis" target="_blank" rel="noopener">Alternative to Atlantis – Spacelift</a></li>
</ul>

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Infrastructure Drift Detection and How to Fix It With IaC Tools

$15M in a new funding round led by Insight Partners will enable  Spacelift to quadruple engineering headcount in Europe and build a sales and marketing organization in the United States.

Peter Christensen

How to Manage Infrastructure as Code at Scale (Examples)

Get the newsletter

Recent posts

Spacelift raises $15m in Series B funding from Insight Partners