Why Generic CI/CD Tools Will Not Deliver Successful IaC

Infrastructure as code (IaC) is still in its infancy. According to the August 2022 Global Market Insights Inc, the market is predicted to grow from $1B to $3.5B by 2030, at an annual rate of around 22%, and other reports predict similar growth. As the market matures, the complexity of adoptions will increase, intensifying the need for adequate processes and tools.

In this post, we focus on the four key elements that successful IaC delivers and generic CI/CD tooling does not:

1. Automation
   
2. Continuous processes
3. Infrastructure orchestration
4. Governance

We outline how a dedicated solution can help businesses attain these four pillars and successfully manage their IaC, debunking the myths around dedicated IaC solutions, and concluding with the benefits of choosing a specialized platform for managing IaC successfully.

IaC emerged from the Automation element of the CALMS (Culture, Automation, Lean, Measurement, and Sharing) framework and is a pillar of the modern DevOps Approach. Automation has always been a goal of engineers, who previously used scripting languages like Bash or even programming languages to try to achieve it. 

Ultimately, Configuration Management emerged as a way to manage operating systems configuration, application deployments, and configurations; and IaC was introduced to manage cloud infrastructure in a way that maximized business value.

Apart from delivering business benefits, IaC allowed Agile teams to manage their own infrastructure more easily and effectively. Interaction with infrastructure (even cloud infrastructure) was no longer siloed but could be equally distributed among multiple teams. However, this could also prompt issues that we will discuss later. 

IaC certainly introduced valuable benefits, but the process only automated single executions — not the entire process. Many organizations implemented CI/CD for applications, but infrastructure was treated separately, and automation (in the form of Configuration Management or IaC) was still executed manually. This led to significant issues, including inconsistent executions, incomplete understanding of the process inside teams, lack of communication, long delays, and lack of control. These made the process for infrastructure unreliable and unpredictable.

The solution was CI/CD processes and tools. 

According to the State of CD report, if developers are involved in DevOps activities, 29% of them manage and provision infrastructure programmatically, with different kinds of IaC tools. The report also states that when using multiple DevOps technologies, companies can decrease lead time, increase deployment frequency, and ultimately decrease the MTTR (Mean Time To Restore).

Combining IaC execution with CI/CD pipelines finally allowed organizations and teams to automate processes, not just executions.

Let’s look at the diagrams below:

In the simplified picture of the process above, we clearly see the discontinuity between change and execution. We need two manual, unconnected steps: One is publishing the code of IaC, and the second concerns deploying to the target environment (in this case, AWS).

This leads to multiple problems, such as gaps in governance and compliance, broken processes, etc. CI/CD allows us to fix these gaps.

Adding an automated process of execution through CI/CD not only completes the flow, but also removes manual steps from the process. We can clearly understand what change triggers execution. We can control the process by understanding the whole flow.

Combining those two approaches — for infrastructure and for delivery — created a continuous flow, which should be instant in terms of trigger. There is no delay between committing the code and action triggered in the CI/CD tool. In this way, the organization significantly reduces one of the Lean’s waste – waiting state.

In the CI/CD world, specialized tools like ArgoCD or Octopus focus on one function or one group of products, whereas generic CI/CD tools like Jenkins, CircleCI, and GitHub Actions cover the whole Continuous Integration/Continuous Delivery (or Deployment) field. 

The main limitation of CI/CD tools derives from the design of the approach – not the specific tool – and is a default behavior of this type of tool. DevOps is pictured as an infinite loop, whereas the CI/CD pipeline is represented by a straight line. This line is a part of the infinite loop, of course, but the problem here is that one execution of the CI/CD pipeline is a single, encapsulated entity, which is not tightly connected to the whole lifecycle.

Let’s look at the simplified view of the generic pipeline:

How do engineering teams build IaC code?

Well, there is no one answer to this question. However, in more complicated solutions we can observe multiple templates and even multiple stacks with very focused functions. Create a network layer, create a database layer, a storage layer and so on.

The diagram below illustrates this approach:

We see a complete disconnection between pipelines, which leads to multiple issues.

As the organization grows, so does the infrastructure. In the case of application-focused pipelines, it requires more work to properly manage the growing number of application pipelines. 

Using the architecture patterns of microservices in the proper way eliminates this obstacle, but it also creates siloed areas of responsibility and management.

In the case of infrastructure, engineering teams apply many solutions, and there is almost always a disconnect between infrastructure elements in different stacks. As the infrastructure grows, the complexity of managing stacks also intensifies because more stacks are created and executed, and the increased connections between these stack executions must be managed.

Modern CI/CD tools allow us to trigger one pipeline from another or configure the pipeline execution in a flexible way, but this is just a workaround for the main problem— a lack of orchestration.

Organizations should enforce three main goals for governance when thinking about IaC: compliance, security, and cost control. These elements are not relevant to scale. All organizations want to be secure and run their business in a cost-effective manner. However, the complexity of governance management grows as companies scale. Furthermore, companies operating in sectors such as finance, energy, or healthcare are governed by specific industry regulations.

Generic CI/CD tooling can be adapted to these restrictions from a technical standpoint, but this results in a siloed approach with risks for collaboration, communication, and understanding of the process.

Proper security for IaC is crucial. Infrastructure is highly sensitive to all vulnerabilities, not just as a potential vector of attack for cybercriminals, but also from the perspective of the risks for the infrastructure’s stability and scalability incorporated in its design. According to Prisma Cloud’s State of Cloud-native Security Report 2023, 90% of organizations struggle with detecting and fixing cyber threads within an hour. Of the respondents to this report, 78% stated that they are expected to have minimal learning for cloud security. 

This should be an “aha” moment for organizations. We enforce a shift-left approach, which is appreciated by teams, but the point is that security for infrastructure is different from security for applications, and development teams shouldn’t be forced to learn a whole new area. And here we hit a huge drawback of generic CI/CD tools, where teams are fully responsible for their pipelines. Even with shared libraries etc., the team that establishes a new pipeline must be aware of the configuration.

Generic CI/CD tools can achieve only very loose coupling between automation, continuous processes, orchestration, and governance.

This is because the main goal of CI/CD tools is the continuous integration and delivery of code to production systems. CI/CD utilizes automation, but from a process perspective, it doesn’t matter for the CI/CD tool what code it ships. CI/CD can orchestrate and govern the entire setup, but it is mostly manual work and requires specialized knowledge from all development teams.

This decoupling is shown in the diagram below:

Dedicated tools like Spacelift encapsulate the whole spectrum of infrastructure SDLC in one tool, as in the diagram below:

With this in mind, organizations can control the whole process from all angles, like tooling, processes themselves, governance, security, and compliance.

Debunk the myths about dedicated tools

It is a common perception that organizations should limit the number of tools in the delivery chain. This does not mean that organizations should not use specialized tools for their chain of operations. We have to analyze the business value of the tool:

• Does the tool decrease the complexity of the SDLC?
• What part of SDLC can be covered by the tool?
• Does the organization have the skills required to use the tool properly?
• Does the tool affect the lead time?
• Does the tool affect service restoration?

Add the proper tool to improve lead time

Chapter 4.3 of the State of the CD Report 2023 mentioned earlier confirms that organizations with diversified tooling are more likely to be top performers in terms of lead time for code changes. The important thing is to use a prudent approach to incorporating tools in the SDLC – not simply add more tools for the sake of it. 

Improve distribution of skills in your teams

Another myth is that more tools means you’ll need more skills. If the organization considers all business needs and opportunities prudently, the opposite might be the case. If continuous automation is characterized by proper orchestration and governance, workloads are decreased for stream-aligned teams.

Pay for the tool, not for work

It is common to view tools, licenses, etc. as costs for the organization. However, if the tool decreases the amount of work needed, the organization lowers its operational costs. Upskilling all development teams to use specific IaC, CI/CD, and orchestration tools will be more time- and effort-consuming than using a dedicated tool within a dedicated team.

Don’t be discouraged by vendor lock-in

The vendor lock-in dilemma is raised frequently, especially in the cloud-native approach. The perception of avoiding vendor lock-in at any cost is unfortunate and might lead to unnecessary issues in processes. Every organization is subject to some vendor lock-in, by selecting a cloud vendor, software vendors, programming languages, or even a specific delivery methodology.

Generic CI/CD tools are simply not up to the task of fully controlling and monitoring infrastructure. As we discussed previously, they don’t know what technology they ship. They don’t know what happens before or after a pipeline is executed. The CI/CD pipeline is designed to deliver the code from VCS to production in an automated, controlled way.

Spacelift’s platform includes delivery pipelines but it is also aware of the technology the organization uses and is able to orchestrate and govern the whole infrastructure.

Let’s take a look at these elements one by one:

Awareness of AUTOMATION

Spacelift is aware of infrastructure technology. During the process of creating the stack, the proper backend technology must be selected – for example, Terraform. This determines the behavior of the run. Applying specific policies also increases this awareness of the technology used in the process.

CONTINUOUS processes

As core functionality, Spacelift stacks are connected to the Version Control System of your choice, and each run can be executed after committing code to the repository. Furthermore, this functionality is also used for the pull requests process. This approach decreases the lead time significantly and allows control of the executions in a very granular way.

ORCHESTRATE the SDLC

Spacelift orchestrates all your stacks, executions and technologies through a couple of mechanisms implemented. With Blueprints, organizations can enforce optimum infrastructure quality and security. Spaces allow organizations to control the executions and privileges in a very granular way. This is achieved by specific configuration of access for users, variables, and variable sets (called contexts). To enforce orchestration, Spacelift offers policies, stack dependencies, and customizations of the workflow.

Control your infrastructure with properly implemented GOVERNANCE

Finally, Spacelift allows organizations to implement and enforce the proper governance models. 

Security of the process can be enhanced through Blueprints, Spaces, policies, and granular access mentioned previously. For Terraform, this can also be done through Terraform registry.

Execution can be secured easily by implementing security frameworks for the runs, and, most importantly, this implementation can be enforced by policies applied to stacks.

Implementation of cost control is possible for Terraform stacks using custom inputs and policies with Infracost. Finally, with full view on the resources deployed, the organization is able to understand and control all infrastructure resources.

Spacelift allows organizations to use one tool to conveniently govern and orchestrate the continuous automation of infrastructure. This allows businesses to increase the cohesion of their infrastructure and decrease lead times and restore times 

The table below sums up the advantages of using Spacelift as a dedicated tool for infrastructure management:

As Eficode stated in their DevOps Trends 2023 guide, organizations move beyond automation to orchestration. This statement is valid for infrastructure too. This type of orchestration allows organizations to control waste and inefficiencies in their value streams (also stated in Eficode’s guide).

Spacelift offers a single pane of glass for your infrastructure. In an industry where understanding and the ability to act quickly is crucial, using this platform provides a significant advantage over generic tools, which were designed mostly for handling application code, not infrastructure.

Are you looking for the best orchestration of your infrastructure? Book a demo with our engineering team to discuss your options in more detail.