Automation is the fuel that propels the world forward today. Irrespective of all the challenges, it has proved to be beneficial for any ecosystem. We have witnessed this for centuries – almost all technological revolutions that have happened have automation as a consistent attribute.
DevOps is a term coined a few years back. However, the efforts to automate the transport of workloads and updated software in production have been carried out for decades. The term DevOps is an acknowledgment from the industry, which has given rise to the whole slew of tools that have become part of the DevOps toolchain.
In my opinion, DevOps can be defined as one of the disciplines of automation that assures reliable delivery of software products and services in today’s ever-growing complex world. In this post, we take a look at some of the most useful DevOps tools that define the state of today’s software CI/CD automation:
Note: The tools mentioned in the following sections are in alphabetical order and not in any order of preference.
Azure DevOps by Microsoft is an all-in-one CI/CD platform that features entire software delivery in one place. As the name suggests, it is more than just a CI/CD tool. Below are some of its features.
- Azure Repos – It is a cloud-hosted private Git repository service. Azure Repos enables team collaboration by providing features like branching, tagging, pull requests, etc., which are essential.
- Azure Boards – Epics and stories that are created in a backlog and made part of the sprints required for agile software delivery are supported by Azure Boards. Azure Boards make it possible to coordinate team efforts. The stories have (sub) tasks that are linked to the relevant commit in Azure Repos. This enables traceability across the development and management planes.
- Azure Pipelines – Azure Pipelines enable the CI/CD automation of the software being developed. It can be integrated with any remote Git repository and not just Azure Repos. The extensions marketplace offers a number of pre-defined tasks which are reused along with custom tasks. The development of the CI/CD pipeline in Azure Pipelines follows an industry standard of YAML syntax.
- Azure Artifacts – Software artifacts like binaries, packages, and files, often need secure storage and enforcement of the right access controls so that only the required automation workflows use these artifacts. The outputs generated by the build processes are usually stored in Azure Artifacts for quick, seamless, and secure access.
- Azure Test Plans – Azure Test Plans offer a great set of tools to run tests on various combinations of host OS and browser suites, and enable end-to-end traceability of bugs. It provides a great UI to showcase indicators like test coverage. Test Plans truly help teams to deliver quality software with speed.
What makes Azure DevOps stand out is its ability to wrap all of the above in a single window. There are hardly any tools out there that encompass the end-to-end software development lifecycle automation like Azure DevOps.
The clean interface presented by Azure DevOps makes it easy to understand, navigate, and implement the CI/CD automation. It also boasts the release management functionality for systematic software releases – all integrated within Azure Pipelines. In the case of container images, it integrates well with Azure Container Repository (ACR) to push and pull images.
Similar to ACR, Azure Pipelines also integrates natively with other Azure cloud services for deployment. Service principals and managed identities provide a secure way to perform deployment activities from Azure DevOps.
Considering the fact that a good percentage of organizations use Microsoft products and Azure for their office work, using Azure DevOps is a natural choice for CI/CD automation in large teams.
There are many containerization tools available in the market, but Docker has stood the test of time. Docker is used to build application images run in standalone mode on virtual machines or in a cluster as pods.
When an application source code is ready to be deployed or when new features are ready to be delivered, the first step is to containerize the application using Docker. The diagram below shows the general process of the same.
Docker’s architecture is mainly divided into three parts. First, the Docker CLI is used to interact with the Docker daemon, which is a process that builds, manages, and runs Docker images. The built images are stored in remote repositories, from where the CI/CD workflows fetch them for deployment purposes.
Finally, in the sub-prod and prod environments, these applications are run in the form of containers, which are instances of the images stored in repositories. The Docker engine is a term that represents the complete Docker core – that includes the CLI and Daemon and supports container runtime.
The Dockerfile contains the steps to build a container image that is understood by the Docker daemon. It usually starts with specifying the base image of the OS, certain configuration and patching tasks, and finally, the source code.
If the compilation steps are required, they are specified as well. The application image thus created contains all the necessary runtime environment variables and OS requirements packaged together. A successfully built image can therefore be run on any machine that hosts a Docker engine.
The Docker engine also takes care of the resource management for each container. Resource limits can be managed in a way that not all resources offered by the host machine are consumed by a single running container. This makes it possible to run multiple instances of multiple applications on the same machine.
Docker helps package the application’s source code into an image that is reused in any environment. This ability offers great reliability in the overall software delivery process. Docker has made it possible to seamlessly package and ship products with speed. See our Docker tutorial to learn how to use this tool.
Any source control system mainly has two parts – local Git to manage the code versions locally and remote repository to enable team collaboration. GitHub has been around for a long time and is a crucial part of the source code management systems available for free on the internet. It acts as a global remote repository for anyone who wants to develop in any language or framework.
GitHub Actions is a CI/CD platform integrated within GitHub repositories and available for all the repositories one may have. Once the code is pushed or merged in a GitHub repo, the same event is used to trigger build and deploy pipelines. Similarly, there are ways to configure a mechanism to select specific events responsible for triggering the GitHub Actions pipeline.
All the pipelines are configured using YAML syntax, and they usually consist of jobs. Jobs, in turn, are executed via a series of steps. GitHub Actions are the steps included in jobs, which are a series of commands and instructions that perform a specific task.
There is an ecosystem of pre-defined actions available to choose from in any GitHub Actions pipeline. These Action templates are provided by various vendors depending on their use cases. It is also possible to create and publish custom GitHub Actions to be used within the organization or published to the GitHub Marketplace so that other developers can take advantage of the same.
The jobs and steps included in any GitHub Action pipeline are executed on runner machines provided by Github. These are Github-managed compute resources. However, it is also possible to host private runners to run, build and deploy private applications.
Here is an example of a GitHub Action that initializes the Terraform code on the Runner system whenever some code is pushed to the main branch of the repo. Comments on each step explain the purpose for the given step.
# Name of Github Action
name: Initialize Terraform
# Trigger condition: Start this action everytime a source code is pushed on main branch.
on:
push:
branches: [ "main" ]
jobs:
# Name of the Job
tf-init:
# OS and version specification of the Runner.
runs-on: ubuntu-latest
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v3
# Using "HashiCorp - Setup Terraform" pre-built Action from Github Marketplace, to install Terraform on Runner machine.
- name: HashiCorp - Setup Terraform
uses: hashicorp/setup-terraform@v2.0.0
with:
terraform_version: latest
terraform_wrapper: true
# Custom Action created to initialize the Terraform code
- name: Init
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_TCF_DEV_KEY }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_TCF_DEV_SECRET }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_TCF_REGION }}
run: |
cd infra
terraform init
Apart from the SCM and CI/CD (Actions) capabilities, Github also provides users with basic project management tools and secrets management features to manage secrets used in pipelines, issue tracking, and insights. Some of these features are spread across Github Team and Enterprise, where organization support is also provided.
Additionally, it is also possible to build GitHub Apps, which can integrate with the CI/CD process to enhance the process itself or fetch feedback from all the GitHub Actions tasks for reporting.
GitHub is used by almost every developer on the planet, and it provides great capabilities. Starting out with GitHub Actions is very easy and leveraging it for automation workflows is valuable. It is one of the few resources available today for free that offers complete CI/CD capabilities and more.
See also: Managing Terraform with GitHub Actions & Scaling Considerations
Managing cloud infrastructure manually to host any product or service manually is tedious and risky. Terraform is an Infrastructure as Code (IaC) tool by Hashicorp that provides automation capabilities for infrastructure provisioning. Terraform uses HCL (Hashicorp Configuration Language) to express the infrastructure in the form of code and manage the end-to-end lifecycle of cloud resources.
Creating and managing infrastructure using code inherently leverages the advantages offered by programming practices and source code management systems. Similar to code, the infrastructure is also versioned, tracked, and rolled back. We can create and publish modules to create infrastructure to support reusability, thus improving the delivery velocity.
Terraform IaC syntax is declarative. Any infrastructure component developed in the form of code is created if it is not yet created. Dependencies are implicitly taken care of, while it is also possible to include explicit dependencies. We essentially express the desired state in Terraform configuration, and Terraform helps us provide the same.
The diagram below represents a typical workflow when managing infrastructure using Terraform. Developers develop the infrastructure using HCL on their local machine in files with extension .tf. The versions of these files are managed in remote git repositories.
To create the real-world resources using this Terraform code/configuration, it needs to be executed (apply command) on a Terraform host. A Terraform host is a machine where the Terraform binary is available that can interpret the configuration and call appropriate cloud provider APIs to create these resources.
Terraform maintains a state file that holds the mapping of real-world objects created to the configuration. If the configuration is changed and applied, the corresponding change is also reflected in the state file. Thus, the state file gains immense importance, and it is managed in a remote backend which is different than the Git repository as a best practice.
Take a look at 8 Popular Terraform Alternatives.
Since Terraform works with all the major cloud providers, it is cloud agnostic. The syntax used to develop infrastructure remains the same. However, the provider’s and module’s names may differ. The documentation on Terraform providers and modules is maintained at the Terraform Registry.
It is possible to build custom modules and publish them to the public or private registry. Modules are also published on the remote SCM systems. Additionally, for private data centers, it is possible to create a custom provider that helps developers develop and deploy infrastructure components in the private data centers.
With cloud adoption being a major factor in any organization’s digital transformation journey, learning Terraform as a skill makes a lot of sense. Configuration management tools like Ansible have been around to manage the lifecycle of the applications. However, Terraform manages the lifecycle of the underlying resources via code – this enhances automation capabilities to newer heights.
Read more about deploying your infrastructure in CI/CD using Terraform.
When we talk about “developing” Infrastructure as Code, we have to acknowledge that the IaC was just the beginning. IaC tools like Terraform, Pulumi, AWS CloudFormation, etc. help us encode our infrastructure and leverage best practices from the programming paradigm. But, IaC alone is not enough to truly manage the infrastructure seamlessly.
Spacelift provides a platform to manage IaC in an automated, easy, controlled, and secure manner. We touched upon remote backends in the last section. Spacelift goes above and beyond the support that is offered by the plain backend system. Spacelift is a sophisticated CI/CD platform for IaC that enables Gitops on infrastructure provisioning workflows.
Cloud infrastructure is prone to changes. There are many attributes that can be “tweaked” unknowingly. The reasons could be anything – human error, overlapping automation processes, 3rd party triggers, etc. The fact is that the infrastructure is prone to unwarranted change, and this can cause uncertainty in the way it is managed by IaC tools. This phenomenon is also known as infrastructure drift.
On the one hand, where we try to automate the lifecycle management of infrastructure, on the other hand, we also need an automated way to easily detect such configuration drifts. Spacelift features automatic discovery and remediation of infrastructure drift (drift detection feature) and alerts the team with possible fixes.
We can define CI/CD flow for infrastructure on Spacelift, and it integrates well with popular version control systems like GitHub, GitLab, Bitbucket, and Azure DevOps. Thus, all the updates of the run triggered by pull requests are recorded and traceable to the responsible commit. It also implements an approval process to regulate the deployment process and avoid surprises.
Spacelift offers a great visual aid to see all the resources and flows in action. The intuitive UI helps quick understanding of the current state and provides actionable insights. Additionally, it also offers a view of cost estimates for the infrastructure being managed via Spacelift.
There are many tools in the DevOps ecosystem, and the ecosystem is constantly growing and improving. Till a few years back, tools like Jenkins dominated the market for automating workloads. With newer capabilities, the DevOps space is constantly evolving. The article covers some of the most relevant and essential tools today
However, it should be noted that automation and DevOps go beyond the tooling or the toolchain. It is more about adopting practices that are beneficial for software delivery in terms of quality, consistency, peace of mind, and costs. In the earlier days, it was often said that adopting DevOps is more about adopting a new mindset. Today, it is not just about adoption but the evolution of the mindset as allowed by the tools with cutting-edge capabilities.
The Most Flexible CI/CD Automation Tool
Spacelift is an alternative to using homegrown solutions on top of a generic CI solution. It allows you to automate, audit, secure, and continuously deliver your infrastructure.
