15 GitOps Best Practices to Improve Your Workflows

GitOps refers to the use of code-based processes for all aspects of software delivery. It extends DevOps practices that have proved successful for app development to also power infrastructure management and operations.

Developers have long-versioned code using Git repositories and pull requests. This makes it easier to review changes, collaborate on projects, and revert problematic deployments. GitOps formalizes this workflow as a framework that can be applied throughout the DevOps lifecycle. The approach lets you manage infrastructure using the same high-level techniques as you use for application code.

Once you’ve defined your infrastructure’s configuration—via IaC tools like Terraform and Pulumi—you can use CI/CD pipelines to automatically provision your resources. Changes are made by modifying the IaC files in your repository, creating a pull request, and then waiting for the CI/CD pipeline to update your live resources. 

App deployments are handled in exactly the same way, such as by using a Git repository to store Kubernetes manifests that are then applied in CI/CD. GitOps ensures your deployments always match the configuration visible in your repositories.

GitOps offers compelling benefits for DevOps teams automating processes at scale. It eliminates manual tasks and reduces uncertainty over what’s deployed. However, it’s important to follow best practices to ensure your implementation is reliable, performant, and capable of adapting to changes in your workflows.

Here are 15 key factors to consider when adopting GitOps in your teams:

1. Fully automate apps and infrastructure
2. Modularize projects and pipelines
3. Decouple development, delivery, and deployment
4. Use declarative configuration for all resources
5. Keep Git as the single source of truth
6. Adopt pull-based, agent-driven deployments
7. Implement immutable deployments and releases
8. Enable drift detection for infrastructure changes
9. Ensure GitOps portability across environments and clouds
10. Nurture a GitOps culture in your organization
11. Apply GitOps to infrastructure and configuration, not just apps
12. Choose tools with native integrations
13. Use trunk-based Git branching for GitOps repositories
14. Implement progressive delivery strategies
15. Enforce compliance with policy-as-code governance

The basic definition of GitOps is simple: the state of your resources is defined by code and config files stored in a Git repository. Tools then process those files to populate your live environments. 

This isn’t an inherently automated workflow, though: in small teams, it may be up to DevOps team members to periodically pull repositories and run terraform apply or kubectl apply against them.

To make GitOps work at scale, you need to fully automate the process. Use CI/CD pipelines to automatically roll out changes when new commits are merged. This ensures your apps and infrastructure always match the state of your repositories. 

It also prevents the errors and conflicts that can occur when two developers manually apply changes at the same time. Moreover, you won’t need to grant developers direct access to cloud credentials.

GitOps workflows can be challenging to maintain at scale. New components added to your stack cause repositories to grow and become disorganized, while CI/CD configurations end up being duplicated to handle small differences in environments.

You can mitigate these problems by splitting your configs into smaller reusable units. For instance, publishing your own Terraform module registry allows you to compose your configs from existing components. This reduces maintenance overheads and allows new resources to be assembled more efficiently. 

Avoid hardcoding values in your configurations by instead reading environment variables you can customize each time a module is used.

GitOps allows you to automate your entire software lifecycle, from development through to delivery and deployment. Nonetheless, you should keep these tasks decoupled so they’re easy to govern and maintain. This means using separate repositories and pipelines for each config type:

• Development: Source code and IaC files should be tested and validated first.
• Delivery: A separate pipeline prepares changes for continuous delivery, running policy-as-code checks stored in another repository.
• Deployment: A continuous deployment flow handles the actual rollout to production, executing commands like terraform apply.

Keeping each phase distinct lets you more easily track progress, enforce access requirements, and pinpoint the causes of failures.

GitOps revolves around declarative configuration. The configuration files in your repositories should describe what you want to be deployed, not how to deploy it. This makes your configs easier to understand, maintain, and audit. 

Declarative configuration is also less prone to errors compared to traditional infrastructure provisioning scripts, where developer mistakes could inadvertently reorder important operations.

Kubernetes and Terraform are two popular tools that are primarily declarative in nature: you specify that you want a certain infrastructure component to exist; the platform then automates the process of achieving that state in your environment.

GitOps only succeeds when it’s the single source of truth. This means your repositories should solely define the states of your environments. Conflicts, inconsistencies, and errors easily occur when other processes are used at the same time.

For example, developers may manually reconfigure resources if they feel it’s quicker or more convenient. However, those changes would be reverted and lost the next time the GitOps pipeline runs unless they’ve been committed to your repository.

GitOps CI/CD pipelines give you a choice of two different high-level strategies: push or pull. 

• Push-based CI/CD is the traditional approach. A script running on your CI/CD server runs commands like kubectl apply to “push” deployments to your infrastructure.
• Pull-based deployment is the modern alternative. Under this model, an agent running inside your target environment periodically polls your Git repositories. The agent “pulls” the changes and automatically applies necessary actions to reconcile your environment’s state.

Compared to push-based deployment, pull-driven strategies offer automation, reliability, and security benefits. In particular, you don’t need to share infrastructure access keys with your CI/CD server, so there’s less chance of live environments being breached.

Immutable deployments make GitOps release processes safer and more predictable. They enable you to treat new releases as fully independent snapshots that will never change once created.

The practice means that instead of continually reusing tags like latest and production, each release should be assigned its own unique identifier. These IDs are typically derived from the Git commit SHA that’s being deployed. Once a release has been tagged, it should be impossible to reuse that tag or change its contents.

Immutable deployments make it easier to revert problematic changes. You can simply redeploy the last good tag instead of pushing another update to the latest. Immutability is also a crucial compliance tool, as it helps prove that releases haven’t been tampered with post-creation.

Drift is one of the biggest issues facing infrastructure teams at scale. It’s the phenomenon that occurs when resources gradually deviate from their expected or desired state over time. In practical terms, drift means your live infrastructure no longer matches the configuration defined in your Git repositories despite your successful adoption of a robust GitOps workflow. It can cause errors, misconfigurations, and compliance breaches.

Automated drift detection scans enable you to find and fix drift as it happens. They compare your infrastructure’s current state to the configuration defined in your Git repository, using IaC tool features like terraform plan. 

Drift usually becomes harder to solve the longer that it exists, so regular scans are crucial to ensure your infrastructure runs reliably. Drift detection and resolution capabilities are typically found in IaC orchestration platforms like Spacelift.

GitOps can enhance operational flexibility by standardizing multi-cloud deployments and enabling concurrent releases to different environments. However, this depends on your GitOps workflows being designed for portability. You should avoid using tools and processes that are tightly coupled to specific cloud providers, as this increases the risk of vendor lock-in.

Infrastructure configurations are inherently specific to individual clouds—the Terraform config for an AWS EC2 instance will look different from one for GCP GCE, for instance—but there are still steps you can take to improve portability. For example, it’s often possible to split common config sections out into reusable modules that reduce the amount of duplication required.

Advanced tool choices such as Crossplane can also help abstract differences between clouds. Crossplane enables you to provision infrastructure components using Kubernetes custom resources. It means developers can create a new ComputeInstance or Database without understanding all the cloud provider interactions involved.

GitOps success depends on your whole team adopting it with confidence and enthusiasm. You need to nurture a mindset that ensures IaC, CI/CD, and declarative configuration are everyone’s default approach to operations.

Team members need to understand how GitOps processes work, why they’re used, and how they benefit from them. For instance, you could highlight how GitOps reduces the frequency of infrastructure incidents, giving developers more time to focus on meaningful work. 

Creating a personal connection makes it less likely that individuals will try to bypass GitOps and continue using legacy processes.

Many DevOps teams have started adopting GitOps for app deployments, such as by using tools like Argo CD to release to Kubernetes. However, this is just one key GitOps use case—the strategy applies equally well to infrastructure management using IaC tools and configuration management via solutions like Ansible.

Aligning your entire software delivery lifecycle around GitOps improves the consistency and maintainability of your projects. It empowers you to manage all aspects of your apps and infrastructure using a single high-level automated process. Not only does this simplify operations, it also helps share skills and knowledge. 

Developers who are not specially trained in infrastructure management could safely apply urgent infrastructure changes by committing IaC files via your GitOps workflow.

GitOps and related topics like IaC and CI/CD are some of the most active fields in the DevOps space. New tools regularly appear, so there are plenty of options to choose from for different use cases. 

However, it’s important to check your platforms can seamlessly integrate with each other as this simplifies setup tasks and supports your security posture.

Combining a major version control system, CI/CD platform, cloud provider, and IaC tool shouldn’t pose a problem, but the situation could be trickier if you’re using more specialist solutions. Make sure you audit each tool to understand how it interacts with the others and any security risks it presents. 

The benefits of native integrations can be seen in Spacelift’s connectivity with cloud providers, for instance, which dynamically generates short-lived cloud credentials to use in your IaC runs.

GitOps works best with trunk-based Git branching. This refers to the practice of developing all changes using short-lived feature branches that are merged into a single “trunk” branch. It avoids the merge conflicts and overheads that occur when developers try to maintain and synchronize multiple adjacent branches like main, develop, and release.

With trunk-based branching, you can simply point your CI/CD and IaC agents at your repository’s trunk. 

Developers work on feature branches and then merge them into the trunk once all necessary approvals have been made. The agents will detect the merge and automatically apply the trunk’s current state to your live environments. This eliminates branching complexity and ensures what’s deployed is always clear.

GitOps makes it easy to implement progressive delivery strategies such as canary and blue-green deployments. These techniques improve release safety by initially limiting new deployments to a subset of users.

Tools such as Argo Rollouts gradually ramp up availability until everyone’s using a new release without requiring any manual intervention. If a failure occurs, it’s possible to automatically revert to the last good state from your repository. GitOps continually provides the source of truth for what the final deployment should look like.

GitOps workflows are powerful and convenient, but this means they also pose security and compliance threats. It’s crucial to enforce granular access controls so that only authorized developers can commit changes. 

Moreover, you need to be able to govern approval conditions and prevent possible misconfigurations from being applied.

Platforms that support Policy-as-Code engines like Open Policy Agent (OPA) provide the versatility and control needed for GitOps workflows at scale. Policy-as-code lets you define precise rules that your IaC, CI/CD, and app deployment configs must meet before they’re merged. 

Because the policies themselves are code, they can also be managed as part of a GitOps workflow. This enables you to automatically test your rules before they’re enabled in your projects.

Spacelift is an IaC management platform that uses GitOps to automate CI/CD for your infrastructure components. It supports OpenTofu, Terraform, Terragrunt, CloudFormation, Pulumi, Kubernetes, and Ansible.

The power of Spacelift lies in its fully automated hands-on approach. Once you’ve created a Spacelift stack for your project, changes to the IaC files in your repository will automatically be applied to your infrastructure. 

Spacelift’s pull request integrations keep everyone informed of what will change by displaying which resources are going to be affected by new merges. Spacelift also allows you to enforce policies and automated compliance checks that prevent dangerous oversights from occurring.

Spacelift includes drift detection capabilities that periodically check your infrastructure for discrepancies compared to your repository’s state. It can then launch reconciliation jobs to restore the correct state, ensuring your infrastructure operates predictably and reliably.

With Spacelift, you get:

• Policies to control what kind of resources engineers can create, what parameters they can have, how many approvals you need for a run, what kind of task you execute, what happens when a pull request is open, and where to send your notifications
• Stack dependencies to build multi-infrastructure automation workflows with dependencies, having the ability to build a workflow that, for example, generates your EC2 instances using Terraform and combines it with Ansible to configure them
• Self-service infrastructure via Blueprints, or Spacelift’s Kubernetes operator, enabling your developers to do what matters – developing application code while not sacrificing control
• Creature comforts such as contexts (reusable containers for your environment variables, files, and hooks), and the ability to run arbitrary code
• Drift detection and optional remediation

If you want to learn more about Spacelift, create a free account today or book a demo with one of our engineers.

GitOps is the process of configuring apps and infrastructure using files in Git repositories. CI/CD pipelines and IaC tools then consume your repository’s content to automatically provision your resources.

GitOps makes software delivery manageable at scale, but there’s more to its implementation than simply pushing IaC configs up to GitHub. The best practices we’ve discussed in this article allow you to fully realize the benefits of GitOps through pull-based deployments, automated drift detection, and policy-as-code. 

Yet it’s also important to ensure GitOps is truly the sole source of truth for your environments—developers and operators shouldn’t be able to bypass GitOps processes, as this can easily result in conflicts and misconfigurations.

Ready to start implementing GitOps for your team’s projects? Check out Spacelift to provision, configure, and govern your IaC resources using one automated workflow. Our platform’s compatible with the top IaC tools and cloud providers, letting you seamlessly apply IaC changes as you commit to your repositories. We’ve also got plenty more GitOps content here on the Spacelift blog—try reading our summary of the top GitOps tools, or learn how to get started using Terraform in a GitOps workflow.