[Virtual Event] Unifying infra and app promotions with Spacelift, OpenTofu, and Kargo

Sign up ➡️

Kubernetes

Kubernetes GitOps: How It Works, Tools, and Best Practices

kubernetes gitops

In modern DevOps practices, manually deploying applications to Kubernetes is no longer scalable or safe. Teams need reliable, automated deployment strategies that work seamlessly across environments. 

That’s where GitOps steps in. GitOps brings infrastructure and deployment definitions under version control and integrates tightly with Kubernetes to make continuous delivery pipelines cleaner and easier to manage.

Kubernetes runs modern apps across many machines. It handles containers with intelligent scaling and recovery. It works well for teams running many services daily. But without proper tooling, Kubernetes gets hard to manage. That is where GitOps makes setup and rollout much easier.

GitOps adds safety and visibility to Kubernetes cluster management. Rolling back means reverting a commit rather than reconstructing what production used to look like. Using pull requests to initiate deployments also means code reviews and policy checks are enforced early. By using Git as a single source of truth, you reduce human error and boost deployment reliability. However, implementing GitOps in Kubernetes has its challenges. 

In this post, we’ll cover the core principles and design of GitOps pipelines through the following topics:

You will know how to build clean and secure GitOps pipelines for Kubernetes by the end.

What is GitOps?

GitOps is a modern approach to continuous delivery that uses Git as the single source of truth. It manages infrastructure and application deployment through declarative files stored in version control. 

GitOps tools continuously watch Git for changes and sync them to Kubernetes. This ensures that the cluster always reflects the desired state defined in Git. All changes go through Git workflows, including pull requests and reviews. This brings automation, control, and traceability to Kubernetes operations.

gitops workflow

GitOps treats operations like software development using standard Git practices. Every change must be committed, reviewed, and merged before deployment. Reconciler tools then apply these changes automatically to the cluster. 

This makes deployments repeatable and easy to audit. Teams gain better control over environments without needing to log into clusters manually. GitOps also aligns with DevSecOps by integrating security and compliance checks early in the pipeline.

Read also: 15 GitOps Best Practices to Improve Your Workflows

Benefits of GitOps

Benefits of implementing GitOps include:

  • Version-controlled deployments: Every change to infrastructure or apps is recorded in Git history, allowing teams to trace, audit, and roll back deployments easily.
  • Improved developer experience: Developers use the same Git workflow for code and operations. They don’t need direct access to Kubernetes, which reduces risk.
  • Stronger security and compliance: All changes are reviewed and approved through Git. This ensures that policy enforcement and access control happen before deployment.
  • Automated and consistent delivery: GitOps tools constantly monitor and apply changes from Git. This ensures environments stay consistent and reduces configuration drift between desired and actual states.

How GitOps works with Kubernetes

Kubernetes is declarative by design. Managing its configurations across environments is very challenging. Manual deployments often lead to human errors and sudden downtime with inconsistent cluster states. Rolling back broken changes is rarely straightforward and may leave clusters in a partial state. 

Security is another concern. 

Every manual interaction increases the surface for misconfigurations. Maintaining reliability and auditability becomes harder as teams grow without a structured approach.

Many teams also struggle with visibility and coordination. YAML sprawl and untracked configuration changes make debugging and reproducing environments complex. CI/CD pipelines lack a consistent way to track infrastructure versions alongside app changes. 

There’s often no clear audit trail of who changed what or why. Enforcing policies manually is slow and error-prone. Without guardrails, developers may unknowingly break production. 

All of this makes scaling Kubernetes safely a challenge for most organizations.

Key reasons to adopt GitOps with Kubernetes

GitOps improves Kubernetes operations by making infrastructure and application delivery declarative, version-controlled, and automated through Git:

  • Consistent and simplified configuration management: Git stores all configuration in one place using declarative files. This eliminates guesswork and centralizes environment control.
  • Instant rollbacks and full version history: Every change is logged and tracked in Git history. Reverting a faulty deployment is as simple as reverting a commit.
  • Higher deployment reliability and control: GitOps enforces approvals and automated sync with Kubernetes, making direct access unnecessary and reducing mistakes.
  • Built-in security and policy enforcement: Policies can be applied through pull request checks. This ensures compliance and reduces security risks before anything reaches the cluster.
  • Team scalability and better collaboration: Git enables collaborative workflows with reviews and approvals. Everyone stays on the same page without interfering with live systems.
  • Automation that reduces errors and manual work: GitOps agents watch for changes and reconcile automatically. This means fewer manual actions and faster, safer deployments across environments.

Managing Kubernetes deployments using GitOps

Let’s explore the essential parts of a GitOps pipeline. A complete GitOps deployment setup typically involves four key components: 

  1. Git repository
  2. Kubernetes cluster
  3. Continuous reconciliation agent 
  4. Optional CI pipeline integrations

The Git repo holds all deployment manifests and configuration files. Kubernetes is the deployment target. The GitOps agent watches Git and syncs state into Kubernetes. CI pipelines may build and push container images and update tags in Git.

1. Standard GitOps setup with environment-mapped branches

The diagram below shows where most teams start. Two Git branches, dev and main, mapped to the development and production clusters. It is easy to reason about, and it is the fastest way to get a first cluster under GitOps control. It also has a shelf life. Read on to the promotion section for where it breaks and what to move to.

diagram showing standard GitOps Kubernetes setup with two Git branches, dev and main, mapped to the development and production clusters
  • The dev branch syncs with the Dev Kubernetes cluster, allowing continuous testing and experimentation. 
  • The main branch feeds into the Prod cluster, ensuring only validated and approved changes reach production. 

This structure gives teams visibility, control, and repeatability for every deployment step.

Once the GitOps foundation is in place, developers can independently test changes using feature branches. Using the diagram below, let’s understand how developers can merge their feature branch into the dev branch to deploy and test changes in the Dev environment.

2. Developer workflow for feature branch testing in dev

As shown below, a developer creates a feature branch from dev, makes the necessary updates (e.g., modifying deployment YAMLs, updating container versions), and submits a pull request. Upon merge, the updated dev branch reflects the new desired state. 

diagram showing GitOps Kubernetes setup feature branch testing in dev

A GitOps controller like Argo CD or Flux picks the change up on its next reconciliation. Argo CD polls Git every three minutes by default, calculated as 120 seconds plus up to 60 seconds of jitter, so a merge reaches Dev within one cycle. Configure a webhook and it lands in seconds. 

Flux works the other way around: you set an explicit interval on each source and Kustomization, so the cadence is yours to choose. Once the sync completes, the Dev cluster is running the new version and your tests can run against it.

This approach streamlines testing and encourages frequent, low-risk changes. Since the Prod environment remains untouched, teams can catch misconfigurations or runtime issues early. 

It also enables multiple developers to test parallel features in the dev cluster before consolidation and promotion.

What happens after testing in dev? Once the changes are validated in dev, the next step is to move them toward production. But how do teams decide when and how to promote these changes safely? That’s where promotion workflows come into play.

3. Promotion workflow from dev to prod via GitOps

This diagram illustrates the promotion workflow in a GitOps pipeline.

diagram showing GitOps Kubernetes setup Promotion workflow

 On the left is a Git repository with two branches. 

  • The main branch corresponds to the production environment. 
  • The dev branch corresponds to the development environment, which contains a Kubernetes cluster with three pods, indicating a deployed and tested running application. 
  • The prod environment also has a Kubernetes cluster with three pods, representing the production deployment that will be updated. 

The diagram shows the dev branch merging into the main branch. It is depicted by the line connecting the two branches with a merge point.

This merge represents promoting changes from the dev environment to the prod environment. In a GitOps workflow, after changes are tested in the dev environment, they are merged into the main branch via a pull request. 

Once the merge occurs, the GitOps tool detects the updated state in the main branch and synchronizes the Prod cluster to match the new configuration in production.

After the changes have been validated in the dev environment, the developer creates a pull request to merge the dev branch into the main branch. Once the pull request is approved and merged, the GitOps tool applies the changes to the Prod cluster. 

Thus, the changes tested in the dev environment (via the dev branch) are merged into the main branch. It triggers the GitOps tool to deploy those changes to the prod environment.

Using GitOps with environment-specific branches also simplifies rollback. A simple git revert restores a previous commit, and the GitOps tool automatically syncs the cluster back to that exact state.

When environment branches stop scaling

The branch-per-environment model works well with two environments. Adding a third is where it starts to strain.

With dev, staging, and production each mapped to its own branch, changes no longer travel in a straight line. A fix that needs to reach production ahead of everything else sitting in staging has to be cherry-picked, and every cherry-pick makes the branches diverge a little further. 

Over time, the diff between staging and production stops describing a release and starts describing the accumulated difference between two branches. Nothing fails loudly. You lose a reliable answer to the question of what is deployed where.

Two adjustments address this.

First, model environments as directories rather than branches. Keep a shared base of manifests and give each environment its own Kustomize overlay or Helm values file, as covered in the section on managing environment-specific configurations below. A change to the staging overlay cannot reach production, because production reads a different path.

Second, separate promotion from deployment. A GitOps controller reconciles one environment against one path in Git. Moving a verified change from one environment to the next is a different job, and it is the part most teams end up scripting in CI. 

Argo CD’s Source Hydrator renders manifests into Git, so the real state of each environment is visible and reviewable in a pull request. Kargo, built by the Argo CD maintainers, models environments as stages and promotes a change into the next stage only once the previous stage passes its verification checks.

Both approaches keep Git as the source of truth. The difference is that promotion becomes an explicit, auditable step rather than a merge that happens to trigger a sync.

Read more: GitOps at Scale: Strategies for Enterprise Adoption & Growth

4. Safe promotion and rollback in a GitOps workflow

This final diagram showcases a complete GitOps pipeline with built-in promotion and rollback mechanisms. It highlights two critical flows: one for safely promoting changes to production, and another for quickly rolling them back if needed.

GitOps Kubernetes setup rollback

The forward flow shows how updates move from the dev branch to the main branch via a pull request, triggering the GitOps controller to sync changes into the prod cluster. This ensures the cluster reflects the new configuration state approved through Git.

Once changes are validated in the dev environment, merging into main acts as a green light for production deployment. The GitOps controller continuously monitors the main branch and applies the updated manifests to the prod Kubernetes cluster. 

Git becomes the single source of truth. Wth every pull request, the team knows exactly what’s running in production. This flow enforces control and improves traceability by tying each deployment to a specific Git commit.

But if something breaks in production, the rollback flow kicks in. Instead of manual intervention, you revert the commit that broke production. The revert is itself a commit, so the audit trail stays intact and nothing is rewritten. The controller picks it up on its next sync and restores the previous state. This ensures quick recovery while maintaining a clear audit trail.

Best practices for setting up GitOps Kubernetes workflows

To set up GitOps with Kubernetes effectively, ensure your setup includes secure, automated synchronization between Git repositories and cluster state. Key practices include:

  • Keep your config repo separate from your application source repo. Argo CD’s own best-practices guide recommends this: you get a clean audit log, you can change manifests without triggering an application build, and you can grant commit access to application code without granting it to production config. Within that config repo, model environments as directories with Kustomize overlays or Helm values files rather than as long-lived branches.
  • Store only declarative configuration in Git, such as YAML for Kubernetes resources and Helm or Kustomize files. Avoid including secrets directly.
  • Automate synchronization with a GitOps operator like Argo CD or Flux, which continuously reconciles cluster state with Git and provides audit trails.
  • Integrate secrets management using external tools like HashiCorp Vault, Sealed Secrets, or SOPS to safely manage sensitive data in GitOps workflows.
  • Enable role-based access control (RBAC) and use signed commits or admission policies to ensure only trusted changes are applied.
  • Test all manifests and configurations in CI pipelines before merging to main branches to prevent misconfigurations from reaching the cluster.

Managing cluster resources, RBAC, and secrets with GitOps

Once teams adopt GitOps for application deployments, the next logical step is to extend it to manage all Kubernetes resources. 

Declarative management of Kubernetes resources

GitOps can declaratively control PersistentVolumeClaims, ConfigMaps, Secrets, Ingress objects, and HorizontalPodAutoscalers, all through Git. 

For instance, PVCs for backing storage, ConfigMaps for environment variables, and Ingress resources for routing are stored as YAML files and versioned in the repo. 

When a ConfigMap is updated to switch a feature toggle or an HPA threshold is tuned to improve responsiveness under load, those changes are made via pull requests. The GitOps controller then automatically applies the updated manifest to the correct cluster, ensuring the environment matches what’s in Git.

Cluster-level operational concerns

Beyond workloads and configuration, GitOps is highly effective for handling cluster-level operational concerns. This includes managing ResourceQuotas to prevent namespace overuse, RBAC policies to define fine-grained access control, and NetworkPolicies to enforce traffic restrictions between pods. 

For example, a team might restrict staging traffic from accessing production services by defining a NetworkPolicy in Git. Or they might onboard a new developer by assigning a specific ClusterRoleBinding through a merge. 

Instead of granting permissions manually with kubectl, changes are proposed, reviewed, and committed in Git and then automatically enforced by the controller. This provides audit trails, peer reviews, and reproducibility for even the most sensitive operational tasks.

Managing environment-specific configurations

Using GitOps patterns, managing environment-specific configurations becomes far more manageable. One widely adopted approach is maintaining a shared base directory with reusable Kubernetes manifests and then applying Kustomize overlays or Helm value files per environment. 

For example, staging might use a different replica count or log level than production or point to separate external service endpoints. These overlays live in environment-specific folders. GitOps tools reconcile each environment’s folder to its respective cluster. 

This structure allows developers to validate changes in dev without touching production configurations, while platform engineers retain centralized control over shared templates and policies.

Kubernetes GitOps tools compared

GitOps tools for Kubernetes automate infrastructure and application delivery by syncing the desired state from a Git repository to the cluster. Core tools include Argo CD and Flux, but platforms like Spacelift also support GitOps workflows for Kubernetes via policy-driven automation and integration with Git-based workflows.

Popular GitOps Kubernetes tools include:

  1. Argo CD: A CNCF graduated project and the default choice for most teams. It runs a control plane with a web UI, watches your config repo, and reconciles clusters to match it. ApplicationSets generate Applications across a fleet from a single template, and the Source Hydrator renders manifests into Git so your diffs show the resources that will actually apply. Best for multi-cluster fleets and developer self-service. The tradeoff is a centralized component you have to run and scale.
  2. Flux: Also CNCF graduated, and built as a set of Kubernetes controllers rather than a central server. Version 2.9 added server-side apply field ignore rules, so you can hand ownership of specific fields, such as a Deployment’s replicas, to another controller instead of fighting it. Best for lightweight, multi-tenant, and edge deployments. The tradeoff is that the UI lives in a separate ecosystem project, Flux Operator.
  3. Spacelift: Spacelift runs Kubernetes deployments through kubectl, with Kustomize as the native path, so anything you can apply with kubectl runs in a Spacelift stack. Pull requests show a plan of what will change before it reaches the cluster, and policy as code gates every run. Helm works through helm template in a before-plan hook. Best for teams who want Kubernetes governed in the same workflow as their Terraform, OpenTofu, and Ansible. Not a replacement for an in-cluster reconciler if continuous drift correction inside the cluster is what you need.
  4. Kargo: Argo CD reconciles one environment. Kargo moves a verified artifact between environments, modelling dev, staging, and production as stages where each stage’s confirmed state becomes the input to the next. Built by the creators of Argo CD. Reach for it when your promotion logic has outgrown a merge and a pile of CI scripts.
  5. Weave GitOps (no longer a live option): You will still find Weave GitOps on GitOps tool lists. Skip it. Weaveworks shut down in February 2024, and the repository now describes itself as transitioning to a community-driven project, with its install instructions pointing at a release candidate rather than a stable release. If you want a UI on top of Flux, use the Flux Web UI that ships with Flux Operator instead.

Read more: Top 8 GitOps Tools You Should Know

How to improve your Kubernetes GitOps workflows with Spacelift

Spacelift is an IaC management platform that uses GitOps to automate CI/CD for your infrastructure components. It supports Kubernetes, OpenTofu, Terraform, Terragrunt, CloudFormation, Pulumi, and Ansible.

The power of Spacelift lies in its fully automated hands-on approach. Once you’ve created a Spacelift stack for your project, changes to the IaC files in your repository will automatically be applied to your infrastructure. 

Spacelift’s pull request integrations keep everyone informed of what will change by displaying which resources are going to be affected by new merges. Spacelift also allows you to enforce policies and automated compliance checks that prevent dangerous oversights from occurring.

gitops best practices example

Spacelift includes drift detection capabilities that periodically check your infrastructure for discrepancies compared to your repository’s state. It can then launch reconciliation jobs to restore the correct state, ensuring your infrastructure operates predictably and reliably.

With Spacelift, you get:

  • Policies to control what kind of resources engineers can create, what parameters they can have, how many approvals you need for a run, what kind of task you execute, what happens when a pull request is open, and where to send your notifications
  • Stack dependencies to build multi-infrastructure automation workflows with dependencies, having the ability to build a workflow that, for example, generates your EC2 instances using Terraform and combines it with Ansible to configure them
  • Self-service infrastructure via Blueprints, enabling your developers to do what matters – developing application code while not sacrificing control
  • Creature comforts such as contexts (reusable containers for your environment variables, files, and hooks), and the ability to run arbitrary code
  • Drift detection and optional remediation

If you want to learn more about Spacelift, create a free account today or book a demo with one of our engineers.

Key points

GitOps gives you control over Kubernetes deployments without manually touching the cluster. You define everything in Git, making deployments predictable and reversible. GitOps also improves the visibility and audibility of your entire infrastructure. 

We learned how GitOps works using tools like Argo CD and Flux. We also saw how pipelines connect to Git to manage environments and structure repositories. 

  • Use the diagrams and patterns in this blog to build your GitOps pipelines. 
  • Start with a single service and apply changes through pull requests. 
  • Organize your Git repositories cleanly. Use CI to build and GitOps to deploy. 
  • Slowly scale the approach across all your Kubernetes environments. 

GitOps helps your teams move faster with fewer mistakes. You spend less time debugging environments or chasing configuration drift. Let Git manage your Kubernetes deployments so you can focus on building features. 

The best GitOps tool

Spacelift is an IaC management platform that uses GitOps to automate CI/CD for your infrastructure components. It works with OpenTofu, Terraform, Terragrunt, CloudFormation, Pulumi, Kubernetes, and Ansible and supports self-hosted on-prem workers, workflow customization, drift detection, policies, and more.

Learn more