What is DevOps Security? Challenges & Best Practices

Security has never been more important. This is equally true for end-user applications and the systems, tools, and processes involved in building and delivering these applications to those end users.

Security in the context of application development (i.e., writing the source code of your applications) is commonly referred to as application security or AppSec. Security related to the systems, tools, and processes involved in building and delivering applications can be referred to as DevOps security.

This blog post will focus on important DevOps security best practices to help your organization build and deliver software and systems to your end users in a secure way.

1. What is DevOps security?
2. DevOps security best practices
3. DevOps security challenges

DevOps involves many aspects of the software development lifecycle (SDLC). You should consider the security aspects of your DevOps environment carefully.

DevOps security is the practice of integrating security measures into the DevOps workflow, ensuring code, infrastructure, and deployments are protected. It leverages automated tools for tasks like vulnerability scanning, access control, and compliance checks to safeguard systems without disrupting the development pipeline.

The central concept of most DevOps environments is a continuous integration and continuous delivery (CI/CD) pipeline. A CI/CD pipeline does not necessarily need to be a single pipeline in a tool such as GitHub Actions or Azure DevOps; it can involve separate interconnected systems and processes. Think of the CI/CD pipeline as a concept that automates the process of building and delivering applications and systems to end users.

Securing the CI/CD pipeline and the systems it integrates with is the main focus of DevOps security. A few of the things that are involved in this work include:

• Handling application and system secrets in a secure and responsible manner
• Tracking and continuously updating application dependencies and scanning for vulnerabilities
• Implementing access control for the CI/CD pipeline to limit who can make modifications to it and who is allowed to trigger the pipeline
• Monitoring and alerting on suspicious activities in your DevOps environment
• Educating everyone involved in the SDLC about security

It is difficult to draw a line between DevOps security and AppSec, which have many overlapping concerns. Writing secure application code is primarily an AppSec responsibility, so this will not be covered in this blog post.

The rest of this blog post will focus primarily on the security of the CI/CD pipeline and its related systems and processes.

The integration of development and operations workflows demands a proactive approach to embedding security, ensuring that resilience is not sacrificed for innovation. Let’s dive into DevOps security best practices.

1. Turn DevOps into DevSecOps

Security should be on everyone’s mind. It is only as strong as the weakest link. A security breach can be devastating to your organization. In the worst case, it can lead to lost sensitive data, lost customers, and irreparable damage to your organization’s reputation.

Historically, security operations have functioned as a silo in the organization. The security operations team has been seen as a police-like function in the organization, with a running joke that their job is to say no to all application developer requests.

This aligns with the way developers and operators existed separately in the organization before the DevOps movement. Like the DevOps movement, a new movement aims to include security operations as a first-class citizen in the DevOps workflow, leading to the DevSecOps movement.

Working with a DevSecOps mindset means recognizing the importance of security and making security a part of everything you do. This includes adopting security-related tooling and processes such as threat modeling and penetration testing.

Read more in our article: DevSecOps: Making Security Central To Your DevOps Pipeline

2. Shift security left

Shifting security left is the process of introducing security-related practices and processes earlier in the SDLC. Instead of bolting security on at the end, security should be built into everything you do from the start. This includes the applications you build, as well as the systems and processes involved in the DevOps environment.

For the CI/CD pipeline, this means introducing security scanning and testing early on. In general, all security-related matters should be prioritized. Obviously, if a security vulnerability in your application is detected early in the pipeline, it is pointless and possibly even insecure to run the pipeline to completion.

Shifting security left also means you should conduct a threat modeling exercise before you write a single line of code for an application or integrate your CI/CD pipeline with a new third-party system, and consider whether this change introduces new security weaknesses or vulnerabilities in your environment.

3. Security automation

Security operations often feel like a burden on your already swamped development and operations teams. When prioritizing between feature development and security fixes, feature development tends to look more attractive and be more fun.

As much security as possible should be automated. If a security scanning or test tool can be run in a CI/CD pipeline, it should be added to the pipeline. Helping your development teams out with security is usually appreciated and leads to higher quality and a more secure environment.

If your organization has a central platform team, one of its tasks should be to build reusable CI/CD pipeline templates for other teams. These templates should include standard security tasks that you want each team to use. Security is everyone’s responsibility, but that does not mean everyone should have to reinvent the wheel.

Manual testing pitfalls

Manual testing— and manual security-related testing, in particular — tends not to be performed or maintained. Relying on manual testing alone will not increase confidence in your environment’s security. 

Investigate whether your current manual testing processes can be automated with modern tools and included as a step in your CI/CD pipeline. Manual security testing can be used as a complementary exercise when appropriate.

What are golden images?

Another key element that can be automated is building golden images for virtual machines and containers your application teams can use as a base image for their own applications. This allows you to have a known-good state for all of your applications. 

The central platform team should be responsible for keeping these golden images up to date. However, the image scanning step is still highly relevant in your pipelines because each team will extend the base images with their applications and dependencies.

4. Implement access control and a least-privilege mindset

The CI/CD pipeline can single-handedly move a change from an initial push to your version control system all the way to a running application in production. Therefore, it is of utmost importance to restrict who can trigger or make changes to your CI/CD pipeline.

All CI/CD systems include some form of access control and permission system. This should be configured according to your needs. Minimize the number of people with permission to modify or trigger CI/CD pipelines that interact with sensitive production systems. Ideally, only automated processes with built-in guardrails should have these permissions.

Implementing GitOps

GitOps is a common approach for deploying applications on Kubernetes. With GitOps, you can avoid giving your CI/CD pipeline direct access to the Kubernetes cluster where your application runs. Instead, you give the cluster access to your source code (i.e. Kubernetes manifests). This can make your environment far more secure, as long as you control who can push changes to the version control system that the GitOps operator monitors.

You should include practices such as requiring at least two people to be involved in moving a change from commit to production. This can be as simple as requiring two people to be involved in code review before a pull request can be merged into the main branch. For highly sensitive and regulated systems, you could even require more than two people to be involved.

The CI/CD pipeline itself will run and integrate with external systems using one or more machine identities. The permissions of these identities should be restricted and scoped to exactly what is needed for the pipeline to perform its job. If the pipeline machine identities are compromised, you need to contain the damage.

Apart from the CI/CD pipeline itself, you must also consider access control and least privileged access in all related systems. It is surprisingly common for organizations to have strict controls around their CI/CD systems and version control systems yet allow everyone to be administrators in their cloud environments. 

Having administrator access to a cloud environment means you can bypass the entire CI/CD pipeline and deploy changes directly to production, which presents obvious risks. 

5. Secure the CI/CD pipeline

Securing the CI/CD pipeline starts with securing access to your version control system. This is where your source code is stored, and anyone who can commit code to the version control system can often set the CI/CD pipelines in motion.

Developers will often need read access to the CI/CD pipeline environment. This is because things can go wrong, and your developers need to be able to troubleshoot the CI/CD environment.  In general, read access is harmless. However, make sure you understand what your developers can read using this access, as your CI/CD environment could contain more than just pipelines (e.g., secrets).

The continuous integration (CI) part of the CI/CD pipeline often produces some form of build artifact. This could be a Docker image, a collection of Kubernetes manifests, or something similar. You should also secure access to the locations where these artifacts are stored.

Many CI/CD pipelines involve custom runners or build machines. These are virtual machines or containers you use to perform some CI/CD pipeline tasks, which could include building an application binary or Docker image, scanning source code for vulnerabilities, or uploading HTML and JavaScript files to an external location. 

These machines often have direct access to your source code and various external systems, so you should make sure they are patched and secured, similar to how you secure your application compute resources. 

Because these machines are often placed in a network partition with access to other parts of your internal networks, you should also strengthen the network security around them.

6. Secure state file

Infrastructure as code (IaC) comes with additional security considerations.

It is often a big part of the DevOps domain. This includes source code management for the IaC source files and dedicated CI/CD pipelines for infrastructure deployments. Many of the other security best practices also cover IaC security.

For Terraform, OpenTofu, Pulumi (and possibly others), you must take care in securing your state file. The state file often contains sensitive information and secrets for your infrastructure. It is also a map of your organization’s cloud infrastructure, which can be helpful for planning an attack targeting your organization.

Securing the state file starts with storing it in a secure location. This location should be guarded by identity and access management, just as you should protect every other system involved in the CI/CD pipeline. 

In general, only the process running the IaC tool should have access to read and write to the state file. The state storage location should support encrypting the state file at rest. Reads and writes of the state file should be encrypted in transit using TLS.

Even if the state file is encrypted at rest in a storage backend such as AWS S3, anyone with read access to the S3 bucket can read its contents. 

If a developer has read access to the state file location, the state file can easily end up on local developer laptops. To take security one step further, you can use features such as the state and plan encryption available in OpenTofu.

If you use Kubernetes with GitOps you must secure access to the version control system the GitOps operator is monitoring, as well as securing the Kubernetes clusters themselves. Securing Kubernetes clusters is a big topic, worthy of a blog post of its own.

7. Secrets management

Secrets management is a key element of both application development and the DevOps CI/CD pipeline. Different steps of the pipeline interact with different external systems, each requiring some form of credentials, which must be protected. 

Most CI/CD systems include a system for secret management. If these integrated secrets management systems do not provide enough features for your use cases, you could use a third-party secrets management system instead (e.g., HashiCorp Vault).

Secrets management should include preferably automated processes for rotating secret values at a schedule. If possible, secrets should be dynamic, single-use, and restricted in time.

You should avoid using a single “master identity” in CI/CD pipelines with broad access to all the external systems it interacts with. Instead, use multiple identities (or credentials), one for each explicit purpose. For example, if you run Terraform or OpenTofu, you could use a separate identity with read access to the state file to run plan operations and a different identity with read/write access to the state file for apply operations.

Many CI/CD systems and the external systems they integrate with support the concept of workload identity federation. This involves setting up a trust relationship between the two systems (i.e., your CI/CD pipeline and the external system you want to integrate with). 

Once this trust relationship has been established, you can interact with the external system using no explicit credentials. This removes the credentials from the equation, greatly improving the overall security of your environment. The recommendation is to use workload identity federation wherever it is supported.

8. Logging and monitoring for security

If your CI/CD pipeline and related systems support audit logging, you should make sure this feature is enabled and that the logs are stored securely in a separate location. 

Audit logs can tell you:

• What has happened in your environment
• Who performed a certain action
• When the action was performed

Audit logs also include metadata such as the IP from which an action was taken, other details related to the action, and the identity that took the action.

Audit logs should be stored securely, and you should restrict who has access to these logs. In general, audit logs are stored in the same system as the actions that are audited. You should also stream the logs to a separate location.

Providing write access to the audit logs introduces the risk that someone can tamper with them to conceal traces of actions that have been performed in your environment. If your system offers a way of validating the logs’ integrity, you should to use this.

An example of a system that has this capability is AWS CloudTrail, which is the audit log for everything that happens in your AWS environment.

Which audit logging alerts to set?

You should create alerts that are triggered when certain high-privileged tasks are performed outside of known patterns. This includes:

• Updates to your CI/CD pipelines
• Reads and writes of secret values
• Modifications to permissions for the machine identities of your CI/CD pipelines
• Tampering with the audit logs
• Self-approvals of pull-requests
• Manual triggers of pipelines that should not be manually triggered
• And more …

You need to decide the exact types of alerts that make sense for your organization. Start small with a set of alerts for the most serious events, and slowly build up your monitoring capabilities.

9. Use policies for compliance

Policies used in policy as code (PaC) can provide an extra layer of protection in your environment. PaC can limit thet changes that can be made to your infrastructure and CI/CD environment.

Learn about how your CI/CD platform can use PaC and write policies to restrict certain behaviors that you want to protect your environment from. 

A popular tool for PaC is Open Policy Agent (OPA), which is integrated into many platforms (e.g., Spacelift). Even if your platform does not have native support for PaC, you can still use OPA in your environment if you put in the work.

Examples of policies you can use in your environment are:

• Deny manual updates to CI/CD pipelines
• Deny switching machine identities for a given CI/CD pipeline
• Require a container scanning step for each CI/CD pipeline that is created
• Deny a given CI/CD pipeline from deploying changes to production on a weekend or outside of business hours
• Deny old versions of application dependencies or completely deny certain dependencies from being used

10. Use AI for security

AI-powered tools such as GitHub Dependabot and Copilot can help your developers write secure code and your platform teams stay on top of vulnerabilities.

Investigate whether your CI/CD platform offers any AI security capabilities and evaluate whether they will help make your environment more secure.

11. Educate and train your teams on security

The preceding list of best practices has focused primarily on the CI/CD platform and other technical pieces of your environment. However, it is supremely important to continuously educate and train your organization on security best practices. Security is a moving target, with constant updates to be aware of.

Security champions program

You can establish a security champions program in which each development team in your organization is represented. The forum should focus on sharing knowledge on security, with a focus on the challenges and solutions specific to your organization. You can also highlight important developments in the security space in this forum. 

Gathering security-minded people together in a forum provides opportunities to hear many different perspectives and ideas on security you might otherwise miss. 

DevOps security challenges often arise due to the integration of development and operations processes. These challenges typically stem from gaps in oversight or misconfigurations, which can unintentionally introduce vulnerabilities. Here are some common challenges: 

1. Balancing speed with security practices

The DevOps movement is about speeding up the delivery of value to your end users through efficient CI/CD pipelines and processes. A challenge is balancing this speed with security practices. Security doesn’t have to slow down the rest of your DevOps processes, but it requires a commitment to automation. 

Automating as many security practices as possible will allow you to retain speed while remaining secure.

2. Monitoring security practices

Once you have established security practices in your organization, a major challenge is to monitor and ensure that these practices and processes are followed and work as intended. If security operations feel like a burden or an obstacle, developers are likely to find creative ways to circumvent them.

3. Managing vulnerabilities

A major challenge with DevOps security is staying on top of vulnerabilities. New vulnerabilities are discovered daily. Most present a risk only in very specific circumstances. However, critical vulnerabilities are occasionally discovered. 

A system that tracks which application dependencies you are using across your organization will alert you to any newly discovered security vulnerability.

4. Maintaining a security culture

Keeping your developers up to date in the security space is a big challenge. How you go about building a strong security culture will vary depending on your initial state and your developers’ security awareness. 

A few ideas on how to get started with building a strong security culture include:

• Holding inspirational presentations around security and secure development practices
• Holding workshops for platform and development teams
• Starting a security champions program in your organization
• Having an outlet for sharing security-related news throughout your organization

5. Neglecting security awareness

Keeping security on everyone’s mind means that the whole organization — not just your DevOps teams — should be involved. 

A leaked password to one of your employees could be the entrypoint that one hacker needs to reach even the most protected part of your environment. When you educate your organization on security practices, make sure everyone is involved in some way.

A platform like Spacelift can help you and your organization fully manage cloud resources within minutes. Spacelift is an infrastructure management platform that supports tools like OpenTofu, Terraform, Ansible, Pulumi, Kubernetes, and more. 

Security is one of Spacelift’s biggest priorities, so state-of-the-art security solutions, such as policy as code, encryption, Single Sign-On (SSO), MFA, and private worker pools, are embedded inside the product.

The power of Spacelift lies in its fully automated hands-on approach. Once you’ve created a Spacelift stack for your project, changes to the IaC files in your repository will automatically be applied to your infrastructure. 

Spacelift’s pull request integrations keep everyone informed of what will change by displaying which resources are going to be affected by new merges. Spacelift also allows you to enforce policies and automated compliance checks that prevent dangerous oversights from occurring.

Spacelift includes drift detection capabilities that periodically check your infrastructure for discrepancies compared to your repository’s state. It can then launch reconciliation jobs to restore the correct state, ensuring your infrastructure operates predictably and reliably.

With Spacelift, you get:

• Policies to control what kind of resources engineers can create, what parameters they can have, how many approvals you need for a run, what kind of task you execute, what happens when a pull request is open, and where to send your notifications
• Stack dependencies to build multi-infrastructure automation workflows with dependencies, having the ability to build a workflow that, for example, generates your EC2 instances using Terraform and combines it with Ansible to configure them
• Self-service infrastructure via Blueprints, or Spacelift’s Kubernetes operator, enabling your developers to do what matters – developing application code while not sacrificing control
• Creature comforts such as contexts (reusable containers for your environment variables, files, and hooks), and the ability to run arbitrary code
• Drift detection and optional remediation

If you want to learn more about Spacelift, create a free account today or book a demo with one of our engineers.

This blog post has covered many DevOps security best practices to help your organization build a secure environment. The focus has been on the DevOps CI/CD pipeline and its integrations with external systems.

Security is a shared responsibility that extends beyond security and development teams, and your defenses are only as strong as the weakest link. This requires a DevSecOps approach that integrates security into daily workflows and emphasizes shifting security left in the development lifecycle. 

Key practices include shifting security left in the SDLC, automating operations, securing CI/CD pipelines with least-privilege access, managing secrets effectively, and using tools like audit logs and policy as code to enforce safeguards and protect against security threats.