In today’s fast-paced software industry, organizations are increasingly relying on the Continuous Integration/Continuous Delivery (CI/CD) pipelines to enhance efficiency, speed, and reliability in software development. However, the rapid adoption of CI/CD has resulted in a significant increase in security threats.
In this article, we will cover:
CI/CD security refers to the practice of integrating security into the CI/CD pipeline to identify and remediate any risks or vulnerabilities throughout all the stages of the CI/CD process. To protect your pipeline against potential threats, it is essential to understand the top security risks associated with CI/CD pipelines.
Why is CI/CD security crucial?
The CI/CD pipeline is the backbone of software development organizations, enabling them to build, test, and deliver any software faster, more frequently, and more reliably.
But what happens when these pipelines break?
This can result in significant chaos, hampered collaboration, and delays in delivering new code to end-users, leading to a loss of revenue, reputation, and even legal consequences.
If this gives you goosebumps, there is something even scarier.
What happens if these pipelines are compromised by attackers? Worse yet, what if this occurs and the security incident is not detected for a long time or at all?” 😲
Consider the SolarWinds CI/CD security incident, where attackers targeted the pipeline, affecting more than 18,000 organizations and gaining access to their networks, systems, and data undetected for over 14 months. This attack alone underscores the critical need for CI/CD security, and it’s not the only one. The StackOverflow incident, where vulnerabilities in the build systems led to stolen source code and leaked secrets, the php backdoor attack, and the CodeCov exploit all serve as examples of the devastating effects of CI/CD security breaches.
The significance of protecting the CI/CD process cannot be overstated, and it has shifted the focus of security teams in recent years.
In this section, we will discuss the top 10 CI/CD security risks and how they can be exploited.
CICD-SEC-1: Insufficient flow control mechanisms
The first CI/CD security risk on the list refers to the ability of an attacker who has gained access to any of the systems in the CI/CD process (SCM, CI, or runners) to push malicious code down the pipeline due to a lack of control mechanisms that enforce additional approval or review.
Why is this bad?
CI/CD pipelines are a direct bridge from the developers’ machine to the production environment. In the absence of flow control like reviews or approvals, a single entity (human or machine) can push malicious code to production and publish packages with malicious code.
This means that once the system is compromised, there are no constraints to the limit of the damage that the malicious actors can do.
How was this exploited?
CICD-SEC-2: Inadequate identity and access management
This refers to the lack of proper identity and access management in CI/CD ecosystems, given the vast number of identities to manage and the complexity of interconnected systems involved, such as SCM and CI, which often have different authentication and authorization methods.
In many cases, user/machine accounts do not follow the principle of least privilege and have overly permissive default permissions.
Why is this bad?
The consequences of inadequate identity and access management in CI/CD processes are severe. The existence of multiple identities paired with poor identity and access control creates an opportunity for hackers, where compromising a single account could lead to them having control over the systems involved in the CI/CD process. In the worst case, it can even serve as a path to reach production, leading to devastating consequences for the organization.
How was this exploited?
Several instances show how this risk has been exploited in the past. For instance:
- StackOverflow team city CI server was compromised, where the attacker was able to escalate their privilege in the environment, as newly registered accounts were granted admin privileges by default.
- A self-managed GitLab server of the New York state government having sensitive secrets was compromised as it allowed anyone to self-register and get access.
CICD-SEC-3: Dependency chain abuse
When it comes to the security of a CI/CD pipeline, one of the biggest risks is dependency chain abuse. It happens when hackers exploit the way in which dependencies are pulled, enabling them to fetch and execute malicious packages.
Why is this bad?
Once downloaded, the package can run itself and perform a variety of nefarious actions, such as stealing sensitive data, compromising credentials, or even sneaking into the production environment.
There are several attack vectors, including dependency confusion, dependency hijacking, typosquatting, brandjacking.
How was this exploited?
Several notable incidents demonstrate the threat of dependency chain abuse.
CICD-SEC-4: Poisoned pipeline execution (PPE)
Poisoned Pipeline Execution (PPE) is a threat to CI/CD pipeline security and refers to the ability of a hacker who has access to the SCM without having direct access to CI/CD pipelines to manipulate the build process by modifying/poisoning the CI configuration files to run malicious commands.
Why is this bad?
The consequences of PPE can be severe, as CI environments are often a central source of sensitive secrets, credentials tokens, and more. The attacker can poison the CI process with malicious commands to compromise and send sensitive secrets like cloud provider credentials, artifact repository credentials, etc, to remote servers by modifying the build process in the CI pipeline.
Build and deploy steps can also be modified to directly promote malicious builds to production.
How was it exploited?
- GitHub Actions were abused to mine cryptocurrency by pull requests that contained malicious code.
- Abusing the terraform plan command for execution of OS commands in the CI/CD, by Alex Kaskasoli.
CICD-SEC-5: Insufficient PBAC (Pipeline-based access controls)
PBAC refers to access controls that are implemented at the pipeline level. These controls determine what resources the pipeline can access and what actions it can perform. Insufficient PBAC means that the access controls are not granular enough, allowing a pipeline stage to have more permissions than it requires.
Why is this bad?
Malicious code that gains access to a pipeline stage can access the same resources and systems as that stage. An overly permissive stage can be used by an attacker to get access to secrets and/or connect to any resources the pipeline has access to. An attacker who gains access to the pipeline can exploit these permissions to move laterally inside and outside the execution environment, accessing sensitive systems and exfiltrating secrets.
How was this exploited?
Recent attacks have highlighted the importance of implementing strong PBAC controls.
- Codecov was compromised to steal environment variables.
- A vulnerability found in Teleport’s CI implementation allowed attackers to run a privileged container and escalate to root privilege — leading to secret exfiltration, the release of malicious artifacts, and access to sensitive systems.
CICD-SEC-6: Insufficient credential hygiene
Insufficient credential hygiene refers to the failure to manage and protect secrets and credentials used in the pipeline due to flaws in access control and over-permissive credentials.
Why is this bad?
Credentials are among the most valuable data that attackers seek, and a leak can give them access to high-value resources, enabling them to modify the build process or push malicious code to production.
How was this exploited?
The recent security breaches at Codecov and Samsung serve as a stark reminder of the importance of proper credential hygiene in CI/CD pipelines.
- Codecov was compromised by attackers, and thousands of credentials were stolen.
- Samsung spilled SmartThings app source code and secret keys.
CICD-SEC-7: Insecure system configuration
Security configurations and hardening refer to the process of configuring systems to minimize vulnerabilities and make them less susceptible to attack. In the case of the CI/CD pipeline, this involves implementing proper access controls, minimizing network exposure, and regularly patching and updating systems. Unfortunately, this is often overlooked, making it one of the easiest targets for attackers.
Why is this bad?
Attackers can exploit the lack of security configurations and hardening to gain unauthorized access to the pipeline, allowing them to move laterally within these systems. This can result in the attacker obtaining sensitive secrets, compromising the build process, or even gaining access to sensitive resources inside and outside the engineering ecosystem. Such unauthorized access can have devastating consequences, not just for the software development process but also for the organization as a whole.
How was this exploited?
In recent years, several high-profile incidents have highlighted the dangers of poor security configurations and hardening.
CICD-SEC-8: Ungoverned usage of 3rd party services
One of the most common vulnerabilities in CI/CD systems is the ungoverned usage of third-party services. CI/CD systems often rely on multiple third-party services and plugins to perform tasks like code scanning and linting. While these integrations can be incredibly useful, they also present a significant risk as they often grant a high level of permissions for resources in the CI/CD system to third-party services. This can significantly expand the attack surface of the organization.
Why is this bad?
If a third-party service or plugin is compromised, having overly permissive access can lead to attackers pushing malicious code and compromising the CI/CD or even the production environment.
How was this exploited?
- Attackers compromised DeepSource granting them full access to the codebase of all clients of DeepSource.
- The database of the Waydev analytics platform was compromised to steal the Github and Gitlab OAuth tokens of their customers.
CICD-SEC-9: Improper artifact integrity validation
Proper artifact integrity validation is essential to ensure that only trusted code is deployed in the CI/CD pipeline. However, attackers can easily exploit the lack of mechanisms to validate artifacts or code, leading to a host of security vulnerabilities.
Why is this bad?
A deployable artifact is built from multiple sources, such as 3rd party dependencies and internal artifacts. These create multiple entry points where the artifacts can be tampered with, and without proper validation, an artifact having malicious code can easily slip through.
How was this exploited?
- The code of the Orion software of SolarWinds was changed in the build system during the build process, which affected more than 18000 organizations.
CICD-SEC-10: Insufficient logging and visibility
One of the biggest challenges in securing the CI/CD pipeline is the lack of visibility into what is happening. Attackers can take advantage of this by carrying out malicious activities undetected.
Why is this bad?
Insufficient logging and visibility can make it difficult to identify the attacker’s tactics, techniques, and procedures as part of any post-incident investigation. This makes remediation more challenging, as there may be limited information available to guide the response.
Now that we have understood the most common CI/CD security risks, let’s understand the measures to mitigate them.
There are several measures we can undertake to build a secure CI/CD pipeline.
Top best practices for securing CI/CD pipeline are:
1. Integrate Source Composition Analysis (SCA)
One of the most critical steps in securing a CI/CD pipeline is integrating SCA tools. SCA is an AppSec methodology that analyzes all open-source components and dependencies in a project to detect any vulnerabilities, issues, or risks. It can help in mitigating dependency chain abuse (CICD-SEC-3), like dependency hijacking, dependency confusion, etc.
SCA tools can be easily integrated with popular CI/CD providers and source control to automatically scan and detect security issues and vulnerabilities.
Some of the most popular SCA providers in the market are Snyk, JFrog Xray, and GitLab.
2. Integrate Static Application Security Testing (SAST)
Static application security testing (SAST) is an AppSec methodology that scans project source code, byte code, and binaries to detect any issues or vulnerabilities like SQL injection, broken access control, insecure design, etc.
SAST goes hand in hand with SCA and helps in detecting and mitigating OWASP top 10 and other vulnerabilities in the source code. By integrating SAST, you can identify vulnerabilities early in the development cycle and save time and resources.
Some of the most popular SAST providers in the market are Checkmarx, Fortify, Sempgrep, Snyk, and SonarQube.
3. Introduce access control
As we saw earlier, inadequate identity and access management (CICD-SEC-2) and insufficient pipeline-based access control (CICD-SEC-5) are two of the top risks for CI/CD.
To address these, it is important to ask 2 questions in the context of access control for the CI/CD pipelines:
- Who has access to the pipelines, and the limit of their access?
- What access do the pipelines have to do their job?
Managing access to the pipelines
Pipelines are at the heart of developing, testing and distributing software, and many identities require access to them. More often than not, these identities are overly permissive, shared, and sometimes even stale (not in use). This may create an opportunity for an attacker to gain access to the pipelines, aka the crown jewels of your organization.
It is, therefore, important to manage them well and grant access based on the principle of least privilege. Moreover, identities should not be shared and regularly audited to clean up any stale identities to mitigate all risks associated with them.
Managing access of the pipelines
CI/CD pipelines are high-value resources, often requiring a high level of access to multiple sensitive resources to do their job. Sometimes, pipelines are granted all the permissions, which poses a severe risk. If an attacker gains access to an overly privileged pipeline, she can exploit this to move laterally in the engineering ecosystem and can do significant damage.
It is important to limit the access of the pipelines to only the necessary permissions to avoid any lateral movement in case they are compromised. Moreover, no pipelines should share the same set of permissions or nodes that they run on unless their job functions are identical.
4. Threat modeling
Threat modeling is a proactive approach that helps to identify potential risks and threats to the CI/CD pipeline. This process involves analyzing the system and identifying potential vulnerabilities and weaknesses that could be exploited by attackers.
Once these threats are identified, appropriate countermeasures can be implemented to prevent or mitigate them, thereby improving the overall security posture of the system. By regularly conducting threat modeling exercises, organizations can ensure that their CI/CD pipelines remain secure and resilient against potential attacks.
5. Secure secrets management
The management of secrets and credentials is a critical aspect of securing a CI/CD pipeline and can help prevent insufficient credential hygiene (CICD-SEC-6).
Avoid hardcoding secrets and ensure that they are not checked into source code. IDE security plugins, pre-commit hooks, and regular repository scans can help detect hard-coded secrets and protect against inadvertently logging sensitive information.
Secrets should be stored in an encrypted format, and the permissions associated with them should follow the principle of least privilege. It is important not to share them with others (users/machines).
Periodically rotate all static secrets and remove outdated ones. Additionally, logging and monitoring can aid in detecting and mitigating the unauthorized use of secrets and other potential threats.
Tools like HashiCorp Vault, AWS Secrets Manager, Knox, etc, can be used to effectively manage secrets and reduce the risk of a security breach.
6. Secure containers and registry
As the use of containers in software development and delivery continues to grow, securing them becomes more critical. Attackers can exploit vulnerabilities in container images or insecure networking and even gain root privileges on the host to compromise CI/CD pipelines.
One way to mitigate the risks associated with containers is to regularly scan images and registries to detect any potential security issues or vulnerabilities and remediate or mitigate them.
Tools such as Aqua Security, Qualys, etc., help to detect and mitigate any security issues around container images or registries.
Read more about Container Security Best Practices.
7. Protect CI/CD configurations
The CI/CD configurations are frequently saved alongside the code. GitHub workflow configurations, for example, are frequently stored in the same repository under the path .github/workflows. Anyone with repository access can easily modify these configurations and manipulate the CI/CD process.
Branch protection rules are therefore required for branches that are configured to trigger a pipeline in the CI/CD system, and any configuration changes must be reviewed before running. Alternatively, these configurations can be managed in a branch separate from the one used for pushing code. The branch that contains the configuration should be marked as protected.
These measures can help in protecting against poisoned pipeline execution (CICD-SEC-4).
8. Dependency management
Dependency management can help against dependency chain abuse (CICD-SEC-3). All packages should be pulled from an internal registry that only contains pre-vetted packages instead of directly pulling them from the internet.
Checksum and signature verification should be enabled to ensure that clients receive the correct dependencies. Additionally, lock the package version and avoid pulling the most recent versions without first ensuring that the most recent version is free of vulnerabilities.
9. Logging, monitoring, observability, and alerts
Lack of visibility can lead to an attack going unnoticed and make it challenging to investigate an incident and determine the extent of the breach.
Effective logging, monitoring, observability, and alerts are crucial for securing a CI/CD pipeline. They help in remediating insufficient logging and visibility (CICD-SEC-10) risk.
Logging events such as user/machine access, configuration changes, triggers, and code execution context can aid in incident investigation. Proper monitoring and observability enable the system to be observed and detect any anomalies indicating a security incident or vulnerability.
Alerts complement logging and monitoring by bringing the detected anomalies to the surface in near real-time. Tools like ELK and Prometheus-Grafana can be used for effective monitoring and observability.
10. Auditing
Auditing is an essential part of maintaining a secure CI/CD pipeline. Regular audits ensure that the systems and processes within the pipeline are compliant with the security standards and policies established by the organization. These audits can identify any weaknesses or vulnerabilities in the system that could be exploited by an attacker and assess whether the security measures in place are effective in protecting the system.
Access control auditing is particularly important in the context of CI/CD pipelines. This involves verifying that the right people have access to the right resources and that they only have the level of access they need to perform their job functions.
Regularly auditing 3rd party integrations help in mitigating ungoverned usage of 3rd party services (CICD-SEC-8), as these integrations often have access to sensitive information and resources, and any vulnerabilities in these integrations can be exploited to compromise the entire pipeline.
All these measures help in securing your CI/CD pipeline. In the next section, we will discuss some best practices to keep in mind for hardening CI/CD pipelines.
11. Configure branch protection rules, artifact validation, and approvals
Configure branch protection rules to enforce security requirements, such as only allowing signed commits and merging only reviewed code to the branches that trigger production deployment.
Another important practice is to have an approval-based system that requires all code changes to be reviewed and validated before they can be merged and trigger deployment. This adds an additional layer of security to the system and ensures that no single entity(human/machine) can push code to production. This helps in resolving the risks of insufficient flow control mechanisms (CICD-SEC-1)
Allow only signed and validated artifacts to flow through the CI/CD pipeline. This ensures that only trusted and legitimate artifacts are deployed to production, thereby minimizing the risk of security breaches. To implement this, it is recommended to use a rigorous validation process that scrutinizes each artifact before it moves further down the pipeline and helps in preventing improper artifact integrity validation (CICD-SEC-9).
With these measures in place, the CI/CD pipeline can operate securely and efficiently.
12. Access control based on the principle of least privilege
Access control is a fundamental aspect of any secure system. The principle of least privilege (PoLP) is a guiding principle in access control that limits users and machines to only the permissions necessary for their respective tasks. By implementing the PoLP, you can reduce the risk of security breaches that could arise from users with excessive privileges.
It is essential to maintain clear visibility of who has access to what in order to maintain proper access control. This visibility ensures that the privileges assigned to each user or machine are limited to their scope of work. Additionally, regular audits and reviews of access control mechanisms can help identify and correct any potential misconfigurations or weaknesses that may exist.
13. Have security and quality gates
Security and quality gates serve as checkpoints in the CI/CD pipeline that detect potential issues, vulnerabilities, or other problems early on. These gates can prevent malicious code from being deployed to production and ensure that only code and artifacts that meet the required security and quality standards move further down the pipeline.
14. Enforce secure coding standards
Enforcing secure coding standards can help in preventing known vulnerabilities in code at the time of development itself by adhering to a set of secure coding rules and guidelines. A good place to start is OWASP secure coding practices.
15. Keep credentials safe
One of the most important aspects of securing CI/CD pipelines is to keep credentials safe. CI/CD pipelines are a gold mine of sensitive secrets and credentials such as access keys, API tokens, and login credentials. These should never be hardcoded or stored in plain text. Instead, they should be encrypted and stored in a secure location with access only given to the scope in which they are needed. Furthermore, just-in-time access should be enabled wherever possible to reduce the exposure of secrets.
16. Clean up temporary resources
CI/CD pipelines are dynamic in nature, which means that temporary resources are frequently created, and these resources can pose a significant security risk.
Nodes perform multiple jobs in the CI/CD pipeline, and these jobs often require temporary resources such as containers and artifacts. If these resources are not properly cleaned up after use, they can linger on the node and create potential security vulnerabilities.
Cleaning up these temporary resources is, therefore, a critical part of hardening CI/CD pipelines. This process involves removing any artifacts, containers, or other resources that were created during the pipeline execution. By doing so, the risk of an attacker gaining access to sensitive data or exploiting vulnerabilities is significantly reduced.
17. Automate security tasks
Automation is the key to improving security by eliminating human errors. Automate routine security tasks, such as scanning for vulnerabilities or applying security patches. This also helps the security team to focus on more complex tasks and allows them to respond more quickly to security incidents.
18. Configure local security plugins, linters, and pre-commit hooks
Shifting security left means implementing security measures that can detect issues at the earliest. Equipping the developer engineering environment with tools like security plugins, linters, etc can detect any security issues at the time of writing the code itself and avoid running this vulnerable code in CI/CD pipelines.
Additionally, pre-commit hooks can be configured with a set of security rules to detect any security issues at the time of committing.
Some examples of such plugins and tools are TFLint, snyk plugin for intellij, pre-commit, etc.
19. Detect misconfigurations
Detecting misconfigurations in the CI/CD process is crucial in ensuring the security and integrity of the system. Regular scanning and monitoring of all systems involved can help identify potential security issues or misconfigurations that may be exploited by an attacker. These misconfigurations can involve network settings, access control policies, and infrastructure security configurations.
It’s important to keep all nodes updated and patched regularly to prevent known vulnerabilities from being exploited. By proactively identifying and addressing misconfigurations, the security posture of the CI/CD pipeline can be strengthened and better protected against attacks and helps in resolving one of the common risks of insecure system configuration (CICD-SEC-7).
20. Have a response plan
It is crucial to acknowledge that despite implementing all the necessary security measures, no system can be considered completely secure. Therefore, it is necessary to have a well-defined response plan in case of any security breach or system failure. Such a plan should aim to mitigate and control the damage caused and recover from the incident as quickly as possible.
Having a set of incident response Standard Operating Procedures (SOPs) is vital in dealing with such situations. These procedures should outline the steps to be taken in case of a breach or system failure depending on the nature of the incident. For example, temporary revocation of admin-level access.
Aside from the best practices mentioned above, keep security in mind when picking your CI/CD platform. For example, security is one of Spacelift’s biggest priorities, and apart from the state-of-the-art security solutions that are embedded inside the product like Policy as Code, Encryption, Single Sign On (SSO), Private Worker Pools, and others. In some organizations, there are tools that engineers were accustomed to and want to have the possibility to integrate them. When it comes to integrating security tools in your workflows, Spacelift has you covered with the Custom Inputs feature. Read more on integrating security tools with Spacelift.
Spacelift also adds an extra layer of policy that allows you to control – separately from your infrastructure project – what changes can be made, when, and by whom. This isn’t only useful to protect yourself from the baddies but allows you to implement an automated code review pipeline. Check out the documentation.
AI can help automate and streamline CI/CD security processes, reduce the time and effort required to detect and respond to security threats and provide greater visibility into the security posture of an organization’s software development pipeline.
- Threat detection: AI and ML can detect security threats in the code, the pipeline, or the infrastructure and alert security teams in real time. It can also improve the accuracy of threat detection by reducing false positives and false negatives.
- AI-powered security tools: AI-powered tools can help enhance the capabilities of existing tools in security tasks such as vulnerability scanning, analyzing logs, real-time monitoring, etc.
- Predictive Analysis: AI and ML algorithms can be used for predictive analysis to detect potential security vulnerabilities based on historical data and assist teams in taking proactive measures to prevent them. These tools may also examine code repositories and identify potential vulnerabilities or weaknesses that attackers could exploit.
- Behavioral analytics and anomaly detection: AI algorithms can monitor and recognize patterns in user behavior, network traffic, logs, etc. and identify unusual activities and anomalies that could indicate a security threat, such as unauthorized access and other security issues.
Protecting your CI/CD pipeline is critical to your organization’s success, reputation, and longevity. The CI/CD pipeline is the lifeline of software development organizations, and it must be guarded at all times.
Adopting CI/CD security best practices, such as integrating security into the pipeline, vulnerability scanning, access control, and more, can help ensure that your pipeline is secure from attacks and can deliver software faster, more reliably, and with greater confidence.
The Most Flexible CI/CD Automation Tool
Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.
