Omkar is a CoFounder at Interleap and a seasoned DevSecOps engineer with a passion for DevOps and security. He specializes in AWS DevOps, Terraform, CDKTF and Docker.

All posts by Omkar Birade - Infrastructure as Code Expert

Omkar Birade

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

General | Spacelift Blog

General

In today’s fast-paced software industry, organizations are increasingly relying on the Continuous Integration/Continuous Delivery (CI/CD) pipelines to enhance efficiency, speed, and reliability in software development. However, the rapid adoption of CI/CD has resulted in a significant increase in security threats. 
In this article, we will cover:
<ol>
<li><a href="https://spacelift.io/blog/ci-cd-security#why-is-cicd-security-crucial">Why is CI/CD security crucial?</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-security#what-is-cicd-security">What is CI/CD security?</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-security#owasp-top-10-cicd-security-risks">OWASP Top 10 CI/CD security risks</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-security#how-to-secure-a-cicd-pipeline">How to secure a CI/CD pipeline</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-security#best-practices-for-hardening-cicd-pipelines">Best practices for hardening CI/CD pipelines</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-security#key-points">How AI/ML can help in CI/CD security?</a></li>
</ol>

The <a href="https://spacelift.io/blog/ci-cd-pipeline" target="_blank" rel="noopener">CI/CD pipeline</a> is the backbone of software development organizations, enabling them to build, test, and deliver any software faster, more frequently, and more reliably.
But what happens when these pipelines break?
This can result in significant chaos, hampered collaboration, and delays in delivering new code to end-users, leading to a loss of revenue, reputation, and even legal consequences.
If this gives you goosebumps, there is something even scarier.
What happens if these pipelines are compromised by attackers? Worse yet, what if this occurs and the security incident is not detected for a long time or at all?” 😲
Consider the <a href="https://www.cyberark.com/resources/blog/the-anatomy-of-the-solarwinds-attack-chain" target="_blank" rel="nofollow noopener">SolarWinds</a> CI/CD security incident, where attackers targeted the pipeline, affecting more than 18,000 organizations and gaining access to their networks, systems, and data undetected for over 14 months. This attack alone underscores the critical need for CI/CD security, and it’s not the only one. The <a href="https://www.theregister.com/2021/01/27/stack_overflow_2019_hack_was/" target="_blank" rel="nofollow noopener">StackOverflow</a> incident, where vulnerabilities in the build systems led to stolen source code and leaked secrets, the <a href="https://www.theregister.com/2021/03/29/php_repository_infected/" target="_blank" rel="nofollow noopener">php</a> backdoor attack, and the <a href="https://blog.gitguardian.com/codecov-supply-chain-breach/" target="_blank" rel="nofollow noopener">CodeCov</a> exploit all serve as examples of the devastating effects of CI/CD security breaches.
The significance of protecting the CI/CD process cannot be overstated, and it has shifted the focus of security teams in recent years.

CI/CD security refers to the practice of integrating security into the CI/CD pipeline to identify and remediate any risks or vulnerabilities throughout all the stages of the CI/CD process. To protect your pipeline against potential threats, it is essential to understand the top security risks associated with CI/CD pipelines.
In the next section, we will discuss the top 10 CI/CD security risks.

<h3><a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-01-Insufficient-Flow-Control-Mechanisms" target="_blank" rel="nofollow noopener">CICD-SEC-1</a>: Insufficient Flow Control Mechanisms</h3>
The first CI/CD security risk on the list refers to the ability of an attacker who has gained access to any of the systems in the CI/CD process (SCM, CI, or runners) to push malicious code down the pipeline due to a lack of control mechanisms that enforce additional approval or review.
Why is this bad?
CI/CD pipelines are a direct bridge from the developers’ machine to the production environment. In the absence of flow control like reviews or approvals, a single entity (human or machine) can push malicious code to production and publish packages with malicious code. 
This means that once the system is compromised, there are no constraints to the limit of the damage that the malicious actors can do.
How was this exploited?
<ul>
<li style="font-weight: 400;" aria-level="1">An attacker planted a backdoor in the <a href="https://news-web.php.net/php.internals/113981" target="_blank" rel="nofollow noopener">PHP</a> git repository by pushing malicious unreviewed code.</li>
<li style="font-weight: 400;" aria-level="1">On Homebrew, a security researcher found a way to <a href="https://brew.sh/2021/04/21/security-incident-disclosure/" target="_blank" rel="nofollow noopener">abuse</a> the auto-merge rules to push malicious code.</li>
</ul>

<h3><a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-02-Inadequate-Identity-And-Access-Management" target="_blank" rel="nofollow noopener">CICD-SEC-2</a>: Inadequate Identity and Access Management</h3>
This refers to the lack of proper identity and access management in CI/CD ecosystems, given the vast number of identities to manage and the complexity of interconnected systems involved, such as SCM and CI, which often have different authentication and authorization methods. 
In many cases, user/machine accounts do not follow the principle of least privilege and have overly permissive default permissions.
Why is this bad?
The consequences of inadequate identity and access management in CI/CD processes are severe. The existence of multiple identities paired with poor identity and access control creates an opportunity for hackers, where compromising a single account could lead to them having control over the systems involved in the CI/CD process. In the worst case, it can even serve as a path to reach production, leading to devastating consequences for the organization.
How was this exploited?
Several instances show how this risk has been exploited in the past. For instance:
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://stackoverflow.blog/2021/01/25/a-deeper-dive-into-our-may-2019-security-incident/" target="_blank" rel="nofollow noopener">StackOverflow</a> team city CI server was compromised, where the attacker was able to escalate their privilege in the environment, as newly registered accounts were granted admin privileges by default.</li>
<li style="font-weight: 400;" aria-level="1">A self-managed <a href="https://techcrunch.com/2021/06/24/an-internal-code-repo-used-by-new-york-states-it-office-was-exposed-online/" target="_blank" rel="nofollow noopener">GitLab</a> server of the New York state government having sensitive secrets was compromised as it allowed anyone to self-register and get access.</li>
</ul>

<h3><a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-03-Dependency-Chain-Abuse" target="_blank" rel="nofollow noopener">CICD-SEC-3</a>: Dependency Chain Abuse</h3>
When it comes to the security of a CI/CD pipeline, one of the biggest risks is dependency chain abuse. It happens when hackers exploit the way in which dependencies are pulled, enabling them to fetch and execute malicious packages.
Why is this bad?
Once downloaded, the package can run itself and perform a variety of nefarious actions, such as stealing sensitive data, compromising credentials, or even sneaking into the production environment.
There are several attack vectors, including <a href="https://dhiyaneshgeek.github.io/web/security/2021/09/04/dependency-confusion/" target="_blank" rel="nofollow noopener">dependency confusion</a>, <a href="https://cyware.com/news/analyzing-the-deadly-rise-in-npm-package-hijacking-78b24364" target="_blank" rel="nofollow noopener">dependency hijacking</a>, <a href="https://snyk.io/blog/typosquatting-attacks/" target="_blank" rel="nofollow noopener">typosquatting</a>, <a href="https://blog.sonatype.com/twilio-npm-is-brandjacking-malware-in-disguise" target="_blank" rel="nofollow noopener">brandjacking</a>.
How was this exploited?
Several notable incidents demonstrate the threat of dependency chain abuse.
<ul>
<li style="font-weight: 400;" aria-level="1">The popular ua-parser-js NPM library, with 9 million downloads a week, was <a href="https://github.com/advisories/GHSA-pjwm-rvh2-c87w" target="_blank" rel="nofollow noopener">hijacked</a> to launch crypto-miners and steal credentials.</li>
<li style="font-weight: 400;" aria-level="1">Amazon, Zillow, Lyft, and Slack NodeJS apps <a href="https://www.bleepingcomputer.com/news/security/malicious-npm-packages-target-amazon-slack-with-new-dependency-attacks/" target="_blank" rel="nofollow noopener">targeted</a> using the dependency confusion vulnerability.</li>
</ul>

<h3><a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-04-Poisoned-Pipeline-Execution" target="_blank" rel="nofollow noopener">CICD-SEC-4</a>: Poisoned Pipeline Execution (PPE)</h3>
Poisoned Pipeline Execution (PPE) is a threat to CI/CD pipeline security and refers to the ability of a hacker who has access to the SCM without having direct access to CI/CD pipelines to manipulate the build process by modifying/poisoning the CI configuration files to run malicious commands.
Why is this bad?
The consequences of PPE can be severe, as CI environments are often a central source of sensitive secrets, credentials tokens, and more. The attacker can poison the CI process with malicious commands to compromise and send sensitive secrets like cloud provider credentials, artifact repository credentials, etc, to remote servers by modifying the build process in the CI pipeline.
Build and deploy steps can also be modified to directly promote malicious builds to production.
How was it exploited?
<ul>
<li style="font-weight: 400;" aria-level="1">GitHub Actions were <a href="https://dev.to/thibaultduponchelle/the-github-action-mining-attack-through-pull-request-2lmc" target="_blank" rel="nofollow noopener">abused</a> to mine cryptocurrency by pull requests that contained malicious code.</li>
<li><a href="https://alex.kaskaso.li/post/terraform-plan-rce" target="_blank" rel="nofollow noopener">Abusing</a> the <a href="https://spacelift.io/blog/terraform-plan" target="_blank" rel="noopener">terraform plan</a><a href="https://spacelift.io/blog/terraform-plan" target="_blank" rel="noopener"> command</a> for execution of OS commands in the CI/CD, by <a href="https://twitter.com/alxk7i" target="_blank" rel="nofollow noopener">Alex Kaskasoli</a>.</li>
</ul>

<h3><a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-05-Insufficient-PBAC" target="_blank" rel="nofollow noopener">CICD-SEC-5</a>: Insufficient PBAC (Pipeline-Based Access Controls)</h3>
PBAC refers to access controls that are implemented at the pipeline level. These controls determine what resources the pipeline can access and what actions it can perform. Insufficient PBAC means that the access controls are not granular enough, allowing a pipeline stage to have more permissions than it requires.
Why is this bad?
Malicious code that gains access to a pipeline stage can access the same resources and systems as that stage. An overly permissive stage can be used by an attacker to get access to secrets and/or connect to any resources the pipeline has access to. An attacker who gains access to the pipeline can exploit these permissions to move laterally inside and outside the execution environment, accessing sensitive systems and exfiltrating secrets.
How was this exploited?
Recent attacks have highlighted the importance of implementing strong PBAC controls.
<ul>
<li style="font-weight: 400;" aria-level="1"><a href="https://about.codecov.io/security-update/" target="_blank" rel="nofollow noopener">Codecov</a> was compromised to steal environment variables.</li>
<li style="font-weight: 400;" aria-level="1">A <a href="https://goteleport.com/blog/hack-via-pull-request/" target="_blank" rel="nofollow noopener">vulnerability</a> found in Teleport’s CI implementation allowed attackers to run a privileged container and escalate to root privilege — leading to secret exfiltration, the release of malicious artifacts, and access to sensitive systems.</li>
</ul>

<h3><a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-06-Insufficient-Credential-Hygiene" target="_blank" rel="nofollow noopener">CICD-SEC-6</a>: Insufficient Credential Hygiene</h3>
Insufficient credential hygiene refers to the failure to manage and protect secrets and credentials used in the pipeline due to flaws in access control and over-permissive credentials.
Why is this bad?
Credentials are among the most valuable data that attackers seek, and a leak can give them access to high-value resources, enabling them to modify the build process or push malicious code to production.
How was this exploited?
The recent security breaches at Codecov and Samsung serve as a stark reminder of the importance of proper credential hygiene in CI/CD pipelines.
<ul>
<li style="font-weight: 400;" aria-level="1">Codecov was <a href="https://about.codecov.io/security-update/" target="_blank" rel="nofollow noopener">compromised</a> by attackers, and thousands of credentials were stolen.</li>
<li>Samsung <a href="https://techcrunch.com/2019/05/08/samsung-source-code-leak/" target="_blank" rel="nofollow noopener">spilled</a> SmartThings app source code and secret keys.</li>
</ul>

<h3><a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-07-Insecure-System-Configuration" target="_blank" rel="nofollow noopener">CICD-SEC-7</a>: Insecure System Configuration</h3>
Security configurations and hardening refer to the process of configuring systems to minimize vulnerabilities and make them less susceptible to attack. In the case of the CI/CD pipeline, this involves implementing proper access controls, minimizing network exposure, and regularly patching and updating systems. Unfortunately, this is often overlooked, making it one of the easiest targets for attackers.
Why is this bad?
Attackers can exploit the lack of security configurations and hardening to gain unauthorized access to the pipeline, allowing them to move laterally within these systems. This can result in the attacker obtaining sensitive secrets, compromising the build process, or even gaining access to sensitive resources inside and outside the engineering ecosystem. Such unauthorized access can have devastating consequences, not just for the software development process but also for the organization as a whole.
How was this exploited?
In recent years, several high-profile incidents have highlighted the dangers of poor security configurations and hardening.
<ul>
<li style="font-weight: 400;" aria-level="1">Mercedes-Benz onboard logic unit (OLU) source code was <a href="https://www.zdnet.com/article/mercedes-benz-onboard-logic-unit-olu-source-code-leaks-online/" target="_blank" rel="nofollow noopener">leaked</a> as a result of opening a self-maintained Gitlab server for self-registration.</li>
<li>Nissan source code was <a href="https://www.zdnet.com/article/nissan-source-code-leaked-online-after-git-repo-misconfiguration/" target="_blank" rel="nofollow noopener">leaked</a> online after Git repo misconfiguration.</li>
</ul>

<h3><a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-08-Ungoverned-Usage-of-3rd-Party-Services" target="_blank" rel="nofollow noopener">CICD-SEC-8</a>: Ungoverned Usage of 3rd Party Services</h3>
One of the most common vulnerabilities in CI/CD systems is the ungoverned usage of third-party services. CI/CD systems often rely on multiple third-party services and plugins to perform tasks like code scanning and linting. While these integrations can be incredibly useful, they also present a significant risk as they often grant a high level of permissions for resources in the CI/CD system to third-party services. This can significantly expand the attack surface of the organization.
Why is this bad?
If a third-party service or plugin is compromised, having overly permissive access can lead to attackers pushing malicious code and compromising the CI/CD or even the production environment.
How was this exploited?
<ul>
<li style="font-weight: 400;" aria-level="1">Attackers <a href="https://discuss.deepsource.io/t/security-incident-on-deepsource-s-github-application/131" target="_blank" rel="nofollow noopener">compromised</a> DeepSource granting them full access to the codebase of all clients of DeepSource.</li>
<li>The database of the Waydev analytics platform was <a href="https://changelog.waydev.co/github-and-gitlab-oauth-security-update-dw98s" target="_blank" rel="nofollow noopener">compromised</a> to steal the Github and Gitlab OAuth tokens of their customers.</li>
</ul>

<h3><a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-09-Improper-Artifact-Integrity-Validation" target="_blank" rel="nofollow noopener">CICD-SEC-9</a>: Improper Artifact Integrity Validation</h3>
Proper artifact integrity validation is essential to ensure that only trusted code is deployed in the CI/CD pipeline. However, attackers can easily exploit the lack of mechanisms to validate artifacts or code, leading to a host of security vulnerabilities.
Why is this bad?
A deployable artifact is built from multiple sources, such as 3rd party dependencies and internal artifacts. These create multiple entry points where the artifacts can be tampered with, and without proper validation, an artifact having malicious code can easily slip through.
How was this exploited?
<ul>
<li>The code of the Orion software of SolarWinds was changed in the build system during the build process, which <a href="https://www.cyberark.com/resources/blog/the-anatomy-of-the-solarwinds-attack-chain" target="_blank" rel="nofollow noopener">affected</a> more than 18000 organizations.</li>
</ul>

<h3><a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-10-Insufficient-Logging-And-Visibility" target="_blank" rel="nofollow noopener">CICD-SEC-10</a>: Insufficient Logging and Visibility</h3>
One of the biggest challenges in securing the CI/CD pipeline is the lack of visibility into what is happening. Attackers can take advantage of this by carrying out malicious activities undetected.
Why is this bad?
Insufficient logging and visibility can make it difficult to identify the attacker’s tactics, techniques, and procedures as part of any post-incident investigation. This makes remediation more challenging, as there may be limited information available to guide the response.
Now that we have understood the most common CI/CD security risks, let’s understand the measures to mitigate them.

<ul>
<li><a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener">Deploying Infrastructure in CI/CD with Terraform</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
<li><a href="https://spacelift.io/blog/why-devops-engineers-recommend-spacelift" target="_blank" rel="noopener">Why DevOps Engineers Recommend Spacelift</a></li>
</ul>

There are several measures we can undertake to build a secure CI/CD pipeline. Some of them are as follows:
<h3>1. Integrating Source Composition Analysis (SCA)</h3>
One of the most critical steps in securing a CI/CD pipeline is integrating SCA tools. SCA is an AppSec methodology that analyzes all open-source components and dependencies in a project to detect any vulnerabilities, issues, or risks. It can help in mitigating dependency chain abuse (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-03-Dependency-Chain-Abuse" target="_blank" rel="nofollow noopener">CICD-SEC-3</a>), like dependency hijacking, dependency confusion, etc.
SCA tools can be easily integrated with popular CI/CD providers and source control to automatically scan and detect security issues and vulnerabilities.
Some of the most popular SCA providers in the market are Snyk, JFrog Xray, and GitLab.
<h3>2. Integrating Static Application Security Testing (SAST)</h3>
Static application security testing (SAST) is an AppSec methodology that scans project source code, byte code, and binaries to detect any issues or vulnerabilities like SQL injection, broken access control, insecure design, etc. 
SAST goes hand in hand with SCA and helps in detecting and mitigating <a href="https://github.com/OWASP/API-Security/tree/master/2023/en/src" target="_blank" rel="nofollow noopener">OWASP top 10</a> and other vulnerabilities in the source code. By integrating SAST, you can identify vulnerabilities early in the development cycle and save time and resources.
Some of the most popular SAST providers in the market are Checkmarx, Fortify, Sempgrep, Snyk, and SonarQube.
<h3>3. Access control</h3>
As we saw earlier, inadequate identity and access management (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-02-Inadequate-Identity-And-Access-Management" target="_blank" rel="nofollow noopener">CICD-SEC-2</a>) and insufficient pipeline-based access control (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-05-Insufficient-PBAC" target="_blank" rel="nofollow noopener">CICD-SEC-5</a>) are two of the top risks for CI/CD.
To address these, it is important to ask 2 questions in the context of access control for the CI/CD pipelines:
<ul>
<li aria-level="1">Who has access to the pipelines, and the limit of their access?</li>
<li aria-level="1">What access do the pipelines have to do their job?</li>
</ul>
Managing access to the pipelines
Pipelines are at the heart of developing, testing and distributing software, and many identities require access to them. More often than not, these identities are overly permissive, shared, and sometimes even stale (not in use). This may create an opportunity for an attacker to gain access to the pipelines, aka the crown jewels of your organization. 
It is, therefore, important to manage them well and grant access based on the principle of least privilege. Moreover, identities should not be shared and regularly audited to clean up any stale identities to mitigate all risks associated with them.
Managing access of the pipelines
CI/CD pipelines are high-value resources, often requiring a high level of access to multiple sensitive resources to do their job. Sometimes, pipelines are granted all the permissions, which poses a severe risk. If an attacker gains access to an overly privileged pipeline, she can exploit this to move laterally in the engineering ecosystem and can do significant damage.
It is important to limit the access of the pipelines to only the necessary permissions to avoid any lateral movement in case they are compromised. Moreover, no pipelines should share the same set of permissions or nodes that they run on unless their job functions are identical.
<h3>4. Threat modeling</h3>
Threat modeling is a proactive approach that helps to identify potential risks and threats to the CI/CD pipeline. This process involves analyzing the system and identifying potential vulnerabilities and weaknesses that could be exploited by attackers. 
Once these threats are identified, appropriate countermeasures can be implemented to prevent or mitigate them, thereby improving the overall security posture of the system. By regularly conducting threat modeling exercises, organizations can ensure that their CI/CD pipelines remain secure and resilient against potential attacks.
<h3>5. Secure secrets management</h3>
The management of secrets and credentials is a critical aspect of securing a CI/CD pipeline and can help prevent insufficient credential hygiene (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-06-Insufficient-Credential-Hygiene" target="_blank" rel="nofollow noopener">CICD-SEC-6</a>).
Avoid hardcoding secrets and ensure that they are not checked into source code. IDE security plugins, pre-commit hooks, and regular repository scans can help detect hard-coded secrets and protect against inadvertently logging sensitive information. 
Secrets should be stored in an encrypted format, and the permissions associated with them should follow the principle of least privilege. It is important not to share them with others (users/machines).
Periodically rotate all static secrets and remove outdated ones. Additionally, logging and monitoring can aid in detecting and mitigating the unauthorized use of secrets and other potential threats.
Tools like HashiCorp Vault, AWS Secrets Manager, Knox, etc, can be used to effectively manage secrets and reduce the risk of a security breach.
<h3>6. Securing containers and registry</h3>
As the use of containers in software development and delivery continues to grow, securing them becomes more critical. Attackers can exploit vulnerabilities in container images or insecure networking and even gain root privileges on the host to compromise CI/CD pipelines.
One way to mitigate the risks associated with containers is to regularly scan images and registries to detect any potential security issues or vulnerabilities and remediate or mitigate them.
Tools such as Aqua Security, Qualys, etc., help to detect and mitigate any security issues around container images or registries.
Read more about <a href="https://spacelift.io/blog/container-security" target="_blank" rel="noopener">Container Security Best Practices</a>.
<h3>7. Protect CI/CD configurations</h3>
The CI/CD configurations are frequently saved alongside the code. GitHub workflow configurations, for example, are frequently stored in the same repository under the path <code>.github/workflows</code>. Anyone with repository access can easily modify these configurations and manipulate the CI/CD process.
Branch protection rules are therefore required for branches that are configured to trigger a pipeline in the CI/CD system, and any configuration changes must be reviewed before running. Alternatively, these configurations can be managed in a branch separate from the one used for pushing code. The branch that contains the configuration should be marked as protected.
These measures can help in protecting against poisoned pipeline execution (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-04-Poisoned-Pipeline-Execution" target="_blank" rel="nofollow noopener">CICD-SEC-4</a>).
<h3>8. Dependency management</h3>
Dependency management can help against dependency chain abuse (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-03-Dependency-Chain-Abuse" target="_blank" rel="nofollow noopener">CICD-SEC-3</a>). All packages should be pulled from an internal registry that only contains pre-vetted packages instead of directly pulling them from the internet.
Checksum and signature verification should be enabled to ensure that clients receive the correct dependencies. Additionally, lock the package version and avoid pulling the most recent versions without first ensuring that the most recent version is free of vulnerabilities.
<h3>9. Logging, monitoring, observability, and alerts</h3>
Lack of visibility can lead to an attack going unnoticed and make it challenging to investigate an incident and determine the extent of the breach.
Effective logging, monitoring, observability, and alerts are crucial for securing a CI/CD pipeline. They help in remediating insufficient logging and visibility (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-10-Insufficient-Logging-And-Visibility" target="_blank" rel="nofollow noopener">CICD-SEC-10</a>) risk.
Logging events such as user/machine access, configuration changes, triggers, and code execution context can aid in incident investigation. Proper monitoring and observability enable the system to be observed and detect any anomalies indicating a security incident or vulnerability.
Alerts complement logging and monitoring by bringing the detected anomalies to the surface in near real-time. Tools like ELK and <a href="https://spacelift.io/blog/prometheus-kubernetes" target="_blank" rel="noopener">Prometheus-Grafana</a> can be used for effective monitoring and observability.
<h3>10. Auditing</h3>
Auditing is an essential part of maintaining a secure CI/CD pipeline. Regular audits ensure that the systems and processes within the pipeline are compliant with the security standards and policies established by the organization. These audits can identify any weaknesses or vulnerabilities in the system that could be exploited by an attacker and assess whether the security measures in place are effective in protecting the system.
Access control auditing is particularly important in the context of CI/CD pipelines. This involves verifying that the right people have access to the right resources and that they only have the level of access they need to perform their job functions.
Regularly auditing 3rd party integrations help in mitigating ungoverned usage of 3rd party services (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-08-Ungoverned-Usage-of-3rd-Party-Services" target="_blank" rel="nofollow noopener">CICD-SEC-8</a>), as these integrations often have access to sensitive information and resources, and any vulnerabilities in these integrations can be exploited to compromise the entire pipeline.
All these measures help in securing your CI/CD pipeline. In the next section, we will discuss some best practices to keep in mind for hardening CI/CD pipelines.

<h3>1. Configuring branch protection rules, artifact validation, and approvals</h3>
Configure branch protection rules to enforce security requirements, such as only allowing signed commits and merging only reviewed code to the branches that trigger production deployment.
Another important practice is to have an approval-based system that requires all code changes to be reviewed and validated before they can be merged and trigger deployment. This adds an additional layer of security to the system and ensures that no single entity(human/machine) can push code to production. This helps in resolving the risks of insufficient flow control mechanisms (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-01-Insufficient-Flow-Control-Mechanisms" target="_blank" rel="nofollow noopener">CICD-SEC-1</a>)
Allow only signed and validated artifacts to flow through the CI/CD pipeline. This ensures that only trusted and legitimate artifacts are deployed to production, thereby minimizing the risk of security breaches. To implement this, it is recommended to use a rigorous validation process that scrutinizes each artifact before it moves further down the pipeline and helps in preventing improper artifact integrity validation (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-09-Improper-Artifact-Integrity-Validation" target="_blank" rel="nofollow noopener">CICD-SEC-9</a>).
With these measures in place, the CI/CD pipeline can operate securely and efficiently.
<h3>2. Access control based on the principle of least privilege</h3>
Access control is a fundamental aspect of any secure system. The principle of least privilege (PoLP) is a guiding principle in access control that limits users and machines to only the permissions necessary for their respective tasks. By implementing the PoLP, you can reduce the risk of security breaches that could arise from users with excessive privileges.
It is essential to maintain clear visibility of who has access to what in order to maintain proper access control. This visibility ensures that the privileges assigned to each user or machine are limited to their scope of work. Additionally, regular audits and reviews of access control mechanisms can help identify and correct any potential misconfigurations or weaknesses that may exist.
<h3>3. Having security and quality gates</h3>
Security and quality gates serve as checkpoints in the CI/CD pipeline that detect potential issues, vulnerabilities, or other problems early on. These gates can prevent malicious code from being deployed to production and ensure that only code and artifacts that meet the required security and quality standards move further down the pipeline.
<h3>4. Enforce secure coding standards</h3>
Enforcing secure coding standards can help in preventing known vulnerabilities in code at the time of development itself by adhering to a set of secure coding rules and guidelines. A good place to start is <a href="https://owasp.org/www-project-secure-coding-practices-quick-reference-guide/" target="_blank" rel="nofollow noopener">OWASP secure coding practices</a>.
<h3>5. Keep credentials safe</h3>
One of the most important aspects of securing CI/CD pipelines is to keep credentials safe. CI/CD pipelines are a gold mine of sensitive secrets and credentials such as access keys, API tokens, and login credentials. These should never be hardcoded or stored in plain text. Instead, they should be encrypted and stored in a secure location with access only given to the scope in which they are needed. Furthermore, just-in-time access should be enabled wherever possible to reduce the exposure of secrets.
<h3>6. Clean up temporary resources</h3>
CI/CD pipelines are dynamic in nature, which means that temporary resources are frequently created, and these resources can pose a significant security risk.
Nodes perform multiple jobs in the CI/CD pipeline, and these jobs often require temporary resources such as containers and artifacts. If these resources are not properly cleaned up after use, they can linger on the node and create potential security vulnerabilities.
Cleaning up these temporary resources is, therefore, a critical part of hardening CI/CD pipelines. This process involves removing any artifacts, containers, or other resources that were created during the pipeline execution. By doing so, the risk of an attacker gaining access to sensitive data or exploiting vulnerabilities is significantly reduced.
<h3>7. Automation of security tasks</h3>
Automation is the key to improving security by eliminating human errors. Automate routine security tasks, such as scanning for vulnerabilities or applying security patches. This also helps the security team to focus on more complex tasks and allows them to respond more quickly to security incidents.
<h3>8. Configure local security plugins, linters, and pre-commit hooks</h3>
Shifting security left means implementing security measures that can detect issues at the earliest. Equipping the developer engineering environment with tools like security plugins, linters, etc can detect any security issues at the time of writing the code itself and avoid running this vulnerable code in CI/CD pipelines.
Additionally, pre-commit hooks can be configured with a set of security rules to detect any security issues at the time of committing.
Some examples of such plugins and tools are <a href="https://github.com/terraform-linters/tflint" target="_blank" rel="nofollow noopener">tflint</a>,<a href="https://snyk.io/blog/secure-coding-with-jetbrains-ide-plugin/" target="_blank" rel="nofollow noopener"> snyk plugin for intellij</a>, <a href="https://pre-commit.com/" target="_blank" rel="nofollow noopener">pre-commit</a>, etc.
<h3>9. Misconfiguration detection</h3>
Detecting misconfigurations in the CI/CD process is crucial in ensuring the security and integrity of the system. Regular scanning and monitoring of all systems involved can help identify potential security issues or misconfigurations that may be exploited by an attacker. These misconfigurations can involve network settings, access control policies, and infrastructure security configurations. 
It’s important to keep all nodes updated and patched regularly to prevent known vulnerabilities from being exploited. By proactively identifying and addressing misconfigurations, the security posture of the CI/CD pipeline can be strengthened and better protected against attacks and helps in resolving one of the common risks of insecure system configuration (<a href="https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-07-Insecure-System-Configuration" target="_blank" rel="nofollow noopener">CICD-SEC-7</a>).
<h3>10. Have a response plan</h3>
It is crucial to acknowledge that despite implementing all the necessary security measures, no system can be considered completely secure. Therefore, it is necessary to have a well-defined response plan in case of any security breach or system failure. Such a plan should aim to mitigate and control the damage caused and recover from the incident as quickly as possible.
Having a set of incident response Standard Operating Procedures (SOPs) is vital in dealing with such situations. These procedures should outline the steps to be taken in case of a breach or system failure depending on the nature of the incident. For example, temporary revocation of admin-level access.
Aside from the best practices mentioned above, keep security in mind when picking your CI/CD platform. For example, security is one of <a href="https://spacelift.io/" target="_blank" rel="noopener">Spacelift</a>’s biggest priorities, and apart from the state-of-the-art <a href="https://docs.spacelift.io/product/security.html" target="_blank" rel="noopener">security solutions</a> that are embedded inside the product like <a href="https://docs.spacelift.io/concepts/policy/index.html" target="_blank" rel="noopener">Policy as Code</a>, Encryption, <a href="https://docs.spacelift.io/integrations/single-sign-on/index.html" target="_blank" rel="noopener">Single Sign On (SSO)</a>, <a href="https://docs.spacelift.io/concepts/worker-pools" target="_blank" rel="noopener">Private Worker Pools</a>, and others, in some organizations, there are tools that engineers were accustomed to and want to have the possibility to integrate them. When it comes to integrating security tools in your workflows, Spacelift has you covered with the Custom Inputs feature. Read more on <a href="https://spacelift.io/blog/integrating-security-tools-with-spacelift" target="_blank" rel="noopener">integrating security tools with Spacelift</a>.
Spacelift also adds an extra layer of <a href="https://docs.spacelift.io/concepts/policy/" target="_blank" rel="noopener">policy</a> that allows you to control &#8211; separately from your infrastructure project &#8211; <a href="https://docs.spacelift.io/concepts/policy/run-initialization-policy" target="_blank" rel="noopener">what code can be executed</a>, <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy" target="_blank" rel="noopener">what changes can be made</a>, <a href="https://docs.spacelift.io/concepts/policy/stack-access-policy" target="_blank" rel="noopener">when, and by whom</a>. This isn&#8217;t only useful to protect yourself from the baddies but allows you to implement an <a href="https://docs.spacelift.io/concepts/policy/terraform-plan-policy#automated-code-review" target="_blank" rel="noopener">automated code review pipeline</a>. Check out the <a href="https://docs.spacelift.io/concepts/policy/" target="_blank" rel="noopener">documentation</a>.

AI can help automate and streamline CI/CD security processes, reduce the time and effort required to detect and respond to security threats and provide greater visibility into the security posture of an organization’s software development pipeline.
<ol>
<li style="font-weight: 400;" aria-level="1">Threat detection: AI and ML can detect security threats in the code, the pipeline, or the infrastructure and alert security teams in real time. It can also improve the accuracy of threat detection by reducing false positives and false negatives.</li>
<li style="font-weight: 400;" aria-level="1">AI-powered security tools: AI-powered tools can help enhance the capabilities of existing tools in security tasks such as vulnerability scanning, analyzing logs, real-time monitoring, etc.</li>
<li style="font-weight: 400;" aria-level="1">Predictive Analysis: AI and ML algorithms can be used for predictive analysis to detect potential security vulnerabilities based on historical data and assist teams in taking proactive measures to prevent them. These tools may also examine code repositories and identify potential vulnerabilities or weaknesses that attackers could exploit.</li>
<li style="font-weight: 400;" aria-level="1">Behavioral analytics and anomaly detection: AI algorithms can monitor and recognize patterns in user behavior, network traffic, logs, etc. and identify unusual activities and anomalies that could indicate a security threat, such as unauthorized access and other security issues.</li>
</ol>

Protecting your CI/CD pipeline is critical to your organization’s success, reputation, and longevity. The CI/CD pipeline is the lifeline of software development organizations, and it must be guarded at all times. 
Adopting CI/CD security best practices, such as integrating security into the pipeline, vulnerability scanning, access control, and more, can help ensure that your pipeline is secure from attacks and can deliver software faster, more reliably, and with greater confidence.

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Automated and Manual Code Testing with DevSecOps

16 DevOps Best Practices Every Developer Should Know

Infrastructure Drift Detection and How to Fix It With IaC Tools

Learn why CI/CD Security is the key to faster and more reliable software delivery with tips on how to keep your pipelines secure.

211-ci-cd-security

CI/CD security

CI/CD Security: What is It, Risks & Best Practices

How to Create AWS IAM Policy Using Terraform

Want to level up your Terraform knowledge? Explore our tutorials and step-by-step guides written by Terraform experts. Learn Terraform best practices and solutions to the most difficult problems.

Terraform Guides and Tutorials | Spacelift Blog

Terraform

IAM policies lie at the heart of AWS access management. They are essentially a set of permissions that can be attached to an AWS identity or resource to manage its access.
Based on the permission set, policies determine what an identity or resource is allowed to do on AWS. They are primarily stored in JSON format and are accessible through the AWS IAM console.

iam-policy-diagram

iam policy diagram

For more information about policies, check out our <a href="https://spacelift.io/blog/iam-policy" target="_blank" rel="noopener">AWS IAM Policies guide</a>.
A simple policy that allows only one action i.e <code>rds:CreateDBInstance</code> (the RDS instance creation action) looks like this:

Before we get started with <a href="https://spacelift.io/blog/what-is-terraform" target="_blank" rel="noopener">Terraform</a> and create our own policies, let’s go over the general structure of policies.

Despite the fact that there are various types of policies on AWS, all policies on AWS have the same structure, which includes a policy version and a list of statement(s). Let’s look at the purpose of each key in a well-formed policy.
<ul>
<li style="font-weight: 400;" aria-level="1">Version: Version of the policy language to use. At the time of writing, the latest version is 2012–10–17.</li>
<li style="font-weight: 400;" aria-level="1">Statement: Each statement represents a permission and has all its details.</li>
<li style="font-weight: 400;" aria-level="1">Sid: An optional name for a statement.</li>
<li style="font-weight: 400;" aria-level="1">Action: The actions that the statement allows or denies.</li>
<li style="font-weight: 400;" aria-level="1">Effect: Indicates whether an action(s) is allowed or denied using the value “Allow” or “Deny”. </li>
<li style="font-weight: 400;" aria-level="1">Principal: Represents the identity to which the policy applies. (Useful in case of resource-based policies and not required in case of IAM permission policies).</li>
<li style="font-weight: 400;" aria-level="1">Resource: Represents the resources to which the actions apply. (Optional in case of resource-based policies).</li>
<li style="font-weight: 400;" aria-level="1">Condition: Specifies a condition that must be fulfilled for the statement to apply.</li>
</ul>
AWS supports a variety of policies, but as previously stated, the above structure applies to all of them.

AWS supports six types of policies, namely:
<ul>
<li style="font-weight: 400;" aria-level="1">Identity-based policies</li>
<li style="font-weight: 400;" aria-level="1">Resource-based policies</li>
<li style="font-weight: 400;" aria-level="1">Permissions boundaries</li>
<li style="font-weight: 400;" aria-level="1">Organizations SCPs</li>
<li style="font-weight: 400;" aria-level="1">ACLs</li>
<li style="font-weight: 400;" aria-level="1">Session policies</li>
</ul>
This article will primarily focus on identity-based policies and how to create them using Terraform. These learnings can be extended to any other type of AWS-supported policy.
If you would like to set up the ability to sign in to your Spacelift account using a SAML 2.0 integration with AWS IAM Identity Center (formerly known as AWS SSO), check out our <a href="https://docs.spacelift.io/integrations/single-sign-on/aws-iam-identity-saml-setup-guide.html" target="_blank" rel="noopener">AWS IAM Identity SAML Setup Guide</a>. Or <a href="https://docs.spacelift.io/integrations/cloud-providers/aws" target="_blank" rel="noopener">learn more about Spacelift integration with AWS</a>, with our new <a href="https://spacelift.io/blog/spacelift-cloud-integrations" target="_blank" rel="noopener">Cloud Integrations section</a> and our update to support account-level AWS integrations.

<h3>Prerequisites</h3>
<ol>
<li style="font-weight: 400;" aria-level="1">An AWS admin user with the ability to create resources on AWS.</li>
<li style="font-weight: 400;" aria-level="1">At least one S3 bucket that can be used to check whether a policy with the <code>s3:ListAllMyBucket</code> action is properly associated with a user.</li>
<li style="font-weight: 400;" aria-level="1">Configuring the AWS credentials to be used with terraform.</li>
</ol>
Once the prerequisites are out of the way, we will quickly understand what identity-based policies are before diving into creating one with Terraform.

Identity-based policies govern what actions an identity (User, Group, <a href="https://spacelift.io/blog/aws-iam-roles" target="_blank" rel="noopener">Role</a>) can perform on which resources and under what conditions.
A simple identity-based policy, such as the one shown below, allows the identity to list all S3 buckets.

Now that we have understood what identity-based policies are, we will start by creating a user that will be the subject of our hands-on exercises on creating and managing IAM policies using Terraform.
<h3>Creating a user using terraform</h3>
<ol>
<li style="font-weight: 400;" aria-level="1">Create a file with the name as <code>main.tf</code> in an empty folder.</li>
<li style="font-weight: 400;" aria-level="1">Copy and paste the below <a href="https://spacelift.io/blog/terraform-commands-cheat-sheet" target="_blank" rel="noopener">HCL code</a> snippet within this file. 
This code snippet creates an AWS user and outputs a randomly generated password that can be used to log in to AWS.</li>
</ol>

<ol start="3">
<li>Initialize the backend and the required provider plugins using the <a href="https://spacelift.io/blog/terraform-init" target="_blank" rel="noopener"><code>terraform init</code></a> command.</li>
</ol>

terraform-iam-policy-terraform-init

terraform iam policy terraform init

<ol start="4">
<li style="font-weight: 400;" aria-level="1">Verify the changes by running the <a href="https://spacelift.io/blog/terraform-plan" target="_blank" rel="noopener"><code>terraform plan</code></a> command.</li>
</ol>

<ol start="5">
<li>After you’ve verified the plan, use the <code>terraform apply</code> command to deploy the changes to AWS. 
A successful run would produce the following output, which includes a randomly generated password that can be used to log into AWS.</li>
</ol>

<ol start="6">
<li>Let’s log in to AWS with the newly created username and password and try to access the s3 dashboard.</li>
</ol>

A new user without any policies attached has no access to any AWS resources because AWS denies everything by default and the <code>spacelift-user</code> user currently has no permissions attached to it. 
 
Note: The S3 console might not necessarily show a 403 error but would definitely not show any existing buckets due to the lack of permissions.

terraform-iam-policy-s3-console

terraform iam policy s3 console

<ol start="7">
<li>Try accessing any other resources on AWS and verify that the new user in fact has no access.</li>
</ol>

terraform-iam-policy-no-access

terraform iam policy no access

Now that we’ve got everything in order let’s create a policy that allows our newly created user to list all <a href="https://spacelift.io/blog/terraform-aws-s3-bucket" target="_blank" rel="noopener">S3 buckets</a> using Terraform.

An inline policy is one that is embedded as part of the identity and has no independent existence. When you delete the identity, you also delete the inline policy.
Inline policies can be created and attached to a user using the <code>aws_iam_user_policy</code> resource. Let’s get started.
<ol>
<li>Create an <code>aws_iam_user_policy</code> resource inside the <code>main.tf</code> file, which attaches an inline policy to the <code>spacelift-user</code>. 
</li>
</ol>

Note: Please keep in mind that we used the HEREDOC syntax here to represent the policy. This allows us to use the JSON policy directly in Terraform without any modifications.
<ol start="2">
<li>Run the <code>terraform plan</code> command to verify the changes.</li>
</ol>

Note: It&#8217;s worth noting that, despite using the HEREDOC syntax to represent the policy, Terraform internally uses the <code>jsonencode</code> function, which is visible in the plan. In the following section, we will go over the <code>jsonencode</code> function in greater detail.
<ol start="3">
<li>Finally, run the <code>terraform apply</code> command to deploy the changes which will attach an inline policy to the <code>spacelift-user</code>. 
</li>
</ol>

terraform-iam-policy-inline-policy

terraform iam policy inline policy

<ol start="4">
<li>Great! Now let&#8217;s again log in to AWS using the <code>spacelift-user</code> and check the S3 console.</li>
</ol>

terraform-iam-policy-inline-policy-s3-console

terraform iam policy inline policy s3 console

Awesome! The policy took effect to grant the <code>spacelift-user</code> permission to use the  <code>s3:ListAllMyBuckets</code> action.
In this exercise, we created an inline policy and represented it using the HEREDOC syntax. In the following section, we will look at all of the different ways to represent policies in Terraform.

Terraform provides multiple ways to represent a policy in HCL, those are:
<ol>
<li aria-level="1">HEREDOC syntax</li>
<li aria-level="1">jsonencode function to convert a policy into JSON</li>
<li aria-level="1">file function to load a policy from a JSON file</li>
<li aria-level="1">aws_iam_policy_document data resource</li>
</ol>
We have already seen the HEREDOC syntax in the earlier tutorial. In the later sections, we will explore all the others with standalone policies.
Note: It is important to remember that any of these methods can be used in place of the others without affecting the <code>terraform plan</code>.

<ul>
<li><a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Deploy your Infrastructure in CI/CD using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Deploy your Infrastructure in CI/CD using Terraform</a></li>
<li><a href="https://spacelift.io/blog/ci-cd-tools" target="_blank" rel="noopener">Top Most Useful CI/CD Tools for DevOps</a></li>
<li><a href="https://spacelift.io/blog/how-specialized-solution-can-improve-your-iac" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;10 Ways to Improve Your Infrastructure as Code&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">10 Ways to Improve Your Infrastructure as Code</a></li>
</ul>

Before we jump into the hands-on, let’s understand what standalone policies are and why they are important.
Take a moment to ponder upon the following questions:
<ol>
<li style="font-weight: 400;" aria-level="1">How can we share the same inline policy with multiple other identities? Should we re-create the inline policy for each identity?</li>
<li style="font-weight: 400;" aria-level="1">How can we delete an identity without also deleting the policies associated with it?</li>
</ol>
This is where standalone policies come into the picture. They can exist independently of any identity or resource and can be attached to any number of identities without recreating them.

<h3>Using the jsonencode function</h3>
<ol>
<li style="font-weight: 400;" aria-level="1">In the <code>main.tf</code> file, create an <code>aws_iam_policy</code> resource, which creates a standalone policy with the name <code>S3ReadOnlyPolicy</code> and uses the <code>jsonencode</code> function to represent the policy.</li>
</ol>

<ol start="2">
<li>Run the <code>terraform plan</code> command to verify the changes.</li>
</ol>

<ol start="3">
<li style="font-weight: 400;" aria-level="1">Finally, run the <code>terraform apply</code> command to deploy the changes to AWS.</li>
<li style="font-weight: 400;" aria-level="1">Check the IAM console to verify if the policy was created. Take note that the policy is not associated with any identity yet.</li>
</ol>

terraform-iam-policy-console

terraform iam policy console

<h3>Attaching a standalone policy to a user</h3>
A standalone policy can be attached to a user using the <code>aws_iam_user_policy_attachment</code> resource.
Please keep in mind that this resource is exclusively for attaching a policy to a user, unlike the <code>aws_iam_user_policy</code> resource, which both creates and attaches an inline policy to a user.
Let’s proceed to attach the newly created policy to the <code>spacelift-user</code> user.
<ol>
<li>Create an <code>aws_iam_user_policy_attachment</code> resource, which has two attributes <code>user</code> and <code>policy_arn</code>. Set the <code>user</code> attribute as the name of the <code>spacelift-user</code> user and <code>arn</code> as the arn of the standalone policy we created. 
</li>
</ol>

<ol start="2">
<li>Run the <code>terraform plan</code> to verify the changes.</li>
</ol>

<ol start="3">
<li>Run the <code>terraform apply</code> command and check if the policy was attached to the <code>spacelift-user</code> in the AWS IAM console.</li>
</ol>

terraform-iam-policy-console-policy-attached

terraform iam policy console policy attached

Awesome! Let’s now see how to achieve the same results using the <code>file</code> function.
<h3>Using the file function</h3>
<ol>
<li>Create a <code>json</code> file and add the below policy to it. 
</li>
</ol>

<ol start="2">
<li>When creating the <code>aws_iam_policy</code> resource use the <code>file</code> function to load the policy from the json file.</li>
</ol>

Rest of the steps are the same as in the previous example of creating the policy using the <code>jsonencode</code> function.
<h3>Using aws_iam_policy_document data resource</h3>
The aws_iam_policy_document data resource generates policies in JSON format that can be used with the <code>aws_iam_policy</code> resource or with any resource that expects a JSON policy.
Create an <code>aws_iam_policy_document</code> data resource in the main.tf file.

<ol start="2">
<li>As usual, run the <code>terraform plan</code> command.</li>
</ol>

terraform-iam-policy-plan

terraform iam policy plan

Take a moment to consider why no changes are being reflected 🤔 
 
This is due to the fact that <code>aws_iam_policy_document</code> is a data resource. Data resources are used to read information and have no effect on the infrastructure. 
 
This implies that we must use this data resource as a reference somewhere in order to actually create the generated policy.
<ol start="3">
<li>Create a new <code>aws_iam_policy</code> resource that refers to the data resource <code>aws_iam_policy_document</code>.</li>
</ol>

<ol start="4">
<li>Again, run the <code>terraform plan</code> command. This time you should see some changes that should match the below output.</li>
</ol>

<ol start="5">
<li>Finally, run the <code>terraform apply</code> command to create the <code>S3WriteOnlyPolicy</code> standalone policy.</li>
</ol>

terraform-iam-policy-console-new-policy

terraform iam policy console new policy

We saw a variety of ways to represent policies using Terraform, which is fantastic. But which one should we use and when? 🤔

In general, you are free to choose any way that you like to represent policies.
However, the <code>aws_iam_policy_document</code> data resource approach is recommended because it allows Terraform to validate any errors without having to <code>apply</code> the changes, which is not possible in other cases.
Let us see how this works in practice.
Below are two policies, one with the HEREDOC syntax and one with the <code>jsonencode</code> function. Both of these policies have a typo in the <code>Action</code> key, which reads <code>MistakeInAction</code> at the moment.

Let’s see if Terraform can detect this error.
To validate the HCL file, use the <a href="https://spacelift.io/blog/terraform-validate" target="_blank" rel="noopener"><code>terraform validate</code></a> command.

terraform-iam-policy-terraform-validate

terraform iam policy terraform validate

Terraform couldn’t detect any problems with either of the policies. 
Let us now try the same thing with <code>aws_iam_policy_document</code> data resource.

Run the <code>terraform validate</code> command again and check if Terraform is able to find the typo.

terraform-iam-policy-terraform-validate-typo

terraform iam policy terraform validate typo

Terraform was able to detect the typo, which is a huge benefit for developers.
Aside from that, when using the <code>aws_iam_policy_document</code> data resource, you can benefit from the auto-completion feature of IDEs, which leads to a better development experience 🙂
This brings us to the final section of the exercise, where we will learn how to associate an AWS managed policy with a user.

AWS managed policies are created and managed by AWS for the most common use cases.
Let’s look at how to attach an AWS managed policy to a user.
<ol>
<li style="font-weight: 400;" aria-level="1">Because AWS managed policies are already present on AWS, we will read the policy from AWS using the data resource <code>aws_iam_policy</code>.</li>
</ol>

<ol start="2">
<li>Let us attach this policy to the <code>spacelift-user</code> user as we did before using the reference to the data resource.</li>
</ol>

Take note that we can directly hardcode the arn of the policy as well instead of using the data resource.
<ol start="3">
<li style="font-weight: 400;" aria-level="1">Run the <code>terraform plan</code> command and verify the planned changes.</li>
</ol>

<ol start="4">
<li>Run the <code>terraform apply</code> command to deploy the changes and verify them in the IAM console.</li>
</ol>

terraform-iam-policy-terraform-apply

terraform iam policy terraform apply

Congratulations, we learned how to use Terraform to manage inline, standalone, and AWS managed policies and numerous ways to represent them. 🎉
As previously stated, all of these learnings can be extended to any other policy type on AWS. As an example, creating resource-based policies using Terraform is shown below.

The method for creating policies remains the same as with identity-based policies; the only difference is the resource used to attach the policy to an entity.

In this article, we learned what IAM policies are, numerous ways to create and manage them using Terraform and how to use them in practice.
AWS IAM policies lie at the heart of access management, and therefore it is critical to master them to secure access control on AWS.

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Build more complex workflows based on Terraform using policy as code, programmatic configuration, context sharing, drift detection, resource visualization and many more.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Build more complex workflows based on Terraform using policy as code, programmatic configuration, context sharing, drift detection, resource visualization and many more.

Initialize/ plan/ apply your IaC, manage modules, state, and more. 

frame-8479

Frame 8479

How to Build AWS VPC using Terraform - Step by Step

Managing AWS Security Groups Through Terraform

Infrastructure Problems that Spacelift Solves

See what IAM Policies are, numerous ways to create and manage IAM Policy using Terraform, and how to use them in practice.

149-create-aws-iam-policy-with-terraform

How to Create AWS IAM Policy Using Terraform (Tutorial)

Managing AWS Security Groups through Terraform, Network Security and more

As per the <a href="https://www.techrepublic.com/article/cloud-misconfigurations-cost-companies-nearly-5-trillion/" target="_blank" rel="nofollow noopener">Divvy Cloud report</a>, cloud misconfiguration breaches cost companies around the world an estimated 5 trillion in 2018 and 2019.
Many organizations fail to install adequate security measures, which are critical for protecting the cloud against breaches and attacks. Having an understanding of security and installing the appropriate security measures minimizes threats and keeps the cloud secure.
Security is a <a href="https://aws.amazon.com/compliance/shared-responsibility-model/" target="_blank" rel="nofollow noopener">shared responsibility</a> between AWS and the customer, where AWS is responsible for the security of the cloud, and the customers are responsible for security in the cloud.
AWS provides many security features, and VPCs are one of the most important. They let us create an isolated private cloud within the public cloud to protect resources from direct access.
In this article, we will go through:
<ol>
<li><a href="https://spacelift.io/blog/terraform-security-group#virtual-private-cloud-vpc">Virtual Private Cloud (VPC)</a></li>
<li><a href="https://spacelift.io/blog/terraform-security-group#security-groups">Security Groups</a></li>
<li><a href="https://spacelift.io/blog/terraform-security-group#creating-our-own-security-group">Creating Our Own Security Group</a></li>
<li><a href="https://spacelift.io/blog/terraform-security-group#important-characteristics-of-security-groups">Important Characteristics of Security Groups</a></li>
<li><a href="https://spacelift.io/blog/terraform-security-group#managing-security-groups-through-terraform">Managing Security Groups Through Terraform</a></li>
<li><a href="https://spacelift.io/blog/terraform-security-group#how-to-use-existing-security-groups" rel="">How to Use Existing Security Groups</a></li>
<li><a href="https://spacelift.io/blog/terraform-security-group#applications-of-security-groups">Applications of Security Groups</a></li>
<li><a href="https://spacelift.io/blog/terraform-security-group#security-groups-vs-nacl">Security groups vs. NACL</a></li>
<li><a href="https://spacelift.io/blog/terraform-security-group#security-groups-and-compliance">Security groups and compliance</a></li>
</ol>

Virtual Private Cloud (VPC) is one of the fundamental building blocks of AWS. VPCs are private virtual networks that resemble traditional networks and provide secure, isolated environments to host and protect AWS resources like EC2 instances.
Security for the resources within the VPC can be configured using security groups and a network access control list (NACL).
Let’s understand how we can use them to protect resources within VPC and improve the overall security posture.
Read more about <a href="https://spacelift.io/blog/terraform-aws-vpc" target="_blank" rel="noopener">how to build AWS VPC using Terraform</a>.

<h3>What are security groups?</h3>
Security groups are one of the most important pillars that support VPC security. They are software-defined network firewalls defined inside a VPC that allow or deny traffic to resources based on the inbound and outbound rules.
They support stateful Layer 3 (<a href="https://en.wikipedia.org/wiki/Network_layer" target="_blank" rel="nofollow noopener">Network layer</a>) and Layer 4 (<a href="https://en.wikipedia.org/wiki/Transport_layer" target="_blank" rel="nofollow noopener">Transport layer</a>) filtering capabilities.
At the resource level, security groups work to secure the flow of traffic allowed to leave and reach them. They serve as the final line of defense.
They are most commonly associated with EC2 instances, but their applications are not limited to them. They are widely used with a variety of other services such as CodeBuild, EKS, and so on.
The figure below shows how security groups secure traffic flow to and from resources.

security-group

security group

<h3>What makes up a security group?</h3>
A security group is composed of rules. A rule consists of the following components:
<ul>
<li style="font-weight: 400;" aria-level="1">Protocol</li>
<li style="font-weight: 400;" aria-level="1">Port range</li>
<li style="font-weight: 400;" aria-level="1">Source</li>
<li style="font-weight: 400;" aria-level="1">Destination</li>
</ul>
A security group has separate rule sets for incoming and outgoing traffic.
An inbound rule consists of a protocol, a port(s), and a source.
For example, the below rule allows all traffic to reach resources associated with the security group.

security-group-inbound-rule

security group - inbound rule

An outbound rule consists of a protocol, a port(s), and a destination.
For example, the below rule allows all traffic to leave resources associated with the security group.

security-group-outbound-rule

security group - outbound rule

Let us inspect an existing security group to understand the rules better.
<h3>Exploring the default security group</h3>
Let us inspect the default security group to understand what the rules look like.
<ol>
<li style="font-weight: 400;" aria-level="1">Open the EC2 dashboard.</li>
</ol>

security-groups-ec2-dashboard

Security groups - EC2 dashboard

<ol start="2">
<li style="font-weight: 400;" aria-level="1">Click on “Security Groups” in the left nav under “Networks and Security”.</li>
<li style="font-weight: 400;" aria-level="1">Explore the inbound and outbound rules for the default security group and other details.</li>
</ol>

default-security-group-details

default security group details

<h3>Key observations</h3>
<ol>
<li style="font-weight: 400;" aria-level="1">The default security group is associated with a VPC. The VPC ID can be found under the “VPC ID” tab.</li>
<li style="font-weight: 400;" aria-level="1">By default, the security group has one inbound and one outbound rule.</li>
<li style="font-weight: 400;" aria-level="1">All traffic can reach the associated resources thanks to the inbound rule.</li>
<li style="font-weight: 400;" aria-level="1">The outbound rule allows all traffic destined for everywhere (denoted by 0.0.0.0) to leave the associated resources.</li>
</ol>
We can always create new security groups with whatever rules we need. In the following section, we will learn how to do so.
Note: Please keep in mind that we can only create <a href="https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html" target="_blank" rel="nofollow noopener">2500 security groups per region</a>.
Security consideration: Keep the number of security groups to a minimum to avoid making mistakes when managing too many security groups.

<ol>
<li style="font-weight: 400;" aria-level="1">Navigate to the “Security groups” panel.</li>
<li style="font-weight: 400;" aria-level="1">Click the “Create security group” button to create a new security group.</li>
<li style="font-weight: 400;" aria-level="1">Give the security group a name as <code>web-server-sg</code> and a description as <code>Allow SSH to the web server.</code></li>
</ol>
Note: The name of the security group has to be unique within the VPC and cannot exceed 255 characters. Another thing to keep in mind is that the security group’s name cannot start with <code>sg-</code>.

create-security-group

create security group

<ol start="4">
<li style="font-weight: 400;" aria-level="1">For the time being, do not add any inbound rules and proceed with the suggested outbound rule, which allows all traffic to flow everywhere (0.0.0.0).</li>
<li style="font-weight: 400;" aria-level="1">Finally, click on the “Create security group” button.</li>
</ol>

first-security-group

first security group

Congratulations! You have created your first security group 🎉
Security groups are only effective when attached to an AWS resource. Next, we will attach it to an EC2 instance to see it in action.

<h3>Attaching Security Groups to AWS Resources</h3>
In this tutorial, we will create an EC2 instance and attach the newly created security group to it. We will then try to connect to this instance via SSH.
Attaching a security group to an EC2 instance
<ol>
<li style="font-weight: 400;" aria-level="1">Navigate to the EC2 dashboard and click on the “Launch instances” button.</li>
<li style="font-weight: 400;" aria-level="1">Assign the instance a name as <code>web-server</code>.</li>
<li>Choose any Amazon Machine Image (AMI) and architecture. 
</li>
</ol>

attaching-a-security-group-to-an-ec2-instance

attaching a security group to an EC2 instance

<ol start="4">
<li style="font-weight: 400;" aria-level="1">Select the instance type as <code>t2.nano</code>.</li>
<li style="font-weight: 400;" aria-level="1">Create a new key and name it <code>web-server-key</code>. The key will be downloaded automatically. We will use it later to SSH into our instance.</li>
<li style="font-weight: 400;" aria-level="1">Make sure to select the default VPC and a public subnet under “Network settings”</li>
<li style="font-weight: 400;" aria-level="1">Enable the “Auto-assign public IP” option.</li>
</ol>

security-groups-auto-assign-public-ip

security groups auto-assign public IP

<ol start="8">
<li>Under the “Firewall (security groups)” tab, choose the option “Select existing security group” and select the security group we created earlier named <code>web-server-sg</code>. 
</li>
</ol>

firewall-security-groups

firewall (security groups)

<ol start="9">
<li>Leave all other options unchanged and launch the instance.</li>
</ol>
With the security group now attached to our EC2 instance, let us try to connect to it via SSH.
<h3>Connecting to the EC2 instance via SSH</h3>
Open the details of the newly created EC2 instance and copy the public IP address.

connecting-to-the-ec2-instance-via-ssh

connecting to the EC2 instance via SSH

<ol start="2">
<li>Open a terminal and modify the permissions for the <code>web-server-key.pem</code> file we downloaded earlier to allow read and write permission for the owner using the below command.</li>
</ol>

<ol start="3">
<li>Next, let us try to connect to the EC2 instance with our <code>pem</code> key using the below command.</li>
</ol>

After a while, the SSH request times out with the following error.

security-groups-ssh-request-time-out

security groups ssh request time out

This is expected as our request is being blocked by the security group that we have attached to the EC2 resource. Remember that the security group we created does not yet have any inbound rules.
We can resolve this issue by configuring an inbound rule to allow incoming traffic on port 22 (SSH) from our IP address.
<h3>Adding rules to an existing security group</h3>
<ol>
<li style="list-style-type: none;">
<ol>
<li style="font-weight: 400;" aria-level="1">Open the security groups panel and select the security group attached to our EC2 instance named <code>web-server-sg</code>.</li>
<li style="font-weight: 400;" aria-level="1">Under the “Inbound rules” tab, click the “Edit inbound rules” button, followed by the “Add rule” button.</li>
</ol>
</li>
</ol>

adding-rules-to-an-existing-security-group

Adding rules to an existing security group

<ol start="3">
<li style="font-weight: 400;" aria-level="1">Choose SSH as the protocol and My IP as the source (allowing SSH through your IP only). Optionally, add the description as Allow SSH from my network.</li>
</ol>
Security consideration: For protocols such as SSH and RDP, only specific IP addresses should be allowed. Specifying 0.0.0.0 opens access to everyone, which is a security risk.
It should be noted that the source and destination do not have to be IP addresses. Other security groups or prefixes can also be used.

security-groups-edit-inbound-rules

security groups edit inbound rules

<ol start="4">
<li>Save the rule and attempt to reconnect to the EC2 instance via SSH.</li>
</ol>
<h3>Reconnecting to the EC2 instance via SSH</h3>
We’ll use the same SSH command as before to see if we can connect after adding the inbound rule to the security group.

reconnecting-to-the-ec2-instance-via-ssh

reconnecting to the EC2 instance via SSH

Awesome! We successfully connected when we added an inbound rule for SSH.
You can also try switching to a different network to confirm that the security group is blocking the traffic.
In the following sections, we’ll go over some of the key characteristics of security groups, as well as how to manage security groups with Terraform.

<ul>
<li><a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale" target="_blank" rel="noopener">5 Ways to Manage Terraform at Scale</a></li>
<li><a href="https://spacelift.io/blog/terraform-automation" target="_blank" rel="noopener">How to Automate Terraform Deployments and Infrastructure Provisioning</a></li>
<li><a href="https://spacelift.io/blog/infrastructure-problems-that-spacelift-solves" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Common Infrastructure Challenges and How to Solve Them&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Common Infrastructure Challenges and How to Solve Them</a></li>
</ul>

When working with security groups, it is critical to be aware of a few key characteristics.
<ul>
<li style="font-weight: 400;" aria-level="1">There can only be “allow rules” in security groups, not “deny rules” 
AWS blocks all traffic to resources associated with security groups by default. The allow rules filter and allow traffic that is relevant.</li>
<li style="font-weight: 400;" aria-level="1">A resource can have multiple security groups attached to it 
When multiple security groups are attached to a single resource, all rules are aggregated to form a single rule set that determines whether or not the traffic is allowed.</li>
<li style="font-weight: 400;" aria-level="1">Security groups are stateful 
Being stateful means that with an outbound rule that allows traffic to flow out of the resource also allows the response to reach back regardless of whether an inbound rule is present or not. To better understand this analogy, please refer to <a href="https://stackoverflow.com/questions/53622935/what-is-meant-by-security-groups-are-stateful" target="_blank" rel="nofollow noopener">this</a> StackOverflow answer.</li>
</ul>

Handling <a href="https://spacelift.io/blog/5-ways-to-manage-terraform-at-scale">infrastructure at scale</a> manually is an extremely difficult task. Terraform and other IaC tools make it much easier for us to create and manage infrastructure at scale thanks to their declarative approach and state management.
Let’s look at how we can easily create and manage security groups with Terraform.
You will need to have <a href="https://spacelift.io/blog/how-to-install-terraform" target="_blank" rel="noopener">Terraform </a><a href="https://spacelift.io/blog/how-to-install-terraform" target="_blank" rel="noopener">installed</a> on your machine and your AWS credentials configured in the shell to follow this tutorial.
Note: We will keep using the same <code>main.tf</code> file and make changes to it.
You can also take a look at how platforms like <a href="https://spacelift.io" target="_blank" rel="noopener">Spacelift</a> can help you and your organization fully manage cloud resources within minutes. Spacelift is a CI/CD platform for infrastructure-as-code that supports tools like Terraform, Pulumi, Kubernetes, and more. For example, it enables <a href="https://docs.spacelift.io/concepts/policy/" target="_blank" rel="noopener">policy-as-code</a>, which lets you define policies and rules that govern your infrastructure automatically. You can even invite your security and compliance teams to collaborate on and approve certain workflows and policies for parts that require a more manual approach.
<h3>Creating a security group</h3>
We’ll set up a security group with an inbound rule that allows only HTTPS traffic and an outbound rule that allows all traffic.
<ol>
<li style="font-weight: 400;" aria-level="1">Create a file with the name as <code>main.tf</code> and copy the below HCL code snippet to this file.</li>
</ol>

<ol start="2">
<li>Run the <a href="https://spacelift.io/blog/terraform-plan" target="_blank" rel="noopener"><code>terraform plan</code></a> command in the terminal where the file resides. The plan should look something like the one below.</li>
</ol>

<ol start="3">
<li>Run <a href="https://spacelift.io/blog/terraform-apply" target="_blank" rel="noopener"><code>terraform apply</code></a> to deploy the security group once the plan has been verified.</li>
</ol>

creating-a-security-group-terraform

creating a security group - terraform

Great work! We have successfully deployed a new security group through Terraform 🎉
<h3>Adding rules to security groups</h3>
There are two ways to add rules to a security group in Terraform:
<ul>
<li aria-level="1">Inline rules: Rules defined with the <code>aws_security_group</code> terraform resource</li>
</ul>
<ul>
<li aria-level="1">Standalone rules: Rules defined separately using the <code>aws_security_group_rule</code> terraform resource</li>
</ul>
Note: Keep in mind that these are the two ways to add rules to a security group through Terraform, which at the end of the day, correspond to the same thing on AWS.
Terraform advises that inline rules should not be used in conjunction with standalone rules. Using both of them together can cause conflicts.

adding-rules-to-security-groups-terraform

adding rules to security groups - terraform

We’ll use standalone rules as we’ve already seen inline rules when creating a security group. Before adding any new rules, we will first convert the existing inline rules to standalone rules in the same file.
<h3>Converting inline security group rules to standalone rules</h3>
<ol>
<li style="font-weight: 400;" aria-level="1">Set the ingress and egress attribute to an empty array. 
Note: Terraform doesn’t detect any changes if we completely remove ingress and egress attributes from <code>aws_security_group</code>. This issue can be tracked <a href="https://github.com/hashicorp/terraform-provider-aws/issues/4399" target="_blank" rel="nofollow noopener">here</a>. The workaround is to set the ingress and egress attributes to <code>[]</code>.</li>
</ol>

<ol start="2">
<li style="font-weight: 400;" aria-level="1">Run the <code>terraform plan</code> command and then the <code>terraform apply</code> command to remove the existing rules before adding any standalone rules to avoid any conflicts.</li>
<li style="font-weight: 400;" aria-level="1">Add standalone security group rules for HTTP ingress and all egress. Do not forget to remove the inline ingress and egress attributes from the <code>aws_security_group</code> resource to avoid any conflicts.</li>
</ol>

<ol start="4">
<li>Run <code>terraform plan</code> again. The changes should reflect the two rules being added.</li>
</ol>

<ol start="5">
<li>Finally run the <code>terraform apply</code> command to deploy the changes.</li>
</ol>

converting-inline-security-group-rules-to-standalone-rules-terraform

Converting inline security group rules to standalone rules - terraform

Awesome! We successfully added standalone rules to our security group.
<h3>Attaching a security group to an EC2 resource</h3>
In this tutorial, we will learn how to attach a security group to an EC2 instance.
<ol>
<li>Add Terraform resource to spin up an EC2 instance. 
</li>
</ol>

Note: We do need to create the <code>web-server-key</code> again as we had previously created it through the console.
<ol start="2">
<li>Add key <code>vpc_security_group_ids</code>  with the value as <code>[aws_security_group.web_server_sg_tf.id]</code> to attach the previously created security group to this EC2 instance.</li>
</ol>

<ol start="3">
<li style="font-weight: 400;" aria-level="1">Run the <code>terraform plan</code> command and review the changes before running the <code>terraform apply</code> command.</li>
<li style="font-weight: 400;" aria-level="1">Check the AWS console to verify if the security group was attached successfully.</li>
</ol>

Deep dive into AWS Security Groups with Terraform. Learn about managing them, network security and more.

Omkar Birade

CI/CD Security: What is It, Risks & Best Practices

Get the newsletter

Recent posts

How to Create AWS IAM Policy Using Terraform (Tutorial)

Managing AWS Security Groups Through Terraform

AWS IAM Roles – Everything You Need to Know & Examples

YAML Tutorial : A Complete Language Guide with Examples

Terraform .tfvars files: Variables Management with Examples