Christophe fell in love with technology at a young age the moment he wrote his first lines of code. He launched Cybr, a training platform and community, to help close the cybersecurity skills gap and help organizations secure their cloud resources.

All posts by Christophe Limpalair - Infrastructure as Code Expert

Christophe Limpalair

Author title

Title category

Get all the latest content on Infrastructure as Code, DevOps, Spacelift features & news directly to your inbox!

Level up your IaC knowledge. Stay up to date with educational devops guides & news in infrastructure as code, cloud & more!

Managing Infrastructure at Scale | Spacelift Blog

applepodcasts

spotify

missionctrl

Welcome to missionCTRL, a show dedicated to exploring the frontiers of modern DevOps. Listen (or watch) as we sit down with the practitioners who are pushing the boundaries of development

DevSecOps: Automated and Manual Code Testing

General | Spacelift Blog

General

In <a href="https://spacelift.io/blog/what-is-devsecops" target="_blank" rel="noopener">this prior article</a>, we talked about what DevSecOps is and why it’s important. The main goal that we discussed was to shift security left, which is all about including security from the very beginning so that problems are found as early in the process as possible. To expand on that article, we’re going to take a closer look at a very important part of the shift-left strategy: code testing.
Code testing is one of the first lines of defense against introducing vulnerabilities, leaking secrets into production or public sources, and overall bugs that, over time, slow down development velocity and end up costing your organization.
A challenge with code testing is that it can end up slowing down your development team, or it can end up being a waste of time if it’s not properly implemented.
In this article, we’re going to explain what code testing is, why you should not delay implementing it any longer, and approaches that you can take to make the transition easier based on your organization’s needs.
<ol>
<li aria-level="1"><a href="https://spacelift.io/blog/devsecops-code-testing#devsecops-automated-and-manual-code-testing">DevSecOps: Automated and Manual Code Testing</a></li>
<li aria-level="1"><a href="https://spacelift.io/blog/devsecops-code-testing#source-code-versioning">Source code versioning</a></li>
<li aria-level="1"><a href="https://spacelift.io/blog/devsecops-code-testing#implementing-standards-and-code-reviews">Standards and code reviews</a></li>
<li aria-level="1"><a href="https://spacelift.io/blog/devsecops-code-testing#linting">Lint Scanning</a></li>
<li aria-level="1"><a href="https://spacelift.io/blog/devsecops-code-testing#static-application-security-testing-sast">SAST (Static Application Security Testing)</a></li>
<li aria-level="1"><a href="https://spacelift.io/blog/devsecops-code-testing#secrets-scanning">Secrets Scanning</a></li>
<li aria-level="1"><a href="https://spacelift.io/blog/devsecops-code-testing#infrastructure-as-code-iac-scanning-and-policy-as-code">Infrastructure as Code (IaC) Scanning and Policy as Code</a></li>
<li aria-level="1"><a href="https://spacelift.io/blog/devsecops-code-testing#container-image-scanning">Container Image Scanning</a></li>
<li aria-level="1"><a href="https://spacelift.io/blog/devsecops-code-testing#software-composition-analysis-sca">SCA (Software Composition Analysis</a></li>
<li aria-level="1"><a href="https://spacelift.io/blog/devsecops-code-testing#prioritization-and-conclusion">Prioritization</a></li>
</ol>
These are all topics we will be talking about in this article since they should be part of our overall security strategy.
At the end of the day, code testing is not a one-size-fits-all, and it requires both automated and manual approaches, so let’s take a look at how you can implement it successfully.

The primary goal of any security initiative is to bring down the risk for that particular organization. Every organization faces unique risks, so while not everyone should implement automated and manual code testing in the exact same ways, there are implementations that most should at least consider.
Let’s bring back up our handy-dandy DevSecOps lifecycle cheat sheet from our prior article to explain what we mean.

devsecops-lifecycle-3

devsecops lifecycle

After we’ve gone through the planning stage, it’s time for developers to start writing code or for our infrastructure engineers to start configuring cloud resources to support those applications. This is when we enter the code section of our DevSecOps lifecycle.
If you’re starting from scratch, though, you’re not going to be able to realistically implement all of this in one fell swoop. You’re going to have to prioritize and set a roadmap that the rest of the organization will have to agree on.
With that said, one of the most important steps will be to implement source code versioning, which is why we’ll start by discussing that topic. If you or your organization is not already using source code versioning, that will be one of the most impactful changes you can implement starting today. Almost everything else we discuss in this article builds on top of this foundation.
Next, we will talk about code reviews, a manual approach to finding issues with committed code. While automation is great and should certainly be part of your overall strategy, it can’t completely replace the need for manual reviews, and we’ll be explaining the why and the how.
We’ll then talk about linting, which is often considered to be a cousin of SAST that focuses on a much more narrow scope — and there’s a reason why we’re talking about it separately in this article.
After that, we’ll talk about how we can automate testing using Static Application Security Testing, or SAST. SAST evokes mixed feelings among developers and security professionals, and over 60% of developers who responded to <a href="https://about.gitlab.com/developer-survey/" target="_blank" rel="nofollow noopener">GitLab’s 2020 DevSecOps survey</a> said that they don’t run any static application security testing.

gitlab-devsecops-survey

gitlab devsecops survey

Why is that? There are many reasons, but one of the primary reasons is that implementing SAST is not easy. It’s not just a matter of picking an existing tool, plugging it into an environment, and calling it a day. It takes months of effort to get everyone on the same page, perform tests, tweak settings based on those test results, etc&#8230;. until it’s an actually helpful solution and not something that gets in the developer’s way.
Because it’s often poorly implemented, which ends up producing more issues than helping, many continue to assume that it’s a problem with SAST and not with the implementation. 
Ultimately, we want SAST to help us:
<ol>
<li style="font-weight: 400;" aria-level="1">Implement early security checks</li>
<li style="font-weight: 400;" aria-level="1">Reduce human errors</li>
<li style="font-weight: 400;" aria-level="1">Help developers fix issues in less time</li>
<li style="font-weight: 400;" aria-level="1">Track valuable metrics</li>
</ol>
If it’s not accomplishing all 4 of those goals, then the implementation should be tweaked.
After talking about SAST, we’re going to explore secrets scanning, IaC scanning, and container image scanning. While many SAST solutions include some level of secrets scanning, there also exist standalone tools dedicated to that purpose. IaC scanning, or Infrastructure as Code scanning, scans infrastructure configuration files for known vulnerabilities.
If you’re using something like Terraform, Ansible, CloudFormation, Kubernetes, etc…then you can benefit from using this. If you use containers of any kind, then you can also benefit from container image scanning, which is the practice of scanning container images for known vulnerabilities. Container scanning is often considered to be a form of SCA, so let’s talk about that next.
SCA stands for Software Composition Analysis. It seems like every other month, we read another news article about how a third-party library, plug-in, or framework has been compromised, and because it’s in use by thousands of organizations, all of them are now vulnerable.
SCA aims to find issues with third-party dependencies before they become problems with our applications and before they can get exploited. That’s easier said than done, which is why we’ll spend some time learning about how SCA works and when it makes sense to implement it.
Now that we’ve gone through a quick description of each topic let’s dive right in.

You can’t secure what you don’t know you have. The same holds true for software, which is why it all starts with using a Version Control System, or VCS, to implement source code versioning.
Suppose your organization is not already using a VCS, like Git. In that case, this is one of the most impactful technological changes you could implement, and it doesn’t make a whole lot of sense to move on to the rest of this article or other security-related tasks until that’s done.
Otherwise, investing time, effort, and money into code security will have a minimal impact.
Once you’ve implemented source code versioning, you can put security findings directly in front of developers where they work (the VCS), instead of making them have to work out of yet another platform or tool.
That’s also why applications such as Spacelift integrate directly with GitHub. That way, you can control and get visibility into your <a href="https://docs.spacelift.io/concepts/stack/" target="_blank" rel="noopener">stacks</a>, <a href="https://docs.spacelift.io/vendors/terraform/module-registry" target="_blank" rel="noopener">modules</a>,<a href="https://docs.spacelift.io/concepts/policy/" target="_blank" rel="noopener"> policies</a>, and overall <a href="https://docs.spacelift.io/concepts/resources" target="_blank" rel="noopener">resources</a> from one central location.
Once you’ve implemented source code versioning, it’s time to implement standards and code reviews.

Ideally, during the planning phase of our DevSecOps pipeline, the organization has written Tech Design Documents (aka Tech Specs).
A great way to include security from the very beginning is to standardize embedding security questions directly in those documents. It’s no longer sufficient only to include designs and features for new products and services. We also need to include security questions that the engineers fill out as they write those technical documents.
Examples of security questions we could embed include:
<ul>
<li style="font-weight: 400;" aria-level="1">Does this service take any inputs from end users? If so, what are you using to protect against common injection attacks like XSS, SQL Injections, etc.?</li>
<li style="font-weight: 400;" aria-level="1">Does this service process/store any sensitive user or customer information? If so, is it encrypted?</li>
<li style="font-weight: 400;" aria-level="1">If encryption/decryption/salting/hashing are needed, which algorithms are being used?</li>
<li style="font-weight: 400;" aria-level="1">Does the service require any secrets to be used (i.e., to establish a database connection), and if so, is the secret coming from a secret management solution?</li>
<li style="font-weight: 400;" aria-level="1">How will proper monitoring and alerting be put in place for this service?</li>
</ul>

devsecops-tech-design-documents

devsecops tech design documents

These are, of course, just a handful of examples of the types of questions we should be asking and answering before work actually begins. That way, we don’t get close to the finish line before realizing that we’re using a deprecated and weak encryption algorithm. Or we totally forgot to handle user inputs properly, and we’re exposing our database. Etc.
In fact, these types of questions are multipurpose:
<ol>
<li style="font-weight: 400;" aria-level="1">They help the engineers think through security while they’re already thinking about technical design.</li>
<li style="font-weight: 400;" aria-level="1">They help the security team gather security-specific information very early on in the process.</li>
<li style="font-weight: 400;" aria-level="1">They give the security team the opportunity to highlight areas of concern or provide recommendations before a single line of code is written.</li>
<li style="font-weight: 400;" aria-level="1">They produce information and diagrams that can be fed into the organization’s Threat Models.</li>
</ol>
An added benefit is that they can then be used during code reviews and/or to set up automated tests since we can look to make sure that specific details were implemented as intended. Speaking of code reviews…
A typical flow for code reviews is to have a development team do a code review for a pull request (PR) before they merge their code. Instead of having the quality of these code reviews be completely dependent on the person doing the reviewing, it’s a good idea to devise a checklist.
That checklist will, in part, be formed by those very questions answered in the tech specs document. In addition, you can have a checklist created based on the language(s) being used, since each language will have its unique quirks.

devsecops-code-checklist

devsecops code checklist

For examples of checklists and for much more detail on how to implement secure code reviews, check out <a href="https://owasp.org/www-pdf-archive/OWASP_Code_Review_Guide_v2.pdf" target="_blank" rel="noopener">this wonderful OWASP resource</a>.
Even with checklists, though, the effectiveness of code reviews still comes down to the reviewers’ skills and knowledge and the time that they have available. If they’re rushing through the code review because they’re trying to complete other priorities, or if they don’t have the knowledge necessary, they may very well miss subtle security problems. That’s where the other recommendations we just talked about can really help.
To improve your secure code reviews, you can:
<ol>
<li style="font-weight: 400;" aria-level="1">Standardize embedding security questions in Tech &amp; Design Documents.</li>
<li style="font-weight: 400;" aria-level="1">Create code review checklists based on those security questions and on the specific language(s)</li>
<li style="font-weight: 400;" aria-level="1">Educate and train your developers on the basics of threat modeling, code reviews, and risk assessments</li>
</ol>
The reason code reviews are still important and needed, even when you can implement automated tools like SAST (which we’ll talk about in just a moment), is because tools aren’t going to find everything. Developers still need to use code reviews, which are a manual process, to find vulnerabilities in code that can’t be scanned by security testing tools, no matter how advanced. With that said, we also can’t just rely on manual code reviews, so let’s talk about automation tools.

In addition to code reviews, we’ll want to implement a linting solution.
Linting tools are often considered to be “cousins” of SAST tools because linting is the process of using static code analysis to identify stylistic and programmatic errors in code.
That makes a lint tool a basic static code analyzer, and while they can be used standalone, they can also often be integrated directly in SAST tools.
Linting is an important step because it helps reduce the number of basic programmatic errors that could result in bugs, and it helps reduce the number of stylistic errors that improve the overall health of your codebase.
That means linting would happen even before code reviews and testing, which helps alleviate the load on reviewers and reduces the likelihood of low-hanging fruit being present during testing.
We’re talking about linting separately from SAST because linting is easier and faster to implement. It often just works out of the box with only a few tweaks necessary, which is definitely not the case for SAST. Instead of waiting to receive linting benefits until SAST tooling has been thoroughly tweaked and tested, you can implement an effective linting solution this week.
Linting is especially useful for languages like Python and JavaScript. Since they aren’t compiled, they’re considered to be dynamically typed languages. That means they don’t usually enforce strict rules, which can make it easier for developers to deviate from language-specific best practices.
Linters can be directly integrated as plugins in your favorite IDE (Integrated Development Environment), like Visual Studio (here’s an example for <a href="https://code.visualstudio.com/docs/python/linting" target="_blank" rel="nofollow noopener">Python linting</a>, and <a href="https://code.visualstudio.com/docs/nodejs/reactjs-tutorial#_linting" target="_blank" rel="nofollow noopener">JavaScript linting</a>), which means that they can warn you about issues before you even run or commit your code.
On top of that, linters can be configured to run with git hooks, which means you can even lint your commit messages! Neat, right?
Overall, linters are fairly limited in their scope of work and don’t go much beyond programmatic and stylistic checks, which means they’re one of the most basic forms of static analysis. It’s a great starting point that will provide early benefits while setting the stage for a more thorough solution.

<ul>
<li><a href="https://spacelift.io/blog/devops-best-practices" target="_blank" rel="noopener">16 DevOps Best Practices for 2023</a></li>
<li><a href="https://spacelift.io/blog/terraform-in-ci-cd" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;How to Deploy your Infrastructure in CI/CD using Terraform&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">How to Deploy your Infrastructure in CI/CD using Terraform</a></li>
<li><a href="https://spacelift.io/blog/drift-detection" target="_blank" rel="noopener"><span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Infrastructure Drift Detection and How to Fix It&quot;}" data-sheets-userformat="{&quot;2&quot;:513,&quot;3&quot;:{&quot;1&quot;:0},&quot;12&quot;:0}">Infrastructure Drift Detection and How to Fix It with IaC Tools</a></li>
</ul>

<h3>What is SAST?</h3>
Static Application Security Testing, or SAST, is the process of analyzing source code before compiling it to validate that it uses secure coding policies.
That makes it a useful approach to performing static code reviews and detecting deviances from default coding standards, which means that this is where you are going to catch the most security flaws as early as possible.

devsecops-sast

DevSecOps SAST

The reason that we started the article by talking about source code versioning is because the ideal place to integrate SAST scanning is on every Pull Request (PR) across the organization’s codebase(s) as the code gets committed to those repositories. If you don’t already have some sort of versioning system in place, you can’t effectively implement SAST.
With that said, SAST can be performed at multiple points in the engineering workflow depending on the tool(s) used:
<ul>
<li style="font-weight: 400;" aria-level="1">Directly in the IDE using plug-ins</li>
<li style="font-weight: 400;" aria-level="1">With pre-commit hooks</li>
<li style="font-weight: 400;" aria-level="1">At every pull request</li>
<li style="font-weight: 400;" aria-level="1">At a set and regular frequency to scan entire repositories</li>
</ul>
The reason you may want to implement it at different points in the workflow is to catch different issues without causing too much delay. For example, if you have linting functionality plugged directly into the IDE, developers will see basic errors before they even commit their code.
With pre-commit hooks, additional issues can be found as the code gets committed to a repository.
With pull requests, we can run more detailed scans that take a little bit longer because most PRs are not reviewed immediately. It can take some time for another team member to get to it.
Finally, by setting a regular frequency to scan entire repositories, we can run much more complete and detailed scans that look at the entire code base. This can take time to run because of the amount of code that needs to get checked, but it can happen at a time when no developer is working on the codebase anyway (like in the middle of the night).
<h3>Benefits of SAST</h3>
The point of SAST is to focus on addressing issues within the codebase before they become technical debt. Once security vulnerabilities turn into technical debt, the cost rises exponentially, and the odds of them becoming resolved get reduced significantly.
SAST can help us find issues like:
<ul>
<li style="font-weight: 400;" aria-level="1">Lack of input validation: this is a significant cause of issues in applications and can lead to all sorts of vulnerabilities like XSS, SQL injections, and others listed in the <a href="https://owasp.org/www-project-top-ten/" target="_blank" rel="nofollow noopener">OWASP Top 10</a>.</li>
<li style="font-weight: 400;" aria-level="1">Hardcoded credentials and secrets: which, if leaked, can give attackers privileged access to your tools, platforms, and cloud environments, for example.</li>
<li style="font-weight: 400;" aria-level="1">Logic bugs: which can lead to unintended issues, such as uncontrolled resource consumption, which is when software doesn’t properly control the allocation and maintenance of resources.</li>
</ul>

sast-logic-bugs

SAST Logic bugs

It can find issues like these by using five analysis types:
<ol>
<li style="font-weight: 400;" aria-level="1">Configuration — examining configuration files to make sure they’re aligned with security best practices and policies</li>
<li style="font-weight: 400;" aria-level="1">Semantic — examining code within its context, examining identifiers, syntax, and types</li>
<li style="font-weight: 400;" aria-level="1">Dataflow — tracking the flow of data through the application to see if user inputs are properly handled</li>
<li style="font-weight: 400;" aria-level="1">Control flow — checking for flaws in program execution (race conditions, resource leaks, etc.)</li>
<li style="font-weight: 400;" aria-level="1">Structural — analyzing for language-specific violations of safe coding practices, improper coding practices (dead code, bad use of variables or functions, bad class design, etc.), and hardcoded secrets</li>
</ol>
One of the benefits of SAST scanning is that it is relatively quick. There is no compiling necessary, which means we can integrate it directly into our <a href="https://spacelift.io/blog/ci-cd-pipeline" target="_blank" rel="noopener">CI/CD</a> so that developers receive alerts of potential bugs as soon as they’re trying to commit code to a repository.
An added benefit is that it actually helps train developers over time. Because they’re able to get near-instant feedback for the code they just wrote, it will help them change their coding practices in response to these findings. Over time, it will become second nature which means they’ll spot the issues before the tooling or even avoid them altogether as they’re writing the code.
<h3>Downsides of SAST and Solutions</h3>
Does SAST sound great so far? Good! But not so fast…a major potential downside of implementing SAST that most organizations face initially is that developers get inundated with false positive alerts. Then, if security is trying to force a policy that blocks moving code forward until the scans come back clean, SAST can end up completely slowing down developer velocity, which delays features and frustrates developers and management. Suddenly, an approach to security that was supposed to help turns out to be another bottleneck that seems to provide little value.
Simply purchasing and/or installing a SAST tool doesn’t cut it. SAST tooling requires thoughtful consideration for how it will be implemented.

downsides-of-sast

downsides of sast

To avoid this situation, a few things should happen when first rolling out a SAST solution:
<ol>
<li style="font-weight: 400;" aria-level="1">SAST scan results should not block code from moving forward. They should only inform.</li>
<li style="font-weight: 400;" aria-level="1">SAST should be rolled out in phases starting with only one or two repositories and/or should be run manually and selectively.</li>
<li style="font-weight: 400;" aria-level="1">SAST settings should be fine-tuned based on results to minimize false positives and false negatives, which is a delicate balance that takes time to refine.</li>
<li style="font-weight: 400;" aria-level="1">Only once the SAST results are consistently helpful should it be introduced to every PR and in a blocking mode.</li>
</ol>
Not rolling out a SAST solution across the organization right away might seem odd at first. It might feel like we are potentially allowing vulnerabilities into production that the tool may have warned us about, but you have to think about it this way: the organization has been pushing to production all of this time without SAST scans, so it’ll survive just fine for now. You need to take it one step at a time to avoid a failed roll-out that ends up harming the organization more in the short and long term.
If you roll out SAST in a way that frequently blocks developers for false positives, they’ll not only start to dislike the tooling but also start to ignore findings. This mentality is hard to undo once it’s started, which we need to avoid.
Instead, start out by implementing SAST scans in a non-blocking mode on a selected few code repositories, which you can determine in collaboration with the development team(s). Triage findings and fine-tune the scanner to increase true positive rates. At the same time, create security tickets for legitimate findings and set due dates for fixes.
This will be a manual and intensive process at first, which is another reason to start off with only one or two repositories. Over time, and with refinement, this will become automated.
<h3>SAST Recap and Additional Advice</h3>
Once you have source code versioning and standards and code reviews in place, investing in SAST towards the beginning of your security journey is a great investment. It will take time, and it will require patience, but your goal should be to implement these scanners in your CI/CD pipeline so that you can set up a constant feedback look for developers at the beginning of the code’s journey and you can use that feedback to fine-tune the tools over time.
You’ll want to enable the scanners and passively monitor, but don’t block PRs yet. At least not until you’ve tweaked the settings enough that you’re confident you’ve reached a good balance between false positives and true positives. Then you can start blocking PRs without having the engineers completely ignore findings. While these scanners will still be seen as a source that creates more work for the developers, they’ll at least know that these vulnerability tickets are legitimate and not waste their time.
A great way to track the progress of your SAST roll-out is to implement KPIs (Key Performance Indicators) and overall metrics. This may need to be a manual effort to start, but over time, you can think of how you could automate it. Focus on metrics that actually matter to you, the development team, and the C-suite, but also don’t try to track everything under the sun, or those metrics will get ignored.

Many SAST scanners can scan for known and generic secrets in code. For that reason, you may decide to invest your time in implementing an overall SAST solution before worrying about adding an additional layer of secrets scanning tools.
Or, if you feel that secrets scanning is a very high-priority task, you can implement tools that focus on this specifically before working on implementing broader SAST tooling.

secrets-scanning

secrets scanning

The reason for worrying about secrets scanning in the first place is to avoid mistakes that commit highly privileged secret keys, passwords, etc. into your version control system. Once that happens, especially if it’s a public or visible repository, you’ve got a problem on your hands that requires immediate action but you may not even realize that it happened for quite a while.
That’s why, ideally, you would want to catch these mistakes before a commit is even made. That makes integrating secrets scanners on a local developer’s workstation the best place. A secondary location we can check is before a pull request gets approved.
Examples of tools that can help are:
<ul>
<li style="font-weight: 400;" aria-level="1">Gitleaks</li>
<li style="font-weight: 400;" aria-level="1">Git-Secrets</li>
<li style="font-weight: 400;" aria-level="1">GitHub Secret Scanning</li>
<li style="font-weight: 400;" aria-level="1">Detect-Secrets</li>
</ul>
There are more than this, but these are some examples to get you started.
Finally, in addition to secrets scanning, you’ll want to consider using a secrets management platform (such as Vault, AWS Secrets Manager, Knox, etc.) as your method of integrating necessary secrets without having to hardcode them.

In addition to scanning the application code base, you can implement Infrastructure as Code (IaC) and Policy as Code scanning via SAST tools. This can help detect infrastructure misconfigurations and policy violations as they get committed to repositories instead of as they get deployed to production environments.
Terraform, Pulumi, AWS CloudFormation, Kubernetes, and Ansible are all examples of modern <a href="https://spacelift.io/blog/infrastructure-as-code" target="_blank" rel="noopener">Infrastructure as Code (IaC)</a> tools that many organizations use and that <a href="https://docs.spacelift.io/" target="_blank" rel="noopener">Spacelift supports</a>. They provide a way of controlling environments and infrastructure that lives in AWS, Azure, GCP, etc.
This infrastructure, if misconfigured, can provide an entry point for malicious actors to infiltrate your cloud environments, which can result in significant problems.
You will want to make sure that your SAST solution supports IaC scanning, and/or you will want to look for standalone solutions that can scan your infrastructure specification files.
Because we have an article planned that will be dedicated to explaining and demonstrating IaC scanning, we’ll leave it at that for this article, and we’ll add a link to it as soon as it’s published.

We’ve seen an explosion in the use of container technology for hosting and deploying applications. The reason is simple: they make developing, deploying, and updating applications a much faster and more efficient process.
With that said, they’ve also introduced security concerns. One of those concerns is that many container images are based on other images, which creates a dependency. This is similar to using a third-party library in your application’s code…and <a href="https://owasp.org/Top10/A06_2021-Vulnerable_and_Outdated_Components/" target="_blank" rel="noopener">we’ve seen what can happen</a> in that situation. There have been many recent examples of organizations becoming vulnerable because of a security issue in one of their third-party dependencies.
By scanning containers and their images, we can identify potential security threats in the containers and their components. Components can include the software inside the container, the configurations for storage and networking, and even how the container interacts with the operating system and other containers.

container-image-scanning

container image scanning

Again, and as a common theme throughout this article, there exist broad SAST tools that offer this functionality, but there are also more focused tools we can use for this purpose. They can be directly integrated in the IDE, and they can scan as the container configuration files get committed for very quick feedback.
Container scanning is oftentimes considered to be part of SCA since it has a lot to do with third-party dependencies. To round out this article, let’s talk about SCA in more depth.

Instead of having to reinvent the wheel every time we need to develop a new feature or product, we can turn to open or closed-source software that already does a great job and fits our needs. This can save hours upon hours of hard work and is a wonderful solution for modern development.
The problem that this approach introduces is that these dependencies can have security flaws in them. As soon as we pull that dependency into our project, we add that flaw into our codebase.
If an exploit is found for this security flaw, then projects using that software can become vulnerable. This is not just theory. Many high-profile cases have occurred recently, including <a href="https://www.cisa.gov/uscert/ncas/alerts/aa21-356a" target="_blank" rel="noopener">Log4Shell</a> and <a href="https://www.cisa.gov/news/2020/12/13/cisa-issues-emergency-directive-mitigate-compromise-solarwinds-orion-network" target="_blank" rel="noopener">SolarWinds</a>.
In an attempt to combat this growing problem, organizations can turn to something called Software Composition Analysis, or SCA. SCA is a larger topic, but the part we’ll discuss in this article is SCA scanning.
SCA typically involves:
<ul>
<li style="font-weight: 400;" aria-level="1">Container scanning</li>
<li style="font-weight: 400;" aria-level="1">Dependency scanning</li>
<li style="font-weight: 400;" aria-level="1">License compliance</li>
<li style="font-weight: 400;" aria-level="1">…and more</li>
</ul>

sca-scanning

sca scanning

SCA scanning tools can help find known vulnerabilities in code dependencies, containers, and issues with licensing. We can integrate this form of tooling as a way of:
<ol>
<li style="font-weight: 400;" aria-level="1">Documenting added dependencies, and</li>
<li style="font-weight: 400;" aria-level="1">Preventing them from being added if something wrong is detected</li>
</ol>
For example, does the library we’re trying to use have a license that we would violate with our use case? Or does the framework we’re trying to add have known vulnerabilities that haven’t been fixed yet?
<h3>How are SCA and SAST Different?</h3>
One of the main differences between SCA and SAST scanning is that SCA is typically meant to address open-source code and third-party dependencies, while SAST addresses your organization’s own proprietary code. Another difference is that fixing issues found with SCA scanning will require patching the vulnerabilities through updates provided by the third party or through your own temporary security measures (like firewall rule updates) while you wait for an official fix. Fixing issues found with SAST scanning means your own organization needs to write better and more secure code.
This goes to show that SCA and SAST focus on and solve different issues, which means that they are not interchangeable.
<h3>SCA Recap and Additional Advice</h3>
While you may be tempted to try and implement all of the previous security scannings we’ve talked about with SAST in addition to implementing scanning for SCA at the same time, keep in mind that this requires a tremendous amount of work. Especially if you are just one security person on the team, or if you are part of a small security team.
While SCA is very important, realistically, it can take some time to implement properly. In that case, you may want to focus on implementing SAST scanning, followed by CI/CD integration, and fine-tuning that before moving on to implementing SCA scanning.
Again, though, this is a recommendation that might go against your own organization’s priorities… so let’s conclude by discussing prioritization.

We’ve talked about a lot of topics in this article, and attempting to implement all of these at the same time would be a recipe for failure. Instead, a better approach is to prioritize.
An approach to prioritizing is to come up with a high-level roadmap and then share it with the rest of the team. Creating this roadmap will not only help you think through everything and get a deeper understanding of what it would take to implement, but it will also give a clear picture for the rest of your team to provide feedback on.
From there, you can revisit a few times until you’ve got a prioritized roadmap that makes sense for your team and your organization. This will also help gather support from management earlier in the process because they’ll be able to see that you’ve got a well-thought-out approach and you’re not just slapping together a set of tools and ideas.
Once you’ve got a structured roadmap, you can start adding what we’ve talked about in this article to improve your DevSecOps pipeline one step at a time, with each step helping your organization reduce risk.
We hope this article helps you make progress toward your shift-left journey! If it did, consider subscribing so that you don’t miss our future DevSecOps articles!

<span data-sheets-value="{&quot;1&quot;:2,&quot;2&quot;:&quot;Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.&quot;}" data-sheets-userformat="{&quot;2&quot;:769,&quot;3&quot;:{&quot;1&quot;:0},&quot;11&quot;:4,&quot;12&quot;:0}">Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Open Policy Agent: What Is OPA and How It Works (Examples)

How to Keep Your Playbooks Secure Using Ansible Vault

Managing AWS Security Groups Through Terraform

Learn what code testing is in the DevSecOps lifecycle and the shift-left strategy. Explore why you should implement it and how to do it.

146-devsecops-testing

Automated and Manual Code Testing with DevSecOps

In this article, we’re going to take a closer look at a very important part of the DevOps pipeline: security.
Because of how important security is to <a href="https://spacelift.io/blog/devops-pipeline" target="_blank" rel="noopener">DevOps pipelines</a>, many have taken to calling it DevSecOps instead. What exactly is it, and how can you properly implement it in your organization? Let’s take a look.
Table of contents:
<ol>
<li><a href="https://spacelift.io/blog/what-is-devsecops#what-is-devsecops" rel="">What is DevSecOps?</a></li>
<li><a href="https://spacelift.io/blog/what-is-devsecops#advantages-of-devsecops">Advantages of DevSecOps</a></li>
<li><a href="https://spacelift.io/blog/what-is-devsecops#organizational-culture" rel="">Organizational Culture</a></li>
<li><a href="https://spacelift.io/blog/what-is-devsecops#the-devsecops-lifecycle">The DevSecOps Lifecycle</a></li>
<li><a href="https://spacelift.io/blog/what-is-devsecops#implementing-the-proper-tools-and-processes">Implementing the Proper Tools and Processes</a></li>
</ol>

The point of DevOps is to increase your team’s productivity by breaking down silos between development teams and operations teams.
The point of DevSecOps, therefore, is to implement security best practices while still maintaining increased team productivity by breaking down silos between teams.
This is important because most people view security as an annoyance and as something that adds extra friction to the development and deployment process. Instead of accepting that and putting security on the back burner, DevSecOps brings security back to the forefront by saying “you know what, security is critical and must be part of our process, so let’s make it fit in as painlessly as possible.”
<h3>Shift-left security</h3>
To make security more impactful without adding much friction, we need to include it from the very beginning. The traditional approach has been to plan, develop, and push changes forward. Only then would a security team become involved. 
There are many issues with this approach.
One issue is that the security team will not have access to most of the context. Lack of context results in questions like these being asked without an easy way to gather answers and typically way too late in the process:
<ul>
<li style="font-weight: 400;" aria-level="1">Why is this being developed the way that it is?</li>
<li style="font-weight: 400;" aria-level="1">Who made that decision?</li>
<li style="font-weight: 400;" aria-level="1">Are they aware it will have an impact on other parts of the system?</li>
</ul>
Another issue is that when the security team finds problems, the pipeline has to go all the way back to the beginning. That causes delays and frustration, especially if the development or infrastructure team has already moved on to another task.
Instead, with DevSecOps, we want to include security from the very beginning so that problems are found as early in the process as possible: closest to the decision makers and to the context.
One way we can do this is by including security team members during planning and then again during product demonstrations in between each development interval. This helps threefold:
<ol>
<li style="font-weight: 400;" aria-level="1">It helps the security team understand business goals for the implementation</li>
<li style="font-weight: 400;" aria-level="1">It involves a team that will be impacted during the development instead of after</li>
<li style="font-weight: 400;" aria-level="1">The security team can provide guidance and feedback as features get created</li>
</ol>
This approach provides security expertise earlier in the development process and reduces the amount of time before issues get discovered.
Another way we can do this is by tracking open security issues in the same tracking systems that development and operations teams use. The traditional approach has been to keep security issues in separate tracking systems, which means Dev and Ops don’t have visibility (and vice versa). By having all open issues (from Dev, Sec, and Ops) in a central location, everyone can see what the priorities are, and they can collaborate.
If we envision our pipeline going from left to right — left being the starting point and right being production — by moving security at the very beginning, we are shifting security left. That’s why this is often referred to as a “shift-left” security strategy.

devsecops-shift-left

shift left devsecops

Ask any developer, infrastructure engineer, or security engineer, whether they’d prefer someone to find issues with their code, infrastructure, etc… while they’re still working on it versus after they’ve pushed it through and moved on to another task and the answer will be the same: show me issues as early as possible so that I can quickly fix them and move on to something else.
It’s much easier and less painful to fix issues while the project is fresh on your mind than after you’ve already switched to a different context. It’s much more efficient.
Let’s take a closer look at how those efficiencies can benefit the organization and team.

The reason that security is so important to include from day one is that it will help:
<ul>
<li style="font-weight: 400;" aria-level="1">Reduce risk and legal liability</li>
<li style="font-weight: 400;" aria-level="1">Identify bugs and vulnerabilities earlier in the process</li>
<li style="font-weight: 400;" aria-level="1">Make security more manageable</li>
</ul>

devsecops-reducing-risk-2

reducing risk devsecops

<h3>Reducing risk and legal liability</h3>
When it comes to reducing risk and legal liability, we’re not just talking about getting sued by shareholders or customers when there’s a costly breach. We’re also talking about being in compliance with the law.
Depending on your business, you may have to comply with GDPR law depending on certain situations. Or, you may have to handle health-related records, which means you may need to comply with HIPAA or other important laws.
If you don’t comply, it could cost your organization a massive amount of money in fees.
Having the proper tools and processes in place can help prevent modifications from going out that would result in violations.
It can also help identify compliance violations as the features get created by the development or infrastructure teams instead of right before we’re trying to deploy them to production.
<h3>Identifying bugs and vulnerabilities early in the process</h3>
It’s important for issues that can cause vulnerabilities to be caught as early as possible in the DevOps pipeline.
The reason is that the further into the pipeline you are, the more expensive it is (in both monetary and time requirement terms) to fix issues.
With the traditional DevOps process, code or infrastructure changes are committed and pushed to the next phase. The QA team and automated tools receive those changes and start testing, and everyone crosses their fingers that no major issues are found so that everything can get deployed to production on time in order to meet a deadline.

devsecops-autmated-testing

automated testing devsecops

Except that more often than not, the QA team or automated testing will find problems. Not just cosmetic problems but potentially major security issues that require costly re-writes.
Now, all of a sudden, the issues are sent back to the development or infrastructure teams, and they need to be fixed as soon as possible because they’re holding up a release. Those teams are under massive pressure to deliver a solution. Once they submit a solution, the QA and security tests start over again.

devsecops-security

security devsecops

Security becomes a source of pain and anxiety, and when stakeholders ask why a project is delayed, they’re told that it’s because the security team found issues.
Instead, if you shift security left and include it earlier in the process, it becomes integrated into the development process rather than being an add-on further down. Those otherwise costly security issues get discovered much faster and closer to the developer or infrastructure engineer so that they can address those issues before moving on to another task.
Our goal is to give Dev and Ops fast feedback on the work they are producing so that they can be notified right away and can address the issue before moving on to another task and switching contexts.
<h3>Making security more manageable</h3>
An added benefit of shifting security left is that it helps make security tasks more manageable in a way that can feel completely natural to development teams.
Enabling IT teams to deliver value faster is not a new concept…development has had very similar problems. How has development fixed it? By going agile and by breaking down large projects into the smallest tasks possible.

devsecops-managable-security

devsecops managable security

Security should be handled in the same way. Instead of keeping huge security tests or projects, we need to take those and break them down into really small tests or projects that happen continuously throughout the DevOps pipeline. Being reactive to changes halfway down our deployment pipeline makes this very difficult. Being proactive to changes at the very beginning enables us to be agile.
<h3>How do we achieve these benefits?</h3>
The way that we achieve these benefits is by getting management buy-in, making security part of everyone’s responsibilities, and by implementing the proper tools and processes.
Let’s talk more about what that means and what it looks like.

<h3>Management buy-in</h3>
Properly implementing DevSecOps is a fundamental shift in how your organization thinks about, builds, and deploys software and infrastructure. Doing that requires far more than just your peers agreeing to make a change…it requires management buy-in.
Software vulnerabilities being exploited in production results in risks for the organization. The person who judges risks vs. revenue needs to see the value in implementing DevSecOps, or else it all falls flat.
We want to build a mindset that everyone is responsible for security. That sounds cliché, but it’s true, and it is practical, not just conceptual. For good security decisions to be made at speed and scale, they simply can’t be centralized.
If you want to see this properly implemented, you need management to not only back it up but also implement it themselves.
<h3>Security is everyone’s responsibility</h3>
DevSecOps makes security part of the development process instead of lobbing it over the fence to a security team and forgetting about it.
Instead of hearing, “I don’t usually worry about vulnerabilities in my code or config files because the security team will take care of it” we want to empower the development and infrastructure teams to write more secure code and configurations.
Of course, security is a profession, and we do need highly skilled people in security-focused roles, but we also need everyone else to be responsible for securing their work instead of lobbing it over the fence and saying, “not my problem.”
Again, achieving this requires management buy-in and measuring the right metrics.
<h3>Security and DevSecOps training</h3>
Getting organizational culture right is critical, but it’s not going to happen overnight. Unless you’re already implementing a lot of these concepts, it’s going to require a fundamental shift. There will be resistance, and it will initially cause frustration because you’re changing people’s traditional ways of thinking and working.
Of course, management buy-in will help alleviate these issues, but so will proper training. Not only does management need to understand these concepts (even if at a high level), but so does everyone else (at a more practical level).
Once everyone understands the DevSecOps approach, they’ll not only understand why it’s important, but they’ll also see how to properly implement it and how it will end up helping them do their jobs more efficiently.

As mentioned before, DevSecOps doesn’t have just one right way of implementing it. With that said, visual learners may find it helpful to visualize the lifecycle.

devsecops-lifecycle-2

As we can see, a lot of verifications should happen before we even reach the test or release stages of the lifecycle…or in other words, on the left.
When we’re in the planning phase, which is where all projects start, we should be looking at:
<ul>
<li style="font-weight: 400;" aria-level="1">Providing secure code training</li>
<li style="font-weight: 400;" aria-level="1">Creating baselines</li>
<li style="font-weight: 400;" aria-level="1">Creating threat models</li>
</ul>
Then, when developers start writing application code or when infrastructure engineers start configuring cloud resources to support those applications, we should be using:
<ul>
<li style="font-weight: 400;" aria-level="1">Source code versioning</li>
<li style="font-weight: 400;" aria-level="1">Code reviews</li>
<li style="font-weight: 400;" aria-level="1">Local secrets scanning</li>
<li style="font-weight: 400;" aria-level="1">Linting</li>
<li style="font-weight: 400;" aria-level="1">Static Application Security Testing (SAST)</li>
<li style="font-weight: 400;" aria-level="1">Software Composition Analysis (SCA)</li>
</ul>
The code then moves on to the build stage, where we should be using:
<ul>
<li style="font-weight: 400;" aria-level="1">Dynamic Application Security Testing (DAST)</li>
<li style="font-weight: 400;" aria-level="1">Interactive Application Security Testing (IAST)</li>
<li style="font-weight: 400;" aria-level="1">Centralized SCA and SAST (instead of local only)</li>
</ul>
Read more about <a href="https://spacelift.io/blog/devsecops-code-testing" target="_blank" rel="noopener">Automated and Manual Code Testing with DevSecOps</a>.
After that, we have the testing stage, which should include:
<ul>
<li style="font-weight: 400;" aria-level="1">Centralized secrets scanning</li>
<li style="font-weight: 400;" aria-level="1">Vulnerability assessments</li>
</ul>
We could even include penetration tests in this stage, depending on the size of the organization.
In the release stage, we can run:
<ul>
<li style="font-weight: 400;" aria-level="1">Compliance checks</li>
</ul>
Then, in the deploy stage, we’ll want to:
<ul>
<li style="font-weight: 400;" aria-level="1">Implement code signing</li>
<li style="font-weight: 400;" aria-level="1">Harden endpoints</li>
<li style="font-weight: 400;" aria-level="1">Harden applications</li>
<li style="font-weight: 400;" aria-level="1">Encrypt everything that needs to be</li>
</ul>
Once changes are in production, we’re not done. We need to implement:
<ul>
<li style="font-weight: 400;" aria-level="1">Centralized logging</li>
<li style="font-weight: 400;" aria-level="1">Process for patching</li>
<li style="font-weight: 400;" aria-level="1">Vulnerability disclosure</li>
<li style="font-weight: 400;" aria-level="1">Vulnerability assessments</li>
<li style="font-weight: 400;" aria-level="1">Penetration Tests</li>
<li style="font-weight: 400;" aria-level="1">Bug bounty programs</li>
</ul>
All of that feeds into monitoring where we should implement:
<ul>
<li style="font-weight: 400;" aria-level="1">Access monitoring</li>
<li style="font-weight: 400;" aria-level="1">IPS/IDS and WAF solutions</li>
<li style="font-weight: 400;" aria-level="1">SIEM</li>
<li style="font-weight: 400;" aria-level="1">Attack surface mapping</li>
</ul>
All of that helps us learn and improve, which we can then use the next time we start a new project, which takes us all the way back to the planning phase.
Another common way of visualizing and calling this process is <a href="https://spacelift.io/blog/ci-cd-pipeline" target="_blank" rel="noopener">CI/CD</a>, or Continuous Integration and Continuous Delivery/Deployment.

devsecops-cicd

devsecops cicd

Continuous Integration helps us move through the Plan, Code, Build, Test, and Release stages. 
Continuous Delivery helps us move through the Deploy, Operate, and Monitor stages so that a member of the team can then press a button to smoothly deploy all of the changes to production. 
Continuous Deployment is the same as Continuous Delivery, except it aims to completely automate deployment without even requiring manual intervention. Getting to this point requires a serious amount of trust in your pipeline and is a great goal to strive towards.

Getting organizational buy-in and detecting security issues as early in the pipeline as possible sounds great and all, but also much easier said than done. Plus, there’s not just one right way of achieving DevSecOps. Every organization is different in the way that they build and deploy. With that said, there are concepts that can be applied across the board.
We’ve barely scratched the surface as you could see from the graphic above. Each stage has a number of actions, processes, and tools we can (and should) implement depending on our needs.

Now that we understand what DevSecOps aims to achieve, why it’s advantageous, and what it looks like, we need to explore how to properly implement processes and tools to achieve it in our organization. This will be covered in the next article of our DevSecOps series.
Be sure to check back soon for the rest of this series! Better yet, subscribe to our newsletter so that we can notify you when the latest articles get released.
In the meantime, go ahead and share this article with your colleagues so they can start learning about DevSecOps, and learn how a platform like <a href="https://docs.spacelift.io/" target="_blank" rel="noopener">Spacelift</a> can help you and your organization fully manage cloud resources within minutes. Spacelift is a CI/CD platform for infrastructure-as-code that supports tools like Terraform, Pulumi, Kubernetes, and more. For example, it enables policy-as-code, which lets you define policies and rules that govern your infrastructure automatically. You can even invite your security and compliance teams to collaborate on and approve certain workflows and policies for parts that require a more manual approach.
While we will cover concepts like these in the rest of our DevSecOps series, get a head start with <a href="https://docs.spacelift.io/" target="_blank" rel="noopener">Spacelift’s documentation</a>.

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities s for infrastructure management.

16 DevOps Best Practices Every Developer Should Know

5 Most Useful CI/CD Tools for DevOps in 2023

SRE vs. DevOps: What’s the Difference Between Them?

Learn what DevSecOps is and what it aims to achieve. Take a closer look at a very important part of the DevOps pipeline: security.

Christophe Limpalair

Automated and Manual Code Testing with DevSecOps

Get the newsletter

Recent posts

DevSecOps: Making Security Central To Your DevOps Pipeline