Automated and Manual Code Testing with DevSecOps

In this prior article, we talked about what DevSecOps is and why it’s important. The main goal that we discussed was to shift security left, which is all about including security from the very beginning so that problems are found as early in the process as possible. To expand on that article, we’re going to take a closer look at a very important part of the shift-left strategy: code testing.

Code testing is one of the first lines of defense against introducing vulnerabilities, leaking secrets into production or public sources, and overall bugs that, over time, slow down development velocity and end up costing your organization.

A challenge with code testing is that it can end up slowing down your development team, or it can end up being a waste of time if it’s not properly implemented.

In this article, we’re going to explain what code testing is, why you should not delay implementing it any longer, and approaches that you can take to make the transition easier based on your organization’s needs.

1. DevSecOps: Automated and Manual Code Testing
2. Source code versioning
3. Standards and code reviews
4. Lint Scanning
5. SAST (Static Application Security Testing)
6. Secrets Scanning
7. Infrastructure as Code (IaC) Scanning and Policy as Code
8. Container Image Scanning
9. SCA (Software Composition Analysis
10. Prioritization

These are all topics we will be talking about in this article since they should be part of our overall security strategy. You can also check out how to integrate security tools with Spacelift using custom inputs.

At the end of the day, code testing is not a one-size-fits-all, and it requires both automated and manual approaches, so let’s take a look at how you can implement it successfully.

The primary goal of any security initiative is to bring down the risk for that particular organization. Every organization faces unique risks, so while not everyone should implement automated and manual code testing in the exact same ways, there are implementations that most should at least consider.

Let’s bring back up our handy-dandy DevSecOps lifecycle cheat sheet from our prior article to explain what we mean.

After we’ve gone through the planning stage, it’s time for developers to start writing code or for our infrastructure engineers to start configuring cloud resources to support those applications. This is when we enter the code section of our DevSecOps lifecycle.

If you’re starting from scratch, though, you’re not going to be able to realistically implement all of this in one fell swoop. You’re going to have to prioritize and set a roadmap that the rest of the organization will have to agree on.

With that said, one of the most important steps will be to implement source code versioning, which is why we’ll start by discussing that topic. If you or your organization is not already using source code versioning, that will be one of the most impactful changes you can implement starting today. Almost everything else we discuss in this article builds on top of this foundation.

Next, we will talk about code reviews, a manual approach to finding issues with committed code. While automation is great and should certainly be part of your overall strategy, it can’t completely replace the need for manual reviews, and we’ll be explaining the why and the how.

We’ll then talk about linting, which is often considered to be a cousin of SAST that focuses on a much more narrow scope — and there’s a reason why we’re talking about it separately in this article.

After that, we’ll talk about how we can automate testing using Static Application Security Testing, or SAST. SAST evokes mixed feelings among developers and security professionals, and over 60% of developers who responded to GitLab’s 2020 DevSecOps survey said that they didn’t run any static application security testing.

With that said, GitLab ran another survey in 2022 and reported that 53% of developers now do run static application security testing (SAST) scans, which is quite a significant jump from 2 years prior. Clearly, static scanning has become more popular in a short period of time, but it does take some time to roll out and implement.

Why is that? There are many reasons, but one of the primary reasons is that implementing SAST is not easy. It’s not just a matter of picking an existing tool, plugging it into an environment, and calling it a day. It takes months of effort to get everyone on the same page, perform tests, tweak settings based on those test results, etc…. until it’s an actually helpful solution and not something that gets in the developer’s way.

Because it’s often poorly implemented, which ends up producing more issues than helping, many continue to assume that it’s a problem with SAST and not with the implementation. 

Ultimately, we want SAST to help us:

1. Implement early security checks
2. Reduce human errors
3. Help developers fix issues in less time
4. Track valuable metrics

If it’s not accomplishing all 4 of those goals, then the implementation should be tweaked.

After talking about SAST, we’re going to explore secrets scanning, IaC scanning, and container image scanning. While many SAST solutions include some level of secrets scanning, there also exist standalone tools dedicated to that purpose. IaC scanning, or Infrastructure as Code scanning, scans infrastructure configuration files for known vulnerabilities.

If you’re using something like Terraform, Ansible, CloudFormation, Kubernetes, etc…then you can benefit from using this. If you use containers of any kind, then you can also benefit from container image scanning, which is the practice of scanning container images for known vulnerabilities. Container scanning is often considered to be a form of SCA, so let’s talk about that next.

SCA stands for Software Composition Analysis. It seems like every other month, we read another news article about how a third-party library, plug-in, or framework has been compromised, and because it’s in use by thousands of organizations, all of them are now vulnerable.

SCA aims to find issues with third-party dependencies before they become problems with our applications and before they can get exploited. That’s easier said than done, which is why we’ll spend some time learning about how SCA works and when it makes sense to implement it.

Now that we’ve gone through a quick description of each topic let’s dive right in.

You can’t secure what you don’t know you have. The same holds true for software, which is why it all starts with using a Version Control System, or VCS, to implement source code versioning.

Suppose your organization is not already using a VCS, like Git. In that case, this is one of the most impactful technological changes you could implement, and it doesn’t make a whole lot of sense to move on to the rest of this article or other security-related tasks until that’s done.

Otherwise, investing time, effort, and money into code security will have a minimal impact.

Once you’ve implemented source code versioning, you can put security findings directly in front of developers where they work (the VCS), instead of making them have to work out of yet another platform or tool.

That’s also why applications such as Spacelift integrate directly with GitHub. That way, you can control and get visibility into your stacks, modules, policies, and overall resources from one central location.

Once you’ve implemented source code versioning, it’s time to implement standards and code reviews.

Ideally, during the planning phase of our DevSecOps pipeline, the organization has written Tech Design Documents (aka Tech Specs).

A great way to include security from the very beginning is to standardize embedding security questions directly in those documents. It’s no longer sufficient only to include designs and features for new products and services. We also need to include security questions that the engineers fill out as they write those technical documents.

Examples of security questions we could embed include:

• Does this service take any inputs from end users? If so, what are you using to protect against common injection attacks like XSS, SQL Injections, etc.?
• Does this service process/store any sensitive user or customer information? If so, is it encrypted?
• If encryption/decryption/salting/hashing are needed, which algorithms are being used?
• Does the service require any secrets to be used (i.e., to establish a database connection), and if so, is the secret coming from a secret management solution?
• How will proper monitoring and alerting be put in place for this service?

These are, of course, just a handful of examples of the types of questions we should be asking and answering before work actually begins. That way, we don’t get close to the finish line before realizing that we’re using a deprecated and weak encryption algorithm. Or we totally forgot to handle user inputs properly, and we’re exposing our database. Etc.

In fact, these types of questions are multipurpose:

1. They help the engineers think through security while they’re already thinking about technical design.
2. They help the security team gather security-specific information very early on in the process.
3. They give the security team the opportunity to highlight areas of concern or provide recommendations before a single line of code is written.
4. They produce information and diagrams that can be fed into the organization’s Threat Models.

An added benefit is that they can then be used during code reviews and/or to set up automated tests since we can look to make sure that specific details were implemented as intended. Speaking of code reviews…

A typical flow for code reviews is to have a development team do a code review for a pull request (PR) before they merge their code. Instead of having the quality of these code reviews be completely dependent on the person doing the reviewing, it’s a good idea to devise a checklist.

That checklist will, in part, be formed by those very questions answered in the tech specs document. In addition, you can have a checklist created based on the language(s) being used, since each language will have its unique quirks.

For examples of checklists and for much more detail on how to implement secure code reviews, check out this wonderful OWASP resource.

Even with checklists, though, the effectiveness of code reviews still comes down to the reviewers’ skills and knowledge and the time that they have available. If they’re rushing through the code review because they’re trying to complete other priorities, or if they don’t have the knowledge necessary, they may very well miss subtle security problems. That’s where the other recommendations we just talked about can really help.

To improve your secure code reviews, you can:

1. Standardize embedding security questions in Tech & Design Documents.
2. Create code review checklists based on those security questions and on the specific language(s)
3. Educate and train your developers on the basics of threat modeling, code reviews, and risk assessments

The reason code reviews are still important and needed, even when you can implement automated tools like SAST (which we’ll talk about in just a moment), is because tools aren’t going to find everything. Developers still need to use code reviews, which are a manual process, to find vulnerabilities in code that can’t be scanned by security testing tools, no matter how advanced. With that said, we also can’t just rely on manual code reviews, so let’s talk about automation tools.

In addition to code reviews, we’ll want to implement a linting solution.

Linting tools are often considered to be “cousins” of SAST tools because linting is the process of using static code analysis to identify stylistic and programmatic errors in code.

That makes a lint tool a basic static code analyzer, and while they can be used standalone, they can also often be integrated directly in SAST tools.

Linting is an important step because it helps reduce the number of basic programmatic errors that could result in bugs, and it helps reduce the number of stylistic errors that improve the overall health of your codebase.

That means linting would happen even before code reviews and testing, which helps alleviate the load on reviewers and reduces the likelihood of low-hanging fruit being present during testing.

We’re talking about linting separately from SAST because linting is easier and faster to implement. It often just works out of the box with only a few tweaks necessary, which is definitely not the case for SAST. Instead of waiting to receive linting benefits until SAST tooling has been thoroughly tweaked and tested, you can implement an effective linting solution this week.

Linting is especially useful for languages like Python and JavaScript. Since they aren’t compiled, they’re considered to be dynamically typed languages. That means they don’t usually enforce strict rules, which can make it easier for developers to deviate from language-specific best practices.

Linters can be directly integrated as plugins in your favorite IDE (Integrated Development Environment), like Visual Studio (here’s an example for Python linting, and JavaScript linting), which means that they can warn you about issues before you even run or commit your code.

On top of that, linters can be configured to run with git hooks, which means you can even lint your commit messages! Neat, right?

Overall, linters are fairly limited in their scope of work and don’t go much beyond programmatic and stylistic checks, which means they’re one of the most basic forms of static analysis. It’s a great starting point that will provide early benefits while setting the stage for a more thorough solution.

What is SAST?

Static Application Security Testing, or SAST, is the process of analyzing source code before compiling it to validate that it uses secure coding policies.

That makes it a useful approach to performing static code reviews and detecting deviances from default coding standards, which means that this is where you are going to catch the most security flaws as early as possible.

The reason that we started the article by talking about source code versioning is because the ideal place to integrate SAST scanning is on every Pull Request (PR) across the organization’s codebase(s) as the code gets committed to those repositories. If you don’t already have some sort of versioning system in place, you can’t effectively implement SAST.

With that said, SAST can be performed at multiple points in the engineering workflow depending on the tool(s) used:

• Directly in the IDE using plug-ins
• With pre-commit hooks
• At every pull request
• At a set and regular frequency to scan entire repositories

The reason you may want to implement it at different points in the workflow is to catch different issues without causing too much delay. For example, if you have linting functionality plugged directly into the IDE, developers will see basic errors before they even commit their code.

With pre-commit hooks, additional issues can be found as the code gets committed to a repository.

With pull requests, we can run more detailed scans that take a little bit longer because most PRs are not reviewed immediately. It can take some time for another team member to get to it.

Finally, by setting a regular frequency to scan entire repositories, we can run much more complete and detailed scans that look at the entire code base. This can take time to run because of the amount of code that needs to get checked, but it can happen at a time when no developer is working on the codebase anyway (like in the middle of the night).

Benefits of SAST

The point of SAST is to focus on addressing issues within the codebase before they become technical debt. Once security vulnerabilities turn into technical debt, the cost rises exponentially, and the odds of them becoming resolved get reduced significantly.

SAST can help us find issues like:

• Lack of input validation: this is a significant cause of issues in applications and can lead to all sorts of vulnerabilities like XSS, SQL injections, and others listed in the OWASP Top 10.
• Hardcoded credentials and secrets: which, if leaked, can give attackers privileged access to your tools, platforms, and cloud environments, for example.
• Logic bugs: which can lead to unintended issues, such as uncontrolled resource consumption, which is when software doesn’t properly control the allocation and maintenance of resources.

It can find issues like these by using five analysis types:

1. Configuration — examining configuration files to make sure they’re aligned with security best practices and policies
2. Semantic — examining code within its context, examining identifiers, syntax, and types
3. Dataflow — tracking the flow of data through the application to see if user inputs are properly handled
4. Control flow — checking for flaws in program execution (race conditions, resource leaks, etc.)
5. Structural — analyzing for language-specific violations of safe coding practices, improper coding practices (dead code, bad use of variables or functions, bad class design, etc.), and hardcoded secrets

One of the benefits of SAST scanning is that it is relatively quick. There is no compiling necessary, which means we can integrate it directly into our CI/CD so that developers receive alerts of potential bugs as soon as they’re trying to commit code to a repository.

An added benefit is that it actually helps train developers over time. Because they’re able to get near-instant feedback for the code they just wrote, it will help them change their coding practices in response to these findings. Over time, it will become second nature which means they’ll spot the issues before the tooling or even avoid them altogether as they’re writing the code.

Downsides of SAST and Solutions

Does SAST sound great so far? Good! But not so fast…a major potential downside of implementing SAST that most organizations face initially is that developers get inundated with false positive alerts. Then, if security is trying to force a policy that blocks moving code forward until the scans come back clean, SAST can end up completely slowing down developer velocity, which delays features and frustrates developers and management. Suddenly, an approach to security that was supposed to help turns out to be another bottleneck that seems to provide little value.

Simply purchasing and/or installing a SAST tool doesn’t cut it. SAST tooling requires thoughtful consideration for how it will be implemented.

To avoid this situation, a few things should happen when first rolling out a SAST solution:

1. SAST scan results should not block code from moving forward. They should only inform.
2. SAST should be rolled out in phases starting with only one or two repositories and/or should be run manually and selectively.
3. SAST settings should be fine-tuned based on results to minimize false positives and false negatives, which is a delicate balance that takes time to refine.
4. Only once the SAST results are consistently helpful should it be introduced to every PR and in a blocking mode.

Not rolling out a SAST solution across the organization right away might seem odd at first. It might feel like we are potentially allowing vulnerabilities into production that the tool may have warned us about, but you have to think about it this way: the organization has been pushing to production all of this time without SAST scans, so it’ll survive just fine for now. You need to take it one step at a time to avoid a failed roll-out that ends up harming the organization more in the short and long term.

If you roll out SAST in a way that frequently blocks developers for false positives, they’ll not only start to dislike the tooling but also start to ignore findings. This mentality is hard to undo once it’s started, which we need to avoid.

Instead, start out by implementing SAST scans in a non-blocking mode on a selected few code repositories, which you can determine in collaboration with the development team(s). Triage findings and fine-tune the scanner to increase true positive rates. At the same time, create security tickets for legitimate findings and set due dates for fixes.

This will be a manual and intensive process at first, which is another reason to start off with only one or two repositories. Over time, and with refinement, this will become automated.

SAST Recap and Additional Advice

Once you have source code versioning and standards and code reviews in place, investing in SAST towards the beginning of your security journey is a great investment. It will take time, and it will require patience, but your goal should be to implement these scanners in your CI/CD pipeline so that you can set up a constant feedback look for developers at the beginning of the code’s journey and you can use that feedback to fine-tune the tools over time.

You’ll want to enable the scanners and passively monitor, but don’t block PRs yet. At least not until you’ve tweaked the settings enough that you’re confident you’ve reached a good balance between false positives and true positives. Then you can start blocking PRs without having the engineers completely ignore findings. While these scanners will still be seen as a source that creates more work for the developers, they’ll at least know that these vulnerability tickets are legitimate and not waste their time.

A great way to track the progress of your SAST roll-out is to implement KPIs (Key Performance Indicators) and overall metrics. This may need to be a manual effort to start, but over time, you can think of how you could automate it. Focus on metrics that actually matter to you, the development team, and the C-suite, but also don’t try to track everything under the sun, or those metrics will get ignored.

Many SAST scanners can scan for known and generic secrets in code. For that reason, you may decide to invest your time in implementing an overall SAST solution before worrying about adding an additional layer of secrets scanning tools.

Or, if you feel that secrets scanning is a very high-priority task, you can implement tools that focus on this specifically before working on implementing broader SAST tooling.

The reason for worrying about secrets scanning in the first place is to avoid mistakes that commit highly privileged secret keys, passwords, etc. into your version control system. Once that happens, especially if it’s a public or visible repository, you’ve got a problem on your hands that requires immediate action but you may not even realize that it happened for quite a while.

That’s why, ideally, you would want to catch these mistakes before a commit is even made. That makes integrating secrets scanners on a local developer’s workstation the best place. A secondary location we can check is before a pull request gets approved.

Examples of tools that can help are:

• Gitleaks
• Git-Secrets
• GitHub Secret Scanning
• Detect-Secrets

There are more than this, but these are some examples to get you started.

Finally, in addition to secrets scanning, you’ll want to consider using a secrets management platform (such as Vault, AWS Secrets Manager, Knox, etc.) as your method of integrating necessary secrets without having to hardcode them.

In addition to scanning the application code base, you can implement Infrastructure as Code (IaC) and Policy as Code scanning via SAST tools. This can help detect infrastructure misconfigurations and policy violations as they get committed to repositories instead of as they get deployed to production environments.

Terraform, OpenTofu, Pulumi, AWS CloudFormation, Kubernetes, and Ansible are all examples of modern Infrastructure as Code (IaC) tools that many organizations use and that Spacelift supports. They provide a way of controlling environments and infrastructure that lives in AWS, Azure, GCP, etc.

This infrastructure, if misconfigured, can provide an entry point for malicious actors to infiltrate your cloud environments, which can result in significant problems.

You will want to make sure that your SAST solution supports IaC scanning, and/or you will want to look for standalone solutions that can scan your infrastructure specification files.

Read more about Infrastructure as Code (IaC) scanning and Policy as Code scanning for vulnerabilities.

We’ve seen an explosion in the use of container technology for hosting and deploying applications. The reason is simple: they make developing, deploying, and updating applications a much faster and more efficient process.

With that said, they’ve also introduced security concerns. One of those concerns is that many container images are based on other images, which creates a dependency. This is similar to using a third-party library in your application’s code…and we’ve seen what can happen in that situation. There have been many recent examples of organizations becoming vulnerable because of a security issue in one of their third-party dependencies.

By scanning containers and their images, we can identify potential security threats in the containers and their components. Components can include the software inside the container, the configurations for storage and networking, and even how the container interacts with the operating system and other containers.

Again, and as a common theme throughout this article, there exist broad SAST tools that offer this functionality, but there are also more focused tools we can use for this purpose. They can be directly integrated in the IDE, and they can scan as the container configuration files get committed for very quick feedback.

Container scanning is oftentimes considered to be part of SCA since it has a lot to do with third-party dependencies. To round out this article, let’s talk about SCA in more depth.

Instead of having to reinvent the wheel every time we need to develop a new feature or product, we can turn to open or closed-source software that already does a great job and fits our needs. This can save hours upon hours of hard work and is a wonderful solution for modern development.

The problem that this approach introduces is that these dependencies can have security flaws in them. As soon as we pull that dependency into our project, we add that flaw into our codebase.

If an exploit is found for this security flaw, then projects using that software can become vulnerable. This is not just theory. Many high-profile cases have occurred recently, including Log4Shell and SolarWinds.

In an attempt to combat this growing problem, organizations can turn to something called Software Composition Analysis, or SCA. SCA is a larger topic, but the part we’ll discuss in this article is SCA scanning.

SCA typically involves:

• Container scanning
• Dependency scanning
• License compliance
• …and more

SCA scanning tools can help find known vulnerabilities in code dependencies, containers, and issues with licensing. We can integrate this form of tooling as a way of:

1. Documenting added dependencies, and
2. Preventing them from being added if something wrong is detected

For example, does the library we’re trying to use have a license that we would violate with our use case? Or does the framework we’re trying to add have known vulnerabilities that haven’t been fixed yet?

How are SCA and SAST Different?

One of the main differences between SCA and SAST scanning is that SCA is typically meant to address open-source code and third-party dependencies, while SAST addresses your organization’s own proprietary code. Another difference is that fixing issues found with SCA scanning will require patching the vulnerabilities through updates provided by the third party or through your own temporary security measures (like firewall rule updates) while you wait for an official fix. Fixing issues found with SAST scanning means your own organization needs to write better and more secure code.

This goes to show that SCA and SAST focus on and solve different issues, which means that they are not interchangeable.

SCA Recap and Additional Advice

While you may be tempted to try and implement all of the previous security scannings we’ve talked about with SAST in addition to implementing scanning for SCA at the same time, keep in mind that this requires a tremendous amount of work. Especially if you are just one security person on the team, or if you are part of a small security team.

While SCA is very important, realistically, it can take some time to implement properly. In that case, you may want to focus on implementing SAST scanning, followed by CI/CD integration, and fine-tuning that before moving on to implementing SCA scanning.

Again, though, this is a recommendation that might go against your own organization’s priorities… so let’s conclude by discussing prioritization.

We’ve talked about a lot of topics in this article, and attempting to implement all of these at the same time would be a recipe for failure. Instead, a better approach is to prioritize.

An approach to prioritizing is to come up with a high-level roadmap and then share it with the rest of the team. Creating this roadmap will not only help you think through everything and get a deeper understanding of what it would take to implement, but it will also give a clear picture for the rest of your team to provide feedback on.

From there, you can revisit a few times until you’ve got a prioritized roadmap that makes sense for your team and your organization. This will also help gather support from management earlier in the process because they’ll be able to see that you’ve got a well-thought-out approach and you’re not just slapping together a set of tools and ideas.

Once you’ve got a structured roadmap, you can start adding what we’ve talked about in this article to improve your DevSecOps pipeline one step at a time, with each step helping your organization reduce risk.

We hope this article helps you make progress toward your shift-left journey! If it did, consider subscribing so that you don’t miss our future DevSecOps articles!