DevSecOps Testing: Automated and Manual

In a previous article, we discussed DevSecOps and its importance. The main goal is to shift security left, which centers on including security from the very beginning so that problems are found as early in the process as possible. To expand on that article, we’re going to take a closer look at a very important part of the shift-left strategy: code testing.

Code testing is one of the first lines of defense against introducing vulnerabilities, leaking secrets into production or public sources, and bringing in bugs that slow down development velocity over time and ultimately cost your organization.

However, code testing can slow down your development team or even be futile if it’s not properly implemented.

In this article, we’ll explain code testing, why you should not delay implementing it, and approaches to make the transition easier based on your organization’s needs.

1. What is the role of code testing in DevSecOps?
2. Strategies to improve the code testing process
3. Types of testing in DevSecOps
4. SAST (Static Application Security Testing)
5. Secrets scanning
6. DAST (Dynamic Application Security Testing)
7. IAST (Interactive Application Security Testing)
8. Infrastructure as Code (IaC) scanning and Policy as Code
9. Container image scanning
10. SCA (Software Composition Analysis)

These approaches should be part of your overall security strategy. You can also learn how to integrate security tools with Spacelift using custom inputs.

Code testing varies depending on the organization and involves both automated and manual approaches. Here’s how to implement it successfully: 

The primary goal of any security initiative is to reduce risk. Every organization faces unique risks, but although not everyone will implement automated and manual code testing in the same way, they should at least consider certain implementations.

Here is the DevSecOps lifecycle cheat sheet from our previous article to explain what we mean:

After the planning stage, developers start writing code for infrastructure engineers to configure cloud resources to support those applications. This is the code section of our DevSecOps lifecycle.

However, anyone starting from scratch won’t be able to realistically implement all of this in one fell swoop. They will have to prioritize and set a roadmap the rest of the organization agrees on.

Even so, source code versioning is a key step. For any organization not using source code versioning already, it is one of the most impactful changes it can implement immediately. Almost everything else we discuss in this article is based on this foundation.

Next, we will discuss code reviews, a manual approach to finding issues with committed code. Automation should certainly be part of your overall strategy, but it can’t replace the need for manual reviews completely.

We’ll then discuss linting, which is often considered a cousin of SAST that focuses on a much narrower scope — and there’s a reason why we discuss it separately in this article.

After that, we’ll discuss how to automate testing using SAST. SAST evokes mixed feelings among developers and security professionals, and over 60% of developers who responded to GitLab’s 2020 DevSecOps survey said they didn’t run any static application security testing.

However, GitLab ran another survey in 2022 and reported that 53% of developers now do run SAST scans, a significant jump in two years. Clearly, static scanning has become more popular in a short period of time, but it does take time to roll out and implement.

One primary reason is that implementing SAST is not easy. It’s not just a matter of picking an existing tool and plugging it into an environment. It takes months of effort to get everyone on the same page, perform tests, and tweak settings based on those test results beforel it’s a helpful solution that doesn’t get in the developer’s way.

Because it’s often poorly implemented, producing more issues than it helps, many continue to assume that the problem is with SAST and not with the implementation. 

Ultimately, we want SAST to help us:

1. Implement early security checks
2. Reduce human errors
3. Help developers fix issues in less time
4. Track valuable metrics

If it’s not accomplishing all those goals, then the implementation should be tweaked.

We will also explore secrets scanning, IaC scanning, and container image scanning. Many SAST solutions include secret scanning and standalone tools also fulfill this purpose. Infrastructure-as-code (IaC) scanning, scans infrastructure configuration files for known vulnerabilities and is helpful if you are using something like Terraform, Ansible, CloudFormation, or Kubernetes. If you use containers of any kind, container image scanning is the practice of scanning container images for known vulnerabilities. Container scanning is often considered to be a form of Software Composition Analysis (SCA).

SCA aims to find issues with third-party dependencies such as libraries, plug-ins, or frameworks before they become problems and can be exploited. That’s easier said than done, so we’ll spend some time learning about how SCA works and when it makes sense to implement it.

Here are some actionable DevSecOps strategies to improve your code testing pipeline.

1. Source code versioning

You can’t secure what you don’t know you have. That is why you need a Version Control System ( VCS) to implement source code versioning for your software.

If you don’t use a VCS such as Git investing time, effort, and money in code security is largely pointless.

Once you’ve implemented source code versioning, you can present security findings directly to developers where they work (the VCS) instead of forcing them to use another platform or tool.

Applications such as Spacelift integrate directly with GitHub so you can control your stacks, modules, policies, and overall resources and get visibility into them from one central location.

With source code versioning in place, you can implement standards and code reviews.

2. Implement standards and code reviews

Ideally, the organization will have written Tech Design Documents (a.k.a. Tech Specs) during the planning phase of the DevSecOps pipeline.

A great way to include security from the very beginning is to standardize the embedding of security questions directly in those documents. It’s no longer sufficient to include designs and features for new products and services only. We also need to include security questions the engineers complete as they write those technical documents.

Examples of security questions we could embed include:

• Does this service take any inputs from end users? If so, what are you using to protect against common injection attacks like XSS, SQL Injections, etc.?
• Does this service process/store any sensitive user or customer information? If so, is it encrypted?
• If encryption/decryption/salting/hashing are needed, which algorithms are being used?
• Does the service require any secrets to be used (e.g., to establish a database connection)? If so, is the secret coming from a secret management solution?
• How will proper monitoring and alerting be put in place for this service?

These are just a few examples of the types of questions we should be asking and answering before work actually begins. These ensure you don’t near the finish line only to discover you are using a deprecated, weak encryption algorithm, or you expose your database because you forget  to handle user inputs properly.

These types of questions are multipurpose:

1. They help the engineers think through security while they’re already thinking about technical design.
2. They help the security team gather security-specific information very early in the process.
3. They give the security team the opportunity to highlight areas of concern or provide recommendations before a single line of code is written.
4. They produce information and diagrams that can be fed into the organization’s threat models.

An added benefit is that they can then be used during code reviews and/or to set up automated tests because they can be reviewed to ensure specific details were implemented as intended. 

A typical code-review flow is for a development team to review a pull request (PR) before merging their code. Instead of making the quality of these code reviews completely dependent on the person doing the reviewing, it’s a good idea to devise a checklist.

That checklist will be dictated partly by the questions answered in the tech specs document. You can also create a checklist based on the language(s) being used because each language will have its unique quirks.

Check this wonderful OWASP resource for examples of checklists and for more detail on how to implement secure code reviews.,.

However, even with checklists, the effectiveness of code reviews still depends on the reviewers’ skills and knowledge and the time they have available. If they rush through the code review because they’re trying to complete other priorities or they lack the necessary knowledge, they may miss subtle security problems. That’s where the other recommendations we just discussed can really help.

To improve your secure code reviews, you can:

1. Standardize embedding security questions in tech and design documents.
2. Create code review checklists based on those security questions and on the specific language(s)
3. Educate and train your developers on the basics of threat modeling, code reviews, and risk assessments

The reason code reviews are still required when you can implement automated tools like SAST is that tools can’t find everything. Developers still need to use manual code reviews to find vulnerabilities in code that can’t be scanned even by the most advanced security testing tools. However, you can’t rely on manual code reviews alone either, so let’s talk about automation tools.

3. Linting

You’ll need to implement a linting solution in addition to code reviews.

Linting tools are often seen as cousins of SAST tools because linting is the process of using static code analysis to identify stylistic and programmatic errors in code.

That makes a lint tool a basic static code analyzer, and although they can be used standalone, they can often be integrated directly into SAST tools.

Linting is an important step because it helps reduce the number of basic programmatic errors that could result in bugs and stylistic errors, improving the overall health of your codebase.

That means linting should happen even before code reviews and testing, helping to alleviate the load on reviewers and reducing the likelihood of low-hanging fruit appearing during testing.

We’re discussing linting separately from SAST because linting is easier and faster to implement. Unlike SAST, it often works out of the box with only a few tweaks. Instead of waiting to receive linting benefits until SAST tooling has been thoroughly tweaked and tested, you can implement an effective linting solution this week.

Linting is especially useful for languages like Python and JavaScript. They aren’t compiled, so they’re considered dynamically typed languages. That means they don’t usually enforce strict rules, which can make it easier for developers to deviate from language-specific best practices.

Linters can be directly integrated as plugins in your favorite IDE (Integrated Development Environment), like Visual Studio (here’s an example for Python linting, and JavaScript linting), which means that they can warn you about issues before you even run or commit your code.

Furthermore, linters can be configured to run with git hooks, which means you can even lint your commit messages.

Overall, linters are fairly limited in their scope of work and don’t go far beyond programmatic and stylistic checks, making them one of the most basic forms of static analysis. A great starting point, they will provide early benefits while setting the stage for a more thorough solution.

Here’s an overview of DevSecOps testing methods:

Static Application Security Testing (SAST) is the process of analyzing source code before compiling it to validate that it uses secure coding policies.

This makes it a useful approach to performing static code reviews and detecting deviances from default coding standards, sot you can catch the most security flaws as early as possible.

We started the article by discussing source code versioning because the ideal place to integrate SAST scanning is on every Pull Request (PR) across the organization’s codebase(s) as the code gets committed to those repositories. If you don’t have a versioning system in place, you can’t implement SAST effectively.

SAST can be performed at multiple points in the engineering workflow depending on the tool(s) used:

• Directly in the IDE using plug-ins
• With pre-commit hooks
• At every pull request
• At a set and regular frequency to scan entire repositories

If you implement it at different points in the workflow, you can catch different issues with minimal delay. For example, if you have linting functionality plugged directly into the IDE, developers will see basic errors before they even commit their code.

With pre-commit hooks, additional issues can be found as the code gets committed to a repository.

With PRs, we can run more detailed scans, which take a little longer because most PRs are not reviewed immediately and can take time for another team member to get to.

Finally, setting a regular frequency to scan entire repositories enables far more complete and detailed scans that look at the entire codebase. They can take time to run because of the amount of code that needs to get checked but can be scheduled at times when no developer is working on the codebase anyway (such as in the middle of the night).

Benefits of SAST

The point of SAST is to focus on addressing issues within the codebase before they become technical debt. Once security vulnerabilities become technical debt, their cost rises exponentially, and the odds of their resolution decrease significantly.

SAST can help us find issues, including:

• Lack of input validation: This is a significant cause of application issues and can lead to vulnerabilities including XSS, SQL injections, and others listed in the OWASP Top 10.
• Hardcoded credentials and secrets: If leaked, these can give attackers privileged access to your tools, platforms, and cloud environments, for example.
• Logic bugs: These can lead to unintended issues, such as uncontrolled resource consumption, which is when software fails to control the allocation and maintenance of resources properly.

It uses five analysis types to find issues:

1. Configuration — examining configuration files to make sure they’re aligned with security best practices and policies
2. Semantic — examining code within its context, examining identifiers, syntax, and types
3. Dataflow — tracking the flow of data through the application to see if user inputs are properly handled
4. Control flow — checking for flaws in program execution (race conditions, resource leaks, etc.)
5. Structural — analyzing for language-specific violations of safe coding practices, improper coding practices (dead code, bad use of variables or functions, bad class design, etc.), and hardcoded secrets

One of the benefits of SAST scanning is that it is relatively quick. Compiling is unnecessary, which means we can integrate it directly into our CI/CD so that developers are alerted to potential bugs as soon as they try to commit code to a repository.

An added benefit is that it actually helps train developers over time. They get near-instant feedback for the code they write, so they can change their coding practices in response. Over time, they will learn to spot issues before the tooling does or even avoid them altogether as they write the code.

Downsides of SAST and solutions

A major potential downside of implementing SAST is that developers get inundated with false positive alerts initially. Then, if security is trying to force a policy that blocks moving code forward until the scans come back clean, SAST can impede developer velocity, which delays features and frustrates developers and management. This could turn it into another bottleneck that seems to provide little value.

Simply purchasing and/or installing a SAST tool is not enough. You need to consider its implementation carefully. 

To avoid this situation, certain things should happen when rolling out a SAST solution:

1. SAST scan results should not block code from moving forward. They should only inform.
2. SAST should be rolled out in phases, starting with only one or two repositories, and/or it should be run manually and selectively.
3. SAST settings should be fine-tuned based on results to minimize false positives and false negatives, which is a delicate balance that takes time to refine.
4. Only when the SAST results are consistently helpful should the solution be introduced to every PR and in a blocking mode.

Not rolling out a SAST solution across the organization right away might seem odd at first. It might feel like you are allowing potential vulnerabilities into production that the tool could have warned you about. However, if the organization has been pushing to production without SAST scans until now, it’ll survive just fine for now. You need to take it one step at a time to avoid a failed roll-out that ends up harming the organization more in the short and long term.

If you roll out SAST in a way that frequently blocks developers for false positives, they’ll start to dislike the tooling and ignore its findings. This mentality is hard to undo once it’s started.

Instead, start by implementing SAST scans in a non-blocking mode on a few selected code repositories, which you can determine in collaboration with the development team(s). Triage findings and fine-tune the scanner to increase true positive rates. At the same time, create security tickets for legitimate findings and set due dates for fixes.

This will be a manual, intensive process at first, which is another reason to start with one or two repositories only. This will become automatic over time and with refinement.

SAST recap and additional advice

Once you have source code versioning and standards and code reviews in place, investing in SAST toward the beginning of your security process is a great investment. It will take time and patience, but your goal should be to implement these scanners in your CI/CD pipeline so that you can set up a constant feedback loop for developers at the beginning of the code’s journey, which you can use to fine-tune the tools over time.

Enable the scanners and passively monitor, but don’t block PRs until you’ve tweaked the settings enough to be confident you’ve reached a good balance between false positives and true positives. Then you can start blocking PRs without engineers completely ignoring findings. These scanners will still be considered a source of more work for the developers, but they’ll know the vulnerability tickets are legitimate and not waste their time.

A great way to track the progress of your SAST roll-out is to implement KPIs (Key Performance Indicators) and overall metrics. This may need to be a manual effort to start, but over time, you can think of ways to automate it. Focus on metrics that actually matter to you, the development team, and the C-suite, but if you try to track everything, the metrics will be ignored.

Many SAST scanners can scan for known and generic secrets in code, so you may decide to spend time implementing an overall SAST solution before adding an additional layer of secret scanning tools.

If secret scanning is a very high-priority task, you can implement tools that focus on this specifically before mplementing broader SAST tooling.

Secret scanning prevents mistakes that commit highly privileged secret keys, passwords, etc., into your VCS. If that happens, it requires immediate action but you may not even know about it for some time.

Ideally, you should catch these mistakes before a commit is even made. A local developer’s workstation is the best place to integrate secret scanners, and ae secondary location you can check is before a pull request gets approved.

Examples of tools that can help are:

• Gitleaks
• Git-Secrets
• GitHub Secret Scanning
• Detect-Secrets

There are more than this, but these are some examples to get you started.

You should also consider using a secrets management platform (such as Vault, AWS Secrets Manager, Knox, etc.) to integrate necessary secrets without having to hardcode them.

Dynamic Application Security Testing (DAST) is a type of security testing that evaluates the security of a running application by simulating real-world attacks. It focuses on identifying vulnerabilities in web applications while they are in their operational state, without requiring access to the source code. DAST examines the application externally, mimicking the behavior of a potential attacker to find security flaws that arise during runtime.

How DAST works:

1. Crawl: The DAST tool crawls the web application to identify all accessible endpoints.
2. Attack simulation: It sends crafted inputs to simulate common attacks on identified endpoints.
3. Detection: The tool monitors responses and detects anomalous or vulnerable behaviors.
4. Reporting: It identifies vulnerabilities, categorizes them by severity, and generates a report.

However, DAST has limitations, such as limited coverage of code paths, an inability to detect logical vulnerabilities, requiring running apps, slower feedback, and difficulty detecting vulnerabilities in APIs or client-side code. It can’t identify issues in code that’s not executed during testing.

Interactive Application Security Testing (IAST) is a security testing method that checks for vulnerabilities in an application while it’s running. It blends elements of two other approaches: SAST, which examines code without running it, and DAST, which tests the app while it’s running externally.

IAST works by embedding an agent inside your application. This agent watches the app as it runs and performs its regular functions. For example, if you’re testing the app or using it normally, the agent will monitor how data flows, which parts of the code are executed, and which inputs are being processed. If it spots a security vulnerability, it reports it immediately.

What makes IAST useful:

1. Fewer false positives: Because IAST sees the code running in real time, it understands the context better. This reduces the number of false alarms compared to purely static or dynamic testing.
2. Immediate feedback: Developers get security feedback while they’re testing the app, making it easier to fix issues quickly.
3. Works in CI/CD pipelines: IAST fits well with modern development practices like DevOps and CI/CD, providing automated security checks during the development process.
4. Detailed insights: When it finds a vulnerability, IAST shows you exactly where the issue is in the code and under what conditions it happens. This makes debugging faster and easier.

IAST provides significant benefits in detecting security vulnerabilities during runtime, but limitations such as dependency on execution paths, test coverage, and potential performance overhead need to be considered when deciding on a security testing strategy. Combining IAST with SAST, DAST, and Software Composition Analysis (SCA) can provide a more comprehensive security testing approach.

In addition to scanning the application code base, you can implement IaC and policy-as-code scanning via SAST tools. This can help detect infrastructure misconfigurations and policy violations as they get committed to repositories instead of as they get deployed to production environments.

Terraform, OpenTofu, Pulumi, AWS CloudFormation, Kubernetes, and Ansible are all examples of modern Infrastructure as Code (IaC) tools that many organizations use and that Spacelift supports. They provide a way of controlling environments and infrastructure that live in AWS, Azure, Google Cloud, etc.

If misconfigured, this infrastructure can provide an entry point for malicious actors to infiltrate your cloud environments, resulting in significant problems.

You should ensure your SAST solution supports IaC scanning and/or look for standalone solutions that can scan your infrastructure specification files.

Read more about IaC scanning and policy as code scanning for vulnerabilities and see a list of the top IaC scanning tools.

Container technology has become extremely popular for hosting and deploying applications. The reason is simple: Containers make developing, deploying, and updating applications much faster and more efficient.

However, they’ve also introduced security concerns, such as the fact that many container images are based on other images, which creates a dependency. This is similar to using a third-party library in your application’s code — and we’ve seen what can happen in that situation. There have been many recent examples of organizations becoming vulnerable because of a security issue in one of their third-party dependencies.

By scanning containers and their images, we can identify potential security threats in the containers and their components. Components can include the software inside the container, the configurations for storage and networking, and even how the container interacts with the operating system and other containers.

General SAST tools offer this functionality, but you can also use more focused tools for this purpose. They can be directly integrated in the IDE, and they can scan as the container configuration files get committed for very quick feedback.

Container scanning is often considered part of SCA because it is closely linked to third-party dependencies. To round out this article, let’s talk about SCA in more depth.

Using appropriate open- or closed-source software Instead of reinventing the wheel every time you need to develop a new feature or product can save hours of hard work, but these dependencies can have security flaws. As soon as you pull a dependency into your project, you add that flaw to your codebase.

If an exploit is found for this security flaw, projects using that software can become vulnerable. This is not just theory. Many high-profile cases have occurred recently, including Log4Shell and SolarWinds.

To combat this growing problem, organizations can turn to Software Composition Analysis (SCA). This article will focus specifically on SCA scanning.

SCA typically involves:

• Container scanning
• Dependency scanning
• License compliance
• …and more

SCA scanning tools can help find known vulnerabilities in code dependencies, containers, and issues with licensing. We can integrate this form of tooling as a way of:

1. Documenting added dependencies, and
2. Preventing them from being added if something wrong is detected

For example, does the library you’re trying to use have a license that you would violate with our use case? Or does the framework you’re trying to add have known vulnerabilities that haven’t been fixed yet?

SCA recap and additional advice

Implementing all the security scannings we’ve discussed with SAST and scanning for SCA requires a tremendous amount of work, especially if you are solely responsible for security or you are part of a small security team.

SCA is very important, but it can take time to implement properly. You may want to focus on implementing SAST scanning, followed by CI/CD integration, and fine-tune that before moving on to implementing SCA scanning.

A platform like Spacelift can help you and your organization fully manage cloud resources within minutes. Spacelift is an infrastructure management platform that supports tools like OpenTofu, Terraform, Ansible, Pulumi, Kubernetes, and more. 

Security is a key Spacelift priority, so state-of-the-art security solutions, such as policy as code, encryption, Single Sign-On (SSO), MFA, and private worker pools, are embedded inside the product.

The power of Spacelift lies in its fully automated hands-on approach. Once you’ve created a Spacelift stack for your project, changes to the IaC files in your repository will be applied automatically to your infrastructure. 

Spacelift’s pull request integrations keep everyone informed of what will change by displaying which resources are going to be affected by new merges. Spacelift also allows you to enforce policies and automated compliance checks that prevent dangerous oversights from occurring.

Spacelift includes drift detection capabilities that periodically check your infrastructure for discrepancies compared to your repository’s state. It can then launch reconciliation jobs to restore the correct state, ensuring your infrastructure operates predictably and reliably.

With Spacelift, you get:

• Policies to control what kind of resources engineers can create, what parameters they can have, how many approvals you need for a run, what kind of task you execute, what happens when a pull request is open, and where to send your notifications
• Stack dependencies to build multi-infrastructure automation workflows with dependencies, having the ability to build a workflow that, for example, generates your EC2 instances using Terraform and combines it with Ansible to configure them
• Self-service infrastructure via Blueprints, or Spacelift’s Kubernetes operator, enabling your developers to do what matters – developing application code while not sacrificing control
• Creature comforts such as contexts (reusable containers for your environment variables, files, and hooks) and the ability to run arbitrary code
• Drift detection and optional remediation

If you want to learn more about Spacelift, create a free account today or book a demo with one of our engineers.

This article discussed many code testing approaches, and attempting to implement all of them simultaneously would fail. A better approach is to prioritize by creating a high-level roadmap and sharing it with the rest of the team. Designing this roadmap will help you think through everything and gain a deeper understanding of what it requires to implement, but it will also provide a clear picture for the rest of your team to deliver feedback on.

Iterate until you’ve got a prioritized roadmap that makes sense for your team and your organization. This will also help gather support from management earlier in the process because they’ll be able to see that you’ve got a well-thought-out approach.

Once you’ve got a structured roadmap, you can start adding what we’ve discussed in this article to gradually improve your DevSecOps pipeline and help your organization reduce risk.