Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Vulnerabilities / Threats

02:00 PM
Alex Haynes
Alex Haynes

Automated Pen Testing: Can It Replace Humans?

These tools have come a long way, but are they far enough along to make human pen testers obsolete?

Cybersecurity automation has increased dramatically in the past few years, but penetration testing has remained stubbornly immune. While crowdsourced security evolved as an alternative to pen testing in the past 10 years, it's not based on automation but simply throwing more humans at a problem (and, in the process, creating its own set of weaknesses). Recently though, automated pen-testing tools have advanced to a point where they are usable under certain conditions. This begs the question: Can these tools replace human pen testers?

I've spent a great deal of the past year testing these tools and comparing them in like-for-like tests against human pen testers, but the caveat is that these automation tools are improving at a phenomenal rate, so this analysis may already be out of date before you read it.

Related Content:

Vulnerability Prioritization Tops Security Pros' Challenges

The Changing Face of Threat Intelligence

New on The Edge: ISP Security: Do We Expect Too Much?

How Automated Pen Testers Work
These tools "deliver" a pen test by using either an agent or a virtual machine (VM) that simulates the pen tester's laptop and/or attack proxy plugging into your network. The pen testing bot then performs reconnaissance on its environment by doing identical scans as a human would do — running a vulnerability scan with a preferred tool or just a ports-and-services sweep with Nmap or Masscan. Once the automated tools have established where they sit within the environment, they will filter through what they've found. This is where the similarities to vulnerability scanners end.

Vulnerability scanners simply list a series of vulnerabilities and potential vulnerabilities they find — with no context about their exploitability. They simply regurgitate Common Vulnerabilities and Exposures (CVE) references and Common Vulnerability Scoring System (CVSS) scores. They sometimes paste "proof" that the system is vulnerable — but they don't cater well to false positives. The automated pen-testing tools then choose the "best" system from these targets to take over, making decisions based on ease of exploit, noise, and such factors. For example, if the bot finds a Windows machine that is vulnerable to EternalBlue, it may favor this over brute-forcing an open SSH port that authenticates with a password, as it's a known quantity and a much faster and easier exploit.

Once the tool gains a foothold, it will propagate through the network, mimicking how a human pen tester or attacker would do it. The difference is that it installs a version of its own agent on the exploited machine and continues its pivot from there. It starts the process again from scratch, but this time it forensically investigates the machine to give it more ammunition to continue its journey through your network. This is where it could dump password hashes or look for hard-coded credentials. It then adds this to its repertoire for the next round. While previously it might have just repeated the scan/exploit/pivot, this time it will try a "pass the hash" attack or connect to an SSH port using the key it just pilfered. Then, it pivots again and so on.

If you notice a lot of similarities to how human pen testers behave, you're absolutely right — a lot of this is exactly how pen testers (and to a lesser extent, attackers) behave. The tool sets are similar and the techniques and vectors used to pivot are identical in many ways. So, what's different?

Advantages of Automated Pen Testing
Automation offers a few advantages over the aging pen-testing methodology (and the equally chaotic crowdsourced methodology).

First, the speed of the test and reporting is magnitudes faster, and the reports are surprisingly readable. (After conferring with some Qualified Security Assessors, I've verified they will pass the various PCI-DSS pen-testing requirements.) No more waiting days or weeks for a report drafted by human hands and a few rounds of QA before it's delivered into your hands.

This is one of the primary weaknesses of human pen tests today — continuous delivery means many reports are out of date before they are delivered. The environment has been updated multiple times since the test, which introduces new potential vulnerabilities and misconfigurations that weren't there during the pen test. This is why traditional pen testing is really a snapshot of your security posture at a point in time.

Automated pen-testing tools get around this limitation by running tests daily, twice daily, or on every change, and delivering a report almost instantly. This means you can pen test your environment and detect potentially exploitable configuration changes on a daily basis, rather than relying on a report delivered weeks later.

Automation's second advantage is the entry point. While you may give a human pen tester a specific entry point into your network, an automated tool can run the same pen test multiple times from different entry points to uncover vulnerable vectors and monitor various impact scenarios depending on the entry point. While this is theoretically possible with a human, it would require a huge budget to pay each time for a different test.

Disadvantages of Automated Pen Testing
Automated pen-testing tools do have downsides. First, they don't understand web applications — at all. While they will detect something like a web server at the ports/services level, they won't understand that you have an insecure direct object reference (IDOR) vulnerability in your internal API or a server-side request forgery (SSRF) in an internal webpage that a human pen tester can use to pivot further. This is because today's web stack is complex, and even specialist scanners (like web application scanners) have a hard time detecting vulnerabilities that aren't low-hanging fruit (such as XSS or SQLi).

This leads to another weakness in automated pen-testing tools: You can only use them "inside" the network. As most exposed company infrastructure is web-based, and automated pen-testing tools don't understand these, you'll still need to stick to good ol' fashioned humans for pen testing from the outside.

Where Things Stand
This technology shows a lot of promise, but it's early days. While it can't make human pen testers redundant just yet, it has a role in meeting today's offensive security challenges that can't be met without automation.

Alex Haynes is a former pentester with a background in offensive security and is credited for discovering vulnerabilities in products by Microsoft, Adobe, Pinterest, Amazon Web Services and IBM. He is a former top 10 ranked researcher on Bugcrowd and a member of the Synack ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Threaded  |  Newest First  |  Oldest First
Aviv Cohen
Aviv Cohen,
User Rank: Apprentice
12/20/2020 | 12:24:28 PM
Automated Peneration Testing is the future
I agree with Alex Haynes in this forward-looking article 100%. It's time for defenders to test their security efficacy with software as DIY and save on 3rd party expenses. 
7 Old IT Things Every New InfoSec Pro Should Know
Joan Goodchild, Staff Editor,  4/20/2021
Cloud-Native Businesses Struggle With Security
Robert Lemos, Contributing Writer,  5/6/2021
Defending Against Web Scraping Attacks
Rob Simon, Principal Security Consultant at TrustedSec,  5/7/2021
Register for Dark Reading Newsletters
White Papers
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you today!
Flash Poll
How Enterprises are Developing Secure Applications
How Enterprises are Developing Secure Applications
Recent breaches of third-party apps are driving many organizations to think harder about the security of their off-the-shelf software as they continue to move left in secure software development practices.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2021-05-13
Piwigo 11.4.0 allows admin/user_list_backend.php order[0][dir] SQL Injection.
PUBLISHED: 2021-05-13
The Flask-Caching extension through 1.10.1 for Flask relies on Pickle for serialization, which may lead to remote code execution or local privilege escalation. If an attacker gains access to cache storage (e.g., filesystem, Memcached, Redis, etc.), they can construct a crafted payload, poison the ca...
PUBLISHED: 2021-05-13
Bitcoin Core 0.12.0 through 0.21.1 does not properly implement the replacement policy specified in BIP125, which makes it easier for attackers to trigger a loss of funds, or a denial of service attack against downstream projects such as Lightning network nodes. An unconfirmed child transaction with ...
PUBLISHED: 2021-05-13
The HTMLSanitizer class in html-sanitizer.ts in all released versions of the Aurelia framework 1.x repository is vulnerable to XSS. The sanitizer only attempts to filter SCRIPT elements, which makes it feasible for remote attackers to conduct XSS attacks via (for example) JavaScript code in an attri...
PUBLISHED: 2021-05-13
An information disclosure vulnerability in ILIAS before 5.3.19, 5.4.12 and 6.0 allows remote authenticated attackers to get the upload data path via a workspace upload.