Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Vulnerabilities / Threats

12/2/2020
02:00 PM
Alex Haynes
Alex Haynes
Commentary
50%
50%

Automated Pen Testing: Can It Replace Humans?

These tools have come a long way, but are they far enough along to make human pen testers obsolete?

Cybersecurity automation has increased dramatically in the past few years, but penetration testing has remained stubbornly immune. While crowdsourced security evolved as an alternative to pen testing in the past 10 years, it's not based on automation but simply throwing more humans at a problem (and, in the process, creating its own set of weaknesses). Recently though, automated pen-testing tools have advanced to a point where they are usable under certain conditions. This begs the question: Can these tools replace human pen testers?

I've spent a great deal of the past year testing these tools and comparing them in like-for-like tests against human pen testers, but the caveat is that these automation tools are improving at a phenomenal rate, so this analysis may already be out of date before you read it.

Related Content:

Vulnerability Prioritization Tops Security Pros' Challenges

The Changing Face of Threat Intelligence

New on The Edge: ISP Security: Do We Expect Too Much?

How Automated Pen Testers Work
These tools "deliver" a pen test by using either an agent or a virtual machine (VM) that simulates the pen tester's laptop and/or attack proxy plugging into your network. The pen testing bot then performs reconnaissance on its environment by doing identical scans as a human would do — running a vulnerability scan with a preferred tool or just a ports-and-services sweep with Nmap or Masscan. Once the automated tools have established where they sit within the environment, they will filter through what they've found. This is where the similarities to vulnerability scanners end.

Vulnerability scanners simply list a series of vulnerabilities and potential vulnerabilities they find — with no context about their exploitability. They simply regurgitate Common Vulnerabilities and Exposures (CVE) references and Common Vulnerability Scoring System (CVSS) scores. They sometimes paste "proof" that the system is vulnerable — but they don't cater well to false positives. The automated pen-testing tools then choose the "best" system from these targets to take over, making decisions based on ease of exploit, noise, and such factors. For example, if the bot finds a Windows machine that is vulnerable to EternalBlue, it may favor this over brute-forcing an open SSH port that authenticates with a password, as it's a known quantity and a much faster and easier exploit.

Once the tool gains a foothold, it will propagate through the network, mimicking how a human pen tester or attacker would do it. The difference is that it installs a version of its own agent on the exploited machine and continues its pivot from there. It starts the process again from scratch, but this time it forensically investigates the machine to give it more ammunition to continue its journey through your network. This is where it could dump password hashes or look for hard-coded credentials. It then adds this to its repertoire for the next round. While previously it might have just repeated the scan/exploit/pivot, this time it will try a "pass the hash" attack or connect to an SSH port using the key it just pilfered. Then, it pivots again and so on.

If you notice a lot of similarities to how human pen testers behave, you're absolutely right — a lot of this is exactly how pen testers (and to a lesser extent, attackers) behave. The tool sets are similar and the techniques and vectors used to pivot are identical in many ways. So, what's different?

Advantages of Automated Pen Testing
Automation offers a few advantages over the aging pen-testing methodology (and the equally chaotic crowdsourced methodology).

First, the speed of the test and reporting is magnitudes faster, and the reports are surprisingly readable. (After conferring with some Qualified Security Assessors, I've verified they will pass the various PCI-DSS pen-testing requirements.) No more waiting days or weeks for a report drafted by human hands and a few rounds of QA before it's delivered into your hands.

This is one of the primary weaknesses of human pen tests today — continuous delivery means many reports are out of date before they are delivered. The environment has been updated multiple times since the test, which introduces new potential vulnerabilities and misconfigurations that weren't there during the pen test. This is why traditional pen testing is really a snapshot of your security posture at a point in time.

Automated pen-testing tools get around this limitation by running tests daily, twice daily, or on every change, and delivering a report almost instantly. This means you can pen test your environment and detect potentially exploitable configuration changes on a daily basis, rather than relying on a report delivered weeks later.

Automation's second advantage is the entry point. While you may give a human pen tester a specific entry point into your network, an automated tool can run the same pen test multiple times from different entry points to uncover vulnerable vectors and monitor various impact scenarios depending on the entry point. While this is theoretically possible with a human, it would require a huge budget to pay each time for a different test.

Disadvantages of Automated Pen Testing
Automated pen-testing tools do have downsides. First, they don't understand web applications — at all. While they will detect something like a web server at the ports/services level, they won't understand that you have an insecure direct object reference (IDOR) vulnerability in your internal API or a server-side request forgery (SSRF) in an internal webpage that a human pen tester can use to pivot further. This is because today's web stack is complex, and even specialist scanners (like web application scanners) have a hard time detecting vulnerabilities that aren't low-hanging fruit (such as XSS or SQLi).

This leads to another weakness in automated pen-testing tools: You can only use them "inside" the network. As most exposed company infrastructure is web-based, and automated pen-testing tools don't understand these, you'll still need to stick to good ol' fashioned humans for pen testing from the outside.

Where Things Stand
This technology shows a lot of promise, but it's early days. While it can't make human pen testers redundant just yet, it has a role in meeting today's offensive security challenges that can't be met without automation.

Alex Haynes is a former pentester with a background in offensive security and is credited for discovering vulnerabilities in products by Microsoft, Adobe, Pinterest, Amazon Web Services and IBM. He is a former top 10 ranked researcher on Bugcrowd and a member of the Synack ... View Full Bio
 

Recommended Reading:

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Aviv Cohen
50%
50%
Aviv Cohen,
User Rank: Apprentice
12/20/2020 | 12:24:28 PM
Automated Peneration Testing is the future
I agree with Alex Haynes in this forward-looking article 100%. It's time for defenders to test their security efficacy with software as DIY and save on 3rd party expenses. 
Attackers Leave Stolen Credentials Searchable on Google
Kelly Sheridan, Staff Editor, Dark Reading,  1/21/2021
How to Better Secure Your Microsoft 365 Environment
Kelly Sheridan, Staff Editor, Dark Reading,  1/25/2021
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Write a Caption, Win an Amazon Gift Card! Click Here
Latest Comment: I can't find the back door.
Current Issue
2020: The Year in Security
Download this Tech Digest for a look at the biggest security stories that - so far - have shaped a very strange and stressful year.
Flash Poll
Assessing Cybersecurity Risk in Today's Enterprises
Assessing Cybersecurity Risk in Today's Enterprises
COVID-19 has created a new IT paradigm in the enterprise -- and a new level of cybersecurity risk. This report offers a look at how enterprises are assessing and managing cyber-risk under the new normal.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2021-21275
PUBLISHED: 2021-01-25
The MediaWiki "Report" extension has a Cross-Site Request Forgery (CSRF) vulnerability. Before fixed version, there was no protection against CSRF checks on Special:Report, so requests to report a revision could be forged. The problem has been fixed in commit f828dc6 by making use of Medi...
CVE-2021-21272
PUBLISHED: 2021-01-25
ORAS is open source software which enables a way to push OCI Artifacts to OCI Conformant registries. ORAS is both a CLI for initial testing and a Go Module. In ORAS from version 0.4.0 and before version 0.9.0, there is a "zip-slip" vulnerability. The directory support feature allows the ...
CVE-2021-23901
PUBLISHED: 2021-01-25
An XML external entity (XXE) injection vulnerability was discovered in the Nutch DmozParser and is known to affect Nutch versions < 1.18. XML external entity injection (also known as XXE) is a web security vulnerability that allows an attacker to interfere with an application's processing of XML ...
CVE-2020-17532
PUBLISHED: 2021-01-25
When handler-router component is enabled in servicecomb-java-chassis, authenticated user may inject some data and cause arbitrary code execution. The problem happens in versions between 2.0.0 ~ 2.1.3 and fixed in Apache ServiceComb-Java-Chassis 2.1.5
CVE-2020-12512
PUBLISHED: 2021-01-22
Pepperl+Fuchs Comtrol IO-Link Master in Version 1.5.48 and below is prone to an authenticated reflected POST Cross-Site Scripting