Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Vulnerabilities / Threats

3/19/2019
10:30 AM
Alex Haynes
Alex Haynes
Commentary
100%
0%

Crowdsourced vs. Traditional Pen Testing

A side-by-side comparison of key test features and when best to apply them based on the constraints within your budget and environment.

Crowdsourced security has recently moved into the mainstream, displacing traditional penetration-testing companies from what once was a lucrative niche space. While several companies have pioneered their own programs (Google, Yahoo, Mozilla, and Facebook), Bugcrowd and HackerOne now carve up the lion's share of what is a fast-growing market.

How does crowdsourced pen testing compare with traditional pen testing, and how does it differ in methodology? Does this disruptive approach actually make things better? Read on for a side-by-side comparison.

Time-Limited vs. Open-Ended Engagements
One of the major downsides of pen testing today is that it doesn't match the development speed of modern applications. Most companies pen test annually, but in today's environments, applications are updated frequently, sometimes once a day, and sometimes even more than that. This results in your pen test being merely a snapshot of your security posture at a particular point in time. That's it. Once you've updated your website or application, those findings are out of date, which means potential new vulnerabilities.

Crowdsourced pen tests are typically open ended, which maps better to how applications are built today and, most importantly, how attackers behave. An attacker can spend three to four months examining one of your assets if he pleases; a traditional pen tester doesn't have that luxury. On the other hand, crowdsourced pen testers do, and it shows as they dig up highly critical bugs from live sites they have been pen testing for years.

I once found a vulnerability that took me over 50 hours to find (way longer than a pen test), and the vulnerability gave me access to the internal company network as well as all its data. This company used to run pen tests, but what surprised me most was that its crowdsourced program had been open for a year without anyone finding this particular bug, which proves another point: The more eyeballs you throw at something, the more things you'll discover.

Proof-of-Concept vs. Theoretical Vulnerabilities
I've read dozens of pen-testing reports over the years filled with "junk" risk, where a vulnerability is listed as "high" just because a system is not on the latest patch but without showing how. When asked for a proof of concept on how this is exploitable, the report's authors usually remove that from the report. This is now referred to as pen-tester syndrome — making things appear worse than they actually are. Garbage such as missing HTTP headers with absolutely no context as to how or why they are exploitable also falls into this category. In a crowdsourced pen test, you will only get exploitable vulnerabilities with actionable proof of concept. This does wonders for preventing companies from chasing phantom risk and focusing their remediation where it matters. Crowdsourced security really shines in this respect.

Pay per Pen Test vs. Pay per Vulnerability
Pen testing, for now, has held its ground against crowdsourced security because it's cheap. Since you pay per day and a typical website will take you between four to five days, you know exactly how much you will pay up front, regardless of how many vulnerabilities are found. Crowdsourced pen tests, on the other hand, can vary, and because you have to pay both a platform fee and, on top of that, pay per vulnerability found, it can get expensive. While different providers now vary their models (some will charge just a platform fee so you don't pay per vulnerability), it can be difficult to budget for the pay-per-vulnerability approach.

Testing Different Types of Assets
If you want someone testing from "inside" your network in a traditional pen-testing environment, a pen tester physically turns up at your office and just plugs in his or her laptop. In a crowdsourced scenario, it can get messy. Some of the engagements I've participated in require VPN or proxy setups, and you're usually in a test environment, not a live environment with real users. This increases the cost for companies, not to mention doing this not just for one but dozens of testers. Other assets such as embedded and Internet of Things devices require a physical asset in hand, and while I have seen a few crowdsourced programs send out devices to testers in the mail, it's more convenient and cheaper just to hand a single device over to a pen tester. For now, if you want to test anything inside your network or an IoT device, pen testing is just more convenient and cheaper!

Salaried Employee vs. Disposable Resource
While rarely considered, there is a glaring difference between both crowdsourced and traditional pen tests: how people are rewarded. In a traditional pen test, you know that work is carried out by a salaried employee who is remunerated correctly and paid regardless of whether he or she finds vulnerabilities or not. It's likely this person has other "soft" benefits such as a pension plan and pen-testing tools paid for by the company, and probably gets regular training and sick pay.

Crowdsourced pen testers do not have any of that because they are paid per vulnerability. Referring back to my previous example of spending over 50 hours on a vulnerability, if I had turned up empty handed, I would have been rewarded nothing at all. Crowdsourced pen testers also have to fund their own training and their own tools. Want to test an iOS app? Better have your own test device set up. You're sick? Too bad. Pension plan? What's that? The crowdsourced industry is acutely aware of this criticism and has started offering standard flat fees for certain tests and certain researchers, so that if you don't find any vulnerabilities, you still get paid.

Related Content:

 

 

Join Dark Reading LIVE for two cybersecurity summits at Interop 2019. Learn from the industry's most knowledgeable IT security experts. Check out the Interop agenda here.

Alex Haynes is a former pentester with a background in offensive security and is credited for discovering vulnerabilities in products by Microsoft, Adobe, Pinterest, Amazon Web Services and IBM. He is a former top 10 ranked researcher on Bugcrowd and a member of the Synack ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
jayd3e
50%
50%
jayd3e,
User Rank: Apprentice
5/18/2019 | 7:02:46 AM
Re: Bug bounty program and maturity
Interesting post.  I noticed that you did not include Synack in your list of providers.  Any particular reason?  We have been using Synack for a while now and have found their team to professional, their service well-designed, and their security reasearchers to be top notch.  We look at the options you mentioned and ended up going with Synack.
jayd3e
50%
50%
jayd3e,
User Rank: Apprentice
5/18/2019 | 6:59:31 AM
Re: Bug bounty program and maturity
Interesting post.  I noticed that you did not include Synack in your list of providers.  Any specific reason?  We have seen a lot of benefit from using Synack and have found their team to be professional, their service well-designed, and their security researchers to be some of the best in the world.
Arkada
100%
0%
Arkada,
User Rank: Author
4/5/2019 | 4:36:28 AM
Re: Interesting, but maybe missing some key points...
Thanks for your comment. All valid points. Crowdsourced security isn't mutually exclusive to traditional pentesting but due to the cost it depends entirely on budget. Like you point out, many 'crowdsourced' pentesters have a day job - I myself am CISO during the day and crowdsourced hacker by night but the advantages you point are valid - a crowdsourced tester will literally have all the time in the world to find a vulnerability, but more importantly you'll get a nice proof of concept on how the vulnerability can be exploited in a real world scenario, which is where crowdsourced pentesting shines. Personally I tired of reading 'traditional' pentesting reports filled with 'your system is out of date, it's vulnerable' - when I enquire as to 'why?' it either gets taken off the report or I get a stock answer 'we prefer to play it safe!'. 
Arkada
50%
50%
Arkada,
User Rank: Author
4/5/2019 | 4:32:57 AM
Re: Bug bounty program and maturity
Thanks for your comment. My take is yes, you do need a more a more 'mature' security level by virtue of the fact that you'll end up paying lots of bounties out to researchers for vulnerabilities that you could have discovered yourself. You need to be at least running regular web application scans and had a few pentests so that you've picked all the 'low hanging fruit' from your application then probably engage a crowdsourced test so that you can pick up the more esoteric ones that aren't going to be found in a pentest. This is if you had a nice fat budget you can use both traditional and crowdsourced in conjunction which each other - they aren't mutually exclusive.
Arkada
50%
50%
Arkada,
User Rank: Author
4/5/2019 | 4:30:52 AM
Re: Missed synack from that list
Thanks for your comment. You're right, there are a quite a few more platforms, I am myself a member of the Synack Red Team also so it is a valid point. Cobalt, Federacy, Yogosha, and even Hacken are all platforms to an extent, I just didn't want this article to become more vendor focused so I picked the 'big two'. 
KevinStanley
50%
50%
KevinStanley,
User Rank: Apprentice
3/25/2019 | 12:53:20 PM
Re: Bug bounty program and maturity
Agreed. Some key points are missing in my opinion.
KevinStanley
50%
50%
KevinStanley,
User Rank: Apprentice
3/25/2019 | 12:52:27 PM
Re: Bug bounty program and maturity
Agreed. Some key points are missing in my opinion
nologic
100%
0%
nologic,
User Rank: Apprentice
3/19/2019 | 8:11:16 PM
Missed synack from that list
Enjoyed the article! I think it's worth mentioning Synack on that list as well. I've worked on their platform. They do a good job at motivating researchers by paying out nice bounties, picking up interesting targets, respecting researchers and, of course, throwing great parties. In general through, these models are great for researchers who want to have more freedom in their day to day activities. A regular pentesting firm will have it's own approaches which may create friction for an individual contributor. Especially, in today's cyber security talent shortage, I feel like it's a good time to be a hacker on Synack's platform or the others that you've mentioned.
CISO Dave
100%
0%
CISO Dave,
User Rank: Apprentice
3/19/2019 | 6:44:13 PM
Interesting, but maybe missing some key points...
Interesting read Alex, thanks. I think on the whole you make some really valid points, but I think that maybe there are a couple of points worthy of highlighting. For me, Crowd sourced testing isn't a direct replacement for traditional pen testing. I think it offers an alternative, more dynamic way of performing more "real life" scenarios to the testing regime. You touch on the fact that traditional pen testing can be routine and not always providing the insight - that is what I like about the crowdsourced testing approach...I get to see a vulnerability and how it can be exploited in the real world. I too have had vulnerabilities highlighted by crowd sourced testers that have gone unnoticed for many years through our traditional pen test approach. I'm not sure how we would ever had seen it had it not been for the "independent" tester who was looking for his reward. Whilst I recognise that running crowd sourced programs internally can be challenging for the lack of reward, what I have seen (with people like Synack) is that their testers often have a day-job and use their personal motivation and drive to test themselves (and ultimately generate rewards) outside of their 9-5 work. I think the crowd source model is evolving too - again, using Synack as an example, they are looking at moving into the compliance space (through their Missions program) as well as creating indicators on your capability and speed to respond and remdiate the vulnerabilities they identify. These to me offer some additional value add that I wouldn't necessarily get from traditional pen test firms. But, as I say, the use of traditional pen testing still plays an important role in testing our estate and driving us to improve the security posture of the organisation, but I do believe there is a real market for good quality, robust crowd sourced testing to work alongside this.
pborghesi
50%
50%
pborghesi,
User Rank: Apprentice
3/19/2019 | 6:39:42 PM
Bug bounty program and maturity
Interesting post. However I had a discussion with my colleagues, and they state that before undertaking a bug bounty program you need to have a very mature security level or it can be too "risky". What is your take on that?
The Problem with Proprietary Testing: NSS Labs vs. CrowdStrike
Brian Monkman, Executive Director at NetSecOPEN,  7/19/2019
How Attackers Infiltrate the Supply Chain & What to Do About It
Shay Nahari, Head of Red-Team Services at CyberArk,  7/16/2019
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Current Issue
Building and Managing an IT Security Operations Program
As cyber threats grow, many organizations are building security operations centers (SOCs) to improve their defenses. In this Tech Digest you will learn tips on how to get the most out of a SOC in your organization - and what to do if you can't afford to build one.
Flash Poll
The State of IT Operations and Cybersecurity Operations
The State of IT Operations and Cybersecurity Operations
Your enterprise's cyber risk may depend upon the relationship between the IT team and the security team. Heres some insight on what's working and what isn't in the data center.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2019-10102
PUBLISHED: 2019-07-22
The Linux Foundation ONOS 1.15.0 and ealier is affected by: Improper Input Validation. The impact is: The attacker can remotely execute any commands by sending malicious http request to the controller. The component is: Method runJavaCompiler in YangLiveCompilerManager.java. The attack vector is: ne...
CVE-2019-10102
PUBLISHED: 2019-07-22
Frog CMS 1.1 is affected by: Cross Site Scripting (XSS). The impact is: Cookie stealing, Alert pop-up on page, Redirecting to another phishing site, Executing browser exploits. The component is: Snippets.
CVE-2019-10102
PUBLISHED: 2019-07-22
Ilias 5.3 before 5.3.12; 5.2 before 5.2.21 is affected by: Cross Site Scripting (XSS) - CWE-79 Type 2: Stored XSS (or Persistent). The impact is: Execute code in the victim's browser. The component is: Assessment / TestQuestionPool. The attack vector is: Cloze Test Text gap (attacker) / Corrections ...
CVE-2019-9959
PUBLISHED: 2019-07-22
The JPXStream::init function in Poppler 0.78.0 and earlier doesn't check for negative values of stream length, leading to an Integer Overflow, thereby making it possible to allocate a large memory chunk on the heap, with a size controlled by an attacker, as demonstrated by pdftocairo.
CVE-2019-4236
PUBLISHED: 2019-07-22
A IBM Spectrum Protect 7.l client backup or archive operation running for an HP-UX VxFS object is silently skipping Access Control List (ACL) entries from backup or archive if there are more than twelve ACL entries associated with the object in total. As a result, it could allow a local attacker to ...