Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Perimeter

3/22/2010
05:19 PM
John H. Sawyer
John H. Sawyer
Commentary
50%
50%

Automated Web Scanners Bring The Noise

One fish, two fish, red fish, skipfish...huh? That was my initial thought. Skipfish is definitely an interesting name for a Web application security scanner. It sounds like it came straight out of a Dr. Seuss book, but instead it's an awesome new tool from Michal Zalewski and Google.

One fish, two fish, red fish, skipfish...huh? That was my initial thought. Skipfish is definitely an interesting name for a Web application security scanner. It sounds like it came straight out of a Dr. Seuss book, but instead it's an awesome new tool from Michal Zalewski and Google.Web security tools aren't anything new to Michal, who previously wrote ratproxy, a passive semiautomated Web application security-auditing tool that acts as a proxy to your Web browser while performing checks. Initial reports state that skipfish is fast but noisy. After some testing of my own, it is screaming fast. And as you might expect from an automated scanner, it is also very noisy.

So here's my question to you: Isn't that what you'd expect from an automated Web application scanner? I've used numerous open-source Web scanning tools from simple scanners, like Nikto, to more powerful and complex tools, like w3af. I even use a commercial solution. Whenever you're scanning and using brute-force file and directory discovery, it's going to be noisy, with tens to hundreds of thousands of 404 (file missing) error messages as it's attempting to discovery interesting locations not linked from the main site.

After reading through some of the comments in blogs and on Twitter about Skipfish's "noisiness," I started comparing a couple of other tools that have brute-force file and directory discovery features. Nikto scans for more than 6100 dangerous files/CGIs and other checks. The documentation even states that it is not designed to be overly stealthy. No big surprise there.

In looking at the "dictionaries" included with skipfish, the complete config includes 1,802 "words" that will get combined with 92 "extensions" to become one monster list of files and directories that get probed.

DirBuster is another well-known discovery brute-forcing tool. I've only used it once in Web penetration testing training session so I forgot just how cool it was because of the way the directory lists are created. As the tool's overview page says, "tools of this nature are often as only good as the directory and file list they come with. A different approach was taken to generating this. The list was generated from scratch, by crawling the Internet and collecting the directory and files that are actually used by developers!"

That just rocks, and it gives DirBuster the largest list I've come across in a Web scanner. My next test will be to integrate the two and see if I can make Skipfish's brute-forcing capability even better, although at some point I can foresee it getting possibly too long making the scan unbearingly long.

Skipfish is definitely a cool tool with a bright future, but don't forget what it is and what it does. Its inherent nature is going to cause it to be a noisy tool because it scans a Web app looking for certain files and all those requests will (or should) be logged, which can generate some hefty log files.

Quick update: The documentation for the dictionaries included with Skipfish discusses the DirBuster approach at the very bottom of the document and why Skipfish doesn't use it. Basically, "keywords related to popular websites and brands are heavily overrepresented," and "some of the most interesting security-related keywords are not commonly indexed." So, I likely won't bother testing DirBuster's list with Skipfish and will trust Michals's design decision. :)

John H. Sawyer is a senior security engineer on the IT Security Team at the University of Florida. The views and opinions expressed in this blog are his own and do not represent the views and opinions of the UF IT Security Team or the University of Florida. When John's not fighting flaming, malware-infested machines or performing autopsies on blitzed boxes, he can usually be found hanging with his family, bouncing a baby on one knee and balancing a laptop on the other. Special to Dark Reading.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
News
Former CISA Director Chris Krebs Discusses Risk Management & Threat Intel
Kelly Sheridan, Staff Editor, Dark Reading,  2/23/2021
Edge-DRsplash-10-edge-articles
Security + Fraud Protection: Your One-Two Punch Against Cyberattacks
Joshua Goldfarb, Director of Product Management at F5,  2/23/2021
News
Cybercrime Groups More Prolific, Focus on Healthcare in 2020
Robert Lemos, Contributing Writer,  2/22/2021
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you today!
Flash Poll
Building the SOC of the Future
Building the SOC of the Future
Digital transformation, cloud-focused attacks, and a worldwide pandemic. The past year has changed the way business works and the way security teams operate. There is no going back.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2021-22861
PUBLISHED: 2021-03-03
An improper access control vulnerability was identified in GitHub Enterprise Server that allowed authenticated users of the instance to gain write access to unauthorized repositories via specifically crafted pull requests and REST API requests. An attacker would need to be able to fork the targeted ...
CVE-2021-22862
PUBLISHED: 2021-03-03
An improper access control vulnerability was identified in GitHub Enterprise Server that allowed an authenticated user with the ability to fork a repository to disclose Actions secrets for the parent repository of the fork. This vulnerability existed due to a flaw that allowed the base reference of ...
CVE-2021-22863
PUBLISHED: 2021-03-03
An improper access control vulnerability was identified in the GitHub Enterprise Server GraphQL API that allowed authenticated users of the instance to modify the maintainer collaboration permission of a pull request without proper authorization. By exploiting this vulnerability, an attacker would b...
CVE-2020-10519
PUBLISHED: 2021-03-03
A remote code execution vulnerability was identified in GitHub Enterprise Server that could be exploited when building a GitHub Pages site. User-controlled configuration of the underlying parsers used by GitHub Pages were not sufficiently restricted and made it possible to execute commands on the Gi...
CVE-2021-21353
PUBLISHED: 2021-03-03
Pug is an npm package which is a high-performance template engine. In pug before version 3.0.1, if a remote attacker was able to control the `pretty` option of the pug compiler, e.g. if you spread a user provided object such as the query parameters of a request into the pug template inputs, it was p...