"It's not like network scanning where you go to an IP address and scan the network," says Danny Allan, director of security research for IBM Rational Software, which sells the AppScan vulnerability scanner. "This is not just a point-and-click product."
Web application vulnerability scanning -- also known as "black box" testing (as opposed to source-code scanning, or white-box testing) -- touches on various levels, transactions, and interactions associated with a Web application. And it requires an experienced hand to run it in order to get the most out of the process of detecting security flaws in Web applications, security experts say.
"The people who are running the scanner matter a lot more than the scanner itself. These are not simple hammers anyone can use. They require the operator to have a significant level of Web security knowledge," says Jeremiah Grossman, CTO of WhiteHat Security, a Web security services firm.
Another misconception about these devices is that the more vulnerabilities they find, the better they are. "Many people go by vuln counts in Web scanners, which is incorrect," notes Caleb Sima, CTO of the application security center at HP, which sells the WebInspect Web app scanner. That's because some products lump together multiple iterations of a specific vulnerability. If one scanner finds 12 SQL injection flaws, and another finds five, it doesn't mean the second one is necessarily missing more bugs, Sima says.
Consider, then, how a scanner counts vulnerabilities rather than how many times it finds SQL injection bugs. Eve more important, IBM's Allan says, is the underlying coding problem that caused the vulnerability. "You may have one cross-site scripting vulnerability and 80 different ways to exploit it," he says. "You need to focus not on the [vulnerability] issue, but on why it happened...that helps prevent security issues from happening again in the future."
That said, it's not so simple to compare these products head-to-head. A European security researcher's recently released test results comparing three major Web app scanners highlighted those challenges given the differences in their approaches. Anantasec performed vulnerability scans against several applications using Acunetix WVS version 6.0, IBM Rational AppScan Version 7.7.620 Service Pack 2, and HP WebInspect Version 7.7.869. He concluded that Acunetix performed the best overall, but as a second layer of analysis, he also used Acunetix's AcuSensor, which looks at source code using a form of white-box testing. So it wasn't actually an apples-to-apples comparison, experts say.
So what should you look for when selecting a Web application vulnerability scanner? IBM's Allan says to first look at how well they test for known vulnerabilities, conceding that most products are fairly equal in the regard. "Most products have similar capabilities in the testing," he says.
Another important feature is the ability to maintain your login state during a scan so that if the person running the scan gets logged out during a test, he doesn't have to start all over again. "The ability to login and maintain login state effectively is hugely important because if the scanner cannot [do so], the scan is invalid because the functionality [would remain] untested," WhiteHat's Grossman says.
It helps to know the features and functions of the scanner before you test it so you can get the most out of your test -- and prevent any problems. Don't just rely on a demo test the vendor provides, either. Test it against your own Website apps.
Not all products have all of their features on by default, so learn about the options and don't just enable them all without considering the impact on your specific application. "Turning on all security tests and scan configuration can be damaging," IBM's Allan notes.
Meanwhile, HP's Sima says many of his company's enterprise customers are going to the next level -- performing recurring Web scanning. "This is where the majority of scanning is done on an after-production scale," he says. "This is a scan of a high-level policy on all Web-facing properties that hold the company's branding, including off-site. It allows the company to identify its risk and quickly identify glaring holes in things it may or may not control, but can cause damage from a branding perspective -- or worse."
One aspect security experts agree on is that black-box testing alone isn't enough. "Black-box testing should be used together with a sensor technology like AcuSensor, and you should also add source-code review into the mix," Anantasec says.
Nick Selby, vice president and research director with The 451 Group, says to look out for false positives and false negatives with these tools, as well.
"At their most basic, Web application vulnerability assessment tools can give you a basic snapshot of known vulnerabilities in Web applications, and some kind of explanation of what it finds, what the severity of a given vuln is, and tips and suggestions on how to fix them," Selby says. "Because this is a very fluid environment, though, false positives and false negatives abound. That's why we suggest vulnerability scanning services in addition to the software -- many firms including White Hat and Cenzic offer these [services]."
Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message