You know that vulnerability assessment and penetration testing are key tools for evaluating the security of your infrastructure. But did you know that these technologies and services also can cause problems in your network as well?

Experts say that, on occasion, they have seen vulnerability assessment and pen testing services wreak havoc in enterprises. Such tests can inadvertently crash routers, switches, and printer devices. In worst-case scenarios, these products have caused whole networks to crash, and penetration tests have even exposed sensitive company data.

These events are rare, and they shouldn't deter enterprises from asking a third party to test the effectiveness of their defenses, experts say. But there are risks in using such services, and more enterprises should be aware of them, they warn.

Vulnerability assessment scans look for holes and flaws to see where patches are installed (or not) and whether a system can be exploited. It provides a wide view of your security posture. "A vulnerability assessment shows you where the problems might be and penetration test tells you what the problems are -- just the ones used to gain access," says HD Moore, head of the Metasploit Project.

Penetration testing is a more targeted test, where a third party tries to break into your systems to find and exploit flaws, many of which aren't the typical vulnerability. "It's things like configuration problems, bad passwords, and custom buggy software," says Moore. "All it takes is one 'bug' to succeed with a penetration test, since the weakest link is all that matters."

The best practice is to first run a vulnerability assessment, update your patches based on the findings, and then have a penetration tester come in and do his or her thing. Sean Kelly, business technology consultant for Consilium1, which performs both types of testing for its clients, says penetration testing is more of a "proof of concept" while vulnerability assessment can help weed out false positives.

An obvious downside of penetration testing is that it's more intrusive. It means letting a white hat attacker view your sensitive data. "An organization may not want you viewing their CEO's email or capturing the flag," Kelly says.

Penetration tests are aggressive and can crash a server or other systems, he says. Penetration tests also tend to focus on a specific point of entry -- not the overall security picture, as vulnerability assessments do.

Moore says even an ethical hacker can inadvertently do damage in a pen test. "They may launch an exploit that normally works perfectly and leaves no traces, but something causes it to go in a loop and take down the server," he says. "Or, in the worst case, corrupt the hard drive."

That's why clients should go over the fine print before inking a pen testing deal. "Before we sign a contract, most folks say that if we can get in without a denial-of-service attack or bringing down their network, go for it -- as long as the risk is minimal," says Steve Stasiukonis, vice president and founder of Secure Network Technologies.

Pen testing isn't for everyone, he says. "If you're doing a hardcore penetration test, you're throwing in the kitchen sink -- and that can be nasty stuff," says Stasiukonis. "Most people I know don't opt for it" unless the test can be conducted without any hostile attack.

Occasionally, vulnerability assessment scanning can also cause problems in the network. "In my experience, penetration testing is less intrusive, simply because the attacks are targeted and launched by a human that knows better," Moore says. With vulnerability scanning, some devices can "just fall over and die when being probed for a completely unrelated issue," he says.

When vulnerability assessments or pen tests do break something, they reveal an unexpected problem in the network. A few years ago, when one of Secure Network Technologies' researchers was running a freeware vulnerability scanning tool on a large banking client's network, it crashed the network switch. The scanning of thousands of addresses filled the switch's memory, says Stasiukonis. "It was bringing the memory of their core switch to a level it had never been."

Stasiukonis admits it made things uncomfortable with the client at first, but the end result was information they wouldn't otherwise have had. "They would never have known about the defective memory on the switch" without the vulnerability test, he says.

Marc Maiffret, CTO and chief hacking officer for eEye Digital Security, which sells vulnerability assessment tools, agrees. "It's a Catch-22: You're bringing down a router by vulnerability assessment, and the business should realize -- a bad guy can do the same thing. So they should fix it."

Maiffret says a customer running eEye's vulnerability scanning tool recently found out the hard way what can happen if you don't keep your router patches up to date: The scan knocked out a Cisco router, which it wasn't even testing for vulnerabilities. "This is all avoidable if companies keep their devices up to date," he says. "And if an automated scanner tool [can crash it], so can a bad guy."

It's really the older, more fragile devices and systems that are the problem, not the vulnerability assessment test itself, says Allen Wilson, vice president of research for SecureWorks, a managed security services provider that does vulnerability assessment and pen testing. "It depends on how robust the device is and how the network stack is," he says, and the chances of a scan hurting other devices is rare.

"In vulnerability assessment products, there are typically settings where you can tune it to be very careful, or to be more aggressive," such as whether to allow a denial of service test, he says.

Some vulnerability and pen-test clients are all for doing what it takes to find their weaknesses before the hackers do, even if it means devices falling over. Others prefer little or no disruption to their network during testing. "Either way, you need to work with the customer up front and make them aware of" potential risks, Wilson says.

In the end, it's what you do with the test results that counts: It's all about remediation. "You have to do something with the report," says eEye's Maiffret.

— Kelly Jackson Higgins, Senior Editor, Dark Reading