Penetration testing, once considered a risky practice for the enterprise and even a tool for evil hacking purposes, is becoming more of an accepted mainstream process in the enterprise mainly due to compliance requirements, more automated, user-friendly tools -- and most recently, the imminent arrival of a commercial offering based on the popular open-source Metasploit tool.

Rapid7's purchase of the Metasploit Project last month and its hiring of the renowned creator of Metasploit, HD Moore, demonstrate just how far penetration testing has come during the past 18 months, security analysts say. While some organizations still confuse penetration testing with the more pervasive vulnerability scanning, which searches for and pinpoints specific vulnerabilities and weaknesses, penetration testing is finally about to enter a new phase of commercial deployment, experts say.

Penetration testing basically puts the tester in the shoes of a would-be attacker, using exploits and attack combinations against a network or application to find where the actual exploitable weaknesses lay.

"This is an exciting time because we're starting see even the edgy [penetration testing providers] look to the enterprise as a viable market," says Nick Selby, managing director of Trident Risk Management, a Dallas-based security and consultancy firm. "The technology is more mature so that the more experienced and skilled penetration testers have better toolsets than ever...and the less experienced ones can do more of the low-hanging fruit work."

Penetration testing traditionally has been the domain of white hat hackers, as well as script kiddies and even black-hat types. But as leading penetration testing vendors Core Security Technologies and Immunity Inc. have struck partnerships with top vulnerability scanning vendors and worked on developing more user-friendly versions of their tools, pen testing has begun to gain broader appeal within the enterprise. Core has partnerships with eEye, GFI, IBM, Lumension, nCircle, Qualys, and Tenable, while Immunity is teamed up with Tenable, as well, for instance.

"Pen testing has been a bit too edgy up until now to fully integrate with the more safe and steady-as-she-goes vulnerability assessment scanning," Selby says. "But that's changing: Over the coming year, we'll see better integration with the existing partnerships Core and Immunity have. And with the rapid integration of Rapid7 and Metasploit, vulnerability assessment and penetration testing will become better defined in the minds of run-of-the-mill security specialists and managers."

Rapid7 has said it plans to enhance its NeXpose vulnerability management products, as well as its penetration testing services, with Metasploit technology. Although Rapid7 is still looking at just how it will combine these technologies, the company is considering keeping Metasploit a separate product that's heavily integrated with its existing vulnerability scanning product and pen-testing services.

Corey Thomas, vice president of products and operations at Rapid7, said recently the goal is for Metasploit's exploit technology to help determine which vulnerabilities found by NeXpose are exploitable.

Rapid7's Moore, who is the chief security officer there, says the old worries of a pen test knocking a server or service offline have been superseded by bigger concerns of what an attacker could do. The risk of an attacker doing serious damage has more enterprises starting to use pen-testing tools, or to hire out pen-testers. "If you have let the malware in, you might as well let the pen tester in," Moore says.

Meanwhile, even the profile of today's pen tester is changing -- albeit less rapidly -- as well. Traditionally the domain of security specialists and hackers, pen-testing duties are starting to fall to others in the enterprise as well.

Ivan Arce, CTO of Core, says when Core first started out, it was an internal security expert or outside consultant who used its tools. "It was a more technical person with knowledge about exploits and attacks and who could manually choose actions one by one," Arce says. "Over time as we've added automation and ease of use, the barrier to entry was lowered. Today, it's not [always] necessarily a pen tester" using the tools, he says. It's a network security professional, a vulnerability specialist, a Web security specialist, a security user, an auditor, or a developer. "I see that trend expanding and increasing in the future," he says.

Selby says the most important thing about the enterprise user of a penetration testing tool is that he or she is not a "thrill seeker." "They have to be curious and creative, but just short of insurrectional. You don't want to turn a loose cannon around free in your network," Selby says. "You want to make sure the person has the intellectual curiosity and capability to run these tools without causing damage -- and [is] not [doing it] just for fun."

Meanwhile, PCI DSS has played a major role in making penetration testing more of a household name in the enterprise. PCI specifically requires penetration testing (PDF) of networks and applications either internally by an experienced pen tester within the enterprise or a third-party tester. Trident's Selby says it makes sense for enterprises -- especially those under PCI rules -- to run both internal penetration tests as well as hiring out external pen tests. "There's a great advantage to building [penetration testing] internally," namely that your organization is more informed when the external pen-testers do the job and provide their results, he says.

But penetration testing tools are still missing some key features to make them more appealing to a broader array of enterprises and users. Core's Arce says pen-testing tools in 2010 will "scale up" in size and in the number of times enterprises deploy them internally. And Rapid 7's Moore says reporting and analysis will improve in these tools: "The ability of scanning products to take it one step further, to should I care about that or not? Hopefully, we'll see some improvements in that," Moore says. "And more reporting/analysis so that when you find a something exploitable you can look around and find out what else you can do" with it, he says, noting that Core's product addresses some of this.

Ideally, the vulnerability scanner would be tied in with the pen-test exploit tools and automatically show you "where to focus" your efforts, Moore says.

Trident's Selby says workflow management and topology assessment need to be improved in pen-testing tools, as well as a better user interface. "Even in Core, which has a great user interface, you need a better graphical representation of where you are in the network and what you're looking at," Selby says.

Meanwhile, Core, Immunity, and Metasploit's penetration testing tools have evolved beyond the early days of mainly network pen tests, to testing Web applications and wireless networks for exploitable holes -- using both exploits and other hacks into weaknesses. "We're currently working on networks, operating systems, applications, client-side applications, users, wireless, and Web applications," Core's Arce says.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message.