When it comes to detecting vulnerabilities in mission-critical applications, security professionals often find themselves in a bind. These are usually the applications that the enterprise can least afford to suffer a hack. But at the same time, they are also the applications whose owners are most likely to balk at security testing or scanning probes while they're live. These opponents to vulnerability scans on production applications point to the near-infinitesimal tolerance for downtime or disruption as reason enough to leave well enough alone. But according to security professionals, someone will eventually find those vulnerabilities and if the organization doesn't do it first odds are it is the bad guys who will ferret out the flaws.

"Scanning production applications is a challenging proposition, as availability and data integrity are paramount for organizations," says Wolfgang Kandek, CTO of Qualys. "However, security has become as important as availability, and anyway, attackers are doing their own scanning to map out the assets of the organizations, whether we like it or not."

The fact is that organizations can't fix what they don't know about, and when it comes to many of their most important production applications, many enterprises just don't have the visibility to discover potentially disastrous flaws.

[How can you start instituting a secure software development lifecycle? See 10 Commandments Of Application Security.]

"If you're not scanning production systems for vulnerabilities, you're almost guaranteed to leave some risk to your most critical assets undiscovered," says Tim Erlin, director of IT security and risk strategy for nCircle. "There is no way to manage and mitigate undiscovered risk. The trend is definitely towards more frequent scanning, but there's no doubt that there are multi-billion dollar companies out there that don't have a consistent scanning program."

Part of the pushback that occurs around scanning running applications is that those organizations with a strong focus on application security have tended to throw their resources at the front-end of the application lifecycle, says John Weinschenk, CEO of Cenzic. While that may be the most cost-effective method of addressing application security, the truth is that it doesn't address the realities of applications running in a production environment

"When you look at an enterprise, traditionally how they have handled application security has been focused at the pre-production level," he says. "But with every company that's been hacked that you've seen in the paper recently, its always the production applications that got hacked."

What's more, it doesn't address the dynamic nature of the vulnerability landscape. "Even though your development team might have given you an application that has no vulnerabilities that are known as of today, it doesn't mean a week from now that that application is still secure," Weinschenk says.

However, security professionals shouldn't sugar coat the potential risks of scanning production applications. Downtime worries aren't completely unfounded.

"You can crash it, you can take it down, you can corrupt the database," Weinschenk says. "You can do a lot of bad things to it, which, by the way, is no different than what a hacker can do to you in real life."

In particular, security organizations must be "very careful" about the way they probe legacy applications, says Vinnie Liu, managing partner for Stach & Liu. Nevertheless, most organizations can greatly reduce the risks with a number of best practices. First among them is to be mindful of when tests are done, he says.

"Folks sometimes forget that if you are testing an app you need to check for specific blackout dates or windows that might occur on certain days where especially sensitive transactions occur, like end of the month payroll on an accounting system," he says. "It's an easy one to overlook because most people just focus on either during business hours or off business hours."

He also highly recommends that security practitioners do a good job of mapping out applications prior to automated testing to look for troublesome functions that might be better suited for manual tests.

"When running a tool a lot of them crawl through the application and exercise all of the functionality in it and some of the functionality might be remove or delete a profile or delete my accounts or delete my entries," Liu says. "So it is important to go through and survey the app first to get a lay of the land."

Organizations may find that the best bet is to ease their way into more comprehensive scans through lightweight discovery scans or monitoring mode.

"The best practice for continuous monitoring is to run scans as often as possible, at least at a weekly level," Kandek says. "Scans should initially be light, in a 'discovery mode' that has minimal impact on infrastructure and applications but allows for accurate enumeration. Once machines and applications are found, scans should become comprehensive."

Additionally, says Weinschenk, organizations could look for vendors that have integrations into virtual systems that would allow them to test a virtual copy of the application rather than the real systems.

"You can do a snapshot or virtual image copy of your production applications and assess that virtual copy," he says.

Weinschenk says that larger organizations may even go so far as to be "old school" about it and stand up a mirrored environment specifically to test applications regularly, a scenario that makes sense for large enterprises like financial organizations that have robust staging environment.

Regardless of how it is done, he believes that it must be done to truly secure the applications businesses depend upon.

"If companies want to stay secure, they're going to have to start testing their production apps," Weinschenk says. "There's no excuse for a company only to test in pre-preduction because this threat model is constantly moving."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.