Searching For Security's Yardstick

There’s an old saying in IT: You can’t manage what you can’t measure. If that’s true, however, security managers must be in a world of hurt.

Across this usually contentious security industry, there is violent agreement about two points: Security departments need better ways to prove that their organizations are safe, and there are no clear-cut numbers that definitively prove that point.

"So you’re in the management meeting, and the sales guy gives specific numbers about orders and gross revenue," says Steve Dauber, vice president of marketing at RedSeal, which makes software designed to monitor security posture. "The networking guy gives numbers about uptime and throughput and response time. Then it comes around to the security guy, and he says, 'Well, we didn’t get hacked today.'"

The basic problem, experts say, is that it’s tough to measure a negative. If security’s primary goal is to prevent outsiders from getting in -- and insider data from getting out -- what numbers are there to measure its success? The only clear metric is a negative: How many times has a compromise been discovered?

"The measure of success in security is that nothing bad happened," says Mike Rothman, an analyst for Securosis, a security consulting firm. "Your best day is going to be that zero bad things occurred. There's never going to be a measurement that shows that good things are happening."

If security is about prevention of leaks and attacks, then what metrics should security departments show their bosses to prove that they are doing their jobs well?

"I think you have to start with things you can control," says Scott Crawford, an analyst at Enterprise Management Associates, a consulting firm that focuses on systems and network management. "If you can't change the controls, then metrics won't do you any good."

Setting a security policy -- and the means to monitor it -- is a good place to start, Crawford says. "If you set a policy and there is a growing number of systems or users that are operating outside the policy, then that's something you can act on, either through education or through greater controls," he observes.

But security professionals should be wary of "dashboards" and artificial measures that don't have meaning for the specific business that their enterprises are in, says Gary Hinson, CEO of security consulting firm iSecT in New Zealand.

"Some companies begin with a long list of 'security things that can be measured' and then try to shoehorn them into some sort of metrics system or dashboard. That, to me, is the wrong way to go about things," Hinson says. "You don't design an aircraft cockpit's information systems with a list of things that can be readily measured on the aircraft. You start by asking what does the pilot need to know -- altitude, azimuth/heading, etc. -- and then prioritizing those things, organizing them into related groupings, and finally filling the dashboard.

"Then you get lots and lots of feedback from pilots about what is missing, superfluous, misleading, wrongly positioned, too big/too small, too annoying/too discreet, etc.," Hinson continues. "In other words, the metrics design process is very interactive, involving the system designers, instrumentation specialists, engineers, and pilots all working together to define, design, and refine the metrics system."

Next: Basic metrics