David Maynor, a security researcher who used to work with SecureWorks, and Robert Graham, former chief scientist for ISS, have quietly launched a new security firm that tests and certifies the security of software, Dark Reading has learned.

The new Atlanta-based company, called Errata Security, plans to test all types of software products as well as managed security service offerings, both for the vendors and potential buyers of the products and services.

"We test for how much due diligence a vendor has put into security," says Maynor, CTO of the new startup. Graham is also president and CEO. "The earlier this kind of testing becomes involved in product development or the development of a service offering, the easier it is to fix the [security] mistakes as they happen, instead of after the fact."

Errata uses homegrown tools to provide security assessment scores and other information on just how secure a product is. "We alert our customers that if a hacker were auditing them for bugs, these are the things that stick out for them, the real problems," he says.

Errata isn't the first organization to offer security certification. ICSA Labs and The NSS Group have been security-testing and certifying products for some time, mainly using publicly available tools. (See EMC Gets Certification.) Most vendors use these services as stamps of approval for additional credibility in their security marketing.

But Errata says it will raise the bar with more stringent tests, using its own testing suites and custom-built exploits, for instance.

"We are also grading the products based on real-world scenarios," Maynor says. "When looking over some reports, the fact that a product stops 21 out of 25 randomly selected exploits might not mean much to a customer. [But] seeing that this product performs well or poorly in a scenario the customer can relate to -- like detecting machines infected with bots -- helps people understand better."

Thomas Ptacek, a researcher with Matasano Security, which helps vendors secure their products but doesn't provide certification, says he's skeptical of any certification processes. "The major labs claim they already do the kind of extensive testing that Errata will be doing. But they don't have anyone like Dave Maynor on staff," Ptacek says. "So the certifications we have today [with the major labs] are just very expensive stickers, available to anyone who ponies up the cash."

Trouble is, educated buyers ignore certifications, he says, and it's tough to explain to a mainstream buyer why existing certifications don't carry much weight. "Errata's trying to be an honest player in a crooked card game. That's a problem."

Maynor says he and Graham launched Errata because they felt there was a need for a trusted source of testing. "A lot of people are feeling that there really isn't a gold standard of testing."

Vendors that want to differentiate themselves may undergo such testing, but they are also understandably shy about any negative feedback published about their products, notes Mike Scher, general counsel and compliance architect for Nexum, an IT security VAR and MSSP, and former head of Neohapsis Labs.

Errata, meanwhile, analyzes binary code, not source code: "A lot of consulting companies [do] source-code analysis, but that doesn't really tell a nontechnical person how secure a product is," Maynor says.

Errata is offering different levels of testing -- fully automated, complete with a scoring system; a combination of automated testing and manual analysis; and a full security audit, which includes the first two levels of testing as well.

And in January, Errata will test and certify managed security service providers' services. "This is a departure from current security testing methods, where you will just run a list of known exploits past devices and give a score based on the number caught and missed," he says. "Our tests are more focused on real-world scenarios that are important to customers -- like can this MSSP service help me detect and shutdown an internal machine that has been infected with a rootkit and is spewing spam."

Errata hopes to provide a report card of how well MSSPs can protect their customers, including metrics to compare them with other MSSPs. "An example of the metrics we are looking for is the average response to critical activities like a host being compromised or detection of a denial-of-service attack," Maynor says. "This test will evaluate a MSSP total service, just not a particular product they are managing."

The company is so new that its Website is still under development but will be available soon, according to Errata officials.

— Kelly Jackson Higgins, Senior Editor, Dark Reading