AMTSO Publishes Guidelines For False Positive Testing

PRESS RELEASE

3rd December 2010 - The Anti-Malware Testing Standards Organization (AMTSO, www.amtso.org), an international organization that promotes improved methodologies for testing security products, has announced its guidelines for false positive testing.

False positives occur when a security product incorrectly flags a file or resource as malicious. Although false positives rarely have significant impact, the consequences of a severe incident can be more damaging than failing to detect a malicious file such as when access to one or more system-critical files is lost, as dramatically indicated by a recent high-profile false positive incident where up to 25,000 files are claimed to have been quarantined. (http://amtso.wordpress.com/2010/12/02/false-positives-testing-paper/)

The new guidelines suggest a series of criteria for testers to use in determining the magnitude of a false positive. Criticality looks at the impact of a false positive on the user. It categorizes the severity depending on the function of the affected resource within a system, network or application and assesses how critical it is to normal operation. Prevalence considers how many users would be impacted by a false positive - is it five or five thousand? Recoverability assesses how difficult it is to remediate the situation: has data been deleted and does the system need to be taken offline?

"False positives tend to have a greater visible impact on the customer than on a security product's protection, so it's surprising that not more anti-malware tests include false positives," says Mark Kennedy of Symantec, who introduced the guidelines on behalf of AMTSO in papers for the Virus Bulletin and AVAR Conferences. "In recent times, the introduction of proactive technologies such as behavior blocking and generic signatures have dramatically increased the likelihood of false positives. The problem with current tests is that they are frequently too simplistic in their approach, presuming that all non-malicious files are equally important. However, when you break down a file's specific function it's clear that it this is simply not the case."

Just as in its guidelines for testing detection rates of malicious files, AMTSO stresses that care must be taken to ensure that all samples to be tested are verified, that they are not misclassified and that the vendor has not added detection intentionally because it regards the file as "greyware", "possibly unwanted" and so on. AMTSO also recommends that testers make it clear when FP testing is performed in conjunction with malware detection testing, as this may bias the results.

The Guidelines for false positive testing are available for free download here: http://www.amtso.org/amtso-download-amtso-false-positive-testing-guidelines. html

###

About AMTSO AMTSO is comprised of 37 members, representing testers, vendors, academics and publishers involved in anti-malware research. Founded in 2008, AMTSO members have cooperatively developed and adopted standards, guidelines, educational materials, and a review analysis process aimed at improving the efficacy and reliability of anti-malware testing.

Media Contact: Sara Claridge Marylebone Media Relations [email protected]