Shortcomings Identified Among Security Vendors in Latest Evaluation of Advanced Threat Solutions

PRESS RELEASE

LONDON, UK (December 4, 2014) – Delta Testing, a specialist in anti-malware testing methodologies and research, today released a new report highlighting shortcomings in the detection rates of leading players in the advanced threat solution market.  Based on independent tests carried out by Delta Testing, the New Approach to Assessing Advanced Threat Solutions report, shows a significant discrepancy between the seven vendors tested, with FireEye achieving a 99.14 per cent detection rate compared to Fidelis 5.17 at the other end of the scale.
 
Unlike established information security tests which rely on legacy malware such as common viruses and worms or malware repositories, Delta Testing runs advance malware that has never before been detected by a product.  This malicious code is sourced from Delta’s network of incident response teams, security researchers and organisations that have been breached.  
 
By using malicious code already in the wild, yet recent enough that no signatures currently exist, products are exposed to an unpredictable environment.  This approach ensures that products that don’t have the ability to detect unknown malware using dynamic analysis can be easily identified. The approach of using actual malicious code along with actual packet captures is a clear advantage over using synthetic malware that, while lab-appropriate, doesn’t mimic real-life examples.
 
In the advanced threat protection solutions test, packet captures were stripped of customer data and replayed through the latest published software version of the advanced malware protection products.  All of the products were allowed time to analyse the packets in their virtual environments or perform lookups in their cloud or threat intelligence.  Each product’s interface was subsequently checked to see if an alert had been triggered to determine detection.   Detection rates were calculated as a percentage by dividing the number of detections against a total number of samples sent. 

{Table 1}
Commenting on the findings, Mark Thomas, commercial director, Delta Testing says, “It would appear that most mainstream vendors are on par when it comes to previously known attacks.  However, as soon as you move towards a methodology that exposes the products to fresh malware samples taken from actual customer environments, detection rates start to show significant variations based on the technology and detection methodology used by the vendor. This clearly demonstrated FireEye’s leadership in detecting advanced/unknown threats.”  
 
“While only one malware sample tested was missed by all seven products, the fact that no vendor scored 100 per cent in identifying all of the samples shows that some attacks are still getting through the net.   As such, in order to meet the shortfall, organisations need to consider a holistic approach to security which doesn’t just rely on detection but addresses threat prevention and response tactics too.”

The full report can be viewed and downloaded here.