Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Endpoint

11/29/2018
08:10 PM
Connect Directly
Google+
Twitter
RSS
E-Mail
50%
50%

MITRE Changes the Game in Security Product Testing

Nonprofit has published its first-ever evaluation of popular endpoint security tools - measured against its ATT&CK model.

There were no grades, scores, or rankings, but today's official release by MITRE of the results from its tests of several major endpoint security products could signal a major shift in the testing arena.

MITRE, a nonprofit funded by the US federal government, in its inaugural commercial tests pitted each product against the well-documented attack methods and techniques used by the Chinese nation-state hacking group APT3, aka Gothic Panda, drawn from MITRE's widely touted - and open - ATT&CK model.

Endpoint detection and response (EDR) vendors Carbon Black, CrowdStrike, CounterTack, Endgame, Microsoft, RSA, and SentinelOne played blue team with their products against a red team of experts from MITRE. Unlike traditional third-party product testing in security, the ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) approach uses open standards, methods, and the vendors perform live defenses with their products.

The testing operates in a collaborative manner. "They invited the vendors in to help them drive the tool and show how they find [attacks]," says Mark Dufresne, vice president of research for Endgame. "MITRE was sitting right there, and the product wasn't just chucked over the wall" and tested like in many other third-party tests.

"It was a collaborative and conversational, versus transactional, model," he says.

MITRE tracks and documents how the tools are tuned and configured, and how they do or don't detect an offensive move by its "APT3" red team, for example. The results get published on MITRE's website for anyone to see and study.

"We really want this to be a collaborative process with the vendors; we want them to be part of the process," says Frank Duff, MITRE's lead engineer for the evaluations program. The goal is both to improve the products as well as share the evaluations publicly so organizations running those tools or shopping for them can get an in-depth look at their capabilities, according to Duff.

MITRE chose APT3's methods of attack, which include credential-harvesting and employing legitimate tools used by enterprises to mask their activity. Each step of the attack is documented, such as how the tool reacted to the attacker using PowerShell to mask privilege-escalation.  ATT&CK is based on a repository of adversary tactics and techniques and is aimed at helping organizations find holes in their defenses. For security vendors, ATT&CK testing helps spot holes or weaknesses in their products against known attack methods.

The collaborative and open testing setup represents a departure from traditional third-party testing. Vendors and labs traditionally have had an uneasy and sometimes contentious relationship over control of the testing process and parameters. Longtime friction in the security product test space erupted into an ugly legal spat in September, when testing firm NSS Labs filed an antitrust lawsuit against cybersecurity vendors CrowdStrike, ESET, and Symantec, as well as the nonprofit Anti-Malware Testing Standards Organization (AMTSO), over a vendor-backed testing protocol.

The suit claims the three security vendors and AMTSO, of which they and other endpoint security vendors are members, unfairly allow their products to be tested only by organizations that comply with AMTSO's testing protocol standard.

"The whole testing landscape is a real mess," Endgame's Dufresne says. "[But] as a vendor, it's important to be there."

NSS Labs, which is a member of AMTSO, was one of a minority of members that voted against the standard earlier this year; the majority of members support it and plan to adopt it. "Our fundamental focus is if a product is good enough to sell, it's good enough to test. We shouldn't have to comply to a standard on what and how we can test," said Jason Brvenik, chief technology officer at NSS Labs, in an interview with Dark Reading after the suit was filed.

Traditional third-party tests, such as those conducted by NSS Labs, AV-Test, and AV-Comparatives, focus mainly on file-based malware, Endgame's Dufresne explains, looking at whether the security product blocks specific malware. "We do participate in those ... but they truly miss a huge swath of overall attacker activity."

MITRE's Duff says there are different security product tests for different purposes. ATT&CK is all about openness and providing context to the evaluation, he says. "All different testing services have their own purpose, value, and approaches. This is our approach, and we are hoping it resonates to the public."

Even so, malware-based testing isn't likely to go away. "I think some buyers like to see a number" like those tests provide, Endgame's Dufresne says.

Greg Sim, CEO of Glasswall Solutions, says third-party testing was overdue for a change. Even when they garner high scores from the antimalware testing labs, some products continue to fail in real-world attacks, he says. "I think there's going to be a different model," he says, noting that his firm has run tests with MITRE, which it considers an example of a reputable third party for testing.

Another security vendor executive who requested anonymity says MITRE's entry into the testing arena came just at the right time. "Emulating tradecraft of a known adversary, nation/state - for us on the vendor side, we’re saying, 'Hallelujah, they are doing it the right way,'" he says.

MITRE's new testing service represents new territory for the nonprofit, but that doesn't mean its federal government work will subside. Vendors pay a fee, which MITRE would not disclose. "MITRE historically has focused on doing testing of solutions for US government customers or sponsors. That role is not going anywhere," Duff notes.

MITRE hasn't yet set a timeframe for its next series of tests, he adds, but the team will pick another APT group to emulate. 

"We are just trying to get at the ground truth on these tools," Duff says.

Related Content:

 

Black Hat Europe returns to London Dec 3-6 2018  with hands-on technical Trainings, cutting-edge Briefings, Arsenal open-source tool demonstrations, top-tier security solutions and service providers in the Business Hall. Click for information on the conference and to register.

Kelly Jackson Higgins is the Executive Editor of Dark Reading. She is an award-winning veteran technology and business journalist with more than two decades of experience in reporting and editing for various publications, including Network Computing, Secure Enterprise ... View Full Bio
 

Recommended Reading:

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
COVID-19: Latest Security News & Commentary
Dark Reading Staff 7/6/2020
Ripple20 Threatens Increasingly Connected Medical Devices
Kelly Sheridan, Staff Editor, Dark Reading,  6/30/2020
DDoS Attacks Jump 542% from Q4 2019 to Q1 2020
Dark Reading Staff 6/30/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
How Cybersecurity Incident Response Programs Work (and Why Some Don't)
This Tech Digest takes a look at the vital role cybersecurity incident response (IR) plays in managing cyber-risk within organizations. Download the Tech Digest today to find out how well-planned IR programs can detect intrusions, contain breaches, and help an organization restore normal operations.
Flash Poll
The Threat from the Internetand What Your Organization Can Do About It
The Threat from the Internetand What Your Organization Can Do About It
This report describes some of the latest attacks and threats emanating from the Internet, as well as advice and tips on how your organization can mitigate those threats before they affect your business. Download it today!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-5595
PUBLISHED: 2020-07-07
TCP/IP function included in the firmware of Mitsubishi Electric GOT2000 series (CoreOS with version -Y and earlier installed in GT27 Model, GT25 Model, and GT23 Model) contains a buffer overflow vulnerability, which may allow a remote attacker to stop the network functions of the products or execute...
CVE-2020-5596
PUBLISHED: 2020-07-07
TCP/IP function included in the firmware of Mitsubishi Electric GOT2000 series (CoreOS with version -Y and earlier installed in GT27 Model, GT25 Model, and GT23 Model) does not properly manage sessions, which may allow a remote attacker to stop the network functions of the products or execute a mali...
CVE-2020-5597
PUBLISHED: 2020-07-07
TCP/IP function included in the firmware of Mitsubishi Electric GOT2000 series (CoreOS with version -Y and earlier installed in GT27 Model, GT25 Model, and GT23 Model) contains a null pointer dereference vulnerability, which may allow a remote attacker to stop the network functions of the products o...
CVE-2020-5598
PUBLISHED: 2020-07-07
TCP/IP function included in the firmware of Mitsubishi Electric GOT2000 series (CoreOS with version -Y and earlier installed in GT27 Model, GT25 Model, and GT23 Model) contains an improper access control vulnerability, which may which may allow a remote attacker tobypass access restriction and stop ...
CVE-2020-5599
PUBLISHED: 2020-07-07
TCP/IP function included in the firmware of Mitsubishi Electric GOT2000 series (CoreOS with version -Y and earlier installed in GT27 Model, GT25 Model, and GT23 Model) contains an improper neutralization of argument delimiters in a command ('Argument Injection') vulnerability, which may allow a remo...