Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Endpoint

2/14/2017
05:00 PM
Connect Directly
Twitter
LinkedIn
RSS
E-Mail
50%
50%

CrowdStrike Fails In Bid To Stop NSS Labs From Publishing Test Results At RSA

NSS results are based on incomplete and materially incorrect data, CrowdStrike CEO George Kurtz says.

A security vendor’s failed attempt to prevent a report containing details on its product’s capabilities from being released at the RSA conference in San Francisco this week has focused attention on the need for more widely accepted testing and validation services for security technologies.

The report is from NSS Labs, a company that bills itself as an independent security product-testing firm, which also sells its own threat protection platform.  NSS recently conducted security tests on Advanced Endpoint Protection (AEP) products from 13 vendors and released the results of its testing at RSA this week.

Each of the products, according to NSS, was tested against multiple attack threat vectors for their resilience against evasion techniques, their tendency to generate false positives, and other metrics. The products were then graded for their overall security effectiveness and total cost of ownership per protected endpoint agent and assigned a rating of “Recommended,” “Security Recommended” or “Caution”.

Security vendor CrowdStrike, one of two vendors in the report to garner a “Caution” rating filed a lawsuit in federal court in Delaware last week seeking to prevent NSS from releasing the report at RSA. In a legal brief, CrowdStrike claimed that NSS’ analysis of its Falcon advanced endpoint protection technology was based on incomplete testing conducted via illegally obtained access to its software and against CrowdStrike’s specific instructions to stop.

The court dismissed CrowdStrike’s request for a temporary restraining order and preliminary injunction against NSS Labs on the grounds that the company had failed to show how it would suffer irreparable harm from the report becoming public. It will now decide whether NSS acted illegally when conducting its tests on CrowdStrike’s Falcon technology.

George Kurtz, president and CEO of CrowdStrike says his company’s dispute with NSS has to do with the incompetent manner in which the tests were initially conducted and the fact that the results are based on incorrect and materially incomplete information.

CrowdStrike first hired NSS last year to conduct private testing on Falcon but quickly put a halt to it over concerns about the competency of the testing and the methods used, Kurtz says. As one example he pointed to NSS flagging legitimate software such as Skype and Adobe as malicious, during testing.

Even after paying a total of $150,000 for two private tests on Falcon, CrowdStrike was left with no confidence about NSS’ abilities and decided not to participate in a subsequent public test of advance endpoint protection products by NSS.

NSS however, proceeded to conduct public tests on Falcon anyway, using access it obtained illegally via a reseller. When CrowdStrike discovered what was going on the company immediately pulled NSS’ access to Falcon. So any subsequent conclusion that NSS made about Falcon’s effectiveness were based on incomplete and materially incorrect information and without all product features being turned on, Kurtz says.

“This is not about trying to silence independent research,” Kurtz says. “We welcome open, fair, transparent and competent testing. We didn’t necessarily see it here. This isn’t the Consumer Reports of cybersecurity. It’s bad tests, bad data and bad results.”

The episode shows why it is important to have more standard processes and organizations for testing the effectiveness of security products, he says. While companies like NSS Labs purport to be independent and unbiased, he says they have a for-profit business model and make money selling test reports and their own security platforms. “Unfortunately, there is no Underwriters Laboratories or testing house for security products,” Kurtz says.

Vikram Phatak, CEO of NSS Labs expressed regret over CrowdStrike’s stance. “We obviously disagree and are disappointed with Crowdstrike's characterization of NSS,” he said in emailed comments.

Though Crowdstike's request for a temporary restraining Order and preliminary injunction were denied by the federal court, the lawsuit itself is still pending.  “So we are limited in what we can say,” Phatak said. “Whether or not it is their intent, their suit has the effect of keeping us from debating the facts publicly.”

Such disputes over product testing and validation are not unusual says Pete Lindstrom, an analyst with IDC. Still, it is slightly unsettling when companies get as aggressive as CrowdStrike did in its dispute with NSS Labs, he says.

Technology buyers generally understand the limitations of product testing and know how difficult it can be for testers to replicate actual real world conditions when assessing the effectives of security products, Lindstrom says.

“In my mind security vendors should be looking for ways to get more information about the efficacy of their products out into the market,” he notes. “Anyone confident in their solution, should be willing to do that even within the limitations of testing.”

As long as all the circumstances of the testing are described fully and vendors have a chance to rebut claims, users are well served, Lindstrom said. An enterprise buyer that is already interested in CrowdStrike for instance is unlikely to view the company negatively based on one report, especially if all details are fully available.

 “These reports are generally used for confirmation rather than negation,” he says.

Related Content:

 

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
B__P
50%
50%
B__P,
User Rank: Apprentice
2/16/2017 | 1:37:26 PM
Conflict of interest
If NSS does testing plus offers its own solution it cannot be impartial.  That's a clear conflict of interest.
US Turning Up the Heat on North Korea's Cyber Threat Operations
Jai Vijayan, Contributing Writer,  9/16/2019
NetCAT Vulnerability Is Out of the Bag
Dark Reading Staff 9/12/2019
Preventing PTSD and Burnout for Cybersecurity Professionals
Craig Hinkley, CEO, WhiteHat Security,  9/16/2019
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Current Issue
7 Threats & Disruptive Forces Changing the Face of Cybersecurity
This Dark Reading Tech Digest gives an in-depth look at the biggest emerging threats and disruptive forces that are changing the face of cybersecurity today.
Flash Poll
The State of IT Operations and Cybersecurity Operations
The State of IT Operations and Cybersecurity Operations
Your enterprise's cyber risk may depend upon the relationship between the IT team and the security team. Heres some insight on what's working and what isn't in the data center.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2019-15943
PUBLISHED: 2019-09-19
vphysics.dll in Counter-Strike: Global Offensive before 1.37.1.1 allows remote attackers to achieve code execution or denial of service by creating a gaming server and inviting a victim to this server, because a crafted map is mishandled during a memset call.
CVE-2019-16413
PUBLISHED: 2019-09-19
An issue was discovered in the Linux kernel before 5.0.4. The 9p filesystem did not protect i_size_write() properly, which causes an i_size_read() infinite loop and denial of service on SMP systems.
CVE-2019-3738
PUBLISHED: 2019-09-18
RSA BSAFE Crypto-J versions prior to 6.2.5 are vulnerable to an Improper Verification of Cryptographic Signature vulnerability. A malicious remote attacker could potentially exploit this vulnerability to coerce two parties into computing the same predictable shared key.
CVE-2019-3739
PUBLISHED: 2019-09-18
RSA BSAFE Crypto-J versions prior to 6.2.5 are vulnerable to Information Exposure Through Timing Discrepancy vulnerabilities during ECDSA key generation. A malicious remote attacker could potentially exploit those vulnerabilities to recover ECDSA keys.
CVE-2019-3740
PUBLISHED: 2019-09-18
RSA BSAFE Crypto-J versions prior to 6.2.5 are vulnerable to an Information Exposure Through Timing Discrepancy vulnerabilities during DSA key generation. A malicious remote attacker could potentially exploit those vulnerabilities to recover DSA keys.