Endpoint

2/14/2017
05:00 PM
Connect Directly
Twitter
LinkedIn
RSS
E-Mail
50%
50%

CrowdStrike Fails In Bid To Stop NSS Labs From Publishing Test Results At RSA

NSS results are based on incomplete and materially incorrect data, CrowdStrike CEO George Kurtz says.

A security vendor’s failed attempt to prevent a report containing details on its product’s capabilities from being released at the RSA conference in San Francisco this week has focused attention on the need for more widely accepted testing and validation services for security technologies.

The report is from NSS Labs, a company that bills itself as an independent security product-testing firm, which also sells its own threat protection platform.  NSS recently conducted security tests on Advanced Endpoint Protection (AEP) products from 13 vendors and released the results of its testing at RSA this week.

Each of the products, according to NSS, was tested against multiple attack threat vectors for their resilience against evasion techniques, their tendency to generate false positives, and other metrics. The products were then graded for their overall security effectiveness and total cost of ownership per protected endpoint agent and assigned a rating of “Recommended,” “Security Recommended” or “Caution”.

Security vendor CrowdStrike, one of two vendors in the report to garner a “Caution” rating filed a lawsuit in federal court in Delaware last week seeking to prevent NSS from releasing the report at RSA. In a legal brief, CrowdStrike claimed that NSS’ analysis of its Falcon advanced endpoint protection technology was based on incomplete testing conducted via illegally obtained access to its software and against CrowdStrike’s specific instructions to stop.

The court dismissed CrowdStrike’s request for a temporary restraining order and preliminary injunction against NSS Labs on the grounds that the company had failed to show how it would suffer irreparable harm from the report becoming public. It will now decide whether NSS acted illegally when conducting its tests on CrowdStrike’s Falcon technology.

George Kurtz, president and CEO of CrowdStrike says his company’s dispute with NSS has to do with the incompetent manner in which the tests were initially conducted and the fact that the results are based on incorrect and materially incomplete information.

CrowdStrike first hired NSS last year to conduct private testing on Falcon but quickly put a halt to it over concerns about the competency of the testing and the methods used, Kurtz says. As one example he pointed to NSS flagging legitimate software such as Skype and Adobe as malicious, during testing.

Even after paying a total of $150,000 for two private tests on Falcon, CrowdStrike was left with no confidence about NSS’ abilities and decided not to participate in a subsequent public test of advance endpoint protection products by NSS.

NSS however, proceeded to conduct public tests on Falcon anyway, using access it obtained illegally via a reseller. When CrowdStrike discovered what was going on the company immediately pulled NSS’ access to Falcon. So any subsequent conclusion that NSS made about Falcon’s effectiveness were based on incomplete and materially incorrect information and without all product features being turned on, Kurtz says.

“This is not about trying to silence independent research,” Kurtz says. “We welcome open, fair, transparent and competent testing. We didn’t necessarily see it here. This isn’t the Consumer Reports of cybersecurity. It’s bad tests, bad data and bad results.”

The episode shows why it is important to have more standard processes and organizations for testing the effectiveness of security products, he says. While companies like NSS Labs purport to be independent and unbiased, he says they have a for-profit business model and make money selling test reports and their own security platforms. “Unfortunately, there is no Underwriters Laboratories or testing house for security products,” Kurtz says.

Vikram Phatak, CEO of NSS Labs expressed regret over CrowdStrike’s stance. “We obviously disagree and are disappointed with Crowdstrike's characterization of NSS,” he said in emailed comments.

Though Crowdstike's request for a temporary restraining Order and preliminary injunction were denied by the federal court, the lawsuit itself is still pending.  “So we are limited in what we can say,” Phatak said. “Whether or not it is their intent, their suit has the effect of keeping us from debating the facts publicly.”

Such disputes over product testing and validation are not unusual says Pete Lindstrom, an analyst with IDC. Still, it is slightly unsettling when companies get as aggressive as CrowdStrike did in its dispute with NSS Labs, he says.

Technology buyers generally understand the limitations of product testing and know how difficult it can be for testers to replicate actual real world conditions when assessing the effectives of security products, Lindstrom says.

“In my mind security vendors should be looking for ways to get more information about the efficacy of their products out into the market,” he notes. “Anyone confident in their solution, should be willing to do that even within the limitations of testing.”

As long as all the circumstances of the testing are described fully and vendors have a chance to rebut claims, users are well served, Lindstrom said. An enterprise buyer that is already interested in CrowdStrike for instance is unlikely to view the company negatively based on one report, especially if all details are fully available.

 “These reports are generally used for confirmation rather than negation,” he says.

Related Content:

 

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
B__P
50%
50%
B__P,
User Rank: Apprentice
2/16/2017 | 1:37:26 PM
Conflict of interest
If NSS does testing plus offers its own solution it cannot be impartial.  That's a clear conflict of interest.
More Than Half of Users Reuse Passwords
Curtis Franklin Jr., Senior Editor at Dark Reading,  5/24/2018
Is Threat Intelligence Garbage?
Chris McDaniels, Chief Information Security Officer of Mosaic451,  5/23/2018
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment: This comment is waiting for review by our moderators.
Current Issue
Flash Poll
[Strategic Security Report] How Enterprises Are Attacking the IT Security Problem
[Strategic Security Report] How Enterprises Are Attacking the IT Security Problem
Enterprises are spending more of their IT budgets on cybersecurity technology. How do your organization's security plans and strategies compare to what others are doing? Here's an in-depth look.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2018-11506
PUBLISHED: 2018-05-28
The sr_do_ioctl function in drivers/scsi/sr_ioctl.c in the Linux kernel through 4.16.12 allows local users to cause a denial of service (stack-based buffer overflow) or possibly have unspecified other impact because sense buffers have different sizes at the CDROM layer and the SCSI layer.
CVE-2018-11507
PUBLISHED: 2018-05-28
An issue was discovered in Free Lossless Image Format (FLIF) 0.3. An attacker can trigger a long loop in image_load_pnm in image/image-pnm.cpp.
CVE-2018-11505
PUBLISHED: 2018-05-26
The Werewolf Online application 0.8.8 for Android allows attackers to discover the Firebase token by reading logcat output.
CVE-2018-6409
PUBLISHED: 2018-05-26
An issue was discovered in Appnitro MachForm before 4.2.3. The module in charge of serving stored files gets the path from the database. Modifying the name of the file to serve on the corresponding ap_form table leads to a path traversal vulnerability via the download.php q parameter.
CVE-2018-6410
PUBLISHED: 2018-05-26
An issue was discovered in Appnitro MachForm before 4.2.3. There is a download.php SQL injection via the q parameter.