Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Endpoint

2/14/2017
05:00 PM
Connect Directly
Twitter
LinkedIn
RSS
E-Mail
50%
50%

CrowdStrike Fails In Bid To Stop NSS Labs From Publishing Test Results At RSA

NSS results are based on incomplete and materially incorrect data, CrowdStrike CEO George Kurtz says.

A security vendor’s failed attempt to prevent a report containing details on its product’s capabilities from being released at the RSA conference in San Francisco this week has focused attention on the need for more widely accepted testing and validation services for security technologies.

The report is from NSS Labs, a company that bills itself as an independent security product-testing firm, which also sells its own threat protection platform.  NSS recently conducted security tests on Advanced Endpoint Protection (AEP) products from 13 vendors and released the results of its testing at RSA this week.

Each of the products, according to NSS, was tested against multiple attack threat vectors for their resilience against evasion techniques, their tendency to generate false positives, and other metrics. The products were then graded for their overall security effectiveness and total cost of ownership per protected endpoint agent and assigned a rating of “Recommended,” “Security Recommended” or “Caution”.

Security vendor CrowdStrike, one of two vendors in the report to garner a “Caution” rating filed a lawsuit in federal court in Delaware last week seeking to prevent NSS from releasing the report at RSA. In a legal brief, CrowdStrike claimed that NSS’ analysis of its Falcon advanced endpoint protection technology was based on incomplete testing conducted via illegally obtained access to its software and against CrowdStrike’s specific instructions to stop.

The court dismissed CrowdStrike’s request for a temporary restraining order and preliminary injunction against NSS Labs on the grounds that the company had failed to show how it would suffer irreparable harm from the report becoming public. It will now decide whether NSS acted illegally when conducting its tests on CrowdStrike’s Falcon technology.

George Kurtz, president and CEO of CrowdStrike says his company’s dispute with NSS has to do with the incompetent manner in which the tests were initially conducted and the fact that the results are based on incorrect and materially incomplete information.

CrowdStrike first hired NSS last year to conduct private testing on Falcon but quickly put a halt to it over concerns about the competency of the testing and the methods used, Kurtz says. As one example he pointed to NSS flagging legitimate software such as Skype and Adobe as malicious, during testing.

Even after paying a total of $150,000 for two private tests on Falcon, CrowdStrike was left with no confidence about NSS’ abilities and decided not to participate in a subsequent public test of advance endpoint protection products by NSS.

NSS however, proceeded to conduct public tests on Falcon anyway, using access it obtained illegally via a reseller. When CrowdStrike discovered what was going on the company immediately pulled NSS’ access to Falcon. So any subsequent conclusion that NSS made about Falcon’s effectiveness were based on incomplete and materially incorrect information and without all product features being turned on, Kurtz says.

“This is not about trying to silence independent research,” Kurtz says. “We welcome open, fair, transparent and competent testing. We didn’t necessarily see it here. This isn’t the Consumer Reports of cybersecurity. It’s bad tests, bad data and bad results.”

The episode shows why it is important to have more standard processes and organizations for testing the effectiveness of security products, he says. While companies like NSS Labs purport to be independent and unbiased, he says they have a for-profit business model and make money selling test reports and their own security platforms. “Unfortunately, there is no Underwriters Laboratories or testing house for security products,” Kurtz says.

Vikram Phatak, CEO of NSS Labs expressed regret over CrowdStrike’s stance. “We obviously disagree and are disappointed with Crowdstrike's characterization of NSS,” he said in emailed comments.

Though Crowdstike's request for a temporary restraining Order and preliminary injunction were denied by the federal court, the lawsuit itself is still pending.  “So we are limited in what we can say,” Phatak said. “Whether or not it is their intent, their suit has the effect of keeping us from debating the facts publicly.”

Such disputes over product testing and validation are not unusual says Pete Lindstrom, an analyst with IDC. Still, it is slightly unsettling when companies get as aggressive as CrowdStrike did in its dispute with NSS Labs, he says.

Technology buyers generally understand the limitations of product testing and know how difficult it can be for testers to replicate actual real world conditions when assessing the effectives of security products, Lindstrom says.

“In my mind security vendors should be looking for ways to get more information about the efficacy of their products out into the market,” he notes. “Anyone confident in their solution, should be willing to do that even within the limitations of testing.”

As long as all the circumstances of the testing are described fully and vendors have a chance to rebut claims, users are well served, Lindstrom said. An enterprise buyer that is already interested in CrowdStrike for instance is unlikely to view the company negatively based on one report, especially if all details are fully available.

 “These reports are generally used for confirmation rather than negation,” he says.

Related Content:

 

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year ... View Full Bio
 

Recommended Reading:

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
B__P
50%
50%
B__P,
User Rank: Apprentice
2/16/2017 | 1:37:26 PM
Conflict of interest
If NSS does testing plus offers its own solution it cannot be impartial.  That's a clear conflict of interest.
COVID-19: Latest Security News & Commentary
Dark Reading Staff 9/21/2020
Cybersecurity Bounces Back, but Talent Still Absent
Simone Petrella, Chief Executive Officer, CyberVista,  9/16/2020
Meet the Computer Scientist Who Helped Push for Paper Ballots
Kelly Jackson Higgins, Executive Editor at Dark Reading,  9/16/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
Special Report: Computing's New Normal
This special report examines how IT security organizations have adapted to the "new normal" of computing and what the long-term effects will be. Read it and get a unique set of perspectives on issues ranging from new threats & vulnerabilities as a result of remote working to how enterprise security strategy will be affected long term.
Flash Poll
How IT Security Organizations are Attacking the Cybersecurity Problem
How IT Security Organizations are Attacking the Cybersecurity Problem
The COVID-19 pandemic turned the world -- and enterprise computing -- on end. Here's a look at how cybersecurity teams are retrenching their defense strategies, rebuilding their teams, and selecting new technologies to stop the oncoming rise of online attacks.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-25826
PUBLISHED: 2020-09-23
PingID Integration for Windows Login before 2.4.2 allows local users to gain privileges by modifying CefSharp.BrowserSubprocess.exe.
CVE-2020-25821
PUBLISHED: 2020-09-23
** UNSUPPORTED WHEN ASSIGNED ** peg-markdown 0.4.14 has a NULL pointer dereference in process_raw_blocks in markdown_lib.c. NOTE: This vulnerability only affects products that are no longer supported by the maintainer.
CVE-2020-3130
PUBLISHED: 2020-09-23
A vulnerability in the web management interface of Cisco Unity Connection could allow an authenticated remote attacker to overwrite files on the underlying filesystem. The vulnerability is due to insufficient input validation. An attacker could exploit this vulnerability by sending a crafted HTTP re...
CVE-2020-3133
PUBLISHED: 2020-09-23
A vulnerability in the email message scanning of Cisco AsyncOS Software for Cisco Email Security Appliance (ESA) could allow an unauthenticated, remote attacker to bypass configured filters on the device. The vulnerability is due to improper validation of incoming emails. An attacker could exploit t...
CVE-2020-3135
PUBLISHED: 2020-09-23
A vulnerability in the web-based management interface of Cisco Unified Communications Manager (UCM) could allow an unauthenticated, remote attacker to conduct a cross-site request forgery (CSRF) attack on an affected device. The vulnerability is due to insufficient CSRF protections for the web-based...