Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


07:00 PM
Connect Directly

NSS Labs Files Antitrust Suit Against Symantec, CrowdStrike, ESET, AMTSO

Suit underscores longtime battle between vendors and labs over control of security testing protocols.

UPDATED 9/20 with comments from ESET, AV-Test, and CrowdStrike

Security product testing firm NSS Labs today filed an antitrust lawsuit against cybersecurity vendors CrowdStrike, ESET, and Symantec as well as the Anti-Malware Testing Standards Organization (AMTSO) over a vendor-backed testing protocol.

The lawsuit accuses the three security vendors and the nonprofit AMTSO, of which they and other endpoint security vendors are members, of unfairly allowing their products to be tested only by organizations that comply with AMTSO's testing protocol standard. NSS Labs, which also is a member of AMTSO, earlier this year voted against adoption of the standard and says it has no plans to comply with it.

A majority of AMTSO members voted in favor of the standard in May of this year, and most plan to adopt the protocol.

Friction between security vendors and independent testing labs is nothing new. Vendors and labs traditionally have had an uneasy and sometimes contentious relationship over control of the testing process and parameters. NSS Labs' suit appears to represent an escalation of that age-old conflict, security experts say.

NSS Labs is calling foul in its lawsuit: "NSS Labs has suffered antitrust injury as a result of the acts herein alleged because it is the direct and principal target of the concerted refusal to deal/group boycott" any testing organizations that don't adopt ATMSO's testing standard, the lawsuit says.

In an interview with Dark Reading, Jason Brvenik, chief technology officer at NSS Labs, said the ATMSO standard falls short. "Our fundamental focus is that if a product is good enough to sell, it's good enough to test," and NSS Labs shouldn't be forced to comply with ATMSO's standard, he says. "It should be an independent test."

Brvenik says the AMTSO standard does not support independent testing. "It's driven by vendors to create a picture of capabilities that are not true," for example, he says. "The standard is more like guidelines to interact with than a standard, and it doesn't make things better for products" or the way they are tested, he says.

According to the NSS Labs suit, other vendors that spoke out against the adoption of AMTSO's standards included AVComparatives, AV-Test, and SKD LABS. None of those vendors are named as parties in NSS Labs' case. Efforts to reach AVComparatives, and SKD Labs were unsuccessful as of this posting.

AV-Test CEO Andreas Marx said he had just heard about the suit and is unable to comment at this time.

CrowdStrike called the suit groundless: "NSS is a for-profit, pay-to-play testing organization that obtains products through fraudulent means and is desperate to defend its business model from open and transparent testing. We believe their lawsuit is baseless," the company said in a statement.

"CrowdStrike supports independent and standards-based testing—including public testing—for our products and for the industry. We have undergone independent testing with AV-Comparatives, SE Labs, and MITRE and you can find information on that testing here," the statement said. "We applaud AMTSO’s efforts to promote clear, consistent, and transparent testing standards."

ESET said it had not been officially contacted about the suit, but that it refutes the allegations. "We are aware of the allegations stated in the blog post from NSS Labs, however, we have yet to receive official legal communication. As legal proceedings appear to have been initiated, we are unable to say more at this time, beyond the statement that we categorically deny the allegations," an ESET spokesperson said. "Our customers should be reassured that ESET’s products have been rigorously tested by many independent third-party reviewers around the world, received numerous awards for their level of protection of end users over many years, and are widely praised by industry-leading specialists." 

Symantec would not comment on the case, and efforts to reach AMTSO  were unsuccessful as of this posting.

In a blog post earlier this month, ATMSO president Dennis Batchelder wrote that the protocol is a voluntary framework for testing anti-malware software "fairly and transparently." 

For enterprises, there aren't many options for vetting security software. Most don't have the resources to perform their own in-house testing of security products, so they rely on consulting firms' recommendations, third-party testing organizations — or the claims of their vendor.

Jon Oltsik, senior principal analyst with consulting firm Enterprise Strategy Group, says he's seen enterprises struggle with the testing dilemma. "Customers don't know how to test the efficacy of next-generation endpoint security products," he says. "No one trusts vendors to test their own product."

The concept of a vetted product testing standard is a "very good idea," says Oltsik, who notes that he has not specifically studied ATMSO's protocol.

Bottom Line
NSS Labs meantime argues that the AMTSO and its standard are anti-competitive. "They claim to try to improve testing but what they're actually doing is actively preventing unbiased testing. Further, vendors are openly exerting control and collectively boycotting testing organizations that don't comply with their AMTSO standards — even going so far as to block the independent purchase and testing of their products," Vikram Phatak, CEO of NSS Labs wrote in a blog post today announcing the suit.

Meanwhile, NSS Labs claims in its lawsuit that AMTSO's efforts have hurt its bottom line. "NSS Labs has lost sales and profits from the sale and license of its public testing reports, including its AEP Group Test reports, because it cannot charge customers who purchase reports that do not include all market participants as much as it could charge for reports that included all market participants." 


Black Hat Europe returns to London Dec. 3-6, 2018, with hands-on technical Trainings, cutting-edge Briefings, Arsenal open-source tool demonstrations, top-tier security solutions, and service providers in the Business Hall. Click for information on the conference and to register.

Kelly Jackson Higgins is the Executive Editor of Dark Reading. She is an award-winning veteran technology and business journalist with more than two decades of experience in reporting and editing for various publications, including Network Computing, Secure Enterprise ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
12/7/2018 | 6:36:46 AM
Re: NSS Antitrust Law Suit
Exactly. I do not see any problem. And if there IS some problem with standard (which really is actually just guideline) then why they do not publish openly what are the problematic parts and how they would like to change them. As somebody said alrerady this brings questions what NSS has to hide by not accepting this open "standard".
User Rank: Strategist
9/21/2018 | 6:57:24 AM
It's important that the NSS Labs Suit should not succeed.

Can't say I'm in the least bit surprised.

It's important that the NSS Labs suit against AMTSO should not succeed, in my view.

What AMTSO is trying to achieve is self-evidently a 'good thing'. Its goals - as supported by a who's-who of pretty much anybody who's anybody in endpoint security - are entirely consistent with bringing more open, transparent, security testing standards to consumers and businesses.

Has AMTSO somewhere in its articles of incorporation or operating practices left some kind of miniscule loophole - some missing word, some un-crossed 't' or some upside down full-stop - that a lawyer can seize upon as a 'gotcha' for their client? I haven't the faintest idea. You have to hope not.

Something else. IT security professionals have long complained about lack of transparency in endpoint security testing. Many look at independent test house results based on proprietary test methods and the first thing the eyes do is look to the clouds (if they don't glaze over first).

You get the politicians you deserve? I don't altogether go along with that. But do enterprise IT organizations get the security testing standards they deserve? Yeah, they do. AMTSO needs support from its lawyers and its members, right now. But it needs a lot more support from enterprise users too.
User Rank: Ninja
9/20/2018 | 5:17:18 PM
Thank you for your insight
I appreciate the background information you provided, it seems that NSS is going after a number of companies (not just Crowdstrike), this seems to be backlash from other encounters they have had outside of this specific incident:

→ Cybersecurity vendors CrowdStrike, ESET, and Symantec

However, the question still remains, if Crowdstrike has nothing to hide and if they have confidence in their product, then why is there such resistance, seems to me that Crowdstrike has something to hide (also, per the recommendation from the courts). NSS labs uses tools from the wild that will more resemble attacks as opposed to running simple tests that don't really give the user a clear indication of how their product will fair when it is attacked using surreptious and/or zero-day attack methods.

In addition, I would like to know which methods they thought did not meet their approval, in the wild, there will be numerous things that don't meet their approval, especially after spending $150K.

I am curious what the courts will rule, again, time will tell.

Kelly Jackson Higgins
Kelly Jackson Higgins,
User Rank: Strategist
9/20/2018 | 3:20:33 PM
Re: NSS Antitrust Law Suit

That's a good question and there a few things going on here. NSS Labs maintains that the AMTSO is skewed to the vendors and not independent, and that its own testing is more independent than one approved by vendors (AMTSO's). There also appears to be some bad blood between NSS Labs and CrowdStrike, publicly dating back to 2017 when CrowdStrike attempted (and failed) to get a restraining order to prevent NSS from publishing test results. According to CrowdStrike, it found NSS's testing methods questionable and the results based on incorrect and incomplete information. So it pulled out of the test, and then NSS accessed CrowdStrike's Falcon product via reseller in a violation of CrowdStrike's EULA. CrowdStrike said the test didn't have all of the product's features enabled, so it wasn't an accurate asssessment: https://www.darkreading.com/endpoint/crowdstrike-fails-in-bid-to-stop-nss-labs-from-publishing-test-results-at-rsa/d/d-id/1328154

CrowdStrike called NSS Labs' antitrust suit "baseless" this morning, and now NSS is firing back and calling out CrowdStrike's "smear tactics" and dismissing AMTSO as a "pay to play" thing. 

So...I really don't think NSS Labs has any plans to go AMTSO. 
User Rank: Ninja
9/20/2018 | 2:35:15 PM
NSS Antitrust Law Suit
What I am confused about is as follows:

→ Why can't NSS Adopt the ATMSO protocol/methodology as part of their overall testing strategy?

→ Why don't they provide ATMSO and their independent testing concept and compare the two?

If they did this, then they would be the defacto standard when it comes to security pentesting.

I think it is about pride and money as opposed to anything else.

Oh well, time will tell.

7 Old IT Things Every New InfoSec Pro Should Know
Joan Goodchild, Staff Editor,  4/20/2021
Cloud-Native Businesses Struggle With Security
Robert Lemos, Contributing Writer,  5/6/2021
Defending Against Web Scraping Attacks
Rob Simon, Principal Security Consultant at TrustedSec,  5/7/2021
Register for Dark Reading Newsletters
White Papers
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you today!
Flash Poll
How Enterprises are Developing Secure Applications
How Enterprises are Developing Secure Applications
Recent breaches of third-party apps are driving many organizations to think harder about the security of their off-the-shelf software as they continue to move left in secure software development practices.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2021-05-16
The JSON web services in Liferay Portal 7.3.4 and earlier, and Liferay DXP 7.0 before fix pack 97, 7.1 before fix pack 20 and 7.2 before fix pack 10 may provide overly verbose error messages, which allows remote attackers to use the contents of error messages to help launch another, more focused att...
PUBLISHED: 2021-05-16
Denial-of-service (DoS) vulnerability in the Multi-Factor Authentication module in Liferay DXP 7.3 before fix pack 1 allows remote authenticated attackers to prevent any user from authenticating by (1) enabling Time-based One-time password (TOTP) on behalf of the other user or (2) modifying the othe...
PUBLISHED: 2021-05-16
The SimpleCaptcha implementation in Liferay Portal 7.3.4, 7.3.5 and Liferay DXP 7.3 before fix pack 1 does not invalidate CAPTCHA answers after it is used, which allows remote attackers to repeatedly perform actions protected by a CAPTCHA challenge by reusing the same CAPTCHA answer.
PUBLISHED: 2021-05-16
Delta Industrial Automation CNCSoft ScreenEditor Versions 1.01.28 (with ScreenEditor Version 1.01.2) and prior are vulnerable to an out-of-bounds read while processing project files, which may allow an attacker to execute arbitrary code.
PUBLISHED: 2021-05-16
Cross-site scripting (XSS) vulnerability in the Asset module's categories administration page in Liferay Portal 7.3.4 allows remote attackers to inject arbitrary web script or HTML via the site name.