Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


02:58 PM
Dark Reading
Dark Reading
Products and Releases

NSS Labs Releases Next Generation Firewall Group Test Reports

NSS' research yielded several key conclusions

AUSTIN, Texas – February 26, 2013 - NSS Labs today released its 2013 Next Generation Firewall (NGFW) Security Value Map and Comparative Analysis Reports, which evaluated 9 of the leading NGFW products on the market for security effectiveness, performance, enterprise management capabilities and total cost of ownership. This was the second group test for NGFW that NSS has conducted and overall there was marked improvement from most vendors' 2012 test scores.

Read the Reports:

NSS 2013 Next Generation Firewall Security Value Map&trade and Comparative Analysis Reports&trade – Performance, Management, Security and Total Cost of Ownership

NSS's research yielded several key conclusions:

Check NGFWs' firmware before deployment: Out of a total of 9 products tested, 6 vendors submitted products that required firmware updates or configuration changes to complete the NSS tests. Only Check Point, Fortinet and Stonesoft submitted products that worked the first time.

New Metric Highlights Enterprise Management Failings: If a device cannot be managed effectively, the security effectiveness of that device is compromised. As part of this test, NSS performed in-depth technical evaluations of all the main features and capabilities of the enterprise management systems offered by each vendor and factored it into the final score as a new and unique metric called "managed security effectiveness". Managed security effectiveness scores ranged from 29.1% to 98.5%.

NGFWs' Security Effectiveness Scores Improve Significantly: In the latest 2013 tests, 8 of the 9 products scored over 90% for security effectiveness (excluding management). This is a marked increase compared to 2012, when only half of tested vendors scored above 90% in this category. The overall scores for security effectiveness in 2013 ranged from 34.2% to 98.5% compared to 18% to 98.9% in 2012.

Total Cost of Ownership Remains Fairly Stable: While the overall range of TCO decreased in 2013 testing, prices per protected megabit per second remained fairly stable with most tested devices costing below $44 per Protected-Mbps. The overall 2013 range was $18 - $124 per Protected Mbps, down from a range of $30 - $375 in 2012 testing.

More Vendors Back their Performance Claims: Only 2 of 9 products tested had throughput rates that were significantly less than their vendors' stated claims. In 2012 testing, 5 of the 8 products tested performed well below their advertised speeds. In 2013, three vendors – Dell SonicWALL, Sourcefire and Palo Alto – performed better in tested performance than their stated throughput and two vendors – Check Point and Stonesoft – had throughputs that were virtually equal to their stated performance.

Commentary: NSS Labs Research Director Francisco Artes

"In 2012, our tests showed that while vendors turned in a good first showing, there was significant room for NGFW technologies as a whole to improve before being widely deployed in large enterprises," said Francisco Artes, Research Director at NSS Labs. "In our 2013 tests, I think we've seen much of the improvement we thought was needed in previous testing. With 7 of the 9 products receiving a `Recommend' rating in this year's tests, it's clear that the vendors are investing a lot of time and effort to address many of the overall stability, leakage, performance and security effectiveness concerns from last year."

The 2013 NGFW Security Value Map&trade, Comparative Analysis Reports&trade, and Product Analysis Reports&trade for each vendor are currently available to NSS Labs' subscribers at www.nsslabs.com.

The products covered in the 2013 NGFW Group Test are:

· Check Point 12600

· Dell SonicWALL SuperMassive E10800

· Fortinet FortiGate 3600C

· Juniper SRX 3600

· Palo Alto PA-5020

· Sourcefire 8250

· Sourcefire 8290

· Stonesoft 3202

· WatchGuard XTM 2050

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Pat Devlin
Pat Devlin,
User Rank: Apprentice
3/7/2013 | 1:56:44 AM
re: NSS Labs Releases Next Generation Firewall Group Test Reports
Our ranking on NSS Labs' recent Next Generation Firewall Security Value Map does not accurately portray the performance of our products.

During the installation, an error in the firmware image required our test engineer to re-download and re-install a different firmware image.- This process took 16 hours to accomplish, but was a one-time anomaly.- However, during their post-test scoring, NSS Labs multiplied that 16 hours by all 50 hypothetical devices, resulting in an unrealistic score.

Our customers also use and love our AD (active directory) integration, a feature we have supported for nearly 5 years. We received a low score in this category and we believe the flaw was in NSS's AD methodology.

See a case study on our AD performance here:


WatchGuardGs "Best in Class" model is optimized for Unified Threat Management (UTM) - a step beyond NGFW -- because we believe that UTM platforms are where customers see the largest benefit and value. Sometimes this means we donGt fit squarely within traditional categories set up for laboratory tests, but our customers donGt operate within a test lab, they operate in the real world.

We are continuing to improve our product to provide the best possible solutions for our customers and will be seeking independent test results to prove our claims.

Patrick Devlin
Regional Director
Watchguard Technologies | Australia & New Zealand

*Edit 7th of March 2013:

When it comes to the AD test bed and firmware, the responsibility ultimately lies with us. We have learned from these tests that better communication is needed when submitting products to be tested. As for the firmware, it will generally take one minute to update across all of our products (probably a little longer when using a 2400 baud modem).

We're extremely confident that our products deliver and they usually achieve excellent results in these tests and weGve had a harsh reminder about how important it is for us to maintain strong contact with the labs during product testing.
Data Leak Week: Billions of Sensitive Files Exposed Online
Kelly Jackson Higgins, Executive Editor at Dark Reading,  12/10/2019
Intel Issues Fix for 'Plundervolt' SGX Flaw
Kelly Jackson Higgins, Executive Editor at Dark Reading,  12/11/2019
Register for Dark Reading Newsletters
White Papers
Current Issue
The Year in Security: 2019
This Tech Digest provides a wrap up and overview of the year's top cybersecurity news stories. It was a year of new twists on old threats, with fears of another WannaCry-type worm and of a possible botnet army of Wi-Fi routers. But 2019 also underscored the risk of firmware and trusted security tools harboring dangerous holes that cybercriminals and nation-state hackers could readily abuse. Read more.
Flash Poll
Rethinking Enterprise Data Defense
Rethinking Enterprise Data Defense
Frustrated with recurring intrusions and breaches, cybersecurity professionals are questioning some of the industrys conventional wisdom. Heres a look at what theyre thinking about.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2019-12-14
There is an improper authentication vulnerability in Huawei smartphones (Y9, Honor 8X, Honor 9 Lite, Honor 9i, Y6 Pro). The applock does not perform a sufficient authentication in a rare condition. Successful exploit could allow the attacker to use the application locked by applock in an instant.
PUBLISHED: 2019-12-14
Some Huawei smart phones have a null pointer dereference vulnerability. An attacker crafts specific packets and sends to the affected product to exploit this vulnerability. Successful exploitation may cause the affected phone to be abnormal.
PUBLISHED: 2019-12-13
There is an information disclosure vulnerability in certain Huawei smartphones (Mate 10;Mate 10 Pro;Honor V10;Changxiang 7S;P-smart;Changxiang 8 Plus;Y9 2018;Honor 9 Lite;Honor 9i;Mate 9). The software does not properly handle certain information of applications locked by applock in a rare condition...
PUBLISHED: 2019-12-13
Huawei CloudUSM-EUA V600R006C10;V600R019C00 have an information leak vulnerability. Due to improper configuration, the attacker may cause information leak by successful exploitation.
PUBLISHED: 2019-12-13
Certain Huawei products (AP2000;IPS Module;NGFW Module;NIP6300;NIP6600;NIP6800;S5700;SVN5600;SVN5800;SVN5800-C;SeMG9811;Secospace AntiDDoS8000;Secospace USG6300;Secospace USG6500;Secospace USG6600;USG6000V;eSpace U1981) have an out-of-bounds read vulnerability. An attacker who logs in to the board m...