Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


02:58 PM
Dark Reading
Dark Reading
Products and Releases

NSS Labs Releases Next Generation Firewall Group Test Reports

NSS' research yielded several key conclusions

AUSTIN, Texas – February 26, 2013 - NSS Labs today released its 2013 Next Generation Firewall (NGFW) Security Value Map and Comparative Analysis Reports, which evaluated 9 of the leading NGFW products on the market for security effectiveness, performance, enterprise management capabilities and total cost of ownership. This was the second group test for NGFW that NSS has conducted and overall there was marked improvement from most vendors' 2012 test scores.

Read the Reports:

NSS 2013 Next Generation Firewall Security Value Map&trade and Comparative Analysis Reports&trade – Performance, Management, Security and Total Cost of Ownership

NSS's research yielded several key conclusions:

Check NGFWs' firmware before deployment: Out of a total of 9 products tested, 6 vendors submitted products that required firmware updates or configuration changes to complete the NSS tests. Only Check Point, Fortinet and Stonesoft submitted products that worked the first time.

New Metric Highlights Enterprise Management Failings: If a device cannot be managed effectively, the security effectiveness of that device is compromised. As part of this test, NSS performed in-depth technical evaluations of all the main features and capabilities of the enterprise management systems offered by each vendor and factored it into the final score as a new and unique metric called "managed security effectiveness". Managed security effectiveness scores ranged from 29.1% to 98.5%.

NGFWs' Security Effectiveness Scores Improve Significantly: In the latest 2013 tests, 8 of the 9 products scored over 90% for security effectiveness (excluding management). This is a marked increase compared to 2012, when only half of tested vendors scored above 90% in this category. The overall scores for security effectiveness in 2013 ranged from 34.2% to 98.5% compared to 18% to 98.9% in 2012.

Total Cost of Ownership Remains Fairly Stable: While the overall range of TCO decreased in 2013 testing, prices per protected megabit per second remained fairly stable with most tested devices costing below $44 per Protected-Mbps. The overall 2013 range was $18 - $124 per Protected Mbps, down from a range of $30 - $375 in 2012 testing.

More Vendors Back their Performance Claims: Only 2 of 9 products tested had throughput rates that were significantly less than their vendors' stated claims. In 2012 testing, 5 of the 8 products tested performed well below their advertised speeds. In 2013, three vendors – Dell SonicWALL, Sourcefire and Palo Alto – performed better in tested performance than their stated throughput and two vendors – Check Point and Stonesoft – had throughputs that were virtually equal to their stated performance.

Commentary: NSS Labs Research Director Francisco Artes

"In 2012, our tests showed that while vendors turned in a good first showing, there was significant room for NGFW technologies as a whole to improve before being widely deployed in large enterprises," said Francisco Artes, Research Director at NSS Labs. "In our 2013 tests, I think we've seen much of the improvement we thought was needed in previous testing. With 7 of the 9 products receiving a `Recommend' rating in this year's tests, it's clear that the vendors are investing a lot of time and effort to address many of the overall stability, leakage, performance and security effectiveness concerns from last year."

The 2013 NGFW Security Value Map&trade, Comparative Analysis Reports&trade, and Product Analysis Reports&trade for each vendor are currently available to NSS Labs' subscribers at www.nsslabs.com.

The products covered in the 2013 NGFW Group Test are:

· Check Point 12600

· Dell SonicWALL SuperMassive E10800

· Fortinet FortiGate 3600C

· Juniper SRX 3600

· Palo Alto PA-5020

· Sourcefire 8250

· Sourcefire 8290

· Stonesoft 3202

· WatchGuard XTM 2050

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Pat Devlin
Pat Devlin,
User Rank: Apprentice
3/7/2013 | 1:56:44 AM
re: NSS Labs Releases Next Generation Firewall Group Test Reports
Our ranking on NSS Labs' recent Next Generation Firewall Security Value Map does not accurately portray the performance of our products.

During the installation, an error in the firmware image required our test engineer to re-download and re-install a different firmware image.- This process took 16 hours to accomplish, but was a one-time anomaly.- However, during their post-test scoring, NSS Labs multiplied that 16 hours by all 50 hypothetical devices, resulting in an unrealistic score.

Our customers also use and love our AD (active directory) integration, a feature we have supported for nearly 5 years. We received a low score in this category and we believe the flaw was in NSS's AD methodology.

See a case study on our AD performance here:


WatchGuardGs "Best in Class" model is optimized for Unified Threat Management (UTM) - a step beyond NGFW -- because we believe that UTM platforms are where customers see the largest benefit and value. Sometimes this means we donGt fit squarely within traditional categories set up for laboratory tests, but our customers donGt operate within a test lab, they operate in the real world.

We are continuing to improve our product to provide the best possible solutions for our customers and will be seeking independent test results to prove our claims.

Patrick Devlin
Regional Director
Watchguard Technologies | Australia & New Zealand

*Edit 7th of March 2013:

When it comes to the AD test bed and firmware, the responsibility ultimately lies with us. We have learned from these tests that better communication is needed when submitting products to be tested. As for the firmware, it will generally take one minute to update across all of our products (probably a little longer when using a 2400 baud modem).

We're extremely confident that our products deliver and they usually achieve excellent results in these tests and weGve had a harsh reminder about how important it is for us to maintain strong contact with the labs during product testing.
COVID-19: Latest Security News & Commentary
Dark Reading Staff 4/7/2020
The Coronavirus & Cybersecurity: 3 Areas of Exploitation
Robert R. Ackerman Jr., Founder & Managing Director, Allegis Capital,  4/7/2020
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment: This comment is waiting for review by our moderators.
Current Issue
6 Emerging Cyber Threats That Enterprises Face in 2020
This Tech Digest gives an in-depth look at six emerging cyber threats that enterprises could face in 2020. Download your copy today!
Flash Poll
State of Cybersecurity Incident Response
State of Cybersecurity Incident Response
Data breaches and regulations have forced organizations to pay closer attention to the security incident response function. However, security leaders may be overestimating their ability to detect and respond to security incidents. Read this report to find out more.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2020-04-08
An issue was discovered in Varnish Cache before 6.0.5 LTS, 6.1.x and 6.2.x before 6.2.2, and 6.3.x before 6.3.1. It does not clear a pointer between the handling of one client request and the next request within the same connection. This sometimes causes information to be disclosed from the connecti...
PUBLISHED: 2020-04-08
An issue was discovered in iXsystems FreeNAS 11.2 and 11.3 before 11.3-U1. It allows a denial of service.
PUBLISHED: 2020-04-08
An issue was discovered in Varnish Cache before 6.0.6 LTS, 6.1.x and 6.2.x before 6.2.3, and 6.3.x before 6.3.2. It occurs when communication with a TLS termination proxy uses PROXY version 2. There can be an assertion failure and daemon restart, which causes a performance loss.
PUBLISHED: 2020-04-08
A flaw was discovered in the way that the KVM hypervisor handled instruction emulation for an L2 guest when nested virtualisation is enabled. Under some circumstances, an L2 guest may trick the L0 guest into accessing sensitive L1 resources that should be inaccessible to the L2 guest.
PUBLISHED: 2020-04-08
A vulnerability in Juniper Networks Junos OS on vMX and MX150 devices may allow an attacker to cause a Denial of Service (DoS) by sending specific packets requiring special processing in microcode that the flow cache can't handle, causing the riot forwarding daemon to crash. By continuously sending ...