Fair warning about intrusion prevention systems (IPSes) products: They are nearly 50 percent less effective at catching threats if left with their default settings, and some operate at only about half of their promised throughput, according to results from a new laboratory test of popular IPS products.

NSS Labs, an independent testing organization, also found a wide range in protection rates among different products; the IPS that stopped the most threats in its testing did so at a rate of 89.5 percent, while the one that blocked the fewest threats did so at a low rate of 17.3 percent. And don't expect to necessarily get the full gigabit or other throughput speed promised by every IPS, according to NSS Labs' findings: "Performance can be overstated by 50 percent," says Rick Moy, president of NSS Labs.

NSS Labs tested the Cisco IPS 4260 Sensor, IBM Proventia Network IPS GX4004, IBM Proventia Network IPS GX6116, Juniper Networks IDP-250, Juniper Networks IDP-600c, Juniper Networks IDP-800, McAfee M-1250, McAfee M-8000, Sourcefire 3D 4500, Stonesoft StoneGate IPS-1030, Stonesoft StoneGate IPS-1060, Stonesoft StoneGate IPS-6105, TippingPoint 10 IPS, TippingPoint 660N IPS, and TippingPoint 2500N IPS. Several other IPS vendors declined NSS Labs' invitation to participate in the tests.

Details of specific vendors' performances in the tests are available only via paid reports sold by NSS, which are available here. Meanwhile, NSS Labs provided some insight into the overall IPS testing results.

NSS Labs tested the IPSes with their default settings and then with "tuned" configurations by on-site technicians from the vendors themselves. "We let the vendors go in and configure their products," Moy says. "We found that you can get up to 44 percent better protection by tuning it. On average, you get 18 percent better protection."

But configuring an IPS isn't just set-it-and-forget-it, he says. "You'd better know what you are doing," Moy says. "Anyone installing an IPS that's setting it and forgetting it is [at] risk. You could be catching more attacks than you are."

Attacks that employ evasion and obfuscation often fool IPSes, and only two vendors' IPS products were able to detect all attacks that tried to mask their true intentions.

Moy says he was surprised the IPSes weren't able to detect evasion techniques, which basically hide a known attack by sending it in a different way in an attempt to evade the filters. "We feel products should catch those. This would be an easy way [to improve the product]," he says.

Actual performance varies a lot among IPSes, meanwhile, and NSS Labs found that vendors overstate their throughput anywhere from 12 to 50 percent. "Watch out for the performance numbers vendors give," Moy says. "If it's a 1-gigabit interface, it's not necessarily protecting 1 gig. Some IPSes were only able to process and inspect up to 50 to 60 percent [of traffic]."

So a 1-gigabit speed IPS may be passing only 600 megabits of traffic, and the rest may go uninspected, Moy says. "It drops all traffic beyond a certain limit or lets it pass through," he says. "This is definitely one of the unknowns [about IPSes]. This is one of the things network administrators need to check on."

And the more signatures an IPS contains, the more it can hurt performance, the NSS tests showed. But that's a necessary tradeoff, according to the lab.

The report also calculated an IPS' total cost of ownership, including the purchase price, product maintenance fees, installation time, upkeep with patches and updates, and tuning policies. Using a rate of $75 per hour (based on the experienced engineers vendors sent to participate in the tests). TCO ranged from $8,579 to $234,924 per year for the IPSes, depending on the model.

"Every product has a different approach. Some of them require a lot of tuning, which is a good thing, but the bad thing is that you need to do so continually," Moy says.

And NSS Labs concluded that an IPS with the lowest sticker price isn't necessarily a better value if it doesn't offer sufficient protection. "Organizations should evaluate products based upon their value (protection, performance, and labor costs) within the context of a three-year Total Cost of Ownership (TCO)," the report says.

Meanwhile, Moy says even the best IPSes let some attacks through. "You can't expect them to be 100 percent," he says. NSS Labs is also offering a new series of reports that includes data on what attacks each product missed.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message.