Some of the top advanced threat detection products failed to catch custom-written malware samples posing as targeted attacks in an independent lab study.

Researchers from the Laboratory of Cryptography and System Security (CrySyS) Lab and MRG Effitas teamed up to test five "well-established" advanced threat detection appliances to see just how effective these technologies are in spotting unknown threats. The goal of the tests was not to determine the detection rates of the products, but rather to see whether they could bypass them. The researchers did not reveal the names of the products.

One of the four custom samples written by the researchers snuck past all five of the products, while another bypassed three of them. The two most basic samples were detected by all five of the products, but in some cases they registered only a low-severity alarm.

The big takeaway from the tests, according to the researchers, is that no security tool is infallible when it comes to new malware samples. "A lot of customers believe these products can detect all advanced attacks. Believing this can provide a false sense of security," Zoltan Balazs of the UK security research firm MRG Effitas, said in an email interview.

Even so, these appliances are a key layer of security: "Defense in depth is still important, as there are always unexpected areas where advanced attackers can be detected," he said. "These products add value, and can detect attacks which won't be detected by other technologies deployed at enterprises."

All the malware test samples were devised with typical RAT features of remote code execution, along with the ability to download and upload files.

The stealthiest of the homemade samples -- dubbed "BAB0"-- that bypassed all five products was downloaded by the "victim" from a web page and was hidden behind an image using steganography. Among other things, the simulated attack hides command and control traffic inside HTTP requests.

The researchers plan to publish some components of BAB0 to help anti-APT/advanced threat protection vendors to beef up their products, as well as to help organizations test the strength of those appliances in their organizations.

"If we were able to bypass all products, then advanced attackers are surely able, too. Maybe not in the same way as we did. Maybe in an even better way," Levente Buttyan of CrySyS Lab said in an email interview. "So it is very important that vendors work together with independent testers more frequently, but our experience is that they are very reluctant to participate in tests. This should change."

However, it won't be easy for vendors to stop advanced threats,, Buttyan said.

Meanwhile, there's a range of effectiveness among various appliances, according to CrySys Labs' Boldizsar Bencsath.

Tom Kellermann, chief cyber security officer with Trend Micro, said his company's Deep Discovery product was not among the tools tested in the study. The problem with many products in this category is that they can't evaluate the lateral movement of malware across more than five protocols, and they lack proper sandboxing and correlation of unknown events, so advanced attacks can sneak by them.

The full lab report was published today.