Starting in late 2011, unknown attackers attempted to install malicious code on a computer belonging to a client of security firm Bit9. The attack, which occurred around 6 a.m. each day, failed because the company's whitelisting technology did not recognize the program as an approved application and so blocked its installation.

Only recently was the attack given another name: Flame.

Although Bit9 and its client, which the company would not name but says is based in the Middle East, did not investigate the routine security incidents last year, recent events convinced Bit9 to search through its database of hashes to identify past executables that its technology had blocked. When it found a match, the company -- with permission -- performed forensics using the client's local database of security events. Bit9 found that a dropper had attempted to install at least two different files on the targeted system.

"Somebody had remotely targeted that system and compromised it enough to try to remotely drop executables on the computer, and we flagged them as unauthorized," says Harry Sverdlove, chief technology officer with Bit9. "It attempted to run. We said no, and that was the end of it."

Following Flame, the most recent targeted attack to hit the headlines, antivirus companies are facing a great deal of criticism for missing signs of the attack for more than four years. Even one of the industry's own, Mikko Hypponen of F-Secure, issued a mea culpa in Wired, saying that the company and its competitors could do better.

"All of us had missed detecting this malware for two years, or more," F-Secure's chief research officer wrote. "That's a spectacular failure for our company, and for the antivirus industry in general."

[ Microsoft issued an emergency patch for all versions of Windows after it discovered the attackers had abused one of its digital certificates to help spread the Flame infection from one machine to others within the targeted organization. See Flame Burns Microsoft With Digital Certificate Hack. ]

Historically, however, antivirus software's strength has been in detecting viruses, worms, and other mass attacks. More recent improvements, such as threat communities and cloud analysis, continue to shorten the delay between detection and the distribution of specific protections. Yet antivirus and anti-malware programs continue to be ill-suited to detect the low-volume threats like targeted attacks.

It's not just nation-state attacks, such as Stuxnet and Duqu, both of which spread for at least 12 months before detection. Cybercriminals routinely run their own targeted attacks against antivirus firms' software to make sure they are not detected. In more than 300 investigations performed by security firm Trustwave in 2011, all involved malware and none were detected by the antivirus software installed on the clients' systems, the company stated in its Global Security Report earlier this year.

"The clients would say, 'We were running antivirus on this system, and we know we updated all of our signatures -- why wasn't this caught?'" says Nick Percoco, senior vice president and head of Trustwave's SpiderLabs. "The vast majority of people don't understand that the bad guys can test target an environment and write a piece of malware to evade detection."

To detect targeted threats, companies must first be more aware of what is going on in their networks, Percoco says. By watching for events -- and not just suspicious activity -- a company can detect the existence of an infection. Known as indicators of compromise, or IOCs, these events can tip a company off that something unwanted is inside the firewall.

"We have found that a chain of three or four positive events -- such as a successful login followed by Web activity and an uptick in disk utilization -- can equal something negative, a compromise," he says.

What works at the network level can also work at the systems level. Because there are so many attack vectors today, it is hard to watch every one; instead, companies can monitor systems and memory for the telltale evidence that something bad is happening, says Pascal Longpre, chief technology officer for anti-malware firm Silicium Security. The company's software analyzes events in the system memory to detect anomalies that may indicate an infection.

"Our approach looks at the behavior of the system," he says. "And then we send that to a central server, where a security expert can make the call."

Finally, companies can take the "deny all" approach to applications, just like the recommended practice for firewall rules. Known as whitelisting, the defensive technology allows only known good programs to run on systems. With millions of variants of malware being generated every year, focusing on the 10,000 to 25,000 programs running on a typical system make more sense, Bit9's Sverdlove says.

"Just trying to keep up with the bad stuff and trying to identify more and more malware is not an effective solution," he says.

Sverdlove stresses that whitelisting has grown up. Once known for its difficulty to maintain the trusted applications lists, whitelisting now focuses on accepted general policies.

In the end, it's not so much that antivirus is not working, but that people are expecting software created to detect commoditized attacks to work against made-to-order targeted attacks. Companies need to use the right defenses for the job, Silicium's Longpre says.

"If you want to protect your office, you put a lock on the door, but there is only so much a lock can do," he says. "Instead, you start adding other defenses, such as video cameras and motion sensors. Thats the approach we need to take."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.