When companies relied on the static analysis of binaries to determine whether a program is malicious, attackers came up with a simple way to bypass defenses: obfuscating the code with packers and other techniques.

Many security firms then moved onto dynamic analysis, allowing a program to run in a sandboxed or virtual environment and looking for signs that it was doing something malicious. The strategy is a significant departure from the past, when authors created code that would noisily attempt to exploit a number of vulnerabilities, says Michael Sutton, vice president of research for Zscaler, a cloud security provider.

"A few years ago, we would see them throw everything at the machine, and hoped that one of them worked," he says. "The downside is that it creates more noise and is more likely be picked up by host-based AV. Now, we see them being more surgical in their attacks and only delivering the payload that will work on the compromised platform."

Attackers are increasingly using evasion techniques to foil automated analysis, the latest moves in an ongoing cat-and-mouse game between malware authors and security analysts. While evasion is far from a standard feature of malware, it is frequently used in exploit packs--the attack toolkits developed and sold by rogue developers--an attempt by the authors to delay the reverse engineering of their latest attacks. Recent malware--such as DarkLeech and the latest variants of Capshaw and Kelihos--are examples of the possibilities of evasions.

[Following a program's evolution back to the author may not yet be a reality, but computer scientists are searching for more accurate measures of the relationships between software versions. See Researchers Seek Better Ways To Track Malware's Family Tree.]

Always checking for vulnerable components and only attempting to compromise the system if those components are present, is one of the three ways in which attackers attempt to sabotage defender's analysis. Attackers can also attempt to detect whether its running in a virtual machine or an analysis environment. in many cases, the attackers know the environments used by defenders to analyze malware, so can create effective method of evading detection, says Giovanni Vigna, director of the Center for CyberSecurity at the University of California at Santa Barbara.

"Evasion is the reaction to dynamic analysis, to the sandbox, and it's very difficult to catch," says Vigna. "Most of the analysis right now is done manually after observing that there is bad stuff that has not been detected--that is, a false negative--which is then analyzed manually to find the evasion."

If the attacker is unfamiliar with the analysis environment, they can still use techniques to fool typical analysis setups, such as sleeping for a long period or waiting for human input. Finally, some attackers are starting to gather intelligence to see if the system which they are trying to infect could be a honeypot or even a known compromised systems. A recent version of Kelihos, for example, check Internet black lists to see if the about-to-be-compromised system will likely be blocked.

"They have multiple evasions that targets each environment," says Alexandros Kapravelos, a PhD student in computer science at UCSB and the co-creator of the Revolver system for detecting evasive malware. "While some of these evasions hit our system, the other ones are designed to hit somebody else."

At the USENIX Security Conference in August, Kapravelos, Vigna, and three other researchers from UCSB and the University of Birmingham presented a method for detecting evasive Web malware. The Revolver system creates abstract representations of a program's function and then uses clustering and other machine learning techniques to match the code with known good and bad software. Malicious software that uses evasions become more visible when looked at from a run-time point of view.

Since September 2012, the researchers collected almost 6.5 million Web pages, of which about 266,000 were malicious. From those pages, the researchers culled more than 700,000 benign scripts and 5,700 malicious scripts, which also included 150 different evasion techniques. Some of the evasions took advantages of differences between the Internet Explorer and the browser implemented by the analysis system, known as Wepawet. Other evasions used differences in the rendering of PDF objects to detect the analysis environment.

When the researchers found the attackers using an evasion technique, they would patch their analysis system. Generally, within a few days, attackers would return with a new evasion, the researchers stated in their paper.

This makes "a tool like Revolver necessary to automatically keep track of this behavior and keep false negative detections as low as possible," the researchers stated.

Keeping up with the attackers requires automation, an approach that is used in some form by many security firm in their own fights with attackers, says Zscaler's Sutton.

"That overall approach is fairly accepted in the AV community," he says. "Otherwise you just can't keep up, because there are millions of pieces of malware every day."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.