[Excerpted from "Malware War: How Malicious Code Authors Battle to Evade Detection," a new, downloadable report available this week on Dark Reading's Advanced Threats Tech Center.]

Reverse engineering is the enemy of trade secrets. Someone in a competitor's laboratory could pour a bottle of Coke into a mass spectrometer and figure out what makes Coke, um, Coke.

Malware "vendors" have, for quite some time, recognized that reverse engineering is their enemy. Every day, an army of security researchers deploys a wide array of mass spectrometer-like tools on thousands of suspicious executables. The results of that work are wrapped into the antimalware tools that run on servers and PCs around the world. And every day, malware vendors alter their products to evade those same security tools and to make their malicious code more difficult to reverse engineer.

With Malware, Inc. churning out more than a million new pieces of malicious code in the first half of 2010 alone, according to SPAMfighter, the odds are already against the anti-malware vendors. If your livelihood depends on your malware going undetected as long as possible, then an enterprising malicious code author will take whatever steps necessary to deter detection.

Face it: Anti-malware is a reactive technology. You can develop a signature (and, like it or not, most anti-malware software is still signature-based) for a specific piece of malicious code only if you’ve seen that code. While security vendors all deploy honeypots and Web-browsing drones, the vast majority of files they investigate still come from user submissions. If you can keep the anti-malware vendors from ever seeing your malicious masterpiece, it likely won’t ever be tagged as malware by their tools.

Malware vendors follow a few simple rules to significantly reduce the likelihood that their malware will be captured by or submitted to an anti-malware company for analysis:

• Limit malware distribution. On the surface, this appears to fly directly in the face of Malware, Inc.’s more general business model. It is, however, characteristic of a growing segment of the business’ product line, "boutique malware," which is tightly targeted toward a single company, a single department, or even a single individual. Boutique malware generally has one goal -- gathering business intelligence -- and its limited-distribution model represents a significant problem for an anti-malware industry built on signatures.

Similarly, we are seeing an increase in the “long-tail” approach to malware distribution. Instead of trying to attack the maximum number of machines with a relatively limited number of malicious programs, Malware, Inc. distributes huge numbers of discrete programs and variants to mask the "mass outbreak" phenomenon.

• Communicate sparingly and only over mechanisms that mimic user behavior. Make it look like a duck, walk like a duck, and talk like a duck, when it’s really a hawk.

• Use the resources of the infected machine in moderation. Few things trigger an investigation more quickly than a slow server or a dodgy end-user machine.

The above "best practices" to avoid the security labs notwithstanding, malware authors employ various techniques to impede security analysts who try to reverse engineer their code.

Code obfuscation techniques have been around almost as long as coding itself, and have a useful life outside of malware; many commercial applications use these same techniques to protect legitimate trade secrets. Many malicious code authors (and reverse engineers) first cut their teeth cracking the techniques used to copy-protect applications and games in the early days of the computer revolution.

Code packers are the executable-code equivalent of this type of script obfuscation. A code packer works by encoding an executable program and then tacking a short executable “decoding stub” at the beginning of the encoded data. When this stub is executed, it decodes the encoded data back into something executable and then launches the decoded program.

The Achilles’ heel of the entire obfuscation process is also pretty easy to grasp: The obfuscated program (whether a script or an executable) must be decoded before it can execute. Whether malware analysts work on an obfuscated executable "by hand" or automatically, with an anti-malware tool the idea is to allow the program to decode itself, then grab a copy of the decoded program immediately before it executes. This can be done using a debugger, such as IDAPro or Ollydbg, or a specialized tool, like LordPE.

The battle between anti-malware vendors and malware authors can, in many cases, be seen as a series of escalations. One side finds some new technique to help its cause, which, in turn, provokes the other side to raise the bar as well.

When antivirus researchers began to develop techniques and then tools to deal with this type of obfuscation, Malware, Inc. stepped up its game, applying multiple layers of encoding, mixing encoding mechanisms, and so on. Unfortunately for the obfuscators, you can alter your code only so much before performance begins to suffer.

Malware, Inc. needed to raise the bar some other way. You really can’t be an effective malware author without knowing something about how the security vendors -- your arch nemesis -- ply their trade. To be proficient at evading reverse engineering, malicious coders need to know about the tools that will be used against them. One of the best ways to evade reverse engineering is for malicious code to do something that breaks those tools.

To find out more about how malware authors attempt to render security analysis tools ineffective -- and the specific tools they use -- download the free report.

Have a comment on this story? Please click "Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.