While the Flashback Trojan has garnered its share of digital ink for its assault on the Mac OS X operating system, security researchers and computer scientists fear a different aspect of the malware -- its ability to lock itself to a specific infected system.

Flashback, which started spreading in September, consists of a number of components: a downloader that infects systems, and modules that are fetched from Internet hosts to add functionality to the Trojan. Such a division of labor is standard for botnets and Trojan downloaders. However, the attack tool's use of encryption to bind downloaded modules to the infected system -- similar to how digital-rights-protected content is licensed and bound to a single playback device -- is new.

The problem for security firms and researchers is that encrypted malware makes automated malware analysis much harder, says Paul Royal, a research scientist at the Georgia Institute of Technology's Information Security Center (GTISC).

"Flashback is the first widespread threat we've seen that employs an analysis-resistant technique to evade malware analysis by making execution of the sample dependent on the unique properties on the host that it infects," he says.

Just as defenders are focused on attacking the economics of cybercrime and making online thefts more difficult to conduct, attackers are focusing on raising the effort needed to defend against their attacks. Already, server-side polymorphism, which uses encryption and data-compression techniques to foil static analysis, has reduced the ability of antivirus scanners to detect "new" attacks. (In reality, the attacks are known malware with bits scrambled to evade detection.)

The next steps involve the attacker producing a countermeasure to foil automated dynamic analysis, a method security firms rely on for classifying malware samples found on the Internet and that have attacked customers' systems. The goal is not to prevent analysis of the malware, but to make automated analysis less efficient.

[The weapon of choice for a cyberspy gaining a foothold inside its target traditionally has been the socially engineered email with a malicious link or attachment, but they are increasingly targeting specific, legitimate websites and injecting them with malware. See Cyberspies Target Victims Via 'Strategic' Drive-by Website Attacks.]

In unpublished research, Royal and researchers at Georgia Tech have identified two possible tactics that could significantly slow down analysis: host identity-based encryption (HIE) and instruction-set localization.

HIE uses information found on the infected system to create a key that is used to encrypt, and therefore bind, some of the attack code to the system.

Instruction-set localization, meanwhile, can use information about the network location to create an interpreter that can decrypt commands sent by a command-and-control (C&C) server. The idea is that only infected systems inside the proper network will be able to decode the instructions.

"You don't actually have to bundle your malicious behavior into the executable that is served to a compromised system," Royal says. Instead, the malware is an interpreter of encrypted commands and the malicious actions are taken by the C&C server. "In that way, it's a little bit like platform-as-a-service," he says.

While the research may have pinpointed better ways to cause trouble for automated analysis, security firms have been dealing with such countermeasures for some time, says Liam O Murchu, manager of operations for Symantec's security response group. Many Russian cybercriminals, for example, have avoided infecting systems in Russia by checking for a localized version of Windows. To deal with such issues, Symantec and other companies host their analysis platforms in various countries -- or use network trickery to accomplish the same effect -- and use virtualized systems in a wide variety of languages.

"It's true that it makes things a little bit more difficult," says O Murchu. "But it just means that you need to have a more advanced automated infrastructure to deal with those scenarios."

Attackers are doing a lot of analysis themselves to identify researchers and their automated analysis platforms. They are doing as much work to detect the defenders as the defenders are doing on them, says Dean De Beer, chief technology officer of malware-analysis firm ThreatGrid.

"As the crimeware industry has grown, you see them moving toward active analysis as well, and asking, 'How are the defenders going to try to detect us?'" he says. "That is why we see a lot of creative ways to detect sandboxes and researchers."

Security companies and researchers have a limited menu of countermeasures, GTISC's Royal says. Companies could push the analysis onto the end users' systems, allowing even programs bound to the system to be analyzed. However, most enterprises will not want to allow their systems to be infected, he acknowledges.

A better solution is to try to replicate the manner in which the system got infected. A drive-by download? Go to the same website. An infected e-mail? Open the same attachment. Yet attackers could still use a one-time URL to serve up malware, foiling such defenses.

"That is, unfortunately, part of the asymmetry when we talk about these topics," Royal says. "The attacker has to do very little work to make it difficult for the defender to analyze the attack."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.