Decoding the methods in malicious code is becoming more difficult, according to reverse-engineering experts. Attacks no longer scramble simple function names, but encrypt entire blocks of code.

Attackers use obfuscation to make it harder to analyze malicious software and stymie security tools, such as intrusion-detection systems, from recognizing the attack. Initially, obfuscation merely scrambled the names of the functions being called by a program, complicating analysis of the binary code.

As automated reverse engineering makes progress, however, malware authors are increasingly scrambling entire blocks of code and using better obfuscation techniques to make analysis and detection that much harder, says Adam Meyers, director of cybersecurity operations for SRA International. "At the business end of the malware, it is getting very complex and confusing," says Adam Meyers, director of cybersecurity operations for SRA International. Meyers will speak at the SOURCE Boston conference next week in a talk on reverse-engineering techniques to deal with obfuscation.

Part of the problem is that attackers are using so many different ways of getting onto systems, experts say. Attacks that use social engineering will use obfuscated Web addresses and code. Drive-by downloads, which infect people when they visit a website, will encrypt their payloads. And more direct measures aimed at servers will scramble the code to evade intrusion-detection systems, says Matt McKinley, director of product management at network security firm Stonesoft.

"The battle from the defenders' side is big," McKinley says. "There is a lot of things that you have to reverse engineer."

Reverse engineering has become an extremely important security function. In March, online giant Google bought reverse-engineering firm Zynamics, the maker of a number of tools to help analyze binary executables.

Currently, most obfuscation is simple, using operations such as XOR-ing bits or rotating through alphanumeric characters, says SRA's Meyers, who spent three years at the U.S. State Department handling security and reverse-engineering attacks. Increasingly, however, the attackers are using better encryption or customized functions to make reverse engineering more difficult.

Oddly, the targeted attacks that most call "advanced persistent threats" (APTs) are not always the most difficult to reverse engineer, Meyers says. The mercenary developers that create software for cybercriminals are more likely to use encryption and other advanced forms of obfuscation. One reason is that competition in the malware market has led to better tools. In addition, commercial malware increasingly uses digital rights management (DRM) technology to prevent customers from becoming competitors.

"I've seen people say things about APT -- targeted attacks -- who may not have been familiar with APT," Meyers says.

The newer the malware category, the more likely that better obfuscation will be used, experts say. Meyers, for example, has encountered DES encryption obfuscating mobile malware. DES stands for data encryption standard, an older method for scrambling data that is no longer considered secure but is more than adequate for obfuscation.

"People that are writing new malware are fixing a lot of the mistakes that had been made before," Meyers says.

Have a comment on this story? Please click "Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.