Creative Approaches To Malware Detection

Cyberwar and advanced persistent threats (APT) are fun terms thrown around a lot lately. Everyone seems to have their own slightly varied opinion on what they each mean. Personally, I don't care all that much what the different nuances of each are as long as I can understand the associated threats and deal with them appropriately.In my recent Tech Insight article, I covered some of the defenses that should be in place if you ever come under fire during cyberwar. I think what it really comes down to is that you can end up being attacked whetheryou are in the energy, finance, or similar critical industry. If you have customers or information that interests the attacker, then you could be a target -- just look at Google and the other companies hacked during the "Aurora" attacks.

Until someone proves otherwise, I keep coming back to the root cause of most attacks: the lack of due diligence in securing the resources. The software, tools, and know-how are there; companies just aren't putting them together in order to be effective. With lax defenses, I think the biggest hurdle of preparation is in the incident response and detection areas.

Traditional means of detecting malware are failing at finding advanced, targeted bots, and backdoors. There are no reliable IDS signatures to detect them (if there are, they're bleeding edge); they're blending in to look like normal software (similar names, not packed and/or not crypted [compressed and/or encrypted], etc.); use normal looking communication protocols (HTTPS/HTTP) to essentially, hide in plain sight.

My blog post about throwing manpower at botnet detection gets at the issue of dealing with detection by learning what's normal on your network, using bleeding edge Snort rules, and dedicating someone to look for anomalies (because malware authors don't code malware communications to meet network standards). But how do we go about detection at the host-level if we can't rely on antivirus?

I hate to mention it again because I've mentioned application whitelisting several times lately, but think about it. If you've whitelisted all of your known-good applications, then there's only a few things that will trigger alerts. The first will be apps you don't want your users running. The other is likely to be malware. Write some correlation rules to detect the attempted execution of files from temporary Internet file folders, system temp folders, and removable media -- you'll find some good stuff...or bad stuff, I should say.

My last blog covered using Software Restriction Policies in Windows to combat malware. You can almost think of SRPs as a poor man's application whitelisting tool that could be used along the same lines. I've even seen one diligent security guy who used the McAfee VirusScan Access Protection rules I mentioned in the same blog for detecting malware infections, or attempted infections even when the antivirus component detected nothing was wrong.

There's something fun about taking a non-traditional approach to security. Some call it "thinking outside the box," but the successful attackers are not only persistent, but often very creative with their approach. It's about time more security professionals took a creative approach to detecting and defeating attacks. Along those lines, I'll take a look at memory analysis and some interesting new approaches to detecting malware in memory.

John H. Sawyer is a senior security engineer on the IT Security Team at the University of Florida. The views and opinions expressed in this blog are his own and do not represent the views and opinions of the UF IT Security Team or the University of Florida. When John's not fighting flaming, malware-infested machines or performing autopsies on blitzed boxes, he can usually be found hanging with his family, bouncing a baby on one knee and balancing a laptop on the other. Special to Dark Reading.