Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Perimeter

6/27/2013
11:37 PM
50%
50%

'BinaryPig' Uses Hadoop To Sniff Out Patterns In Malware

At Black Hat next month, researchers will release new set of big-data tools that can find patterns in the data among security firms' massive databases of malware

As the menagerie of malware collected by security firms continues to multiply, researchers are looking for new ways to analyze the massive data sets to find interesting information in their malware zoos.

Click here for more of Dark Reading's Black Hat articles.

At the Black Hat Security Briefings in late July, one trio of researchers plans to release a framework for using Hadoop and the Apache Pig platform for parallelization that makes analyzing large sets of malware programs easier. The three researchers -- Zachary Hanif, Telvis Calhoun, and Jason Trost of Endgame Systems -- developed the framework, dubbed BinaryPig, while trying to analyze a quickly growing collection of millions of malware samples collected by the company over the past three years.

Originally, the researchers wanted to mine their collection of malware data for historical trends, but the number of malware binaries -- now at 20 million -- made it difficult to process. By moving to big-data analytics, the researchers can now analyze trends in patterns in the executable headers, look for specific file features and even do entropy analysis, says Hanif, a senior researcher with Endgame.

"It is comparatively shallow analysis compared to what heavyweight reverse-engineers do, but at scale we can take that shallow analysis and extract deep insight," he says.

The security industry has begun focusing on using big-data analytics techniques to find intelligence in their security data, from enterprises looking for signs of breaches in their log data to security companies looking for patterns in attack data from their sensor networks.

Malware analysis is an appropriate application of the techniques because attackers are generating so many variants of their programs, as a way to dodge defenses, that security firms' malware zoos have become overpopulated. McAfee's zoo, for example, topped 128 million malicious programs after gaining more than 14 million in the first quarter of 2013, according to the firm's quarterly report.

Using Hadoop and other big-data analysis methods, McAfee and other companies can reduce the tens of thousands of malware samples arriving each day into a more manageable number, says Adam Wosotoswsky, messaging security architect for McAfee.

"You are able to say, 'Here are the things that we definitely think are bad, here are the things that we definitely think are good, and here is the gray area,'" he says.

The advantage of Hadoop in working with big data is that the movement of data between machines is minimized. Instead, the processing functions are moved to where the data resides, which takes less time to accomplish.

[For big companies looking to spend big budgets, the Big Data pitch for security information and event management (SIEM) systems is a good fit. But other improvements are on the way. See More Improvements To SIEM Than Big Data.]

The approach allows novel approaches to malware analysis. For example, the Endgame researchers have analyzed the bitmaps used by malware for icons, buttons, and controls to find commonalities in families of malware. Some attempt to display the old Windows XP icon for PDF files, for example, while others use some far more recognizable images, such as skulls, says Endgame's Hanif.

"There are a handful of malware authors out there, or at least malware families, that seem to have differences in which icon they use to masquerade as a different file type," he says. "We are trying to see what the possibilities are for doing some clustering and classification based on those images."

The three researchers plan to release the framework as open-source tools at the Black Hat Security Briefings, allowing others to use and build on the work.

Dean De Beer, chief technology officer for malware analysis firm ThreatGRID, sees the release of the tools as important to helping researchers and open-source intelligence projects deal with big-data problems. While ThreatGRID has created a non-Hadoop framework to allow the storing of features culled from the static and binary analysis of malware, De Beer says the BinaryPig approach can help improve analysis.

"If there is a way that people can find a far more efficient means to search, query, and extract content, than I think that is a very very powerful tool," he said. "It would be nice to see it evolve from static extraction to handle dynamic feature extraction, however."

For Trost, Hanif, and Calhoun, however, the release of the framework means giving more tools to help the security industry adopt big-data analysis tools.

"Big data technology is going to revolutionize the security industry," Endgame's Trost says. "A lot of other industries have started to ride this wave, and I really am hoping that the security industry will take advantage of this."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message. Veteran technology journalist of more than 20 years. Former research engineer. Written for more than two dozen publications, including CNET News.com, Dark Reading, MIT's Technology Review, Popular Science, and Wired News. Five awards for journalism, including Best Deadline ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
COVID-19: Latest Security News & Commentary
Dark Reading Staff 9/21/2020
Cybersecurity Bounces Back, but Talent Still Absent
Simone Petrella, Chief Executive Officer, CyberVista,  9/16/2020
Meet the Computer Scientist Who Helped Push for Paper Ballots
Kelly Jackson Higgins, Executive Editor at Dark Reading,  9/16/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Latest Comment: Exactly
Current Issue
Special Report: Computing's New Normal
This special report examines how IT security organizations have adapted to the "new normal" of computing and what the long-term effects will be. Read it and get a unique set of perspectives on issues ranging from new threats & vulnerabilities as a result of remote working to how enterprise security strategy will be affected long term.
Flash Poll
How IT Security Organizations are Attacking the Cybersecurity Problem
How IT Security Organizations are Attacking the Cybersecurity Problem
The COVID-19 pandemic turned the world -- and enterprise computing -- on end. Here's a look at how cybersecurity teams are retrenching their defense strategies, rebuilding their teams, and selecting new technologies to stop the oncoming rise of online attacks.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-4643
PUBLISHED: 2020-09-21
IBM WebSphere Application Server 7.0, 8.0, 8.5, and 9.0 is vulnerable to an XML External Entity Injection (XXE) attack when processing XML data. A remote attacker could exploit this vulnerability to expose sensitive information. IBM X-Force ID: 185590.
CVE-2020-4590
PUBLISHED: 2020-09-21
IBM WebSphere Application Server Liberty 17.0.0.3 through 20.0.0.9 running oauth-2.0 or openidConnectServer-1.0 server features is vulnerable to a denial of service attack conducted by an authenticated client. IBM X-Force ID: 184650.
CVE-2020-4731
PUBLISHED: 2020-09-21
IBM Aspera Web Application 1.9.14 PL1 is vulnerable to cross-site scripting. This vulnerability allows users to embed arbitrary JavaScript code in the Web UI thus altering the intended functionality potentially leading to credentials disclosure within a trusted session. IBM X-Force ID: 188055.
CVE-2020-4315
PUBLISHED: 2020-09-21
IBM Business Automation Content Analyzer on Cloud 1.0 does not set the secure attribute on authorization tokens or session cookies. Attackers may be able to get the cookie values by sending a http:// link to a user or by planting this link in a site the user goes to. The cookie will be sent to the i...
CVE-2020-4579
PUBLISHED: 2020-09-21
IBM DataPower Gateway 2018.4.1.0 through 2018.4.1.12 could allow a remote attacker to cause a denial of service by sending a specially crafted HTTP/2 request with invalid characters. IBM X-Force ID: 184438.