Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Perimeter

6/27/2013
11:37 PM
50%
50%

'BinaryPig' Uses Hadoop To Sniff Out Patterns In Malware

At Black Hat next month, researchers will release new set of big-data tools that can find patterns in the data among security firms' massive databases of malware

As the menagerie of malware collected by security firms continues to multiply, researchers are looking for new ways to analyze the massive data sets to find interesting information in their malware zoos.

Click here for more of Dark Reading's Black Hat articles.

At the Black Hat Security Briefings in late July, one trio of researchers plans to release a framework for using Hadoop and the Apache Pig platform for parallelization that makes analyzing large sets of malware programs easier. The three researchers -- Zachary Hanif, Telvis Calhoun, and Jason Trost of Endgame Systems -- developed the framework, dubbed BinaryPig, while trying to analyze a quickly growing collection of millions of malware samples collected by the company over the past three years.

Originally, the researchers wanted to mine their collection of malware data for historical trends, but the number of malware binaries -- now at 20 million -- made it difficult to process. By moving to big-data analytics, the researchers can now analyze trends in patterns in the executable headers, look for specific file features and even do entropy analysis, says Hanif, a senior researcher with Endgame.

"It is comparatively shallow analysis compared to what heavyweight reverse-engineers do, but at scale we can take that shallow analysis and extract deep insight," he says.

The security industry has begun focusing on using big-data analytics techniques to find intelligence in their security data, from enterprises looking for signs of breaches in their log data to security companies looking for patterns in attack data from their sensor networks.

Malware analysis is an appropriate application of the techniques because attackers are generating so many variants of their programs, as a way to dodge defenses, that security firms' malware zoos have become overpopulated. McAfee's zoo, for example, topped 128 million malicious programs after gaining more than 14 million in the first quarter of 2013, according to the firm's quarterly report.

Using Hadoop and other big-data analysis methods, McAfee and other companies can reduce the tens of thousands of malware samples arriving each day into a more manageable number, says Adam Wosotoswsky, messaging security architect for McAfee.

"You are able to say, 'Here are the things that we definitely think are bad, here are the things that we definitely think are good, and here is the gray area,'" he says.

The advantage of Hadoop in working with big data is that the movement of data between machines is minimized. Instead, the processing functions are moved to where the data resides, which takes less time to accomplish.

[For big companies looking to spend big budgets, the Big Data pitch for security information and event management (SIEM) systems is a good fit. But other improvements are on the way. See More Improvements To SIEM Than Big Data.]

The approach allows novel approaches to malware analysis. For example, the Endgame researchers have analyzed the bitmaps used by malware for icons, buttons, and controls to find commonalities in families of malware. Some attempt to display the old Windows XP icon for PDF files, for example, while others use some far more recognizable images, such as skulls, says Endgame's Hanif.

"There are a handful of malware authors out there, or at least malware families, that seem to have differences in which icon they use to masquerade as a different file type," he says. "We are trying to see what the possibilities are for doing some clustering and classification based on those images."

The three researchers plan to release the framework as open-source tools at the Black Hat Security Briefings, allowing others to use and build on the work.

Dean De Beer, chief technology officer for malware analysis firm ThreatGRID, sees the release of the tools as important to helping researchers and open-source intelligence projects deal with big-data problems. While ThreatGRID has created a non-Hadoop framework to allow the storing of features culled from the static and binary analysis of malware, De Beer says the BinaryPig approach can help improve analysis.

"If there is a way that people can find a far more efficient means to search, query, and extract content, than I think that is a very very powerful tool," he said. "It would be nice to see it evolve from static extraction to handle dynamic feature extraction, however."

For Trost, Hanif, and Calhoun, however, the release of the framework means giving more tools to help the security industry adopt big-data analysis tools.

"Big data technology is going to revolutionize the security industry," Endgame's Trost says. "A lot of other industries have started to ride this wave, and I really am hoping that the security industry will take advantage of this."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message. Veteran technology journalist of more than 20 years. Former research engineer. Written for more than two dozen publications, including CNET News.com, Dark Reading, MIT's Technology Review, Popular Science, and Wired News. Five awards for journalism, including Best Deadline ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
When It Comes To Security Tools, More Isn't More
Lamont Orange, Chief Information Security Officer at Netskope,  1/11/2021
US Capitol Attack a Wake-up Call for the Integration of Physical & IT Security
Seth Rosenblatt, Contributing Writer,  1/11/2021
IoT Vendor Ubiquiti Suffers Data Breach
Dark Reading Staff 1/11/2021
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
2020: The Year in Security
Download this Tech Digest for a look at the biggest security stories that - so far - have shaped a very strange and stressful year.
Flash Poll
Assessing Cybersecurity Risk in Today's Enterprises
Assessing Cybersecurity Risk in Today's Enterprises
COVID-19 has created a new IT paradigm in the enterprise -- and a new level of cybersecurity risk. This report offers a look at how enterprises are assessing and managing cyber-risk under the new normal.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-15864
PUBLISHED: 2021-01-17
An issue was discovered in Quali CloudShell 9.3. An XSS vulnerability in the login page allows an attacker to craft a URL, with a constructor.constructor substring in the username field, that executes a payload when the user visits the /Account/Login page.
CVE-2021-3113
PUBLISHED: 2021-01-17
Netsia SEBA+ through 0.16.1 build 70-e669dcd7 allows remote attackers to discover session cookies via a direct /session/list/allActiveSession request. For example, the attacker can discover the admin's cookie if the admin account happens to be logged in when the allActiveSession request occurs, and ...
CVE-2020-25533
PUBLISHED: 2021-01-15
An issue was discovered in Malwarebytes before 4.0 on macOS. A malicious application was able to perform a privileged action within the Malwarebytes launch daemon. The privileged service improperly validated XPC connections by relying on the PID instead of the audit token. An attacker can construct ...
CVE-2021-3162
PUBLISHED: 2021-01-15
Docker Desktop Community before 2.5.0.0 on macOS mishandles certificate checking, leading to local privilege escalation.
CVE-2021-21242
PUBLISHED: 2021-01-15
OneDev is an all-in-one devops platform. In OneDev before version 4.0.3, there is a critical vulnerability which can lead to pre-auth remote code execution. AttachmentUploadServlet deserializes untrusted data from the `Attachment-Support` header. This Servlet does not enforce any authentication or a...