Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Analytics //

Security Monitoring

6/28/2013
11:37 PM
50%
50%

Machine-Learning Project Sifts Through Big Security Data

As the volume of data created by security and network devices multiplies, researchers look for ways to teach computer to better highlight attack patterns

As an information-security consultant, Alexandre Pinto spent 12 years helping companies set up difficult-to-configure systems to cull security intelligence from logs and security events.

Click here for more of Dark Reading's Black Hat articles.

Yet configuring the systems required months of work and even then needed constant maintenance to enable them to detect the latest threats and pinpoint likely malicious traffic. He realized that while companies may want to monitor their networks for threats, they typically have too few security people to work through data from far too many logs -- a problem that will only get worse as companies seek to sift through more operational data to detect threats. Big data could be the downfall of security if companies don't find better ways of dealing with the growing volumes, he says.

"What chance do we have: We can't find the needle in the haystack as it is now, and now the haystack is 100,000 times larger," he says. "We are going to need help."

To help solve the problem, Pinto has worked during the past six months on a machine-learning system that can take logs and identify traffic that originates from suspicious neighborhoods of the Internet. Dubbed MLSec, the project uses supervised learning algorithms to identify networks that are home to malicious actors. Pinto plans to demonstrate the tool at the Black Hat Security Briefings in July.

The independent researcher started with data from the SANS Institute's DShield project, which gathers firewall logs from participating community members. Pinto trained the system on 1.2 million events from 30 million log entries as well as other data submitted by volunteers. When comparing his results to known blacklists, the machine-learning algorithm appeared to be accurate in 92 to 95 percent of cases. Unlike blacklists, however, the systems does not need to be told which networks are malicious; it creates its own representation of the Internet.

Such a system can help information security managers by more accurately flagging traffic coming from questionable areas of the Internet, says Johannes Ullrich, dean of research for the SANS Technology Institute. In addition, the system can give administrators the best guess of the maliciousness of incoming traffic based on incomplete information, he says.

"It really helps to direct the attention of the security administrator," Ullrich says. "The big-data approach filters the data down to a subset, so you know what is worth looking at."

[Rather than watching for communications between infected systems and command-and-control servers, companies can detect stealthy malware when it attempts to spread. See Researcher To Open-Source Tools For Finding Odd Authentication Behavior.]

For companies with overworked staff, the ability to cull the run-of-the-mill data from the interesting -- potentially malicious -- traffic can be a great benefit. In addition, a machine-learning system can be constructed to adapt far faster than a human as the attackers change their tactics, Pinto argues.

"The model will outperform the expert because the model does not forget the data," he says. "It selectively diminishes the weight of what happened before, as time goes by, but it does not forget it."

Pinto plans to make the system available as a service to anyone to upload their firewall logs. In exchange, the people will get a report that summarizes the findings of the system.

In the end, the more people who use the system, the better the results should be, Pinto says.

"In machine learning, of course, the algorithm is important, but the more data that you throw at it, the better," he says. "This is the perfect fit with data security. The more you are attacked, the better your defenses should get."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message. Veteran technology journalist of more than 20 years. Former research engineer. Written for more than two dozen publications, including CNET News.com, Dark Reading, MIT's Technology Review, Popular Science, and Wired News. Five awards for journalism, including Best Deadline ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Florida Town Pays $600K to Ransomware Operators
Curtis Franklin Jr., Senior Editor at Dark Reading,  6/20/2019
Pledges to Not Pay Ransomware Hit Reality
Robert Lemos, Contributing Writer,  6/21/2019
AWS CISO Talks Risk Reduction, Development, Recruitment
Kelly Sheridan, Staff Editor, Dark Reading,  6/25/2019
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
Building and Managing an IT Security Operations Program
As cyber threats grow, many organizations are building security operations centers (SOCs) to improve their defenses. In this Tech Digest you will learn tips on how to get the most out of a SOC in your organization - and what to do if you can't afford to build one.
Flash Poll
The State of IT Operations and Cybersecurity Operations
The State of IT Operations and Cybersecurity Operations
Your enterprise's cyber risk may depend upon the relationship between the IT team and the security team. Heres some insight on what's working and what isn't in the data center.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2019-10133
PUBLISHED: 2019-06-26
A flaw was found in Moodle before 3.7, 3.6.4, 3.5.6, 3.4.9 and 3.1.18. The form to upload cohorts contained a redirect field, which was not restricted to internal URLs.
CVE-2019-10134
PUBLISHED: 2019-06-26
A flaw was found in Moodle before 3.7, 3.6.4, 3.5.6, 3.4.9 and 3.1.18. The size of users' private file uploads via email were not correctly checked, so their quota allowance could be exceeded.
CVE-2019-10154
PUBLISHED: 2019-06-26
A flaw was found in Moodle before versions 3.7, 3.6.4. A web service fetching messages was not restricted to the current user's conversations.
CVE-2019-9039
PUBLISHED: 2019-06-26
The Couchbase Sync Gateway 2.1.2 in combination with a Couchbase Server is affected by a previously undisclosed N1QL-injection vulnerability in the REST API. An attacker with access to the public REST API can insert additional N1QL statements through the parameters ?startkey? and ?endkey? of the ?_a...
CVE-2018-20846
PUBLISHED: 2019-06-26
Out-of-bounds accesses in the functions pi_next_lrcp, pi_next_rlcp, pi_next_rpcl, pi_next_pcrl, pi_next_rpcl, and pi_next_cprl in openmj2/pi.c in OpenJPEG through 2.3.0 allow remote attackers to cause a denial of service (application crash).