Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Operations

4/18/2016
08:00 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

MIT AI Researchers Make Breakthrough On Threat Detection

New artificial intelligence platform offers 3x detection capabilities with 5x fewer false positives.

Researchers with MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) believe that can offer the security world a huge boost in incident response and preparation with a new artificial-intelligence platform it believes can eventually become a secret weapon in squeezing the most productivity from security analyst teams.

Dubbed AI2, the technology has shown the capability to offer three times more predictive capabilities and drastically fewer false positive than todays analytics methods.

CSAIL gave a sneak peek into AI2 in a presentation to the academic community last week at the IEEE International Conference on Big Data Security, which detailed the specifics of a paper released to the public this morning. The driving force behind AI2 is its blending of artificial intelligence with what researchers at CSAIL call "analyst intuition," essentially finding an effective way to continuously model data with unsupervised machine learning while layering in periodic human feedback from skilled analysts to inform a supervised learning model.

"You can think about the system as a virtual analyst,” says CSAIL research scientist Kalyan Veeramachaneni, who developed AI2 with former CSAIL postdoc Ignacio Arnaldo, who is now a chief data scientist at PatternEx. “It continuously generates new models that it can refine in as little as a few hours, meaning it can improve its detection rates significantly and rapidly.”

This offers the best of both worlds in what has become a bright line division in security analytics today. For the most part, security systems today either depend on analyst-driven solutions that rely on rules created by human experts or they lean heavily on machine-learning systems for anomaly detection that trigger highly disruptive false positive rates.

Gain insight into the latest threats and emerging best practices for managing them. Attend the Security Track at Interop Las Vegas, May 2-6. Register now!

In the paper released today, Veeramachaneni, Arnaldo and their team showed how the system did when tested with 3.6 billion pieces of log data generated by millions of users over three months. During this test, the platform was able to detect 85% of attacks, three times better than previous benchmark, while at the same time reducing false positives by a factor of five.

The approach of melding together human- and computer-based approaches to machine learning has long run into stumbling blocks due to the challenge of manually labeling cybersecurity data for algorithms. The specialized nature of analyzing the data makes it a difficult data set to crack with typical crowdsourcing strategies employed in other arenas of big data analysis. The average person on a site like Amazon Mechanical Turk would be hard-pressed to apply accurate labels for data indicating DDoS or exfiltration attacks, Veermachaneni explained.

Meanwhile, security experts have already tried several generations worth of supervised machine learning models only to find that 'feeding' these systems ends up creating more work rather than saving an analyst time. This is what has lead many organization to dump early analytics solutions in the proverbial waste bin after experiencing those frustrations.

AI2 is able to perform better by bringing together three different unsupervised learning models to sift through raw data before presenting data to the analyst. So on day one, that system offers 200 of the most abnormal events to an analyst, who then manually sifts through those to identify the real attacks. That information is fed back into the system and even within a few days the unsupervised system is presenting as few as 30 to 40 events for verification.

“The more attacks the system detects, the more analyst feedback it receives, which, in turn, improves the accuracy of future predictions,” Veeramachaneni says. “That human-machine interaction creates a beautiful, cascading effect.”

Check out this video for a quick overview of the way AI2 works.

Related Content: 

 

Ericka Chickowski specializes in coverage of information technology and business innovation. She has focused on information security for the better part of a decade and regularly writes about the security industry as a contributor to Dark Reading.  View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Oldest First  |  Newest First  |  Threaded View
ONIHadoop
100%
0%
ONIHadoop,
User Rank: Apprentice
4/19/2016 | 5:46:28 PM
Open Network Insight
This is very interesting work, thanks for reporting on it.  We agree that there needs to be a new approach to cyber security that leads with machine learning against large datasets.  Our open source project, Open Network Insight, was launched on this same premise and provides insight and operational analytics for network flows, domain name service data and full packet captures.  

If anyone reading this article is interested in learning more please visit our website or github.
Mike Anders
50%
50%
Mike Anders,
User Rank: Apprentice
6/7/2016 | 12:24:22 AM
Re: Open Network Insight
Object Based Production (OBP) and Activity Based Intelligence (ABI) can assist in achieving "the best of both worlds" with respect to detection when both OBP and ABI are brought to bear on the cyber problem. Not a lecture, just an observation!
GitHub Named in Capital One Breach Lawsuit
Dark Reading Staff 8/14/2019
The Mainframe Is Seeing a Resurgence. Is Security Keeping Pace?
Ray Overby, Co-Founder & President at Key Resources, Inc.,  8/15/2019
The Flaw in Vulnerability Management: It's Time to Get Real
Jim Souders, Chief Executive Officer at Adaptiva,  8/15/2019
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Current Issue
7 Threats & Disruptive Forces Changing the Face of Cybersecurity
This Dark Reading Tech Digest gives an in-depth look at the biggest emerging threats and disruptive forces that are changing the face of cybersecurity today.
Flash Poll
The State of IT Operations and Cybersecurity Operations
The State of IT Operations and Cybersecurity Operations
Your enterprise's cyber risk may depend upon the relationship between the IT team and the security team. Heres some insight on what's working and what isn't in the data center.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2019-5034
PUBLISHED: 2019-08-20
An exploitable information disclosure vulnerability exists in the Weave Legacy Pairing functionality of Nest Cam IQ Indoor version 4620002. A set of specially crafted weave packets can cause an out of bounds read, resulting in information disclosure. An attacker can send packets to trigger this vuln...
CVE-2019-5035
PUBLISHED: 2019-08-20
An exploitable information disclosure vulnerability exists in the Weave PASE pairing functionality of the Nest Cam IQ Indoor, version 4620002. A set of specially crafted weave packets can brute force a pairing code, resulting in greater Weave access and potentially full device control. An attacker c...
CVE-2019-5036
PUBLISHED: 2019-08-20
An exploitable denial-of-service vulnerability exists in the Weave error reporting functionality of the Nest Cam IQ Indoor, version 4620002. A specially crafted weave packets can cause an arbitrary Weave Exchange Session to close, resulting in a denial of service. An attacker can send a specially cr...
CVE-2019-8103
PUBLISHED: 2019-08-20
Adobe Acrobat and Reader versions, 2019.012.20035 and earlier, 2019.012.20035 and earlier, 2017.011.30142 and earlier, 2017.011.30143 and earlier, 2017.011.30142 and earlier, 2015.006.30497 and earlier, and 2015.006.30498 and earlier have an out-of-bounds read vulnerability. Successful exploitation ...
CVE-2019-8104
PUBLISHED: 2019-08-20
Adobe Acrobat and Reader versions, 2019.012.20035 and earlier, 2019.012.20035 and earlier, 2017.011.30142 and earlier, 2017.011.30143 and earlier, 2017.011.30142 and earlier, 2015.006.30497 and earlier, and 2015.006.30498 and earlier have an out-of-bounds read vulnerability. Successful exploitation ...