Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Analytics //

Security Monitoring

10/10/2013
06:54 PM
50%
50%

Big Data Detectives

Could big data be the key to identifying sophisticated threats? Security experts are on the case.

For Vigilant, it started in 2009. And as with most companies, it started small.

The security services startup, now part of audit and consulting firm Deloitte, wanted a way to bring information about external threats to clients that were using SIEM (security information and event management) systems to monitor their own environments. The Vigilant team knew that the combination of external threat data with internal security event data could be a powerful way to improve enterprise defenses, but crunching all that data would be a monumental task.

Vigilant began combining threat intelligence feeds, filtering the data to pull out the most important information for each client, and then transmitting the data to their clients' SIEM systems. The company started with two threat lists: domains serving malware, and domains compromised by the Trojans SpyEye and Zeus. To reduce false alarms and aid in analysis, the company began adding more data feeds.

Vigilant's analysts quickly became addicted to the analysis. Each new source of data gave them the ability to tease out additional information on threats. By 2011, the company was processing about 50 to 100 GBs per day. But the company's systems couldn't keep up with the flow of data, and it started missing performance deadlines, says Joe Magee, co-founder and former CTO of Vigilant, who is now a director at Deloitte.

"We were not able to catch up," Magee says. "We were not able to process the information and push it out fast enough, and that's when it became a big data issue for us. We needed to be able to rip through this data in Google-like fashion."

The volume of data and rate of change caused the problem, because most of the data came in the form of feeds updated daily with gigabytes of data. It overwhelmed the company's initial database built on top of Postgres. In 2011, Vigilant moved to Hadoop and became one of many companies -- both vendors and enterprises -- that are advocating the use of big data analytics to improve the response to security threats.

Big Data Still Just A Promise

For security teams, the use of analytics on massive quantities of security data -- from device and application logs to collections of captured network packets and operational business data -- promises better visibility into the security threats that elude current defenses.

Big data analytics can be more complex than the log collection and analysis conducted by most SIEM systems, so automating the number crunching is often needed to let security pros more easily use statistical correlations to discover trends and anomalies. Tracking days or weeks of business activity allows the system to find outliers -- a user who accesses far more data on a daily basis than the average employee, or a system that has a sudden spike in resource consumption. Analysts then can dig deeper into the large data sets of security information for any flagged events.

"Big data is not just about gaining insights, it's about helping remediate issues faster," says Jason Corbin, director of security intelligence strategy for IBM Security Systems. "The big problem is that [security teams] are overwhelmed with information they have. All that information goes to some guy who has to sift through tons of incidents or vulnerability reports and decide what they need to patch or virtually patch or fix. Security teams fall behind, and that's how companies suffer breaches based on known but unpatched vulnerabilities."

But for many companies, the promise of big data in security is just that -- a promise. While security teams hope to gain more awareness of what is going on in their networks by collecting and analyzing more of their data, the technology is still in its adolescence. "Hadoop has been around for a while, but it is still figuring out what it is and what is wants to be," says Adrian Lane, CTO for security consultancy Securosis.

Still, the potential is huge, Lane adds. Companies that kick off a big data project for security can collect an immense volume of data and have a security analyst poke through the information, ask queries of the data and make important discoveries.

chart: Which of these big data tools are in use at your company

Veteran technology journalist of more than 20 years. Former research engineer. Written for more than two dozen publications, including CNET News.com, Dark Reading, MIT's Technology Review, Popular Science, and Wired News. Five awards for journalism, including Best Deadline ... View Full Bio

Previous
1 of 3
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Chuck Brooks
50%
50%
Chuck Brooks,
User Rank: Apprentice
10/19/2013 | 6:20:48 PM
re: Big Data Detectives
Situational awareness on both internal and external threats can be enhanced via data analytics. False alarms a major issue and the predictive and forensic aspects of data analysis is an imperative. Vigilant is a good acquisition for Deloitte.
AnonymousMan
50%
50%
AnonymousMan,
User Rank: Moderator
10/14/2013 | 10:01:35 PM
re: Big Data Detectives
Uhg, another Big Data article. "APT" must be running out of steam. I'd like to see some discussion of the NEW analytics being performed. Don't just backend your [SIEM] solution with Hadoop and call it a "Big Data" solution. This re-branding has been going on for as long as I've been in the security industry. This industry makes me want to move into the woods and make moonshine for a living.
COVID-19: Latest Security News & Commentary
Dark Reading Staff 8/10/2020
Researcher Finds New Office Macro Attacks for MacOS
Curtis Franklin Jr., Senior Editor at Dark Reading,  8/7/2020
Exploiting Google Cloud Platform With Ease
Dark Reading Staff 8/6/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Current Issue
Special Report: Computing's New Normal, a Dark Reading Perspective
This special report examines how IT security organizations have adapted to the "new normal" of computing and what the long-term effects will be. Read it and get a unique set of perspectives on issues ranging from new threats & vulnerabilities as a result of remote working to how enterprise security strategy will be affected long term.
Flash Poll
The Changing Face of Threat Intelligence
The Changing Face of Threat Intelligence
This special report takes a look at how enterprises are using threat intelligence, as well as emerging best practices for integrating threat intel into security operations and incident response. Download it today!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-8720
PUBLISHED: 2020-08-13
Buffer overflow in a subsystem for some Intel(R) Server Boards, Server Systems and Compute Modules before version 1.59 may allow a privileged user to potentially enable denial of service via local access.
CVE-2020-12300
PUBLISHED: 2020-08-13
Uninitialized pointer in BIOS firmware for Intel(R) Server Board Families S2600CW, S2600KP, S2600TP, and S2600WT may allow a privileged user to potentially enable escalation of privilege via local access.
CVE-2020-12301
PUBLISHED: 2020-08-13
Improper initialization in BIOS firmware for Intel(R) Server Board Families S2600ST, S2600BP and S2600WF may allow a privileged user to potentially enable escalation of privilege via local access.
CVE-2020-7307
PUBLISHED: 2020-08-13
Unprotected Storage of Credentials vulnerability in McAfee Data Loss Prevention (DLP) for Mac prior to 11.5.2 allows local users to gain access to the RiskDB username and password via unprotected log files containing plain text credentials.
CVE-2020-8679
PUBLISHED: 2020-08-13
Out-of-bounds write in Kernel Mode Driver for some Intel(R) Graphics Drivers before version 26.20.100.7755 may allow an authenticated user to potentially enable denial of service via local access.