Analytics // Security Monitoring
10/10/2013
06:54 PM
50%
50%

Big Data Detectives

Could big data be the key to identifying sophisticated threats? Security experts are on the case.

For Vigilant, it started in 2009. And as with most companies, it started small.

The security services startup, now part of audit and consulting firm Deloitte, wanted a way to bring information about external threats to clients that were using SIEM (security information and event management) systems to monitor their own environments. The Vigilant team knew that the combination of external threat data with internal security event data could be a powerful way to improve enterprise defenses, but crunching all that data would be a monumental task.

Vigilant began combining threat intelligence feeds, filtering the data to pull out the most important information for each client, and then transmitting the data to their clients' SIEM systems. The company started with two threat lists: domains serving malware, and domains compromised by the Trojans SpyEye and Zeus. To reduce false alarms and aid in analysis, the company began adding more data feeds.

Vigilant's analysts quickly became addicted to the analysis. Each new source of data gave them the ability to tease out additional information on threats. By 2011, the company was processing about 50 to 100 GBs per day. But the company's systems couldn't keep up with the flow of data, and it started missing performance deadlines, says Joe Magee, co-founder and former CTO of Vigilant, who is now a director at Deloitte.

"We were not able to catch up," Magee says. "We were not able to process the information and push it out fast enough, and that's when it became a big data issue for us. We needed to be able to rip through this data in Google-like fashion."

The volume of data and rate of change caused the problem, because most of the data came in the form of feeds updated daily with gigabytes of data. It overwhelmed the company's initial database built on top of Postgres. In 2011, Vigilant moved to Hadoop and became one of many companies -- both vendors and enterprises -- that are advocating the use of big data analytics to improve the response to security threats.

Big Data Still Just A Promise

For security teams, the use of analytics on massive quantities of security data -- from device and application logs to collections of captured network packets and operational business data -- promises better visibility into the security threats that elude current defenses.

Big data analytics can be more complex than the log collection and analysis conducted by most SIEM systems, so automating the number crunching is often needed to let security pros more easily use statistical correlations to discover trends and anomalies. Tracking days or weeks of business activity allows the system to find outliers -- a user who accesses far more data on a daily basis than the average employee, or a system that has a sudden spike in resource consumption. Analysts then can dig deeper into the large data sets of security information for any flagged events.

"Big data is not just about gaining insights, it's about helping remediate issues faster," says Jason Corbin, director of security intelligence strategy for IBM Security Systems. "The big problem is that [security teams] are overwhelmed with information they have. All that information goes to some guy who has to sift through tons of incidents or vulnerability reports and decide what they need to patch or virtually patch or fix. Security teams fall behind, and that's how companies suffer breaches based on known but unpatched vulnerabilities."

But for many companies, the promise of big data in security is just that -- a promise. While security teams hope to gain more awareness of what is going on in their networks by collecting and analyzing more of their data, the technology is still in its adolescence. "Hadoop has been around for a while, but it is still figuring out what it is and what is wants to be," says Adrian Lane, CTO for security consultancy Securosis.

Still, the potential is huge, Lane adds. Companies that kick off a big data project for security can collect an immense volume of data and have a security analyst poke through the information, ask queries of the data and make important discoveries.

chart: Which of these big data tools are in use at your company

Robert Lemos is a veteran technology journalist of more than 16 years and a former research engineer, writing articles that have appeared in Business Week, CIO Magazine, CNET News.com, Computing Japan, CSO Magazine, Dark Reading, eWEEK, InfoWorld, MIT's Technology Review, ... View Full Bio

Previous
1 of 3
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Chuck Brooks
50%
50%
Chuck Brooks,
User Rank: Apprentice
10/19/2013 | 6:20:48 PM
re: Big Data Detectives
Situational awareness on both internal and external threats can be enhanced via data analytics. False alarms a major issue and the predictive and forensic aspects of data analysis is an imperative. Vigilant is a good acquisition for Deloitte.
AnonymousMan
50%
50%
AnonymousMan,
User Rank: Moderator
10/14/2013 | 10:01:35 PM
re: Big Data Detectives
Uhg, another Big Data article. "APT" must be running out of steam. I'd like to see some discussion of the NEW analytics being performed. Don't just backend your [SIEM] solution with Hadoop and call it a "Big Data" solution. This re-branding has been going on for as long as I've been in the security industry. This industry makes me want to move into the woods and make moonshine for a living.
Register for Dark Reading Newsletters
White Papers
Cartoon
Current Issue
Dark Reading, January 2015
To find and fix exploits aimed directly at your business, stop waiting for alerts and become a proactive hunter.
Flash Poll
Video
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2013-7402
Published: 2014-12-17
Multiple unspecified vulnerabilities in request.c in c-icap 0.2.x allow remote attackers to cause a denial of service (crash) via a crafted ICAP request.

CVE-2014-5437
Published: 2014-12-17
Multiple cross-site request forgery (CSRF) vulnerabilities in ARRIS Touchstone TG862G/CT Telephony Gateway with firmware 7.6.59S.CT and earlier allow remote attackers to hijack the authentication of administrators for requests that (1) enable remote management via a request to remote_management.php,...

CVE-2014-5438
Published: 2014-12-17
Cross-site scripting (XSS) vulnerability in ARRIS Touchstone TG862G/CT Telephony Gateway with firmware 7.6.59S.CT and earlier allows remote authenticated users to inject arbitrary web script or HTML via the computer_name parameter to connected_devices_computers_edit.php.

CVE-2014-7170
Published: 2014-12-17
Race condition in Puppet Server 0.2.0 allows local users to obtain sensitive information by accessing it in between package installation or upgrade and the start of the service.

CVE-2014-7285
Published: 2014-12-17
The management console on the Symantec Web Gateway (SWG) appliance before 5.2.2 allows remote authenticated users to execute arbitrary OS commands by injecting command strings into unspecified PHP scripts.

Best of the Web
Dark Reading Radio
Archived Dark Reading Radio
Join us Wednesday, Dec. 17 at 1 p.m. Eastern Time to hear what employers are really looking for in a chief information security officer -- it may not be what you think.