Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Analytics //

Security Monitoring

10/10/2013
06:54 PM
50%
50%

Big Data Detectives

Could big data be the key to identifying sophisticated threats? Security experts are on the case.

For Vigilant, it started in 2009. And as with most companies, it started small.

The security services startup, now part of audit and consulting firm Deloitte, wanted a way to bring information about external threats to clients that were using SIEM (security information and event management) systems to monitor their own environments. The Vigilant team knew that the combination of external threat data with internal security event data could be a powerful way to improve enterprise defenses, but crunching all that data would be a monumental task.

Vigilant began combining threat intelligence feeds, filtering the data to pull out the most important information for each client, and then transmitting the data to their clients' SIEM systems. The company started with two threat lists: domains serving malware, and domains compromised by the Trojans SpyEye and Zeus. To reduce false alarms and aid in analysis, the company began adding more data feeds.

Vigilant's analysts quickly became addicted to the analysis. Each new source of data gave them the ability to tease out additional information on threats. By 2011, the company was processing about 50 to 100 GBs per day. But the company's systems couldn't keep up with the flow of data, and it started missing performance deadlines, says Joe Magee, co-founder and former CTO of Vigilant, who is now a director at Deloitte.

"We were not able to catch up," Magee says. "We were not able to process the information and push it out fast enough, and that's when it became a big data issue for us. We needed to be able to rip through this data in Google-like fashion."

The volume of data and rate of change caused the problem, because most of the data came in the form of feeds updated daily with gigabytes of data. It overwhelmed the company's initial database built on top of Postgres. In 2011, Vigilant moved to Hadoop and became one of many companies -- both vendors and enterprises -- that are advocating the use of big data analytics to improve the response to security threats.

Big Data Still Just A Promise

For security teams, the use of analytics on massive quantities of security data -- from device and application logs to collections of captured network packets and operational business data -- promises better visibility into the security threats that elude current defenses.

Big data analytics can be more complex than the log collection and analysis conducted by most SIEM systems, so automating the number crunching is often needed to let security pros more easily use statistical correlations to discover trends and anomalies. Tracking days or weeks of business activity allows the system to find outliers -- a user who accesses far more data on a daily basis than the average employee, or a system that has a sudden spike in resource consumption. Analysts then can dig deeper into the large data sets of security information for any flagged events.

"Big data is not just about gaining insights, it's about helping remediate issues faster," says Jason Corbin, director of security intelligence strategy for IBM Security Systems. "The big problem is that [security teams] are overwhelmed with information they have. All that information goes to some guy who has to sift through tons of incidents or vulnerability reports and decide what they need to patch or virtually patch or fix. Security teams fall behind, and that's how companies suffer breaches based on known but unpatched vulnerabilities."

But for many companies, the promise of big data in security is just that -- a promise. While security teams hope to gain more awareness of what is going on in their networks by collecting and analyzing more of their data, the technology is still in its adolescence. "Hadoop has been around for a while, but it is still figuring out what it is and what is wants to be," says Adrian Lane, CTO for security consultancy Securosis.

Still, the potential is huge, Lane adds. Companies that kick off a big data project for security can collect an immense volume of data and have a security analyst poke through the information, ask queries of the data and make important discoveries.

chart: Which of these big data tools are in use at your company

Veteran technology journalist of more than 20 years. Former research engineer. Written for more than two dozen publications, including CNET News.com, Dark Reading, MIT's Technology Review, Popular Science, and Wired News. Five awards for journalism, including Best Deadline ... View Full Bio

Previous
1 of 3
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Chuck Brooks
50%
50%
Chuck Brooks,
User Rank: Apprentice
10/19/2013 | 6:20:48 PM
re: Big Data Detectives
Situational awareness on both internal and external threats can be enhanced via data analytics. False alarms a major issue and the predictive and forensic aspects of data analysis is an imperative. Vigilant is a good acquisition for Deloitte.
AnonymousMan
50%
50%
AnonymousMan,
User Rank: Moderator
10/14/2013 | 10:01:35 PM
re: Big Data Detectives
Uhg, another Big Data article. "APT" must be running out of steam. I'd like to see some discussion of the NEW analytics being performed. Don't just backend your [SIEM] solution with Hadoop and call it a "Big Data" solution. This re-branding has been going on for as long as I've been in the security industry. This industry makes me want to move into the woods and make moonshine for a living.
Mobile Banking Malware Up 50% in First Half of 2019
Kelly Sheridan, Staff Editor, Dark Reading,  1/17/2020
Exploits Released for As-Yet Unpatched Critical Citrix Flaw
Jai Vijayan, Contributing Writer,  1/13/2020
Microsoft to Officially End Support for Windows 7, Server 2008
Kelly Sheridan, Staff Editor, Dark Reading,  1/13/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment: This comment is waiting for review by our moderators.
Current Issue
The Year in Security: 2019
This Tech Digest provides a wrap up and overview of the year's top cybersecurity news stories. It was a year of new twists on old threats, with fears of another WannaCry-type worm and of a possible botnet army of Wi-Fi routers. But 2019 also underscored the risk of firmware and trusted security tools harboring dangerous holes that cybercriminals and nation-state hackers could readily abuse. Read more.
Flash Poll
[Just Released] How Enterprises are Attacking the Cybersecurity Problem
[Just Released] How Enterprises are Attacking the Cybersecurity Problem
Organizations have invested in a sweeping array of security technologies to address challenges associated with the growing number of cybersecurity attacks. However, the complexity involved in managing these technologies is emerging as a major problem. Read this report to find out what your peers biggest security challenges are and the technologies they are using to address them.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-7227
PUBLISHED: 2020-01-18
Westermo MRD-315 1.7.3 and 1.7.4 devices have an information disclosure vulnerability that allows an authenticated remote attacker to retrieve the source code of different functions of the web application via requests that lack certain mandatory parameters. This affects ifaces-diag.asp, system.asp, ...
CVE-2019-15625
PUBLISHED: 2020-01-18
A memory usage vulnerability exists in Trend Micro Password Manager 3.8 that could allow an attacker with access and permissions to the victim's memory processes to extract sensitive information.
CVE-2019-19696
PUBLISHED: 2020-01-18
A RootCA vulnerability found in Trend Micro Password Manager for Windows and macOS exists where the localhost.key of RootCA.crt might be improperly accessed by an unauthorized party and could be used to create malicious self-signed SSL certificates, allowing an attacker to misdirect a user to phishi...
CVE-2019-19697
PUBLISHED: 2020-01-18
An arbitrary code execution vulnerability exists in the Trend Micro Security 2019 (v15) consumer family of products which could allow an attacker to gain elevated privileges and tamper with protected services by disabling or otherwise preventing them to start. An attacker must already have administr...
CVE-2019-20357
PUBLISHED: 2020-01-18
A Persistent Arbitrary Code Execution vulnerability exists in the Trend Micro Security 2020 (v160 and 2019 (v15) consumer familiy of products which could potentially allow an attacker the ability to create a malicious program to escalate privileges and attain persistence on a vulnerable system.