Threat Intelligence
3/15/2016
04:00 PM
Connect Directly
Google+
Twitter
RSS
E-Mail
50%
50%

Threat Intelligence's Big Data Problem

Security teams are drowning in often useless threat intel data, but signs of maturity are emerging in what IT-Harvest predicts will be a $1.5 billion market by 2018.

First in a series on the evolution of threat intelligence

Something’s gotta give: nearly three-fourths of enterprises today say they ignore security events because they’re overwhelmed by the deluge of alerts. And that doesn’t even take into account the firehose of threat intelligence data they’re funneling today, a new report shows.

Mega-retailer Target was the poster child for security alert awareness gone bad—the needle in the haystack Target dismissed was actually the clue that it was under a major attack in the fall of 2013. Nearly three years after that epic data breach, security events, alerts, and threat intelligence feeds are exploding in many enterprises hungry for hints that they are in the bullseye. The tradeoff is that this deluge of data is drowning security teams who must sift, separate, and correlate the real threats from the false positives or irrelevant information.

Security event overload alone is causing some dramatic fallout: more than half of all security events get ignored by IT security pros due to the overload of information, according to a new Enterprise Strategy Group (ESG) report that surveyed 125 IT security pros on the state of incident response in their organizations. Around 30% of those organizations say they also have some 11 different threat intelligence feeds flowing in as well, the Phantom-commissioned report—published today--found.

Threat intelligence data is all about helping enterprises block or protect against the newest threats by providing in-the-wild attack and threat artifacts and intel that companies can compare and correlate with their security. But for many organizations, the deluge of this type of information isn’t much help if they can’t triage and apply it effectively. 

The threat intelligence market itself is booming, growing at a rapid clip at 84% annually, according to new data published today by IT-Harvest. The threat intel market—which was at $251 million in 2015—is expected to reach more than $460 million this year, says Richard Stiennon, chief research analyst for IT-Harvest.

Threat intelligence platform products such as those of ThreatConnect, ThreatStream (now Anomali), ThreatQuotient, and BrightPoint Security, made up $61 million of 2015’s total threat intel market revenues, according to IT-Harvest. The market is on track to hit $1.5 billion in 2018 at the current rate of growth, according to the report, which includes a look at more than 20 threat intelligence vendors, including FireEye’s iSIGHT Partners, Cyveillance+LookingGlass, Digital Shadows, and Flashpoint Intel.

“I expect a lot of churn and also a lot of startups,” Stiennon says of the threat intelligence space.

Signs of churn started to show in the past month, with Norse Corp.’s mass layoffs and executive shakeout. Security experts attributed Norse’s plight more to its own internal managerial problems and lack of a solid product as well some weak analysis reports, rather than as a bellwether of the threat intel space.

‘Threat’ Rebrand

Meanwhile, recent moves by other threat intel vendors show signs of a logical evolution of making threat intel more useful and manageable.

Late last month, ThreatStream dropped the “threat” moniker and rebranded itself as Anomali, now focusing on not just delivering threat intel, but also prioritizing and matching it for individual organizations. Threat intel has its own big data problem, according to executives at Anomali, which now is filtering down indicators of compromise (IOCs) and other threat intel for security event and information management (SIEM) systems, which it says weren’t built to process millions of IOCs.

“When we started [out], the volume of threat intelligence coming from feed vendors and open communities versus now was more manageable. There were hundreds of thousands of indicators of compromise, and now there are tens of millions,” says Hugh Njemanze, CEO of Anomali. “We expect this year to [reach] 100 million IoCs. There’s been an explosion.”

That kind of threat intel volume isn’t conducive for most in-house SIEM tools today. “Even the most robust SIEM is not able to ingest more than 1 million IOCs,” he says. Anomali’s new cloud-based products basically match event flows with IOCs, for example, and then feed contextual information about the incident to the SIEM.

“We’re taking on the burden of discovery and matching and letting the SIEM do what it’s good at: analyzing the millions of events they are collecting,” Njemanze explains. Security operations center teams need to know which IOCs are relevant, so that’s what Anomali is offering.

Anomali still offers ThreatStream Optic, its threat intel feed, in addition to its new Harmony Breach Analytics and Anomali Reports products. “We still see ourselves as a threat intelligence player, but we’re radically shifting how threat intel can be operationalized,” he says.

“I’m convinced TI platforms like ThreatStream’s [Anomali’s] have an opportunity. I haven’t seen anyone targeting dealing with the data. Building a distiller takes the good stuff out, and turns the SIEM into a log manager,” IT-Harvest’s Stiennan says.

ThreatConnect, meanwhile, has upgraded its ThreatConnect platform to better integrate a company’s security incidents with threat intelligence. “The goal of my platform is to bring the two together: every data set and correlate it with events and incidents that are unfolding so human beings don’t have to look at the noise. Instead, the most important things bubble up to the top, based on the underlying analytics,” says Adam Vincent, CEO of ThreatConnect.

ThreatConnect has partnered with Splunk, Palo Alto Networks, and others, to integrate threat intel with an organization’s incident detection and response processes. Version 4.0 of the ThreatConnect platform also lets companies customize reports for all levels of users, including C-level executives who want to see a map of which regions are targeting their company, for example, Vincent says.

Threat intelligence is about empowering decision-making, he says. “It’s not the end goal in itself.”

So rather than a retailer looking at 100 events in the order in which they occur, the threat intel platform would flag and prioritize events that appear to be connected or related to other attacks in the wild. “It would say this event is important because it looks coordinated, and it’s against equipment that has known vulnerabilities,” Vincent says. “And it looks at what type of techniques and tradecraft the [attacker] is using ... As the [company] investigates it, they are collecting additional information that is going to inform their decision-making.”

Most security vendors now offer some level of threat intelligence, and there are several open-source threat intel feeds as well. “The challenge right now is to tell high-quality threat intelligence from low-quality threat intelligence. It’s tough to distinguish, given the abundancy of options” out there, says Oliver Friedrichs, founder and CEO of startup Phantom.

“One of the biggest challenges is how to reconcile all the various feeds and how to actually make sense of them. The threat intelligence platform space is really striving to solve that,” says Friedrichs, whose firm offers an automation and “orchestration engine” for an organization’s security tools.

 

Related Content:

 

Interop 2016 Las VegasFind out more about security threats at Interop 2016, May 2-6, at the Mandalay Bay Convention Center, Las Vegas. Register today and receive an early bird discount of $200.

Kelly Jackson Higgins is Executive Editor at DarkReading.com. She is an award-winning veteran technology and business journalist with more than two decades of experience in reporting and editing for various publications, including Network Computing, Secure Enterprise ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
ACThomson
50%
50%
ACThomson,
User Rank: Author
3/25/2016 | 9:20:15 AM
An opportunity for artificial intelligence
There are quite a number of startups appearing that are using artificial intelligence techniques to help manage this overload, allowing the most important threats to be prioratised.
ecrutchlow
50%
50%
ecrutchlow,
User Rank: Apprentice
3/21/2016 | 12:38:59 PM
Target Example Completely Wrong
Your use of Target was completely wrong. This was Target's first excuse for why they didn't catch this early, but as the story evolved the true events came out resulting in the CIO and the CEO resigning.

The FireEye device did detect and alert the subcontractor team in India and FireEye saw the alert as well. The subcontractor team was contacted immediately by FireEye and informed of the significance of the breach. In turn, the subcontractor notified Target and they did not act on the information. For a good write-up on the events, please look up the SANS report titled, "Case Study: Critical Controls that Could Have Prevented Target Breach"

The article is correct regarding the amount of data organizations must review. It's really the story of the boy who cried wolf one too many times. Except we have systems that cry wolf thousands of times a day. Solutions that can reduce the noise is essential. Security operations centers should be focusing on real events and not on vetting false-positives.
MattDevost
50%
50%
MattDevost,
User Rank: Apprentice
3/17/2016 | 4:27:59 PM
An evolution, hopefully based on the fundamentals of intelligence
Look forward to tracking this new series.  The evolution of threat intelligence is an important topic as organizations migrate towards intelligence driven security programs.  Hopefully you spend a little time looking at the fundamentals of the intelligence process and how they need to be incorporated and evolved into the cyber security domain.  

Having built intel organizations at companies like iDEFENSE, iSIGHT, the Terrorism Research Center, and within 56 U.S. cities, this is a topic that I track closely.  Many organizations don't give enough attention to how they need to consume and act on intelligence to drive decisions.  It isn't just about how many feeds you can consume, but how those feeds fit into and drive an internal intelligence process that is iterative and has a robust feedback loop.
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment: This comment is waiting for review by our moderators.
Current Issue
Security Operations and IT Operations: Finding the Path to Collaboration
A wide gulf has emerged between SOC and NOC teams that's keeping both of them from assuring the confidentiality, integrity, and availability of IT systems. Here's how experts think it should be bridged.
Flash Poll
New Best Practices for Secure App Development
New Best Practices for Secure App Development
The transition from DevOps to SecDevOps is combining with the move toward cloud computing to create new challenges - and new opportunities - for the information security team. Download this report, to learn about the new best practices for secure application development.
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2017-0290
Published: 2017-05-09
NScript in mpengine in Microsoft Malware Protection Engine with Engine Version before 1.1.13704.0, as used in Windows Defender and other products, allows remote attackers to execute arbitrary code or cause a denial of service (type confusion and application crash) via crafted JavaScript code within ...

CVE-2016-10369
Published: 2017-05-08
unixsocket.c in lxterminal through 0.3.0 insecurely uses /tmp for a socket file, allowing a local user to cause a denial of service (preventing terminal launch), or possibly have other impact (bypassing terminal access control).

CVE-2016-8202
Published: 2017-05-08
A privilege escalation vulnerability in Brocade Fibre Channel SAN products running Brocade Fabric OS (FOS) releases earlier than v7.4.1d and v8.0.1b could allow an authenticated attacker to elevate the privileges of user accounts accessing the system via command line interface. With affected version...

CVE-2016-8209
Published: 2017-05-08
Improper checks for unusual or exceptional conditions in Brocade NetIron 05.8.00 and later releases up to and including 06.1.00, when the Management Module is continuously scanned on port 22, may allow attackers to cause a denial of service (crash and reload) of the management module.

CVE-2017-0890
Published: 2017-05-08
Nextcloud Server before 11.0.3 is vulnerable to an inadequate escaping leading to a XSS vulnerability in the search module. To be exploitable a user has to write or paste malicious content into the search dialogue.

Dark Reading Radio
Archived Dark Reading Radio
In past years, security researchers have discovered ways to hack cars, medical devices, automated teller machines, and many other targets. Dark Reading Executive Editor Kelly Jackson Higgins hosts researcher Samy Kamkar and Levi Gundert, vice president of threat intelligence at Recorded Future, to discuss some of 2016's most unusual and creative hacks by white hats, and what these new vulnerabilities might mean for the coming year.