Analytics // Security Monitoring
04:55 PM
Connect Directly

A State of Security Event Overload

As many as 150,000 security events are logged each day in some enterprises, new data shows.

Target isn't the only enterprise getting inundated with security events:  The average enterprise receives more than 10,000 events a day that may or may not be malware-related, and for some of the biggest enterprises, that number jumps to more than 150,000 per day, according to new data from Damballa Labs.

It could happen to anyone, but Target has become a poster child for how easy it is to dismiss the wrong event as a false positive among the heavy volume generated by today's security tools. Target's security team evaluated the "activity" that was flagged and concluded it was not relevant for action. "With the benefit of hindsight, we are investigating whether, if different judgments had been made, the outcome may have been different," a Target spokeswoman said in the aftermath.

Damballa Labs' new data on network events, logged in the first quarter of this year, demonstrates how easy it would be for information overload to complicate the ability to respond to real threats among the benign events.

"There are lots of events each day, and [organizations] can't check on each one" individually, says Brian Foster, CTO at Damballa. "There are not enough smart people to go around. The industry needs to make humans smarter and more efficient, and then they can deal with more events... It eventually leads to automatable defenses."

Foster says the risk of missing a real event among a bunch of false positives is such that some organizations are taking a more holistic approach that looks at risk, prevention and detection, and response. "How many active infections are those alerts resulting in?" he asks, and how much data is going out the door as the attackers steal it?

"Security teams must be able to automate infection 'hunting' and prioritize their response. Otherwise they will find the wolf is already inside their network," Damballa's new Q1 2014 State of Infections Report says.

The full report is available here for download.

Kelly Jackson Higgins is Executive Editor at She is an award-winning veteran technology and business journalist with more than two decades of experience in reporting and editing for various publications, including Network Computing, Secure Enterprise ... View Full Bio

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
6/12/2014 | 10:49:27 AM
Re: Overload indeed.
The issue of training, pay and retention are certainly important.  But that is ultimately just short term supply and demand.  It doesn't really address the larger issue:

How many security staff do you have?  For a typical company...

Rev x % spend on IT x % of that spent on security x % of that spent on security staff x % of those doing actual analysis

For example....

$1B rev company x 5% x 10% x 30% X 25% = $375K

Assuming a loaded labor cost of $200K, that's about two headcount.

Now get these two guys to stay on top of 150,000 alerts a day - and also fix the CEO's expired password?


Based on some research by the U.S. intelligence, the total number of registered hackers in China is approaching 400,000.  Source:  (Ok, it's a number, but even if it's off by an order of magnitude or get the point)

And these guys only need ONE window a jar in your network...

So we are hugely outnumbered.

The only answer is leverage.  And leverage will have to come from machines that can learn to  tie together indicators of compromise (not produce more malware signatures) that increasingly get better at separating signal from noise. Then our two lone staff can be pointed to the right data (not big data, please), analyze it fast for patterns that - if presented visually -  enable humans to see an activity pattern faster than a machine, then teach the machine this new analytic.  Rinse and repeat as in GTD.

Shameless plug - this is what we are working on at
User Rank: Apprentice
5/19/2014 | 7:37:49 AM
Staffing and Training
I see inside a lot of companies.  The number one security issue I see is with staffing.  Companies will spend thousands to millions of dollars on tools that no one knows how to use.  Managers say, 'I don't want to train people because they leave'.  These qualified people leave because they make more elsewhere.  I don't buy the story about there are not enough qualified people.  Unemployment has been high for years and especially for new graduates.  HR needs to work with IT managers to have a plan to immediately give the high acheiving employees raises.  It could be bring in employees as contractors first at a low wage.  If they prove themselves by passing tests and receiving high marks by managers, they are brought on full time with a significant increase in income.  On line training is cheap. 

Companies are receiving too many alerts because no one has the time to take a long look at all the alerts and start filtering the noise.  Where is your critical data?  Know where it is and prioritize your alerts.  Take a day or two and think about what is really at risk in your organization then go protect what is important. 
Robert McDougal
Robert McDougal,
User Rank: Ninja
5/16/2014 | 11:53:22 AM
Overload indeed.
This is a major issue that faces all security professionals.  There is simply too much traffic to be able to evaluate it all.  Therefore, we must rely on signatures to detect security events.  Even that only narrows down the results from billions to hundreds of thousands.  As a result, many of those events are never investigated becauser there simply isn't enough man power to properly investigate.

Also, this doesn't even take into account the security events for which there are no signatures.
Register for Dark Reading Newsletters
White Papers
Current Issue
Five Emerging Security Threats - And What You Can Learn From Them
At Black Hat USA, researchers unveiled some nasty vulnerabilities. Is your organization ready?
Flash Poll
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
Published: 2015-10-15
The Direct Rendering Manager (DRM) subsystem in the Linux kernel through 4.x mishandles requests for Graphics Execution Manager (GEM) objects, which allows context-dependent attackers to cause a denial of service (memory consumption) via an application that processes graphics data, as demonstrated b...

Published: 2015-10-15
netstat in IBM AIX 5.3, 6.1, and 7.1 and VIOS 2.2.x, when a fibre channel adapter is used, allows local users to gain privileges via unspecified vectors.

Published: 2015-10-15
Cross-site request forgery (CSRF) vulnerability in eXtplorer before 2.1.8 allows remote attackers to hijack the authentication of arbitrary users for requests that execute PHP code.

Published: 2015-10-15
Directory traversal vulnerability in QNAP QTS before 4.1.4 build 0910 and 4.2.x before 4.2.0 RC2 build 0910, when AFP is enabled, allows remote attackers to read or write to arbitrary files by leveraging access to an OS X (1) user or (2) guest account.

Published: 2015-10-15
Cisco Application Policy Infrastructure Controller (APIC) 1.1j allows local users to gain privileges via vectors involving addition of an SSH key, aka Bug ID CSCuw46076.

Dark Reading Radio
Archived Dark Reading Radio
According to industry estimates, about a million new IT security jobs will be created in the next two years but there aren't enough skilled professionals to fill them. On top of that, there isn't necessarily a clear path to a career in security. Dark Reading Executive Editor Kelly Jackson Higgins hosts guests Carson Sweet, co-founder and CTO of CloudPassage, which published a shocking study of the security gap in top US undergrad computer science programs, and Rodney Petersen, head of NIST's new National Initiative for Cybersecurity Education.