Analytics
4/22/2014
12:00 PM
Joshua Goldfarb
Joshua Goldfarb
Commentary
Connect Directly
Twitter
RSS
E-Mail
50%
50%

7 Tips To Improve 'Signal-to-Noise' In The SOC

When security analysts are desensitized to alerts because of sheer volume, they miss the true positives that can prevent a large-scale data breach. Here's how to up your game.

Reports on the recent Target and Neiman Marcus breaches have indicated that, in both cases, there were numerous alerts fired as a result of the intrusion activity. Yet, according to news accounts, the alerts were not properly handled, allowing system compromises to go undetected and giving attackers the crucial footholds needed to pull off large-scale breaches.

Though the exact reasons for these oversights remain unclear, it is likely that security operation centers (SOCs) responsible for monitoring retailers' networks had low "signal-to-noise ratios." By that, I mean that analysts in the SOC were adept at collecting vast amounts of security information, but they faced challenges in discerning the most severe, imminent threats -- and responding to them in an effective, timely manner.

A low signal-to-noise ratio means that, more than likely, these SOCs had daily work queues inundated with false positives, to the point where true positives were lost in the clutter of noise. Commonly in these scenarios, analysts attempt to review only the alerts of the highest priority. But because of the large volume of even the highest priority-flagged alerts, analysts are not able to successfully review all of them. When analysts become desensitized to alerts because of the sheer volume of false positives, they tend to miss the true positives.

What surprises me is not that organizations have a low signal-to-noise ratio, but rather, the fundamental assumption that it has to be this way. I know from experience that there is a better way. Here are seven tips that have worked well for me throughout my career:

Tip No. 1: Go for the "Money Shot." It's possible to detect attacks at multiple stages (e.g., exploit, payload delivery, command and control, etc.) using indicators of compromise (IoCs). Different stages of attack will have different false positive ratios, and a given attack may have multiple IoCs that can be used to identify it. Whenever possible, the most reliable IoCs with the lowest false positive rates should be used to benefit the signal-to-noise ratio.

Tip No. 2: Take a scalpel approach. Alerting technologies have tremendous potential to identify suspicious or malicious activity when used like a scalpel. Unfortunately, many organizations use them less like a scalpel and more like a hatchet. It pays to assess risk, security needs, operational needs, and business needs at each deployment location and focus alerting technologies selectively. Singling out only the most relevant alerts lowers the number of false positives and improves the signal-to-noise ratio.

Tip No. 3: Use correlation. Sometimes, an individual alert is not particularly interesting until observed in conjunction with one or more other alerts or activity of interest. In those cases, the alert should be sent to the work queue only when all the correlation criteria are met. This increases the fidelity of the alerts and reduces the number of false positives, both of which help the signal-to-noise ratio.

Tip No. 4: Write intelligent alerting. As the saying goes, "Ask a stupid question, get a stupid answer." Today's threats are sophisticated and require intelligent, targeted, incisive alert logic to extract activity of concern while minimizing false positives. In other words, today's threats require intelligent alert logic. Working to tighten this logic goes a long way toward an optimal signal-to-noise ratio.

Tip No. 5: Be picky with intelligence. Different intelligence sources will bring different fidelity, relevance, and value to security operations. Blindly integrating intelligence feeds without evaluating their fidelity and false positive rates can have a detrimental effect on security operations by significantly lowering the signal-to-noise ratio.

Tip No. 6: Prioritize appropriately. Prioritization is one of the greatest tools a security team can utilize. Alerts that are of higher fidelity and detect higher-risk activity can be assigned a higher priority, while alerts that are of lower fidelity and detect lower-risk activity can be assigned a lower priority. Analysts work the queue from highest priority to lowest priority, ensuring that the most reliable alerts covering the activity of the greatest risk are addressed first.

Tip No. 7: Review every alert. Why fill the queue with alerts that are never reviewed? The whole point is to draw the attention of an analyst for review. Using the techniques I've described, an organization can regain control of its queue with the ultimate goal that every alert will be reviewed. This ensures that nothing is overlooked.

(Source: Wikipedia)
(Source: Wikipedia)

To illustrate these concepts, consider the case of SOC A and SOC B. In SOC A, the daily work queue contains approximately 100 reliable, high-fidelity, usable alerts. Each one is reviewed by an analyst. If incident response is necessary for a given alert, it is performed.

In SOC B, the daily work queue contains approximately 100,000 alerts, almost all of which are false positives. Analysts attempt to review them according to priority. Because of the large volume (even for alerts of the highest priority), analysts cannot successfully review all of the highest-priority alerts. Additionally, because of the large number of false positives, SOC B's analysts become desensitized to alerts and do not take them particularly seriously.

One day, 10 additional alerts relating to payment-card stealing malware fire within a few minutes of one another.

In SOC A, where every alert is reviewed by an analyst, where the signal-to-noise ratio is high, and where 10 additional alerts seem like a lot, analysts successfully identify the breach in less than 24 hours. SOC A's team can perform analysis, containment, and remediation within the first 24 hours of the breach. The team can stop the bleeding before any payment card data is exfiltrated. Though there has been some damage, it can be controlled. The organization can assess the damage, respond appropriately, and return to normal business operations.

In SOC B, where an extremely small percentage of the alerts are reviewed by an analyst, where the signal-to-noise ratio is low, and where 10 additional alerts doesn't even raise an eyebrow, the breach remains undetected. Months later, SOC B will learn of the breach from a third party. The damage will be extensive, and full recovery will take months or years.

Which SOC would you rather have inside your organization?

Joshua Goldfarb is an experienced cyber security analyst with over a decade of experience building, operating, and running security operations centers (SOCs). Before joining nPulse Technologies as its Chief Security Officer, he worked as an independent consultant, applying ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
jg@npulsetech.com
50%
50%
jg@npulsetech.com,
User Rank: Apprentice
4/23/2014 | 9:58:54 AM
Re: Lowering the noise level -- gut feelings
Another excellent question Marilyn, thank you.  Gut feelings, or put another way, intuition, can certainly catalyze and inspire investigation and analysis.  Not every feeling or intuition is going to lead in a productive direction of course, and only experience can really help determine what may be a promising intuition vs. what may not be a good use of resources.  For example, it is reasonable to say "I don't think we have any legitimate business traffic to any .ce.ms domains."  Following this hypothesis, the organization should seek the ground truth that is contained in the data.  In my experience, what creates a dangerous feeling or intuition is when decisions are based solely on that feeling or intuition.  If decisions are made based upon investigation and analysis of the data catalyzed by those initial feelings or intuitions, then that can be a good thing in my opinion.
Marilyn Cohodas
50%
50%
Marilyn Cohodas,
User Rank: Strategist
4/23/2014 | 9:45:40 AM
Re: Lowering the noise level -- gut feelings
Phrases like "this feels like it's less actionable" or "I think this is always a false positive" can be dangerous.

How do you quantify those "gut feelings." And is there no place for them in the SOC?
jg@npulsetech.com
50%
50%
jg@npulsetech.com,
User Rank: Apprentice
4/22/2014 | 2:07:52 PM
Re: Lowering the noise level
Excellent question, Marilyn, and thank you.  In my experience, organizations too often take an "all or nothing" approach to managing the noise level.  For example, an organization may say "we get too many alerts from source X, so we are just going to ignore or turn off all alerts from source X."  I think a better approach would be "let's think about what business, operational, and security needs we can address through alerts from source X and tune source X delicately to address those needs."  Additionally, organizations will sometimes emote or intuit noise reduction.  Phrases like "this feels like it's less actionable" or "I think this is always a false positive" can be dangerous.  The best source for educated decisions is the data.  The data do not lie and form the basis for good security decision making (and the tips provided in the article of course).
Marilyn Cohodas
50%
50%
Marilyn Cohodas,
User Rank: Strategist
4/22/2014 | 12:43:17 PM
Lowering the noise level
Thanks for sharing you wisdom with the Dark Reading community Joshua. In your experience, what do you think are the biggest mistakes SOCs make in loweirng the noise level?
Register for Dark Reading Newsletters
Partner Perspectives
What's This?
In a digital world inundated with advanced security threats, Intel Security seeks to transform how we live and work to keep our information secure. Through hardware and software development, Intel Security delivers robust solutions that integrate security into every layer of every digital device. In combining the security expertise of McAfee with the innovation, performance, and trust of Intel, this vision becomes a reality.

As we rely on technology to enhance our everyday and business life, we must too consider the security of the intellectual property and confidential data that is housed on these devices. As we increase the number of devices we use, we increase the number of gateways and opportunity for security threats. Intel Security takes the “security connected” approach to ensure that every device is secure, and that all security solutions are seamlessly integrated.
Featured Writers
White Papers
Cartoon
Current Issue
Dark Reading's October Tech Digest
Fast data analysis can stymie attacks and strengthen enterprise security. Does your team have the data smarts?
Flash Poll
Threat Intel Today
Threat Intel Today
The 397 respondents to our new survey buy into using intel to stay ahead of attackers: 85% say threat intelligence plays some role in their IT security strategies, and many of them subscribe to two or more third-party feeds; 10% leverage five or more.
Video
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2013-4594
Published: 2014-10-25
The Payment for Webform module 7.x-1.x before 7.x-1.5 for Drupal does not restrict access by anonymous users, which allows remote anonymous users to use the payment of other anonymous users when submitting a form that requires payment.

CVE-2014-0476
Published: 2014-10-25
The slapper function in chkrootkit before 0.50 does not properly quote file paths, which allows local users to execute arbitrary code via a Trojan horse executable. NOTE: this is only a vulnerability when /tmp is not mounted with the noexec option.

CVE-2014-1927
Published: 2014-10-25
The shell_quote function in python-gnupg 0.3.5 does not properly quote strings, which allows context-dependent attackers to execute arbitrary code via shell metacharacters in unspecified vectors, as demonstrated using "$(" command-substitution sequences, a different vulnerability than CVE-2014-1928....

CVE-2014-1928
Published: 2014-10-25
The shell_quote function in python-gnupg 0.3.5 does not properly escape characters, which allows context-dependent attackers to execute arbitrary code via shell metacharacters in unspecified vectors, as demonstrated using "\" (backslash) characters to form multi-command sequences, a different vulner...

CVE-2014-1929
Published: 2014-10-25
python-gnupg 0.3.5 and 0.3.6 allows context-dependent attackers to have an unspecified impact via vectors related to "option injection through positional arguments." NOTE: this vulnerability exists because of an incomplete fix for CVE-2013-7323.

Best of the Web
Dark Reading Radio
Archived Dark Reading Radio
Follow Dark Reading editors into the field as they talk with noted experts from the security world.