Analytics
8/9/2016
11:15 AM
Giora Engel
Giora Engel
Commentary
Connect Directly
Facebook
Twitter
LinkedIn
Google+
RSS
E-Mail vvv
100%
0%

Building A Detection Strategy With The Right Metrics

The tools used in detecting intrusions can lead to an overwhelming number of alerts, but they're a vital part of security.

Traditional security used to be focused exclusively on prevention and policy enforcement. While these mechanisms are still important, there is increasing attention paid to detection in order to be able to more rapidly spot potentially malicious activities that have circumvented preventative security.

The leading security teams are shifting their mission to detect a potential attack and respond appropriately and quickly. To achieve such a mission, organizations typically implement the following detection (or incident response) process:

  1. Start with an alert.
  2. Investigate the alert and determine if it is benign or unwanted.
  3. Make a proper decision and remediate or respond.

This detection process can be implemented at different levels of detail in various security organizations. I have seen implementations conducted by a single-person security team all the way to large security operations centers.

Of course, an effective detection strategy isn’t easy to achieve with a limited security budget and constrained staff. A major factor determining the effectiveness of the detection process is that security teams have to deal with the massive inefficiency and imprecision of security tools. For most organizations, the total alert volume is far beyond what any group can handle. Most alerts aren’t useful because either they don’t highlight anything of importance or are nonspecific or unsubstantiated, making an already impossible situation even worse. This situation is created by three main factors.

First, most tools are still primarily malware-oriented and based on technical artifacts, such as a signature, hash, domain, or a predefined list of software behaviors. This means that alerts will be firing all the time.

Second, many tools are designed to detect isolated “trees” and don’t understand that they are part of an overall “forest” of an orchestrated campaign being run by an attacker. It’s common to see a singular event — let’s say a scan inside the network — set off a hundreds or even thousands of individual alerts.

Third, many traditional tools are based on a do-it-yourself mentality. They’re created assuming a completely manual style of operational security. Part of this recognizes that most tools are rather low in fidelity and not intended to offload the work of a security professional by conducting some level of investigation or making a judgment call.

If organizations are going to be successful with detection, they need to measure and improve their efficiency. The following questions can help develop an effective detection strategy, or evaluate a specific new or existing detection tool:

  1. Detection coverage: If an active attacker were operating inside my network, would my systems see an operational activity and set off an alert?
  2. Detection quantity: Can my team investigate all the relevant alerts? Is it clear which are relevant?
  3. Detection quality: When an alert is investigated, can I reach a conclusion?

Detection coverage is determined by what is being sought — is a tool looking for malware? Reconnaissance? Lateral movement? The best approach here is to define a few use cases that you’d like to detect and make sure that they’re covered. Most data breaches are ultimately the result of an attacker that operates inside the internal network. The use cases can either be defined theoretically or tested live as an attack simulation or penetration test.

Detection quality and quantity can be measured in a very concrete way using standardized metrics:

  1. Detection quantity: efficiency — Count the total number of alerts per day and calculate how many alerts you have per 1,000 hosts.
  2. Detection quality: accuracy — Percentage of alerts handled by the team that are useful (not whitelisted or ignored) over the total number of alerts.

A Ponemon Institute study of 630 enterprises showed that the average number of alerts a security team receives per week from their security tools — firewalls, intrusion-detection systems, intrusion-prevention systems, sandboxing systems, security information and event management (SIEM) systems, etc. — was a shocking 172 per 1,000 hosts per day (2,420 alerts for an average 14,000 endpoints in a network). Where does one start with that volume of alerts, particularly given that the large majority of these are false-positives? The same study shows that, on average, two-thirds of the security staff’s time is wasted because their tools are grossly inefficient, so only 4% of alerts are typically investigated, leaving the possibility that somewhere around 96% of those untouched may be something important.

This is clearly an inefficient, inaccurate process because there is no good way to guarantee a focus on the right alerts. There are examples where signs of an attacker were hidden in a flood of alerts. In the Target breach, according to reported details, signals were detected by existing systems but buried under thousands of similar alerts without any way to know that this particular alert was important.

It’s not enough to have a haystack with some needles that can only be discovered with hindsight after it’s too late; we need a process that can get to the important needles to proactively detect attacks, before damage is done. Realistically, security teams can actively handle a handful of alerts per 1,000 hosts per day. This is based on the Ponemon data (4% of handled events equals 6.8 alerts/1,000 hosts/day) and also on my experience with different chief information security officers. If a process generates many more, they will be ignored.

Using metrics enables evaluation and comparison of different tools and methods, including overlapping products. For example, you can compare SIEM-based event correlation to network traffic analytics, sandbox-based detection, user and entity behavior analytics, endpoint detection, malware analysis, and so on, with these questions:

  1. Does the type of new alerts add significant coverage or attack detection capability?
  2. How many alerts per day does the system normally generate? Can my team investigate that amount?
  3. What percentage of the alerts can be handled and are relevant to handle?

With this perspective, we can ask questions such as, “Will this increase detection accuracy?” “How is this security tool making the team more efficient?” and “Can this security solution reduce staffing requirements?”

Related Content:

 

Giora Engel, vice president, product & strategy at LightCyber is a serial entrepreneur with many years of technological and managerial experience. For nearly a decade, he served as an officer in an elite technological unit in the Israel Defense Forces, where he initiated and ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
Security Operations and IT Operations: Finding the Path to Collaboration
A wide gulf has emerged between SOC and NOC teams that's keeping both of them from assuring the confidentiality, integrity, and availability of IT systems. Here's how experts think it should be bridged.
Flash Poll
New Best Practices for Secure App Development
New Best Practices for Secure App Development
The transition from DevOps to SecDevOps is combining with the move toward cloud computing to create new challenges - and new opportunities - for the information security team. Download this report, to learn about the new best practices for secure application development.
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2013-7445
Published: 2015-10-15
The Direct Rendering Manager (DRM) subsystem in the Linux kernel through 4.x mishandles requests for Graphics Execution Manager (GEM) objects, which allows context-dependent attackers to cause a denial of service (memory consumption) via an application that processes graphics data, as demonstrated b...

CVE-2015-4948
Published: 2015-10-15
netstat in IBM AIX 5.3, 6.1, and 7.1 and VIOS 2.2.x, when a fibre channel adapter is used, allows local users to gain privileges via unspecified vectors.

CVE-2015-5660
Published: 2015-10-15
Cross-site request forgery (CSRF) vulnerability in eXtplorer before 2.1.8 allows remote attackers to hijack the authentication of arbitrary users for requests that execute PHP code.

CVE-2015-6003
Published: 2015-10-15
Directory traversal vulnerability in QNAP QTS before 4.1.4 build 0910 and 4.2.x before 4.2.0 RC2 build 0910, when AFP is enabled, allows remote attackers to read or write to arbitrary files by leveraging access to an OS X (1) user or (2) guest account.

CVE-2015-6333
Published: 2015-10-15
Cisco Application Policy Infrastructure Controller (APIC) 1.1j allows local users to gain privileges via vectors involving addition of an SSH key, aka Bug ID CSCuw46076.

Dark Reading Radio
Archived Dark Reading Radio
In past years, security researchers have discovered ways to hack cars, medical devices, automated teller machines, and many other targets. Dark Reading Executive Editor Kelly Jackson Higgins hosts researcher Samy Kamkar and Levi Gundert, vice president of threat intelligence at Recorded Future, to discuss some of 2016's most unusual and creative hacks by white hats, and what these new vulnerabilities might mean for the coming year.