Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Analytics

7/9/2015
10:30 AM
Raffael Marty
Raffael Marty
Commentary
Connect Directly
Twitter
RSS
E-Mail vvv
100%
0%

Creating Your Own Threat Intel Through ‘Hunting’ & Visualization

How security analysts armed with a visual interface can use data science to find hidden attacks and the 'unknown unknowns.'

As 'cyber defenders' we can no longer work with external threat intelligence alone. We have to stop relying on how attacks have been conducted in the past and start hunting for signs of compromises and anomalies in our environments. This is the only way that we have a fighting chance to find advanced perpetrators.

While I don't have a silver bullet, my recent observations have shown that traditional security devices like firewalls, intrusion detection systems, security information event management (SIEM),  and most notably anti virus are all based on signatures -- knowledge of past attacks and methods. Back in the days of script kiddies and worms, those approaches worked fairly well. But the threat landscape has changed drastically. Today’s attacks are surgically planned and executed by very skilled adversaries. No two attacks are the same. And even worse, there is an entire industry of cybercriminals who make money by selling services and custom attack code, making it possible for anyone to become an "advanced adversary."

The problem with threat intel
Many companies today subscribe to some sort of threat intelligence feeds. They often get lists of malicious IP addresses and other indicators of compromise (IOC). These IOCs have the same problems as the attack signatures from yesterday because indicators are only useful for broad-stroke attacks where maybe an entire industry is hit with the same malware. It won't help if you are dealing with a targeted attack that concerns only your company.

So what can you do? Generate your own threat intelligence by going on the hunt! Hunting is a tremendous asset for threat intelligence teams, letting security analysts look for signs of anomalies and attacks in as much data (logs) as they can get their hands on. The hunting function is one that relies on both security analysts’ knowledge of their organization and environment and their security experience. Hunting is not about algorithms or pre-canned rules and signatures that describe past attacks. For effective hunting, in addition to good analysts, you also need data and some analytical capabilities. This is not a tool you download off the Internet or buy off the shelf.

[Learn more from Raffael about using visual analytics to deliver actionable security intelligence during his training sessions at Black Hat 2015, Las Vegas August 1-2 & 3-4.]

Where to begin
Start with collecting as much data as you can. Get data from your SIEM, your log management tools, log files, etc. and collect it in a big data lake. If you already have a columnar data store containing your security data, that's a great starting point. If you don't, go get one. For scalability I recommend storing your data on Hadoop in a columnar data format. Unfortunately, your SIEM is not suited for hunting. It doesn't have the right scalability and is too closed off when it comes to adding analytics and visualization on top of it (see below).

 Simplified data flow that supports the hunting process through interactive visualization.

Simplified data flow that supports the hunting process through interactive visualization.

Once the data is in a fast data store, you are ready to let the analysts interact with the data. Don't expose your analysts to the textual data and have them write SQL queries. You want to empower your analysts, not slow them down. Give them a visual interface to interactively explore the data. Visually displaying large amounts of data requires the use of aggregation. We simply don't have enough pixels to display all of the data. This falls in the camp of data science with summarization on the one end of the spectrum and complex unsupervised learning algorithms on the other end. In any case, make sure the data science complexities are staying hidden from the analysts. You don't want them having to understand how these complex algorithms work. You need your analysts to be experts in security.

 A visualization of network traffic using a data science approach to color and plot traffic based on its similarity.

A visualization of network traffic using a data science approach to color and plot traffic based on its similarity.

The unknown unknowns
The hunting or exploratory process is all about finding unknown and hidden attacks. If you already know what you are looking for, visualization can help identify those instances quicker and easier, but generally, visualization is used to find the unknown unknowns. Once identified, the unknowns then get translated into one of the traditional analytics approaches: rules, statistics, or behaviors. This way we can automate the process of finding them in the future.

Some of the insights the hunting process discovers can't be described with a traditional analytics approach. The parameters are not clear or change ever so slightly. However, visually those outliers are quite apparent. In those cases it is necessary to continue using visualization on a regular basis to keep finding new instances of the same type of attacks. Below is a visual representation of end point data from a simple "hunt" that uncovered rogue DNS traffic used to exfiltrate information.  

 Graph 1 is a representation of how processes on an end point interact with external machines. The bar chart on the left shows the ports used in the communications. Graph 2 shows a zoom in on the DNS traffic, which is highlighted in red. Instead of using the default DNS server, we see that Chrome used a secondary DNS server as well. In a further drill down into the original records below, we see in Graph 3 the exact details of the traffic.

Graph 1 is a representation of how processes on an end point interact with external machines. The bar chart on the left shows the ports used in the communications. Graph 2 shows a zoom in on the DNS traffic, which is highlighted in red. Instead of using the default DNS server, we see that Chrome used a secondary DNS server as well. In a further drill down into the original records below, we see in Graph 3 the exact details of the traffic.

  Black Hat USA is next month. Register here.

Raffael Marty is vice president of security analytics at Sophos. He is one of the world's most recognized authorities on security data analytics, big data and visualization. Previously, Marty launched pixlcloud, a visual analytics platform, and Loggly, a cloud-based log ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Small Business Security: 5 Tips on How and Where to Start
Mike Puglia, Chief Strategy Officer at Kaseya,  2/13/2020
Architectural Analysis IDs 78 Specific Risks in Machine-Learning Systems
Jai Vijayan, Contributing Writer,  2/13/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
6 Emerging Cyber Threats That Enterprises Face in 2020
This Tech Digest gives an in-depth look at six emerging cyber threats that enterprises could face in 2020. Download your copy today!
Flash Poll
How Enterprises Are Developing and Maintaining Secure Applications
How Enterprises Are Developing and Maintaining Secure Applications
The concept of application security is well known, but application security testing and remediation processes remain unbalanced. Most organizations are confident in their approach to AppSec, although others seem to have no approach at all. Read this report to find out more.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-9308
PUBLISHED: 2020-02-20
archive_read_support_format_rar5.c in libarchive before 3.4.2 attempts to unpack a RAR5 file with an invalid or corrupted header (such as a header size of zero), leading to a SIGSEGV or possibly unspecified other impact.
CVE-2019-20479
PUBLISHED: 2020-02-20
A flaw was found in mod_auth_openidc before version 2.4.1. An open redirect issue exists in URLs with a slash and backslash at the beginning.
CVE-2011-2498
PUBLISHED: 2020-02-20
The Linux kernel from v2.3.36 before v2.6.39 allows local unprivileged users to cause a denial of service (memory consumption) by triggering creation of PTE pages.
CVE-2012-2629
PUBLISHED: 2020-02-20
Multiple cross-site request forgery (CSRF) and cross-site scripting (XSS) vulnerabilities in Axous 1.1.1 and earlier allow remote attackers to hijack the authentication of administrators for requests that (1) add an administrator account via an addnew action to admin/administrators_add.php; or (2) c...
CVE-2014-3484
PUBLISHED: 2020-02-20
Multiple stack-based buffer overflows in the __dn_expand function in network/dn_expand.c in musl libc 1.1x before 1.1.2 and 0.9.13 through 1.0.3 allow remote attackers to (1) have unspecified impact via an invalid name length in a DNS response or (2) cause a denial of service (crash) via an invalid ...