Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


10:30 AM
Raffael Marty
Raffael Marty
Connect Directly
E-Mail vvv

Creating Your Own Threat Intel Through ‘Hunting’ & Visualization

How security analysts armed with a visual interface can use data science to find hidden attacks and the 'unknown unknowns.'

As 'cyber defenders' we can no longer work with external threat intelligence alone. We have to stop relying on how attacks have been conducted in the past and start hunting for signs of compromises and anomalies in our environments. This is the only way that we have a fighting chance to find advanced perpetrators.

While I don't have a silver bullet, my recent observations have shown that traditional security devices like firewalls, intrusion detection systems, security information event management (SIEM),  and most notably anti virus are all based on signatures -- knowledge of past attacks and methods. Back in the days of script kiddies and worms, those approaches worked fairly well. But the threat landscape has changed drastically. Today’s attacks are surgically planned and executed by very skilled adversaries. No two attacks are the same. And even worse, there is an entire industry of cybercriminals who make money by selling services and custom attack code, making it possible for anyone to become an "advanced adversary."

The problem with threat intel
Many companies today subscribe to some sort of threat intelligence feeds. They often get lists of malicious IP addresses and other indicators of compromise (IOC). These IOCs have the same problems as the attack signatures from yesterday because indicators are only useful for broad-stroke attacks where maybe an entire industry is hit with the same malware. It won't help if you are dealing with a targeted attack that concerns only your company.

So what can you do? Generate your own threat intelligence by going on the hunt! Hunting is a tremendous asset for threat intelligence teams, letting security analysts look for signs of anomalies and attacks in as much data (logs) as they can get their hands on. The hunting function is one that relies on both security analysts’ knowledge of their organization and environment and their security experience. Hunting is not about algorithms or pre-canned rules and signatures that describe past attacks. For effective hunting, in addition to good analysts, you also need data and some analytical capabilities. This is not a tool you download off the Internet or buy off the shelf.

[Learn more from Raffael about using visual analytics to deliver actionable security intelligence during his training sessions at Black Hat 2015, Las Vegas August 1-2 & 3-4.]

Where to begin
Start with collecting as much data as you can. Get data from your SIEM, your log management tools, log files, etc. and collect it in a big data lake. If you already have a columnar data store containing your security data, that's a great starting point. If you don't, go get one. For scalability I recommend storing your data on Hadoop in a columnar data format. Unfortunately, your SIEM is not suited for hunting. It doesn't have the right scalability and is too closed off when it comes to adding analytics and visualization on top of it (see below).

Once the data is in a fast data store, you are ready to let the analysts interact with the data. Don't expose your analysts to the textual data and have them write SQL queries. You want to empower your analysts, not slow them down. Give them a visual interface to interactively explore the data. Visually displaying large amounts of data requires the use of aggregation. We simply don't have enough pixels to display all of the data. This falls in the camp of data science with summarization on the one end of the spectrum and complex unsupervised learning algorithms on the other end. In any case, make sure the data science complexities are staying hidden from the analysts. You don't want them having to understand how these complex algorithms work. You need your analysts to be experts in security.

The unknown unknowns
The hunting or
exploratory process is all about finding unknown and hidden attacks. If you already know what you are looking for, visualization can help identify those instances quicker and easier, but generally, visualization is used to find the unknown unknowns. Once identified, the unknowns then get translated into one of the traditional analytics approaches: rules, statistics, or behaviors. This way we can automate the process of finding them in the future.

Some of the insights the hunting process discovers can't be described with a traditional analytics approach. The parameters are not clear or change ever so slightly. However, visually those outliers are quite apparent. In those cases it is necessary to continue using visualization on a regular basis to keep finding new instances of the same type of attacks. Below is a visual representation of end point data from a simple "hunt" that uncovered rogue DNS traffic used to exfiltrate information.  

  Black Hat USA is next month. Register here.

Raffael Marty is vice president of security analytics at Sophos. He is one of the world's most recognized authorities on security data analytics, big data and visualization. Previously, Marty launched pixlcloud, a visual analytics platform, and Loggly, a cloud-based log ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Oldest First  |  Newest First  |  Threaded View
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Current Issue
2020: The Year in Security
Download this Tech Digest for a look at the biggest security stories that - so far - have shaped a very strange and stressful year.
Flash Poll
Assessing Cybersecurity Risk in Today's Enterprises
Assessing Cybersecurity Risk in Today's Enterprises
COVID-19 has created a new IT paradigm in the enterprise -- and a new level of cybersecurity risk. This report offers a look at how enterprises are assessing and managing cyber-risk under the new normal.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2021-01-25
An XML external entity (XXE) injection vulnerability was discovered in the Nutch DmozParser and is known to affect Nutch versions < 1.18. XML external entity injection (also known as XXE) is a web security vulnerability that allows an attacker to interfere with an application's processing of XML ...
PUBLISHED: 2021-01-25
When handler-router component is enabled in servicecomb-java-chassis, authenticated user may inject some data and cause arbitrary code execution. The problem happens in versions between 2.0.0 ~ 2.1.3 and fixed in Apache ServiceComb-Java-Chassis 2.1.5
PUBLISHED: 2021-01-22
Pepperl+Fuchs Comtrol IO-Link Master in Version 1.5.48 and below is prone to an authenticated reflected POST Cross-Site Scripting
PUBLISHED: 2021-01-22
Pepperl+Fuchs Comtrol IO-Link Master in Version 1.5.48 and below is prone to an authenticated blind OS Command Injection.
PUBLISHED: 2021-01-22
Pepperl+Fuchs Comtrol IO-Link Master in Version 1.5.48 and below is prone to a NULL Pointer Dereference that leads to a DoS in discoveryd