Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


10:30 AM
Raffael Marty
Raffael Marty
Connect Directly
E-Mail vvv

Creating Your Own Threat Intel Through ‘Hunting’ & Visualization

How security analysts armed with a visual interface can use data science to find hidden attacks and the 'unknown unknowns.'

As 'cyber defenders' we can no longer work with external threat intelligence alone. We have to stop relying on how attacks have been conducted in the past and start hunting for signs of compromises and anomalies in our environments. This is the only way that we have a fighting chance to find advanced perpetrators.

While I don't have a silver bullet, my recent observations have shown that traditional security devices like firewalls, intrusion detection systems, security information event management (SIEM),  and most notably anti virus are all based on signatures -- knowledge of past attacks and methods. Back in the days of script kiddies and worms, those approaches worked fairly well. But the threat landscape has changed drastically. Today’s attacks are surgically planned and executed by very skilled adversaries. No two attacks are the same. And even worse, there is an entire industry of cybercriminals who make money by selling services and custom attack code, making it possible for anyone to become an "advanced adversary."

The problem with threat intel
Many companies today subscribe to some sort of threat intelligence feeds. They often get lists of malicious IP addresses and other indicators of compromise (IOC). These IOCs have the same problems as the attack signatures from yesterday because indicators are only useful for broad-stroke attacks where maybe an entire industry is hit with the same malware. It won't help if you are dealing with a targeted attack that concerns only your company.

So what can you do? Generate your own threat intelligence by going on the hunt! Hunting is a tremendous asset for threat intelligence teams, letting security analysts look for signs of anomalies and attacks in as much data (logs) as they can get their hands on. The hunting function is one that relies on both security analysts’ knowledge of their organization and environment and their security experience. Hunting is not about algorithms or pre-canned rules and signatures that describe past attacks. For effective hunting, in addition to good analysts, you also need data and some analytical capabilities. This is not a tool you download off the Internet or buy off the shelf.

[Learn more from Raffael about using visual analytics to deliver actionable security intelligence during his training sessions at Black Hat 2015, Las Vegas August 1-2 & 3-4.]

Where to begin
Start with collecting as much data as you can. Get data from your SIEM, your log management tools, log files, etc. and collect it in a big data lake. If you already have a columnar data store containing your security data, that's a great starting point. If you don't, go get one. For scalability I recommend storing your data on Hadoop in a columnar data format. Unfortunately, your SIEM is not suited for hunting. It doesn't have the right scalability and is too closed off when it comes to adding analytics and visualization on top of it (see below).

Once the data is in a fast data store, you are ready to let the analysts interact with the data. Don't expose your analysts to the textual data and have them write SQL queries. You want to empower your analysts, not slow them down. Give them a visual interface to interactively explore the data. Visually displaying large amounts of data requires the use of aggregation. We simply don't have enough pixels to display all of the data. This falls in the camp of data science with summarization on the one end of the spectrum and complex unsupervised learning algorithms on the other end. In any case, make sure the data science complexities are staying hidden from the analysts. You don't want them having to understand how these complex algorithms work. You need your analysts to be experts in security.

The unknown unknowns
The hunting or
exploratory process is all about finding unknown and hidden attacks. If you already know what you are looking for, visualization can help identify those instances quicker and easier, but generally, visualization is used to find the unknown unknowns. Once identified, the unknowns then get translated into one of the traditional analytics approaches: rules, statistics, or behaviors. This way we can automate the process of finding them in the future.

Some of the insights the hunting process discovers can't be described with a traditional analytics approach. The parameters are not clear or change ever so slightly. However, visually those outliers are quite apparent. In those cases it is necessary to continue using visualization on a regular basis to keep finding new instances of the same type of attacks. Below is a visual representation of end point data from a simple "hunt" that uncovered rogue DNS traffic used to exfiltrate information.  

  Black Hat USA is next month. Register here.

Raffael Marty is vice president of security analytics at Sophos. He is one of the world's most recognized authorities on security data analytics, big data and visualization. Previously, Marty launched pixlcloud, a visual analytics platform, and Loggly, a cloud-based log ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
COVID-19: Latest Security News & Commentary
Dark Reading Staff 7/9/2020
Introducing 'Secure Access Service Edge'
Rik Turner, Principal Analyst, Infrastructure Solutions, Omdia,  7/3/2020
Russian Cyber Gang 'Cosmic Lynx' Focuses on Email Fraud
Kelly Sheridan, Staff Editor, Dark Reading,  7/7/2020
Register for Dark Reading Newsletters
White Papers
Current Issue
Special Report: Computing's New Normal, a Dark Reading Perspective
This special report examines how IT security organizations have adapted to the "new normal" of computing and what the long-term effects will be. Read it and get a unique set of perspectives on issues ranging from new threats & vulnerabilities as a result of remote working to how enterprise security strategy will be affected long term.
Flash Poll
The Threat from the Internet—and What Your Organization Can Do About It
The Threat from the Internet—and What Your Organization Can Do About It
This report describes some of the latest attacks and threats emanating from the Internet, as well as advice and tips on how your organization can mitigate those threats before they affect your business. Download it today!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2020-07-09
An information leak was discovered on Yubico YubiKey 5 NFC devices 5.0.0 to 5.2.6 and 5.3.0 to 5.3.1. The OTP application allows a user to set optional access codes on OTP slots. This access code is intended to prevent unauthorized changes to OTP configurations. The access code is not checked when u...
PUBLISHED: 2020-07-09
In TimelineJS before version 3.7.0, some user data renders as HTML. An attacker could implement an XSS exploit with maliciously crafted content in a number of data fields. This risk is present whether the source data for the timeline is stored on Google Sheets or in a JSON configuration file. Most T...
PUBLISHED: 2020-07-09
The tough library (Rust/crates.io) prior to version 0.7.1 does not properly verify the threshold of cryptographic signatures. It allows an attacker to duplicate a valid signature in order to circumvent TUF requiring a minimum threshold of unique signatures before the metadata is considered valid. A ...
PUBLISHED: 2020-07-09
A reflected Cross-Site Scripting (XSS) Vulnerability in the KingComposer plugin through 2.9.4 for WordPress allows remote attackers to trick a victim into submitting an install_online_preset AJAX request containing base64-encoded JavaScript (in the kc-online-preset-data POST parameter) that is execu...
PUBLISHED: 2020-07-09
IBM Guardium Activity Insights 10.6 and 11.0 does not set the secure attribute on authorization tokens or session cookies. Attackers may be able to get the cookie values by sending a http:// link to a user or by planting this link in a site the user goes to. The cookie will be sent to the insecure l...