Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Threat Intelligence

10/1/2019
10:00 AM
Randy Caldejon
Randy Caldejon
Commentary
Connect Directly
LinkedIn
RSS
E-Mail vvv
50%
50%

AIOps: The State of Full Packet Capture Enters the Age of Practicality

How machine learning and artificial intelligence are changing the game of acting on large volumes of network data in near real time.

It's a great time to be a security analyst, but those who serve in the role today are facing much higher expectations from their organizations compared with when I started out. Many are teetering on the edge of burnout because their companies need to get to the truth sooner, leaving analysts stuck with traditional approaches and tactics associated with full packet capture as the high-speed network's bandwidth increases by the day.

The state of full packet capture — fundamental to enabling security analysts to hunt for threats, discover anomalies, or respond to incidents — has seen a few incremental advancements over the several decades but nothing that has allowed the analyst to allocate less time to it because there is still a bit of heavy lifting required.

As a security analyst in the military, my first experience with full packet capture in the late '90s was the SHADOW system, an open source project dubbed an intrusion-detection system but really a full packet capture system designed for retrospective analysis, also known as threat hunting. The project was essentially a framework built with tcpdump and a collection of Perl scripts. However, SHADOW lacked any form of indexing, so mining the data was quite painful.

The next breakthrough in full packet capture was Time Machine, which introduced the notion of connection cutoff and indexing for faster search and retrieval. A sister project to Zeek (formerly known as Bro), Time Machine was an interesting project with lots of promise. Unfortunately, Time Machine did not scale beyond a few gigabits per second. Finally, there is Moloch, a full packet capture and search application integrated with advanced visualization that scales to 10Gbps and more. Moloch represents the state-of-the-art in open source, full packet capture, but it is yet to be determined if it can scale to 100Gbps.

These incremental improvements were made in the background while the high-speed network expanded and has grown in importance within the organization. While the number of servers on-premises might have decreased, the quantity of mobile devices, Internet of Things sensors and cloud applications that organizations are utilizing today to improve operations is increasing to create an even more complex network environment, making the traditional approaches to full packet capture even more impractical.

Adding to the problem is the recent rise in overall traffic. which is forecasted to continue. According to Cisco, companies can expect to see their network traffic triple by 2022. This will require organizations to make a proportional increase in data storage and maintain a brute force, record-everything approach for network forensics that will cost companies significantly more in terms of time and money. This runs counter to most companies' digital transformation journeys where the bigger objective is to save on operational costs, increase IT agility, and improve responsiveness.

Fortunately, full packet capture is finally entering the age of practicality because of the introduction of AIOps. Gartner defines AIOps as the application of machine learning (ML) and data science to IT operations problems. The firm also predicts that large enterprises use of AIOps tools will reach 30% by 2023. The adoption of AIOps will pave the way to security automation like intelligent packet capture. This is an exciting development that our company is pursuing, along with others in the industry, to enable security analysts to utilize AIOps for network forensics.

The advancement of machine learning (ML) and artificial intelligence (AI) is enabling new innovations in full packet capture to bring some needed relief to the security analyst. When a machine learning engine 'learns' to classify packets to predict those that need to be recorded, the security analyst benefits by having data with higher fidelity allowing he or she to conduct more meaningful and expedient forensics. As noted security analyst and trainer, Chris Sanders says in his blog post, "if you can distill a PCAP down to key events then you'll have a much more manageable set of data points to aid your investigation."

Thanks to AIOps, security analysts now have an opportunity to utilize more open source technologies and experiment with ML and AI to make packet capture work better for them and their organizations. Before it was unrealistic to expect a group of analysts in a security operations center to proactively ferret through petabytes of data in search of an anomaly or indicator of compromise in a timely manner. Normally, this would be — at best — a week-long exercise without ML or AI. Access to these enabling technologies represent a significant improvement in the state of full packet capture, making them practical and invaluable resources for security analysts.

Related Content:

Check out The Edge, Dark Reading's new section for features, threat data, and in-depth perspectives. Today's top story: "5 Disruptive Trends Transforming Cybersecurity."

Randy Caldejon leads the company's innovation and product development. Prior to CounterFlow, Randy was the CTO of Enterprise Forensics at FireEye. He is a widely-respected authority in network security monitoring and sensor technology. A military veteran, engineer, and serial ... View Full Bio
 

Recommended Reading:

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
COVID-19: Latest Security News & Commentary
Dark Reading Staff 7/14/2020
Omdia Research Launches Page on Dark Reading
Tim Wilson, Editor in Chief, Dark Reading 7/9/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
Special Report: Computing's New Normal, a Dark Reading Perspective
This special report examines how IT security organizations have adapted to the "new normal" of computing and what the long-term effects will be. Read it and get a unique set of perspectives on issues ranging from new threats & vulnerabilities as a result of remote working to how enterprise security strategy will be affected long term.
Flash Poll
The Threat from the Internetand What Your Organization Can Do About It
The Threat from the Internetand What Your Organization Can Do About It
This report describes some of the latest attacks and threats emanating from the Internet, as well as advice and tips on how your organization can mitigate those threats before they affect your business. Download it today!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-6164
PUBLISHED: 2020-07-15
In SilverStripe through 4.5.0, a specific URL path configured by default through the silverstripe/framework module can be used to disclose the fact that a domain is hosting a Silverstripe application. There is no disclosure of the specific version. The functionality on this URL path is limited to ex...
CVE-2020-6165
PUBLISHED: 2020-07-15
SilverStripe 4.5.0 allows attackers to read certain records that should not have been placed into a result set. This affects silverstripe/recipe-cms. The automatic permission-checking mechanism in the silverstripe/graphql module does not provide complete protection against lists that are limited (e....
CVE-2020-8958
PUBLISHED: 2020-07-15
Guangzhou 1GE ONU V2801RW 1.9.1-181203 through 2.9.0-181024 and V2804RGW 1.9.1-181203 through 2.9.0-181024 devices allow remote attackers to execute arbitrary OS commands via shell metacharacters in the boaform/admin/formPing Dest IP Address field.
CVE-2020-9309
PUBLISHED: 2020-07-15
Silverstripe CMS through 4.5 can be susceptible to script execution from malicious upload contents under allowed file extensions (for example HTML code in a TXT file). When these files are stored as protected or draft files, the MIME detection can cause browsers to execute the file contents. Uploads...
CVE-2020-9311
PUBLISHED: 2020-07-15
In SilverStripe through 4.5, malicious users with a valid Silverstripe CMS login (usually CMS access) can craft profile information which can lead to XSS for other users through specially crafted login form URLs.