Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Perimeter

1/4/2011
12:06 PM
John H. Sawyer
John H. Sawyer
Commentary
50%
50%

Mining Web Proxy Logs For Interesting, Actionable Data

Simple statistical analysis of Web proxy logs provides wealth of information and incidents missed by AV

The importance of system logging and log analysis is often overlooked. I know it's easy to say that off-the-cuff, but I'll back it up with one of my favorite data points to come out of the Verizon Data Breach Investigation Report: 86% of the victims had evidence of the breach in their logs. As a result of that statistic, Verizon made the recommendation to "change your approach to event monitoring and log analysis."

In the past I've discussed centralized log collection and monitoring of Windows environments and the value of tools like Splunk. There's a plethora of logs within an organization that can provide insight into what's going on and when bad things are starting to happen. The problem is those logs are regularly ignored until it's too late and IT is scrambling to figure out what happened.

Antivirus logs often go unchecked with the assumption that they're working, but they can be useful in spotting attack trends and problematic users who regularly visit malicious sites. Likewise, Web proxy logs hold similar value and can be mined for a lot of useful, actionable data, like daily summaries of malicious HTTP User Agents, content types (think "executables"), and more.

Some recent research into proxy log mining turned up an interesting presentation, from Matthew Myrick of the Lawrence Livermore National Laboratory, titled "Mining Proxy Logs: Finding Needles In Haystacks." Matthew provided some excellent examples of how his team leverages its BlueCoat Web proxy logs to find "bad guys" through simple statistics, User Agents, content types, and compound searches. It's a great presentation that provides ideas of how easy it is to develop these tools in-house and perform daily analysis with little effort.

Another find during my research was a cool Ruby-based tool called LightBulb, which was created to help find automated traffic in BlueCoat Web proxy logs. The idea behind it was that malware must phone home and often does so based on a set interval. The beacon home can be found in the proxy logs by analyzing the amount of randomness of traffic to a website. Traffic with little to no randomness would indicate a regular beacon.

There are numerous other ways to slice and dice Web proxy logs to find bad things. For example, comparing a list of currently known malicious domains or Zeus malware domains and IPs to the proxy logs can help find hosts that have been attacked or infected, but not blocked by the Web proxy. And based on your experience and environment, you'll likely come up with other ways.

I think what ultimately has to happen for organizations is the realization that the logs are there and it doesn't take much work to pull interesting bits of data that can help provide better situational awareness. And, hopefully, it will help them catch something bad before they end up being another statistic in the Verizon report.

John Sawyer is a Senior Security Analyst with InGuardians. The views and opinions expressed in this blog are his own and do not represent the views and opinions of his employer. He can be reached at [email protected]

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Edge-DRsplash-10-edge-articles
7 Old IT Things Every New InfoSec Pro Should Know
Joan Goodchild, Staff Editor,  4/20/2021
News
Cloud-Native Businesses Struggle With Security
Robert Lemos, Contributing Writer,  5/6/2021
Commentary
Defending Against Web Scraping Attacks
Rob Simon, Principal Security Consultant at TrustedSec,  5/7/2021
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you today!
Flash Poll
How Enterprises are Developing Secure Applications
How Enterprises are Developing Secure Applications
Recent breaches of third-party apps are driving many organizations to think harder about the security of their off-the-shelf software as they continue to move left in secure software development practices.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2021-31922
PUBLISHED: 2021-05-14
An HTTP Request Smuggling vulnerability in Pulse Secure Virtual Traffic Manager before 21.1 could allow an attacker to smuggle an HTTP request through an HTTP/2 Header. This vulnerability is resolved in 21.1, 20.3R1, 20.2R1, 20.1R2, 19.2R4, and 18.2R3.
CVE-2021-32051
PUBLISHED: 2021-05-14
Hexagon G!nius Auskunftsportal before 5.0.0.0 allows SQL injection via the GiPWorkflow/Service/DownloadPublicFile id parameter.
CVE-2021-32615
PUBLISHED: 2021-05-13
Piwigo 11.4.0 allows admin/user_list_backend.php order[0][dir] SQL Injection.
CVE-2021-33026
PUBLISHED: 2021-05-13
The Flask-Caching extension through 1.10.1 for Flask relies on Pickle for serialization, which may lead to remote code execution or local privilege escalation. If an attacker gains access to cache storage (e.g., filesystem, Memcached, Redis, etc.), they can construct a crafted payload, poison the ca...
CVE-2021-31876
PUBLISHED: 2021-05-13
Bitcoin Core 0.12.0 through 0.21.1 does not properly implement the replacement policy specified in BIP125, which makes it easier for attackers to trigger a loss of funds, or a denial of service attack against downstream projects such as Lightning network nodes. An unconfirmed child transaction with ...