Perimeter
1/4/2011
12:06 PM
John H. Sawyer
John H. Sawyer
Commentary
Connect Directly
RSS
E-Mail
50%
50%

Mining Web Proxy Logs For Interesting, Actionable Data

Simple statistical analysis of Web proxy logs provides wealth of information and incidents missed by AV

The importance of system logging and log analysis is often overlooked. I know it's easy to say that off-the-cuff, but I'll back it up with one of my favorite data points to come out of the Verizon Data Breach Investigation Report: 86% of the victims had evidence of the breach in their logs. As a result of that statistic, Verizon made the recommendation to "change your approach to event monitoring and log analysis."

In the past I've discussed centralized log collection and monitoring of Windows environments and the value of tools like Splunk. There's a plethora of logs within an organization that can provide insight into what's going on and when bad things are starting to happen. The problem is those logs are regularly ignored until it's too late and IT is scrambling to figure out what happened.

Antivirus logs often go unchecked with the assumption that they're working, but they can be useful in spotting attack trends and problematic users who regularly visit malicious sites. Likewise, Web proxy logs hold similar value and can be mined for a lot of useful, actionable data, like daily summaries of malicious HTTP User Agents, content types (think "executables"), and more.

Some recent research into proxy log mining turned up an interesting presentation, from Matthew Myrick of the Lawrence Livermore National Laboratory, titled "Mining Proxy Logs: Finding Needles In Haystacks." Matthew provided some excellent examples of how his team leverages its BlueCoat Web proxy logs to find "bad guys" through simple statistics, User Agents, content types, and compound searches. It's a great presentation that provides ideas of how easy it is to develop these tools in-house and perform daily analysis with little effort.

Another find during my research was a cool Ruby-based tool called LightBulb, which was created to help find automated traffic in BlueCoat Web proxy logs. The idea behind it was that malware must phone home and often does so based on a set interval. The beacon home can be found in the proxy logs by analyzing the amount of randomness of traffic to a website. Traffic with little to no randomness would indicate a regular beacon.

There are numerous other ways to slice and dice Web proxy logs to find bad things. For example, comparing a list of currently known malicious domains or Zeus malware domains and IPs to the proxy logs can help find hosts that have been attacked or infected, but not blocked by the Web proxy. And based on your experience and environment, you'll likely come up with other ways.

I think what ultimately has to happen for organizations is the realization that the logs are there and it doesn't take much work to pull interesting bits of data that can help provide better situational awareness. And, hopefully, it will help them catch something bad before they end up being another statistic in the Verizon report.

John Sawyer is a Senior Security Analyst with InGuardians. The views and opinions expressed in this blog are his own and do not represent the views and opinions of his employer. He can be reached at johnhsawyer@gmail.com

Comment  | 
Print  | 
More Insights
Register for Dark Reading Newsletters
Partner Perspectives
What's This?
In a digital world inundated with advanced security threats, Intel Security seeks to transform how we live and work to keep our information secure. Through hardware and software development, Intel Security delivers robust solutions that integrate security into every layer of every digital device. In combining the security expertise of McAfee with the innovation, performance, and trust of Intel, this vision becomes a reality.

As we rely on technology to enhance our everyday and business life, we must too consider the security of the intellectual property and confidential data that is housed on these devices. As we increase the number of devices we use, we increase the number of gateways and opportunity for security threats. Intel Security takes the “security connected” approach to ensure that every device is secure, and that all security solutions are seamlessly integrated.
Featured Writers
White Papers
Cartoon
Current Issue
Dark Reading's October Tech Digest
Fast data analysis can stymie attacks and strengthen enterprise security. Does your team have the data smarts?
Flash Poll
Title Partner’s Role in Perimeter Security
Title Partner’s Role in Perimeter Security
Considering how prevalent third-party attacks are, we need to ask hard questions about how partners and suppliers are safeguarding systems and data.
Video
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2014-3409
Published: 2014-10-25
The Ethernet Connectivity Fault Management (CFM) handling feature in Cisco IOS 12.2(33)SRE9a and earlier and IOS XE 3.13S and earlier allows remote attackers to cause a denial of service (device reload) via malformed CFM packets, aka Bug ID CSCuq93406.

CVE-2014-4620
Published: 2014-10-25
The EMC NetWorker Module for MEDITECH (aka NMMEDI) 3.0 build 87 through 90, when EMC RecoverPoint and Plink are used, stores cleartext RecoverPoint Appliance credentials in nsrmedisv.raw log files, which allows local users to obtain sensitive information by reading these files.

CVE-2014-4623
Published: 2014-10-25
EMC Avamar 6.0.x, 6.1.x, and 7.0.x in Avamar Data Store (ADS) GEN4(S) and Avamar Virtual Edition (AVE), when Password Hardening before 2.0.0.4 is enabled, uses UNIX DES crypt for password hashing, which makes it easier for context-dependent attackers to obtain cleartext passwords via a brute-force a...

CVE-2014-4624
Published: 2014-10-25
EMC Avamar Data Store (ADS) and Avamar Virtual Edition (AVE) 6.x and 7.0.x through 7.0.2-43 do not require authentication for Java API calls, which allows remote attackers to discover grid MCUser and GSAN passwords via a crafted call.

CVE-2014-6151
Published: 2014-10-25
CRLF injection vulnerability in IBM Tivoli Integrated Portal (TIP) 2.2.x allows remote authenticated users to inject arbitrary HTTP headers and conduct HTTP response splitting attacks via unspecified vectors.

Best of the Web
Dark Reading Radio
Archived Dark Reading Radio
Follow Dark Reading editors into the field as they talk with noted experts from the security world.