Zebras are the employees, and their computers, who are doing something odd. Defenders are right to want to protect the zebras in their networks, but defenders should occasionally "radio tag" and follow their zebras to see where they go, he said.
On one day, for example, Larsen saw more than 700 users had visited a malicious domain at least once. So he focused on the nine users with more than six hits each, finding one who visited a malicious domain whose name consisted of more than 50 characters. That's suspicious, he said.
"You don't just want to send the box off to get reimaged; you want to know what this is," he told attendees. "This potentially could be something scary. And that is what we are looking for: Things that could be advanced and targeted."
Larsen's data came from crunching the anomalies from Blue Coat's K9 Web Protection browser plug-in, which warns users of malicious Web sites and enforces parental controls. Yet companies can mine the information from firewall logs in the same way to turn the mass of log data into much more focused intelligence on the potential threat in their networks, he said.
Firewall and managed-security experts weighed in on the best ways for security professionals to find the unusual activity -- the zebras -- in their networks.
1. Know the network.
Before crunching any numbers, companies need to know what "normal" looks like. Larsen only got about 5 percent of the data from Blue Coat's K9 network -- anonymized, of course -- because those were the zebras showing abnormal behavior.
Companies need to do the same. By profiling their networks over time, companies can know what behavior seems strange and find the 5 percent to which they need to pay attention, says Jeff Williams, director of security strategy for managed-security provider Dell SecureWorks.
"If you know what you have in your network, and what systems should be talking to what other systems, and what those conversations should look like, and how often they should be occurring, that helps you understand what is normal," he says. "Only once you understand what is normal can you spot those anomalies."
[For big companies looking to spend big budgets, the Big Data pitch for security information and event management (SIEM) systems is a good fit, but other improvements are on the way. See More Improvements To SIEM Than Big Data.]
2. Collect all the data.
Companies also need to configure their firewalls and other devices to collect the right data. In many cases, a company will store only the dropped traffic, arguing that such data is most interesting. But the most serious attacks are the ones that get through the firewall, says Jody Brazil, chief technology officer and co-founder of firewall management firm FireMon.
Companies will commonly disable the logs on their most used firewall rules, many times because their firewalls are overtaxed, he says.
"If the firewall is doing its job and dropping traffic, and you trust the technology that you have purchased, why are we focusing all of our attention on the traffic that is being dropped and not on the traffic that is getting through?" Brazil says.
3. Find the foolish zebras.
Many security teams attempt to find every threat that enters their networks and quickly become overwhelmed. Instead, companies should look for the low-hanging fruit -- the foolish zebras -- and figure out what is going on there first.
Blue Coat's Larsen pays attention to only the most blatantly anomalous traffic to cut down on his team's workload. In his RSA presentation, for example, he looked at users who had gone to sites classified as "suspicious," but raised the bar even higher and checked out the 10 users who had hit more than 30 suspicious sites each. One user visited a domain consisting of 35 x's and the .com top-level domain name 37 times.
"There are a bunch of zebras that have the same kind of infection, the same kind of behavior, but I'm really interested in the abnormal-abnormal," Larsen said. "In my little group of foolish zebras, if there is one guy that is red-and-black striped, that is where I want to spend my time, because that is where I may find something really interesting and really targeted."
4. Combine with threat intelligence.
Much of the time, it's not large volumes of traffic that will tip off a security team to malicious activity, but where the traffic is coming from or going to. Free blacklists and commercial sources of threat data, when combined with a company's firewall logs, can find the malicious attacks that may otherwise escape notice, FireMon's Brazil says.
There are a lot of decent threat sources out there today, and inexpensive tools that can be used to combine them with firewall data, he says.
"For someone that is low on budget, you can perform this with existing log aggregation tools, but I would not try to do this by hand," says Brazil, who is a big proponent of security information and event monitoring (SIEM) systems.
5. Check back on your foolish zebras.
Gathering intelligence on attacks can reveal the motives of the attackers and help train the security team and incident responders at the same time. Yet even after a system has been cleaned and the investigation completed, checking up on the infected users can return dividends, Blue Coat's Larsen said.
"Once you have found a good foolish zebra, they are worth their weight in gold," he said. "It's not just this investigation. Give that zebra a week or two, go back and see where they have been lately."
In his experience, zebras rarely change their stripes.
Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.