An international group of researchers has built a prototype system for detecting botnets on a large scale and that can sniff out previously undiscovered botnet command-and-control (C&C) servers.

Botnet hunters traditionally focus on inspecting individual botnets or botnet activity within organizations, for example, the researchers say. The new prototype, called Disclosure, expands the view of botnet activity to a wider scale and detects botnet C&C traffic in real-time, inspecting billions of flows of datasets each day, they say. It uses the NetFlow network protocol created by Cisco that gathers IP traffic data, plus some custom features they added that allow the tool to differentiate between C&C traffic and legitimate traffic based on flow size and behavior patterns of the clients, as well as time frames of the traffic. They also integrated it with some external reputation scoring services.

"I think the main contribution is that it's operating at such a large scale that you could have much broader [botnet] protection of the Internet at large," says William Robertson, assistant professor at the College of Computer and Information Science at Northeastern University, who, along with Engin Kirda of Northeastern, Leyla Bilge of Symantec Research Labs, Davide Balzarotti of Eurecom, and Christopher Kruegel of UC Santa Barbara, built and tested Disclosure.

"It's very efficient: It can process a day's worth of data in less than a day," Robertson says.

The prototype also was able to detect several botnet C&C servers that had been previously unknown, he says. "We manually verified those: We had some students probe those sites to discover if they were likely C&C servers or not."

Today's tools for botnet hunters provide them the ability to detect C&C channels between the botnet operator and the infected bots, or to detect botnets based on behavior among a group of machines that indicates they are bots, the researchers say.

"Once bots or, ideally, C&C servers have been identified, a number of actions can be performed, ranging from removal of infected endpoints from the network, to filtering C&C channels at edge routers, to orchestrated take-downs of the C&C servers themselves," the researchers wrote in their paper, which they will present in December at the Annual Computer Security Applications Conference in Orlando, Fla.

"Unfortunately, while previous botnet detection approaches are effective under certain circumstances, none of these approaches scales beyond a single administrative domain while retaining useful detection accuracy. This limitation restricts the application of automated botnet detection systems to those entities that are informed or motivated enough to deploy them," they wrote. "Thus, we have the current state of botnet mitigation, where small pockets of the Internet are fairly well protected against infection while the majority of endpoints remain vulnerable."

The prototype is not the first large-scale botnet protection approach, however: Damballa, for instance, offers DNS-based reputation filtering for protecting large customers such as ISPs.

Meanwhile, in tests of the tool in a university network and a Tier 1 ISP network, the researchers found that Disclosure spotted some 65 percent of known botnet C&C servers, with a 1 percent false-positive rate. It also caught new botnet C&C servers that weren't previously known.

NetFlow data is valuable in botnet detection, but NetFlow analysis alone has its limitations in an enterprise environment, where network address translation and IPSes can wreak havoc on detection there, security experts say. "But even in the ISP environment, flow-based systems have problems keeping up with the traffic. Therefore, as the authors of the paper discuss, they will have to do sampling of the overall NetFlow traffic. It is clear that by sampling the traffic, a large portion of the botnet traffic will not be observed due to the sampling," says Manos Antonakakis, principal scientist and director of academic sciences at Damballa. "Therefore, the particular flow-based botnet detection system will most likely detect quite noisy botnets" such as spam, DDoS, and peer-to-peer botnets, he says.

The researchers say their prototype is not meant to detect targeted attacks of mini-botnet C&C systems. "This approach is not for more targeted attacks. We are trying to look at characteristics of large-scale attacks," says Kirda, who is associate professor for information assurance at the College of Computer and Information Science and the Department of Electrical and Computer Engineering at Northeastern University. The researchers also previously had built a tool called Exposure that detects DNS anomalies.

[A large peer-to-peer botnet known for its resilience was spotted sniffing out potential victim voice-over-IP (VoIP) servers using an advanced stealth technique of camouflaging its efforts to recruit new bots. See Botnet Spotted Silently Scanning IPv4 Address Space For Vulnerable VoIP.]

Damballa's Antonakakis says Disclosure is yet another tool for botnet defenders. "New detection tools are useful in botnet research. I think research should focus more on how we can defend against emerging threats. To that extent, I consider this paper a step toward the right direction, however quite incremental, to already existing techniques," says Antonakakis, who while at Georgia Tech co-developed Notos (PDF), a dynamic reputation system for DNS traffic that helps spot botnet activity and that is used today by Damballa.

The Disclosure research paper is available here (PDF) for download.

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.