The University of Missouri-Columbia was starting to get a reputation among the anti-spam community a year ago. Its zombified client machines were unknowingly spewing spam around the Internet, landing the university on several email blacklists.

So the university's security and IT groups, who support some 28,000 students and 10,000 faculty and staff, decided to try blocking Internet Relay Chat (IRC) traffic -- the method most botnets use to control bot-infected machines.

"We wanted to reduce complaints from outside about spam on our campus going out" and to remove the university from spam blacklists, says Allen Brokken, Principal Systems Security Analyst for MU.

That meant adding the then-new Quarantine Protection option to the university's TippingPoint IPS last year. Quarantine works hand in hand with the university's device registration system, which ensures that the clients students bring to the campus network are properly registered -- and therefore quarantinable. "We determined the most efficient way was to block IRC using the IPS, and to use the Quarantine action on it," Brokken says.

When the IPS Quarantine detects a client machine trying to initiate an IRC session, the user is automatically quarantined and redirected to a MU device registration Web page that informs the user his or her machine is infected. The site then gives instructions and links for cleaning up the machine before the client is allowed on the network. Brokken says the IRC block, which the university configured with the system, isn't a typical application of TippingPoint's Quarantine.

A machine is quarantined for one hour, and then admitted back on the network if it's clean.

The only catch with the IRC block is that sometimes users have legitimate reasons to be running an IRC session with other users for gaming or academic reasons. Brokken admits there's really no way for the system to know for sure if it's a bot communiqué. "We look at how many times a user is getting quarantined for IRC. If you do something that violated policy, we will kick you off for one hour to see if you are still doing it," he says. "If you were infected and clean it up, an hour later you're back on the network."

At first the university did inadvertently snare some legit IRC users. But since then, the false positives have dropped to few or none. "We figure that when people using IRC as a normal business practice learn about the restriction on it, they do something else instead. So now we don't have a lot of false positives," he says.

The IRC block also decreased spam complaints against MU. "We went from polluting people with spam to capturing [the offenders]," Brokken says. "That was our biggest win."

Still, it's not a perfect system for catching bots. Botnet operators are increasingly moving away from IRC and using less-conspicuous channels to communicate, such as HTTP and point-to-point links. But Brokken says there are other rule set options in Quarantine that the university can eventually use to block other botnet vectors. (See Black Hat: Botnets Go One-on-One and Botnets Don Invisibility Cloaks.)

The university also runs firewalls in front of each residence hall, which helps minimize botnet infection, Brokken says. "If you're a machine on that network, you can communicate out, but no one can talk to you." That helps prevent a botnet controller from polling a potential zombie, although a user could still fall victim via an infected email message. "We see more bot-related traffic on the administrative side of the network," which is more open.

The downside with TippingPoint's Quarantine feature is that the non-default features require more manual configuration than the ones that are enabled by default, Brokken says. "They have a large rule set that's not deployed by default," he says. And it's time-consuming to add these rules and ensure they don't generate lots of false positives, as the IRC block initially did. "It took us a month to get the IRC block completely deployed."

The university is currently evaluating several different network access control (NAC) products to help secure and manage its client machines, especially with the potential for BlackBerry and other PDAs joining the network. A NAC solution may or may not replace the existing registration and quarantine system, Brokken says.

Meanwhile, Brokken expects the bot threat to continue, despite the university's measures to combat it. Until about a year ago, the university's IPS was mostly catching machines that were infected from other machines, such as residual worm infestations. "Then the demographics started to change, with less hits from Sasser/Blaster, and more external spam complaints," he says. "I have a feeling most of what we are dealing with now are bot machines, or someone downloading or doing something silly. My gut tells me that's where things are headed, with more sophisticated" attacks.

— Kelly Jackson Higgins, Senior Editor, Dark Reading