Why Don't Firewalls Work?

A large enterprise recently realized one of its firewalls had suddenly gone quiet, and was no longer requesting policy changes or updates as usual. After a careful audit, the enterprise discovered the reason: A hacker had inserted an "allow any to any" rule in the middle of the firewall's policy, leaving the doors wide open to all traffic, both in and out of the company.

Such blatant hacks are rare, but the story -- relayed by firewall maker Cisco Systems -- is a cautionary tale, experts say. A poorly configured firewall can be worse for data security than no firewall at all.

"We see customers that have hundreds of rules and thousands of objects defined on their firewalls," says Matt Dryer, product marketing manager for the Security Technology Business Unit at Cisco. "They go in and add rules and objects, but they never delete anything. They don't follow a structured change control process. They don't have an unmanageable amount of gear, but the configuration and change management process just keeps getting more complex."

The proliferation of firewalls in large enterprises is making management even more difficult, notes Mike Lloyd, chief scientist at RedSeal, which makes software that aids in enterprise firewall change and security posture management. "The problem with firewalls is that they were originally designed to secure a closed environment, like you'd secure a bank. But today's enterprise is more like a city than a bank. There needs to be some fundamental change in the way enterprises think about their firewalls."

The problem, experts say, is not in the firewall technology itself, but in the way the firewalls are administered. In most companies, firewall administrators have a wide variety of other responsibilities, and they simply don't have the time or information they need to set all of the rules properly.

"About 95 percent of firewall issues are configuration errors, not vulnerabilities in the firewalls themselves, says Nimrod Reichenberg, vice president of marketing at AlgoSec, which makes tools for firewall configuration and change management. "Most of the issues are caused by human error."

Mike Rothman, an analyst at security consulting firm Securosis, agrees. "Most of the issues with firewalls relate to the user opening something they shouldn't," he says. "This could be because a user asks for a port to be opened, and the admin doesn't realize what the impact of that is. Or it could involve adding [or removing] a rule, which obviates more stringent controls lower in the rule base. The bad guys are constantly doing reconnaissance to figure out which ports and protocols are open, and then attacking them. So if a perimeter firewall is inadvertently opened, there is a pretty high likelihood the issue will be discovered and exploited quickly."

For most companies, problems with the firewall emerge not because of a breach, but because of a compliance audit. Firewalls are a key element in many audits that involve compliance with SOX, PCI, or other regulations, and a misconfigured firewall is more likely to be discovered by an auditor than a hacker, experts say.

"The companies that are regulated generally have to do an audit on an annual basis," Dreyer observes. "That's when the issues usually come out."

In a perfect world, companies would audit their firewalls much more frequently to delete unnecessary rules and test the impact of rule changes, Dreyer says. But most IT organizations operate in a world that's far from perfect.

"In a typical enterprise, you'll see at least 10 change requests a day, going up to as many as 200," RedSeal's Lloyd says. "It's more than a human can handle."

Avishai Wool, CTO and co-founder of AlgoSec, concurs. He notes that when security staff turns over in organizations, much of the knowledge about how firewalls are configured is lost. Most IT organizations don't have time to do research on how and why firewall rules are set, and most of them don't have time to test new rules and changes to see what their impact might be on the overall security picture, he notes.

At the recent Computer Security Institute conference in Washington, D.C., Lloyd outlined a plethora of ways in which a single error in syntax in a single line of firewall code can create major security vulnerabilities. "What may seem arcane might be exactly the sort of thing that an attacker is looking for," he says.

In many cases, the errors occur not on the inbound side of the firewall, but on the outbound side, Lloyd observes. "They've written reasonable controls for traffic coming into the environment, but they write rules that allow just about anything out," he says. "The assumption is that outbound traffic is trusted, but they don't put enough thought into what might be exposed. A credit card database doesn't need to surf the Internet. But a lot of them are exposed to it."