Lessons Learned In San Francisco Hack Job

An IT administrator with the city of San Francisco is alleged to have created a private administrative account on systems within the city's FiberWAN project.

Mike Fratto, Former Network Computing Editor

July 17, 2008

3 Min Read

At one time or another, it happens in almost every IT shop: A handful of people, or even one person, has the sole responsibility for and knowledge of critical systems. So much so that if any one of them were hit by a bus, your IT department would be severely hampered. Virtually every IT manager recognizes this problem, but far fewer do anything about it. San Francisco is now learning its lesson the hard way.

Terry Childs, an IT administrator with the city of San Francisco, whom the city had tried to fire in recent months, is alleged to have created a private administrative account on systems within the city's FiberWAN project. According to prosecutors, he's keeping the password a secret. In so doing, he's locked everyone else out of the system's networking gear and landed himself in jail, with bail set at $5 million. The city is trying to regain access to its network and has brought in experts from Cisco to help break into the compromised systems. The details are murky but also not vital to learning from this painful incident. Simply put, a trusted employee is accused of using that trust to hose his employer.

Organizations must trust employees to act honestly and ethically while at the same time acknowledge that employees pose one of the greatest threats. Employees must have access to systems, and that means they can do damage. InformationWeek's 2008 Strategic Security study found that 53% of respondents considered authorized users/employees one of the greatest threats, and 48% of respondents considered all employees one of the greatest threats. While most employees are ethical, all it takes is one disgruntled person to cause you headache and misery.

No technology or process can fully stop an insider, especially a knowledgeable insider like Childs, from causing harm--but what can be managed, and what's shocking about this incident, is its apparent magnitude. Technology can help. Strong authentication systems could have made creating unique credentials more difficult, and change management products might have alerted someone to Childs' activities. But technology alone isn't the solution; good management plays a part, too.

Jonathan Feldman, director of IT for Asheville, N.C., says problems like these can crop up when the IT department relies on heroes versus using a team approach. He describes the IT hero as a lone wolf, a go-to person who is solely responsible for critical systems. IT heroes often don't like to give up control over their systems, Feldman points out. These employees pose a real risk to the organization because if they leave the job, are unable to work, or simply decide to strike back, no one else has the knowledge to work around them.

What's At Stake • NO ADMINS
Childs changed router and switch administrator passwords

The network serves 60% of the city's needs, including some police systems

Childs is suspended with pay while he sits in jail

The solution is to cross-train IT staff so that no one person alone understands and controls key systems; spread responsibility for systems over several people; and adopt a practical change management process. By doing those three things, the harm that any one person can inflict on your IT department is lessened because there are redundant skills and knowledge in play.

Building an IT team rather than relying on lone wizards requires management discipline and may seem a more costly approach; IT heroes are an easy shortcut for even the best-intentioned managers. However, the payoff in building a team is a more resilient IT department. "You know you are successful when you can let IT personnel go on vacation without having to ask for their cell phone and hotel numbers in case something comes up," says Feldman.

Rogue employees can still wreak havoc if they want. The best you can do is to take steps to minimize the damage.

About the Author(s)

Mike Fratto

Former Network Computing Editor

Mike Fratto is a principal analyst at Current Analysis, covering the Enterprise Networking and Data Center Technology markets. Prior to that, Mike was with UBM Tech for 15 years, and served as editor of Network Computing. He was also lead analyst for InformationWeek Analytics and executive editor for Secure Enterprise. He has spoken at several conferences including Interop, MISTI, the Internet Security Conference, as well as to local groups. He served as the chair for Interop's datacenter and storage tracks. He also teaches a network security graduate course at Syracuse University. Prior to Network Computing, Mike was an independent consultant.

Keep up with the latest cybersecurity threats, newly discovered vulnerabilities, data breach information, and emerging trends. Delivered daily or weekly right to your email inbox.

You May Also Like

More Insights