Security's School of Hard Knocks

"The first cut is the deepest."

While we were never exactly sure what that old song lyric had to with trying to love again, it certainly comes to mind as IT security professionals discuss the one incident that taught them the most painful lesson of their careers.

It's not easy to get people to talk about their mistakes. "So how did it feel when you hit your thumb with the hammer? What do you wish you’d done differently?"

Fortunately, a handful of enterprise IT folk stepped forward to discuss their experiences in character building. Each did so in the spirit of sparing their peers similar scars and experiences.

In a couple instances, they requested anonymity, less out of pride than the potential legal exposure their war stories might create. What's interesting, though, is that technology and hackers and nefarious new malware weren't the typical instrument of these lessons. Human nature, politics, and carelessness were. Seasoned IT pros know that effective security sometimes means knowing something about people as well as technology.

Here are five lessons in IT security that helped these IT pros open their eyes and see their jobs differently. We hope they help you learn something, too -- in a less painful way.

Just Hate When That Happens
The CIO of a major e-commerce site knew that third-party penetration tests would be good for his network and would improve its overall security. But in retrospect, he wishes he'd asked the white hat company he hired to avoid his CEO's email account.

The pen testers bombarded the company's servers with all sorts of garbage designed to expose the holes. They took out the CEO's account and the director of human resources as well. But that's not why he asked us to suppress his name and company.

The worst actually came after the tests were concluded. As part of the contract, the pen testers were to deliver a CD-ROM with all the testing and scanning information -- email addresses, IP addresses, server names, locations -- to the e-commerce company, noting what they found and how each segment performed.

The CD-ROM came in the mail. It had all the required information about internal networks and systems -- for another customer.

"It was a government customer, but not one with a three-letter acronym," the CIO said, who can laugh about it now. "Either way, I was pretty sure I shouldn't be looking at this info. I then asked, 'Where the hell is our data?' They claimed it was a one-way error and the person copying the file hadn’t switched the disks and sent ours to the government agency."

The CIO took the misrouted CD-ROM and put it in the company's fire safe until the issue of payment was resolved. "I told them we would pay them nothing. And both parties agreed to walk away like nothing happened," he said. The issue took less than two weeks to settle. "The firm realized this was the sort of event that could put them out of business."

The point here is to know who you're getting into bed with, network-wise. "We have a lot more questions about who will do testing -- one person or a team of people? And what type of tests -- will you target individual users, and internal or external systems? We also reach agreement on which IP address ranges, or these mail servers, but not individual mail attacks against a specific user," the CIO said. "We just really try to refine expectations."

The company still does pen testing twice a year, with the grim knowledge that any outside firm you work with increases your level of risk. But while it's always a gamble, there are ways to mitigate the risk and exposure of network assets and information.

Parasites, Be Gone
In the spring of 1996, in his then-position of managing the Website for database vendor Sybase, Michael Schaffer got an unexpected call. It was someone in the data center, alerting him that a server was running out of disk space. It was a Sparc 5 machine that Schaffer used for the Web access and for FTP.

"It was odd, because I hadn't put that much stuff on it," said Schaffer, who is now CTO of media e-commerce site Alibris, Emeryville, Calif. He started to investigate.

"Out of the corner of my eye and I saw a file called '...' [three dots]. We had '.' and '..' files," but these were clearly nothing he or his staff had created. "Once my pulse came down, I started to delete the files, at which point I saw a file created with the title, 'Who’s deleting all my files?' "

Schaffer didn't hesitate in response. "I created a file called 'Get off my server, you parasite!' "

When he began opening the illicitly stored files, he found "cracks," and "warez," which can be used to circumvent or break the licensing on commercial software. "When I went behind the looking glass, I found the Sybase FTP site was being used as a distribution site underneath our anonymous FTP site."

The solution was relatively easy; Schaffer and his team created a nailed-down directory so that other users couldn't see what had been uploaded. "You have to be careful when you set up an FTP site to allow uploads, but not downloads. Otherwise, it becomes a public space," he said. "You have to separate the two functions and not let them anonymously upload and download the same material."

It's still a good way to keep parasites from taking over your network.

Politics and Passive Aggression
Upgrades to improve security and automate manual tasks aren't always greeted with open arms. Some people worry they might be automated right out of a job. Others don't like any kind of change. And some people just don't like other people.

Ryan Mann never quite got to the bottom of why a particular department resisted testing the proof-of-concept process he oversaw prior to purchasing and deployment. Mann, who now works as IT security officer for food giant Del Monte Corp., Pittsburgh, said he experienced an odd sort of pushback from a group of users on a server at his last job when it came time to test a security event management (SEM) product across the unnamed enterprise.

"I know that SEM technology in general is seen outside the security department as big brother technology. Most other admins don’t care for it, since SEM has visibility into everything they're doing," Mann explained.

Still, Mann was under a short deadline to get the proof of concept done. And the resistant group/server in question had some spanning-tree problems that were tough to troubleshoot, and incredibly time consuming, given the scope of the project. Staff in the affected department proved indifferent, at best.

"The takeaway here for me was when you do things that are going to step on toes, you need both buy-in and prioritization," Mann said. He had the buy-in, he said, but for whatever reason, these other admins wouldn't make his test a priority. "We ended up having to make a [purchase] decision without all the information we wanted on the SEM product" and how it would impact existing systems and networks.

There wasn't much of a white-knuckle period between purchasing and deployment. The presumed problem with the server in question turned out to be a non-issue. But for Mann, it was a good lesson in the politics of technology.

This Could Get Violent
As social engineer extraordinaire Steve Stasiukonis recalls, that idea didn't occur to him until much, much later. Long after the adrenaline had subsided and the sweat had dried from a fight over a rogue user's laptop. (See Let's Wrestle for It.)

Timing's everything, according to Stasiukonis, VP and founder of Secure Network Technologies. The user had been caught red-handed breaking company security policy, and it was Stasiukonis' job to take the computer away.

Stasiukonis tried to detach the laptop and cabling while the miscreant stepped away from his desk. But before long, the accused user underwent a Jekyll-and-Hyde transformation that still takes Stasiukonis' breath away.

The easy-going, compliant user suddenly began channeling the rage of the Incredible Hulk and the menacing strength of Hulk Hogan. "When something gets taken from people that they value -- like their identity and personal communications and life -- they change in front of your eyes. They turn from mild mannered and meek to WWE [World Wrestling Entertainment]," Stasiukonis said.

For his trouble, Stasiukonis got chided both here and offline for taking needless risks, a lack of planning, and no contingency if the rogue had pulled a knife or a gun.

"I don't agree with those criticisms, but I certainly learned something about human nature," he said. "Never underestimate the individual. Always assume he's going to be hostile and is going to react in a questionable and potentially physically dangerous way."

No security incident is worth risking life and limb, especially when other options are available.

Inadvertent Spamming
Anyone with a Website will tell you: Ports and addresses get scanned multiple times during any 24-hour period, like warehouses where petty thieves try the doors and windows.

But a commercial Website operator who asked to remain anonymous said it was a CGI script written for the site that provided inadvertent entry for a Chinese spammer. "The customer-facing part of our site collected customer email, which then got exported to our internal data warehouse," said the IT executive. "But we neglected to lock it down. It didn’t have any links anywhere, but it was still an application to transfer data from outside into our universe."

Fast forward a few months, when the exec fielded a complaint from a customer, who said the site had sold his email address to spammers. "Some customers create an email address for e-commerce sites like ours, so they know that if they get spam on that address, it came from a specific ecommerce company."

After reviewing log files and traffic patterns, the company found that a Chinese server had captured a bunch of customer email addresses. No financial data was involved, but the exposure was an unpleasant wakeup call.

"My lesson was this: Code that you write for internal use had better not be usable externally," he said, especially since state laws require disclosure now when such leaks occur. "If you're ever running a public Website, take some time to tail the logs. Watch the kind of traffic that goes across the site -- you'll be astonished at how your server get pounded on. Make sure you've done your job of closing off all the holes."

— Terry Sweeney, Special to Dark Reading