Your Data Is Gone, Have A Nice Day

A client called our office the other day, needing help with an emergency data restoration from the backup system it manages in-house. The main server had crashed and was unrecoverable. It happens.

The client's IT manager, who I'll call "Bob" for the sake of confidentiality, had used a reputable, commercial-grade off-site backup service. Bob told us the backup service had a courier deliver a hard drive containing the server's backup.

Problem 1: After reloading the operating system and server-based applications, the applications were unable to recognize their backup data for reload.

Problem 2: There were no local data backups. The off-site backup -- the unrecognizable one -- was also the only backup.

Problem 3: Despite strong advice to do so, the in-house IT staff had never performed a full-scale test of the backup-and-recovery system. There was no benchmark to suggest a recovery had ever even been possible.

Problem 4: By the time we were called, companywide email had been offline for five days. Employees were resorting to their personal email systems for business correspondence, creating a compliance, security, and reputation nightmare.

Problem 5: As time passed, the client grew understandably concerned about the time and money being spent attempting a data recovery that might not be possible. At what point do you accept the loss and start over your systems without restored data, even though the cost of doing so is also unknowable?

By now the company was facing an operational meltdown. Restoring the company's email became top priority, as in, "Just get it running, and we'll clean up later." With no documented, tested process for rebuilding the server, the approach could have best been described as "flying by the seat of the pants on fire," loading the applications first to restore the data, and worrying about updates and patches later. The data remained unrecovered.

Fortunately, we were able to diagnose the problem and help this story end on a happier note. The unpatched applications did not recognize backup data created by up-to-date, fully patched systems. The well-intentioned bypass of running updates before data restoration created a false, and actually very dangerous, belief that the data backups were corrupted. Once we updated the server applications, the backup data promptly and correctly loaded without any data loss.

Documented, tested methodology is essential for proper compliance and security. It's also the difference in a problem that can be solved in five hours and one that lasts for five anxiety-filled days. They are too often ignored, but effective compliance and security efforts are usually far less expensive than the disasters they can help prevent.

Glenn S. Phillips wants you to protect your data. He is the president of Forte' Incorporated where he works with business leaders who want to leverage technology and understand the often hidden risks awaiting them. Glenn is the author of the book Nerd-to-English and you can find him on twitter at @NerdToEnglish.