Flat-File Databases Often Overlooked In Security Schemes

While many enterprises continue to build systems and strategies to protect their relational databases, many organizations haven't done enough to secure the information in the most common database format: the flat-file database, experts say.

It's a dirty little secret of data protection" Most enterprises have sensitive data stored in flat files and floating around the network. Many of these files remain unencrypted and unprotected by the stringent controls that are typically applied to relational databases.

"If you asked a group of IT people, 'Can you think of a flat-file database on your network?,' most people would say, 'Yeah, they're all over the place,'" says Rob Ayoub, global program director of network security for Frost & Sullivan. "But ask, 'How do you protect that data?,' and you might not get an answer.

"We've gotten very good at protecting the critical data within our main databases. But how many times have organizations had employees pull subsets of that data -- without wondering how that data is stored and handled?"

Flat files are a way of life for organizations that need to manipulate data on different systems, migrate data between applications and platforms (think mainframes and COBOL), and even push files across different database platforms, says David Friedland, vice president of business development for Innovative Routines International (IRI).

IRI is an independent software vendor that specializes in data manipulation software -- and, more recently, security products for flat files. "They're just ubiquitous," Friedland says. "Typically, flat files are involved when you're doing large extracts or bulk loads; it's really for very high volume databases and data staging within data warehouses. They call it the 'sequential file stage' in the data transformation world.

"But that's just one way in which flat files are used. Other people just have them around because they feed spreadsheets, or they come out of mainframe applications."

But flat-file data stores are usually less secure than their relational counterparts, experts agree.

"The biggest kind of risk is usually that the implementations are generally not handled well," says Rich Mogull, analyst for Securosis. "Frequently, it will be an application programmer who has just kind of built their own little thing -- as opposed to using an actual managed system -- and they basically have built pretty poor, if any, security around that.

"With a regular database, there's at least some level of security built in. You're, of course, going to have authentication and authorization because those are core elements of the platform. With flat files, you may or may not get that."

Flat files are also a challenge because they move quickly around the network, but can be difficult to locate.

"I think that's one of the real weaknesses of flat files -- they're not transparent, you don't know as much about the data as you do about data that is in another kind of system," says David Stodder, analyst for Perceptive Information Strategies and Ventana Research. "Flat files tend to be the type of thing where organizations will take pieces of it and move it into some other data source -- for example, a data manipulation engine. That in itself is a challenge, just being able to maintain the security as it is traveling around the organization."

Mogull says that as organizations continue their efforts to comply with PCI mandates, many of them are starting to recognize that flat files could be the Achilles' heel of their strategies for protecting personally identifiable information (PII). If the flat file is not part of a managed database solution, then organizations must find ways to encrypt PII in flat files, he advises.

"For sensitive data, you will, of course, need to encrypt that," Mogull says. "If it's not part of your managed database solution, then it's something that you're going to have to go ahead and handle yourself using a crypto toolkit or something else.

"And, boy, I really recommend people be cautious and avoid doing that if they can. Because one of the biggest problems is if someone tries to doing their own crypto -- even using a toolkit -- frequently, they'll screw it up."

This problem is the reason that IRI recently introduced its own field-level encryption solution, FieldShield, which is designed specifically for flat files, Friedland says. However, encryption is only part of the flat-file security solution. Before you can encrypt sensitive information, you'll need to find a way to discover flat files across the infrastructure -- and then find the sensitive information within those files that needs encryption.

"I think the discovery is a huge part of the flat files' dilemma," Ayoub says. "You get a programmer who is working with some old data, dumps the data into a flat file just for testing, and then no one else knows it's out there. There are just huge issues with the tracking and finding of data, and I think discovery is probably an even bigger problem for flat files [than it is for] object-oriented databases." Like several other experts, Ayoub believes organizations should not only implement technological controls to protect data within flat-file databases, but they also need better governance over when, how, and who creates these data dumps.

"In a lot of organizations the policies around data-handling may not be very well defined," Ayoub says. "Some have become more savvy and have set up formal polices around the kind of live data you can dump. But I think it's something that not a lot of people think about."

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message.