Big Data Detectives

Could big data be the key to identifying sophisticated threats? Security experts are on the case.

Waiting For Maturity

While big data analysis holds promise for security, a number of factors have slowed its adoption. First, most enterprises don't have a line item in the budget for big data security projects. "Big data is about solving business problems, and security is generally, in the beginning, not one of those business problems," says Hadi Nahari, chief security architect for graphics chipmaker Nvidia. Some companies are also concerned that big data projects might introduce risk by forcing changes to the way security systems collect and report data, he notes.

Another major obstacle is the shortage of experts with the skills to mine large security databases for information. In addition to having the abilities of a data scientist, any big data security project leader also needs security expertise and a focus on usability, says Teradata's Harris.

The lack of skilled personnel was the third most significant barrier to a strong security posture among enterprises, according to the Ponemon Institute's "Big Data Analytics In Cyber Defense" report, commissioned by Teradata.

The top two barriers, according to the report, were a lack of effective security technology and an insufficient view into business processes -- chosen by 43% and 42% of respondents, respectively. During its RSA 2012 presentation, Zions Bancorporation introduced a team of three employees, including a data scientist, who created and run the company's big data project. But most companies can't afford to hire so many people for a big data security project.

Protecting Big Data
Using big data could be a boon to security, but enterprises should not forget about protecting the big data itself.

Because big data can be a complete record of a business's operations, it's important to lock it down, says Erik Jarlstrom, VP of technology solutions at Dataguise. Companies need to secure big data stores early to avoid delaying the project.

Big data resides in highly distributed clusters of computers, so securing the entire systems is a challenge, according to Adrian Lane, CTO of security consultancy Securosis, which recently released a research paper on big data security. Because data is distributed among the nodes and distributed in multiple copies, it's difficult to know where your data resides. In most cases, there is no generally available encryption for repositories, and no role-based administrative controls.

Lane advises that companies should use the Kerberos protocol to authenticate big data nodes and add file encryption. "We hear [from security architects] the most popular security model is to just hide the entire cluster within their infrastructure," Lane writes. "But those repositories are now Web accessible and very attractive targets."

Another hurdle to using big data in security is the relative immaturity of the market. While a number of security products now tout some tie-in with big data analytics, they require a great deal of expertise to use and maintain. "Big data has been around for a while, but it's only in its second generation," Securosis's Lane says. "It's not ready for prime time for many companies."

The easiest way for a company to get started in analyzing its security data is to buy a large server and start collecting information, says Vigilant's Magee. Many Vigilant clients are considering buying a large 32- or 64-CPU server and a fast data store, and some of them work with business teams that are already familiar with Hadoop.

"We can leverage Moore's Law to get out in front of this problem. We can start putting data into it and analyze it," Magee says. "While that may seem like a very simple or mundane version of SIEM, companies want that ability. They want to ask questions of their data."

For small and midsize businesses that don't have the resources to start up their own big data project, the only likely solution is to settle for services that incorporate external feeds and security analytics, says Jon Oltsik, senior principal analyst with the Enterprise Strategy Group. While big data analytics can be more effective than SIEM, it isn't easy to incorporate into a business.

"Easy is the key word," Oltsik says. "Big data is too complex and too costly for most midsize businesses, so the question is who can deliver the intelligence of big data at a lower cost than doing it themselves. For most smaller companies, that will be a service provider."

chart: When will use big data analytics for cyber defense?