News
6/10/2011
03:46 PM
George Crump
George Crump
Commentary
Connect Directly
RSS
E-Mail
50%
50%

Big Data A Big Backup Challenge

Backing up Big Data requires a system that is fast, cost effective, and reliable. These are conflicting terms in the world of storage.

Big Data is, well, big, and size is not the only challenge it places on backup. It also is a backup application's worst nightmare because many Big Data environments consist of millions or even billions of small files. How do you design a backup infrastructure that will support the Big Data realities?

First, examine what data does not have to be backed up at all because it can be easily regenerated from another system that is already being backed up. A good example is report data generated from a database.

Once this data is identified, exclude it. Next, move on to the real problem at hand--unique data that can't be re-created. This is often discrete file data that is feed into the environment via devices or sensors. It is essentially point-in-time data that can't be regenerated. This data is often copied within the Big Data environment so that it can be safely analyzed. As a result, there can be a fair amount of redundancy in the Big Data environment. This is an ideal role for disk backup devices. They are better suited for the small file transfers and, with deduplication, can eliminate redundancy and compress much of the data to optimize backup capacity.

Effective optimization is critical since Big Data environments are measured in the 100's of terabytes and will soon be measured in the dozens of petabytes. It is also important to consider just how far you want to extend disk backup's role in this environment.

Clearly deduplicated disk is needed, but it probably should be used in conjunction with tape--not in replacement of it. Again, often a large section of this data can't be regenerated. Loss of this data is permanent and potentially ruins the Big Data sample. You can't be too careful and, at the same time, you have to control capacity costs so that the value of the decisions that Big Data allows are not overshadowed by the expense of keeping the data that supports them. We suggest a Big Data backup strategy that includes a large tier of optimized backup disk to store the near-term data set for as long as possible, seven to 10 years worth of data being ideal, then using tape for the decades worth of less frequently accessed data.

Alternatively you could go with the suggestion we made in a recent article "Tape's Role in Big Data" and combine the two into a single active archive--essentially a single file system that seamlessly marries all of these media types. This would consist of fast but low capacity (by Big Data standards) primary disk for data ingestion and active analytical processing, optimized disk for more near term data that is not being analyzed at that moment, and tape for long-term storage. In this environment data can be sent to all tiers of storage as it is created or modified so that less or even no backups need to be done.

Big Data is a big storage challenge, not only to store the data but to put it on a fast enough platform that meaningful analytics can be run while at the same time, being cost effective and reliable. These are conflicting terms in the world of storage. Resolving that conflict is going to require a new way of doing things.

Follow Storage Switzerland on Twitter

George Crump is lead analyst of Storage Switzerland, an IT analyst firm focused on the storage and virtualization segments. Storage Switzerland's disclosure statement.

Comment  | 
Print  | 
More Insights
Register for Dark Reading Newsletters
Partner Perspectives
What's This?
In a digital world inundated with advanced security threats, Intel Security seeks to transform how we live and work to keep our information secure. Through hardware and software development, Intel Security delivers robust solutions that integrate security into every layer of every digital device. In combining the security expertise of McAfee with the innovation, performance, and trust of Intel, this vision becomes a reality.

As we rely on technology to enhance our everyday and business life, we must too consider the security of the intellectual property and confidential data that is housed on these devices. As we increase the number of devices we use, we increase the number of gateways and opportunity for security threats. Intel Security takes the “security connected” approach to ensure that every device is secure, and that all security solutions are seamlessly integrated.
Featured Writers
White Papers
Cartoon
Current Issue
Dark Reading's October Tech Digest
Fast data analysis can stymie attacks and strengthen enterprise security. Does your team have the data smarts?
Flash Poll
10 Recommendations for Outsourcing Security
10 Recommendations for Outsourcing Security
Enterprises today have a wide range of third-party options to help improve their defenses, including MSSPs, auditing and penetration testing, and DDoS protection. But are there situations in which a service provider might actually increase risk?
Video
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2013-4594
Published: 2014-10-25
The Payment for Webform module 7.x-1.x before 7.x-1.5 for Drupal does not restrict access by anonymous users, which allows remote anonymous users to use the payment of other anonymous users when submitting a form that requires payment.

CVE-2014-0476
Published: 2014-10-25
The slapper function in chkrootkit before 0.50 does not properly quote file paths, which allows local users to execute arbitrary code via a Trojan horse executable. NOTE: this is only a vulnerability when /tmp is not mounted with the noexec option.

CVE-2014-1927
Published: 2014-10-25
The shell_quote function in python-gnupg 0.3.5 does not properly quote strings, which allows context-dependent attackers to execute arbitrary code via shell metacharacters in unspecified vectors, as demonstrated using "$(" command-substitution sequences, a different vulnerability than CVE-2014-1928....

CVE-2014-1928
Published: 2014-10-25
The shell_quote function in python-gnupg 0.3.5 does not properly escape characters, which allows context-dependent attackers to execute arbitrary code via shell metacharacters in unspecified vectors, as demonstrated using "\" (backslash) characters to form multi-command sequences, a different vulner...

CVE-2014-1929
Published: 2014-10-25
python-gnupg 0.3.5 and 0.3.6 allows context-dependent attackers to have an unspecified impact via vectors related to "option injection through positional arguments." NOTE: this vulnerability exists because of an incomplete fix for CVE-2013-7323.

Best of the Web
Dark Reading Radio
Archived Dark Reading Radio
Follow Dark Reading editors into the field as they talk with noted experts from the security world.