News
6/19/2012
11:45 AM
George Crump
George Crump
Commentary
50%
50%

Deduplication Performance: More Than Processing Power

Storage performance problems can't be solved by just throwing more processing power at them.

Deduplication, the process of identifying redundant data segments between separate files, is moving its way through the storage infrastructure. The next deduplication targets are primary storage systems and all-flash arrays. As we saw in a recent poll Storage Switzerland conducted, the number one concern is performance impact. You want deduplication, but you don't want it to impact your applications' performance.

Overcoming performance problems is more than just throwing more processing power at the problem. You need intelligent design of the deduplication logic itself. Certainly more processing power helps. As most storage systems upgrade to the latest Intel processors, they will have a new lease on life when it comes to providing all the storage services that you expect from your storage system, like snapshots, replication, and cloning.

Deduplication is a little different though, because there is a database-like lookup that must occur with most deduplication technologies. Anytime there is a lookup, the processor's speed becomes less important because it has to wait on the device that holds the table that performs the lookup. The device's speed becomes more important than processing power.

[ Learn about The Storage Problem Technology Can't Solve. ]

What causes the deduplication lookup? Most deduplication technologies create a table to store information about unique and similar data. As data is sent to a device with deduplication enabled, that data is segmented and then given a hash code. Think of it as a unique serial number. The data is then stored and the serial number or hash code is stored in the table mentioned above. As more data is sent to the device, it is also segmented and given a code. If that code matches a previous code then it is identical data. The table is updated, but the redundant data is not stored. This is where your capacity savings comes from. It is also where your potential performance bottleneck comes from.

As more and more data is stored on the system, that table grows in size. The more unique the data is, the more that table will grow, and, of course, the more total data there is, the more that table will grow. This is important when we talk about deduplication in primary storage. Primary storage does not have near the level of redundancy that backup data does. As a result, the number of entries in the hash table can be dramatically larger than it would be in a backup environment.

Why is this important? It impacts the type of storage device where you can store the lookup table. Most vendors, aware that the speed of the device where the deduplication table is stored is an issue, try to store the entire table in RAM. There is no waiting, but there is a cost and capacity issue

What happens when the number of entries, or the total dataset under management, becomes so large that the deduplication lookup table gets too big to store in RAM? You have to page to a hard disk or SSD, so, essentially, RAM becomes a cache. The problem is that normal cache logic won't work in this situation. This is not a first-in, first-out scenario. You need to verify uniqueness across the entire dataset, not just the most recently stored data.

As we will cover in our upcoming webinar "What is Breaking Deduplication?" if RAM is not used properly, then that means a repeated trip to the hard disk or SSD AND it means that the powerful processor in the storage system has to wait. Its extra processing power--and the money you spent on it--goes to waste.

Follow Storage Switzerland on Twitter

George Crump is lead analyst of Storage Switzerland, an IT analyst firm focused on the storage and virtualization segments. Storage Switzerland's disclosure statement.

Big data places heavy demands on storage infrastructure. In the new, all-digital Big Storage issue of InformationWeek Government, find out how federal agencies must adapt their architectures and policies to optimize it all. Also, we explain why tape storage continues to survive and thrive.

Comment  | 
Print  | 
More Insights
Register for Dark Reading Newsletters
White Papers
Cartoon
Current Issue
Dark Reading December Tech Digest
Experts weigh in on the pros and cons of end-user security training.
Flash Poll
10 Recommendations for Outsourcing Security
10 Recommendations for Outsourcing Security
Enterprises today have a wide range of third-party options to help improve their defenses, including MSSPs, auditing and penetration testing, and DDoS protection. But are there situations in which a service provider might actually increase risk?
Video
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2014-1421
Published: 2014-11-25
mountall 1.54, as used in Ubuntu 14.10, does not properly handle the umask when using the mount utility, which allows local users to bypass intended access restrictions via unspecified vectors.

CVE-2014-3605
Published: 2014-11-25
** REJECT ** DO NOT USE THIS CANDIDATE NUMBER. ConsultIDs: CVE-2014-6407. Reason: This candidate is a reservation duplicate of CVE-2014-6407. Notes: All CVE users should reference CVE-2014-6407 instead of this candidate. All references and descriptions in this candidate have been removed to pre...

CVE-2014-6093
Published: 2014-11-25
Cross-site scripting (XSS) vulnerability in IBM WebSphere Portal 7.0.x before 7.0.0.2 CF29, 8.0.x through 8.0.0.1 CF14, and 8.5.x before 8.5.0 CF02 allows remote authenticated users to inject arbitrary web script or HTML via a crafted URL.

CVE-2014-6196
Published: 2014-11-25
Cross-site scripting (XSS) vulnerability in IBM Web Experience Factory (WEF) 6.1.5 through 8.5.0.1, as used in WebSphere Dashboard Framework (WDF) and Lotus Widget Factory (LWF), allows remote attackers to inject arbitrary web script or HTML by leveraging a Dojo builder error in an unspecified WebSp...

CVE-2014-7247
Published: 2014-11-25
Unspecified vulnerability in JustSystems Ichitaro 2008 through 2011; Ichitaro Government 6, 7, 2008, 2009, and 2010; Ichitaro Pro; Ichitaro Pro 2; Ichitaro 2011 Sou; Ichitaro 2012 Shou; Ichitaro 2013 Gen; and Ichitaro 2014 Tetsu allows remote attackers to execute arbitrary code via a crafted file.

Best of the Web
Dark Reading Radio
Archived Dark Reading Radio
Now that the holiday season is about to begin both online and in stores, will this be yet another season of nonstop gifting to cybercriminals?