News
5/6/2013
10:28 AM
George Crump
George Crump
Commentary
50%
50%

Active Data Vs. Active Archive

We need better metrics to help us decide what data should be on primary storage and what should be on archive storage.

In my last column I discussed how what we used to consider active data is changing. We now have to look at the potential working set instead of the actual working set. Thanks to initiatives like real-time analytics, some data that we used to classify as archivable now needs to be at the ready. If this is the case, what is the role of archive? How do disk and tape archives participate in an increasingly active world?

The key to a balanced storage strategy, even with all this active data, is to change how we decide to archive a certain set of data. Under the current archive methodology the most common decision point was last modification date. In other words, data that is X days/years old can be archived, everything else has to stay on primary storage. The problem with this methodology is it is not compatible with real-time analytics and not even really compatible with the way users use data.

We need better metrics to help us decide what data should be on primary storage and what should be on archive storage. A key criteria is going to be what data, if it needs to be accessed, will need to be delivered instantly -- in other words, something that may need to be analyzed in the future. This data should probably not go to an archive no matter how old it gets since it could have a statistical probability of value.

[ Learn more about virtual desktop infrastructure. Read VDI Performance And Cost: A Deeper Dive. ]

However, if we know for sure that a certain data set will not be part of a real-time processing application or be needed for analytics then lets archive it as soon as possible and not even wait for it to age. Maybe some of this data could even spend all of its data lifecycle on archive storage because the performance of the archive is "good enough" for the use case.

There is also the need to understand relationships between files. As a simple example, I am writing a couple of books right now. Each of those books have multiple iterations on the file name but large chunks of the content within those files are the same. Each draft gets a different file name. When I get to the end of any of these books, I really don't think I will need all of these drafts but, because all data has become a "you never know" situation, I will want to keep all of them around but I doubt I will ever access them again.

The question is how many of these drafts will I require instant access to and how many could I wait 10 minutes before I view them? For my purposes, all I really will need is the final copy and maybe a couple of the iterations. It would be nice to have software analyze this data and keep versions of the files with the most significant internal changes and then archive the rest.

Interestingly, one of the things we are learning from our primary storage deduplication test is how big of a role this technology can play in these circumstances. Essentially, I can keep all of the files with minimal impact on space utilization. And since they can be disk-based, retrieval time is excellent.

Another classification point is how is that data acted on when recovered? From beginning to end, or at some random point in the file? Basically, can the data be utilized sequentially? If this is the case, then just the front section of that data needs to be stored on primary storage, enough so that it can start being accessed while the back end catches up and the users see no delay in response time. This capability will require a file system intelligent enough to deliver data from two different sources at the same time.

When these attributes of the data are known and understood, then it can be properly placed in the proper types of storage in the data center. Data that whose recovery need is random and unpredictable will need to go on fast storage if analytics are being used. Data that is very similar to other data can be archived or deduplicated.

This archive, depending on what the known recovery need is, can easily be tape based because for a large chunk of the data set how quickly it is recovered is less import than how cost effectively can it be stored.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
dave@qstar
50%
50%
dave@qstar,
User Rank: Apprentice
5/10/2013 | 1:40:58 PM
re: Active Data Vs. Active Archive
Hi George - very interesting article (as always). I am a board member of the Active Archive Alliance and SVP Sales for QStar. My main comments are with this paragraph.

"We need better metrics to help us decide what
data should be on primary storage and what should be on archive storage.
A key criteria is going to be what data, if it needs to be accessed,
will need to be delivered instantly -- in other words, something that
may need to be analyzed in the future. This data should probably not go
to an archive no matter how old it gets since it could have a
statistical probability of value".

This assumes that archive storage is slow and primary storage is fast, which is not necessarily correct. Active Archive solutions can use tape but can also use disk or object storage, which is not slow. Accessibility and instant delivery can be provided by Object Storage solutions as an active archive. The key point is getting data away from the primary storage environment, and the constant backup regime that is associated with it, once data is no longer changing. Archives secure data through copy or replication at the time of ingestion removing the on-going need for backup.

Creating hybrid archives is the answer (using disk-based and tape based technology) to your question, and using "Versioning" to store multiple iterations of a file over time is possible and included in many archive solutions. As you point out versions can be stored without significantly consuming capacity. IF you now your data you can move it (perhaps automatically) to the correct archive technology for long-term preservation.

I agree that metrics can always be improved, currently file metadata is about the only way you can decide which files should be moved to fast archive and which to slower, but for many organizations, that is enough.
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Current Issue
Security Operations and IT Operations: Finding the Path to Collaboration
A wide gulf has emerged between SOC and NOC teams that's keeping both of them from assuring the confidentiality, integrity, and availability of IT systems. Here's how experts think it should be bridged.
Flash Poll
The Dark Reading Security Spending Survey
The Dark Reading Security Spending Survey
Enterprises are spending an unprecedented amount of money on IT security where does it all go? In this survey, Dark Reading polled senior IT management on security budgets and spending plans, and their priorities for the coming year. Download the report and find out what they had to say.
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2017-0290
Published: 2017-05-09
NScript in mpengine in Microsoft Malware Protection Engine with Engine Version before 1.1.13704.0, as used in Windows Defender and other products, allows remote attackers to execute arbitrary code or cause a denial of service (type confusion and application crash) via crafted JavaScript code within ...

CVE-2016-10369
Published: 2017-05-08
unixsocket.c in lxterminal through 0.3.0 insecurely uses /tmp for a socket file, allowing a local user to cause a denial of service (preventing terminal launch), or possibly have other impact (bypassing terminal access control).

CVE-2016-8202
Published: 2017-05-08
A privilege escalation vulnerability in Brocade Fibre Channel SAN products running Brocade Fabric OS (FOS) releases earlier than v7.4.1d and v8.0.1b could allow an authenticated attacker to elevate the privileges of user accounts accessing the system via command line interface. With affected version...

CVE-2016-8209
Published: 2017-05-08
Improper checks for unusual or exceptional conditions in Brocade NetIron 05.8.00 and later releases up to and including 06.1.00, when the Management Module is continuously scanned on port 22, may allow attackers to cause a denial of service (crash and reload) of the management module.

CVE-2017-0890
Published: 2017-05-08
Nextcloud Server before 11.0.3 is vulnerable to an inadequate escaping leading to a XSS vulnerability in the search module. To be exploitable a user has to write or paste malicious content into the search dialogue.

Dark Reading Radio
Archived Dark Reading Radio
In past years, security researchers have discovered ways to hack cars, medical devices, automated teller machines, and many other targets. Dark Reading Executive Editor Kelly Jackson Higgins hosts researcher Samy Kamkar and Levi Gundert, vice president of threat intelligence at Recorded Future, to discuss some of 2016's most unusual and creative hacks by white hats, and what these new vulnerabilities might mean for the coming year.