
Join Now

Does the software do any data de-duplication?

Helpdesk icon
Spiceworks Help Desk

The help desk software for IT. Free.


2 Replies

· · ·
Steven (StorageCraft)
Thai Pepper
Steven (StorageCraft)

Brand Representative for StorageCraft

Hello Robert,

There's a lot going on behind that simple question. Let me answer it by starting with some information to put my answer in context.

There are typically two types of deduplication, namely client-side and server-side deduplication. Client-side (or source) deduplication has the advantage of processing data at the source so that only unique data is sent over the network. This potentially reduces network traffic.

Server-side deduplication processes data at the storage destination. A backup software with server-side deduplication is essentially managing data stored at the destination the same way Windows Server 2012 manages deduplication on a file server. Both are looking for duplicated data patterns that can be reduced to one orginal and one or more pointers to the original.

This is why deduplication has the greatest affect on environments where there is obvious file duplication. For example, if I were backing up 10 Windows Vista PCs, it's reasonable that I could keep the OS files/folders from one of those PCs and keep pointers to that copy for the other 9 PCs.

This is also why deduplication doesn't work as well when data is random with statistically fewer incidents of duplicated data patterns, as is typically the case with image-based backups. An image-based backup collects information at the byte/sector level rather than the file level. This means that image-based backups are typically smaller as they would only copy the words that have changed in an MS Word document and not the entire document each time a backup takes place. It also means that the binary patterns that make up an image-based backup are more random than the patterns of entire files in a file/folder backup.

In short, StorageCraft realizes the importance of reducing long-term storage. This is why the ShadowProtect backup software is an image-based software with very fast and very small backups which are further compressed, collapsed, and consolidated in your backup chain to reduce storage as much as possible. It is possible to further deduplicate ShadowProtect backup images by storing them on a network volume that uses deduplication; however, this deduplication process would rely on the technology managing the storage volume and not on a StorageCraft product.

We are looking into deduplication and how we could effectively incorporate it into our product, but at this time we do not provide any deduplication processes native to StorageCraft's products. We simply want to make sure that when we do dedupe that we do it the right way, in the context of image-based data.


· · ·

Thanks for the comprehensive response. I am progressing with getting a 30 day demo of the program.. Looking forward to the trial...


This discussion has been inactive for over a year.


Oops, something's wrong below.