Question Block Level Dedupe on BTRFS volumed

[mgrobins]

I would like experienced opinions from those who have used block level deduplication to assess the benefit of running it offline on the DS3616

Having no native dedupe except the block level implemented in Business backup I figure those of us using our NAS for DAS/file serving kind of get shafted.

A delitghful fellow has a system where I can run a container in Docker and operate Dedupe offline in that. Resource intensive and slow...

A high portion of my file volume is video. A high number of files are audio (smaller files) and a reasonable number but small volume of journal articles etc (say 10,000).

FOr block level what would likely work well and what would not show any compression?

How do we go about enticing Synology to offer native BTRFS block level dedupe?

 

[Rusty]

How do we go about enticing Synology to offer native BTRFS block level dedupe?

I would say we are still way off, but there might be something in DSM7 down the line. Sure they would say something about it on the Syno 2020 keynotes, but there was nothing on that topic, so not sure we will get it out the gate any time soon.

Some horsepower is also needed, so again maybe not all models will get support for it even if they support BTRFS.

FOr block level what would likely work well and what would not show any compression?

Almost certain no benefit on the media side (large files with no changes).

 

[mgrobins]

Almost certain no benefit on the media side (large files with no changes).

I disagree. BLOCK level Dedupe looks for common blocks which can easily exist within massive files of different content. FILE level is obviously pointless for video. Use of a file for hash info would offset need for huge RAM and doing task offline would be fine as BTRFS does not have native inline dedupe like ZFS.

I guess as mentioned... nothing in the 2020 Keynote.

I just find it perplexing that they release quite powerful NAS units with only 4-6 drive bays. Dedupe would be a powerful tool to get the most from these without spending on huge HDD's with higher statistical chance of failure at several points.

I guess I'll need to look elsewhere - maybe ZFS forum , but I am attempting to get an idea of how much practical space block level achieves on media. If it's useful I'll set up a docker container and run it through there.

 

[itsjasper]

You'd get hardly any space saving on dedupe for video, even with block level. It just doesn't have the repeatable data.

Dedupe requires more computing horsepower than what you'd typically find in a NAS, as Rusty said.

 

[mgrobins]

Ok thanks for the input .

 

[jeyare]

agree with CPU performance impact
creating and handling of every single block fingerprints is magic only when you will use min. Xeon based CPUs with +8-cores include Turbo at +3GHz.
You can forget for massive dedupe with i3 2C.