Question Block Level Dedupe on BTRFS volumed

Currently reading
Question Block Level Dedupe on BTRFS volumed

5
1
NAS
3615+ , 1812+
Operating system
  1. Windows
I would like experienced opinions from those who have used block level deduplication to assess the benefit of running it offline on the DS3616

Having no native dedupe except the block level implemented in Business backup I figure those of us using our NAS for DAS/file serving kind of get shafted.

A delitghful fellow has a system where I can run a container in Docker and operate Dedupe offline in that. Resource intensive and slow...

A high portion of my file volume is video. A high number of files are audio (smaller files) and a reasonable number but small volume of journal articles etc (say 10,000).

FOr block level what would likely work well and what would not show any compression?

How do we go about enticing Synology to offer native BTRFS block level dedupe?
 

Rusty

Moderator
NAS Support
2,503
754
www.blackvoid.club
NAS
DS412+, DS718+, DS918+, 2x RS3614RPxs+ with expansions
Router
  1. RT1900ac
  2. RT2600ac
  3. MR2200ac
Operating system
  1. macOS
Mobile operating system
  1. iOS
How do we go about enticing Synology to offer native BTRFS block level dedupe?
I would say we are still way off, but there might be something in DSM7 down the line. Sure they would say something about it on the Syno 2020 keynotes, but there was nothing on that topic, so not sure we will get it out the gate any time soon.

Some horsepower is also needed, so again maybe not all models will get support for it even if they support BTRFS.

FOr block level what would likely work well and what would not show any compression?
Almost certain no benefit on the media side (large files with no changes).
 
5
1
NAS
3615+ , 1812+
Operating system
  1. Windows
Almost certain no benefit on the media side (large files with no changes).

I disagree. BLOCK level Dedupe looks for common blocks which can easily exist within massive files of different content. FILE level is obviously pointless for video. Use of a file for hash info would offset need for huge RAM and doing task offline would be fine as BTRFS does not have native inline dedupe like ZFS.

I guess as mentioned... nothing in the 2020 Keynote.

I just find it perplexing that they release quite powerful NAS units with only 4-6 drive bays. Dedupe would be a powerful tool to get the most from these without spending on huge HDD's with higher statistical chance of failure at several points.

I guess I'll need to look elsewhere - maybe ZFS forum , but I am attempting to get an idea of how much practical space block level achieves on media. If it's useful I'll set up a docker container and run it through there.
 
107
48
NAS
2x DS920+
Router
  1. RT2600ac
  2. MR2200ac
Operating system
  1. Linux
  2. macOS
  3. Windows
  4. other
Mobile operating system
  1. iOS
You'd get hardly any space saving on dedupe for video, even with block level. It just doesn't have the repeatable data.

Dedupe requires more computing horsepower than what you'd typically find in a NAS, as Rusty said.
 

jeyare

Subscriber
1,613
544
agree with CPU performance impact
creating and handling of every single block fingerprints is magic only when you will use min. Xeon based CPUs with +8-cores include Turbo at +3GHz.
You can forget for massive dedupe with i3 2C.
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Welcome to SynoForum.com!

SynoForum.com is an unofficial Synology forum for NAS owners and enthusiasts.

Registration is free, easy and fast!

Trending threads

Top