Currently on DS918+: my plan for storage expansion & request for comments/constructive feedback

[febox69]

Hey everyone, I’m a beginner to intermediate user of Synology DS918+ so please forgive my ignorance or lack of knowledge in many areas of this whole space of data redundancy, backups, NAS hardware and home networking/cybersecurity - I’m here to ask for help from this knowledgeable community about how to best go about my current situation. Thank you in advance to all of you who make it till the end and provide your valuable feedback, knowledge and experience and to all the trolls - skip this one out please; thanks! Get ready for a long read and without further ado, let’s jump straight in.

Background:

Bought my first Synology DS918+ in 2019 and populated it with 4× WD 12TB Ultrastar DC HC520 SATA HDDs and created one RAID6 Volume on EXT4 filesystem (had no clue back then that BTRFS might be a better idea - didn’t know any better at the time...). Been running regular monthly data scrubs, auto S.M.A.R.T. tests and extended drive tests and thus far there were no errors detected on HDDs or anything else that my DS918+ would complain about in terms of RAID volume or drive health.

The main issue is that I’m currently at 82% (17.8 TB/21.7 TB) of usable capacity on my RAID 6 Pool. I currently still have about 7 TB of data spread across 5 different hard drives and SSDs and would like to move all that to RAID to protect the data from disk failure - some of the drives are 5+ years old and I now have mental bandwidth (and courage to create this Reddit post) and adequate amount of money saved purposely to sort the current situation properly.

My aim:

I had a couple of HDD failures with data loss in the last 15 years (resulting in almost complete loss of data) and getting a Synology DS918+ in 2019 was a game changer as I built my first RAID 6 array and was able to move all of the data I had back then to DS918+. I am aware that DS918+ by itself does not replace backup and I am also aware I have not yet implemented a proper 3-2-1 backup strategy because of the lack of time and finances to do it (will address that later in this post). To accommodate about 7 TB extra data and move them to redundancy storage, here’s my plan:

1. Buy 8-bay Synology DS1821+
2. Buy brand new 5× WD Gold 22TB drives
3. Upgrade stock 4GB RAM with these 2×32GB ECC RAM sticks: https://www.amazon.com/gp/product/B08MT3SLN6/?tag=synoforum-20 (other Reddit posts indicate that these should work well) and run RAM memory test to check everything is ok with RAMs then proceed to inserting new HDDs
4. Insert 4 drives and save 5th drive as a hot spare in case one of the drives fails, leave other 4 bays to populate them later on (once I save up enough money for purchase of the next 4×22TB WD Gold HDDs)
5. Run Synology HDD db script (https://github.com/007revad/Synology_HDD_db) to prevent Synology annoying me with drive incompatibility messages and reduced functionality that comes with that
6. Once Synology HDD db script successfully does its magic and Synology DSM is happy with new drives being fully compatible, then run extended tests on all 4 drives - wait for that to complete without an error on all drives
7. Create SHR2 BTRFS Volume (44 TB usable space) and run extended tests on all 4 drives again - just to be 100% certain
8. Copy DS918+ 17.8 TB Volume to DS1821+ with software that would do file verification of copied data and wouldn’t change file & folder creation dates (I was looking at rclone https://rclone.org/ - any suggestions on which other software with similar reliability and GUI would also do the job properly?)
9. Use the remainder of approx. 26.2 TB on newly created 44TB BTRFS Volume to copy approx. 7TB of data currently spread across different HDDs/SSDs using same or similar method as in previous step
10. I would leave the remainder of approx. 19.2 TB for more data to come (my current estimations based on current data production are 10TB in the next 5-7 years which would also leave me with about 20% of volume space for proper hybrid RAID functioning) and a couple of VMs + Docker containers (to keep this post shorter, I’ll open a separate topic for discussion around that)

Use case - now and plans for the future:

• Current use case:

I use DS918+ as primarily as an archive NAS for critical data (family videos, photos and work-related large video and other data files). One user of DS918+ (no kids or other tech-unsavvy users fiddling around those two PCs), just storage, no bells and whistles (e.g. mail server, VMs, Docker containers etc.) - NAS is merely a device to look after RAID6 and disks. NAS is connected to PC and laptop via Cat6 UTP cabling (via home router) from which files are copied onto NAS via Windows Network Drives feature. Files are uploaded remotely onto PC via FTPS file server, where they are thoroughly scanned with ESET NOD32 Antivirus primarily (I also use Malwarebytes and hashes of potentially dodgy files to be uploaded directly to VirusTotal for screening) and I also use Quad9 as default DNS provider across my networks to reduce risk of any viruses/malware/ransomware. UPS-es are connected consecutively as shown on the diagram below.

• Future use case:

Similar as currently but I would use 8-bay DS1821+ where I currently have DS918+ and I would relocate DS918+ to a different location as per diagram below. After this intro (hopefully clear enough), please see my questions below regarding the proposed setup.

My questions (labelled A to E):

A) Applying 3-2-1 backup rule in practice, here’s what my short- to mid-term plan is:

• Move all my data to the newly created SHR2 BTRFS 44TB Volume spread across 4×22TB WD Golds on DS1821+ which would be my first (primary) copy of data.
• Second copy of crucial data would at this stage remain on the original DS918+; would that be wise given that volume has less than 20% free space? What is the lowest tolerable percentage of free space that Synology does what it’s supposed to do (maintain healthy RAID6) and that parity calculations are done properly? Would very low percentage of remaining space on the volume in any way affect RAID6 parity calculations? I read somewhere on Reddit that 15% is the critical still tolerable space for RAID to function normally.
• Third copy for all my data might be an overkill so I would appreciate community sanity check here: Third copy of data would eventually be created as I get a hold of at least 4×22TB WD Gold HDDs (would aim to squeeze in 1 spare 22TB WD Gold as hot spare) and stick them into the remaining 4 bays in DS1821+; once that would be done, I would create a mirror of Volume 1 (first pool of 4×22TB WD Gold HDDs) to Volume 2 (second pool of 4×22TB WD Gold HDDs). Would that be a complete overkill and if so, what would be reasonable and acceptably more risky alternatives for a second copy of the 44TB Volume 1? By acceptably more risky alternatives I want to acknowledge that any RAID higher than 1 is better than no RAID (no fault tolerance) so I guess simply a huge copy of data to 22TB WD Gold sitting in an external enclosure would do the job. I also acknowledge that there may be a different alternative that I am not aware of and would greatly appreciate community feedback on this.
• I would sell current 4×12TB drives and use that money again towards buying 4×22TB WD Gold HDDs (again, would aim to squeeze in 1 spare) and repeat steps 5 and 6 as above on DS918+ to create a third 44TB SHR2 BTRFS pool. I would then sync DS918+ with DS1821+ mirrored volumes and move DS918+ to a different location with fast FTTH connection for incremental backups (via Tailscale or similar but that’s a story for another time). That would ultimately conclude my 3-2-1 backup plan. Again, all comments and suggestions on alternative and equally sound/”safe” approaches to this would be greatly appreciated.

B) In terms of HDDs - what are the main differences between WD Gold Series and WD Ultrastar Series? Are all the drives in both series CMR drives and helium filled? I was unable to find a proper technical breakdown of differences and similarities between two Series and they seem to be the same (just different sticker and target audience) but for some reason there’s a higher price tag on Golds - does anyone have any insider info on what’s really the main difference (and if so - is it worth the extra $$?) between Gold and Ultrastar Series?

C) What are the potential risks of upgrading RAM from stock Synology 4GB ECC RAM to Non-Synology 2×32GB ECC RAM? Suppose I get the sticks, run a memory test and it turns out ok after a first run. Are there any other risks in the long term in terms of RAM failing or causing data corruption? Is it a case of binary type of outcome - it either works flawlessly or it doesn’t at all or is there a spectrum of working/working up to some point (e.g. up to, say, 48GB used and then there could be some corrupted memory cells on RAM that would be undetected by standard memory test and the machine would reboot amidst some data reading/writing/disk scrubbing/ testing)? I would really appreciate some valuable insights around this - so far, a couple of people upgraded their RAMs in DS1821+ to 64GB and have had no issues thus far but I would like to know more about some theoretical scenarios of what could go wrong so I can do my own risk assessment around whether that upgrade would be worth the money/risk.

D) What would be the optimal BTRFS file system settings for archival/mostly storage purposes (data that are rarely changed so revisions of files would not be frequent)? I don’t want too much overhead (parity/checksum?) data for file healing as I believe that SHR2 pool with regular scrubbing and extended disk tests would take care of that part equally well or I am not understanding the benefits of BTRFS fully? I would appreciate some thoughts around that.

E) The idea behind 2 consecutively/successively connected UPS-es is clearly if one fails the other one takes over (1 had one such situation before, luckily without any noticeable consequences, and I’m a bit paranoid because of that). UPS 1 is connected to the power plug and then UPS2 directly to UPS1 and Synology, both computers, my home router and ISP modem to UPS2. What are the potential flaws of this setup and what can I do to improve power redundancy?

Thanks again for all the valuable feedback peeps, much appreciated in advance!

 

[strikes2k]

Welcome. I'm not going to address everything in your post but just post what I do. I have two Synology's - a DS1813+ that I've had since 2013 and an RS2423rp+ that I got a few months ago. I had 8 TB usable in a RAID 5 configuration in my 1813+ and wanted more space. My RS2423rp+ has 110 usable TB of space, which was moved from the older NAS to the newer one.

It seems data integrity is a large concern for you, as it should be. Both for that reason and to facilitate data migration between my two Synology's, I use Hyperbackup:

1) For backup purposes, I have two external hard drives connected to my NAS and scheduled backups that run weekly for various subsets of the data on the volumes. This ensures that my worst case scenario if I have a full failure would be that my data might be up to 7 days old. I used to keep an offsite copy of a hyperbackup in my desk at work but haven't done that lately. At some point I'll get an offsite copy again but at the moment that's a shortcoming of my backup plan.

2) Hyperbackup also allows you to backup/restore the data between the two NAS systems. You can either back it up to the external hard drives and then restore it to the new NAS, or you can just do it straight across the network between the two NAS systems directly. Here is a link to a Synology article discussing these methods:

How do I migrate data between Synology NAS via Hyper Backup (DSM 6.0 and later)? - Synology Knowledge Center

Regarding upgrading RAM, I upgraded the RAM in both my NAS systems. As long as you get similar spec'ed RAM as what's in there, in a supported size (not larger), and it passes the RAM test you should be good.

 

[Rusty]

Copy DS918+ 17.8 TB Volume to DS1821+ with software that would do file verification of copied data and wouldn’t change file & folder creation dates (I was looking at rclone Rclone - any suggestions on which other software with similar reliability and GUI would also do the job properly?)

Personally, I would recommend using Hyper Backup and do a NAS to NAS single file scenario transfer.

Second copy of crucial data would at this stage remain on the original DS918+; would that be wise given that volume has less than 20% free space?

Try not to push it too much as if you will ever need to rebuild and repair a bit of breathing room will not hurt. Also, DSM will be welcome to have some more space for its internal operations.

I would create a mirror of Volume 1 (first pool of 4×22TB WD Gold HDDs) to Volume 2 (second pool of 4×22TB WD Gold HDDs). Would that be a complete overkill

backup volume on the same NAS is just bad practice not to mention a waste of money for those drives. Would recommend getting another smaller NAS and populating those drives inside it for a proper backup.

people upgraded their RAMs in DS1821+ to 64GB

I would like to know why the need for 32+ GB or RAM (or 32 to start off)?

What would be the optimal BTRFS file system settings for archival/mostly storage purposes (data that are rarely changed so revisions of files would not be frequent)? I don’t want too much overhead (parity/checksum?) data for file healing as I believe that SHR2 pool with regular scrubbing and extended disk tests would take care of that part equally well or I am not understanding the benefits of BTRFS fully?

Essentially you don't want to use EXT4 unless you are running a volume for surveillance alone. BTRFS is a prerequisite for almost all advanced features in the current version of DSM7.2 as well as for most packages inside the Package Center. So in short, just use it and don't think about it.

The idea behind 2 consecutively/successively connected UPS-es is clearly if one fails the other one takes over (1 had one such situation before, luckily without any noticeable consequences, and I’m a bit paranoid because of that). UPS 1 is connected to the power plug and then UPS2 directly to UPS1 and Synology, both computers, my home router and ISP modem to UPS2. What are the potential flaws of this setup and what can I do to improve power redundancy?

Sounds like you have your reasons. Having a UPS redundancy will be better than running on a single UPS that is practically mandatory.

 

[febox69]

Welcome. I'm not going to address everything in your post but just post what I do. I have two Synology's - a DS1813+ that I've had since 2013 and an RS2423rp+ that I got a few months ago. I had 8 TB usable in a RAID 5 configuration in my 1813+ and wanted more space. My RS2423rp+ has 110 usable TB of space, which was moved from the older NAS to the newer one.

It seems data integrity is a large concern for you, as it should be. Both for that reason and to facilitate data migration between my two Synology's, I use Hyperbackup:

1) For backup purposes, I have two external hard drives connected to my NAS and scheduled backups that run weekly for various subsets of the data on the volumes. This ensures that my worst case scenario if I have a full failure would be that my data might be up to 7 days old. I used to keep an offsite copy of a hyperbackup in my desk at work but haven't done that lately. At some point I'll get an offsite copy again but at the moment that's a shortcoming of my backup plan.

2) Hyperbackup also allows you to backup/restore the data between the two NAS systems. You can either back it up to the external hard drives and then restore it to the new NAS, or you can just do it straight across the network between the two NAS systems directly. Here is a link to a Synology article discussing these methods:

How do I migrate data between Synology NAS via Hyper Backup (DSM 6.0 and later)? - Synology Knowledge Center

Regarding upgrading RAM, I upgraded the RAM in both my NAS systems. As long as you get similar spec'ed RAM as what's in there, in a supported size (not larger), and it passes the RAM test you should be good.

Hi strikes2k - thank you for your feedback!

Data integrity is a big deal for me indeed - if I understand correctly, HyperBackup keeps all file and folder original creation and modification dates?

Could you tell me more about your external drive backup setup? What drives (models, capacities) do you use and what type of enclosure do you use?

In terms of RAM upgrade - did you use original Synology RAM sticks or other RAM manufacturer? Should I buy ECC RAM?
[automerge]1690211779[/automerge]

Personally, I would recommend using Hyper Backup and do a NAS to NAS single file scenario transfer.


Try not to push it too much as if you will ever need to rebuild and repair a bit of breathing room will not hurt. Also, DSM will be welcome to have some more space for its internal operations.


backup volume on the same NAS is just bad practice not to mention a waste of money for those drives. Would recommend getting another smaller NAS and populating those drives inside it for a proper backup.


I would like to know why the need for 32+ GB or RAM (or 32 to start off)?


Essentially you don't want to use EXT4 unless you are running a volume for surveillance alone. BTRFS is a prerequisite for almost all advanced features in the current version of DSM7.2 as well as for most packages inside the Package Center. So in short, just use it and don't think about it.


Sounds like you have your reasons. Having a UPS redundancy will be better than running on a single UPS that is practically mandatory.

Hi Rusty, thank you for your comprehensive reply!

Did you perform NAS to NAS volume transfer? Does it retain file and folder creation and modification dates? Does it also do data integrity check like rclone (using SHA256 for example)?

With regards to EXT4 volume space - how low would in your opinion be still acceptable? ≥ 10% or ≥ 15%?

In terms of backup strategy - you would suggest getting two 4-bay Synology NAS-es instead of one 8-bay? Is that in case my 8-bay Synology goes out of business suddenly so risk would be lower if I had two separate 4-bay NAS-es running?

With regard to RAM upgrade - my idea was to run two or three VMs on it so I guess extra RAM would only be beneficial for that and overall snappiness of Synology NAS would also improve?

Regarding EXT4 - what are the major pitfalls of EXT4 I'm currently keeping my data on my DS918+? I understand that BTRFS allows for file healing and file revisions that EXT4 does not offer or are there any other advantages besides the ones you also mentioned?

I'm currently running two UPS-es because I had a big failure of one UPS just recently. Out of the blue, battery just died and luckily no data was lost. Hence my currrent setup - I just don't trust a single UPS configuration.

Thank you for your responses in advance - much appreciated!

 

[Rusty]

Did you perform NAS to NAS volume transfer?

Volume backup is not possible via HB but rather a shared folder backup. Saying this HB 4.1, does include NAS to NAS backup (os and data) so it also could be used for NAS to NAS migration or disaster recovery.

Does it retain file and folder creation and modification dates? Does it also do data integrity check like rclone (using SHA256 for example)?

It does on both counts.

With regards to EXT4 volume space - how low would in your opinion be still acceptable? ≥ 10% or ≥ 15%?

Personal preference would be 20 but that’s me.

In terms of backup strategy - you would suggest getting two 4-bay Synology NAS-es instead of one 8-bay? Is that in case my 8-bay Synology goes out of business suddenly so risk would be lower if I had two separate 4-bay NAS-es running?

That would be my choice but then again if 4bays would not be adequate for you in terms of storage it is understandable to go with a larger model. My personal preference is multiple devices if possible.

With regard to RAM upgrade - my idea was to run two or three VMs on it so I guess extra RAM would only be beneficial for that and overall snappiness of Synology NAS would also improve?

Without sounding like a di*k but unless you are running some RS models having multiple VMs on a NAS, regardless how much ram you have the choke point will be cpu. Even with nvme and ram full on, cpu will get you. Number of cores to start with and ofc the overall performance of the cpu.

One VM fine, but anything over that, especially if it’s some Windows crap, or any full blow UI OS, it will not be what you expect.

RS, SA units, fine, DS ones, maintain your expectations.
Here are some tests on a mid to high range SA unit

Synology SA3400 - Part 3 - virtualization

The third part of the SA3400 series of articles talks about virtualization via VMM and ESXi platforms and contains tests of Win 10 virtual machines as well as 10G network transfers.

www.blackvoid.club

As well as a latest 4bay DS model:

Synology DS923+ review

Brand new #Synology #DS923 is here! A long-awaited 4-bay #NAS comes with an optional #10G, up to 32GB of RAM, and an #AMD Ryzen.

www.blackvoid.club

Like I said, manage your expectations

Regarding EXT4 - what are the major pitfalls of EXT4 I'm currently keeping my data on my DS918+? I understand that BTRFS allows for file healing and file revisions that EXT4 does not offer or are there any other advantages besides the ones you also mentioned?

In short apart from missing on all the security and performance elements as well as packages, keep in mind that ext4 might end up being deprecated at one point like ext3. Sure it will not be soon and it might not be at all, but Synology has only one file system in its sights for the foreseeable future and that’s btrfs.

just don't trust a single UPS configuration

Nothing wrong with that!

 

[strikes2k]

Hi strikes2k - thank you for your feedback!

Data integrity is a big deal for me indeed - if I understand correctly, HyperBackup keeps all file and folder original creation and modification dates?

Could you tell me more about your external drive backup setup? What drives (models, capacities) do you use and what type of enclosure do you use?

In terms of RAM upgrade - did you use original Synology RAM sticks or other RAM manufacturer? Should I buy ECC RAM?
[automerge]1690211779[/automerge]

My external drives are both Western Digital Elements. They come with their own enclosure and power supply. One is 8 TB and the other which I just got a few months ago is 18 TB. They've worked well for backup purposes.

The following details the upgrades I've done to my RS2423rp+, including part numbers for the RAM:

RS2423RP+ compatible devices

So I got an RS2423RP+ to replace/supplement my DS1813+ that is still running strong after almost 10 years. This NAS hit all my check boxes for what I wanted in an upgrade/replacement. Here is what I'm running in the new NAS: A couple of Western Digital 2TB WD Red SN700 NVMe Internal Solid...

www.synoforum.com

My NAS came with ECC RAM so I went with ECC. I don't see any reason not to if the NAS supports it. The cost difference is minimal IIRC.

 

[Coop777]

Is there a reason why you are running SHR-2? SHR-1 gives you more space. And unless you for some reason are not able to be in front of the NAS, a hot spare doesn't make a lot of sense as you are putting needless wear and tear on a drive. A cold spare on the shelf is a better idea.

Did you experience drive failures with the Ultrastars? I have been running on both HGST Deskstars, and now Ultrastars for 8 years and have had no drive failures (knock on wood..)

 

[febox69]

Is there a reason why you are running SHR-2? SHR-1 gives you more space. And unless you for some reason are not able to be in front of the NAS, a hot spare doesn't make a lot of sense as you are putting needless wear and tear on a drive. A cold spare on the shelf is a better idea.

Did you experience drive failures with the Ultrastars? I have been running on both HGST Deskstars, and now Ultrastars for 8 years and have had no drive failures (knock on wood..)

I think SHR-2 would offer me greater peace of mind in case I wouldn't be around NAS if/when disk failure occurs as I would know I have no data loss and reasonable time to act and replace faulty drive without data loss on the four drive array. I plan to have my NAS-es on two different locations away from where I reside so SHR-2 would give me that "psychological" safety - I am well aware that comes at a cost of usable space where I am basically buying two drives for data and two for redundancy purposes. Thank for the suggestion of keeping a cold rather then a hot spare which in combination with SHR-2 really makes a lot more sense.

On that note - I'm a happy camper with my current Ultrastars in DS918+ as with regular S.M.A.R.T., short & extended drive tests and 4+ years of operation, there are so far no signs of failing on any four drives in DS918+. I guess HGST/WD Enterprise drives really live up to expectations and are true "troopers". I also have an "ancient" 2TB WD Raid Edition (one of my current drives that I plan to move data from and retire as soon as I get a new Synology NAS) which has been been working without a glitch for 9+ years now and all S.M.A.R.T. data and regular short & extended drive test look good (knock on wood as well.. ).

In your understanding of WD Gold and Ultrastar Series - do you think they are the same drives under the hood? What do you think accounts for the difference in price (Gold Series come with a higher price tag overall and per TB basis across the board)?

Thank you in advance and thank for your valuable input!
[automerge]1690305685[/automerge]

My external drives are both Western Digital Elements. They come with their own enclosure and power supply. One is 8 TB and the other which I just got a few months ago is 18 TB. They've worked well for backup purposes.

The following details the upgrades I've done to my RS2423rp+, including part numbers for the RAM:

RS2423RP+ compatible devices

So I got an RS2423RP+ to replace/supplement my DS1813+ that is still running strong after almost 10 years. This NAS hit all my check boxes for what I wanted in an upgrade/replacement. Here is what I'm running in the new NAS: A couple of Western Digital 2TB WD Red SN700 NVMe Internal Solid...

www.synoforum.com

My NAS came with ECC RAM so I went with ECC. I don't see any reason not to if the NAS supports it. The cost difference is minimal IIRC.

Thank you strikes2k for an overview of your setup.

Do you think it's better to assemble DIY version of external drive backup - buying the enclosure with power supply and inserting a proper CMR Enterprise drive to ensure that it doesn't die (too soon) or just buy something similar like your WD Elements? I had really bad experience with WD Elements in the past and that was one of the reasons why I also decided to go get Synology and buy Enterprise HDDs.

I guess the Synology won't work if you plug in ECC RAM if not supported by the processor/motherboard?

Your RS2423rp+ with an upgrade looks nice BTW!

 

[febox69]

Volume backup is not possible via HB but rather a shared folder backup. Saying this HB 4.1, does include NAS to NAS backup (os and data) so it also could be used for NAS to NAS migration or disaster recovery.


It does on both counts.


Personal preference would be 20 but that’s me.


That would be my choice but then again if 4bays would not be adequate for you in terms of storage it is understandable to go with a larger model. My personal preference is multiple devices if possible.


Without sounding like a di*k but unless you are running some RS models having multiple VMs on a NAS, regardless how much ram you have the choke point will be cpu. Even with nvme and ram full on, cpu will get you. Number of cores to start with and ofc the overall performance of the cpu.

One VM fine, but anything over that, especially if it’s some Windows crap, or any full blow UI OS, it will not be what you expect.

RS, SA units, fine, DS ones, maintain your expectations.
Here are some tests on a mid to high range SA unit

Synology SA3400 - Part 3 - virtualization

The third part of the SA3400 series of articles talks about virtualization via VMM and ESXi platforms and contains tests of Win 10 virtual machines as well as 10G network transfers.

www.blackvoid.club

As well as a latest 4bay DS model:

Synology DS923+ review

Brand new #Synology #DS923 is here! A long-awaited 4-bay #NAS comes with an optional #10G, up to 32GB of RAM, and an #AMD Ryzen.

www.blackvoid.club

Like I said, manage your expectations


In short apart from missing on all the security and performance elements as well as packages, keep in mind that ext4 might end up being deprecated at one point like ext3. Sure it will not be soon and it might not be at all, but Synology has only one file system in its sights for the foreseeable future and that’s btrfs.


Nothing wrong with that!

Hi Rusty, thank you for your comprehensive reply - really deeply appreciated!

With regard to HB - does it also support incremental backups? If I did for example a volume NAS-to-NAS backup/transfer and then update the original shared folder with some new or modified, would HB allow me to do incremental backups by comparing files and folders and then perform incremental backup if certain conditions are met (e.g. new or modified file on the original shared folder etc.)?

Great to know that HB keeps files & folders as per source and does file integrity checks.

Thank you for your opinion on the lowest acceptable EXT4 available space.

In terms of upgrading existing setup - would you prefer to keep free bays on, say, DS1821+ for future expansion (use 4 bays with HDDs/RAID array running and leave 4 bays to populate later as storage needs grow) or would you consider getting compatible expansion units (an expansion unit for, say, DS918+ that I currently have) and then create a separate self-standing volume on the expansion unit (I read that creating volume across different devices is a very bad idea which makes common sense to me but would like to know if you had any horror stories around such setup in the past)?

Thanks again for your in-depth explanation of the bottlenecks on NAS systems which really seems to be the CPU. Read through your published reviews of virtualization capabilities and I now understand what you mean by "managing my expectations" - thank you for the time and effort to do the testing, really helps in terms of making an informed decision!

I know this may be a topic to open on it's own but could you give me a couple of good points in terms of why EXT4 presents a security problem and could you provide me with some good sources of information on where Synology currently stands with BTRFS and whether BTRFS is "mature enough" to be used as likely default in the future? I read, and again, this may be really old information, that there were some issues with RAID6 and BTRFS in the past (which, again, have most likely been properly addressed and are no longer an issue now) in terms of reliability and data corruption.

In terms of UPS-es I think the same rule applies to them as it does to HDDs - it never hurts to have some extra UPS power on your setup in case sh*t hits the fan!

 

[Rusty]

With regard to HB - does it also support incremental backups? If I did for example a volume NAS-to-NAS backup/transfer and then update the original shared folder with some new or modified, would HB allow me to do incremental backups by comparing files and folders and then perform incremental backup if certain conditions are met (e.g. new or modified file on the original shared folder etc.)?

Yes and yes. This is a default behavior.

In terms of upgrading existing setup - would you prefer to keep free bays on, say, DS1821+ for future expansion (use 4 bays with HDDs/RAID array running and leave 4 bays to populate later as storage needs grow)

This. Definitely expand as needed. Not to mention that it would also mean you would for sure not get the disk from the same lot and that way also mitigate any potential series issues.

would you consider getting compatible expansion units (an expansion unit for, say, DS918+ that I currently have) and then create a separate self-standing volume on the expansion unit (I read that creating volume across different devices is a very bad idea which makes common sense to me but would like to know if you had any horror stories around such setup in the past)?

Personal preference would be hell no in this case. First, the expansion unit can’t function on it own. If something happens to the main unit you stilll can’t easily get to the expansion data. Second they cost almost as a stand alone decide and again, are dependent on the main unit. Stripping the volume is indeed a bad idea as once again something happens to the main unit or expansion you will for sure lose all data on such a stripped volume.

No horror stories as I have always avoided using the expansion

why EXT4 presents a security problem and could you provide me with some good sources of information on where Synology currently stands with BTRFS

The mere fact that DSM features are geared towards btrfs system tells us that Synology is not investing any “smart” or “secure” options on the ext4. Synology KB articles and release notes will give you enough idea what those features and scenarios are.

whether BTRFS is "mature enough" to be used as likely default in the future? I read, and again, this may be really old information, that there were some issues with RAID6 and BTRFS in the past (which, again, have most likely been properly addressed and are no longer an issue now) in terms of reliability and data corruption.

I have several BTRFS raid6 volumes (one with 12 drives) ever since btrfs was introduced and haven’t had any issues with it

 

[Telos]

I read, and again, this may be really old information, that there were some issues with RAID6 and BTRFS

Synology's btrfs implementation is limited/robust and is not exposed to the risks you may have read about btrfs in general.

 

[febox69]

Yes and yes. This is a default behavior.


This. Definitely expand as needed. Not to mention that it would also mean you would for sure not get the disk from the same lot and that way also mitigate any potential series issues.


Personal preference would be hell no in this case. First, the expansion unit can’t function on it own. If something happens to the main unit you stilll can’t easily get to the expansion data. Second they cost almost as a stand alone decide and again, are dependent on the main unit. Stripping the volume is indeed a bad idea as once again something happens to the main unit or expansion you will for sure lose all data on such a stripped volume.

No horror stories as I have always avoided using the expansion


The mere fact that DSM features are geared towards btrfs system tells us that Synology is not investing any “smart” or “secure” options on the ext4. Synology KB articles and release notes will give you enough idea what those features and scenarios are.


I have several BTRFS raid6 volumes (one with 12 drives) ever since btrfs was introduced and haven’t had any issues with it

Perfect, thank you for your answer regarding HB and for strong preference (with sensible arguments) for single-unit storage capacity increase without expansion units (but rather populating NAS bays).

With regard to strategy of HDD acquisition - would you recommend getting same model from two different online stores to mitigate the fact that the drives may be from the same batch and potentially faulty/prone to faults or would only good strategy be getting drives month or two apart to try to get drives manufactured at different times? What tests on Synology NAS would you recommend I do when I get new drives to try to establish whether there are any technical faults with HDDs?

Thank you for pointing me towards resources about BTRFS - DSM release notes weren't really helpful but other Synology KB articles shed some light on what Synology favors and features as better alternative to EXT4. I don't use Surveillance Station so I don't see any reason not to use BTRFS for all my data. Quick question - data scrubbing therefore doesn't actually do anything on EXT4 volumes of am I wrong? My DS918+ reports that data scrubbing has been performed but I am now confused as to what exactly is happening when data scrub is supposedly performed.

I'm really happy to hear from an expert like yourself that BTRFS is safe and preferable file system for Synology NAS - thanks again for sharing your vast knowledge and experience with all of us on the forum!
[automerge]1690313838[/automerge]

Synology's btrfs implementation is limited/robust and is not exposed to the risks you may have read about btrfs in general.

Thank you for that Telos - do you have any useful resources/articles/whitepapers I could read through on how Synology is approaching their implementation of BTRFS?

How long have you been using BRTFS on Synology NAS-es and/or on other Linux workstations and have you had any bad experience with it?

 

[Telos]

Thank you for that Telos - do you have any useful resources/articles/whitepapers I could read through on how Synology is approaching their implementation of BTRFS?

How long have you been using BRTFS on Synology NAS-es and/or on other Linux workstations and have you had any bad experience with it?

It seems you may be over-thinking btrfs. What evidence do you have the Synology's btrfs implementation is less than reliable?

 

[Coop777]

"In your understanding of WD Gold and Ultrastar Series - do you think they are the same drives under the hood? What do you think accounts for the difference in price (Gold Series come with a higher price tag overall and per TB basis across the board)?"

The WD Gold is their manufactured Enterprise drive. The Ultrastar is a driveline acquired from HGST - and to my knowledge no changes were made to it after the acquisition was compete. My preference is the Ultrastar. I do wonder what WD did with the Deskstar NAS driveline but have not been able to figure it out. That had a 3 year warranty but also very stable and more affordable. I have not looked at costs to compare the Gold vs Ultrastar but they should be close?

"With regard to strategy of HDD acquisition - would you recommend getting same model from two different online stores to mitigate the fact that the drives may be from the same batch and potentially faulty/prone to faults or would only good strategy be getting drives month or two apart to try to get drives manufactured at different times? What tests on Synology NAS would you recommend I do when I get new drives to try to establish whether there are any technical faults with HDDs?"

While this could be a concern in some situations, I've not worried about this. I purchase mainly through Amazon and have had good luck. If you end up having to go 3rd party through Amazon, do pay attention to whom the supplier is. You definitely want boxed retail drives, not "bare" or "refurbrished" for configuring disk arrays. I do no testing with new drives as I've not experienced issues (knocks on wood again..).

 

[febox69]

It seems you may be over-thinking btrfs. What evidence do you have the Synology's btrfs implementation is less than reliable?

Fair point - I'm just trying to make the right decision and I learned over time to always work with evidence and data rather than doing "what feels right". I guess at some point you need to trust the authority and smart people like yourself and "outsource" the overthinking part to people who have done their share of research and have experience in filesystems such as BTRFS.

I have my share of bad decisions in setting up backup and data storage strategies - I hope you understand where I'm coming from and that's certainly not to say I doubt your experience and expertise. Thank you for your advice and comments!

 

[Rusty]

With regard to strategy of HDD acquisition

Well in that regard it might be difficult to be certain the lots are different just by buying from a separate vendor or with some time difference. But a combination of the two would almost definitely increase those odds.

What tests on Synology NAS would you recommend I do when I get new drives to try to establish whether there are any technical faults with HDDs?

Let me just say that back in 2011 I got 4 brand new drives and one of them died in the first 24 hours while the array was building. So that’s one way. Another is running extensive tests before making the pool/volume, and finally data transfers. I have a practice when I’m about to use a new device with new drives to test it for about a month before actually starting to utilize it in production. During that time, a good amount of data is transferred repeatedly to stress the volume and drives to see if there are any issues.

While this will not grantee anything it will help to keep my mind at peace about using the setup in the long run.

data scrubbing therefore doesn't actually do anything on EXT4 volumes of am I wrong? My DS918+ reports that data scrubbing has been performed but I am now confused as to what exactly is happening when data scrub is supposedly performed.

data scrubbing on an ext4 volume has the following benefits:

detecting silent data corruption Meaning, scrubbing helps identify and correct errors in stored data that may have been silently corrupted over time. This is crucial for maintaining data integrity and avoiding potential issues.

On top of that, enhancing data reliability. By regularly checking and repairing corrupted data, scrubbing improves the overall reliability of the ext4 volume. It helps prevent data loss and ensures the accuracy of information stored on the NAS.

It will help with identifying failing drives. By monitoring the volume for errors, it allows early detection of drive issues, enabling timely replacement and reducing the risk of data loss.

Finally, improving performance. Scrubb helps optimize system performance by identifying and remediating inconsistencies in data. This ensures that data accessed from the ext4 volume is accurate and reliable, enhancing the overall nas/storage performance.

Btrfs on top of this has several tricks. One would be detecting and repairing data errors. Meaning fixing silent data corruption, ensuring data integrity and reducing the risk of data loss or inconsistencies. Also, recovering and identifying bit rot. Btrfs uses checksums for data and metadata, allowing data scrubbing to detect and recover from where drives silently corrupt data over time. So scrubbing ensures that corrupted data is identified and replaced with the correct information.

 

[Coop777]

Well in that regard it might be difficult to be certain the lots are different just by buying from a separate vendor or with some time difference. But a combination of the two would almost definitely increase those odds.


Let me just say that back in 2011 I got 4 brand new drives and one of them died in the first 24 hours while the array was building. So that’s one way. Another is running extensive tests before making the pool/volume, and finally data transfers. I have a practice when I’m about to use a new device with new drives to test it for about a month before actually starting to utilize it in production. During that time, a good amount of data is transferred repeatedly to stress the volume and drives to see if there are any issues.

While this will not grantee anything it will help to keep my mind at peace about using the setup in the long run.


data scrubbing on an ext4 volume has the following benefits:

detecting silent data corruption Meaning, scrubbing helps identify and correct errors in stored data that may have been silently corrupted over time. This is crucial for maintaining data integrity and avoiding potential issues.

On top of that, enhancing data reliability. By regularly checking and repairing corrupted data, scrubbing improves the overall reliability of the ext4 volume. It helps prevent data loss and ensures the accuracy of information stored on the NAS.

It will help with identifying failing drives. By monitoring the volume for errors, it allows early detection of drive issues, enabling timely replacement and reducing the risk of data loss.

Finally, improving performance. Scrubb helps optimize system performance by identifying and remediating inconsistencies in data. This ensures that data accessed from the ext4 volume is accurate and reliable, enhancing the overall nas/storage performance.

Btrfs on top of this has several tricks. One would be detecting and repairing data errors. Meaning fixing silent data corruption, ensuring data integrity and reducing the risk of data loss or inconsistencies. Also, recovering and identifying bit rot. Btrfs uses checksums for data and metadata, allowing data scrubbing to detect and recover from where drives silently corrupt data over time. So scrubbing ensures that corrupted data is identified and replaced with the correct information.

agree. run scrubbing once a month for both my production and backup boxes. BTFRS is the way to go.

 

[febox69]

"In your understanding of WD Gold and Ultrastar Series - do you think they are the same drives under the hood? What do you think accounts for the difference in price (Gold Series come with a higher price tag overall and per TB basis across the board)?"

The WD Gold is their manufactured Enterprise drive. The Ultrastar is a driveline acquired from HGST - and to my knowledge no changes were made to it after the acquisition was compete. My preference is the Ultrastar. I do wonder what WD did with the Deskstar NAS driveline but have not been able to figure it out. That had a 3 year warranty but also very stable and more affordable. I have not looked at costs to compare the Gold vs Ultrastar but they should be close?

Based on basically specs (comparing WD Gold and Ultrastar 22TB) one would argue that WD Gold is Ultrastar in disguise and WD put a heftier price tag on it (because Gold sounds better than Ultrastar in marketing terms I guess)? They both have 5 year warranty and are CMR drives which I guess are major ticks on the list in terms of durability and suitability for long term NAS operation. Adding SHR2 on top of that I guess I should sleep well knowing my data won't be lost (easily) because of drive failure.

 

[Telos]

would you recommend getting same model from two different online stores to mitigate the fact that the drives may be from the same batch and potentially faulty/prone to faults

Are you prescient? Why would drives from different lots be less likely to include a bad drive? Do you have statistics to back that thinking? Do you suppose that manufacturer QC is somehow slipshod?

Hitachi, Seagate, Synology, Western Digital.... it does not matter. Choose a quality seller , and place an order. It's that simple. Will you get a faulty drive? Maybe. If so, return it.

 

[febox69]

agree. run scrubbing once a month for both my production and backup boxes. BTFRS is the way to go.

Thank you for you input Coop777 - I'm running scrubbing once a month as well and I'll move from EXT4 to BRTFS as soon as I have new Synology NAS with drives.
[automerge]1690358053[/automerge]

Are you prescient? Why would drives from different lots be less likely to include a bad drive? Do you have statistics to back that thinking? Do you suppose that manufacturer QC is somehow slipshod?

Hitachi, Seagate, Synology, Western Digital.... it does not matter. Choose a quality seller , and place an order. It's that simple. Will you get a faulty drive? Maybe. If so, return it.

Thank you for your thoughts on this - would in my case looking at WD Distributors page (applicable to my area) be a good way of approaching buying drives from reputable seller: Distributors | Western Digital
[automerge]1690358383[/automerge]

Well in that regard it might be difficult to be certain the lots are different just by buying from a separate vendor or with some time difference. But a combination of the two would almost definitely increase those odds.


Let me just say that back in 2011 I got 4 brand new drives and one of them died in the first 24 hours while the array was building. So that’s one way. Another is running extensive tests before making the pool/volume, and finally data transfers. I have a practice when I’m about to use a new device with new drives to test it for about a month before actually starting to utilize it in production. During that time, a good amount of data is transferred repeatedly to stress the volume and drives to see if there are any issues.

While this will not grantee anything it will help to keep my mind at peace about using the setup in the long run.


data scrubbing on an ext4 volume has the following benefits:

detecting silent data corruption Meaning, scrubbing helps identify and correct errors in stored data that may have been silently corrupted over time. This is crucial for maintaining data integrity and avoiding potential issues.

On top of that, enhancing data reliability. By regularly checking and repairing corrupted data, scrubbing improves the overall reliability of the ext4 volume. It helps prevent data loss and ensures the accuracy of information stored on the NAS.

It will help with identifying failing drives. By monitoring the volume for errors, it allows early detection of drive issues, enabling timely replacement and reducing the risk of data loss.

Finally, improving performance. Scrubb helps optimize system performance by identifying and remediating inconsistencies in data. This ensures that data accessed from the ext4 volume is accurate and reliable, enhancing the overall nas/storage performance.

Btrfs on top of this has several tricks. One would be detecting and repairing data errors. Meaning fixing silent data corruption, ensuring data integrity and reducing the risk of data loss or inconsistencies. Also, recovering and identifying bit rot. Btrfs uses checksums for data and metadata, allowing data scrubbing to detect and recover from where drives silently corrupt data over time. So scrubbing ensures that corrupted data is identified and replaced with the correct information.

Thanks again Rusty for your comprehensive reply - a lot of valuable information here, much appreciated!

How do you do testing of drives - do you keep running extended disk tests for a month or populating drives/array with actual data (e.g. moving a backed up copy of large amount of data to new array)?

Thank you for in-depth explanation of benefits of scrubbing EXT4 drives/array - so major difference between EXT4 and BTRFS data scrubbing is that in case of EXT4, data scrubbing cannot detect bit rot but data scrubbing on BTRFS can?