Active Backup for Google Workspace : NAS unresponsive although backup stalled

[Romano2K]

Hi!

I'm a long time Synology user at home, but I'm just starting to use Synology in a work environment. As the IT manager of a small company (30 collabs), I'm managing a RS1219+ with 8 x 4 TB drives in RAID 5 (unfortunately not SHR) and Btrfs.

Until now it was serving only as a shared storage server for 4 people.

Now I want to backup our Google Workspace using Active Backup.

I started the initial backup last saturday morning. I was off, but on monday morning it wasn't finished. One of my colleagues, who found the NAS very unresponsive over both SMB and the DSM web interface, and who wasn't aware of the backup process, forced a reboot using the power button.

The backup restarted at 4am last night, but the NAS is once again very unresponsive, while the "Backup status" has been showing "Backing up... 85.71%" for hours (stuck on that exact percentage), even after reconnecting to the DSM web interface.

I've tried to understand the problem with the Resource monitor, but I don't understand where the bottleneck is.
Memory usage is at 84% (I think it's the base RS1219+, so only 2GB of RAM?!), but CPU "utilization" is less than 5%, while I/O Wait is more than 90% and load average above 20! It downloads quite slowly, and in small peaks.

What's wrong? What should I do? Thank you in advance!

 

[Rusty]

while I/O Wait is more than 90% and load average above 20

Welcome to the forum!

Well there are several problems here, and you have zeroed in on the one that explains why you are stuck on a certain %. With a high volume utilization it is no wonder that it is slow, but also with only 2GB of ram there is also a lot of swapping going on especially with a large number of file (such as a backup from cloud).

I would suggest to wait and let it do its thing. RAM upgrade would be a great addition, but also 8x4 in RAID5 is quite risky I have to say.

 

[Romano2K]

Thank you Rusty!

After one more night, it went from around 85.71% to 87.50%. At a rate of 2% every 8 hour, it would take 6 more nights.

But I'm not confortable with the NAS being so unresponsive, so I've already asked the colleague who uses most of the storage to make room, and I'm going to upgrade the server (RAM and M.2 drives). I hadn't noticed that it had only 2GB of RAM.

This is a different matter so I'm taking the liberty of creating a new topic.
Edit: RAM and M.2 SSD upgrades for RS1219+

In the meantime, what's wrong with 8x4TB in RAID5?

 

[Rusty]

In the meantime, what's wrong with 8x4TB in RAID5?

It's nothing wrong per say but with 8 drives and only one drive as a redundant one, a lot of data is riding on a single drive failure. Rebuild of such a large array could result in another drive dying mid rebuild, and then data is lost.

Ofc backup will replace the lost data but business continuity is essentially hanging on a thread on a single drive failing, or not.

RAID 6 with hot spare would be my choice (or maybe just RAID6), but that is just me.

 

[Romano2K]

I was wondering if you were talking about a performance issue.

I agree with you regarding the lack of redundancy. (It was not me who did the initial configuration of this NAS.)
Is there a way to switch from RAID5 to RAID6 or even SHR-2? And upgrade storage space?

 

[Rusty]

Is there a way to switch from RAID5 to RAID6 or even SHR-2?

RAID5 can only go to RAID6 not SHR but you will need to have an extra drive added, and you can't as you are full atm.

Change the RAID Type of a Storage Pool | DSM - Synology Knowledge Center

Synology Knowledge Center provides you with answers to frequently asked questions, troubleshooting steps, software tutorials, and all the technical documentation you may need.

kb.synology.com

 

[Romano2K]

So my only option would be to restart from scratch and restore a backup?

 

[7Natives]

Hi!

I'm a long time Synology user at home, but I'm just starting to use Synology in a work environment. As the IT manager of a small company (30 collabs), I'm managing a RS1219+ with 8 x 4 TB drives in RAID 5 (unfortunately not SHR) and Btrfs.

Until now it was serving only as a shared storage server for 4 people.

Now I want to backup our Google Workspace using Active Backup.

I started the initial backup last saturday morning. I was off, but on monday morning it wasn't finished. One of my colleagues, who found the NAS very unresponsive over both SMB and the DSM web interface, and who wasn't aware of the backup process, forced a reboot using the power button.

The backup restarted at 4am last night, but the NAS is once again very unresponsive, while the "Backup status" has been showing "Backing up... 85.71%" for hours (stuck on that exact percentage), even after reconnecting to the DSM web interface.

I've tried to understand the problem with the Resource monitor, but I don't understand where the bottleneck is.
Memory usage is at 84% (I think it's the base RS1219+, so only 2GB of RAM?!), but CPU "utilization" is less than 5%, while I/O Wait is more than 90% and load average above 20! It downloads quite slowly, and in small peaks.

What's wrong? What should I do? Thank you in advance!

In response to your original post, the NAS is most probably indexing the new workload and this issue should resolve once the indexing is done. Large-scale or first-time indexing is a very time consuming process, even on SSDs, and worse on HDDs.

It looks very much like your company is using one of these Amazon setups. Assuming it has HDD's, adding the optional M.2 SSD adapter card and dual SSD cache drives plus increasing your DDR3 memory to 16GB will significantly improve the indexing performance and reduce the amount of time this sort of thing takes. As an alternative, you could stop the ABB task (or even delete it), re-work the ABB task around your work schedule to run in off-hours and take full advantage of down days (weekends?). The indexing will take more calendar time, but you should avoid the degraded operation performance during normal day-to-day operations. In any case, this will most probably be a very long process. Rest assured, however, once it's done ABB should be quite speedy.

Our first time out with this same sort of task with an all SSD array and maxed RAM, but no cache took something on the order of a couple of weeks (if memory serves me). The initially degraded performance forced us to delete and start over with a better-tuned Task. After that, overall the ABB task took longer to fully complete but we had no degradation of performance during the workdays. For us a cache would not have helped overall performance once indexing was done due to other bottlenecks in our sites — again due to all SSD arrays. YMMV due to HDDs, I would certainly think so.

Also, you might want to loop-in your co-worker. An orderly shut down & reboot works fine, but indexing doesn't just pick right back up and get after it when the machine comes back on; it stalls and coughs and sputters awhile making things — well — worse.

If you do decide to upgrade the RAM and/or fill out the M.2 cache, be sure to use compatible hardware so you don't run into more unexpected issues. All of the peculiarities of the RS1219+, including compatible M.2 and RAM can be found at Synology.com here.

 

[Romano2K]

Thank you for your detailed answer.

I asked my co-worker to free up some storage space, he went from 20% to 50% free space.
I have restarted the Active Backup task at midnight, and its much more snappier: DSM is responsive and Resource Monitor shows lower I/O Wait (around 50%), lower load average (around 15) and lower RAM usage. I'm fairly confident it will be done in a few hours, before business hours.

The backup task is set for a daily backup at 4am. I will see how fast those daily backups will be.


Regarding upgrades, I have indeed checked the official Synology Compatibility List, as you can see in the dedicated thread:

RAM and M.2 SSD upgrades for RS1219+

Hi again! I would like to upgrade the Synology RS1219+ of the small company I'm working for. Until now it was serving only as a storage server for 4 people. But I'm starting to use resource intensive applications such as Active Backup for Google Workspace. Regarding RAM, while we're at it, I...

www.synoforum.com

The list lists old SSD models which aren't available anymore. I like Crucial and it lists the MX300. What would you do in this situation?

 

[7Natives]

Thank you for your detailed answer.

I asked my co-worker to free up some storage space, he went from 20% to 50% free space.
I have restarted the Active Backup task at midnight, and its much more snappier: DSM is responsive and Resource Monitor shows lower I/O Wait (around 50%), lower load average (around 15) and lower RAM usage. I'm fairly confident it will be done in a few hours, before business hours.

The backup task is set for a daily backup at 4am. I will see how fast those daily backups will be.


Regarding upgrades, I have indeed checked the official Synology Compatibility List, as you can see in the dedicated thread:

RAM and M.2 SSD upgrades for RS1219+

Hi again! I would like to upgrade the Synology RS1219+ of the small company I'm working for. Until now it was serving only as a storage server for 4 people. But I'm starting to use resource intensive applications such as Active Backup for Google Workspace. Regarding RAM, while we're at it, I...

www.synoforum.com

The list lists old SSD models which aren't available anymore. I like Crucial and it lists the MX300. What would you do in this situation?

I kitted out a duplicate DS920+ NAS for a remote site and used incompatible drives on Btrfs in SHR2 expecting them to work just fine.

F-A-I-L

In < 90-days two of the drives failed, 600 miles away from me. During replacement as each new drive came fully online, another of the existing incompatible drives failed. By the end 100% of the brand new Samsung 870 EVO SSDs had failed, but were successfully replaced with WD Red drives off the compatibility list W/ aid from Synology's very helpful tech support. Be advised: Even though these were "on" the compatibility list, their FW revision was newer — which was a small risk. (Next time, we'll buy Synology's drives because their value-add pencils for us.)

My answer is necessarily prejudiced by this uncomfortable event. I would (1) update the compatibility list on the NAS to ensure you're looking at the latest one, then (2) phone Synology to see what they say. Be forewarned, their mantra may be, "buy our drives". They're expensive so if that presents as an alternative you'll want to get the full low down on their value-add. It's actually quite impressive and does bring them a bit closer to a realistic price point for what's needed and what you get.

I see on your other thread you went with Crucial for the SODIMMs. Keep us posted there.

 

[Telos]

Samsung 870 EVO SSDs

These have a reputation for early failure on all devices... unrelated to the compatibility list.

 

[Rusty]

So my only option would be to restart from scratch and restore a backup?

For that particular case, yes.

 

[7Natives]

These have a reputation for early failure on all devices... unrelated to the compatibility list.

At the time I installed those they did not have said reputation; the 860's were riding at the peak of their rep and the 870's appeared to be running the same. My point was that had I checked —and abided by— the compatibility list I most likely would have avoided my very, very obvious failure. It was a penny-wise, pound-foolish short sighted mistake on my part.

 

[Telos]

Notably, Syniology has not added the 870s to the Incompatible Models list. For now, it's best to stay with Samsung EVO 860s or WD Reds.

 

[Deleted member 5784]

Notably, Syniology has not added the 870s to the Incompatible Models list. For now, it's best to stay with Samsung EVO 860s or WD Reds.

Reports re the 870EVO failures seem to be coalescing around production dates from early to mid 2021 for the problem models. It is suspected that a batch of contaminated NAND is the culprit, though Samsung has said nothing about the whole saga publically which hasn't helped matters.
I am running a pair of 2TB 870EVOs from Q1 or Q2 2022 in a server and have had no issues so far fwiw.

As you said earlier @Telos, this issue has nothing to do with Synology's 'compatibility' list, which imo is little more than cynical and intentional FUD put out to push nervous users into buying overpriced Synology drives. Which ofc are simply drives made by Toshiba / WD etc in any case. The fact that Synology went out of their way to amend and misreport the SMART data of drives other than their own in order to create the FUD is surely proof enough of their user-hostile intentions in this regard.

It would be a mistake imo to take the EVO870 episode as evidence in favour of buying Synology-approved drives only in teh future; as these are made by the same 3rd party manufacturers, they're just as likely to suffer form things like bad NAND / bad fw / bad controllers as any other brands of drives.

 

[Ames]

I was just looking at 8 TB EVO 870. I really need the 8 TB. Doesn't look like I have any alternative. So are these going to throw the fake warnings on RS820+ or can I check that anywhere?

 

[Deleted member 5784]

They dont make an 870 EVO in 8TB size; you'd need the 870 QVO if you want 8TB.
This isn't on the incompatible list for the RS820+, so you should be ok: https://www.synology.com/en-uk/comp...im&not_recommend_mode=true&p=1&change_log_p=1

You may get a 'not oem' type nag in DSM which can be ignored. The level of nagging seems to vary from model to model.