DSM 6.2 Hyper Backup lock

Currently reading
DSM 6.2 Hyper Backup lock

40
4
NAS
DS1819+ DS1817+
Operating system
  1. macOS
  2. Windows
Mobile operating system
  1. iOS
Last edited:
Hi.

I have a Hyper Backup rsync copy (single-version) task, which is stuck. By stuck I mean that it shows progress 11% and doesn't move anymore. The remote is online.
Worse ... cancelling the task is the only way out, but this gets stuck as well, meaning Cancelling is shown for one day now.
Worse 2 ... restarting the NAS is not possible since it tells me that I have to Cancel a running Hyper Backup first.

What can I do using DSM?
I don't want to:
- use the root shell to restart (ugly)
- hard-power off the NAS (very ugly)

Any ideas regarding why this happens (not the first time) and how to get out of it?

Thanks
 
It is most probably caused by a network or other connection issue along the line.
It happened to one of my backup tasks.

It takes a lot of time, rollback of a backup is obviously a huge task as the system needs to sort out what was completed, and what not.
My lesson learned: create multiple, smaller backup tasks to reduce risk.
 
A network/connection problem cannot end up like this (lockup), otherwise the Hyper Backup developers failed to implement some catch code for network/connection issues.

When Hyper Backup is locked, there must be a way to kill the task (via DSM).
Also the fact that a Cancel locks up Hyper Backup is a bug.

A backup rollback isn't necessary. Neither my backup data, nor my source data is gone or corrupted. It's "just" that the Hyper Backup (client) locks!

How can I kill the Hyper Backup task without pulling the plug?
 
How can I kill the Hyper Backup task without pulling the plug?
Maybe you can try and log via SSH and use htop to open up "task manager", and try and list HB tasks. Then use kill command for a specific PID and stop the running task
 
A backup rollback isn't necessary. Neither my backup data, nor my source data is gone or corrupted. It's "just" that the Hyper Backup (client) locks!

How can I kill the Hyper Backup task without pulling the plug?
Cannot comment on your data integrity remark.

Pressing cancel leads to heavy work, as rollback is needed, and previous versions should be restored. That is the way it works If you maintain full backup integrity. I agree that this is an extremely inefficient process, in my case it looked frozen for almost 8 h before it finished.
 
Last edited:
Maybe you can try and log via SSH and use htop to open up "task manager", and try and list HB tasks. Then use kill command for a specific PID and stop the running task
Good idea, but I prefer top. Actually ps -e would even be better.
Bash:
root@mynas:~# ps -e | grep -i backup
10319 ?        00:00:11 img_backupd
15077 ?        00:01:06 synobackupd
Hmmm ... is that Hyper Backup?
Can someone (with a working Hyper Backup) check this for me.
I'm stuck with my locked Hyper Backup task and can't to that.
Pressing cancel leads to heavy work, as rollback is needed, and previous versions should be restored. That is the way it works If you maintain full backup integrity. I agree that this is an extremely inefficient process, in my case it looked frozen for almost 8 h before it finished.
Ah ... so you say that pressing Cancel starts a rollback. What do you mean by that? The present backup is cancelled means to me that the backup process is stopped, and that the already backed up data is removed at the destination. Do we agree on that? Besides this, I do not see any network/disk activity after Cancel. Neither on the source NAS, nor the destination NAS.

Which kind of backup did you cancel (I use rsync copy (single-version))?

... later ...

When I start a Hype Backup task from another machine, I see:
Code:
geohei@anothernas:~$ ps -e | grep -i backup
 6223 ?        00:00:00 synobackupd
30939 ?        00:00:00 SYNO.Backup.Con
30969 ?        00:00:26 img_backup
32656 ?        00:00:00 SYNO.Backup.Tas
So I believe img_backup (w/o the d, daemon) is the task stuck on mynas?

Thanks,
 
my experience is if you cancel a backup, it needs a lot of time to recover.
it does not just stop halfway a backup.
A versioned backup will undo all writes to the backup, and roll back the previous version as well, so you end up in exact the same situation as before the backup.

Not exactly sure how cancelling the single version works as it overwrites existing files on the destination. are these left for what it is after a cancel?
 
After lots of testing, I'm a bit further.

Some findings, answers and clarifications.
  • It's not a network issue.
  • It's also no cancel lock issue, like initially supposed by me. It's the backup itself, which locks.
  • The task which resets (by kill command) Hyper Backup is synobackupd.
  • The problem is on the client side, not the server side.
  • It's not a uid:gid issue (tested).
  • The problem is related to permissions.
  • Cancelling a rsync copy (single-version) only deletes the file in progress at destination. It doesn't rollback the entire backup.
Here the point of the filesystem which creates the backup lockup!

Code:
root@mynas:/volume1/Backups/rsync# ls -l
total 0
d---------+ 1 geohei users 34 Jan 4 2021 parents
root@atlas:/volume1/Backups/rsync# ls -l parents/
total 0
drwxr-xr-x 1 root root 32 Mar 16 10:32 child1
drwxrwxr-x 1 root root 46 Feb 5 2021 child2

In Hyper Backup, the parents directory is enabled (checked) to be backed up, hence the files inside the directory, including all sub-directories are supposed to be backed up. This locks up Hyper Backup in my case !!!

When I disable the parents directory (unchecked), but leave child1 and child2 enabled (checked), the backup works correctly.

It seems (?!), that Hyper Backup is unable to search for files inside an ACL directory structure (parents), if that directory doesn't include files and/or sub-directories with ACL permissions.

That's all a big shot in the dark from my side, but what now ... ?!
 
Last edited:
Update on my analysis.

My previous assumptions regarding issue root were wrong, but all can be explained by what I found until now.

The locked Hyper Backup only occurs when the task configuration is as follows:
- rsync copy (single version)
- server type : rsync-compatible server, Synology rsync server works
- Reserve the backed up file at destination is enabled, disabled makes it work

To get the lock, a folder with many (e.g. >150.000) files has to be selected for backup. The lock nevertheless stops after +/- 6 hours (no data transfer at all during that time) and the backup is completed successfully. If you don't want to wait, kill on synobackupd's PID. A subsequent backup with no file change works fine (= takes less than minute). However as soon as 1 file is changed (added or removed), the lock occurs again. For my tests, all files were very small (+/- 5 GB in total for +/- 150.000 files). So it's not the shear mass of data which contributes to the lock.

During the lock, the Performance Utilization shows around 30% on DSM (while the rest of the NAS is +/- idle), while top shows 99% for the synonetbkp task.

For info, my /etc/rsync.conf (required on server side for rsync-compatible server) looks like this:
Code:
[Synology TEST]
path = /volume1/Backups/Test
read only = false
uid = root
gid = root
This setup is not required if Synology rsync server is used.

I reported all this to Synology Support. They were very reactive and managed to replicate today after 2 weeks of testing on both sides. It's under investigation now.

To me, this looks like a bug.
I hope this helps users who run into the the same issue.
 
Update on my analysis.

My previous assumptions regarding issue root were wrong, but all can be explained by what I found until now.

The locked Hyper Backup only occurs when the task configuration is as follows:
  • rsync copy (single version)
  • server type : rsync-compatible server, Synology rsync server works
  • Reserve the backed up file at destination is enabled, disabled makes it work

To get the lock, a folder with many (e.g. >150.000) files has to be selected for backup. The lock nevertheless stops after +/- 6 hours (no data transfer at all during that time) and the backup is completed successfully. If you don't want to wait, kill on synobackupd's PID. A subsequent backup with no file change works fine (= takes less than minute). However as soon as 1 file is changed (added or removed), the lock occurs again. For my tests, all files were very small (+/- 5 GB in total for +/- 150.000 files). So it's not the shear mass of data which contributes to the lock.

During the lock, the Performance Utilization shows around 30% on DSM (while the rest of the NAS is +/- idle), while top shows 99% for the synonetbkp task.

For info, my /etc/rsync.conf (required on server side for rsync-compatible server) looks like this:
Code:
[Synology TEST]
path = /volume1/Backups/Test
read only = false
uid = root
gid = root
This setup is not required if Synology rsync server is used.

I reported all this to Synology Support. They were very reactive and managed to replicate today after 2 weeks of testing on both sides. It's under investigation now.

To me, this looks like a bug.
I hope this helps users who run into the the same issue.
Did you ever get this resolved? I have a HB from teh NAS toe a USB and it freezes the 0.00B/s
 
Did you ever get this resolved? I have a HB from teh NAS toe a USB and it freezes the 0.00B/s

This is what support wrote.
I complied and it worked.

...

Do you mean that if a Synology NAS is used as backup destination, that rsync-compatible server must not be used?

The developers confirmed that the behavior was expected.
In our case, they encourage us to use Synology Rsync Server type when you want to Backup over a Synology Rsync Server

To summarize :
If you perform a Backup to Synology Rsync Server, please Use Synology Rsync Server
HTH
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Similar threads

that is a good question...i do not remember :) maybe when i was experimenting 2-3 years ago i assigned...
Replies
9
Views
522
  • Question
I thought and did the same. I did fresh run and created new backup job...Ongoing...
Replies
7
Views
825
  • Question
HB/HBV updates are tied to the underlying DSM and may not be available unless your DSM is current. For...
Replies
3
Views
941
The documentation in that DSM 7 article is dated. It has evolved over time. There are still issued with...
Replies
1
Views
756
  • Question
I was trying 'entire system' backup now I'm trying again. 1 reinstalled hb on 1621 and hb vault on 1515...
Replies
2
Views
1,235
That option will work ofc and if time is a factor then use this method. USB copy also have several options...
Replies
1
Views
834
I bought some storage in a server to use Hyper Backup to drop my offsite backup in. However, I have not...
Replies
0
Views
1,101

Welcome to SynoForum.com!

SynoForum.com is an unofficial Synology forum for NAS owners and enthusiasts.

Registration is free, easy and fast!

Back
Top