HyperBackup & Errors at Backup Destination

[revv]

I'm having a problem with my backups via HyperBackup. I don't know who the culprit is, but I thought I'd ask here.

I currently use Hyperbackup to backup my NAS to different destinations. I recently put together a server which is running Proxmox. Inside Proxmox I have an Open Media Vault machine running the zfs plugin. I use HyperBackup to backup my NAS to this server every day via rsync.

Since I put this server together and started backuping to it, I've gotten an error twice. Each time, it results in me having to generate the entire backup from scratch. This time around I'm wondering if there's anything I can do to avoid it.

This is the error I get from HyperBackup:

Backup task was unable to run due to errors found at the backup destination.
The following files were found broken in the latest integrity check and cannot be restored.
There may be other broken files which were not detected this time.
If you have further questions, please contact Synology support.

[Long list of files]

I'm not sure what is happening. I ran a scrub on the zfs pool in Open Media Vault and it returned no repairs. All hard drives are "new" (refurbs from Server Part Deals) and I have no other problem other than this one. HyperBackup is in yellow state and says "Restore Only". All other backups on HyperBackup are going through as expected.

Anyone have any ideas?

 

[revv]

Does anyone have any insight?

 

[Rusty]

Have you tried to open a ticket with synology and provide them with necessary logs? Maybe there are more info inside those that would pin point to an exact problem.

By the looks of it, there are changes happening on the destination side that are affecting this backup process. What those might be it is hard to say as one would have to have similar or identical setup to yours in order to potentially get the same issue.

 

[revv]

They reviewed it and tried to blame it on the use of a RAM stick that is not on their recommended list, which I don't buy because all other backups are going through just fine. I am using ECC RAM. When pressed, they were not able to find evidence in my logs that a bad RAM stick could be causing this.

I can't figure out what could be making changes at the destination. I'm running Proxmox with passed-through hard drives to OMV and only the backup user can read/write to that destination.

 

[Rusty]

Ok considering that unofficial RAM was detected from their end you can forget about Syno support for sure. Anything short of pulling the RAM and then testing and sending new evidence that without 3rd party RAM you have same issues, they will not assist further.

Have you maybe asked on the Proxmox forum? Maybe someone has a similar setup there and negative experience?

 

[revv]

Thanks. Yes, I'm not counting on Synology support for this one. It is sad that they've come to this, but I'm sure those guys sit in the board room and think this hardware upsell strategy is great.

I will ask in PM forums and see what is happening. Right now I'm generating a new backup as I did not find a solution, but I'm starting to wonder if this is related to OMV data scrub. The last thing I did before realizing the backup was broken was a scrub. It might have broken earlier than that, but I didn't realize it, but it is on the list of things to test.

 

[Rusty]

hardware upsell strategy is great

Actually for them it is, but that's not the point of this thread and sarcasm will not solve this problem. No matter how we spin it .

The last thing I did before realizing the backup was broken was a scrub. It might have broken earlier than that, but I didn't realize it, but it is on the list of things to test

That is a smart way to test this, because as much as HB is not a perfect tool, and there are issues from time to time, this could actually be an issue for your particular case.

I'm sure we will all appreciate your time to investigate this as well as any positive or negative (hopefully not) outcome of this problem.

Hope you get to the bottom of it!

 

[Telos]

They reviewed it and tried to blame it on the use of a RAM stick that is not on their recommended list

Reinstall the stock RAM. If problem persists, contact Synology.

 

[revv]

I posted an update in another forum. I thought I'd update here as well:

Updating this thread.

I nuked my OMV and installed Unraid again, created ZFS data pools, and ran Hyper Backup to it. I managed to create a first backup in a day. I then immediately ran an Integrity Check on it and it was found corrupted.
I have a few theories and observations:

1) I have a fairly large collection of photos and videos. Broken files are aways coming back from these photos & videos (phone-taken). This time around, I had fewer files that were broken (just 3 vs the 15-20 that I had with OMV). I find it odd that it is always these files that are broken. None of the hundreds of files I have in other directories (Word, PDF, PPTX, DOCX, etc.). The other constant is that there are always files that were generated by the Synology Photos app present.

These are the ones from today:

Backup task was unable to run due to errors found at the backup destination.
The following files were found broken in the latest integrity check and cannot be restored.
There may be other broken files which were not detected this time.
If you have further questions, please contact Synology support.


Broken file(s):
Version: 2023-10-22 04:23:48-0700, shared folder: homes

  File    30320526     2023-01-22 02:22:10  username/Photos/MobileBackup/Other/Google Photos 2016/@eaDir/VID_20160713_102141297.mp4/SYNOPHOTO_FILM_M.mp4
  File    14029        2023-01-22 02:21:34  username/Photos/MobileBackup/Other/Google Photos 2016/@eaDir/VID_20160713_102141297.mp4/SYNOPHOTO_THUMB_SM.jpg
  File    222404246    2023-01-15 03:33:46  username/Photos/MobileBackup/Other/Google Photos 2016/VID_20160713_102917577.mp4

As a reminder, I have 5 other backups of the same content that are working properly. One inside the NAS, one in an external drive attached to the NAS, one on an offline Rpi with a drive attached to it, one on Google Photos, one on Hetzner. None of these present any issues.

2) This might be related to the use of ZFS? Both OMV and Unraid ran ZFS file systems.

3) Use of non-ECC memory at destination computer. OMV/Unraid run on a cheap N5105 board that does not support ECC memory.

4) Use of unofficial ECC memory on the NAS. I think this is unlikely to be the issue.

II think I'm going to nuke this Unraid install and try Unraid with BTRFS and see if anything changes. If that fails too, I'll have to take the whole thing offline to run a memtest, but at that point I think I might as well just replace the whole thing with something else.

 

[Deleted member 5784]

I don't have a definitive answer for you, but I doubt it is a ZFS issue that corrupts specific file types. This filesystem is used in numerous huge deployments and if there was an issue like this caused by ZFS it would be known about.

When I used HB for backups, I also encountered very frequent errors of the same type you're recieving. This was when backing up to a USB-drive attached directly to the NAS. The backups would run fine, then doing an integrity check produced the errors you're reporting.

I suspected that at some point the multiple version backup tree in HB got too complex and corrupted, but never got to the bottom of things before I ditched the Syno and went with a home-rolled Linux server (using ZFS as it happens) instead. I use Restic to backup on this and it has always been rock solid, including with the very same USB external drive that I was using on the Syno.

TLDR: I think HB is flaky at best.

Addendum: IIRC the files reported by HB as being corrupted are not particularly reliable either. When I ws troubleshooting this, I temporarily deleted the files listed as erroring, re-ran the HB from scratch, and simply got a different list of files that were corrupted. It's as though HB ony lists some of the 'corrupt' files in its error log before giving up.

 

[revv]

Updating.

I tried backing up to a single disk with BTRFS to the same server and got the same problem. So at least I can discard the ZFS theory.

I doubt I have ransomware in here. I've always been very careful with these files and none of the other backups are failing integrity checks.

A backup to the Raspberry Pi I keep mostly offline was successful. It doesn't seem to be related to the NAS or its network.

This leaves me with:

1) Server (destination) memory issues
2) Server (destination) hard drives

I'm going to attach an external hard drive to the server and pass it through to Unraid and try to backup there. If that works, it has to be related to the server's hard drives? If it doesn't, I plan on runing memtest on this thing. If that also works, I guess I'm out of options?

I'm starting to suspect the hard drives themselves, but I don't have any other problems as far as I can tell. I bought four Seagate Exos X16 manufacturer refurbished drives. They all pass S.M.A.R.T.

Addendum: IIRC the files reported by HB as being corrupted are not particularly reliable either. When I ws troubleshooting this, I temporarily deleted the files listed as erroring, re-ran the HB from scratch, and simply got a different list of files that were corrupted. It's as though HB ony lists some of the 'corrupt' files in its error log before giving up.

I agree with you. Every time, the problem is with a different set of files, so it is not the files.

 

[revv]

Updating again.

I tried running a backup to a known and good external USB drive. This backup failed the integrity check. On one hand, this is good because it rules out the other hard drives in the server. On the other, it is bad because I still have no clue as to what is causing these backups to fail while the other backups to different destinations continue to work and pass integrity checks.

I now pulled the server from the rack and am running Memtest86+ on it. I'm also running a memory test on the NAS. If these both pass, I don't really know what else to do other than give up...

 

[Deleted member 5784]

I don't really know what else to do other than give up...

As a test, try using something other than HB to backup the NAS to this server. Eg install restic in a suitable VM on your proxmox server, mount the NAS share(s) in question to this VM using SMB/NFS and run a local restic backup of the problematic files to the internal HDDs on the server.

This will remove HB from the equation, whilkst keeping all the hardware - RAM, HDDs, network etc - consistent. My prediction per my experience above is that this backup will work.

 

[revv]

Well, I think I have finally identified the problem. Bad RAM stick in the Synology NAS. Synology support may have been right all along.

For the record (and for those who Google), as I was running the memory test on the NAS, I noticed it got stuck at 75.xx% and did not move for a while. I went to the unit, rebooted it, and tried again. Again, it got stuck at a different point of the test. I decided to pull the aftermarket memory I had put in there and ran the test again. It passed.

2023-10-28T04:07:13-07:00 Jupiter findhostd[14449]: util_fhost.c:1195 Memtest passed!

While the NAS was doing its memory test, I went ahead and put the suspected faulty stick into another machine that I have here and ran Memtest86+. It crashed Memtest within a few minutes and rebooted the machine. I did this a few more times with the same result, which I believe means that it is a bad stick.

The server (or target computer where HyperBackup was backuping to) is on its 3rd Memtest86+ pass and zero errors. I'm runnining all 10 tests on it (6 hours so far) and I will leave it running overnight for good measure.

I only wish I would have run memtest earlier in the game, but the laziness of pulling the server out of the rack beat me every time. I should have at least suspected the aftermarket ram on the NAS... By the way, the aftermarket RAM is NEMIX 16GB (1X16GB) 1.2V DDR4 2666MHZ PC4-21300 ECC UDIMM 2RX8. I will now do a trial by fire of their RMA process...

 

[revv]

Last post, just confirming that I was able to generate a backup and it also passed the integrity check. So it was the bad RAM on the NAS.

 

[Rusty]

Last post, just confirming that I was able to generate a backup and it also passed the integrity check. So it was the bad RAM on the NAS.

Thank you for sharing and confirming!