Hyperbackup Rotation Question

[SynUser117]

I have a Hyperbakup task setup to backup all of my files and applications and currently have it set to backup daily, I suspect completely overkill for my setup but I wanted to see how it works and all seems fine there. I can certainly change the frequency.

I'm curious about enabling the rotation schedule and how that works, the Synology literature leaves a lot to be desired so I really understand how best to use in my circumstance.

Am I correct in understanding that no matter the schedule chosen the actual files, no matter how old they are and if they never get modified after initially created will never get deleted? I would think that would be the case but just want to make sure that enabling rotation does never delete the actual files themselves.

So my scenario is that the NAS is used primarily as a media server so for things like my CD rips, movie rips, GoPro video and photos. Most of these files one initially created will never be modified again. Sometimes I modify the meta data of the CD rips just to accommodate various media players better. I also have Synology Drive setup on my PC and all family members phones so that files are backed up to the NAS. I'm using Drive not so much for it's team sharing ability but to simply backup the data, I actually have the settings to upload only and to delete NAS file if file on source device is deleted. So this data is data that would be more susceptible to files being modified over time. The NAS is not being used for business, just home use.

The backup task is saving the hbk file to an attached external USB drive. The hbk is currently a bit over 13TB and the external drive is 16TB. I setup the backup task back towards the end of July and it has been running daily for almost the entire time. Was off line for a week while I backed up the hbk file to another USB drive then reformatted the external drive connected to the NAS from NTFS to ext4 so I could properly manage the hbk permissions then the hbk was copied back.

Any suggestions for an appropriate rotation schedule given my use case? If offering suggestions if you wouldn't mind explaining why you think that solution would be ideal that would be appreciated to help me better understand the rotation going on.

Thanks

 

[Rusty]

Am I correct in understanding that no matter the schedule chosen the actual files, no matter how old they are and if they never get modified after initially created will never get deleted?

Short answer is yes. But this is only if you have no versioning/rotations active. As soon as you activate versioning at some point certain version might get deleted. Ofc if there are no changes to the files then those will not get deleted. Also you have the option to "lock" a specific version. In that case, those files/versions will be completely skipped in any clean up.

Any suggestions for an appropriate rotation schedule given my use case?

This might be a bit difficult as you have given an overall and general use case for your NAS. In terms of how often to run the backup it will still be a question just how much changes are happening during the day to a) justify a daily backup and b) to plan for a rotation/versioning.

So you are asking about how often to backup your already backed up data. While backing up your endpoint devices might be justified to be backed up daily, doesn't necessarily mean that should be done on the NAS > USB end.

One final point from my end is that you are having a single HB task that backups everything? If so that is extremly risky imho. If anything happens to that drive or that task (and it can happen), you will have the entire backup hanging in limbo.

Would it be an option to separate and divide that single task to multiple ones? From most to less frequent by change? Or some other way?

This way you will have a few smaller batch processes happening but some will happen more often then others. This way you will also lower the wear and tear on the drives and shorten the backup time overall.

 

[Telos]

As soon as you activate versioning at some point certain version might get deleted. Ofc if there are no changes to the files then those will not get deleted. Also you have the option to "lock" a specific version.

Maybe I misunderstand your post. but this is not my experience. Let's day I have 5 day rotation... so my backups are

Jan 1
Jan 2
Jan 3
Jan 4
Jan 5

Between Jan 1 and Jan 2 I deleted "video.mkv". Looking through Hyper Backup Explorer, that file is present on the Jan 1 backup, but absent from the Jan 2 backup (as expected).

With my next daily backup ("Jan 6") the "Jan 1" backup is removed, leaving Jan 2, Jan 3, Jan 4, Jan 5, and Jan 6. The file "video.mkv" is gone permanently with the deletion of the Jan 1 backup.

Unlike Synology Drive versioning, Hyper Backup retention is basically a snapshot in time (similar to drive imaging). Try it with a small 1 folder/1file sample with successive backup runs.

 

[SynUser117]

Yes, thus far only a single backup task thus the enormous hbk size. As I am new this this whole backup schedule world and in turn rotation schedules there is a lot to learn. I only have the one NAS and although I do have 2 larger external USB drives, a 16TB and 18TB only the 16TB is recognized by my NAS. That's a result of me having an older DS415play which only has a 24TB max capacity and a 16TB max volume limit, even on external drives. I have 4 6TB drives installed and am running SHR so effectively down to 18TB usable internal space and because 18TB exceeds the 16TB volume limit I was forced to setup two volumes. I originally purchased the 18TB drive and was shocked I couldn't use it with the NAS and had to purchase the 2nd so I spent a lot more to begin my endeavor into backups then I had originally anticipated. Currently I have nowhere else to backup my data to.

It hadn't occurred to me to make separate tasks based on the specific data being backed up and there would be nothing stopping me from doing that however it would mean permanently deleing my existing 13+TB hbk file in order to accommodate multiple files which together would equal in size my current file. Also very good point about the hbk file becoming corrupted and there I would be with my backup essentially gone. Also understand that the external drive itself could fail, had that happen to a smaller drive I had for many years.

I have been really wanting to backup my physical movie collection (DVD's, Blu-Ray's and UHD's) but with my NAS at capacity that will require me purchasing an additional NAS and at that point I guess I could backup contents on this NAS to the new NAS. Getting a new NAS would greatly increase my storage capacity getting past the now meager 24TB ceiling I have and if getting say something like a 423+, 923+, 1522+, 1621+, 1821+ or even go big and get like a 2422+, any of those would allow me to go beyond 16TB volume limits up to 108TB I beleive and a single 22TB drive in one of those easily holds the entire contents of my 415play with room to spare not to mention adding additional 22TB drives, so 3 22TB drives to start with SHR would instantly give my 44TB I believe.

 

[Fortran]

I have been really wanting to backup my physical movie collection (DVD's, Blu-Ray's and UHD's) but with my NAS at capacity that will require me purchasing an additional NAS and at that point I guess I could backup contents on this NAS to the new NAS.

As you're new to this; it's worth mentioning that as well as splitting backups into separate tasks, it is also common after a certain point to choose not to backup everything on the NAS. Eg, many people with large ripped movie collections don't back up these rips, the logic being that they take a lot of HDD space and could be re-ripped again if the worst happened, albeit at a considerable cost in time.

The point being, there comes a time where the simplistic 'backup the whole NAS' logic that everyone starts out with becomes less practical, so one starts to make finer judgements about what needs backing up / at what frequency etc. The details of those judgements are obv down to you and your use case, but its worth pointing this general principle out as it's easy to stick with the 'back everything up, all the time' mindset for longer than is helpful.

 

[Rusty]

Maybe I misunderstand your post. but this is not my experience

Well, I said "no change", and you actually deleted a file in the example. That is not what I meant. I was referring to files that exist but had some changes done. Ofc that after the rotation is done the file will be gone following the FIFO method. This is another reason why I said it will depend on the number of versions and rotation schedule.

I think we are actually talking about the same thing, and I agree with what you wrote.

 

[SynUser117]

As you're new to this; it's worth mentioning that as well as splitting backups into separate tasks, it is also common after a certain point to choose not to backup everything on the NAS. Eg, many people with large ripped movie collections don't back up these rips, the logic being that they take a lot of HDD space and could be re-ripped again if the worst happened, albeit at a considerable cost in time.

The point being, there comes a time where the simplistic 'backup the whole NAS' logic that everyone starts out with becomes less practical, so one starts to make finer judgements about what needs backing up / at what frequency etc. The details of those judgements are obv down to you and your use case, but its worth pointing this general principle out as it's easy to stick with the 'back everything up, all the time' mindset for longer than is helpful.

Thanks

That makes sense. For movies and items like my CD rip collection once ripped there is little to no change ever in the data so I see where not having those on a schedule makes perfect sense. Just having a backup task saved and ready to manually execute would suffice to ensure capturing of new items that get added over time. For me for example, after I went through the very time consuming process of ripping my CD collection a number of years ago, ever since then I only periodically purchase new CD's and rip them. So even if a backup task was set to once a month there would be months where no changes would take place in that data as I don't purchase CD's very often. It would be the same with my movie collection when I start that. Having never done movie rips before I suspect each rip takes considerably longer than a CD rip so getting a backup of the data would be important to me. They will take up considerably more storage as well. I currently have appx. 60 UHD discs, appx. 300 BD's and appx 300 DVD's. This doesn't include all of the bonus discs many have so those counts probably double but I suspect the bonus disc storage requires aren't anywhere near the actual movie sizes. I still have to figure out what the best rip format of the movies would be as I suspect there are a number of formats to chose from. I rip CD's to lossless FLAC to preserve quality so I'll attempt to do the same for the movie discs. So possibly ISO format may be the best for those.

 

[Coop777]

As you're new to this; it's worth mentioning that as well as splitting backups into separate tasks, it is also common after a certain point to choose not to backup everything on the NAS. Eg, many people with large ripped movie collections don't back up these rips, the logic being that they take a lot of HDD space and could be re-ripped again if the worst happened, albeit at a considerable cost in time.

The point being, there comes a time where the simplistic 'backup the whole NAS' logic that everyone starts out with becomes less practical, so one starts to make finer judgements about what needs backing up / at what frequency etc. The details of those judgements are obv down to you and your use case, but its worth pointing this general principle out as it's easy to stick with the 'back everything up, all the time' mindset for longer than is helpful.

Regarding the above - the cost of "time" can not be understated, especially with TB's of content.

 

[Fortran]

Regarding the above - the cost of "time" can not be understated, especially with TB's of content.

My 20yo student former self was time-rich and cash poor, ditto my retired parents. My 35yo mid-career self was the inverse. IME the value of one's time is not a constant throughout one's life.

As I said in the following para, everyone's use case is different and one makes such judgements accordingly.

 

[SynUser117]

Thanks for the continued responses.

Out of curiosity I ripped a few of my Blu-rays via MakeMKV using the complete backup option and on average it took appx. an hour with my current drive in my now old laptop. Doing the same for 1 DVD to create the .ISO file completed in about half the time more or less. My drive doesn't read UHD discs so couldn't do a test to get a feel for the time there but with a few hundred DVD's and same for Blu-rays the time to rip everything would be significant, certainly more than it took to rip my few hundred CD's.

Getting back to rotation schedules, for the average user, in a non business environment, would there even be a need to enable rotation? Just thinking the rotation feature seems more business orientated giving the ability to look at and restore older versions of files (while still accessible before being permanently purged via the schedule) that get continually modified over time and/or deleted and business cases may arise where being able to see that older version or previously deleted file are important. Certainly a home user could benefit from the same.

 

[Coop777]

Thanks for the continued responses.

Out of curiosity I ripped a few of my Blu-rays via MakeMKV using the complete backup option and on average it took appx. an hour with my current drive in my now old laptop. Doing the same for 1 DVD to create the .ISO file completed in about half the time more or less. My drive doesn't read UHD discs so couldn't do a test to get a feel for the time there but with a few hundred DVD's and same for Blu-rays the time to rip everything would be significant, certainly more than it took to rip my few hundred CD's.

Getting back to rotation schedules, for the average user, in a non business environment, would there even be a need to enable rotation? Just thinking the rotation feature seems more business orientated giving the ability to look at and restore older versions of files (while still accessible before being permanently purged via the schedule) that get continually modified over time and/or deleted and business cases may arise where being able to see that older version or previously deleted file are important. Certainly a home user could benefit from the same.

"Out of curiosity I ripped a few of my Blu-rays via MakeMKV using the complete backup option and on average it took appx. an hour with my current drive in my now old laptop. Doing the same for 1 DVD to create the .ISO file completed in about half the time more or less. My drive doesn't read UHD discs so couldn't do a test to get a feel for the time there but with a few hundred DVD's and same for Blu-rays the time to rip everything would be significant, certainly more than it took to rip my few hundred CD's" - which makes perfect sense, the larger the size of the disc, the longer this would take....