Backing up large data.... (and also scheduling)

Currently reading
Backing up large data.... (and also scheduling)

135
51
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
Okay a question guys. At the moment I have two Hyper Backup tasks, one backing up to a local 8Tb Seagate backup drive and a second backing up to Synology C2. On the 8Tb backup job I backup most of the content on the NAS, including packages etc. The C2 backup (encrypted) is used to backup less content due to the 1Tb content limit and also the poor broadband I'm suffering - my upload speed is around 7-8 megs.

So to the questions:

1) I'm looking to add my ebook collection to the C2 backup but that is many Gb of books and that'd take a long time to upload. The other issue is that part of the directory tree has within it more than 1000 folders within so I can't use the folder browser in Hyper Backup to only select parts of the collection to do it in stages. If I threw the entire collection in there en-mass I'm presuming the C2 backup job would go on for ever until it was completed and block the other backup job? Any suggestions for getting around this? I can't see a solution.

2) Backup scheduling - I'm finding that C2 backup jobs sometimes fail as the 'destination is busy' which is when they are reconciling the backed up data and in effect pruning snapshots. Synology support advised that I reduced the backup frequency and change the retention policy. I was running the backups every hour, I'n now running every 4 hours. The question here is there any recommended interval?

Thanks as always for any thoughts.
 

jeyare

Subscriber
1,587
537
In general:
4hours interval is useful for business, for home data 24h cycle is enough


You need to evaluate what you will lose in case of damage when you lose your entire external disk or some important data in the disk (as first backup layer). Then you have the second layer in C2. Use paper and pencil and try to draw a group of data, then create simple evaluation.

Next consideration is about the data (lost in such case):
- can I download again the data from another source (Internet)? Do I need to store them in the C2?
- is necessary to use C2 only? Could I use automatic Syno Cloud sync to another data target (Google drive, Dropbox,...)?

An example:
- your personal photos is yours, then you can’t download them from another public sources.
- e-books. This is another example. Many of them you can download anytime. Then backup of them to the C2 (when you reach C2 limits) is questionable.
- ...
 

Rusty

Moderator
NAS Support
2,380
705
www.blackvoid.club
NAS
DS412+, DS718+, DS918+, 2x RS3614RPxs+ with expansions
Router
  1. RT1900ac
  2. RT2600ac
  3. MR2200ac
Operating system
  1. macOS
Mobile operating system
  1. iOS
I can't use the folder browser in Hyper Backup to only select parts of the collection to do it in stages
Are you sure this can’t be done? Since a few versions back HB now finally supports file level selection. So I guess you could select that 1K folder content and divide it into multiple tasks.

is necessary to use C2 only? Could I use automatic Syno Cloud sync to another data target (Google drive, Dropbox,...)?
Why would you put them into those services if you can download from C2 even without you NAS?
 
135
51
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
Thanks guys, I've altered the scheduling to every 6 hours and will see how that goes.

The books are the result of many years of collecting / purchasing etc and lot of the original sources are no longer around. So yes I'd be mortified if I lost it. I'd not be looking to store them on Onedrive etc however as the C2 backup is encrypted I feel a lot happier using that.

There are actually more than 10,000 files in the structure (not 1,000) and I get the error:
Screenshot 2020-05-16 at 18.38.48.png


So looks like I need to decide whether to just go for it.
 

fredbert

Moderator
NAS Support
Subscriber
1,619
674
NAS
DS1520+, DS218+, DS215j
Router
  1. RT2600ac
  2. MR2200ac
Operating system
  1. macOS
Mobile operating system
  1. iOS
To pull back a bit from being NAS-centric, have you considered alternative approaches?


Haven't tried this myself but a test with these will be free (CCC has a 30 day free trial and is my go to Mac clone and backup app). Idea:
  1. OneDrive Mac agent with files on demand enabled.
  2. Using Cryptomator create a vault within OneDrive. The vault should then be mounted as a Finder drive.
  3. With CCC create a task that mounts the NAS's ebook folder as source and has a destination of the Cryptomator vault drive (or folder in it).
  4. Run CCC task.
Downside is that OneDrive files will be accessible all the time and to schedule the CCC teask will require the vault to be mounted (contents accessible) all the time.

But could be a weekly task to do manually.


7/8Mbps outbound? Lucky, I have 6Mbps but should be a whooping 10Mbps if only Virgin Media could fix it. So I know the pain of needing to limit outbound speeds. You can use Traffic Control in DSM (Network settings) to limit the max. bandwidth to destination port TCP 6281 (Hyper Backup which is Syno C2 too).

I did this initially but I also use HB to another LAN NAS, so this wasn't ideal. Instead I've used Traffic Control in my Syno RT2600ac router and that has a blanket outbound limit from that NAS to the Internet. This limit leaves 1Mbps for outbound traffic from other devices. This isn't an issue for me because I only run HB to Syno C2 at night when there's little other contention.
 
135
51
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
@fredbert Okay a few questions here. I did look at CCC when I got the Mac last year, so a trial may not be available - I'll check later.
  1. The Onedrive Mac agent you reference is the standard Onedrive sync agent? I don't run that with Files on Demand turned on as I've some large data sets I don't want to be visible on the Mac at all. Is that an issue with what you propose?
  2. Cryptomator - I'd not seen this before, a definite one to try out. Is the recommended Fuse app needed for this?
The rest sounds viable.... If you can advise I'll then move it on from here.
 
135
51
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
Okay I've progressed a little. Due to the amount of data I need to backup I've had to set up Cloud Sync to upload the destination folder containing the vault(s) to Onedrive. That way I can use the NAS to carry the load. Okay I'll end up with a set of duplicated data as the books are stored on the NAS. I can live with that though.

So I've set up Cryptomator and created the vault. Fuse was loaded onto the system too.

The problem is when I try to use CCC to copy the data from the share to the share. I get 'CCC cannot back up to or from an OS X FUSE volume mounted without the "allow_root" or "allow_other" option.'

I can't see that option anywhere? So stuck for the moment.....
 

fredbert

Moderator
NAS Support
Subscriber
1,619
674
NAS
DS1520+, DS218+, DS215j
Router
  1. RT2600ac
  2. MR2200ac
Operating system
  1. macOS
Mobile operating system
  1. iOS
Okay a few questions here
OneDrive agent is the OneDrive Mac app (whatever it's called, it sits in the menubar and syncs in the background: hence why I generally refer to these things as agents). I hardly use OneDrive even though I've 1TB with an O365 subscription so if you know how to OneDrive on Finder without having the whole lot of cloud data on the Mac ... that's what I was meaning (though it would be the on demand feature).

CCC trial allows all features but after that you can still use it without incremental sync (all data, every time) as a small test.

With Cryptomator I think it's an all-in-one install, but I've had Fuse installed for years anyway.


The problem is when I try to use CCC to copy the data from the share to the share. I get 'CCC cannot back up to or from an OS X FUSE volume mounted without the "allow_root" or "allow_other" option.'
Hmm, does CCC have full disk access permission? If not you can try dragging the app to System Prefs security bit for Full Disk Access (in Privacy tab).

I just tried this with test vault (files on my Mac not OneDrive). The FUSE connection didn't work but changing mount type in prefs to WebDAV did. When requested by CCC for username/password for the 'WebDAV' I just entered 'test' for both ... so seems these could be anything.
 
135
51
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
Okay all is good with Onedrive & Cryptomator, still having the CCC issue though.

CCC does indeed have full disk access, so it's not that. Okay I went into Cryptomator and changed the mount method to WebDAV.

Weirdness...... I deleted the backup task and recreated it exactly as it was before and it's now running :) Sigh mysteries eh? :) Will see how it goes but it looks like I may well be buying CCC :)
 

fredbert

Moderator
NAS Support
Subscriber
1,619
674
NAS
DS1520+, DS218+, DS215j
Router
  1. RT2600ac
  2. MR2200ac
Operating system
  1. macOS
Mobile operating system
  1. iOS
If you don't do a bootable clone backup of the Mac disk then having CCC will sort that out. I've used CCC for quite a few years, and SuperDuper! before that. It's much easier to do a clone of a Mac to external, mount it and then restore (ie. backup) back to the internal disk than doing a Time Machine recovery.

I use CCC to shift data around between devices and also do backups to and of external drives too. I have a cheap licence for Get Backup Pro but to be really honest I only use CCC.
 
135
51
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
I don't do a bootable clone at all and I've no drive free to do that unless I could use the 8Tb external drive which is mounted on the NAS for Hyper Backup purposes? That has 4Tb of disk space free so would that work if I turned on SafetyNet to ensure files already held on the drive are not deleted?
 

fredbert

Moderator
NAS Support
Subscriber
1,619
674
NAS
DS1520+, DS218+, DS215j
Router
  1. RT2600ac
  2. MR2200ac
Operating system
  1. macOS
Mobile operating system
  1. iOS
You can try it.

I partition external disks (Disk Utility) to create various discrete areas for backing up each disk volume. The partition doesn't need to be much bigger that the source volume, and snapshot retention will use the spare space until it's needed for current data.

Having a bootable clone is a good recovery tool for when a OS upgrade goes wrong!
 
135
51
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
I just remembered I do have a 1Tb external drive which holds Apple iPad backups (before beta iOS install) so I'll copy those to the NAS (just in case) and use that for the clone backup :)

Is the CCC bootable clone process easy to do / schedule? I'll not look at it until the ebook backup is complete....
 

fredbert

Moderator
NAS Support
Subscriber
1,619
674
NAS
DS1520+, DS218+, DS215j
Router
  1. RT2600ac
  2. MR2200ac
Operating system
  1. macOS
Mobile operating system
  1. iOS
It's very easy to do as that's the original (primary) job of CCC. It must be an HFS+ or APFS volume and the disk's scheme must be GUID Partition Map. You can check this with Disk Utility but to change scheme (e.g. from Master Boot Record) needs a disk format.

More info here.

I have a schedule to run every Sunday night: two tasks/disks, one weekly and the other monthly. I've also got Time Machine for those times in between plus a lot of data is now in my Drive folder being sync'ed continuously.
 
135
51
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
Right thanks for that. I'll check out and reformat if needed.

The initial clone job just failed BTW as Finder locked up and disconnected the destination share.... Grrr. I had to completely turn off the Mac (as it'd not restart) and I'm now rerunning the job to see what happens this time. Sigh.....

Fingers crossed.
 

fredbert

Moderator
NAS Support
Subscriber
1,619
674
NAS
DS1520+, DS218+, DS215j
Router
  1. RT2600ac
  2. MR2200ac
Operating system
  1. macOS
Mobile operating system
  1. iOS
Best laid plans etc.
 
135
51
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
LOL tell me about it :) Will see how it goes and I may just bite the bullet and back up to C2 in the end.....

So far the re-run CCC job has been going for 36 minutes and is still file checking......
 
135
51
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
Well I woke this morning to find that CCC had completed the backup of the books taking a total of 6:33 hours to copy over the books to the vault :)
I've now re-started Cloud Sync but put it on a schedule to copy only between 20:00 and 06:00 otherwise it just kills the internet connection.
I'll continue to monitor how it goes......
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Welcome to SynoForum.com!

SynoForum.com is an unofficial Synology forum for NAS owners and enthusiasts.

Registration is free, easy and fast!

Similar threads

Similar threads

Trending threads

Top