Question Docker migration to new NAS

[alfabravo]

Hi,

i'd like to migrate from DS218+ ext4 2x6TB SHR to a DS 718+ BTRFS 2x8TB SHR.
I've read a lot and understood that hyper backup doesn't make a 1:1 copy. Does it correct?
What is the best way to migrate with Docker (8 Containers) and other Packages that not supported by
Hyper Backup?

Thank you
Regards

 

[one-eyed-king]

This is how I would do it:
- ui: shutdown container
- ui: export the container configuration
- shell: create an archive of the volumes: tar czvf myarchive.tar.gz folder (I would suggest to create the archives in the parent folder of your volumes)
- copy configuration json and tar.gz file to the new machine
- shell: extract the archive to the same location on the new ds and extract the archive: tar xzvf myarchive.tar.gz
- ui: import the container configuration (if you get an error the image does not exist, you will need to empty the value for "id:". if the tag was replaced by watchtower, you might need to fix the tag in "images:" as well.
- ui: check the configuration of the container, make sure the volumes are properly recognized and/or adjust the path, if you extracted the volume data in a different location.
- start the container
That's about it

If you created your container configuration using docker-compose.yml you can skip the steps to export/import the configuration and the steps will be reduced to stopping the container, creating the archive, copying to the new machine, extracting it and starting your docker-compose deployment.[/icode]

 

[one-eyed-king]

Maybee I should have mentioned why using tar to create the archive is prefered: the archive will preserve ownership and permissions. If the volume folders are just copied over from a file share you will end up lossing those details. Some container are very particular about file ownership and permissions and might end up working completly or will have odd behavior.

 

[alfabravo]

sounds pretty complicated for me))

 

[one-eyed-king]

For someone inexperienced it would seem like that. I moved my container zoo a couple of times so far. If the volume data is below one gigabyte, the whole operation takes something between 3-5 minutes per container.

 

[alfabravo]

if i would keep ext4 on the new machine, it would be easier to migrate?
RAID backup or clone HDD?

 

[one-eyed-king]

I told you what I effectivly did a couple of times myself.

If you find another approach that preserves file ownership and permissions, feel free to use it ... whatever works

 

[Telos]

- shell: create an archive of the volumes: tar czvf myarchive.tar.gz folder (I would suggest to create the archives in the parent folder of your volumes)

ICBW, but I believe the Synology Docker GUI does this if you chose to export container contents and settings.

 

[one-eyed-king]

Sounds to simple to be true... container content != content of a volume source

An export of a container is only usefull if the container was used without volumes. If all persistent data is "mapped out" to a volume, then the container itself is worth nothing - containers are ment to be disposable. Let me rephrase it: you only have to take care of the "trash", if you used it wrong in the first place ^^

 

[alfabravo]

- shell: create an archive of the volumes: tar czvf myarchive.tar.gz folder (I would suggest to create the archives in the parent folder of your volumes)

which volume do you mean? the whole volume? or specific part for Docker?
like this?

/volume1/@docker/volumes/dc89257b95c2ebe273cfc6b466b2b554f1909e7cc6a616612f72bc7a1602f530
i have only two of them and 13 containers, could it be?
appreciate your help, will try it anyway when new machine has arrive.

 

[one-eyed-king]

A volume in the docker context is always a docker volume or a bind-mount (this is what synolgy uses).

You can check for each containers if a volume was used:
- open container details
- change to the tab volume (in the left buttom corner)
- if volume mappings exist, you should see mappings from host paths as (shown as "File/Folder") to a container path (shown as "Mount path").

If the list is empty and the container requires a volume to persist its data, then Telos approach is the only one going to help you. For the sake of reliability I hope this is not the case...

 

[alfabravo]

ok. got it. i will try to archive the whole folder

/volume1/@docker/

and then extract to the new ds. and parallel export like Telos says or is it too much?
thank you for your patience!

 

[one-eyed-king]

Uh, that's a risky endavor and not what I ment.

It may work to archve the whole docker root-data folder and extract it on the new system. If it works, the whole metadata including images, containers, network declarations and volume declarations will be available on the new host. Hint: a volume declaration is not the volume itself, it is merely a pointer to the real storage - in case of bind-mount there is not even a volume declaration. This will NOT cover the data of volumes.


If you are willing to spend a lot of time, you can try whatever commes to mind. If everything fails, you can still use my approach to cleanly migrate to the new ds.

 

[Rusty]

It may work to archve the whole docker root-data folder and extract it on the new system

It will work. I use it like this via snapshot and replication tool. No problem at all. Then just import container config on the new NAS unit, download the image and boot it up. Like @one-eyed-king said, a matter of minutes.

 

[alfabravo]

ok you mean only archive for e.g. this volume folder:
/volume1/@docker/volumes/dc89257b95c2ebe273cfc6b466b2b554f1909e7cc6a616612f72bc7a1602f530
and use exactly the same name of the volume? does the name random?

 

[Rusty]

ok you mean only archive for e.g. this volume folder:
/volume1/@docker/volumes/dc89257b95c2ebe273cfc6b466b2b554f1909e7cc6a616612f72bc7a1602f530
and use exactly the same name of the volume? does the name random?

I ment Snapshot and replication of the /volume#/docker root destination. That destination holds all my bind-mount.

 

[alfabravo]

i guess Snapshot and replication ie only works on btrfs, isn't it?
i moving from ext4 to btrfs.

 

[Telos]

From Synology...

To apply current container settings to a container on a different Synology NAS:

1. In the Container tab, select the container you want to export.
2. Click Export settings.
3. Log into another Synology NAS and create a container with the same image file.
4. Select the container and click Settings > Import Settings.
5. Choose the file you previously downloaded and click Select to import settings.

Easy peasy.

Related....
How to export and import containers with Docker

 

[Rusty]

i guess Snapshot and replication ie only works on btrfs, isn't it?
i moving from ext4 to btrfs.

BTRFS is a requirement for S&R

 

[one-eyed-king]

Moving docker's root-data folder will not work, if the storage driver on the source and target system are not identical. On ext4 the storage driver AuFS used, on btrfs the storage driver btrfs is used. Docker will completly ignore the metadata from the old storage driver, which basicly will look like a fresh installed state.

Exporting the container, regardless which way you use, does not backup and restore data from inside bind-mounts or volumes. It will cover the container, with all the read-only image layers of the image and the copy on write layer on top that is responsible to store changes (changes as in new files, modified files, deleted files). The volumes (including bind-mounts) are what stores the state - those are not covered by a container export.


Let me summarize the three different approaches that have been proposed so far:
- move the whole root-data folder
- export/import container data
- export/import container setting

The first approach works well for Rusty, because he has his bind-mound folders underneath the root-data folder. Only because of this his volume state is migrated as well. Appart from that it is still not going to work for you, as the storage driver will use a different storage driver than your old system.

None of the approaches covers volumes, which usualy are the place where configuration and application data is stored. Well mine does, but is appearently to complicated.

@alfabravo: I would suggest to start your migration and report back which approach you used and what actualy stopped working or is missing on the new installation. Time for action!