Container Manager > A definitive guide for backup & recovery

[JOINSO]

Hello!

I've been using Container Manager for years and I'm very happy with it. I have several Docker containers running without any problems.

Due to a human error, an update was accidentally performed on two linked Docker containers that shouldn't have been updated. I was able to recover them, but it wasn't quick; it tooks me several hours.

I would like that this post can be a general guide on how to manage Docker backups in Container Manager, as well as disaster recovery.

We all know that Container Manager isn't backed up as an "app" in HyperBackup.

First, I will explain my current setup and we will look at its weaknesses:

1. A HyperBackup task that backs up a shared folder called "docker".

This folder basically contains two subfolders:

exports
volumes
As you can deduce, within volumes there's a folder for the volumes of each Docker container I have.

The export folder is detailed below.

2. Some daily tasks I've defined where I perform similar tasks:

cd /volume1/docker/exports/
docker container stop DOCKER_NAME
cd DOCKER_NAME
rm -Rf .
docker export DOCKER_NAME > DOCKER_NAME.tar
gzip DOCKER_NAME.tar
docker container start DOCKER_NAME

What's the problem with this method of creating backups?

Hyperbackup works perfectly: you can recover any volume from the Docker folder.

The problem lies in how each Docker container is backed up.

With docker export, only the current state of the entire filesystem is saved.

If you then try to perform a docker import, it won't allow you to restart the Docker container you may have lost. Synology just creates an image so it can be restarted, but of course, you have to reconfigure everything and is not easy.

Therefore, this process is not efficient.

In the event of a disaster, I can restore the affected Docker containers by finding the official Docker image, recreating the configuration, and using the volume from a HyperBackup restore.

So, how could we improve this?

How can we run a script that creates a copy of the filesystem state associated with each Docker container, while also including the configuration?

The desired result is that in case of a disaster, I can:

• Restore the HyperBackup copy to have the latest volume available.
• Without having the affected Docker container in Container Manager, I can restart it in a matter of minutes using the backup I have.

Thanks in advance
Regards,
JOINSO

 

[one-eyed-king]

Hmm, are you sure it works that way. From what I know an exported container looses the association to an image and looses all metadata.

I am using docker for 11 years now and docker export never was part of a backup solution, as it's an unclean solution: it breaks the relation to the image - so updating the base image to get a security and bug fixed version of the image doesn't work anymore.

Furthermore, export will not export the content of bind mounted container folders or named volume, so in reality you are exporting a state of the container itself, something that is supposed to be disposable by design..

A better approach would be to:

• stop the container
• backup the volumes using a temporary container or backup the host folders of binds
• optional: use docker save to save the image + all metadata to a tar
• start the container again

Your backup should contain the backup of all volumes, if wanted the exported image and a docker compose file to restore everything in case recovery is necessary.

 

[JOINSO]

Hi!
Yes, you are right: docker export it is not the better solution as I discover with the incident.
With docker export, I have a tar, where you can find all the files that it will be used by the container and are mapped into the volumes, but it is only this.
Developing your idea, this will be the ideal script to backup?

• Hyperbackup: copies all the shared folder that holds all the volumes. This is always a good option.
• Stop the container.
• Use docker save: docker save -o TAR_DOCKER_FILE DOCKER_NAME
• Star the container

For recover:

Imagine that all is deleted!
The shared folder for docker is empty.
The image is not in the Container Manager!
Total disaster!

It is correct to do this?
1.- Restore the last copy of Hyperbackup.
2.- Use docker load: docker load --input TAR_DOCKER_FILE ?????
And then? Here I am lost...

Regards,
Jordi

 

[Telos]

Hyper Backup cannot adequately backup running database containers... It may work, but often won't if the database is active.

 

[JOINSO]

It will work if we stop the container and then execute HyperBackup?
Or you suggest to avoid it?
Or just use HyperBackup to keep the copies of the tar files created by the script that we are talking above.

Regards,
Jordi

 

[Telos]

It will work if we stop the container and then execute HyperBackup?

Yes.

 

[JOINSO]

So the correct backup process will be:

• Stop the containers.
• Hyperbackup: copies all the shared folder that holds all the volumes. This is always a good option.
• Use docker save: docker save -o TAR_DOCKER_FILE DOCKER_NAME
• Star the containers.

Now we need to know how to implement the recover process in the most dramatic scenario.

 

[Telos]

Now we need to know how to implement the recover process in the most dramatic scenario.

And test it today, before it becomes a must do.

 

[JOINSO]

Hi!
This is my "backup" process:
1.- Stop dockers
2.- Star dockers save --> store in "saves" folder
3.- Star dockers exports --> store in "exports" fodlder (not necessary but still keeping...)
4.- Hyperbackup all the "docker" shared folder that containts:

• saves
• exports
• configs
• volumes

|- container1
|- container2
....
5.- Start dockers

Detail:

For setup correctly the docker save of every container i do this first:

# docker ps -a
This will get a list with this info:

CONTAINER ID | IMAGE | .... | NAMES
xxxxxxx aaaa:bbbb my_container_name
...................
Then I do the save's in this way:

docker save -o my_container_name_save.tar aaaa:bbbb
gzip my_container_name_save.tar

I am in the correct way?

Regards,
Jordi

 

[one-eyed-king]

You can gzip it in one go:

 docker save <image> | gzip > <filename for exported image>.tar.gz

I liked the approach from the previous post better. Use Hyperbackup on the host directories. You could easily write a bash script that does something along these lines:

• loop over all container
  • stop current container
  • loop over docker inspect --format '{{.HostConfig.Binds}}' <current element from container loop>
    • run tar czf backup tar -C <current element from bind loop> . to create backup of the host volumes
  • start current container
  • export compose file for current container: docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/red5d/docker-autocompose <current element from container loop> > docker-compose.yml
  • export image: docker save $(docker inspect --format '{{.Config.Image}}' <current element from container loop> | gzip <current element from contaiern loop>.tar.gz

I guess it makes more sense to create a hyperbackup per container. The last two steps can be performed when the container is already running again.

 

[JOINSO]

@one-eyed-king : Thanks for the suggestions!
I will improve my script getting your ideas.

Now i am focus on how to do the recover process and I have one doubt that I get from your suggestions.

Imagine that there is a total disaster or I have a new Synology and I want to move things.

What I have with my backup process?
1.- An Hyper Backup copy.
2.- A tar.gz file from the docker save command.

In the new synology:
1.- Move the hyperbackup copy: Now you have the "docker volume" of your container.
2.- Obviously install Container Manager
2.- And then you do:
docker load < my_container_name_save.tar.gz

And what happens?
Now you have your container running again?

Then I remember this from your suggestion:

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/red5d/docker-autocompose <current element from container loop> > docker-compose.yml

We need the "docker-compose.yml" file?

Regards,
Jordi

 

[one-eyed-king]

Are you sure that you are using docker volumes as in named volumes, listed by docker volume ls? My assumption is that you don't use named volumes, but use bind volume instead (where a host folder is bind mounted into a container folder).

The recovery process would be:

• import image in local cache with docker load (this step is optional, because the image can usually be pulled from a registry)
• restore the host folders from hyperbackup that are used as bind volume by the container
• deploy the compose file that contains the configuration of the container (this can be done in container manager, or in the cli/a script).

If named volumes are used, the approach is slightly more complicated

We need the "docker-compose.yml" file?

Its is the easiest way to create a container with a specific configuration. Instead, you could run commands to export the container configuration from container manager, and restore the config in container manager again. The latter approach is more complicated, and a vendor lock-in. That's why I prefer compose, as it work on every docker engine.

 

[JOINSO]

"Are you sure that you are using docker volumes as in named volumes, listed by docker volume ls? My assumption is that you don't use named volumes, but use bind volume instead (where a host folder is bind mounted into a container folder)."

# docker volumes ls
docker: 'volumes' is not a docker command.
See 'docker --help'

Yes, I think I am using "bind".

From one of my exported configs using Container Manager, I see this in the JSON:
....................
"volume_bindings" : [
{
"host_volume_file" : "/docker/volumes/digikam",
"is_directory" : true,
"mount_point" : "/config",
"type" : "rw"
},
{
"host_volume_file" : "/photo",
"is_directory" : true,
"mount_point" : "/config/Pictures",
"type" : "rw"
}
]
....................

About the recover process:

• deploy the compose file that contains the configuration of the container (this can be done in container manager, or in the cli/a script).

I have some container_name.json files generated with Container Manager > Container > (select the desired) > Action > Export > Export container config.

That's right?

And this can be done with a daily script to always keep the current config and not doing manually?

About your last thoughts:

"The latter approach is more complicated, and a vendor lock-in. That's why I prefer compose, as it work on every docker engine."

Yes, you are right. That will be ideal.
But when I consider moving away from Synology and looking at more neutral systems, I might also consider using third-party services.
For now, thanks to everyone's input, we've made significant improvements.

Note: sorry for not using the correct format in replies, or using the formatted way of "someone says".

Regards,
Jordi

 

[Telos]

# docker volumes ls
docker: 'volumes' is not a docker command.

docker volume ls

 

[JOINSO]

# docker volume ls
DRIVER VOLUME NAME
local 2b17e8459050e5d3a3a4315c772491e9afa9e4a1e23e79bd9ec4666005b0d578
local 6f051a124d21e543ba87a29d72d45ffabc61552d4503b5735291f1953856e61c
local 7e763af3adc13b1dd1d2d991c5d1f525905fa07b4083815b497774bbc2236ca9
local 8f9c40a0ad1fd969ee344600e429fe91cf286949a2b3f72e92b3d9bcd2e29410
local 17af1ae8c1bd777d52e524f7c89f01c85a5a15de80044ca14d961715548bc49e
local 73d6795744a93c3cead6598de458bb0fcfc3d55ce45e79ed0ae08e0fd5bac94d
local 74add8050f68be3ed8afcd5a64069670940450ebed5df4b6aed7e640f5367c10
local 95de8e1d6b705380498673ae913fdc42e85dd5bc3cd4617025650a6d36f0d1fb
local 444b01843987a396d2ffa686363ea563e931b16c1a1c767367f2b2982fd9f701
local 956c5643c1ca75104544ea37b13d99544ff874c6d3f38d93de4e46ba4e480bea
local 8233d9ffb88ae7d2e7220359525ace22c06cd40361477389b06135bd7ea4b076
local bd54ce737747bf98a21b3559ad735d755481383c77bfad524f0193702c4517e5
local ce582387802b897d873595b08fdcdadd2945ccb65c13c49d7ea3c8edd6cde973
local e1a0cb50ac41db8ab6592acc3c7b84b392d07bbba2fd2d898b961fb402286560
local ee3736d110d6e1cbbc7a27086092e6328351efc69b7cf68d8335b7d4a5bf4f86
local f29310f3b1ce024a76b326885a5020ec5dd7aa2da2dd7ca5f5bf9a57faa2cc75
local fa3be0815b87d4e0e5a4b0f12b568cf9c732bfab721be283f4efa7c423bbedb2

 

[one-eyed-king]

# docker volumes ls
docker: 'volumes' is not a docker command.
See 'docker --help'

This is almost the command i shared
Try again without the extra "s":

docker volume ls

{
"host_volume_file" : "/docker/volumes/digikam",
"is_directory" : true,
"mount_point" : "/config",
"type" : "rw"
},

This looks like a volume.

I have some container_name.json files generated with Container Manager > Container > (select the desired) > Action > Export > Export container config.

That's right?

And this can be done with a daily script to always keep the current config and not doing manually?

Yup. looks like this (must be run as root afair):

/usr/syno/bin/synowebapi --exec api=SYNO.Docker.Container.Profile method=export version=1 outfile="/volume1/docker/test.json" name="test"

outfile is the target path where the container-config json should be stored
name is the name of the container, in the loop it would be <current element from container loop>[/code]

 

[one-eyed-king]

# docker volume ls
DRIVER VOLUME NAME
local 2b17e8459050e5d3a3a4315c772491e9afa9e4a1e23e79bd9ec4666005b0d578
local 6f051a124d21e543ba87a29d72d45ffabc61552d4503b5735291f1953856e61c
....

Those are all anonymous volumes, they are created when a VOLUME is declared in the Dockerfile of a container image is specified for a container path, but the container based on the image didn't mount a volume into this container directory.

It is usually a sign that you missed out on configuring volumes....

Note: Probably most of the volumes are orphaned.

{
"host_volume_file" : "/docker/volumes/digikam",
"is_directory" : true,
"mount_point" : "/config",
"type" : "rw"
},

I am surprised to not see a volume named digikam in the list of volumes. Please tell me that you didn't specify this folder yourself. Please share the output of sudo ls -la /docker/volumes/digikam so we can see if this is really not a volume.

 

[JOINSO]

root@NAS-SYNOLOGY-3:~# sudo ls -la /volume1/docker/volumes/digikam
total 793332
drwxr-xr-x+ 1 ad242_sds213 users       618 Nov  8 18:47 .
drwxr-xr-x+ 1 ad242_sds213 users       120 Nov 10 18:21 ..
-rwxr-xr-x+ 1 root         root        108 Nov  8 18:47 .bash_history
-rwxr-xr-x+ 1 ad242_sds213 users        37 Jul 26 20:20 .bashrc
drwxr-xr-x+ 1 ad242_sds213   911       486 Sep 10  2023 .cache
drwxr-xr-x+ 1 ad242_sds213   911       102 Nov 12 07:23 .config
drwxr-xr-x+ 1 ad242_sds213 users        22 Jul 26 20:20 .dbus
drwxr-xr-x+ 1 ad242_sds213 users         0 Jul 26 20:20 Desktop
-rwxr-xr-x+ 1 ad242_sds213 users  40345600 Nov  8 11:43 digikam4.db
...............

As you can see:
"host_volume_file" : "/docker/volumes/digikam",

is relative to the volume1 of the NAS.

 

[JOINSO]

# /usr/syno/bin/synowebapi --exec api=SYNO.Docker.Container.Profile method=export version=1 outfile="/volume1/docker/digikam.json" name="digikam"
[Line 265] Not a json value: digikam
[Line 295] Exec WebAPI:  api=SYNO.Docker.Container.Profile, version=1, method=export, param={"name":"digikam"}, runner=SYSTEM_ADMIN, outfile=/volume1/docker/digikam.json
{
   "data" : {},
   "httpd_restart" : false,
   "success" : true
}

 

[one-eyed-king]

Ah, right. I confused it with the root-data folder (you can check it with docker info --format '{{.DockerRootDir}}'). My bad. Of course, It's not a volume.