Container Manager > A definitive guide for backup & recovery

[JOINSO]

Hello!

I've been using Container Manager for years and I'm very happy with it. I have several Docker containers running without any problems.

Due to a human error, an update was accidentally performed on two linked Docker containers that shouldn't have been updated. I was able to recover them, but it wasn't quick; it tooks me several hours.

I would like that this post can be a general guide on how to manage Docker backups in Container Manager, as well as disaster recovery.

We all know that Container Manager isn't backed up as an "app" in HyperBackup.

First, I will explain my current setup and we will look at its weaknesses:

1. A HyperBackup task that backs up a shared folder called "docker".

This folder basically contains two subfolders:

exports
volumes
As you can deduce, within volumes there's a folder for the volumes of each Docker container I have.

The export folder is detailed below.

2. Some daily tasks I've defined where I perform similar tasks:

cd /volume1/docker/exports/
docker container stop DOCKER_NAME
cd DOCKER_NAME
rm -Rf .
docker export DOCKER_NAME > DOCKER_NAME.tar
gzip DOCKER_NAME.tar
docker container start DOCKER_NAME

What's the problem with this method of creating backups?

Hyperbackup works perfectly: you can recover any volume from the Docker folder.

The problem lies in how each Docker container is backed up.

With docker export, only the current state of the entire filesystem is saved.

If you then try to perform a docker import, it won't allow you to restart the Docker container you may have lost. Synology just creates an image so it can be restarted, but of course, you have to reconfigure everything and is not easy.

Therefore, this process is not efficient.

In the event of a disaster, I can restore the affected Docker containers by finding the official Docker image, recreating the configuration, and using the volume from a HyperBackup restore.

So, how could we improve this?

How can we run a script that creates a copy of the filesystem state associated with each Docker container, while also including the configuration?

The desired result is that in case of a disaster, I can:

• Restore the HyperBackup copy to have the latest volume available.
• Without having the affected Docker container in Container Manager, I can restart it in a matter of minutes using the backup I have.

Thanks in advance
Regards,
JOINSO

 

[JOINSO]

Hi!
I opened a Synology Support ticket asking this:
If this command:

/usr/syno/bin/synowebapi --exec api=SYNO.Docker.Container.Profile method=export version=1 outfile="/volume1/docker/container_name.json" name="container_name"

makes an export of the config of the container like we can using DSM > Container Manager > Container > Export conf.
There is a command equivalent to export CONF+DATA we can do in DSM?

The support agent says that he does not know, and does not found in KB, and he will ask internally.

Waiting for response.

 

[JOINSO]

Hi!

The correct command is:

synowebapi --exec api=SYNO.Docker.Container method=export version=1 name=CONTAINER_NAME path=PATH_TO_EXPORT_CONTAINER_SYNO_TXZ_FILE 1>YOUR_OUTPUT_LOG 2>YOUR_ERROR OUTPUT_LOG

where PATH_TO_EXPORT_CONTAINER_SYNO_TXZ_FILE is the path of shared folder WITHOUT the "/volumeX/" start string.
Note: remember that this only exports config and the container. You need also to preserve the volumes.
An Hyperbackup task is your best friend here.

The problem is that command launch a new subprocess, and if you are iterating over all the containers it will launch all the exports at the same time.

I tried to handle with pids/waits but it does not work.

 

[JOINSO]

This a better solution, not very elegant but it works:

synowebapi --exec api=SYNO.Docker.Container method=export version=1 name=$element path=/docker/exports/$element 1>$standard_output 2>$error_output
pid=`ps -edaf | grep "synowebapi --exec api=SYNO.Docker.Container" | grep $element | awk '{print $2}'`
echo "     Waiting for PID $pid completion..." >> $logOutput
while [ -e /proc/$pid ]
do   
    sleep 1m
done

Note: element holds the container name inside a loop over all the containers.

 

[JOINSO]

Hi all!!!

I'm going to give a brief summary of the solution I've implemented to have a good backup of our containers.

First of all, I want to thank @one-eyed-king and @Telos for the help provided!

Also, remember that the provided scripts could be improved and everyone should adapt them to their needs.
As you'll see, I haven't adapted the automatic container list because I was too lazy, because I wanted to manually control what it did, and because I don't have many containers. If anyone follows this thread, you'll see that it's very easy to do.

First of all, I recommend that everyone create a new spreadsheet called "synology_tasks".
In it, we'll create the following columns:

DESCRIPTION | TYPE | STATE | CATEGORY | TASK | PERIOD | EXECUTION TIME | MAX TIME SEEN | USER

Then get a list of all the taks that you have running on your Synology device:

synowebapi --exec api=SYNO.Core.TaskScheduler method=list

And put your tasks in the spreadsheet.
You must also need to get some extra data from DSM > Config > Task Manager.

Then also categorize your tasks with your custom names.
In my case, I categorize with DOCKER all the tasks that are related to docker operations, like stop and start containers, doing config exports, complete exports (Synology way), save, hyperbackup the dockers, etc.

Why is this important?

As you know you can schedule all the tasks without taking a look at the execution time, but sometimes it is better that you distribute some tasks in a specific range time and with enough execution time to each one to avoid execution collision.

Also some tasks needs to be executed first to prepare some scenario to another tasks, like stop containers that involves databases, and then make secure saves and exports (Synology way in my case).

Remember that for you, perhaps is the first time that execute the task that I will show you below and you are not sure what is the execution time needed.
Also, you need to know in which hour range you will execute all the tasks related to DOCKER and then move all secondary tasks to another time range.
For example, Backup Integrity Check, that can be done on weekends in a time range where no other tasks can be running.

When you have this spreadsheet filled, then you can order the tasks by execution time, and adjust the time ranges to your desired configuration based on the time needed to run each tasks to complete.

Now is time to put some order in the shared folder that you use for Docker.

In my case I use only one shared folder.

Some people split each container setup in different shared folders and perhaps in different volumes.
This is a personal decision.
In my case, I have a shared folder named "docker".

This is my "docker" structure:

/docker
|- configs
|- my_container_name_1
|- my_container_name_2
|- my_container_name_3
...
|- my_container_name_N
|- exports
|- my_container_name_1
|- my_container_name_2
|- my_container_name_3
...
|- my_container_name_N
|- saves
|- my_container_name_1
|- my_container_name_2
|- my_container_name_3
...
|- my_container_name_N
|- volumes
|- my_container_name_1
|- my_container_name_2
|- my_container_name_3
...
|- my_container_name_N

When I setup and run a new container, I am only focused in the setup of the volume.
and the creation of the related sub folders in configs, exports and saves.
The automated tasks that I will explain later will fill automatically this folders.

Then you need to setup a task that will backup this docker shared folder.
Hyper Backup is a good option but other people can use tools like rsync?

In my case I use Hyper Backup to backup to a local external USB big disk, and also to backup to AWS S3.
You can backup also to another volume instead of the USB option.
You must adjust with your desired configuration.

Ok, now is time to automate things with a specific order!

Here is my task list ordered by execution time:

/volume1/YOUR_DESIRED_SHARED_FOLDER/tools/sh/stop-docker-containers.sh | Daily | 00:30:00
/volume1/YOUR_DESIRED_SHARED_FOLDER/tools/sh/docker-configs-exports.sh | Daily | 00:40:00
/volume1/YOUR_DESIRED_SHARED_FOLDER/tools/sh/save-docker-images.sh | Daily | 00:45:00
/volume1/YOUR_DESIRED_SHARED_FOLDER/tools/sh/export-docker-images.sh | Daily | 01:15:00
HyperBackup task to local folder (USB/another volume...) | Daily | 03:15:00
HyperBackup task to local folder (AWS S3) | Daily | 04:30:00
/volume1/YOUR_DESIRED_SHARED_FOLDER/tools/sh/start-docker-containers.sh | Daily | 05:00:00

Why we "run" in this order?

Is important to stop containers if there are some of them are using databases, before we "backup" them.
Then do the "backup tasks", and finally start with calm.
Another important think: perhaps we only need to setup "save" script and not the "export" ones.
Or only the "export" and not the "save" one.
I execute both.
It wouldn't hurt to perform both tasks.

Let show all the scripts:

stop-docker-containers.sh

# DEFINE YOUR LIST OF CONTAINERS IN list
# Keep in mind that the order is important!
# If you have linked containers that will start automatically,
# It is important to stop the container that needs dependencies first,
# and then the containers that do not need them.
list='my_container_name_1 ... my_container_name_N'
for element in $list;
do
    docker container stop $element
done

docker-configs-exports.sh:

list='my_container_name_1 ... my_container_name_N'
for element in $list;
do
  standard_output="/volume1/docker/configs/"$element"/"$element"_config_export_output.txt";
  error_output="/volume1/docker/configs/"$element"/"$element"_config_export_error.txt";
  /usr/syno/bin/synowebapi --exec api=SYNO.Docker.Container.Profile method=export version=1 outfile="/volume1/docker/configs/$element/$element.json" name="$element" 1>$standard_output 2>$error_output;
done

save-docker-images.sh:

# Define your list of containers and their images in CONTAINERS_AND_IMAGES
# You can see in DSM > Container Manager > Container : Name and Image
# Also you can get with 'docker ps'
declare -A CONTAINERS_AND_IMAGES
NUM_ROWS=N # REPLACE N WITH YOUR CONTAINERS COUNT
CONTAINERS_AND_IMAGES[1,1]=my_container_name_1
CONTAINERS_AND_IMAGES[1,2]=your_container_1_image_id
# ....
# REPLACE N IN THE NEXT LINES WITH YOUR CONTAINERS COUNT
CONTAINERS_AND_IMAGES[N,1]=my_container_name_N
CONTAINERS_AND_IMAGES[N,2]=your_container_N_image_id
for ((i=1;i<=NUM_ROWS;i++))
do
    cd /volume1/docker/saves/${CONTAINERS_AND_IMAGES[$i,1]}
    rm -Rf *.*
    docker save -o ${CONTAINERS_AND_IMAGES[$i,1]}_save.tar ${CONTAINERS_AND_IMAGES[$i,2]}
    gzip ${CONTAINERS_AND_IMAGES[$i,1]}_save.tar
done

export-docker-images.sh:

logOutput=/volume1/docker/exports/config_container_export.log
# DEFINE YOUR LIST OF CONTAINERS IN list
list='my_container_name_1 ... my_container_name_N'
echo "Starting at: "`date` > $logOutput
for element in $list;
do
    cd /volume1/docker/exports/$element
    rm -Rf *.*
    echo "   Start "$element" at "`date` >> $logOutput
    standard_output="/volume1/docker/exports/"$element"/"$element"_config_container_export_output.txt";
  error_output="/volume1/docker/exports/"$element"/"$element"_config_container_export_error.txt";
    synowebapi --exec api=SYNO.Docker.Container method=export version=1 name=$element path=/docker/exports/$element 1>$standard_output 2>$error_output
    pid=`ps -edaf | grep "synowebapi --exec api=SYNO.Docker.Container" | grep $element | awk '{print $2}'`
    echo "     Waiting for PID $pid completion..." >> $logOutput
    while [ -e /proc/$pid ]
    do
    sleep 1m
    done
    echo "   End "$element" at "`date` >> $logOutput
done
echo "Ending at: "`date` >> $logOutput

Notes:

1) Is important that the first time that you execute this scripts, check the file /volume1/docker/exports/config_container_export.log
It will show the time needed to do this tasks.
And then you can adjust the execution time of the task that we fill in the spreadsheet and "move".

2) Why the "waiting setup"?
If we not do that....:
The export of conf+image in DSM way launch a new process and continues the execution script.
That's mean that all the "exports" are done at the same time.
It could work, but I prefer to do it one by one.

start-docker-containers.sh:

# DEFINE YOUR LIST OF CONTAINERS IN list
# Keep in mind that the order is important!
# If you have linked containers that will start automatically,
# It is important to start first the container dependencies
# and then the 'master' container that relies on them.
list='my_container_name_1 ... my_container_name_N'
for element in $list;
do
    docker container start $element
done

And now all is working perfectly!

What it will be the recover process?

I will explain only the way is more easy to me, but someone can add how to do it in the docker pure way.

One of the scenarios can be the "moving" one.
Imagine that you are buying a new Synology device to replace the old one.

In Synology source (your old device):

- Be sure that you have a correct Hyper Backup task that backup your docker shared folder.

In Synology target (your new device):

• Install Hyper Backup.
• Install Hyper Backup Vault.
• Copy source backups to new device.
• Created a new docker shared folder in the new device.
• Restore the last copy and you will get all the volumes from the source.
• Check if you have all the correct folder infrastructure, specially the folder that every container needs to run (volume bindings)
• Import the exported container file from /docker/exports/my_container_name_X: container_name_X.syno.txz
• Star your container.

You can do the same if you are recovering in the same device.
In this case you do not need Hyper Backup Vault.

And that's all!

Remember to the keep your tasks spreadsheet in a secure place.
Every time you need to add new tasks or adjust execution time, first take look at this document.
Make the necessary modifications and when it is clear, setup in Synology Task Scheduler.

I repeat again: thanks to everyone that help me in setup this!

Regards!!!!