Problems with Docker after the latest update

Currently reading
Problems with Docker after the latest update

86
26
Are there people here who are experiencing problems with Docker since the last update?

I notice that after the last update I have more problems with Docker. For example, if I update a container via CLi, the container disappears after it has been updated. The container starts up but no longer appears in the GUI list. Via portainer the container is visible and I see that it has started. I have to restart Docker to make it visible again in the GUI list.

Last weekend I had problems with Baikal, who has been running for years on ports 8443 and 8008 but suddenly the container didn't want to start anymore and I got a message that there was a port conflict. I then changed the ports and Baikal finally started up. At the end of the day I tried again to put Baikal on the old ports (8443 and 8008) and everything worked normally again.

This morning all the containers just started up normally, but I couldn't reach anything. All proxy https:// links didn't work anymore and I got the synology error page everywhere and apps didn't work either, they couldn't connect to the corresponding containers. None of the containers gave an error message in the logs everything looked normal. Then I decided to restart Docker and everything worked again.

Very strange things are happening and I wouldn't know where to look for it or how to fix it. Maybe someone here has similar experiences or knows a solution.
 

Rusty

Moderator
NAS Support
2,350
700
www.blackvoid.club
NAS
DS412+, DS718+, DS918+, 2x RS3614RPxs+ with expansions
Router
  1. RT1900ac
  2. RT2600ac
  3. MR2200ac
Operating system
  1. macOS
Mobile operating system
  1. iOS
Can't say that I have had similar situations are you described here, but a while back I did had a network problem with some containers that in the end were fixed by docker restart. Could be similar problem that you have described but nothing close. Running about 50 containers total on 2 docker hosts with the latest version of Docker that Synology offers.
 
Upvote 0
Are there people here who are experiencing problems with Docker since the last update?
You almost made me think there was an updated package. The latest update is more than 6 months old now.

In the german Synology forum there was a length discussion about containers complaining about port conflicts (beware of german :giggle:).

In dmesg, if you see output like unregister_netdevice: waiting for veth1 to become free. Usage count = 1 (device and count will differ), than you are affected by a kernel bug.

The bug is said to be fixed with Kernel 4.14.114. Afair Synologys latest is 4.4.x?
Another reason to update to DSM7, which is going to address this issue by updating the kernel.


Side note: Ubuntu, RHEL and CentOS backport the fix to their older kernels as well - never experienced the problem on those systems so far.
 
Upvote 0
86
26
Can't say that I have had similar situations are you described here, but a while back I did had a network problem with some containers that in the end were fixed by docker restart. Could be similar problem that you have described but nothing close. Running about 50 containers total on 2 docker hosts with the latest version of Docker that Synology offers.
Thanks for your message. I have only 25 containers running on two hosts one of them is VDSM both on the latest Docker version and the problem I have described especially the one that is about the disappearing containers after an update I experience on both hosts. So that’s why I concluded that it’s related to the update, but as @one-eyed-king is stating it could be kernel related makes more sense to me..
-- post merged: --

You almost made me think there was an updated package. The latest update is more than 6 months old now.

In the german Synology forum there was a length discussion about containers complaining about port conflicts (beware of german :giggle:).

In dmesg, if you see output like unregister_netdevice: waiting for veth1 to become free. Usage count = 1 (device and count will differ), than you are affected by a kernel bug.

The bug is said to be fixed with Kernel 4.14.114. Afair Synologys latest is 4.4.x?
Another reason to update to DSM7, which is going to address this issue by updating the kernel.


Side note: Ubuntu, RHEL and CentOS backport the fix to their older kernels as well - never experienced the problem on those systems so far.
Thanks for your suggestions I will have a look on the German forum with the help of google translate ;). And have a look at the kernel bug thing...
 
Upvote 0
86
26
You can easily check if the kernel bug is the root cause:
ssh into your box, execute dmesg -T | grep unregister_netdevice and check the output.
I have ssh into my synology box dmesg -T | grep unregister_netdevice
And there is no output? Does this mean that I don't have the bug? Or am I doing something wrong
 
Upvote 0
My bad, seems dmesg does not include the messages.

We need to look at the kernel.log instead:
Bash:
sudo grep unregister_netdevice /var/log/kern.log
 
Upvote 0
86
26
My bad, seems dmesg does not include the messages.

We need to look at the kernel.log instead:
Bash:
sudo grep unregister_netdevice /var/log/kern.log
Okay So tried it and this is the output
Code:
[email protected]:~# sudo grep unregister_netdevice /var/log/kern.log
2020-09-16T19:34:32+02:00 base kernel: [48857.200970] unregister_netdevice: waiting for lo to become free. Usage count = 1
2020-09-16T19:34:42+02:00 base kernel: [48867.293822] unregister_netdevice: waiting for lo to become free. Usage count = 1
2020-09-17T23:03:43+02:00 base kernel: [61408.441566] unregister_netdevice: waiting for lo to become free. Usage count = 1
2020-09-17T23:03:54+02:00 base kernel: [61418.508422] unregister_netdevice: waiting for lo to become free. Usage count = 1
2020-09-17T23:04:04+02:00 base kernel: [61428.613270] unregister_netdevice: waiting for lo to become free. Usage count = 1
2020-09-18T20:13:44+02:00 base kernel: [51208.086620] unregister_netdevice: waiting for lo to become free. Usage count = 1
2020-09-18T21:18:57+02:00 base kernel: [55121.789077] unregister_netdevice: waiting for lo to become free. Usage count = 1
2020-09-18T21:26:31+02:00 base kernel: [55574.913315] unregister_netdevice: waiting for lo to become free. Usage count = 1
2020-09-18T21:26:41+02:00 base kernel: [55585.024163] unregister_netdevice: waiting for lo to become free. Usage count = 1
2020-09-18T21:26:51+02:00 base kernel: [55595.106009] unregister_netdevice: waiting for lo to become free. Usage count = 1

What does this mean?
 
Upvote 0
86
26
That you are affected by the kernel problem.
Lets pray DSM7 will ship with a kernel that does not suffer from the bug.
OMG Thats bad, So it means I will keep having problems with Docker and the containers I'm running from Time to Time. Is there a workaround for the moment?
 
Upvote 0
I realize that you didn't read the kernel bug link, which points to a blogpost describing the problem in detail: When the blue whale sinks

The only workarround is to wait until it clears up - He could be talking about minutes, hours, days here or restart the nas. This is a kernel bug! ... not a docker bug.
 
Upvote 0
86
26
Btw, I read something on the German Forum that running more then 20+ container and If you are regularly installing and deleting containers you could trigger the bug. Are you familiar with this?
 
Upvote 0
From the german forum it gets clear that Synology is aware of this problem and a fix is expected with DSM7. Till then: nothing you can do. The problem will kick in randomly is not restricted to a specific number of containers. Race conditions usualy don't follow rules *cough* ... and this is a terrible race condition.
 
Upvote 0
86
26
I realize that you didn't read the kernel bug link, which points to a blogpost describing the problem in detail: When the blue whale sinks

The only workarround is to wait until it clears up - He could be talking about minutes, hours, days here or restart the nas. This is a kernel bug! ... not a docker bug.
Yes I read the blog "When the blue ...." but I though seeing that it was an old article (2017) that there maybe would be a workaround for now
 
Upvote 0
86
26
From the german forum it gets clear that Synology is aware of this problem and a fix is expected with DSM7. Till then: nothing you can do. The problem will kick in randomly is not to a specific number of containers. Race conditions usualy don't follow rules *cough* ... and this is a terrible race condition.
Okay, well thanks for your time and help figuring out the problem..
 
Upvote 0
Sorry there isn't a better answer to the situation and no way to mittigate the problem. Synology simply oversleept to add a fix. Nothing we can do on our side.
 
Upvote 0
86
26
Sorry there isn't a better answer to the situation and no way to mittigate the problem. Synology simply oversleept to add a fix. Nothing we can do on our side.
No worries m8, I can only hope Synology will issue an update with a Fix before DSM 7.0 otherwise we'll have to wait:rolleyes:
Thanks again for helping
 
Upvote 0
1,057
353
NAS
DS418play, DS213j, DSM 7.0.4-11091
I can only hope Synology will issue an update with a Fix before DSM 7.0
It would be worthwhile to submit a support ticket to Synology. If you can link it to a security issue it may get implemented. Other than for security, I'm unaware Synology patches their kernel.
 
Upvote 0

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Welcome to SynoForum.com!

SynoForum.com is an unofficial Synology forum for NAS owners and enthusiasts.

Registration is free, easy and fast!

Trending threads

Top