Docker daemon eating up disk I/O

[Shadow]

I'm asking this question on behalf of someone from another (Dutch) forum.

Anyone seen this before and have any idea what is causing this?

User is only running this container, and this disk activity only seems to hapen when this container is started:
GitHub - qdm12/gluetun: VPN client in a thin Docker container for multiple VPN providers, written in Go, and using OpenVPN or Wireguard, DNS over TLS, with a few proxy servers built-in.

author if the image has no idea why this person is getting such high disk I/O when this container is started....

Original post..

 

[jeyare]

disk activity + container is frequently integrated with a network traffic

this is my attitude to new container tests (when the person doesn't have Portainer):

Network traffic check

simple dashboard:

docker stats

you can find there block I/O also

or directly with the container name:

docker stats <name>

then more deeply and straight forward (get IP of the container)

docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $INSTANCE_ID

you need the $INSTANCE_ID from the container
or

docker network ls

then

docker network inspect <name> from the 'ls'

you will get IP address of the container

then check of the traffic (in these days most important (without your data, and after)

tcpdump host <IP container address> -n -s 65535 -w /path to somewhere/wireshark-log.pcap

analyze the traffic

then find the traffic and exact process ID:

netstat -tunp | grep <port_number>

for "unexpected" events you can use:

docker events --since <'x'> --filter 'container=<name>' --format 'Type={{.Type}}  Status={{.Status}}  ID={{.ID}}'

where the <'x'> is:
- value in minutes, then use '5m'
- value in date, then use '2021-12-20'
- or a combination of them e.g. '2021-12-20T15:45:00'
and many other combinations

 

[ppeterr]

@Shadow

Thanks for the reference to this forum, and for opening the topic with my problem.
I thought it would be useful to comment here as well.

In the topic start, the problem has already been explained well. But to put it briefly again.

- DSM7 docker --> gleutun container
- wireguard in kernel or userspace

Start container and directly everey few seconds 300/400 kbs R/W IO

Together with @Shadow I found out that this is caused by the dockerD.

@jeyare Thanks a lot for your input. Although I am not new to linux and cli commands, I am not an advanced user either. But - docker stats - gave me some usefull info.

I can see the container show up at start.
And I immediately see the live stats of the suggested network traffic. I see there the initial start of the wiresguard client. It grabs some data through the tunnel, and then the traffic comes to a standstill.

When I open websites via the http proxy I see the Net IO rise. Then nothing again.
The block IO remains at zero. So, i can conclude that there is no hard drive activity at all ?
Sorry, the rest of your suggested commands are a little to advanced for me.

CONTAINER ID   NAME        CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
daed89fef666   gluetun     0.00%     339.2MiB / 3.678GiB   9.01%     7.97MB / 2.58MB   0B / 0B     0

So, this morning i deployed a new container with minimal arguments in portainer.
This gave me the same effect. Just to be sure, i stopped every other service. And
anything else that could cause HDD I/O.

Stopping the container, and the drives goes silent. Any other container does not give this effect.

Any input is welcome, and thanks a lot!

 

[Shadow]

I'm hoping to also get the view of @one-eyed-king 's expertise...

 

[jeyare]

First:

grype qmcgaw/gluetun:latest

✔ Vulnerability DB [no update available]
✔ Parsed image
✔ Cataloged packages [64 packages]
✔ Scanned image [0 vulnerabilities]
No vulnerabilities found

Second:
I'm trying to replicate the problem in my test lab, so far nothing serious
nothing suspicious there

need time to understand the logic of internal container services (responsible for the iops)

 

[Shadow]

test lab

Wat is your test lab?

Just thinking, could this issue only occur on Synology NAS'es? I know that would be absolutely against what Docker was originally designed for..

 

[ppeterr]

Wat is your test lab?

Just thinking, could this issue only occur on Synology NAS'es? I know that would be absolutely against what Docker was originally designed for..

For now, till we know. Blame the guy behind the keyboard

 

[jeyare]

it needs a time for me - now in the preparation phase for Christmas

 

[one-eyed-king]

I'm hoping to also get the view of @one-eyed-king 's expertise...

I must admit, I have no theory so far, just a hunch.

Just thinking, could this issue only occur on Synology NAS'es? I know that would be absolutely against what Docker was originally designed for..

Synology's docker is not a vanila docker. The proprietary Synology log driver is not part of the docker eco system. The log driver might be a candidate if the container itself writes plenty of logs (to the console or into the filesystem)- or if it includes a forward proxy (squid?) that caches artifacts in the container's filesystem.

N.B.: My experience is that vanila docker behavior can only be expected from a vanila docker engine (=from docker's own apt/rpm repositories). Even though forks shared the same code base at one point, doesn't necessarily mean they behave like "the original" until infity... For instance the snap docker package is so mutilated that I don't even consider it as docker.

 

[ppeterr]

Oke guys. I've not found a reason or a solution. But the IO rumble on the drives
irritated me a lot. So, i made a dirty workaround.

Migrated the docker package and the container to a 2Tb Nvme. Noise is gone, IO is still there.

Hopefully one day i will find out what i did wrong. Or, what is going on with Gluetun giving me extra IO.

Thanks for now.