Tutorial Synology Reverse Proxy under the hood

[jeyare]

Tried iptables (nat) re-flush and bridge reconstruction - nothing.

pkill docker   # it's better to manually shutdown each container (just to be sure)
iptables -t nat -F
ifconfig docker0 down
brctl delbr docker0
synoservice --restart pkgctl-Docker

Chain DEFAULT_OUTPUT
& Chain DOCKER
& Chain DEFAULT_POSTROUTING
were successfully recreated. But still out of reaching the containers by RP. Available from LAN (all).


Send me pls a screenshot of a single line from Chain DOCKER/DNAT for a comparison of correct definition.
One from mine:

DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:LOCAL-PORT to:CONTAINER-IP:INTERNAL-CONTAINER-PORT

 

[jeyare]

several tests was done today:

I can use telnet from any container to anywhere.
I can use telnet from any container to any container by NAS-IP:container port.
Default Docker gateway is working well.
There is just a single variable the nginx NAT.

 

[one-eyed-king]

nginx doesn't do NAT. It's not its job.

nginx simply listens on a port of the NAS, checks if the servername matches the one used in the URL, if so checks for location and serves either the content of the location or in case of a reverse proxy rule, send the request to the backend server.

If the backend server is a container, then the publishes NAS-IP:HOST port should be used, not the container-ip:container:port. You already acknowledge you don't use the later case.

If the backend server is not in the same network, than NAS the network's router is responsible to have the routes set to the other networks and point to a router that routes between the networks.

I start to think that I don't understand what you objective is.

 

[jeyare]

@one-eyed-king

I can use telnet from any container to any container by NAS-IP:container port.

still about the container (exposed) public port (not the internal container port). Sorry for the misunderstanding. No need to worry about my mental health (need to explain to my doctor) .

Sometimes is better to use paper and pencil or something like that.

This is a simplification of my point of view on the Netfilter packet flow + Nginx + Docker. I can't find any scheme on the internet; take it as my draft of the big picture.
However, it will be helpful for others to attach it (after tunning) to this resource.

Note: Yes, I understand, that the architecture is more complex. Again - this is drafted for the simplification purpose.

1. When config: server.ReverseProxy.conf is right (reconstructed from a scratch, never manually tuned)

and
2. When config: nginx.conf is right (it is working, include logging/SSL termination/... )
Note: in my case is the /etc/nginx/conf.d/main.conf empty .... is it OK?

and
3. NAS firewall is OFF (avoid possible wrong IPTABLES rules, just to be sure)

then
there is just a single possible wrong point (environment):
the Ngnix communication with the NAT definitions in IPTABLES for the docker networks (just for cases when RP was used).
No errors in Nginx Access/Error log ---> so, seems to be the Nginx is convinced that the RP is sending packets to the right destination/port.
Because all of the container targets defined by RP records are routed to a single destination ---> localhostSM-https-port
The <DSM-https-port> is the port defined for DSM UI.
But in the

iptables -t nat -L

there isn't such rule = reason why all of the containers are accessible when no RP is used.

Conclusion
It doesn't matter what is defined in the RP config (location/proxy_pass):
- localhost:public-container-port
or
- IP address of the NAS interface (both of them in my case), instead of the localhost def. (still it is landing to the localhost)
or
- http/https as destination protocol
or
- even ... I know, I know (just for a test purpose) IP address of the container (based on the Docker bridge interface), , instead of the localhost def.

All of the containers are working and accessible from LAN through NAS interfaces and their public ports.
Also accessible from WAN by direct Router NAT to NAS interfaces and their public ports (out od the RP).
There is valid SSL/TLS test of my wildcard cert and ngnix.cof + server.ReverseProxy.conf contains path to its cert/key. Then no possible SSL termination damage. As was written the same cert is working with the same SSL termination directly (out of the RP).

 

[Rusty]

Have you tried running a separate RP instance instead of built-in syno nginx?

 

[jeyare]

NPM?
yes, it works easy on my TrueNAS
But I need to understand the reason for the issue in this NAS. Because it is over my mind.
I'm starting to think about this nginx instability in conjunction with the latest DSM update 6.2.4-25556 Update 2 (two weeks ago in this NAS).
Syno 2nd level support contacted me just a few minutes ago.
[automerge]1636376885[/automerge]
and it was an interesting exercise because I found by Wireshark accidentally some strange packets sent from Syno AB4B to Amazon AWS IP .

 

[jeyare]

@Rusty
NPM container is now running on the NAS. Working as expected.
Tested containers are accessible from WAN.
I will wait for 2nd level support check of the internal nginx issue.

 

[Rusty]

Cool. Well at least you are absolutely sure its an nginx problem.

 

[one-eyed-king]

The situation seems crazy by a lot. There must be logical explaination for the missbehavior.

When I see macvlan, first thing that commes to mind is the kernel restriction that prevent macvlan child interfaces to communicate with their parent interface (unless you attach a macvlan child interface to the nas as well), but I would expect the reverse proxy to starve or close the connection, but not to serve the dsm-ui instead.

I am currious about what the l2 support will identify as root cause.

 

[jeyare]

There must be logical explaination for the missbehavior

yes:
“Everything's completely different”, said an old man.

I am currious about what the l2 support will identify as root cause.

in first round they asked “full root account” for a remote SSH to my NAS.
So I wrote them that in decided time slot, my computer with TeamViewer and a full SSH / SFTP client to the NAS will be waiting for them. Under my supervision.
Same setup as last time.

 

[jeyare]

it's been 10 days from my ticket, no action
Yes this is the Achilles’ tendon of Synology = a support.
When something is for free, resp. in cost of the box purchased it can’t work.

A consideration:
when from 6M active NAS users
just 5% = 300k will be willing to contribute 5€ monthly subscription
it will be 18M revenue for advanced support channel (direct 2nd level)
when just 10% of them yearly will ask for the support = 30k tickets issued
it means 136 tickets per working day (220 WD/y)
when 50% of them will have same topic (ofc it will be more) = 68 tickets daily
and 80% of them was resolved in the near past, it will remain = 14 tickets daily for resolving
5 days for single ticket resolving = 5d x 14tickets = 70 people in team

when team of 70 2nd level engineers + 4 coordinator will spend 15M yearly in total staff cost
they will get 18M - 15M = 3M € yearly for tuning of such services

It will cover >3k heavy issues yearly. With proper knowledge base maintenance it’s really useful for a power users.

However, at a time when the world is full of average people, this will not be provided from Syno. Because they need more time for new sh.ts discovering.

 

[jeyare]

Finally, I have a long time awaiting a deep dive check from Synology support (ticket issuing date: 2021-11-05):

One thing which I can ask you to check by yourself it's DNS resolution. Reverse Proxy will check the target link and generates the config file after DSM reboot. However, the check will fail due to DNS is not ready. To refresh nginx Reverse Proxy settings please click the save button again to regenerate the config file.

include a guide - how to setup RP:

Definitely, no one from the Syno support had read the ticket description with an understanding.

 

[robotnik]

Hey Synoforum! It's my first post I'm new here but work with my NAS for a few years now.

First of all: Thanks @one-eyed-king for the amazing tutorial on how to work with Synologys RP.

Since I am running Odoo 14.0 for our small company in docker, I was looking for a way to fix the problem, that longpolling just wouldn't work. After researching the issue, I found out that it actually was a nginx RP config Problem, since the DSM 6.2 (as well as the 7.0) UI would only allow ONE location per RP.
(Found out about that here: Longpolling problems with multiple workers · Issue #283 · odoo/docker)

But I would need root domain / as well as /longpolling.
Finding your tutorial I created a new http.{my-domain}.conf and included the settings (I already had the RP from the UI), that I found in /etc/nginx/app.d/server.ReverseProxy.conf into a new file; added the /longpolling location correctly.
Tested the config, all good, deleted the UI RP, restartet nginx, all good. The new RP works --> I actually fixed my longpolling problem with odoo. Yeay!!

BUT: DSM shot me a message saying that it cannot start, but works (of course, as long as nginx is running). I checked with sudo nginx -t and this it what I get:

nginx: [emerg] open() "/etc/nginx/conf.d/main.conf" failed (2: No such file or directory) in /etc/nginx/nginx.conf:7
nginx: configuration file /etc/nginx/nginx.conf test failed

Oh damn....Well, try not restarting my server I guess!

I can't find an answer to how this happened, do you maybe have an idea? I'm struggling here xD

 

[Rusty]

do you maybe have an idea? I'm struggling here xD

Maybe running a more feature rich RP (nginx) as a separate container and bypass the built in one? You will solve all your problems, current and future ones.

NGINX proxy manager

Run your own NGINX reverse proxy manager in Docker on your Synology NAS and open up to a lot more features then what the built-in version offers

www.blackvoid.club

 

[robotnik]

Maybe running a more feature rich RP (nginx) as a separate container and bypass the built in one? You will solve all your problems, current and future ones.

NGINX proxy manager

Run your own NGINX reverse proxy manager in Docker on your Synology NAS and open up to a lot more features then what the built-in version offers

www.blackvoid.club

Well, yeah... that's pretty good advice. Looks like the kind of container that would have saved my life in this case xD
Will definitely set that up after surviving this hickup...

Still, I am mostly wondering what would happen, if I just reboot the NAS .... would it recreate the conf.d directory?
And how could it disappear in the first place?

 

[jeyare]

nginx rp from last upgrade <DSM6> is suitable just for LAN purposes. The NgRPM is another level.
So have you lost the entire ./conf.d directory ?
or just you don’t see the directory within
/etc/nginx/ ?

Check core directory:
/usr/local/etc/nginx/conf.d/

 

[robotnik]

nginx rp from last upgrade <DSM6> is suitable just for LAN purposes. The NgRPM is another level.
So have you lost the entire ./conf.d directory ?
or just you don’t see the directory within
/etc/nginx/ ?

Check core directory:
/usr/local/etc/nginx/conf.d/

Yes! Everything's there under /usr/local/etc/nginx/conf.d/ even my RP config! So I guess I just don't see the dir under /etc/nginx
How can I restore the "visibility"? I think there is something about the Synology System I don't understand here (sorry, for being a newb)....
Could you help me on how to restore the directory @jeyare ?

Also @Rusty - funny thing I already used a tutorial of yours about a year ago to get the "Force SSL" thing working with your Virtual Host + htaccess method in Syno, which works like a charm. Anyways, in case I use NPM, I would have to Gateway-map ports 80 and 443 to different internal ports, meaning, could I still use my WebStation VirtualHost Website(s)? If I understand it right, all HTTPS 443 traffic would only be caught by the NPM Docker Container....am I missing something here?

 

[jeyare]

you need to recreate soft link

cd /etc/nginx/
# then
ls -l /usr/local/etc/nginx/conf.d conf.d

 

[robotnik]

you need to recreate soft link

cd /etc/nginx/
# then
ls -l /usr/local/etc/nginx/conf.d conf.d

Thanks a lot for the help! I should learn more about Linux Links..... also I'm wondering how the link got lost anyways!

Just there seems to be a problem...

sudo ls -l /usr/local/etc/nginx/conf.d/ conf.d

gives me

ls: cannot access conf.d: No such file or directory

Maybe it should be

ln -s /usr/local/etc/nginx/conf.d conf.d

?

EDIT:

OK I linked it, but running nginx -t I got

nginx: [emerg] cannot load certificate "/usr/syno/etc/certificate/ReverseProxy/3396f71a-c599-4f14-a1e5-f27ba0f76b88/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/usr/syno/etc/certificate/ReverseProxy/3396f71a-c599-4f14-a1e5-f27ba0f76b88/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: configuration file /etc/nginx/nginx.conf test failed

Also, when opening the RP config in the UI and changing anything in one of my RPs there, the conf.d link is destroyed again and DSM shoots an error! "DSM cannot start normally because of an error....." --> as a Notification in dsm

 

[jeyare]

seems to be - your nginx is broken more than I can expect

create the link again
and restart the nginx