Downgrade Nextcloud using Docker to solve a problem?

[DeltaGolf]

I have Nextcloud working on my Synology (DSM 6.2.4) using Docker. Yesterday I was unable to login via its website. I get an error "405 Not Allowed" which appears to be an Nginx message. I am able to login and use my Nextcloud via its iOS app. But anything via http/s seems to fail. I have another Nextcloud on another Synology which is working fine.

So what happenned? Did my updating Nextcloud via Docker break it? I want to find out by downgrading from Nextcloud:latest to, say. Nextcloud 19.

I know how to upgrade a Docker app but how to downgrade? I can download Nextcloud:19 but if I delete Nextcloud:latest am I also deleting all its settings (the Environmental Variables etc, obviously my user data is stored elsewhere)?

(Side question, where does Nginx come from? As far as I knew I'm running Apache?)

 

[fredbert]

I did this for a PostgreSQL container. This may not be the prettiest way of doing it and @one-eyed-king or @Rusty may chime it with a better way, but here's what I did.

I was using 'postgres:latest' image and had started on major version 12 when 'latest' moved to version 13. I exported the container's settings (not the full contents) to the .json file. Next I edited the file, find the 'image' line:
"image" : "postgres:latest", to
"image" : "postgres:12",

Next downloaded the matching image file from Docker Hub (using DSM Docker).

Renamed the original container. Stopped any containers that used it. Then imported the edited configuration file. I think I had to delete and redo the Links to it within the other containers. Then restarted the new and stopped containers.

I'm assuming you have configured the volumes you have persistent data in to DSM folders.

 

[DeltaGolf]

I did this for a PostgreSQL container. This may not be the prettiest way of doing it and @one-eyed-king or @Rusty may chime it with a better way, but here's what I did.

Yes, that sound complex.... You'd think I could 'roll back' to last week's / month's version of Docker and that would fix things. But I don't think I have a backup of Docker. Lesson being learned here. I don't see it in the Applications of HyperBackup. Yes. all the data is backed up, just not the actual Docker.

Unless there's another way, I will look at this solution. But given my skills, it may be just as easy to save all the data and destroy the current NextCloud Docker and start again. First question there is could I set it up again using the same settings and it will connect to the current database?

Of course, I still don't know this will solve the issue.

-- post merged: 1. Sep 2021 --

Down a rabbit hole and I'm reading here NGINX proxy manager that the Synology Proxy Manager is in fact Nginx. I asked above where my Nginx error message was coming from. Could it be a problem with my reverse proxy?

 

[Rusty]

Could it be a problem with my reverse proxy?

If you are using Synology reverse then try and poke around the error log file for it via ssh. Log in and go to /var/log/nginx/error.log and check it out for errors. Maybe you will find more details there.

I want to find out by downgrading from Nextcloud:latest to, say. Nextcloud 19.

While migrating the version of a docker image is easy (just use the older image and create a new container connecting to the same DB and volumes), in this case, you might get into trouble. Any app that runs a dedicated DB in the back usually makes updates to the DB as well.

So while you could run an older version of NC, you might get into a conflict with the DB side. Meaning, downgrading the front end side of the NC solution (web app) you will probably have to downgrade (or revert to an older backup) your database side as well.

 

[DeltaGolf]

If you are using Synology reverse then try and poke around the error log file for it via ssh. Log in and go to /var/log/nginx/error.log and check it out for errors. Maybe you will find more details there.

Seems to say a timeout. Is that passing blame to Nextcloud?

2021/09/02 09:08:30 [error] 25443#25443: *30390 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 12.34.56.78, server: nextcloud.mydomain.tld, request: "PROPFIND /remote.php/dav/principals/users/myusername/ HTTP/1.1", upstream: "http://127.0.0.1:8080/remote.php/dav/principals/users/myusername/“, host: "nextcloud.mydomain.tld"

(12.34.56.78 is a censored version of my public IP address, nextcloud.mydomain.tld has replaced my real server address and myusername, well you get the picture...)

Does this help me in any way? Should I be pointing the finger at Nextcloud? What else don't I know I don't know?

 

[Rusty]

Seems to say a timeout. Is that passing blame to Nextcloud?

2021/09/02 09:08:30 [error] 25443#25443: *30390 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 12.34.56.78, server: nextcloud.mydomain.tld, request: "PROPFIND /remote.php/dav/principals/users/myusername/ HTTP/1.1", upstream: "http://127.0.0.1:8080/remote.php/dav/principals/users/myusername/“, host: "nextcloud.mydomain.tld"

(12.34.56.78 is a censored version of my public IP address, nextcloud.mydomain.tld has replaced my real server address and myusername, well you get the picture...)

Does this help me in any way? Should I be pointing the finger at Nextcloud? What else don't I know I don't know?

From this error it seems that NC was the problem yes. Standard 110 and 111 errors from nginx would suggest that there was a problem getting a response from a service that reverse proxy contacted, in this case, NC.

 

[DeltaGolf]

From this error it seems that NC was the problem yes. Standard 110 and 111 errors from nginx would suggest that there was a problem getting a response from a service that reverse proxy contacted, in this case, NC.

What's odd is that I can access my Nextclould perfectly using the iOS app. I'm looking at its log and its connecting using the same server settings I'm trying from a desktop browser.

Of course I'm logged in on the iOS app but its the login page I can't get past on the desktop. So perhaps this is not a reverse proxy problem but a login page problem? I don't want to log out of iOS because...

 

[Rusty]

Stabbing in the dark here from my end. Personally I don't use NC so can't say for sure what can and can't be the problem, but if you can login using your mobile device that would say that the mechanics of NC is working fine.

What could be the exact problem here from the desktop side I'm not sure.

Maybe post web browser log (developer tools) when trying to access the page and there will be some details in the console view that might help detect the problem.

 

[DeltaGolf]

What could be the exact problem here from the desktop side I'm not sure.

I've worked out that the iOS version uses WebDAV. If I try to access via WebDAV on my desktop I'm told I can't do that, WebDAV is only for the app. So, as you say, Nextcloud is working its just access via HTTP/S that's the trouble.

Maybe post web browser log (developer tools) when trying to access the page and there will be some details in the console view that might help detect the problem.

I'm not seeing anything here. But I may not know what to look out for a be looking at the wrong thing.

My only recent clue is this page which says

Warning
This article is obsolete.
Sometimes after upgrading packages or NextCloud, login will fail with no apparent error.

This may be due to incorrect permissions on the PHP session directory. Fix like this, and loging in should be able again instantly:

chown nginx:nginx -R /var/lib/php/session/


I could go and look at permissions, but it may be another false trail. Would that location be somewhere to look?

 

[Rusty]

I could go and look at permissions, but it may be another false trail. Would that location be somewhere to look

That would depend how your NC was configured as a container and what parameters were defined as volumes. So the idea here is that you have a permission problem but I wouldn't say thats the case as your mobile experience works fine. Still, if the mobile access is different then the desktop one then it could end up being a problem/potential solution.

Still, as I do not use NC or have troubleshooted it in the past, I can't say for sure that this will be of any help or not.

I'm not seeing anything here. But I may not know what to look out for a be looking at the wrong thing.

Depending on your browser of choice, it will be called differently. In firefox its called "console" and looks like this:

 

[DeltaGolf]

Depending on your browser of choice, it will be called differently. In firefox its called "console" and looks like this:

When I open up Console in Safari it prevents the page from fully loading. Microsoft's Edge only tells me "login:1 Failed to load resource: the server responded with a status of 405 ()". Which we already knew...

 

[Rusty]

When I open up Console in Safari it prevents the page from fully loading. Microsoft's Edge only tells me "login:1 Failed to load resource: the server responded with a status of 405 ()". Which we already knew...

hmm and just to be sure this happens with any browser yes?

 

[DeltaGolf]

hmm and just to be sure this happens with any browser yes?

Yes. But I know Nextcloud is working, I can access my files using the iOS app (which connects via WebDAV). The web login page works, but there's no response to my sending my credentials via https.

I'm unclear if Apache or Nginx is in charge here. I don't see anything interesting in the Appache error log. Nginx's says

2021/09/03 15:37:58 [error] 18272#18272: *72268 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client: 12.34.56.78, server: nextcloud.myserver.tld, request: "PROPFIND
/remote.php/dav/principals/users/myusername/ HTTP/2.0", upstream:
"http://127.0.0.1:8080/remote.php/dav/principals/users/myusername/", host:
"nextcloud.myserver.tld"

I don't know about PROPFIND so I'll read up on that. I notice the connection to 127.0.0.1 is made over http not https. Is that usual / correct/ I know its an internal connection but I have things set up to go to https so http shouldn't work.

Or is that another red herring?

-- post merged: 3. Sep 2021 --

OK, I see PROPFIND is part of WebDAV. I'm now going to say my iOS app is working because its already logged in, not because its WebDAV. I'm not going to log out to find out.

 

[Rusty]

unclear if Apache or Nginx is in charge here

My guess is that nginx (reverse) is the problem here on some level as if it was apache that runs nc you would have much more problems.

Then again I can’t be sure as I didn’t and don’t run nc.

Have you tried posting on the NC forum?

 

[one-eyed-king]

I get an error "405 Not Allowed" which appears to be an Nginx message.

This is a http error code that indicates that a client tries to access a ressource with a method (=GET, PUT, POST, DELETE, UPDATE,...) that is not allowed by the server configuration. Typicaly seen in combination with rest-services. Not a RP problem.

Also may I suggest to post more under the hood details? Help us to help you. I am not sure if I missed which exact image you used, which ports you configured and exposed and how the reverse proxy is configured and what other configuration might be usefull to understand the flow from WAN to the container and back. Or maybe the client is just outdated and not compatible with breaking changes in the backend?

 

[DeltaGolf]

Have you tried posting on the NC forum?

Yes but no replies

 

[DeltaGolf]

Also may I suggest to post more under the hood details?

DSM 6.2.4
Using the Synology Docker and getting "nextcloud:latest" from the Docker Hub. Docker Hub
I'm not using the standard Synology ports, I changed them because, security? But access to my DSM etc works fine on the new port. I set up the Reverse Proxies using advice given here. Again, these have worked for a year or so. In fact it continues to work, I can easily access nextcloud.mydomain.tld from my browser. I have two RPs:

nextcloud-https
Source
Protocol: HTTPS
Hostname: nextcloud.mydomain.tld
Port: 443
√ Enable HSTS
√ Enable HTTP/2
Destination
Protocol: HTTP
Hostname: localhost
Port: 8080

nextcloud-http
Source
Protocol: HTTP
Hostname: nextcloud.mydomain.tld
Port: 80
Destination
Protocol: HTTPS
Hostname: nextcloud.mydomain.tld
Port: 443

Other settings at default as far as I know

But as I say, I don't think the PR is the problem. I can access my Nextcloud website. On the login page if I enter a wrong username and password I get a normal warning "wrong username or password" instantly. A correct username and password should take me to the 'files' page, a list of files I store in my NC. I never get there. So NC is running, the login part of the database is running. I can access all my files from my already logged in iOS device.

Doesn't that sound like a NC error, I'm not getting the page I request on a correct login? But nobody's helping on the NC support page...

Honestly, I'd give up, throw it all away and start again. I have all my files saved locally. But... NC is also my Calendar and Contacts server. At least one of my Calendars isn't synchronised with my desktop app so I can't 'export' it. I'm not clear where NC stores the Calendar and Contacts data - in the MariaDB database? If I could extract my data from there somehow...

 

[one-eyed-king]

Right now the 2nd rule uses http from client to nextcloud-http <-> from nextcloud-http to nextcloud-https <-> from nextcloud-https to the nc container. Thus the nextcloud-http connection your client is using, transfers data in plain text and the https security context only takes place within the reverse proxy... doesn't make much sense, does it?

What you realy want is a redirect based on a 301 or 302 status code that actualy forwards the browser (or whatever the client is) to the https url. Afair @Rusty wrote a tutorial on how to configure it.

Do the NC version that work and the NC version that doesn't work use the same image version? What if it's just a simple bug. It's not uncommon that things change in backends and frontends (=the javascript libraries in your browser embedded in the NC ui) an lead to a breakage if the implementations are not updated at the same time. Typicaly those kind of bugs are sorted out quickly.

 

[DeltaGolf]

Do the NC version that work and the NC version that doesn't work use the same image version? What if it's just a simple bug.

No, I daren't update the working NC as that really is in a production environment. Deep do do if that breaks. You'd think if there's a bug in the nextcloud:latest I'm using we'd have heard a couple more people asking questions. Either here or on the NC community forums. Also I'm clinging to the fact its not a browser issue as my desktop apps and iOS apps aren't getting synchronised (CalDAV and CardDAV).

There's a long standing 'issue' in Nextcloud (stemming from OwnCloud) regarding DAV and 'well known hosts'. That's broken for many people in many apps (macOS's Contacts I think) but who knows. As far as I know that shouldn't be an issue for a desktop browser.

Meanwhhile, as you were replying I deleted my Reverse Proxies (which I'm sure I remember setting up on advice given on these forums - so the can't be wrong

Again I can get to the login page at http://192.168.1.2:8080 but then I can get to the files page because I'm rediverted to an https page and "Safari can't establish a secure connection to the server 192.168.1.2" Now if I could that would prove/disprove the PR question. Where in DSM can I set things to allow me a secure connection - if at all possible?

-- post merged: 4. Sep 2021 --

OK, I followed this @Rusty tutorial Tutorial - Synology Reverse Proxy and I get the same result. So my RPs may have been wrong but not the wrong sort of wrong.

 

[one-eyed-king]

Also I'm clinging to the fact its not a browser issue as my desktop apps and iOS apps aren't getting synchronised (CalDAV and CardDAV).

Missed out on that part. Any updates on the client applications available? Like I wrote 405 indicates the client tries to access the server in a way the server does not allow - this is not related to user permissions, but rather how the client commuciates with the server. A rough analogy in the real word would be that they used philps head screws before and switched to torx screws, which makes your existing philps head screwdriver unusable on the new torx screws...

There's a long standing 'issue' in Nextcloud (stemming from OwnCloud) regarding DAV and 'well known hosts'. That's broken for many people in many apps (macOS's Contacts I think) but who knows. As far as I know that shouldn't be an issue for a desktop browser.

That particular set of reverse proxy rules can not be configured using the ui. Though, they can be configured via shell. I wrote a tutorial on how the integration needs to be done.

Meanwhhile, as you were replying I deleted my Reverse Proxies (which I'm sure I remember setting up on advice given on these forums - so the can't be wrong

Glad you deleted it, because it provides a wrong feeling of security, while at the same time not providing any security at all.

gain I can get to the login page at http://192.168.1.2:8080 but then I can get to the files page because I'm rediverted to an https page and "Safari can't establish a secure connection to the server 192.168.1.2" Now if I could that would prove/disprove the PR question. Where in DSM can I set things to allow me a secure connection - if at all possible?

Then it's a redirect NC implements, but still the approach is wrong as the behavior is identical as wiring nextcloud-http directly to the container. You will want to have that redirect as early as possible to make sure not even a single file is served using http.