Cannot login into my NAS nor shutdown it safely

[daghemo]

Hi all,
I spent a life with QNAP, but some months ago I've bought a RackStation 1219+ and started to play with 3 IronWolf 8 TB disks from Seagate... and I was quite satisfied with that!

Some days ago a PC may have filled /volume1 because of a True Image backup via Samba/Windows share. I don't know if this may be the root cause, but since then I was not able to login again into my NAS, either via web or SSH. Both login just timeout! And this is tru for both admin and my own user.

That is, I can ping the IP address of my NAS and also Nmap reveals a lot of ports are listening:

• Ports 22 and 2222 for OpenSSH 7.4.
• Ports 80, 443, 8080 and 8443 for nginx.
• Port 873 for rsync.
• Ports 139 and 445 for Samba smbd 3.x-4.x.
• Port 3261 for Sysnology DSM Snapshot Replication iSCSCI LUN.
• Port 5566 for Synology Snapshot Replication shared folder.
• Port 3493 for Network UPS Tools upsd 2.6.1.
• Port 5510 for secureidprop.

On port 8443 I may see a Synology-branded page with "Sorry, the page you are looking for is not found." from time to time, but I usually just can't connect.
I've also tried to connect via the RS232 but I have the same problem: the login timeouts afeter 60 seconds.

But I can actually browse my files via Windows share! Since I do this after authenticating with my admin credentials, I can say that both user and password are fine. But no SSH, web and RS232 access!

How can I get back in control of my NAS? How can I safely shutdown/reboot?
I've opened a support case, but... not much of help...

I've tried to press the power button to start the shutdown. I have not heard a "beep", but the power LED is now blinking. But the system is still powered of after many hours.

I've done the same with the reset button, but, again, I have not heard a "beep" and it just seems not to work.

Any suggestion?

Obviously I cannot accept to lose my data, nor I want to buy another NAS with other disks to try to backup the first one.

 

[Rusty]

synology assistant? What’s reporting?

 

[daghemo]

Synology Assistant just reports the same thing. That is, it sees my Synology, but it says it is offline.
Same result from the DS Finder on my smartphone.

 

[Rusty]

offline... well, resetting DSM (using the reset button on the back and a KB described method) you can retain your data and still get DSM up and running by reinstalling it.

 

[daghemo]

I've already tried that, but with no success at all, as I wrote.
Long pressing power or reset button not worked for me! I was not able to hear a "beep" when pressing then, but after pressing the power button for a while it started to blink as expected.
I'm not sure, but it seems that the CPU is exhausted, so the NAS cannot react to some events.

 

[jeyare]

then there is just last try, that help to all equipment dependents from electricity. Power of by cable unplugging. you can still regenerate your disk by disk rescue technique. then on.

 

[Rusty]

also... you can try and power on the nas without any drives just to see if it will boot at all

 

[jeyare]

also... you can try and power on the nas without any drives just to see if it will boot at all

good point m8,
1. when DSM will start and will be reachable, then step by step put your storage pool by storage pool (to same bay) and check status
2. when DSM is still not reachable, hard reset of the NAS (by reset button)
3. when point no. 2 in any case will fall to same problem then it is time to ask for a replacement of the NAS (guarantee)
4. when point no. 2 will make your DSM reachable then use point no. 1

 

[daghemo]

Yes, this is exactly what I will do.
Maybe I will just buy another disk before, in order to backup some data elsewhere.
Thanks @jeyare and @Rusty!

But I'm curious... is this kind of problem, well, frequent? Or is this first time you hear of it?

It sounds very strange to me... and it smell like a big problem on a NAS, where big disks may suffer URE and a RAID rebuild may be problematic...

 

[Rusty]

But I'm curious... is this kind of problem, well, frequent? Or is this first time you hear of it?

Not the one I hear often (if at all). Personally, I have 5 syno NAS (58 bays in total) and never had this problem (so far).

 

[jeyare]

same as Rusty, never had such problem, oldest machine HA cluster is in 6+ years in 7x24x365
look, sometime is not great day for our hw/sw
send us a feedback, when you will finished
all fingers crossed!!

PS: I remember for one of similar problem, it was my fault when I accidentally setup port aggregation between two Ethernet ports on my Unifi switch for single NAS, but with physical connection of two Ethernets to two different NASs. Then the NAS was totally out of reaching by browser or all Syno possible way. Idiots everywhere , me too. It was hard 10 minutes of audit of all possible scenarios.

 

[notrui]

Hi all,
I spent a life with QNAP, but some months ago I've bought a RackStation 1219+ and started to play with 3 IronWolf 8 TB disks from Seagate... and I was quite satisfied with that!

Some days ago a PC may have filled /volume1 because of a True Image backup via Samba/Windows share. I don't know if this may be the root cause, but since then I was not able to login again into my NAS, either via web or SSH. Both login just timeout! And this is tru for both admin and my own user.

That is, I can ping the IP address of my NAS and also Nmap reveals a lot of ports are listening:

• Ports 22 and 2222 for OpenSSH 7.4.
• Ports 80, 443, 8080 and 8443 for nginx.
• Port 873 for rsync.
• Ports 139 and 445 for Samba smbd 3.x-4.x.
• Port 3261 for Sysnology DSM Snapshot Replication iSCSCI LUN.
• Port 5566 for Synology Snapshot Replication shared folder.
• Port 3493 for Network UPS Tools upsd 2.6.1.
• Port 5510 for secureidprop.

On port 8443 I may see a Synology-branded page with "Sorry, the page you are looking for is not found." from time to time, but I usually just can't connect.
I've also tried to connect via the RS232 but I have the same problem: the login timeouts afeter 60 seconds.

But I can actually browse my files via Windows share! Since I do this after authenticating with my admin credentials, I can say that both user and password are fine. But no SSH, web and RS232 access!

How can I get back in control of my NAS? How can I safely shutdown/reboot?
I've opened a support case, but... not much of help...

I've tried to press the power button to start the shutdown. I have not heard a "beep", but the power LED is now blinking. But the system is still powered of after many hours.

I've done the same with the reset button, but, again, I have not heard a "beep" and it just seems not to work.

Any suggestion?

Obviously I cannot accept to lose my data, nor I want to buy another NAS with other disks to try to backup the first one.

Hi.
Can you now have web acess to your NAS ?
I have an ds418 and i cant have web acess to phpmyadmin with chrome, so i tested and it works with firefox, edge.
Try with a different browser in order to have web acess.

 

[daghemo]

@jeyare, me too I have aggregated all 4 GbE ports from my RS1219+, but it is OK: I can ping it and I get a valid SSH prompt too.
IMHO, something is exahusting the CPU... or maybe the memory... that is, I can almost complete the login, but the system cannot start my shell... well, it could also be the root filesystem being full, thus making some logging operations fail and the login too.
Again, IMHO, we have a Linux here, ad a it should not be too hard to make these things not happen.

@notrui, no web access and it's not a matter of browser. As I wrote, both web, SSH and RS232 login files because of timeout.

@Rusty and @jeyare, I've just had another idea (but I have to wait this evening to try): SysRq! Do you think that they are supported out of the box on recent DSMs on my RS1219+? That is, I may try again with a RS232 cable (just to see the output) and a keyboard attached to the USB... and i can try to Reboot Even If System Utterly Broken.

 

[Rusty]

SysRq! Do you think that they are supported out of the box on recent DSMs on my RS1219+?

Good question but I can't say. Never tried it and never had the need for it. Guessing it might work, but if you connect with 232 I'm sure you will find out really quick.

 

[jeyare]

Never tried,
my Ubuntu is running in Oracle VM box, the I have more possible ways

 

[jeyare]

Good luck and don’t forget to inform us

 

[daghemo]

@Rusty, @jeyare, just try to guess.

 

[jeyare]

forgotten small guy from synology support, locked in your NAS (Men in Black) fell asleep, then your DSM has been unreachable. Reasonable

 

[daghemo]

Thumbs up for @jeyare!

Hooked a keyboard to an USB port on my RS1219+ and my computer via the RS232 port.
After almost three hours of meditation I decided to go with AL+SysRq+I in order to kill all non-init processes on my NAS. On the RS232 I saw a lot of processes being killed. As expected. Now I can get into my NAS via SSH. Or serial. Not via web, but this seems reasonable to since, just because I've killed a lot of processes, for real.

The / filesystem was fine. That is, it was not full. It was just some crazy process keeping my system busy, maybe exhausting the CPU or eating up all available inodes. Don't know.

I've not rebooted my system, yet.
Have you got any idea what loglife/whatever I can investigate into?

 

[Rusty]

All logs are in /var/logs. Focus on messages 1st