DS1821+ Crashed Volumes and Unable to Log In

Currently reading
DS1821+ Crashed Volumes and Unable to Log In

60
36
drfrankenstein.co.uk
NAS
1821+, 1815+
Operating system
  1. Linux
  2. Windows
Mobile operating system
  1. Android
Last edited:
Update:
TLDR - DSM Version: 7.2-64570 (I assume also below) has a nasty bug that will effectively brick your volumes when doing a drive upgrade, make sure you have Version: 7.2-64570 Update 1 before attempting a drive swap.



Excuse the long post but want to make sure I don't miss anything.. Also I have backups of all my valuable data so not a complete loss except some Linux ISOs and potentially a lot of time!

I just hit 4TB free on my circa 36TB volume so decided to swap out a 6TB drive for a new 10TB drive, Due to the location of my NAS in a cupboard I have to shut it down to perform disk upgrades at the moment, not ideal but generally not an issue.

I normally go into Storage Manager and de-activate the drive I am removing and then perform the swap - however due to multitasking I did not do this... Not sure if this was the true cause of what is coming..

So I shutdown put the new drive in and boot back up. The login screen appears however no beep and I get the 'System is busy - try to login again later' so I waiting 30 mins and still nothing. Also no beeps to suggest any issues with drives/pools...

I SSH in and looked at DSMESG which correctly showed a missing drive from the pool and no other activity to suggest boot has been completed. So I eventually forced a login by telling DSM the boot had been completed

sudo synobootseq --set-boot-done
sudo synobootseq --is-ready

Upon login I see disaster. In all the years I have been using DSM (209j way back) I have never had this kind of complete breakdown.

Pool 1 (Volume 1) - 8 drive SHR1 with Read Only Cache
Pool 2 (Volume 2) - Single NMVE for Docker / VMs

The pool allowed me to do a repair with the 10TB drive so that completed over night but a reboot does not allow me to login to DSM without forcing the boot sequence.

So onto my question!

What are my options - how can I try to mount the main volume is it a complete loss? The second volume while it would be nice to get it going I can always pull from last nights backup.

I can dump logs etc if needed I am fairly competent with Linux so feel free to get techincal!

PrivateBin-DrFrankenstein.co.uk - DMESG

2023-06-18_18-36.png



2023-06-18_18-35.png

-- post merged: --

Sorry quick add on - I removed the NVME Cache out of precaution.
 
Yeah - I have nightly backups that go back months of critical data - I don't backup media so not to concerned with that.
However before removing the volumes hoping I could get mounted even Read Only as have a load of empty drives to offlload.

The fact the volume 2 also crashed is weird.
 
Well I’m glad to hear you can at least get back to that backup state. It does sound like some glitch and something got mid-timed with the process. What I don’t have an intuition for is did the process Bork something permanently, or is it a glitch in the rebuild process that can be somehow reset.
 
The more I play around I feel like DSM is corrupted - I tried to do a DSM reset - 4 seconds reset button wait then 4 Seconds - zero reponse, not a single beep

But a blank drive in and it boots and allows me to install DSM (not done that yet) currently digging through reddit posts on the Wayback Machine (thank goodness for the project - glad I had the Archive Team project running for the cause)
 
The more I play around I feel like DSM is corrupted - I tried to do a DSM reset - 4 seconds reset button wait then 4 Seconds - zero reponse, not a single beep

But a blank drive in and it boots and allows me to install DSM (not done that yet) currently digging through reddit posts on the Wayback Machine (thank goodness for the project - glad I had the Archive Team project running for the cause)

If it boots fine with a clean drive, it sounds like something got corrupted on the main drive. I wonder if you boot with that blank drive and put the other drives back in, could you mount/fix them that way? Apologies if that's obvious and youre already doing that, but thought it's worth making explicit.
 
Last edited:
If it boots fine with a clean drive, it sounds like something got corrupted on the main drive. I wonder if you boot with that blank drive and put the other drives back in, could you mount/fix them that way? Apologies if that's obvious and youre already doing that, but thought it's worth making explicit.
Good point - At this point I am trying anything if I don't fix today going to wipe and start over.

Edit you are onto something here!! Put 7 of the 8 drive in and the pool is detected... Going to try an Online Assemble!
2023-06-19_08-24.png

-- post merged: --

I never thought I would love the sound of that Beep, Beep, Beep... @zombiephysicist thanks for the idea!!

Just need to restore all my configs and fix the array - I may as well use the other 10TB drive which was going to be my second upgrade if I had not had all this fun..

2023-06-19_08-41.png
 
Good point - At this point I am trying anything if I don't fix today going to wipe and start over.

Edit you are onto something here!! Put 7 of the 8 drive in and the pool is detected... Going to try an Online Assemble!

-- post merged: --

I never thought I would love the sound of that Beep, Beep, Beep... @zombiephysicist thanks for the idea!!

Just need to restore all my configs and fix the array - I may as well use the other 10TB drive which was going to be my second upgrade if I had not had all this fun..

If it boots fine with a clean drive, it sounds like something got corrupted on the main drive. I wonder if you boot with that blank drive and put the other drives back in, could you mount/fix them that way? Apologies if that's obvious and youre already doing that, but thought it's worth making explicit.

Kudos to @zombiephysicist thanks for the idea!!!

My DS1019 + just survived an almost identical event like Dr_Frankenstein's. After the new 7.2 UP1, the SSD cache suddenly reported damaged/crashed ( said "due to unrepairable hardware issues" ) and the only volume of my 5 bay HDDs reported crashed, while all the HDs are Healthy.

After many attempts and reboots, I decided to start over and plan to erase and start over. A split second before i moved on, I accidently saw this post and followed the advice from Mr. zombiephysicist.

Here is what I did:

Removed disk 2-5; Only use the main drive ( disk 1 ) to boot up, with the M2. SSD still in place.
The DSM reports insufficient disks, then I pushed disk 2-5 back in. The DSM immediately recognized them as available and start Online Assemble. This indeed rebuilt the crashed volume and saved my TBs files.

Hopefully this help fellow members stuck in the same dire situations!
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Similar threads

Nas went shutdown when opening the webui i fix this error by manual insmod synobios, note solution for...
Replies
0
Views
1,145
Personal suggestion would be: vol1: all SSDs with no cache on them vol2: all HDD with nvme cache on them...
Replies
1
Views
2,352
https://kb.synology.com/en-us/DSM/tutorial/How_to_reset_my_Synology_NAS_7
Replies
8
Views
1,967
Yes, that is why you need to create two pools with 1 volume in each pool.
Replies
7
Views
3,563
With that number of drives, I'd strongly suggest switching to SHR2, or at the very least a hot spare...
Replies
12
Views
3,644

Welcome to SynoForum.com!

SynoForum.com is an unofficial Synology forum for NAS owners and enthusiasts.

Registration is free, easy and fast!

Trending threads

Back
Top