DSM 6.2 DSM cannot start, Volume1 completely full, btrfs tasks blocked

[BagronkeN]

Hi,

My DS1821+ cannot boot properly after Volume1 filled up completely.

This was probably due to me running a defrag in the DSM 6 GUI, which maybe made metadata/files duplicate or otherwise fill up the disk.
I aborted the defrag, and DSM was running fine, even after a reboot, but it reported free space as 0 bytes.
This was seemingly not true, because I could still write to the volume without issue, and btrfs filesystem usage showed "Free (estimated)" as 300 GB free which was the same amount of free space I saw last before running defrag.

Searching online gave me the impression running btrfs balance could reclaim the seemingly missing space and clear up the discrepancy, but this isn't available in DSM 6.
I could not upgrade to DSM 7 since it saw Volume1 as full. (Catch 22.)

I also found many docker subvolumes that I wanted to prune (or just remove completely), which AI told me could cause a lot of metadata overhead (but might be completely false).
While backup up Docker settings, I found the DSM and NAS in general starting to run extremely slow.
I rebooted the NAS, and DSM came back online as well as SMB access, but was running even slower, and Docker didn't start fully.
After another reboot, DSM or SMB access no longer comes back.

I have SSH access, but trying to access any share under /volume1 just results in a block/time-out, for example an ls here:

[ 4246.166601] INFO: task ls:12053 blocked for more than 120 seconds.
[ 4246.172799]       Tainted: P           O    4.4.59+ #25556
[ 4246.178285] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4246.186116] ls              D ffff880122a8bcf8     0 12053      1 0x00000004

Other tasks are getting same kind of blocks.

I'd be happy to delete a lot of data on the volume if it needs disk space to finish some task, but I can't even run rm on volume1.

Synology Assistant finds the NAS, but gets stuck in "Checking quota" since no btrfs tasks seems to be able to run.

I'm not running any snapshots or Synology Drive with version history, which seems to be culprits in disks/metadata filling up quickly.
(Hyper Backup runs, but that doesn't use snapshots from what I understand.)

Is there a way out of this state..? Any help or guidance appreciated as I'm a novice at Linux.

Disk status right now:

df -h shows "Avail" = 0 bytes remaining:

df -h /volume1
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/vg1000-lv   74T   74T     0 100% /volume1

btrfs filesystem usage still shows "Free (estimated)" = 300 GB:

btrfs filesystem usage /volume1
Overall:
    Device size:                  76.37TiB
    Device allocated:             76.37TiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Used:                         75.27TiB
    Free (estimated):            306.22GiB      (min: 306.22GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                2.00GiB      (used: 0.00B)

Data,single: Size:75.40TiB, Used:75.10TiB
   /dev/mapper/vg1000-lv          75.40TiB

Metadata,DUP: Size:493.01GiB, Used:86.56GiB
   /dev/mapper/vg1000-lv         986.02GiB

System,DUP: Size:8.00MiB, Used:2.08MiB
   /dev/mapper/vg1000-lv          16.00MiB

Unallocated:
   /dev/mapper/vg1000-lv           1.00MiB

Volume1 is still in rw mode at least FWIW:

mount | grep volume1
/dev/mapper/vg1000-lv on /volume1 type btrfs (rw,noatime,ssd,synoacl,space_cache=v2,auto_reclaim_space,metadata_ratio=50,subvolid=257,subvol=/@syno)

This post includes affiliate links. As an Amazon Associate, SynoForum.com may earn a commission if you make a purchase — at no extra cost to you.
It helps support our community! Learn more...

 

[BagronkeN]

I thought I had read something about write protection. So, the reboot didn't help?

Well, I couldn't uninstall any packages so there's no change in the system since last the reboot (after deleting logs, which didn't make any changes to Volume 1).

 

[DSfuchs]

The hard drives are showing as healthy in the StorageManger GUI, aren't they?

 

[DSfuchs]

Do you have now enough space running:

# delete log archives
# run as root
find / -name "*.log.*.xz" | xargs -P 3 rm

Or is something else running that requires a little space on the system partition, to see whether it is working normaly.

 

[BagronkeN]

The hard drives are showing as healthy in the StorageManger GUI, aren't they?

As described DSM does not boot, so there is no GUI.
But yes, all drives are healthy.

 

[DSfuchs]

If nothing else works, reset the DSM version to reinstall DSM 6.2.4 as an update.

echo "majorversion="6"
minorversion="2"
buildphase="GM"
buildnumber="24922"
smallfixnumber="2"
packing="official"
packing_id="18"
derived="nano"
derived_id="1"
productversion="6.2.2"
os_name="DSM"
builddate="2019/10/30"
buildtime="07:07:54"
unique="synology_v1000_1821+"
extractsize=899608
partitionversionlimit=0
syno_supported_hwrevision="r0"" > /etc.defaults/VERSION

tye in "reboot" for the restart and watch the assistant

 

[BagronkeN]

Do you have now enough space running:

# delete log archives
# run as root
find / -name "*.log.*.xz" | xargs -P 3 rm

Or is something else running that requires a little space on the system partition, to see whether it is working normaly.

I cannot run these commands due to most processes being locked by the system, seemingly since /volume1 is completely full. / is not completely full so that's not an issue here I think.

 

[DSfuchs]

Have you been able to perform a "reboot" from the command line yet?

 

[BagronkeN]

Have you been able to perform a "reboot" from the command line yet?

I cannot use reboot command due to the process locks, but I can use
echo 1 > /proc/sys/kernel/sysrq && echo b > /proc/sysrq-trigger

So yes, the NAS has rebooted, but since we have not changed anything, there is no change to the problem.

 

[DSfuchs]

We might have to try it without the hard drives and then patch the version using the assistant. We will then plug in one (the first one, for example) and patch it using a trick.

I'll be afk for lunch for an hour in a moment.

 

[BagronkeN]

We might have to try it without the hard drive(s) and then patch the version using the assistant. We will then plug in one (the first one, for example) and patch it using a trick.

I'll be away for lunch for an hour in a moment.

Yes, I'll try booting without the drives, and insert them after boot, to see if I can access Volume1 then, or if it just locks up again.

 

[DSfuchs]

Yes, I'll try booting without the drives, and insert them after boot, to see if I can access Volume1 then, or if it just locks up again.

You will have to follow instructions.

 

[DSfuchs]

Rename a PDF file on your workstation to
"DSM-VirtualDSM_25556.pat"
Then try to install it during the update. After the error, your telnet port is opened, to which you have to telnet by the terminal program.

login:
root
101-0101

mdadm --assemble --scan
mount /dev/md0 /mnt
cd /mnt

echo "majorversion="6"
minorversion="2"
buildphase="GM"
buildnumber="24922"
smallfixnumber="2"
packing="official"
packing_id="18"
derived="nano"
derived_id="1"
productversion="6.2.2"
os_name="DSM"
builddate="2019/10/30"
buildtime="07:07:54"
unique="synology_v1000_1821+"
extractsize=899608
partitionversionlimit=0
syno_supported_hwrevision="r0"" > etc.defaults/VERSION

cd /root
umount /dev/md0
reboot and follow the assistant again
after installation of DSM 6.2.4 from here

https://global.synologydownload.com/download/DSM/release/6.2.4/25556/DSM_DS1821%2B_25556.pat

Insert HDDs 2-8 and reboot
We need to see how it behaves after the restart.

Good Speed!

 

[DSfuchs]

Back now at keyboard, and I have fleshed out the commands above.

Did you mount /dev/md0 to /mnt and patch VERSION?

 

[BagronkeN]

back now at keyboard

did you mount /dev/md0 to /mnt ?

Yes, I'll try booting without the drives, and insert them after boot, to see if I can access Volume1 then, or if it just locks up again.

Removing the drives and booting without them solved the "Checking quota" lock that occurred immediately after boot, which stopped all attempts at removing files, uninstalling packages, (most) CLI commands, as well as locking the use of the Synology Assistant.

Like @DSfuchs hinted at, this also locked both SSH and Telnet access, and his "failed upgrade trick" might have opened up that access.

But Synology Assistant could now access the NAS to do a fresh install of DSM, and it even managed to install DSM 7 on my 2.3 GB system drive.

After this, Volume1 is still at 0 bytes, but I now have regained access to the volume to be able to clear space (which was impossible before with all btrfs processes locking the kernel), and DSM can start again even with the volume at 0 bytes.

I don't know exactly what triggered the NAS to enter the deadlocked state. Maybe it's a problem limited to DSM 6 (and its older CLI commands) and the way it handles a completely full btrfs volume, that maybe has since then been fixed.

Maybe another way to have gotten past this state was a hard reset (instead of the no-disk boot and hot-insert), but I was still hoping to solve it via CLI, if I could only get past the btrfs locks by unmounting volume1 via software instead of pulling disks, and hopefully regaining access to the old DSM install before reset. But seemingly there was no way to get past those locks happening immediately after boot -- at least not in DSM 6.

Thank you @Rusty and @DSfuchs for engaging in this topic. I hope my write-up might help others in the future.

 

[DSfuchs]

"a hard reset" will leave you without applications and configuration!
So it is no option.
However, if you were to install DSM 6 according to the instructions, the problems would disappear completely. We would have also increased the potential available capacity to 3% free space.

Good Speed!

 

[Telos]

running a defrag

Never defrag DSM unless instructed by Synology.