DSM 6.2 DSM cannot start, Volume1 completely full, btrfs tasks blocked

[BagronkeN]

Hi,

My DS1821+ cannot boot properly after Volume1 filled up completely.

This was probably due to me running a defrag in the DSM 6 GUI, which maybe made metadata/files duplicate or otherwise fill up the disk.
I aborted the defrag, and DSM was running fine, even after a reboot, but it reported free space as 0 bytes.
This was seemingly not true, because I could still write to the volume without issue, and btrfs filesystem usage showed "Free (estimated)" as 300 GB free which was the same amount of free space I saw last before running defrag.

Searching online gave me the impression running btrfs balance could reclaim the seemingly missing space and clear up the discrepancy, but this isn't available in DSM 6.
I could not upgrade to DSM 7 since it saw Volume1 as full. (Catch 22.)

I also found many docker subvolumes that I wanted to prune (or just remove completely), which AI told me could cause a lot of metadata overhead (but might be completely false).
While backup up Docker settings, I found the DSM and NAS in general starting to run extremely slow.
I rebooted the NAS, and DSM came back online as well as SMB access, but was running even slower, and Docker didn't start fully.
After another reboot, DSM or SMB access no longer comes back.

I have SSH access, but trying to access any share under /volume1 just results in a block/time-out, for example an ls here:

[ 4246.166601] INFO: task ls:12053 blocked for more than 120 seconds.
[ 4246.172799]       Tainted: P           O    4.4.59+ #25556
[ 4246.178285] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4246.186116] ls              D ffff880122a8bcf8     0 12053      1 0x00000004

Other tasks are getting same kind of blocks.

I'd be happy to delete a lot of data on the volume if it needs disk space to finish some task, but I can't even run rm on volume1.

Synology Assistant finds the NAS, but gets stuck in "Checking quota" since no btrfs tasks seems to be able to run.

I'm not running any snapshots or Synology Drive with version history, which seems to be culprits in disks/metadata filling up quickly.
(Hyper Backup runs, but that doesn't use snapshots from what I understand.)

Is there a way out of this state..? Any help or guidance appreciated as I'm a novice at Linux.

Disk status right now:

df -h shows "Avail" = 0 bytes remaining:

df -h /volume1
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/vg1000-lv   74T   74T     0 100% /volume1

btrfs filesystem usage still shows "Free (estimated)" = 300 GB:

btrfs filesystem usage /volume1
Overall:
    Device size:                  76.37TiB
    Device allocated:             76.37TiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Used:                         75.27TiB
    Free (estimated):            306.22GiB      (min: 306.22GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                2.00GiB      (used: 0.00B)

Data,single: Size:75.40TiB, Used:75.10TiB
   /dev/mapper/vg1000-lv          75.40TiB

Metadata,DUP: Size:493.01GiB, Used:86.56GiB
   /dev/mapper/vg1000-lv         986.02GiB

System,DUP: Size:8.00MiB, Used:2.08MiB
   /dev/mapper/vg1000-lv          16.00MiB

Unallocated:
   /dev/mapper/vg1000-lv           1.00MiB

Volume1 is still in rw mode at least FWIW:

mount | grep volume1
/dev/mapper/vg1000-lv on /volume1 type btrfs (rw,noatime,ssd,synoacl,space_cache=v2,auto_reclaim_space,metadata_ratio=50,subvolid=257,subvol=/@syno)

This post includes affiliate links. As an Amazon Associate, SynoForum.com may earn a commission if you make a purchase — at no extra cost to you.
It helps support our community! Learn more...

 

[Rusty]

Run df -h while in root, not just volume1. The potential issue here is that you run out of space on the system partition.

 

[BagronkeN]

Run df -h while in root, not just volume1. The potential issue here is that you run out of space on the system partition.

That's a good idea, but it seems to have some space.

df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        2.3G  1.5G  729M  68% /