DSM 6.2 DSM cannot start, Volume1 completely full, btrfs tasks blocked

[BagronkeN]

Hi,

My DS1821+ cannot boot properly after Volume1 filled up completely.

This was probably due to me running a defrag in the DSM 6 GUI, which maybe made metadata/files duplicate or otherwise fill up the disk.
I aborted the defrag, and DSM was running fine, even after a reboot, but it reported free space as 0 bytes.
This was seemingly not true, because I could still write to the volume without issue, and btrfs filesystem usage showed "Free (estimated)" as 300 GB free which was the same amount of free space I saw last before running defrag.

Searching online gave me the impression running btrfs balance could reclaim the seemingly missing space and clear up the discrepancy, but this isn't available in DSM 6.
I could not upgrade to DSM 7 since it saw Volume1 as full. (Catch 22.)

I also found many docker subvolumes that I wanted to prune (or just remove completely), which AI told me could cause a lot of metadata overhead (but might be completely false).
While backup up Docker settings, I found the DSM and NAS in general starting to run extremely slow.
I rebooted the NAS, and DSM came back online as well as SMB access, but was running even slower, and Docker didn't start fully.
After another reboot, DSM or SMB access no longer comes back.

I have SSH access, but trying to access any share under /volume1 just results in a block/time-out, for example an ls here:

[ 4246.166601] INFO: task ls:12053 blocked for more than 120 seconds.
[ 4246.172799]       Tainted: P           O    4.4.59+ #25556
[ 4246.178285] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4246.186116] ls              D ffff880122a8bcf8     0 12053      1 0x00000004

Other tasks are getting same kind of blocks.

I'd be happy to delete a lot of data on the volume if it needs disk space to finish some task, but I can't even run rm on volume1.

Synology Assistant finds the NAS, but gets stuck in "Checking quota" since no btrfs tasks seems to be able to run.

I'm not running any snapshots or Synology Drive with version history, which seems to be culprits in disks/metadata filling up quickly.
(Hyper Backup runs, but that doesn't use snapshots from what I understand.)

Is there a way out of this state..? Any help or guidance appreciated as I'm a novice at Linux.

Disk status right now:

df -h shows "Avail" = 0 bytes remaining:

df -h /volume1
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/vg1000-lv   74T   74T     0 100% /volume1

btrfs filesystem usage still shows "Free (estimated)" = 300 GB:

btrfs filesystem usage /volume1
Overall:
    Device size:                  76.37TiB
    Device allocated:             76.37TiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Used:                         75.27TiB
    Free (estimated):            306.22GiB      (min: 306.22GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                2.00GiB      (used: 0.00B)

Data,single: Size:75.40TiB, Used:75.10TiB
   /dev/mapper/vg1000-lv          75.40TiB

Metadata,DUP: Size:493.01GiB, Used:86.56GiB
   /dev/mapper/vg1000-lv         986.02GiB

System,DUP: Size:8.00MiB, Used:2.08MiB
   /dev/mapper/vg1000-lv          16.00MiB

Unallocated:
   /dev/mapper/vg1000-lv           1.00MiB

Volume1 is still in rw mode at least FWIW:

mount | grep volume1
/dev/mapper/vg1000-lv on /volume1 type btrfs (rw,noatime,ssd,synoacl,space_cache=v2,auto_reclaim_space,metadata_ratio=50,subvolid=257,subvol=/@syno)

This post includes affiliate links. As an Amazon Associate, SynoForum.com may earn a commission if you make a purchase — at no extra cost to you.
It helps support our community! Learn more...

 

[Rusty]

Run df -h while in root, not just volume1. The potential issue here is that you run out of space on the system partition.

 

[BagronkeN]

Run df -h while in root, not just volume1. The potential issue here is that you run out of space on the system partition.

That's a good idea, but it seems to have some space.

df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        2.3G  1.5G  729M  68% /

 

[DSfuchs]

Try this

# delete log archives
# run as root
find / -name "*.log.*.xz" | xargs -P 3 rm

What does "df -h /" show afterwards?

 

[DSfuchs]

Upgrading to DSM 7 without a migration plan was a big mistake. Modern DSM 7 versions use an 8 GB system partition, but you only have 2.4 GB coming from DSM 6.
Questions:
Do you still have a hard drive with your 6.2.4 system partition?
Which DSM version is installed now?
Do you have SHR running?
Is there a spare HDD available, which is greater, as you disk1?

 

[Rusty]

That's a good idea, but it seems to have some space.

df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        2.3G  1.5G  729M  68% /

It does, but for an upgrade, it looks like its insufficient.

 

[BagronkeN]

Try this

# delete log archives
# run as root
find / -name "*.log.*.xz" | xargs -P 3 rm

What does "df -h /" show afterwards?

It seemingly can't run find -- that command seems to lock even on / -- just like ls and rm commands on /volume1 does.

ls command on / works fine though, but find does not.

ps aux | awk '$8 ~ /D/'
root        48  0.7  0.0      0     0 ?        D    13:35   0:03 [kworker/u16:1]
root       121  1.0  0.0      0     0 ?        D    13:35   0:04 [kworker/u16:4]
root     11385  0.0  0.0      0     0 ?        D    13:36   0:00 [btrfs-transacti]
root     11386  0.0  0.0  15868   576 ?        Ds   13:36   0:00 /sbin/btrfs quota enable /volume1
root     11482  0.0  0.1  62192  5712 ?        D    13:36   0:00 synologset1 sys info 0x11801321 LAN 2 169.254.164.192 255.255.0.0
root     11625  0.0  0.1  62192  5764 ?        D    13:36   0:00 synologset1 sys info 0x11801321 LAN 3 169.254.210.157 255.255.0.0
root     11924  0.0  0.1  62192  5708 ?        D    13:36   0:00 synologset1 sys info 0x11801321 LAN 4 169.254.8.109 255.255.0.0
root     12087  0.1  0.0  21712  2628 pts/12   D+   13:38   0:00 find / -name *.log.*.xz

 

[BagronkeN]

It does, but for an upgrade, it looks like its insufficient.

Yes, I was really only looking to upgrade to be able to run btrfs balance, but I'm not sure a balance is even a solution here.

Right now everything just seems locked up, seemingly due to lack of space on /volume1, but I can't clear space due to everything being locked up...

I'm fine with trying a complete reinstall of DSM 7, but without having to completely clear the data from /volume1 preferably.

 

[BagronkeN]

Upgrading to DSM 7 without a migration plan was a big mistake. Modern DSM 7 versions use an 8 GB system partition, but you only have 2.4 GB coming from DSM 6.
Questions:
Do you still have a hard drive with your 6.2.4 system partition?
Which DSM version is installed now?
Do you have SHR running?
Is there a spare HDD available, which is greater, as you disk1?

Upgrade to DSM 7 never worked since it complained that space on volume1 was lower than ~2.6 GB, so we're still on DSM 6.2.4.

cat /etc.defaults/VERSION
majorversion="6"
minorversion="2"
major="6"
minor="2"
micro="4"
productversion="6.2.4"
buildphase="GM"
buildnumber="25556"
smallfixnumber="8"
nano="0"
base="25556"
builddate="2024/11/25"
buildtime="14:53:46"

I think this is running Standard RAID 5 with 8 x 12 TB drives. (I thought I used SHR, but maybe that was only on my other NAS running DSM7.)

cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
md2 : active raid5 sata1p5[0] sata8p5[7] sata7p5[6] sata6p5[5] sata5p5[4] sata4p5[9] sata3p5[8] sata2p5[1]
      81998354816 blocks super 1.2 level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

I don't have a spare 12 TB+. (Best I can do is a spare 8 TB drive.)

 

[DSfuchs]

Please show the whole output of

cat /proc/mdstat

 

[BagronkeN]

Please show the whole output of

cat /proc/mdstat

cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
md2 : active raid5 sata1p5[0] sata8p5[7] sata7p5[6] sata6p5[5] sata5p5[4] sata4p5[9] sata3p5[8] sata2p5[1]
      81998354816 blocks super 1.2 level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

md1 : active raid1 sata1p2[0] sata2p2[1] sata3p2[2] sata4p2[3] sata5p2[4] sata6p2[5] sata7p2[6] sata8p2[7]
      2097088 blocks [8/8] [UUUUUUUU]

md0 : active raid1 sata1p1[0] sata2p1[1] sata3p1[2] sata4p1[3] sata5p1[4] sata6p1[5] sata7p1[6] sata8p1[7]
      2490176 blocks [8/8] [UUUUUUUU]

unused devices: <none>

 

[DSfuchs]

These are my files:
# find / -name ".log..xz"
/var/log/systemd.log.5.xz
/var/log/systemd.log.18.xz
/var/log/systemd.log.8.xz
/var/log/synoplugin.log.1.xz
/var/log/systemd.log.17.xz
/var/log/systemd.log.11.xz
/var/log/synopkg.log.6.xz
/var/log/synopkgmgr.log.3.xz
/var/log/systemd.log.2.xz
/var/log/synopkg.log.1.xz
/var/log/systemd.log.1.xz
/var/log/synopkg.log.4.xz
/var/log/systemd.log.12.xz
/var/log/systemd.log.6.xz
/var/log/systemd.log.10.xz
/var/log/systemd.log.4.xz
/var/log/systemd.log.3.xz
/var/log/systemd.log.19.xz
/var/log/systemd.log.14.xz
/var/log/synopkg.log.5.xz
/var/log/synopkgmgr.log.2.xz
/var/log/rm.log.4.xz
/var/log/systemd.log.15.xz
/var/log/synopkgmgr.log.1.xz
/var/log/systemd.log.16.xz
/var/log/systemd.log.7.xz
/var/log/rm.log.1.xz
/var/log/systemd.log.13.xz
/var/log/rm.log.3.xz
/var/log/systemd/nginx.service.log.1.xz
/var/log/synopkg.log.3.xz
/var/log/rm.log.2.xz
/var/log/synopkg.log.2.xz
/var/log/systemd.log.9.xz
/var/log/systemd.log.20.xz
/var/log/synopkg.log.7.xz

You have to enter the directory and run the following:

cd /var/log
rm *.log.*.xz

 

[DSfuchs]

Or purge all logs with:

rm -R /var/log