Install the app
How to install the app on iOS

Follow along with the video below to see how to install our site as a web app on your home screen.

Note: This feature may not be available in some browsers.

DSM 6.2 DSM cannot start, Volume1 completely full, btrfs tasks blocked

B

BagronkeN

13
0
NAS
DS1821+
Hi,

My DS1821+ cannot boot properly after Volume1 filled up completely.

This was probably due to me running a defrag in the DSM 6 GUI, which maybe made metadata/files duplicate or otherwise fill up the disk.
I aborted the defrag, and DSM was running fine, even after a reboot, but it reported free space as 0 bytes.
This was seemingly not true, because I could still write to the volume without issue, and btrfs filesystem usage showed "Free (estimated)" as 300 GB free which was the same amount of free space I saw last before running defrag.

Searching online gave me the impression running btrfs balance could reclaim the seemingly missing space and clear up the discrepancy, but this isn't available in DSM 6.
I could not upgrade to DSM 7 since it saw Volume1 as full. (Catch 22.)

I also found many docker subvolumes that I wanted to prune (or just remove completely), which AI told me could cause a lot of metadata overhead (but might be completely false).
While backup up Docker settings, I found the DSM and NAS in general starting to run extremely slow.
I rebooted the NAS, and DSM came back online as well as SMB access, but was running even slower, and Docker didn't start fully.
After another reboot, DSM or SMB access no longer comes back.

I have SSH access, but trying to access any share under /volume1 just results in a block/time-out, for example an ls here:

Code: 
[ 4246.166601] INFO: task ls:12053 blocked for more than 120 seconds.
[ 4246.172799]       Tainted: P           O    4.4.59+ #25556
[ 4246.178285] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4246.186116] ls              D ffff880122a8bcf8     0 12053      1 0x00000004

Other tasks are getting same kind of blocks.

I'd be happy to delete a lot of data on the volume if it needs disk space to finish some task, but I can't even run rm on volume1.

Synology Assistant finds the NAS, but gets stuck in "Checking quota" since no btrfs tasks seems to be able to run.

I'm not running any snapshots or Synology Drive with version history, which seems to be culprits in disks/metadata filling up quickly.
(Hyper Backup runs, but that doesn't use snapshots from what I understand.)

Is there a way out of this state..? Any help or guidance appreciated as I'm a novice at Linux.

Disk status right now:

df -h shows "Avail" = 0 bytes remaining:

Code: 
df -h /volume1
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/vg1000-lv   74T   74T     0 100% /volume1

btrfs filesystem usage still shows "Free (estimated)" = 300 GB:

Code: 
btrfs filesystem usage /volume1
Overall:
    Device size:                  76.37TiB
    Device allocated:             76.37TiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Used:                         75.27TiB
    Free (estimated):            306.22GiB      (min: 306.22GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                2.00GiB      (used: 0.00B)

Data,single: Size:75.40TiB, Used:75.10TiB
   /dev/mapper/vg1000-lv          75.40TiB

Metadata,DUP: Size:493.01GiB, Used:86.56GiB
   /dev/mapper/vg1000-lv         986.02GiB

System,DUP: Size:8.00MiB, Used:2.08MiB
   /dev/mapper/vg1000-lv          16.00MiB

Unallocated:
   /dev/mapper/vg1000-lv           1.00MiB

Volume1 is still in rw mode at least FWIW:

Code: 
mount | grep volume1
/dev/mapper/vg1000-lv on /volume1 type btrfs (rw,noatime,ssd,synoacl,space_cache=v2,auto_reclaim_space,metadata_ratio=50,subvolid=257,subvol=/@syno)

This post includes affiliate links. As an Amazon Associate, SynoForum.com may earn a commission if you make a purchase — at no extra cost to you.
It helps support our community! Learn more...

 
Sort by date Sort by votes
Run df -h while in root, not just volume1. The potential issue here is that you run out of space on the system partition.
 
Upvote 0
Rusty said:
Run df -h while in root, not just volume1. The potential issue here is that you run out of space on the system partition.
Click to expand...

That's a good idea, but it seems to have some space.

Code: 
df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        2.3G  1.5G  729M  68% /
 
Upvote 0
Try this
Bash: 
# delete log archives
# run as root
find / -name "*.log.*.xz" | xargs -P 3 rm

What does "df -h /" show afterwards?
 
Upvote 0
Last edited:
Upgrading to DSM 7 without a migration plan was a big mistake. Modern DSM 7 versions use an 8 GB system partition, but you only have 2.4 GB coming from DSM 6.
Questions:
Do you still have a hard drive with your 6.2.4 system partition?
Which DSM version is installed now?
Do you have SHR running?
Is there a spare HDD available, which is greater, as you disk1?
 
Upvote 0
Last edited:
DSfuchs said:
Try this
Bash: 
# delete log archives
# run as root
find / -name "*.log.*.xz" | xargs -P 3 rm

What does "df -h /" show afterwards?
Click to expand...

It seemingly can't run find -- that command seems to lock even on / -- just like ls and rm commands on /volume1 does.

ls command on / works fine though, but find does not.

Code: 
ps aux | awk '$8 ~ /D/'
root        48  0.7  0.0      0     0 ?        D    13:35   0:03 [kworker/u16:1]
root       121  1.0  0.0      0     0 ?        D    13:35   0:04 [kworker/u16:4]
root     11385  0.0  0.0      0     0 ?        D    13:36   0:00 [btrfs-transacti]
root     11386  0.0  0.0  15868   576 ?        Ds   13:36   0:00 /sbin/btrfs quota enable /volume1
root     11482  0.0  0.1  62192  5712 ?        D    13:36   0:00 synologset1 sys info 0x11801321 LAN 2 169.254.164.192 255.255.0.0
root     11625  0.0  0.1  62192  5764 ?        D    13:36   0:00 synologset1 sys info 0x11801321 LAN 3 169.254.210.157 255.255.0.0
root     11924  0.0  0.1  62192  5708 ?        D    13:36   0:00 synologset1 sys info 0x11801321 LAN 4 169.254.8.109 255.255.0.0
root     12087  0.1  0.0  21712  2628 pts/12   D+   13:38   0:00 find / -name *.log.*.xz
 
Upvote 0
Rusty said:
It does, but for an upgrade, it looks like its insufficient.
Click to expand...

Yes, I was really only looking to upgrade to be able to run btrfs balance, but I'm not sure a balance is even a solution here.

Right now everything just seems locked up, seemingly due to lack of space on /volume1, but I can't clear space due to everything being locked up... 😅

I'm fine with trying a complete reinstall of DSM 7, but without having to completely clear the data from /volume1 preferably.
 
Upvote 0
DSfuchs said:
Upgrading to DSM 7 without a migration plan was a big mistake. Modern DSM 7 versions use an 8 GB system partition, but you only have 2.4 GB coming from DSM 6.
Questions:
Do you still have a hard drive with your 6.2.4 system partition?
Which DSM version is installed now?
Do you have SHR running?
Is there a spare HDD available, which is greater, as you disk1?
Click to expand...

Upgrade to DSM 7 never worked since it complained that space on volume1 was lower than ~2.6 GB, so we're still on DSM 6.2.4.

Code: 
cat /etc.defaults/VERSION
majorversion="6"
minorversion="2"
major="6"
minor="2"
micro="4"
productversion="6.2.4"
buildphase="GM"
buildnumber="25556"
smallfixnumber="8"
nano="0"
base="25556"
builddate="2024/11/25"
buildtime="14:53:46"

I think this is running Standard RAID 5 with 8 x 12 TB drives. (I thought I used SHR, but maybe that was only on my other NAS running DSM7.)

Code: 
cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
md2 : active raid5 sata1p5[0] sata8p5[7] sata7p5[6] sata6p5[5] sata5p5[4] sata4p5[9] sata3p5[8] sata2p5[1]
      81998354816 blocks super 1.2 level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

I don't have a spare 12 TB+. (Best I can do is a spare 8 TB drive.)
 
Upvote 0
Please show the whole output of
Code: 
cat /proc/mdstat
 
Upvote 0
DSfuchs said:
Please show the whole output of
Code: 
cat /proc/mdstat
Click to expand...

Code: 
cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
md2 : active raid5 sata1p5[0] sata8p5[7] sata7p5[6] sata6p5[5] sata5p5[4] sata4p5[9] sata3p5[8] sata2p5[1]
      81998354816 blocks super 1.2 level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

md1 : active raid1 sata1p2[0] sata2p2[1] sata3p2[2] sata4p2[3] sata5p2[4] sata6p2[5] sata7p2[6] sata8p2[7]
      2097088 blocks [8/8] [UUUUUUUU]

md0 : active raid1 sata1p1[0] sata2p1[1] sata3p1[2] sata4p1[3] sata5p1[4] sata6p1[5] sata7p1[6] sata8p1[7]
      2490176 blocks [8/8] [UUUUUUUU]

unused devices: <none>
 
Upvote 0
Last edited:
These are my files:
# find / -name ".log..xz"
/var/log/systemd.log.5.xz
/var/log/systemd.log.18.xz
/var/log/systemd.log.8.xz
/var/log/synoplugin.log.1.xz
/var/log/systemd.log.17.xz
/var/log/systemd.log.11.xz
/var/log/synopkg.log.6.xz
/var/log/synopkgmgr.log.3.xz
/var/log/systemd.log.2.xz
/var/log/synopkg.log.1.xz
/var/log/systemd.log.1.xz
/var/log/synopkg.log.4.xz
/var/log/systemd.log.12.xz
/var/log/systemd.log.6.xz
/var/log/systemd.log.10.xz
/var/log/systemd.log.4.xz
/var/log/systemd.log.3.xz
/var/log/systemd.log.19.xz
/var/log/systemd.log.14.xz
/var/log/synopkg.log.5.xz
/var/log/synopkgmgr.log.2.xz
/var/log/rm.log.4.xz
/var/log/systemd.log.15.xz
/var/log/synopkgmgr.log.1.xz
/var/log/systemd.log.16.xz
/var/log/systemd.log.7.xz
/var/log/rm.log.1.xz
/var/log/systemd.log.13.xz
/var/log/rm.log.3.xz
/var/log/systemd/nginx.service.log.1.xz
/var/log/synopkg.log.3.xz
/var/log/rm.log.2.xz
/var/log/synopkg.log.2.xz
/var/log/systemd.log.9.xz
/var/log/systemd.log.20.xz
/var/log/synopkg.log.7.xz

You have to enter the directory and run the following:
Bash: 
cd /var/log
rm *.log.*.xz
 
Upvote 0
Or purge all logs with:
Bash: 
rm -R /var/log
 
Upvote 0
BagronkeN said:
Upgrade to DSM 7 never worked since it complained that space on volume1 was lower than ~2.6 GB, so we're still on DSM 6.2.4.

Code: 
cat /etc.defaults/VERSION
majorversion="6"
minorversion="2"
major="6"
minor="2"
micro="4"
productversion="6.2.4"
buildphase="GM"
buildnumber="25556"
smallfixnumber="8"
nano="0"
base="25556"
builddate="2024/11/25"
buildtime="14:53:46"

I think this is running Standard RAID 5 with 8 x 12 TB drives. (I thought I used SHR, but maybe that was only on my other NAS running DSM7.)

Code: 
cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [raidF1]
md2 : active raid5 sata1p5[0] sata8p5[7] sata7p5[6] sata6p5[5] sata5p5[4] sata4p5[9] sata3p5[8] sata2p5[1]
      81998354816 blocks super 1.2 level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

I don't have a spare 12 TB+. (Best I can do is a spare 8 TB drive.)
Click to expand...
Then you will need to buy at least two 14TB HDDs to upgrade later to DSM 7. The first two drives are replaced one after the other in the process. Drives 3–8 must be removed from md0 beforehand the expansion. Subsequently, both the md0 RAID and the ext4 file system must be expanded.
 
Upvote 0
DSfuchs said:
Or purge all logs with:
Bash: 
rm -R /var/log
Click to expand...

I purged all logs. It didn't do much space-wise though, if that was what we're after.

rm.log won't delete, but it's only 857K + 54K in total.

Code: 
rm -R /var/log
rm: cannot remove ‘/var/log/rm.log’: Operation not permitted
rm: cannot remove ‘/var/log/rm.log.1.xz’: Operation not permitted
df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        2.3G  1.5G  771M  66% /
 
Upvote 0
For me a number of unnecessary packages could once again be removed to declutter the system:
Bash:
Bash: 
synopkg uninstall ActiveInsight
synopkg uninstall DhcpServer
synopkg uninstall OAuthService
synopkg uninstall ScsiTarget
synopkg uninstall SecureSignIn
synopkg uninstall SupportService
synopkg uninstall SynoAnalytics
synopkg uninstall SynoFinder
synopkg uninstall SynoOnlinePack_v2
 
Upvote 0
After that, try restarting the DiskStation.
....
So, what about the write protection on Volume 1? You might need to remove the protection via the GUI.
 
Upvote 0
DSfuchs said:
After that, try restarting the DiskStation.
....
So, what about the write protection on Volume 1? You might need to remove the protection via the GUI.
Click to expand...

With the processes locked I can't even uninstall packages to clear up the smallest of space.

Code: 
ps aux | awk '$8 ~ /D/'
root      3778  0.2  0.0      0     0 ?        D    14:44   0:03 [kworker/u16:5]
root      3899  0.0  0.0      0     0 ?        D    14:44   0:00 [kworker/u16:10]
root      7578  0.0  0.2 313608  8568 ?        Dsl  14:45   0:00 /usr/syno/sbin/synoconfd -D
root     11380  0.0  0.0      0     0 ?        D    14:45   0:00 [btrfs-transacti]
root     11381  0.0  0.0  15868   564 ?        Ds   14:45   0:00 /sbin/btrfs quota enable /volume1
root     11471  0.0  0.1  62192  5656 ?        D    14:45   0:00 synologset1 sys info 0x11801321 LAN 2 169.254.164.192 255.255.0.0
root     11612  0.0  0.1  62192  5556 ?        D    14:45   0:00 synologset1 sys info 0x11801321 LAN 3 169.254.210.157 255.255.0.0
root     11911  0.0  0.1  62192  5712 ?        D    14:45   0:00 synologset1 sys info 0x11801321 LAN 4 169.254.8.109 255.255.0.0
root     12130  0.0  0.5 322384 20160 pts/4    D+   15:03   0:00 synopkg uninstall SynoFinder

What do you mean about write protection on Volume 1? Volume 1 is still in rw mode so it's not write protected.
I have tried to set it to read only to see if that would stop any btrfs processes that's constantly locking up, but I'm not able to do that because of the locks.
And editing /etc/fstab doesn't seem to work, it seems to reset on reboot.

Code: 
mount | grep volume1
/dev/mapper/vg1000-lv on /volume1 type btrfs (rw,noatime,ssd,synoacl,space_cache=v2,auto_reclaim_space,metadata_ratio=50,subvolid=257,subvol=/@syno)
 
Upvote 0
Since all the DiskStation's HDD slots are occupied, we don't have the option of switching to larger 14TB drives. If two slots were free, we could use two small SSDs.
Alternatively, the process could be handled via an unload/reload outstage, which might take about a week.
 
Upvote 0
I thought I had read something about write protection. So, the reboot didn't help?
 
Upvote 0

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Register

Log in

Already have an account? Log in here.

Popular tags from this forum

advice backup cache configuration data disk drive ds218 ds218+ ds918+ ds920+ dsm expansion hdd migration raid shr ssd storage update upgrade upgrading ups volume

Similar threads

M
  • Question Question
DSM 7.2 Migrating DSM + Surveillance Station from DS920+ -> DS1825+ w/ New Drives
So what exactly does MA transfer? One benefit of doing the parallel setup and manual config is that I can...
Replies
2
Views
235
muzicman82
M
zoinggg
DSM 7.2.2. to 7.3.2 update: Need > 514MB free memory on system partition?
Thanks, where were these files located?
Replies
9
Views
1,319
NigeNigeNige
N
R
  • Solved
DSM Update Stuck?
Thank you Telos. I unplugged the external HD after my initial post and then left for a few hours. I...
Replies
2
Views
1,214
rlreese74
R
N
  • Question Question
Unable to create Hyper Backup task with error "This operation failed. Please in to DSM again and retry."
[Apologies in advance, this is a cross post from the official synology community forums, but SynoForum...
Replies
0
Views
451
nassar
N
MrClicky
Cannot expand volume after adding new drive
Agree. RAID 5 or SHR-1 is optimal.
2
Replies
36
Views
621
Coop777
Coop777
C
Cannot install new DS416Play
Coincidentally finds.synology.com didn't work for me either, yet my NASs are up and reachable by browser...
Replies
1
Views
505
Telos
Telos

Thread Tags

Tags Tags
btrfs dsm filesystem nas

Welcome to SynoForum.com!

SynoForum.com is an unofficial Synology forum for NAS owners and enthusiasts.

Registration is free, easy and fast!

Register

Trending content in this forum

Back
Top