What should I do? Existing drive dying midway through SHR1 expansion...

[TonyS]

Hi everyone,

I've added three 6TB drives to an existing 6+6+6+8+8 SHR1 array on my DS1819+ (I read now that adding one at a time is recommended, but that bird flew a couple of days ago unfortunately).

Its about 50% done two days in, but now one of the existing 6TB drives in the array is throwing repeated bad sector errors. Its up to 30 or so, and is clearly dying.

What should I do? Let it finish the expansion while hoping for the best, or stop that somehow and replace the dying drive? I have an extra spare 6TB drive onhand, that's no problem.

Thanks for any advice!

Tony

 

[Rusty]

Let it finish and see the outcome. Considering that you are running 1 drive redundancy array you can’t afford to mess with it atm.

 

[Robbie]

I agree as it is a case of 'we are where we are' and everything is best left undisturbed.

Do you happen to have the data backed-up somewhere else, or have you gone 'all in' with your raid roulette?

 

[TonyS]

Hi everyone,

I've added three 6TB drives to an existing 6+6+6+8+8 SHR1 array on my DS1819+ (I read now that adding one at a time is recommended, but that bird flew a couple of days ago unfortunately).

Its about 50% done two days in, but now one of the existing 6TB drives in the array is throwing repeated bad sector errors. Its up to 30 or so, and is clearly dying.

What should I do? Let it finish the expansion while hoping for the best, or stop that somehow and replace the dying drive? I have an extra spare 6TB drive onhand, that's no problem.

Thanks for any advice!

Tony

Just an update for anyone facing the same sort of issue.

Advice to leave it alone was well founded. The 1819+ waded through through this issue like a champ, simultaneously continuing its SHR1 expansion while also failing the dying drive and removing it from the storage pool. With the expansion finished the next day, the NAS went into "degraded" mode and invited me to replace the broken drive for a rebuild, which I have done. All indications are we have dodged this bullet, which is great obviously. A few learnings for others:-
1. DSM has a setting to give operations like expansion a greater processor share to complete faster, at the potential risk of reducing NAS performance meanwhile. This is pretty worthwhile.
2. Despite what turned out to be THIRTY new failed sectors in the days previous to the expansion being initiated the DSM's default health status report, both for the array and the individual drive, remained "Healthy". Next time I will definitely check the logs for each drive manually first. I have been running professional video editing facilities for many years; in my experience modern drives don't have bad sectors at all until they are about to fail... that's definitely the house standard here anyway, any drive that starts throwing sectors is immediately replaced. Is there a way to greatly increase the sensitivity of failure notifications?
3. As advised by my kind advisors, if this sort of drama is happening to you, its best to just leave it alone and let the Synology solve the issue. My experience is very encouraging about the quality of the underlaying programming, and the system has remained robust even under duress. Even loosing the UI just meant it was working hard; just leave it alone.

Thanks everyone!

Tony

 

[Telos]

any drive that starts throwing sectors is immediately replaced

Maybe... if these are "hard" failures... but if "soft" failures, the door to investigation is open.

 

[Robbie]

I think most of us have earned our scars with this stuff in the past but the explosion in capacity of HDDs without a corresponding increase in transfer rate has made expansion and raid rebuilds a very dangerous game. We buy capacity the size of a house and then try and empty all the contents through the letterbox when the flood arrives.

I don't touch RAID 5 / SHR 1 on HDDs and even RAID 6 / SHR 2 has the pulse racing when the stress of a rebuild hits. I still use SHR 1 on SSD arrays though with archived stuff going to 2 single-disk spinners on different machines and with different formatting.

Any storage that requires multiple hours or even days to restore or rebuild is just not worth it. Back in the old work data centre everything was clean and certainly nothing allowed that could be a contaminant. But for a major rebuild the formal work instruction noted that whisky could be brought into the centre. Whoever wrote those rules knew his stuff.

Glad this worked out ok for you.