Slow copy from folder to folder

Currently reading
Slow copy from folder to folder

239
35
NAS
DS918+
Operating system
  1. macOS
Mobile operating system
  1. iOS
I'm moving some files (in some folders) from one shared folder to another. I've dragged 'n' dropped in File Station.

Currently running at ~120 KB/sec. That seems a bit slow. I thought Btfrs copied 'instantly' like Apple's APFS. I have just under 400 GB to copy, its going to take over a day?
 
What kind of data has been moved? Lot of small, large? Avg size, how much?
Is it exactly data transfer within single NAS? Not from a computer to/from NAS?
Target is a Indexed folder?

HDD Part Numbers?
All the HDDs have same part numbers? Same firmware? When you don’t know how to, use this advice for info by SSH + hdparm

Then you can test your disk performance by this guide

Then write all results here.
 
What kind of data has been moved? Lot of small, large? Avg size, how much?
Office files, mostly various text file formats some drawing files. Some JPEGs. No databases. A mix of sizes. One Shared Folder, for example, is 281 GB, 196,506 files in 26,249 folders.
Is it exactly data transfer within single NAS? Not from a computer to/from NAS?
Target is a Indexed folder?
Yes, moving, not copying, files within a single NAS. No computer involved!
HDD Part Numbers?
Four IronWolf 10 TB drives, all bought on the same day from the same supplier
All the HDDs have same part numbers? Same firmware?
Yes to both. I'd copy them here but DSM doesn't let me copy anything...
Then you can test your disk performance by this guide
While I work that out, how long would you expect my 400 GB copy to take?
 
Bunch of small files will kill performance anytime. RAID6 is not the right array for the speed of light operation. There is really high write penalty factor for I/O vs other kind of RAIDs.
Specially when you read and write small files within same RAID6 it’s slow. This kind of Array is prepared for an Archive purpose mainly = sequential writing. But you have random read and write heavy task.

Additional question set:
- what is the disk group capacity (%) utilisation?
- not answered till now - indexed target folder? You can hold the indexation process.
- send the Part no. of all HDDs used in the array.
- you can try copy same files from LAN to the NAS (just 1G of the same sample). Check the speed.
- you can move single large file (>4GB) within the Shared folders. Just for test.

Re your question about possible speed:
- there isn’t general answer for your question. Needs deep Analyze of running environment during the data transfer, ...
- I have performance first oriented operation in my primary NASes + stable enterprise drives. Then I use RAID1 as standard. My Seagate Constellation (enterprise drives) is running with 180MB/s for such data move (within the array). Even for the small docs. BTRFS as well.
- if the entire device capacity is not used for Archive purposes only I would like propose more faster Array as is RAID5/SHR and useful backup scheme. I don’t know your operation model.
 
RAID6 is not the right array for the speed of light operation. There is really high write penalty factor for I/O vs other kind of RAIDs.
Specially when you read and write small files within same RAID6 it’s slow. This kind of Array is prepared for an Archive purpose mainly = sequential writing.

I didn't choose this RAID for fast access but security. You're definition of "Archive" may be more technical or different, but some of the contents of this RAID are archive files, files which are not accessed often. But other are in current use. Some are large 20-40 MB, others 4 K. Its a compromise.

- what is the disk group capacity (%) utilisation?
It was around 30%. Now around 35% as I add more data. But it won't increase much more in the short term.
- not answered till now - indexed target folder? You can hold the indexation process.
I can't see where that setting is for the minute... I don't remember setting it up.
- send the Part no. of all HDDs used in the array.
Seagate ST10000VN0008-2JJ101 (all four the same)
- if the entire device capacity is not used for Archive purposes only I would like propose more faster Array as is RAID5/SHR and useful backup scheme. I don’t know your operation model.
I had my first experience with RAID5 and it wasn't good. A drive failed and was replaced. A second drive failed during rebuild. Yes, I had a backup but it was a very big PIA. So for my new Synologys I choose two disc redundancy. RAID10 would be sensible perhaps and faster but I read (Wikipedia) that Btrfs had it planned but not implemented.

We're storing files, some, as I say, accessed very infrequently, some daily. All sizes. Running a mail server - very light usage and backing up from Mac/PCs as well as one Synology to the other. Normal office use, not a data centre!
 
seems to be we have another example of 10TB IronWolf write cache disabled.
you need find new firmware in this page or here

re RAID5:
I don’t know background of your bad experiences with the RAID5. Many things possible. But I will pay a glass of beer, that no reason at RAID side.

re your RAID6:
- 4x10TB
- 35% utilization of total capacity = 7TB

For mentioned usage and your 4bay NAS you can use also 2x RAID1, you can get:
- really faster speed of IO than for RAID6
- really faster speed for degraded array rebuild
- tiered 2 independent disk group (diff performance for diff data)
- then you can use smaller HDDs, then you can get faster disk rebuild speed = higher availability = better variability
With smart backup plan you can get also better sleep. More music for less cost.

Btw - RAID10 with BTRFS you can create on Synology immediately, because there is different architecture (no BTRFs RAID). More you will find in Resources part.
 
seems to be we have another example of 10TB IronWolf write cache disabled.
Super! But before I go down that rabbit hole I finally followed your advice and checked the speeds.

Code:
root@MyNAS:/# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   4280 MB in  2.00 seconds = 2140.93 MB/sec

root@MyNAS:/# dd if=/dev/zero of=/tmp/output conv=fdatasync bs=384k count=1k; rm -f /tmp/output
1024+0 records in
1024+0 records out
402653184 bytes (403 MB) copied, 0.236118 s, 1.7 GB/s

root@MyNAS:/# sync ; time sh -c "dd if=/dev/zero of=testfile bs=100k count=1k && sync" ; rm testfile
1024+0 records in
1024+0 records out
104857600 bytes (105 MB) copied, 0.110215 s, 951 MB/s

real    0m0.962s
user    0m0.005s
sys    0m0.123s

I only have a Gigabit Ethernet so increasing the speed of my read or write by a few percent isn't something I need to do. Can I run the same commands on my Mac and see its results, just as a matter of interest?

Yes I do have firmware SC60. Is updating it really going to improve anything in my real world of use. I mean once everything is in place and copied, in day to day use. Updating firmware is - if even possible from a Mac - a pain.

I've set this and another off-site DS918+ up. Changing the RAID would be a really big PIA. What are the real life advantages of moving to RAID 10 for example? On a Gigabit Ethernet network and where remote workers depend on a 20 Mb upload connection?

I think I've made my bed.
 
follow your test values, I see few reasons of your troubles = really slow speed of data move within NAS shared folders

Reasons:
+ 281 GB, 196,506 files in 26,249 folders .... mentioned by you (even 400GB above)
it's about avg 1.46MB/file .... it's really painful data structure for a move between single disk group = read from same disk structure and write to same disk structure

+ based on heavy RAID6 write speed penalty (more robust parity). Each write operation requires the disks to read the data, read the first parity, read the second parity, write the data, write the first parity and then finally write the second parity. This comes out to be a six times write penalty, which is pretty dramatic = reason why is RAID6 not a right solution for "production" systems. Reason why is such RAID6 better for the backup (archive) target primary only.

+ include problematic IronWolf 10TB firmware with 99,9% possible write cache disabled.

As you can see from the test, you can get 951 MB/s for 105MB single file transfer (with 100kB block) within the disk group. But when you will use such mass of small data your performance must goes down. Especially in above mentioned environment.
 
+ include problematic IronWolf 10TB firmware with 99,9% possible write cache disabled.
I checked the Seagate site for my serial numbers and there is no firmware update for my drives.

Once the big copies are done, they're done. I don't need speed for video production, I only have a Gigabit connection. I should be fine. Thanks.
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Similar threads

  • Question
There is nothing wrong here with the setup and configuration, but the fact is that you essentially have a...
Replies
2
Views
840
Are you absolutely certain your cables are good for GB. I went so far as only use CAT6.
Replies
9
Views
914
I know this is late - and it is for your info only, but just an FYI: I bought two of these switches and...
Replies
7
Views
1,515
Welcome! To start, the switch here is probably not the issue but just to be on the safe side, check in...
Replies
1
Views
1,023
I think they all get used for data redundancy... but someone more knowledgeable can confirm
Replies
6
Views
1,354
Need to read the blog once I get some time... Interestingly...the NAS benchmark (new feature in DSM --...
Replies
12
Views
1,995
Thanks so much guys. I don't want to give up on this device just yet - my needs are pretty modest compared...
Replies
14
Views
3,217

Welcome to SynoForum.com!

SynoForum.com is an unofficial Synology forum for NAS owners and enthusiasts.

Registration is free, easy and fast!

Back
Top