I am using 18 TB drives. Currently a full disk check takes roughly 36 hours.
Iʻd like to know how to physically order the files on my drives in order to have maximum check and sync performance. By doing this I believe I can shorten the check time to 24 hours or less.
I know this question was asked once long in the past: at that time, you said it was strictly by directory order, but were considering changing it so that the files were processed in first cluster order.
After I reorder my files I assume a --force-realloc is sufficient and necessary to get the files reordered on the parity as well. (If I have two parity drives, does --force-realloc become safe by doing one parity drive before the other so if there is a failure the second parity drive is still valid? Or is it better to backup the content file and then rebuild using only one parity drive, then add the 2nd parity drive after the 1-drive parity reconstruction is complete?)
The goal is to have the fastest transfer rate files and parity all in sync rather than reading/writing to slow parts of some drives and fast parts of others.
Thanks.
Last edit: Chiem Ma 2021-12-15
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The files are ordered by the physical address of the first sector of the file. This is from version 4.0.
It's always better to use --force-realloc than rebuilding the parity, because with force-realloc you keep the file hash already computed.
Anyway, before doing so, you can do a sync with the --test-io-stats option that will show some runtime stats on which is the disk that slow down the process. If it's already a parity disk, the reallocation won't have any benefit.
Instead, you can try to increase the number of buffers used for each disk. Like with:
--test-io-cache=128
Worth to try also the --test-io-advise-* options. There is some information in the HISTORY file about them.
Ciao,
Andrea
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am using 18 TB drives. Currently a full disk check takes roughly 36 hours.
Iʻd like to know how to physically order the files on my drives in order to have maximum check and sync performance. By doing this I believe I can shorten the check time to 24 hours or less.
I know this question was asked once long in the past: at that time, you said it was strictly by directory order, but were considering changing it so that the files were processed in first cluster order.
After I reorder my files I assume a --force-realloc is sufficient and necessary to get the files reordered on the parity as well. (If I have two parity drives, does --force-realloc become safe by doing one parity drive before the other so if there is a failure the second parity drive is still valid? Or is it better to backup the content file and then rebuild using only one parity drive, then add the 2nd parity drive after the 1-drive parity reconstruction is complete?)
The goal is to have the fastest transfer rate files and parity all in sync rather than reading/writing to slow parts of some drives and fast parts of others.
Thanks.
Last edit: Chiem Ma 2021-12-15
The files are ordered by the physical address of the first sector of the file. This is from version 4.0.
It's always better to use --force-realloc than rebuilding the parity, because with force-realloc you keep the file hash already computed.
Anyway, before doing so, you can do a sync with the --test-io-stats option that will show some runtime stats on which is the disk that slow down the process. If it's already a parity disk, the reallocation won't have any benefit.
Instead, you can try to increase the number of buffers used for each disk. Like with:
--test-io-cache=128
Worth to try also the --test-io-advise-* options. There is some information in the HISTORY file about them.
Ciao,
Andrea