My sync used to run at 100~170MiB/s and I could run a sync in an hour to two.
I recently started backing up a large archive with a lot of individual small zip archives and the number of these small zip archives is only going to grow.
Now my sync starts at 170MiB/s but after 20 minutes is reduced to 20MiB/s and soon after drops to 10MiB/s with a starting ETA of 60 hours :\
Some info:
PC: i5 750, 16gb ram, 4x3tb (1 parity), 2x2tb, no data disk over 75%, ubuntu 14.04 on 16gb usb not backed up.
Four CPU cores runs at 100% during sync.
Disk usage thrashes high and low during sync, it's inconsistent.
My parity partition is ntfs, I will convert to ext4 with -T largefile4, but first I need to finish sync, spit parity file up, shuffle parts around to reformat the ntfs to ext4
Status result:
WARNING! With 5 disks it's recommended to use two parity levels.
Self test...
Loading state from /mnt/disk2/snapraid.content...
269583 files
27050398 blocks
0 hardlinks
0 symlinks
527 empty dirs
Using 1185 MiB of memory.
I suggest you investigate what is causing your "Four CPU cores to run at 100% during sync".
Ideally, you want SnapRAID to run in disk-throughput limited mode (not CPU-limited), and you want to have your drives reading and writing as fast as they are capable of (and entirely or mostly sequential reads and writes if possible).
If your CPU is maxed out, then you are not running in disk-throughput limited mode. However, since SnapRAID is single-threaded, it cannot be responsible for maxing out 4 CPU cores.
The first thing I would look at would be reading and writing to NTFS. I don't know if that is capable of maxing out multiple CPU cores, but I have heard that linux can be quite inefficient at NTFS throughput.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The NTFS issue might be the culprit.
In htop the top cpu% users are my mounts using
/sbin/mount.ntfs-3g
I didn't have the problem before, but maybe now opening and closing all the files is causing the issue.
I kept the data drives as ntsf so I could pull them out and mount them on my other windows machines to transfer files much faster than through a network. I'm currently at about 50% disk usage. Shuffling everything around to convert to ext4 is going to be a massive undertaking :\
Thanks for the help jwill
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello,
My sync used to run at 100~170MiB/s and I could run a sync in an hour to two.
I recently started backing up a large archive with a lot of individual small zip archives and the number of these small zip archives is only going to grow.
Now my sync starts at 170MiB/s but after 20 minutes is reduced to 20MiB/s and soon after drops to 10MiB/s with a starting ETA of 60 hours :\
Some info:
PC: i5 750, 16gb ram, 4x3tb (1 parity), 2x2tb, no data disk over 75%, ubuntu 14.04 on 16gb usb not backed up.
Four CPU cores runs at 100% during sync.
Disk usage thrashes high and low during sync, it's inconsistent.
My parity partition is ntfs, I will convert to ext4 with -T largefile4, but first I need to finish sync, spit parity file up, shuffle parts around to reformat the ntfs to ext4
Status result:
WARNING! With 5 disks it's recommended to use two parity levels.
Self test...
Loading state from /mnt/disk2/snapraid.content...
269583 files
27050398 blocks
0 hardlinks
0 symlinks
527 empty dirs
Using 1185 MiB of memory.
Files: 269583
Fragmented files: 322
Excess fragments: 1096
Files size: 6556 GiB
Parity size: 2153 GiB
I suggest you investigate what is causing your "Four CPU cores to run at 100% during sync".
Ideally, you want SnapRAID to run in disk-throughput limited mode (not CPU-limited), and you want to have your drives reading and writing as fast as they are capable of (and entirely or mostly sequential reads and writes if possible).
If your CPU is maxed out, then you are not running in disk-throughput limited mode. However, since SnapRAID is single-threaded, it cannot be responsible for maxing out 4 CPU cores.
The first thing I would look at would be reading and writing to NTFS. I don't know if that is capable of maxing out multiple CPU cores, but I have heard that linux can be quite inefficient at NTFS throughput.
The NTFS issue might be the culprit.
In htop the top cpu% users are my mounts using
/sbin/mount.ntfs-3g
I didn't have the problem before, but maybe now opening and closing all the files is causing the issue.
I kept the data drives as ntsf so I could pull them out and mount them on my other windows machines to transfer files much faster than through a network. I'm currently at about 50% disk usage. Shuffling everything around to convert to ext4 is going to be a massive undertaking :\
Thanks for the help jwill