Very few users will see this behavior, but, with some stripped-down, or mis-configured, Linux kernels, the initial (and subsequent) sync run will appear to hang. Right after snapraid reports "Syncing…", it just stalls. It doesn't quickly proceed to display its progress. Not only that, but ^C (control-C) has no effect. However, you do notice there is still disk activity. You can interrupt it with ^\ (control-backslash).
If you never saw this, great! You never will! If you did, and were really(!!) patient, it would eventually recover and complete the sync just fine; but the (mis)behavior would recur on subsequent sync runs.
For anyone affected, the quick-fix/workaround is simple:
1) edit the file config.h and comment out (or just delete) the line
#define HAVE_POSIX_FALLOCATE 1
2) delete the file parity.o (the .o file not the .c or .h file)
3) rebuild snapraid with the command "make" and then re-install with "sudo make install"
I'll send the gory details to Andrea. Though the glitch is rare, the permanent fix is simple, and it will prevent a few new snapraid users from leaving at the start. "You don't get a second chance to make a good first impression."
-UhClem
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I forgot to mention (above post) that this behavior took place with the parity file on an ext2/ext3 filesystem. The prolonged delay (apparent hang) would be 5-6 hours if the eventual parity file was ~2TB!! Then, once the sync actually started to progress, it would take an additional 6+ hours (depending on how macho your system is) to do all the hashing and parity generation (normal behavior/performance).
Am I correct in assuming that most others (with Linux, and parity on an ext2/ext3 fs) did not experience the loooong delay between "Syncing…" and the Progress_Display during their initial sync run???
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
A little more testing and it looks like ext2 and ext3 have this problem for the filesystem containing parity. Also, it has nothing to do with stripped-down or mis-configured kernels.
The kernel I am using, initially, for snapraid is a small footprint one, with only ext2 support, but I tried a simple test program using posix_fallocate() under a full-function 2.6.37 kernel, and both ext2 and ext3 exhibit the slow-extend problem with posix_fallocate(). I also ran the test on ext4 and it worked just fine.
Although I haven't tried it, I believe that statfs() can be used to detect that parity is on a ext2 or ext3 filesystem, and, if so, the extension of the parity file can be accomplished, quickly, using ftruncate(); for non-ext2/ext3, posix_fallocate() can be used as presently.
I don't know anything about JFS, but if it includes a decent implementation of ftruncate(), it too can be accomodated by including it in the statfs() test for ext2 and ext3.
-UhClem
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You are correct! It's really ext3 that doesn't support the posix_fallocate() operation. I was fooled by the fact that this operation automatically fallback to a slow workaround. Really too slow for a big parity file.
I'm now thinking to switch to the specific Linux fallocate() that should fail, and use ftruncate() as fall back.
Ciao,
Andrea
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
After re-thinking my statfs() approach, I was going to suggest exactly what you describe.
Note that fallocate() does NOT set errno, but a non-zero return will be the ERRNUM. Other than ENOSPC, I found other return values confusing. In one case, I got ret=95; something about "protocol". (Someone should correct "RETURN VALUE" section for various man pages for fallocate.)
The fallocate() … ftruncate() approach has the added advantage of not being tightly coupled to specific filesystems. For example, if ext2/ext3 are enhanced with a working fallocate() (unlikely), an existing snapraid binary (with the discussed mod) will take advantage of it.
UhClem
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm using Win 7x64 and I also have alarge delay in the Sync process (using 1.7)
It got to 52% of a process and then the screen looked like it was locked,
But the Parity drive was being written to really slowly,
But none of the source disks where being read.
Could this a manifestation of the above but on NTFS (Win 7) File systems?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Very few users will see this behavior, but, with some stripped-down, or mis-configured, Linux kernels, the initial (and subsequent) sync run will appear to hang. Right after snapraid reports "Syncing…", it just stalls. It doesn't quickly proceed to display its progress. Not only that, but ^C (control-C) has no effect. However, you do notice there is still disk activity. You can interrupt it with ^\ (control-backslash).
If you never saw this, great! You never will! If you did, and were really(!!) patient, it would eventually recover and complete the sync just fine; but the (mis)behavior would recur on subsequent sync runs.
For anyone affected, the quick-fix/workaround is simple:
1) edit the file config.h and comment out (or just delete) the line
2) delete the file parity.o (the .o file not the .c or .h file)
3) rebuild snapraid with the command "make" and then re-install with "sudo make install"
I'll send the gory details to Andrea. Though the glitch is rare, the permanent fix is simple, and it will prevent a few new snapraid users from leaving at the start. "You don't get a second chance to make a good first impression."
-UhClem
I forgot to mention (above post) that this behavior took place with the parity file on an ext2/ext3 filesystem. The prolonged delay (apparent hang) would be 5-6 hours if the eventual parity file was ~2TB!! Then, once the sync actually started to progress, it would take an additional 6+ hours (depending on how macho your system is) to do all the hashing and parity generation (normal behavior/performance).
Am I correct in assuming that most others (with Linux, and parity on an ext2/ext3 fs) did not experience the loooong delay between "Syncing…" and the Progress_Display during their initial sync run???
Which is it? ext2 or ext3?
I've tested snapraid with the parity file on ext4, XFS and JFS, but fallocate on JFS is very slow. XFS and ext4 are fine.
According to the FAQ
http://snapraid.sourceforge.net/faq.html
ext3 should be fine also. But the FAQ does not say anything about ext2.
A little more testing and it looks like ext2 and ext3 have this problem for the filesystem containing parity. Also, it has nothing to do with stripped-down or mis-configured kernels.
The kernel I am using, initially, for snapraid is a small footprint one, with only ext2 support, but I tried a simple test program using posix_fallocate() under a full-function 2.6.37 kernel, and both ext2 and ext3 exhibit the slow-extend problem with posix_fallocate(). I also ran the test on ext4 and it worked just fine.
Although I haven't tried it, I believe that statfs() can be used to detect that parity is on a ext2 or ext3 filesystem, and, if so, the extension of the parity file can be accomplished, quickly, using ftruncate(); for non-ext2/ext3, posix_fallocate() can be used as presently.
I don't know anything about JFS, but if it includes a decent implementation of ftruncate(), it too can be accomodated by including it in the statfs() test for ext2 and ext3.
-UhClem
Hi uhclean,
You are correct! It's really ext3 that doesn't support the posix_fallocate() operation. I was fooled by the fact that this operation automatically fallback to a slow workaround. Really too slow for a big parity file.
I'm now thinking to switch to the specific Linux fallocate() that should fail, and use ftruncate() as fall back.
Ciao,
Andrea
After re-thinking my statfs() approach, I was going to suggest exactly what you describe.
Note that fallocate() does NOT set errno, but a non-zero return will be the ERRNUM. Other than ENOSPC, I found other return values confusing. In one case, I got ret=95; something about "protocol". (Someone should correct "RETURN VALUE" section for various man pages for fallocate.)
The fallocate() … ftruncate() approach has the added advantage of not being tightly coupled to specific filesystems. For example, if ext2/ext3 are enhanced with a working fallocate() (unlikely), an existing snapraid binary (with the discussed mod) will take advantage of it.
UhClem
I'm using Win 7x64 and I also have alarge delay in the Sync process (using 1.7)
It got to 52% of a process and then the screen looked like it was locked,
But the Parity drive was being written to really slowly,
But none of the source disks where being read.
Could this a manifestation of the above but on NTFS (Win 7) File systems?
No, that is some other issue.