#50 Speed issue

[sheldrak]

Hello,
These days I've tested some hash command line utilities on big files and the results regarding read speed from storage (in this case from a RAID0 array made with 2 OCZ Vector SSDs with 128 GB each) are listed below.
Read speeds are those displayed in Windows 8.1 64-bit Task Manager, in Performance tab, Disk 0 (C:). I retain the peak speed for each hash utility. For large files, read speeds from storage are almost constant during hash calculation, for all command line hash utilities, because of using the RAID0 SSD array.
Please note that after each test, I used RAMMap.exe utility from Sysinternals to clear the cached memory (RAM), because my PC have 16 GB RAM and the entire file is copied in RAM after I copied to partition C (RAID0 array) and after each hash utility done its job.

With these utilities, I calculate only the SHA-1 sum of a single file (10 GB size):
hashutils: 586 MB/s
checksum from corz.org: 440 MB/s
fsum: 415 MB/s
FileVerifier++: 386 MB/s
rhash: 293 MB/s
AutoIt _Crypt_HashFile function: 175 MB/s
exactfile: 70 MB/s

I like rhash for it's clean output to file and for the option to display the progress of calculation in percents, in the command prompt window.
Command line used with rhash.exe:
rhash.exe --sha1 --percents --output="C:\FILE.sha1" "C:\FILE.7z" --printf="%%h *%%f"\n

I guess that rhash don't use the potential of a RAID0 array, because 293 MB/s is the maximum read speed from a single OCZ Vector 128 GB SSD, when a single read operation is made on it. More operations done simultaneously will increase the read speed from a single OCZ Vector 128 GB up to 508 MB/s (for RAIDO array with 2xOCZ Vector 128 GB, the top read speed is 1GB/s, for large files).
That is not a bug, maybe a feature that can be added (internal) to use the potential of the RAID0 array.

Thanks!

[sheldrak]

Updated tests done on a RAID0 SSD array:
Read speeds listed below, are those displayed in Windows 8.1 64-bit Task Manager,
in Performance tab, Disk 0 (C:).

With these utilities, I calculate only the SHA-1 sum of a single file (10 GB size):

hashutils: 586 MB/s
powershell v4: 515 MB/s
fciv: 468 MB/s
checksum from corz.org: 440 MB/s
fsum: 415 MB/s
FileVerifier++: 386 MB/s
rhash: 293 MB/s
AutoIt _Crypt_HashFile function: 175 MB/s
exactfile: 70 MB/s

[Aleksey]

It would be great if somebody have found the reason. I have no RAID hardware, plus now I have too little time time to investigate the problem.

[Aleksey]

The problem is fixed by commit edc311890.
Fix will be included into RHash v1.4.6.

Benchmarks:

SHA1, 1.6GB file, Windows, Core i5 2.60GHz

Speed	Program
457 MBps	rhash dev (with bugfix)
353 MBps	sha1sum (hashutils)
249 MBps	rhash v1.4.5

CRC32, 11GB file, Windows, Core i5 2.60GHz

Speed	Program
612 MBps	rhash dev (with bugfix)
380 MBps	crc32sum (hashutils)
364 MBps	rhash v1.4.5

CRC32, 526MB file, Linux, Core i7 2.90GHz

Speed	Program
1974 MBps	rhash dev (with bugfix)
1830 MBps	rhash v1.4.5

[Aleksey]

The fix is included into RHash v1.4.6 (github.com).