md5cp Blog
Copies files conditionally using MD5 checksums
Status: Beta
Brought to you by:
umkopmanis
During testing, version 0.4 processed 79183 files in a little over 12 minutes. The files were a mix of small text files and larger .ps and .png files. The test was conducted using the --nocopy option, so it largely exercised the SQLite capabilities and the file I/O. Nonetheless, I'm pretty happy with the performance.
As a benchmark, the PHP script upon which this is based performed that task is about 100 minutes. Admittedly, PHP isn't as fast, and I used MySQL instead of SQLite, so there was a network and socket I/O component that added overhead.