Duff is a Unix command-line utility for quickly finding duplicates in a given set of files. Duff is written in C and should build and run on most modern Unices.
This tool is written in pure C, compile and install was just standard (tested opensuse 12.2). It finds duplicates faster than other tools find files. This might be due to the builtin hash algorithms. Functionality is quite basic, but easy to extend in C, or just customize the output format and write a console or GUI wrapper. I use it mainly to find duplicate pictures in a big collection, you easily get duplicates when copying from different cameras or cell phones, leaving some files on the phone to show them there. Recommended.