lsdup

lsdup, short for "list duplicates", is a command line tool which can assist in finding and removing files with identical contents.

Unlike many file deduplication tools, this has a "comparison mode" which lets you tell lsdup to "only read the files in this directory, do not consider any for duplicates... but, the files in that other directory may have duplicates, so delete (or move) any of those."

lsdup also has normal deduplication, where, if given a single directory, it will sort the originals from the duplicates.

lsdup works by first looking through all of the directories you specified, and creates a text file which contains all of the "originals" and "duplicates". You can look through and modify the list in your favorite text editor. Finally, feed the file back into lsdup to move/delete the files as needed.