ParDRe

ParDRe is a parallel tool to remove duplicate reads. Duplicate reads can be seen as identical
or nearly identical sequences with some mismatches. This tool will let the users to avoid the analysis of not necessary reads, reducing the time of subsequent procedures with the dataset (e.g., assemblies, mappings, etc.).

The tool is implemented with MPI in order to exploit the parallel capabilities of multicore
clusters. It is faster than multithreaded counterparts (end of 2015) for the same number of cores and, thanks to the message-passing technology, it can be executed on clusters.

There also exists a MapReduce counterpart of ParDRe, called MarDRe (see the link above).

UPDATE: From version 2.0.5 ParDRe also provides support to remove only optical duplicates (and leave biologically interesting duplicates) as well as to work with compressed input/output with .gz format.