pguniq

This utility suppresses or reports duplicate rows without the need to sort them first.
It calculates CRC32 hash for every row and saves it in bit matrix. By default it uses 4-byte hash which consumes up to 512MB of RAM. Memory is allocated by 1K chunks as needed.