pguniq

Pretty Good Unique - report or omit repeated lines without sorting

Status: Alpha

Brought to you by: antondemidov

Downloads: 0 This Week

Last Update: 2021-09-30

Browse Code

Get an email when there's a new version of pguniq

Linux

This utility suppresses or reports duplicate rows without the need to sort them first.
It calculates CRC32 hash for every row and saves it in bit matrix. By default it uses 4-byte hash which consumes up to 512MB of RAM. Memory is allocated by 1K chunks as needed.

Project Activity

See All Activity >

License

GNU General Public License version 3.0 (GPLv3)

Follow pguniq

pguniq Web Site

Other Useful Business Software

MongoDB Atlas runs apps anywhere Icon

MongoDB Atlas runs apps anywhere

Deploy in 115+ regions with the modern database for every enterprise.

MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.

Start Free

Rate This Project

Login To Rate This Project

User Reviews

Be the first to post a review of pguniq!

Additional Project Details

Operating Systems

Programming Language

Registered

2021-09-29

Report inappropriate content

Recommended Projects

Miller
Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed. With Miller, you get to use named fields without needing to count positional indices, using familiar formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed....
Files Hash code Verifier
A Security tool that will help to verify the hash codes of files . Currently,Our Hash Code verifier tool supports the following Hash types: MD5,SHA,CRC32. More details about the tool can be found in the project homepage.
CyoHash
CyoHash is a simple shell extension that is used from within Windows Explorer to calculate the MD5 hash, SHA1 hash, or CRC32 checksum of a file. (Additional SHA256, SHA384, and SHA512 algorithms are available for users of Windows XP SP3 or newer.)
UFFS
UFFS: Ultra-low-cost Flash File System, designed for NAND flash working in embedded system. UFFS typically consumes less then 200K ram for 1Gb(page 512)/4Gb(page 2K) NAND flash. It supports direct flash interface, works with or without OS.
WinHash
Helps you to verify the integrity of the files. Hash Calculator for files. Using: CRC32 MD5 RIPEMD-160 SHA-1 SHA-256 SHA-384 SHA-512 Update: Added CRC32 Now supports big files (more then 4Gb) *Requires .NET 4.5