#6 *TIP* A faster way to blow out all of the duplicates

[speagle]

This tool has saved us loads of wasted space and resulting in real dollars saved and avoiding purchasing more disk space.

The following is a useful tip for erasing large sets of duplicates that commonly occur when parsing large volumes, volume roots and subdirectories and volumes littered with smaller files such as %homedrive%'s.

1. Using the GUI version (dupfgui.exe) and when the scan is complete, save (Store) the results into a text file.
The results will look similar to the example:

- 2 equal files of size 290816
"s:\System Volume Information\_restore{5D527826-05BD-4A83-8416-28ACXTY4001}\RP170\A0014626.DLL"
"s:\System Volume Information\_restore{5D527826-05BD-4A83-8416-28ACXTYA14001}\RP170\A0014839.DLL"
- 2 equal files of size 286720
"s:\System Volume Information\_restore{5D527826-05BD-4A83-8416-28YUTDA14001}\RP170\A0015120.DLL"
"s:\System Volume Information\_restore{5D527826-05BD-4A83-8416-28YUTDA14001}\RP170\A0015168.DLL"

When the list contains thousands+ files, using the GUI to select patterns and clicking can be tedious.

We've noticed consistent characteristics that every group begins with the hyphen and ends with a numerical value between 0-9 indicating file size.

We use gsar.exe to parse the list and deleting the carriage return and linefeed between the end of every group line numerical value and convert it into a REM statement, append the first file in the group to preserve it from deletion then prepend "del" commands at every subsequent line.

This method has created useable batch files out of our results and quickly peels out dup's.

2. Download gsar.exe from multiple sources, JSI's reference and redirect is here: http://windowsitpro.com/article/articleid/77105/jsi-tip-5801-general-search-and-replace-gsar-freeware.html

3. The following commands assume you've stored your dupfgui.exe results as "results.txt", this command peels out numerical values that aren't bracketed by quotes since dupfgui.exe is conscientious about sealing all file names inside quotes for accurate parsing. The section headers displaying number of dup's and sizes can be consistently parsed and gsar.exe removes the carriage return and line feed for the appending function to protect at least one version of your duplicates, the last gsar.exe command prepends the "del" instructions.

WARNING: THE FINAL GSAR COMMAND FOR PREPENDING DEL HAS TO BE EXECUTED LAST OR RISK CORRUPTING YOUR RESULTS FILE BY PREPENDING DELETE N

(most contemporary versions of Win32 command shelsl will process these commands when copied and pasted as a single block and avoiding the tediousness of pasting line by line / save it as a gsar script for reuse)

gsar -s1:013:010:032:032:034 -r1:032REM:034 -o results.txt
gsar -s2:013:010:032:032:034 -r2:032REM:034 -o results.txt
gsar -s3:013:010:032:032:034 -r3:032REM:034 -o results.txt
gsar -s4:013:010:032:032:034 -r4:032REM:034 -o results.txt
gsar -s5:013:010:032:032:034 -r5:032REM:034 -o results.txt
gsar -s6:013:010:032:032:034 -r6:032REM:034 -o results.txt
gsar -s7:013:010:032:032:034 -r7:032REM:034 -o results.txt
gsar -s8:013:010:032:032:034 -r8:032REM:034 -o results.txt
gsar -s9:013:010:032:032:034 -r9:032REM:034 -o results.txt
gsar -s0:013:010:032:032:034 -r0:032REM:034 -o results.txt

gsar -s:032:032:034 -rdel:032:034 results.txt -o

....will result in:

- 2 equal files of size 290816 REM "s:\System Volume Information\_restore{5D527826-05BD-4A83-8416-28ACXTY4001}\RP170\A0014626.DLL"
del "s:\System Volume Information\_restore{5D527826-05BD-4A83-8416-28ACXTYA14001}\RP170\A0014839.DLL"
- 2 equal files of size 286720 REM "s:\System Volume Information\_restore{5D527826-05BD-4A83-8416-28YUTDA14001}\RP170\A0015120.DLL"
del "s:\System Volume Information\_restore{5D527826-05BD-4A83-8416-28YUTDA14001}\RP170\A0015168.DLL"

Save the results file as a batch/cmd script and the header lines will just error while the rest of the DEL commands execute and pare down your duplicates.

Other switches like /q can be inserted using GSAR, since *NIX offers infinite shell options the GSAR options are beyond the scope of this posting but more than likely a UNIX/LINUX shell user is versed in the nuances of their console and will adopt accordingly.

[João Paulo]

It would be nice to send a command to make a hard link of every duplicate found, or at least every duplicate selected with the mouse.