#28 duplicate enteries in the directory-lists

[Ted McCormick]

./DirBuster-0.12$ cat directory-list-2.3-big.txt | wc -l
1273833
./Desktop/DirBuster-0.12$ cat directory-list-2.3-big.txt | sort -u |wc -l
1273796

Looks like there are a number of duplicates in the lists provided. You should probably throw them though sort -u. On a side note, 1.2 million seems really small to me.