#20 Provide MD5 hashes on visible content, not entire page

[Simon Wilkinson]

Another spin-off from Bug #102147

Using MD5 hashes for duplicate detection doesn't work too well on
auto-generated pages which include session information in their URL
strings.

A possible solution to this would be to extend the MD5 hash generation
so that it is done on some subset of the page information when dealing
with HTML content - one possibility would be to use just the visible
text, not the URLs or markup. Disadvantage to this, obviously, is that
it may miss genuinely updated pages, and that it still falls foul of pages
which change their text as well.