#3 Problems handling UTF-8 sites

[Michael Rock]

Any site where the content type is UTF-8 and the &
symbol is encountered, the output from the scraper will
add an extra amp; after &.

For example: & instead of just &

A scrape of links from this page
http://today.reuters.co.uk/news/default.aspx

will add an extra amp; after every & symbol, making the
url invalid.

http://today.reuters.com/news/newsArticle.aspx?
type=worldNews&storyID=2005-10-
05T125502Z_01_MAR536990_RTRUKOC_0_US-IRAQ-
REFERENDUM.xml

The url should be left alone.

http://today.reuters.com/news/newsArticle.aspx?
type=worldNews&storyID=2005-10-
05T125502Z_01_MAR536990_RTRUKOC_0_US-IRAQ-
REFERENDUM.xml

I tried 'link = link.replace('&','&')' but for some
reason it does not work.

If anyone has a solution please post here. Thanks