#5 memory leaks in crawler

[Igor Nekrestyanov]

Crawler is memory hog !!!

I tryied it on local web server
(proxy was disabled so it could only access local
web server and speed was fast).

In several minutes crawler grew from 3M process
to 200+M process before i killed it!!!

BTW, at the end
/var/state/oasis/OASIS-Crawler/state/closed/Recent_visited_URLs.crawler
has only 471 entry!

[Igor Nekrestyanov]

Logged In: YES
user_id=91810

I fixed most ugly leaks and now situation is much
better.

However, crawler still leaky - while some leaks
seems to be system ones (for example one
introduced by localtime()) there are some
leaks to fixed yet.

We have to get some numbers characterising it.

BTW, i used dmalloc to trace leaks.