hi, i've build a very small webcrawler, just to crawl a hrml site. someone is updating it in dreamweaver (...i know, but i cant do anything:)) and he want to have a full-text-search-engine, so i build a small crawler that fetches links and text from a page, and wrote everything in a database, works great, BUT:
atm i use:
to fetch the text and the links, but than the file is downloaded twice.
is there a way to have this part done, with only a single download, without changing the snoopyclass?
I'vwe done it now with
it is not very nice to use 'privat' methods but it works great.
if there is a better solution, give it to me ;)