Re: [Gnugle-developers] web scanner built in?

[Marco L. <ma...@ma...>]

Hi!

> One of the issues that I've been interested in is "web scanning"
> software -- for example, a program that would automatically begin a
> search starting at the current page you are browsing and then return
> pages that match certain keywords that lie within some neighborhood
> of that page.
> 
> Perhaps Gnugle could have something like this built in.

Yes, such a feature is definitely planned, implemented as a HTTP proxy. 
By combined the web crawling with a proxy server, we have several 
advantages for the user:

   - The web crawling can also be used for pre-caching documents linked
     by the current page, thereby being able to return the web page
     from its cache when the user clicks the link.

   - The web crawling bandwidth is not wasted from the user's point of
     view, the user has no disadvantage of having his computer doing the
     web crawling.

   - This proxy function can also be used to generate access statistics
     and ratings for the indexed web pages.

But the whole proxy thing has one big disadvantage: It is not anonymous. 
Potential adversaries in the network could be able to track your web 
surfing. We somehow have to avoid this and do not yet have a idea how.

We surely plan to implement such a feature, but not in the first version 
of Gnugle, as the implementation of a proxy server is rather technical. 
For the first version we want to concentrate on an efficient protocol 
and indexing.

Best regards,
   Marco