Re: [Sprawler-devel] client check.

[Eric A. <and...@ce...>]

Mojo B. Nichols wrote:

>>Actually I think it may be my perl and just the sockets... can
>>somebody else try this on linux?  I said client because that fails,
>>but upon closer inspection it doesn't seem that simple.
>>    
>>
>
>Whew no not my sockets. Basically the problem was two fold: One my
>client database didn't have my client in there.  If I add it blindly
>to the database that takes it past that point.  Then my url seed db
>was either empty or broken or something. removing that index allowed
>it to start working (reseeded it etc). I'm going to shuffle this off
>to the side and see if I can figure out where it went wrong. Perhaps
>the seeded url db  is in cvs? I'll check it out.  
>  
>

Glad to hear you got it working!  That makes me feel better anyway.. :)

>I'm curious about this client db and its intended use.
>

Basically, to keep clients from being able to check in/upload data for 
*ANY* arbitrary url they desire.  That way, an evil indexer can't fake 
an index for it's own website, with all kinds of fake or misleading data 
in it, causing our index to be invalid.  They request URL's to index, 
then they must check in those URL's.  You can't check in URL's you have 
not checked out..

Maybe it's time we have someone write up some documentation on all 
this?  How to use each piece, with example and syntax, etc..  What do 
you think?

Eric


-- 
------------------------------------------------------------------
Eric Anderson     Sr. Systems Administrator    Centaur Technology
Talk sense to a fool and he calls you foolish.
------------------------------------------------------------------