[Grub-develop] Indexing plans for the data? Users want to know.

[Kord C. <ko...@gr...>]

Hi,

I copied the general list on this email as I thought everyone
might get something out of the explanation that I give in
response to Travis' concerns.

1.  Is there any indexing happening right now?

First, and as many of you may know, we do NOT index the results
from the crawls that are done by the clients.  However, we do
keep the status info of the URLs and the returned data for the
last 24 hour crawl cycle.

1a.  What is being done with the client results?

The URL meta data (update rate, update time, down rate, etc.)
is available through a XML interface with our SQL server, and
the crawl data is available via an ftp site.  We have, on
occasion, had people request access to this data.  If anyone
wishes access to these resources, we will try to oblige.  Of
course people wishing to pull a full feed from us or do 1,000s
of queries to the database (small server here folks) will need
to discuss other options with us.

Please also keep in mind that we are still in TESTING, and that
the results returned right now are NOT 100% reliable.  This means
if someone were using our data, we couldn't guarantee that the
data was good, and that the crawl rate would be stable.

Time will fix this, of course.  ;)

2.  What database platform are you using?

MySQL.  It's quite fast - seriously.

3.  What rules you are setting for ranking keywords, ranking pages, etc?

Again, we are a CRAWLING engine, not a search engine.  When the
time comes, we expect other search engines to pull data from
the service.  This means they don't have to crawl their own set
of URLs, which decreases crawl bandwidth on the net, and increases
the crawl rate of the sites - which also increases the quality and
relevance of a search done on those sites.

If anyone has any questions or comments about any of this, please
feel free to post to the list!

Happy holidays!

Kord

>
> Message: 1
> Date: Sun, 29 Dec 2002 16:01:58 -0700 (MST)
> From: tr...@sp...
> To: gru...@li...
> Subject: [Grub-develop] Search page
>
> What's the plans for this area?  Is anybody working on indexing and getting the actual search page going?  I'm finding it kind of useless to be running the client for no purpose.  Like what's the point of running it right now if nobody can reap the benefits?
>
> So here's some questions:
> 1.  Is there any indexing happening right now?  What is being done with the client results?
> 2.  What database platform are you using?
> 3.  What rules you are setting for ranking keywords, ranking pages, etc?
>
> Travis Reeder
> Space Program
> http://www.spaceprogram.com
>
>
>
> --__--__--
>
> _______________________________________________
> Grub-develop mailing list
> Gru...@li...
> https://lists.sourceforge.net/lists/listinfo/grub-develop
>
>
> End of Grub-develop Digest
>

-- 
--------------------------------------------------------------
Kord Campbell                                       Grub, Inc.
President                      5500 North Western Avenue #101C
                                       Oklahoma City, OK 73118
ko...@gr...                            Voice: (405) 848-7000
http://www.grub.org                        Fax: (405) 848-5477
--------------------------------------------------------------