RE: [Phpwiki-talk] Two new queries on search engine activity for AnalyseAccessLog plugin

[Charles C. <ch...@ru...>]

On 07 February 2005 18:40, Reini Urban [mailto:ru...@x-...] wrote

>>         ."WHERE (remote_host LIKE '%googlebot.com' "
>>         ."OR remote_host LIKE '%alexa.com' "
>>         ."OR remote_host LIKE '%inktomisearch.com' "
>>         ."OR remote_host LIKE '%msnbot.msn.com') "
>
>For search bots you can constrcut the hosts string from 
>lib/ExternalReferrer.php

I took a look - this handles the URL for main site of the search engine but not necessarily the crawler engine URL.  For example:
google.com vs googlebot.com. Also, some of these sites use hand built indices rather than crawlers to automatically generate the raw
material.

I could add a new column to the array in ExternalReferrer.php for the crawler URLs but I only have the 4 crawlers above so far...

Regards,
Charles