Re: [Phplib-users] robots and session-id's

[Lindsay H. <fm...@fm...>]

One thing I've found is that some search engines _won't_ go into pages for
which there's session data passed back to the client.  I'm not sure if the
rejection is on account of GET data or attempts to set cookies, or both.  To
get at least the top page of such sites indexed, I generally take a snapshot
of the top page and serve it as a straight HTML.  The site doesn't try to
establish a session until a visitor goes to satellite pages.

This doesn't answer your question, nor I'm sure is it encouraging news w.
regard to getting your product catalog indexed, but I thought I'd pass it
on.

Thus spake Sascha Weise on Thu, Mar 07, 2002 at 10:58:37AM CST
> Hi out there,
> 
> I'm coding a shop with the help of phplib (thanks a lot to all the 
> developers at this point).
> One purpose is, that every page with products-infos can be indexed by 
> search-engines to make the rare products more findable - so I can't lock 
> all pages for robots.
> 
> At this point I was pretty astonished, that I found no standard-solution to 
> prevent urls with session-id's to be indexed by robots!
> 
> All hints I found (most in the archive of this list) recommended to check 
> $HTTP_USER_AGENT against a list of known user-agents and then not to start 
> phplib-page-management-functions if a robot was identified.
> For me this seems not to be very usable as I think it's hard to maintain an 
> actual list of ALL relevant search-engines.
> 
> 
> Now my questions:
> 
> - Does anyone know a better (perfect and simple) solution for this problem?
> 
> OR
> 
> - Wouldn't it be more simple and effective to check $HTTP_USER_AGENT 
> against some valid user-agent-strings from browsers?
> (At least every browser does have something like "...compatible; MSIE..." 
> in his string and I guess this list is shorter and easyier to maintain as a 
> list of robots)
> Does anyone know some PROs and CONs for this assumption?
> 
> 
> Thanks in advance,
> 
> Sascha.
> 
> 
> _______________________________________________
> Phplib-users mailing list
> Php...@li...
> https://lists.sourceforge.net/lists/listinfo/phplib-users

-- 
Lindsay Haisley       | "Everything works    |     PGP public key
FMP Computer Services |       if you let it" |      available at
512-259-1190          |    (The Roadie)      | <http://www.fmp.com/pubkeys>
http://www.fmp.com    |                      |