[htdig-dev] New proposed nofollow robot exclusion on links

[Neal R. <ne...@ri...>]

Read this!

http://www.searchenginejournal.com/index.php?p=1254

Basically they are proposing to support puting nofollow robot directives 
in individual links.  This is great.. I've been wishing for this for a 
while.  This makes it possible to more carefully direct the spider.. ie 
follow some links on a page, but not all without having to write huge 
list of exclusion patterns.

We should probably implement this since it looks like it could become a 
defacto standard.

Thanks.

-- 
Neal Richter 
Knowledgebase Developer
RightNow Technologies, Inc.
Customer Service for Every Web Site
Office: 406-522-1485