[Phpwiki-talk] Re: Another use for Google's 'rel='nofollow''?

[Charles C. <ch...@ru...>]

On Mon, January 31, 2005 17:43, Charles Corrigan said:
> On Thu, January 27, 2005 17:31, Charles Corrigan said:
>> I just looked at the access log for my site and noticed
>> that google was indexing it. And it was trying every
>> possible link from every page!
>>
>> Would it make sense to add a "rel='nofollow'" to links
>> such as "edit text" where it makes no sense for a robot
>> to follow?
>
> When I wrote that, it was already 7 days after Reini had started putting
> the nofollow attribute onto some of the links! My only excuse (hah!) is
> that CVS and then the lists had problems last week.

Google just re-indexed my site and, again, followed all links, including
action=edit etc.  I looked into Google's spec and realised that the
rel="nofollow" only means that the link does not contribute to pagerank.
It does not mean that the link will not be followed.

It looks like the only way to handle this is via the robots.txt.  Google
support an extension to the specification that allows wildcards to be
specified in the Disallow field (see
http://www.google.com/intl/en/webmasters/3.html ).

The new contents of my robots.txt file follow,

regards,
Charles

# robots.txt - Charles Corrigan - 14/2/2005
# This robots.txt file uses a non-standard format that is followed by
# Google. This format allows the use of wildcards in the page names
# and is used here to advise robots not to follow links in PhpWiki that
# are not relevant.
User-agent: *
Disallow: /*action=chmod
Disallow: /*action=chown
Disallow: /*action=create
Disallow: /*action=DebugInfo
Disallow: /*action=diff
Disallow: /*action=edit
Disallow: /*action=EditMetaData
Disallow: /*action=EditMetaInfo
Disallow: /*action=loadfile
Disallow: /*action=lock
Disallow: /*action=PageDump
Disallow: /*action=PageHistory
Disallow: /*action=PageInfo
Disallow: /*action=PhpWikiAdministration%2FChmod
Disallow: /*action=PhpWikiAdministration%2FChown
Disallow: /*action=PhpWikiAdministration%2FRemove
Disallow: /*action=PhpWikiAdministration%2FRename
Disallow: /*action=PhpWikiAdministration%2FReplace
Disallow: /*action=PhpWikiAdministration%2FSetAcl
Disallow: /*action=remove
Disallow: /*action=rename
Disallow: /*action=replace
Disallow: /*action=setacl
Disallow: /*action=TranslateText
Disallow: /*action=unlock
Disallow: /*action=upgrade
Disallow: /*action=viewsource
Disallow: /*action=zip
Disallow: /*action=ziphtml