Re: [htdig] URLS containing ? and & - difference between 3.1.5 and 3.1.6

[Owen B. <ob...@bo...>]

Owen Boyle wrote:
> ...many of the navigation files (which
> contained the links to the rest of the site) contained the meta tag:
> 
>         <meta name="htdig-noindex">
>
> My understanding is that this meta tag causes the file containing it
> *not* to be indexed *but* that the file is still parsed for hrefs. 
> So, either, my understanding of the meta-tag is wrong, or there's a
> bug...

Guess which - my understanding is wrong...

Having further RTFM'd,it would appear that the above meta tag was
working by accident - it should not have been. It looks like 3.1.6
implements the spec more precisely. So the correct procedure is:

- to ignore the document entirely:

<meta name="htdig-noindex">

- do not index words, but find hrefs:

<meta name="robots" content="noindex, follow">

Yours, embarrassed but wiser,

Owen Boyle.