[htdig-dev] Subtle problem with meta robots noindex

[Bill C. <wca...@vh...>]

Hey all,

I ran across a logical problem when handling <META name="robots" 
content="noindex"> on a page. The behavior expected is that links on the 
page will be followed and indexed. This works fine on the initial index.

Let's call the page that shouldn't be indexed TOC (Tables Of Contents, a
typical application)  and pages linked to the TOC are the content.

If the only link to a page of the content is on the TOC, later indexing 
will not index that page as the bridging TOC is dropped from the list of 
documents (this assumes any pages linking to the TOC have not been 
modified since the last run and hence are not re-fetched). This causes the 
page to drop from the database, it will only be picked up on the next 
full index and dropped again on the next partial index.

I didn't see that this issue had been discussed before, would this still 
be an issue for 3.2x?

Later,

Bill Carlson
-- 
Systems Programmer    wca...@vh...         | Anything is possible,
Virtual Hospital      http://www.vh.org/      | given time and money.
University of Iowa Hospitals and Clinics      |       
Opinions are mine, not my employer's.         |