I am having a little trouble indexing a database based document
management system, which I am not allowed to modify, as it is supported
by another company.
There is no single index page, so I have created a hard-coded version,
which points to every single ID that is possible on the system. Not
surprisingly, many of these URL's don't lead to a valid document, but
rather than getting a 404, or even a valid HTML page, I just get a plain
text error message, with no html header or anything.
The only thing that I can see to latch onto is the fact that this is
always 140bytes, but otherwise I am at a loss to think of a way of
keeping these documents from being indexed by htdig 3.1.6
Does anyone have any ideas?
Get latest updates about Open Source Projects, Conferences and News.