Re: [htdig-dev] picking up dynamic php pages

[Gabriele B. <g.b...@co...>]

Hi John and welcome here!

Il mar, 2004-10-05 alle 08:07, John Hanley ha scritto:
> I have successfully installed htdig and have a working search. However
> only static html pages are included in the database. I need to include
> company listings from records in a MySQL database. Normally the user
> accesses details for a particular company from a list of results via a
> search page. This information needs to be included in the web site
> search.

ht://Dig's spider can't follow (X)HTML forms. This means that if your
listings are reachable only through user inputs and not via normal
links, the spider (and every spider on earth) can't index them.

Normally, when dealing with dynamic pages, developers make available to
users a very simple (but comprehensive) listing of all the entries, in
order to make them indexable.

If you don't want to make them visible to all the users, you can issue a
hidden link in the HEAD section of the document from, for instance, the
home page. Like:

<link href="listing/index.php" />

I also suggest you don't index the listings, but make spiders follow
them so they can index the single entries. In order to achieve this,
just include in the HEAD section of the listing:

<meta name="robots" content="noindex, follow" />

Hope this helps,
-Gabriele