htdig-dev

[htdig-dev] picking up dynamic php pages

[John H. <joh...@co...>]

I apologize in advance as this is probably a common problem and =
question. I have read through the docs and played with the config files, =
but I am unable to find a answer. I'm new to the mailing list so here =
goes.

I have successfully installed htdig and have a working search. However =
only static html pages are included in the database. I need to include =
company listings from records in a MySQL database. Normally the user =
accesses details for a particular company from a list of results via a =
search page. This information needs to be included in the web site =
search.

Can someone suggest how to get the search functioning as desired?

Thanks.

Re: [htdig-dev] picking up dynamic php pages

[Gabriele B. <g.b...@co...>]

Hi John and welcome here!

Il mar, 2004-10-05 alle 08:07, John Hanley ha scritto:
> I have successfully installed htdig and have a working search. However
> only static html pages are included in the database. I need to include
> company listings from records in a MySQL database. Normally the user
> accesses details for a particular company from a list of results via a
> search page. This information needs to be included in the web site
> search.

ht://Dig's spider can't follow (X)HTML forms. This means that if your
listings are reachable only through user inputs and not via normal
links, the spider (and every spider on earth) can't index them.

Normally, when dealing with dynamic pages, developers make available to
users a very simple (but comprehensive) listing of all the entries, in
order to make them indexable.

If you don't want to make them visible to all the users, you can issue a
hidden link in the HEAD section of the document from, for instance, the
home page. Like:

<link href="listing/index.php" />

I also suggest you don't index the listings, but make spiders follow
them so they can index the single entries. In order to achieve this,
just include in the HEAD section of the listing:

<meta name="robots" content="noindex, follow" />

Hope this helps,
-Gabriele

Re: [htdig-dev] picking up dynamic php pages

[John H. <joh...@co...>]

Hi Gabriele,

Thanks for the clarification.

Your solution is great! I will create a page as you described and add the
needed HEAD section information.

Sincerely,
John

----- Original Message ----- 
From: "Gabriele Bartolini" <g.b...@co...>
To: "John Hanley" <joh...@co...>
Cc: "ht://Dig - Dev" <htd...@li...>
Sent: Monday, October 04, 2004 11:31 PM
Subject: Re: [htdig-dev] picking up dynamic php pages


> Hi John and welcome here!
>
> Il mar, 2004-10-05 alle 08:07, John Hanley ha scritto:
> > I have successfully installed htdig and have a working search. However
> > only static html pages are included in the database. I need to include
> > company listings from records in a MySQL database. Normally the user
> > accesses details for a particular company from a list of results via a
> > search page. This information needs to be included in the web site
> > search.
>
> ht://Dig's spider can't follow (X)HTML forms. This means that if your
> listings are reachable only through user inputs and not via normal
> links, the spider (and every spider on earth) can't index them.
>
> Normally, when dealing with dynamic pages, developers make available to
> users a very simple (but comprehensive) listing of all the entries, in
> order to make them indexable.
>
> If you don't want to make them visible to all the users, you can issue a
> hidden link in the HEAD section of the document from, for instance, the
> home page. Like:
>
> <link href="listing/index.php" />
>
> I also suggest you don't index the listings, but make spiders follow
> them so they can index the single entries. In order to achieve this,
> just include in the HEAD section of the listing:
>
> <meta name="robots" content="noindex, follow" />
>
> Hope this helps,
> -Gabriele
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out
more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> ht://Dig Developer mailing list:
> htd...@li...
> List information (subscribe/unsubscribe, etc.)
> https://lists.sourceforge.net/lists/listinfo/htdig-dev