From: Christophe D. <du...@tr...> - 2004-10-06 12:14:37
|
I've installed 3.22 recently on red hat 7.3 in order to have the phrase matching option. But I've problems on indexing some sites and I don't know why. I try with the htdig.conf and I just replace the start_url (http://www.colloc.minefi.gouv.fr/) for example but when I run rundig it seems to block to the first page. Heres the log (-vvv) : 1:1:http://www.colloc.minefi.gouv.fr/ New server: www.colloc.minefi.gouv.fr, 80 - Persistent connections: enabled - HEAD before GET: enabled - Timeout: 30 - Connection space: 0 - Max Documents: -1 - TCP retries: 1 - TCP wait time: 5 - Accept-Language: Trying to retrieve robots.txt file Making HTTP request on http://www.colloc.minefi.gouv.fr/robots.txt Header line: HTTP/1.1 404 Not found Header line: Server: Netscape-Enterprise/3.6 Header line: Date: Wed, 06 Oct 2004 11:49:20 GMT Header line: Content-type: text/html Header line: Content-length: 207 Request time: 0 secs pushed pick: www.colloc.minefi.gouv.fr, # servers = 1 > www.colloc.minefi.gouv.fr supports HTTP persistent connections (infinite) 0:2:0:http://www.colloc.minefi.gouv.fr/: Making HTTP request on http://www.colloc.minefi.gouv.fr/ Header line: HTTP/1.1 500 Server Error Header line: Server: Netscape-Enterprise/3.6 Header line: Date: Wed, 06 Oct 2004 11:49:20 GMT Header line: Content-length: 305 Header line: Content-type: text/html Request time: 0 secs not found pick: www.colloc.minefi.gouv.fr, # servers = 1 > www.colloc.minefi.gouv.fr supports HTTP persistent connections (infinite) ht://dig End Time: Wed Oct 6 13:50:20 2004 Deleted, not found: ID: 2 URL: http://www.colloc.minefi.gouv.fr/ Preamble text: Postamble text: Note: This message will be sent again if you do not change or take away the notification of the above mentioned HTML page. Find out more about the notification service at http://www.htdig.org/meta.html Cheers! ht://Dig Notification Service Is anyone has this kind of problems ? Thanks for any help. |