Re: [htdig] www. urls not found

[Greg F. <gre...@ya...>]

--- olinux <ol...@ya...> wrote:
> I brought this up a few weeks back and still have not
> found a solution. when i set start_url to
> http://www.domain.com htdig immediately returns a 404
> error and does not index anyhting.


404 is Page Not Found being returned from the *SERVER*, so it appears
to me that the DNS entry is not at fault.  htdig is getting TO the
server, but it is getting back a not-so-good answer.

From the *same machine* that htdig runs on, try this:

1. Enter command:  telnet www.domain.com 80
2. After the line "Escape character is ^]" type in:  GET /

See what the output from the server is.

A better solution (in my opinion) is to use a command-line URL fetcher
like "wget" or "curl":

  wget -Sv http://www.domain.com/

Hope this helps,
greg_fenton.

=====
Greg Fenton
gre...@ya...

__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com