Re: [htdig] problem with chunked input in 3.2 (was: no subject)

[Gilles D. <gr...@sc...>]

According to Franck Horlaville:
> Here's the trace and log on 3.2.0b4-123001 crawling just 
> www.aicha.com and hanging.
> 
> [Switching to thread 1 (process 23089 thread 0x1603)]
> ^C
> Program received signal SIGINT, Interrupt.
> 0x70004c28 in bzero ()
> (gdb) bt
> #0  0x70004c28 in bzero ()
> #1  0x00018b30 in Connection::Read_Partial (this=0x64e9e0, 
> buffer=0x64e9e4 "\r\nffb\r\nheight=17> \r\n    <td 
> class=xl1510352><font face=\"Verdana, Arial, Helvetica, sans-serif\" 
> size=\"2\">TOUIL \r\n      AYDA</font></td>\r\n    <td 
> class=xl1510352><font face=\"Verdana, Arial, Helvetica, s"..., 
> maxlength=8192) at Connection.cc:645
> #2  0x00018698 in Connection::Read_Char (this=0x64e9e0) at Connection.cc:419
> #3  0x00018764 in Connection::Read_Line (this=0x64e9e0, 
> s=@0xbfffced8, terminator=0xad860 "\r\n") at Connection.cc:466
> #4  0x00014078 in HtHTTP::ReadChunkedBody (this=0x64e7f0) at HtHTTP.cc:1049
> #5  0x00012b6c in HtHTTP::HTTPRequest (this=0x64e7f0) at HtHTTP.cc:411
> #6  0x00012140 in HtHTTP::Request (this=0x64e7f0) at HtHTTP.cc:161
> #7  0x0000390c in Document::Retrieve (this=0x648840, server=0x64bcf0, 
> date={Ht_t = 0, local_time = true, static days = 0xd1790, _vptr$ = 
> 0xf9584}) at Document.cc:487
> #8  0x00008f04 in Retriever::parse_url (this=0xbffff378, urlRef=@0x0) 
> at Retriever.cc:567
> #9  0x00008934 in Retriever::Start (this=0xbffff378) at Retriever.cc:427
> #10 0x0000ff4c in main (ac=6, av=0xbffffb28) at htdig.cc:338
> #11 0x000026f8 in _start ()
> #12 0x00002528 in start ()
> 
> ===
> 
> 7:12:2:http://www.aicha.com/lacollec/c/tirelire_010630.htm: Making 
> HTTP request on http://www.aicha.com/lacollec/c/tirelire_010630.htm
>    Making a HEAD call before the GET
> Header line: HTTP/1.1 200 OK
> Header line: Date: Fri, 01 Feb 2002 16:51:43 GMT
> Header line: Server: Apache/1.3.12 OpenSSL/0.9.6a (Unix)
> Header line: Content-Type: text/html
> No modification time returned: assuming now
> Retrieving document /lacollec/c/tirelire_010630.htm on host: www.aicha.com:80
> Http version      : HTTP/1.1
> Server            : HTTP/1.1
> Status Code       : 200
> Reason            : OK
> Access Time       : Fri, 01 Feb 2002 16:51:26 GMT
> Modification Time : Fri, 01 Feb 2002 16:51:43 GMT
> Content-type      : text/html
> Request time: 1 secs
> Header line: HTTP/1.1 200 OK
> Header line: Date: Fri, 01 Feb 2002 16:51:45 GMT
> Header line: Server: Apache/1.3.12 OpenSSL/0.9.6a (Unix)
> Header line: Transfer-Encoding: chunked
> Header line: Content-Type: text/html
> No modification time returned: assuming now
> Retrieving document /lacollec/c/tirelire_010630.htm on host: www.aicha.com:80
> Http version      : HTTP/1.1
> Server            : HTTP/1.1
> Status Code       : 200
> Reason            : OK
> Access Time       : Fri, 01 Feb 2002 16:51:43 GMT
> Modification Time : Fri, 01 Feb 2002 16:51:46 GMT
> Content-type      : text/html
> Transfer-encoding : chunked

OK, this is consistent with the problem you encountered on the other site
you were digging.  3.2 does seem to have problems with chunked input.

What are your settings of the persistent_connections and head_before_get
attributes?  Does changing either of these avoid the problem?

Gabriele, are you still around?  If so, please look into this problem, or
at least give us your opinion.

-- 
Gilles R. Detillieux              E-mail: <gr...@sc...>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba  Phone:  (204)789-3766
Winnipeg, MB  R3E 3J7  (Canada)   Fax:    (204)789-3930