Re: [htdig-dev] head_before_get attribute (was: 3.2RC1 Feature Freeze)

[Gabriele B. <bar...@in...>]

At 14.39 14/10/2003 -0500, Gilles Detillieux wrote:
>It would be a good idea, in the general case, for us all to learn how
>to properly override config parameters in the code, so that a server
>block or URL block definition doesn't override an internal override

Maybe I am missing something. I am not aware of a way that allows us to 
override blocks definitions through the Configuration classes. Can you 
please point it out to me? Sorry.

>doing an initial dig, then it will only ever take effect when doing an
>update (or incremental) dig with persistent connections turned on.

No no ... wait. I have never talked about turning off head before get. Let 
me try and give an explanation about this parameter.

I remember we issued the 'head_before_get' attribute because of this: when 
requesting a non-parsable document we generally had 3 options:

1) persistent connections on:
         a) head before get on: we perform a HEAD call and notice that the 
document's content-type is not what we want so we simply avoid doing the 
GET call
         b) head before get off: we perform a GET call but in this case we 
must receive all the content returned by the server, otherwise we have to 
close the connection - that's not what we want in general.
2) persistent connections off: we perform a GET call and if the document is 
not what we want we simply close the connection (we anticipate it).

IMHO the 'head_before_get' could make the difference in some cases with 
persistent connections on and only the webmaster can see the difference in 
performances between turning it on or off. If we don't have many multimedia 
files we could simply turn it off (avoiding a 'double' call), whereas if we 
have big files to be downloaded (especially from the Internet) this 
attribute could make the difference, as a pre-emptive HEAD call would let 
us know about the type of document we are being requested and eventually 
save us a big download.

>not as versed in HTTP/1.1 as you are.  It seems to me that htdig should
>always be doing a HEAD before a GET when doing incremental digs through
>persistent connections.

Yes. And not only there. Even when performing an initial dig, if the user 
wants it, we must enable it.

I can think of this possible solution. The scenario above is still valid 
(IMHO) for the initial dig case; I would modify it for the incremental dig 
as mentioned yesterday, as follows:

if "persistent_connections" (on a server basis) is set to on:
         enable persistent connections
else
         disable them

if incremental or ("head_before_get" and "persistent_connections" are both 
set to on)  - I have to modify yesterday's patch a bit
         enable head before get
else
         disable head before get

In this way, for initial dig the user can choose whether activate 
persistent connections and head before get, whether for incremental digs 
the users' settings get overridden.

For me this sounds good. There can be issues regarding the way of doing it; 
I thought that adding some object variables in the Retriever and Document 
class would be fine. Unless there is a way of overriding specific settings 
through the Configuration classes.

Please let me know.

>By the way, Gabriele, good call on the Accept-Encoding header.  It's a
>simple, elegant fix to a troublesome bug.  You're right that adding
>support for gzip encoding is a feature request, and not a bug fix,
>and should be done after the upcoming release (not before).  Good work.

Thank you. However, by following Neal's directives, could someone of you 
try it and let me know so I can close the bug?

Ciao ciao
-Gabriele


>--
>Gilles R. Detillieux              E-mail: <gr...@sc...>
>Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
>Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: SF.net Giveback Program.
>SourceForge.net hosts over 70,000 Open Source Projects.
>See the people who have HELPED US provide better services:
>Click here: http://sourceforge.net/supporters.php
>_______________________________________________
>ht://Dig Developer mailing list:
>htd...@li...
>List information (subscribe/unsubscribe, etc.)
>https://lists.sourceforge.net/lists/listinfo/htdig-dev

--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check 
maintainer
Current Location: Melbourne, Victoria, Australia
bar...@in... | http://www.prato.linux.it/~gbartolini | ICQ#129221447
 > "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The 
Inferno