Re: [fetchmail-devel] 24h comparison of old and new UID handling code

[Matthias A. <mat...@gm...>]

Rainer Weikusat wrote on 2010-05-28:

> "Matthias Andree" <mat...@gm...> writes:

>> D'oh. A std dev' that's thrice the average. There's a considerable
>> chance  for a negative number. :-}
>
> It is conceivable that I am miscalculating the number. But this really
> just communicates that the distribution is very asymmetric.

I'd rather trust your numerics than the underlying assumption it were  
normally distributed data :-)
(Normalverteilung/Gauss'sche). Ok, there's always room to miss out on a  
"2" in an exponent and inadvertently confuse variance and standard  
deviation, but still why would the count of messages in the mailboxes be  
normally distributed?  There isn't a single reason I could see, and this  
voids all chances that the arith. average were in 1-sigma, 2-sigma,  
3-sigma confidence intervals, or anywhere near the expected value.

>>> UID handling is still the dominant operation (processor-time wise)
>>> performed by fetchmail.
>>
>> No it's not, see below -- you're not grouping other code properly. If
>> oprofile samples call stacks so that it can cumulate per caller (like
>> du(1) does) you'll see it's not. Otherwise, we'll resort to the
>> preclusion  reasoning below.
>
> This seems to be a disagreement about the meaning of the term
> 'dominant'. It may well have some defined meaning I am not aware of.
> I am not a repurposed physicist or person with any other kind of
> degree , just a programmer and even this just because I spent
> something like twenty years learning about this stuff whatever I could
> gather from sources available to me.

I'm not sure if there is a formal definiton, but I'd say it's "dominant"  
if you can neglect other terms in a sum, for instance. Or if you'd write  
the >> or << relation operators, often considered as possible with one  
order of magnitude in between. That we don't have after your fix.

That doesn't make your fix less useful, however. As written for my 16,000  
message inbox, fetchmail starts fetching immediately, whereas the  
unpatched fetchmail ponders UIDL more than a minute.

> I will nevertheless do that since it is a fairly simple operation,
> which I can put into a few idle minutes in my (presently) usual 10h+
> workdays and something I can absolutely not do is put much more effort
> into this, except maybe two o'clock in the morning after working for
> ten hours. Since the code exists, works and (as far as I could
> determine) mitigates the problem it was supposed to mitigate, any
> justification I could conceivable use to explain to my boss why I am
> even looking at this during the time I am neither eating nor sleeping
> is gone :-(.

Which is good enough. I've always been meaning to do that, but there was  
always something else more important to do, so your final patch will be  
most welcome. I expect to make minor changes to fix compiler warnings and  
add side comments to NEWS and thereabouts (it's always interesting to see  
what you get when you compile on different Linux flavours, Solaris,  
NetBSD, FreeBSD, MacOS X), but from what I've seen so far, not much will  
be needed.

Possibly the merge with my existing advance development branch (not yet  
published) for 6.4.0 or 6.5.0 may take another hour - I don't think I'll  
put this feature in a 6.3.X release. This 6.3 branch was originally just a  
drop-in compatible stopgap branch so nobody had an excuse to sidestep the  
>100 fixes between 6.2.5 and 6.3.0, but as we all know, makeshift  
arrangements often prove to be long-lived. I would never have expected it  
to spawn 18 nontrivial releases to date, and 6.3.18 would appear, too.

The fetchmail code however really needs updating and leave compatibility  
with pre-2000 operating systems (with before C-89, before-POSIX2001)  
behind - especially the way it's sprayed all over the source with a  
gazillion of #ifdef makes maintaining the code a time-consuming and  
error-prone process, and my experience is if you don't have good  
improvements as selling points users will just stick to the old versions  
that you've left behind long ago.  I know I may disappoint some users that  
way, but in the end I need to see to real life -- which means I'm not  
pampering up systems where vendor support ceased years ago.

Efficiency improvements that users can see are one such selling point to  
motivate a migration... :)

-- 
Matthias Andree