fetchmail-devel — Development related discussion

Re: [fetchmail-devel] Some PATRICIA numbers

[Matthias A. <mat...@gm...>]

Am 16.05.2010, 23:33 Uhr, schrieb Rainer Weikusat:

> The trie implementation is meanwhile complete and I have done some
> preliminary benchmarks. I fear that the 'insert' cost will turn out to
> be somewhat higher, but the results for searches are promising. The
> largest .fetchids file I have here contains 7275 UIDs. Assuming the
> trie has already been created, iterating over the input file a
> thousand times and searching for every UID needs 3.93s on average,
> meaning the average time for a single search is about 5.4E-7 seconds
> (3Ghz P4). Doing a plain linear search for every UID needs about 16.6
> seconds on average[*] or about 2.3E-4 seconds per search.
>
> 	[*] For ten searches. I didn't have the patience to wait for
> 	1000 :->

Hi Rainer,

this looks good.  Without trying to downplay your efforts, I'd also say  
that this was somewhat expected. In other words, your numbers confirm that  
the trie works as expected and does not degenerate into a linear list in  
your test cases.
   Insertion into the data structure is fastest for a pre-allocated array,  
then appending to a list that has a tail pointer. Inserting into trees,  
tries, heaps, and thereabouts always takes a bit longer because it  
encloses a search for the node where you'll be forking the branches. It is  
some sort of "ordered" insertion. This is the price we (happily!) pay for  
quicker retrievals and searches.

The important part here is that, given N UIDs in a list, the costly  
operations "insert all UIDs" and "for each UID retrieved from the server,  
check if we've already seen it" are reduced to O(N log N) when they used  
to cost O(N^2) for larger N (i. e. asymptotically).

I'm really looking forward to your code :-)

Best regards

-- 
Matthias Andree

[fetchmail-devel] Some PATRICIA numbers

[Rainer W. <rwe...@ms...>]

The trie implementation is meanwhile complete and I have done some
preliminary benchmarks. I fear that the 'insert' cost will turn out to
be somewhat higher, but the results for searches are promising. The
largest .fetchids file I have here contains 7275 UIDs. Assuming the
trie has already been created, iterating over the input file a
thousand times and searching for every UID needs 3.93s on average,
meaning the average time for a single search is about 5.4E-7 seconds
(3Ghz P4). Doing a plain linear search for every UID needs about 16.6
seconds on average[*] or about 2.3E-4 seconds per search.

	[*] For ten searches. I didn't have the patience to wait for
	1000 :->

Re: [fetchmail-devel] save_str problems.

[Matthias A. <mat...@gm...>]

Am 13.05.2010, 23:24 Uhr, schrieb Rainer Weikusat:

> Rainer Weikusat <rwe...@ms...> writes:
>> Matthias Andree <mat...@gm...> writes:
>
> [...]
>
>>> - to make sure that your employer either waives his copyright and
>>> allows you to claim it, or your employer is willing to contribute
>>> under a compatible license (GPL "v2 or later" preferred).
>
> [...]
>
>> the usual answer whenever I asked for something like this in the
>> past was 'sure'.
>
> [...]
>
>> I could probably devote at least an hour of (unpaid) overtime to this
>> per day, meaning, a usable implementation should be available at the
>> end of this week or the beginning of next week.
>
> Just as a quick update on that: I've meanwhile talked to my boss about
> this and I am free to decide on a suitable license for publishment
> according to whatever makes the most sense and I am allowed to spend
> some amount of 'recreational programming time' on this per day.

Excellent.

If you want your own public Git repository for that, it's nearly there:  
you can clone the project through Gitorious's web interface, check out  
 from your branch ("git clone"), edit, commit there, and when done, ask me  
to merge, either as a series of commits, or squashed to one single commit.

Given all of the circumstances, I'd say MSS GmbH also deserves mention in  
the changelogs; I'd propose that you edit a NEWS file entry accordingly  
and propose a wording you feel adequate and that is in line with your  
regular PR policies.

-- 
Matthias Andree

Re: [fetchmail-devel] save_str problems.

[Rainer W. <rwe...@ms...>]

Rainer Weikusat <rwe...@ms...> writes:
> Matthias Andree <mat...@gm...> writes:

[...]

>> - to make sure that your employer either waives his copyright and
>> allows you to claim it, or your employer is willing to contribute
>> under a compatible license (GPL "v2 or later" preferred).

[...]

> the usual answer whenever I asked for something like this in the
> past was 'sure'.

[...]

> I could probably devote at least an hour of (unpaid) overtime to this
> per day, meaning, a usable implementation should be available at the
> end of this week or the beginning of next week.

Just as a quick update on that: I've meanwhile talked to my boss about
this and I am free to decide on a suitable license for publishment
according to whatever makes the most sense and I am allowed to spend
some amount of 'recreational programming time' on this per day.

Re: [fetchmail-devel] save_str problems.

[Matthias A. <mat...@gm...>]

Rainer Weikusat schrieb am 2010-05-11:

> I apologize for the misdirected mail ... 'wrong button'

No big deal.

> > Sounds good, although I'm not quite sure if you're working on two distinct
> > issues here (DIMAP and POP3 improvements).
> 
> I do. IMAP UIDs are defined to be strictly monotonically increasing
> unsigned 32 bit numbers (eventual wrap-around to be signalled via
> UIDVAILDITY change) and for the application I need this for, it is ok
> to not apply 'policy changes' (such as a changed size limit)
> retroactively. This means that I basically just record the highest
> 'seen' UID + 1 and then do a SEARCH UID <recorded>:<UIDNEXT> to get
> information about yet 'unseen' mails. 

A changed size limit only hurts if it means that more mail would be
downloaded than before, and in that case, you can as well invalidate
UIDVALIDITY (at the cost of re-fetching everything).

OTOH the POP3 UID stuff could then be reused to do an exhaustive search,
and we will finally be able to fetch from read-only mailboxes either way
in --keep configurations.

> > I have plans to attack POP3/UIDL scalability myself soon -- I could
> > devote my time in other ways if I know your work covers that and
> > reduces complexity to O(n log n) worst-case.
> 
> I actually started to write some code for this. So far, I was planning
> to use two vectors to be able to access records either by position or
> by 'associated fetch number' and to do ID searches based on a PATRICIA
> trie (reasonably easy to implement and I've already done a few in the
> past). This would imply that searching for a string of n characters
> would, at worst, need n * 8 single bit tests[*] (and a final complete
> comparison). Based on my present oprofile numbers[**] (based on 101
> .fetchids files with an average length of ~303 lines), the time spent
> with loading the file data is IMHO not large enough to warrant using a
> disk-based database (and this would be complicated because of the need
> to be able to find 'UIDL records' based on three different critetria).

Hadn't thought of Patricia tries, actually a decent idea (yet possibly
what a C++ STL map type might do under the hood if the key is suitable).

I wonder how we handle the case if two messages (i. e. different message
number) that share the same UIDL (this happens on some sites if the recipient
is listed more than once, or is being Cc:d mailing list mail, and if the
POP3 server uses a hash of few header items to construct the UIDL).
  Such appearance in a mailbox happens in the wild and should not be treated
an error that would abort the fetch. getmail used to fail in such
situations, we don't want to follow it down that route...

At any rate, this looks quite reasonable overall, so I'll keep my hands
off that code for the next days to not step on your toes. I'm looking
forward to it. Good speed!

Thanks & best regards
Matthias

Re: [fetchmail-devel] save_str problems.

[Matthias A. <mat...@gm...>]

Am 09.05.2010 23:02, schrieb Rainer Weikusat:
> Rainer Weikusat <rwe...@ms...> writes:
>> Matthias Andree <mat...@gm...> writes:
>>
>> [...]
>>
>>> it also pretended that someone had already fixed the complexity of
>>> the UID handling (which hasn't happened to the best of my knowledge;
>>> I have started hacking a bit on it, but not sure if I'll make it, or
>>> drop it and leave it for a later release using C++, if it turns out
>>> it's too much of a hassle to do in C).
>>
>> I am done with my last 'firebrigade' assignment which means that I
>> will now continue to work on the UID support for imap
> 
> I'm now the 'proud owner' of a fetchmail variant which works (mostly)
> like a proper 'disconnected imap client'. The number of POP3 accounts
> registered with the product I am using fetchmail for (except pulling
> my mail of my employer's mailserver) has grown to 99 and can be
> expected to increase further.

Sounds good, although I'm not quite sure if you're working on two distinct
issues here (DIMAP and POP3 improvements).

> Because of this, I'm determined to get
> rid of the present UID management code 'soon' (probably, during the
> next couple of 'Sunday shifts'). I've been a content fetchmail user
> ever since I first got 'internet' in 1999 and I would happily direct
> some useful work into it because of this. OTOH, my experience with OSS
> projects so far is a) one gets ignored, b) all kinds of ml lurkers
> start to enage in wild flaming and c) after the need has arisen, the
> powers-that-be grudingly implement the missing features themselves,
> just to ensure that only the worthy ones get any 'due credit', and this
> means some amount of additional work for me (of which I already have
> plenty). So, is there a chance for such a change to be accepted?

Hi Rainer,

yes, I think there is, the only substantial concern I have is that of copyright
and licensing.

There will be proper credit to the copyright holder in the NEWS and/or THANKS
files, and if the author is distinct from the copyright owner, this could read
"(C) Copyright 2010 ACME Software Writing, written by Corey Coder", or similar.
 As long as I don't need to include your marketing brochure or slogan, but just
company name and place, that's fine.


Given this will likely be a nontrivial contribution according to our German
Urheberrechtsgesetz, and it apparently touches interests of your employer, we need:

- to make sure that your employer either waives his copyright and allows you to
claim it, or your employer is willing to contribute under a compatible license
(GPL "v2 or later" preferred). I am not trained in law and justice sciences, but
my take of German copyright law is that even if you're doing things in the spare
time that are closely related to your employment, your employer can still claim
the so-called Urheberschaft.

- state whether coverage under the Affero GNU GPL v3 would be OK, because I plan
to let future versions use that license unless the price to be paid (in ripping
out features where no such licensing is possible) would be too high, in that
case I'd try to use GNU GPL v3 in the long run.



With respect to the concerns you've raised,

(a) we're already talking, but I've not taken your past patch because of
technical concerns (as discussed earlier),

(b) is not something that has happened in the past few years on the fetchmail
list, and I trust Rob MacGregor as list operator to politely, but also
distinctly tell people not to go into such unfruitful discussions (or
bikeshedding); and he has my full backing for that,

(c) is a concern if we cannot reach an agreement on the licenses.  I feel no
need to adorn myself with somebody else's plumes, continued maintenance for half
a decade, fixing more than 200 bugs, speaks for itself.


I'd be grateful if we could make such DIMAP and scalable POP3 possible, and I
thank you for letting me know, because I have plans to attack POP3/UIDL
scalability myself soon -- I could devote my time in other ways if I know your
work covers that and reduces complexity to O(n log n) worst-case.

Please let me know your decision, or if there are major delays to be expected
because your bosses haven't got time to decide the licensing soonish.

Looking forward to your reply.

Best regards
Matthias

-- 
Matthias Andree

Re: [fetchmail-devel] save_str problems.

[Rainer W. <rwe...@ms...>]

Rainer Weikusat <rwe...@ms...> writes:
> Matthias Andree <mat...@gm...> writes:
>
> [...]
>
>> it also pretended that someone had already fixed the complexity of
>> the UID handling (which hasn't happened to the best of my knowledge;
>> I have started hacking a bit on it, but not sure if I'll make it, or
>> drop it and leave it for a later release using C++, if it turns out
>> it's too much of a hassle to do in C).
>
> I am done with my last 'firebrigade' assignment which means that I
> will now continue to work on the UID support for imap

I'm now the 'proud owner' of a fetchmail variant which works (mostly)
like a proper 'disconnected imap client'. The number of POP3 accounts
registered with the product I am using fetchmail for (except pulling
my mail of my employer's mailserver) has grown to 99 and can be
expected to increase further. Because of this, I'm determined to get
rid of the present UID management code 'soon' (probably, during the
next couple of 'Sunday shifts'). I've been a content fetchmail user
ever since I first got 'internet' in 1999 and I would happily direct
some useful work into it because of this. OTOH, my experience with OSS
projects so far is a) one gets ignored, b) all kinds of ml lurkers
start to enage in wild flaming and c) after the need has arisen, the
powers-that-be grudingly implement the missing features themselves,
just to ensure that only the worthy ones get any 'due credit', and this
means some amount of additional work for me (of which I already have
plenty). So, is there a chance for such a change to be accepted?

Re: [fetchmail-devel] save_str problems.

[Rainer W. <rwe...@ms...>]

Matthias Andree <mat...@gm...> writes:

[...]

> it also pretended that someone had already fixed the complexity of
> the UID handling (which hasn't happened to the best of my knowledge;
> I have started hacking a bit on it, but not sure if I'll make it, or
> drop it and leave it for a later release using C++, if it turns out
> it's too much of a hassle to do in C).

I am done with my last 'firebrigade' assignment which means that I
will now continue to work on the UID support for imap (which is
presently at the state that it uses UIDNEXT to detect new mails while
still using \Seen for determining their message sequence numbers and
to distinguish between old and new mails). The UID code isn't really
suitable for public consumption and won't become this without a major
additional effort (and given that my regular worktime is presently
around 53 hours/week, I won't have the time to do that). I will also
hopefully be able to get rid of the linked-list representation for the
POP3 UIDL database at latest until end of June (I hope), possibly
earlier, and I would be willing to base this on a 'public'
fetchmail-tree and share it if their was any interest in that (I am
also supposed to work on a completely different project which is
supposed to be demoed in June, but I think that I will be able to
interleave both).

[fetchmail-devel] fetchmail 6.3.17 security/regression fix release

[Matthias A. <mat...@gm...>]

The 6.3.17 release of fetchmail is now available at the usual locations,
including <http://developer.berlios.de/projects/fetchmail>.

The source archive is available at:
<http://developer.berlios.de/projects/fetchmail/fetchmail-6.3.17.tar.bz2>

Here are the release notes:

fetchmail-6.3.17 (released 2010-05-06, 25767 LoC):

# SECURITY FIX
* CVE-2010-1167: Fetchmail before release 6.3.17 did not properly sanitize
  external input (mail headers and UID). When a multi-character locale (such as
  UTF-8) was in use, this could cause memory exhaustion and thus a denial of
  service, because fetchmail's report.c functions assumed that non-success of
  [v]snprintf was due to insufficient buffer size allocation. It would then
  repeatedly reallocate a larger buffer and fail formatting again.
  See fetchmail-SA-2010-02.txt.

# FEATURES
* Fetchmail now supports a --sslcertfile <file> option to specify a "CA bundle"
  file (a file that contains trusted CA certificates). Since these bundled CA
  files do not require c_rehash to be run, they are easier to use and immune to
  OpenSSL library updates that affect the hash function.
* Fetchmail now supports a FETCHMAIL_INCLUDE_DEFAULT_X509_CA_CERTS
  environment variable to force loading the default SSL CA certificate
  locations even if --sslcertfile or --sslcertpath is used.
  If neither option is in effect, fetchmail loads the default locations.

# REGRESSION FIX
* Fix string handling in rcfile scanner, which caused fetchmail to misparse a
  run control file in certain circumstances.  Fixes BerliOS bug #14257.
  Patch by Michael Banack.  This fixes a regression introduced before 6.3.0.

# BUG FIXES
* Plug memory leak when using a "defaults" entry in the run control file.
* Do not print SSL certificate mismatches unless verbose or --sslcertck is
  enabled.
* Do not lose "set invisible" in fetchmailconf. (Michael Barnack)

# CHANGES
* Usability: SSL certificate chains are fully printed in -v -v mode, and there
  are now helpful pointers to --sslcertpath and c_rehash for "unable to get
  local issuer certificate" and self-signed certificates -- these usually hint
  to missing root signing CAs in the certs directory.
* Several fixes for compiler (GCC, Intel C++, CLang) and autotools warnings
* Memory allocation failures will now cause abnormal program abort (SIGABRT),
  no longer an exit with unspecified code.

# DOCUMENTATION
* Fix table of global option to read "set softbounce" where there used to be a
  2nd copy of "set spambounce".  Patch by Michael Banack, BerliOS Bug #17067.
* In the --sslcertpath description, mention that OpenSSL upgrade (and a 0.9.X
  to 1.0.0 upgrade in particular) may require running c_rehash.

# TRANSLATION UPDATES
  [zh_CN] Chinese/simplified (Ji Zheng-Yu)
  [cs]    Czech (Petr Pisar)
  [nl]    Dutch (Erwin Poeze)
  [fr]    French (Frédéric Marchal)
  [de]    German
  [id]    Indonesian (Andhika Padmawan)
  [it]    Italian (Vincenzo Campanella)
  [ja]    Japanese (Takeshi Hamasaki)
  [pl]    Polish (Jakub Bogusz)
  [sk]    Slovak (Marcel Telka)
  [vi]    Vietnamese (Clytie Siddall)

# KNOWN BUGS AND WORKAROUNDS:
  (this section floats upwards through the NEWS file so it stays with the
  current release information - however, it was stuck with 6.3.8 for a while)
* fetchmail does not handle messages without Message-ID header well
  (See sourceforge.net bug #780933)
* BSMTP is mostly untested and errors can cause corrupt output.
* Sun Workshop 6 (SPARC) is known to miscompile the configuration file lexer in
  64-bit mode.  Either compile 32-bit code or use GCC to compile 64-bit
  fetchmail.  Note that fetchmail doesn't take advantage of 64-bit code,
  so compiling 32-bit SPARC code should not cause any difficulties.
* fetchmail does not track pending deletes over crashes
* the command line interface is sometimes a bit stubborn, for instance,
  fetchmail -s doesn't work with a daemon running


By popular demand, diffs from the previous release have been omitted.

-- 
Matthias Andree

[fetchmail-devel] New Chinese (simplified) PO file for 'fetchmail' (version 6.3.17-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Chinese (simplified) team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/zh_CN.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Polish PO file for 'fetchmail' (version 6.3.17-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Polish team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/pl.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Italian PO file for 'fetchmail' (version 6.3.17-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Italian team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/it.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

Re: [fetchmail-devel] save_str problems.

[Matthias A. <mat...@gm...>]

Am 03.05.2010 14:33, schrieb Rainer Weikusat:
> Sunil Shetye <sh...@bo...> writes:
>> Quoting from Rainer Weikusat's mail on Mon, May 03, 2010:
>>>> Matthias, you have said that the regular uidl code is faster than the
>>>> fastuidl code for 10000 uids. Do you mean while processing 10000 uids
>>>> on the fetchmail side or while getting 10000 uids from the server?
>>>>
>>>> The fastuidl code downloads less UIDs from the server and hence has to
>>>> compare less UIDs to check for new mail. If there are very few new
>>>> mails, then the fastuidl code is expected to save a lot on the string
>>>> comparisons.
>>>
>>> It is conceivable that the string comparisons take less wallclock
>>> time than repeatedly asking a server for an UID and waiting for the
>>> replies does. IMHO, this is still a bad tradeoff because 'e-mail
>>> download' is a background batch job and even on a single-user
>>> workstation, minizing e-mail download time at the expense of directing
>>> more resources away from the interactive jobs the user interacts with
>>> doesn't really make sense except if one is using a uplink whose cost
>>> is proportional to the time spent 'online'.
>>
>> As Matthias has already said, the better solution is to use the
>> appropriate data structures like trees or hashes or vectors in order
>> to cut down both searches and modifications.
> 
> Sorry, but you are seriously lost in space here. I have no idea which
> thought could have motivated you to reply with this to my attempt at
> an explanation of the 'faster with 10000 UIDs' statement.

Guys, I was making that confusing statement, and it referred only to the
network and wallclock consideration, assumed networks with big
bandwidth-delay product, and it also pretended that someone had already
fixed the complexity of the UID handling (which hasn't happened to the
best of my knowledge; I have started hacking a bit on it, but not sure
if I'll make it, or drop it and leave it for a later release using C++,
if it turns out it's too much of a hassle to do in C).

fastuidl will reduce the network traffic considerably, and will also
save time on low-latency links.

Whether we do fastuidl or not, for any nontrivial amounts of messages
kept on the server for POP3, we need to fix the code to get rid of the
O(n^2) complexity.

Re: [fetchmail-devel] save_str problems.

[Rainer W. <rwe...@ms...>]

Sunil Shetye <sh...@bo...> writes:
> Quoting from Rainer Weikusat's mail on Mon, May 03, 2010:
>> > Matthias, you have said that the regular uidl code is faster than the
>> > fastuidl code for 10000 uids. Do you mean while processing 10000 uids
>> > on the fetchmail side or while getting 10000 uids from the server?
>> >
>> > The fastuidl code downloads less UIDs from the server and hence has to
>> > compare less UIDs to check for new mail. If there are very few new
>> > mails, then the fastuidl code is expected to save a lot on the string
>> > comparisons.
>> 
>> It is conceivable that the string comparisons take less wallclock
>> time than repeatedly asking a server for an UID and waiting for the
>> replies does. IMHO, this is still a bad tradeoff because 'e-mail
>> download' is a background batch job and even on a single-user
>> workstation, minizing e-mail download time at the expense of directing
>> more resources away from the interactive jobs the user interacts with
>> doesn't really make sense except if one is using a uplink whose cost
>> is proportional to the time spent 'online'.
>
> As Matthias has already said, the better solution is to use the
> appropriate data structures like trees or hashes or vectors in order
> to cut down both searches and modifications.

Sorry, but you are seriously lost in space here. I have no idea which
thought could have motivated you to reply with this to my attempt at
an explanation of the 'faster with 10000 UIDs' statement.

Re: [fetchmail-devel] save_str problems.

[Sunil S. <sh...@bo...>]

Quoting from Rainer Weikusat's mail on Mon, May 03, 2010:
> > Matthias, you have said that the regular uidl code is faster than the
> > fastuidl code for 10000 uids. Do you mean while processing 10000 uids
> > on the fetchmail side or while getting 10000 uids from the server?
> >
> > The fastuidl code downloads less UIDs from the server and hence has to
> > compare less UIDs to check for new mail. If there are very few new
> > mails, then the fastuidl code is expected to save a lot on the string
> > comparisons.
> 
> It is conceivable that the string comparisons take less wallclock
> time than repeatedly asking a server for an UID and waiting for the
> replies does. IMHO, this is still a bad tradeoff because 'e-mail
> download' is a background batch job and even on a single-user
> workstation, minizing e-mail download time at the expense of directing
> more resources away from the interactive jobs the user interacts with
> doesn't really make sense except if one is using a uplink whose cost
> is proportional to the time spent 'online'.

As Matthias has already said, the better solution is to use the
appropriate data structures like trees or hashes or vectors in order
to cut down both searches and modifications. Once done, regular uidl
and fastuidl will not eat up resources. Note that fastuidl is not the
source of this problem.

-- 
Sunil Shetye.

Re: [fetchmail-devel] save_str problems.

[Rainer W. <rwe...@ms...>]

Sunil Shetye <sh...@bo...> writes:
> Quoting from Sunil Shetye's mail on Fri, Apr 30, 2010:
>> Matthias, you have said that the regular uidl code is faster than the
>> fastuidl code for 10000 uids. Do you mean 10000 uids in the .fetchids
>> file or 10000 uids (i.e. 10000 mails) on the server?
>
> [ This should be read as ]
>
> Matthias, you have said that the regular uidl code is faster than the
> fastuidl code for 10000 uids. Do you mean while processing 10000 uids
> on the fetchmail side or while getting 10000 uids from the server?
>
> The fastuidl code downloads less UIDs from the server and hence has to
> compare less UIDs to check for new mail. If there are very few new
> mails, then the fastuidl code is expected to save a lot on the string
> comparisons.

It is conceivable that the string comparisons take less wallclock
time than repeatedly asking a server for an UID and waiting for the
replies does. IMHO, this is still a bad tradeoff because 'e-mail
download' is a background batch job and even on a single-user
workstation, minizing e-mail download time at the expense of directing
more resources away from the interactive jobs the user interacts with
doesn't really make sense except if one is using a uplink whose cost
is proportional to the time spent 'online'.

[fetchmail-devel] New Vietnamese PO file for 'fetchmail' (version 6.3.17-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Vietnamese team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/vi.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

Re: [fetchmail-devel] save_str problems.

[Sunil S. <sh...@bo...>]

[ Correction in previous mail ]

Quoting from Sunil Shetye's mail on Fri, Apr 30, 2010:
> Matthias, you have said that the regular uidl code is faster than the
> fastuidl code for 10000 uids. Do you mean 10000 uids in the .fetchids
> file or 10000 uids (i.e. 10000 mails) on the server?

[ This should be read as ]

Matthias, you have said that the regular uidl code is faster than the
fastuidl code for 10000 uids. Do you mean while processing 10000 uids
on the fetchmail side or while getting 10000 uids from the server?

The fastuidl code downloads less UIDs from the server and hence has to
compare less UIDs to check for new mail. If there are very few new
mails, then the fastuidl code is expected to save a lot on the string
comparisons.

-- 
Sunil Shetye.

[fetchmail-devel] New Indonesian PO file for 'fetchmail' (version 6.3.17-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Indonesian team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/id.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Czech PO file for 'fetchmail' (version 6.3.17-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Czech team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/cs.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Japanese PO file for 'fetchmail' (version 6.3.17-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Japanese team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/ja.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New French PO file for 'fetchmail' (version 6.3.17-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the French team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/fr.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New Dutch PO file for 'fetchmail' (version 6.3.17-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'fetchmail' has been submitted
by the Dutch team of translators.  The file is available at:

    http://translationproject.org/latest/fetchmail/nl.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/fetchmail/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/fetchmail.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

[fetchmail-devel] New template for 'fetchmail' made available

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.  (If you have
any questions, send them to <coo...@tr...>.)

A new POT file for textual domain 'fetchmail' has been made available
to the language teams for translation.  It is archived as:

    http://translationproject.org/POT-files/fetchmail-6.3.17-pre1.pot

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

Below is the URL which has been provided to the translators of your
package.  Please inform the translation coordinator, at the address
at the bottom, if this information is not current:

    http://www.dt.e-technik.tu-dortmund.de/~ma/fetchmail/fetchmail-6.3.17-pre1.tar.bz2

We can arrange things so that translated PO files are automatically e-mailed
to you when they arrive.  Ask at the address below if you want this.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

Re: [fetchmail-devel] save_str problems.

[Sunil S. <sh...@bo...>]

Hi,

> >which has the nice bonus property that it doesn't have the conditional
> >which goes one way during the first iteration and the other way during
> >all that follow inside the loop (the idea isn't mine, I originally
> >learnt about it because of some years-old USENET posting of a guy
> >whose name I've unfortunately forgotten). I am also rather concerned
> >about use of bandwidth than speed. Presently, I am dealing with 76
> >POP3 accounts and about a meg of stored UIDs I'd need to download
> >every five minutes in order to determine that presently, nothing needs
> >to be downloaded (this refers to fastuidl in general, assuming I
> >understood the principle correctly without really analyzing the code).
> 
> Makes sense, although I'm wondering if it really makes that much of
> a difference for fastuidl.
> fastuidl appears to be opportunistically harvesting message numbers,
> I wonder if that's of any use.
> I think Sunil wrote that fastUIDL code, I'm Cc:ing him.

The whole idea of the fastuidl code (and the unrelated fetchsizelimit
code) is to get the first mail fast. Previously, fetchmail would get
all the UIDs and sizes right at the start of the transaction.

POP3> STAT
POP3< +OK 10000 ...
POP3> UIDL
(gets 10000 uids)
(10000 is the first unseen)
POP3> LIST
(gets 10000 sizes)

With fastuidl and the delayed size information, the fetching of the
first mail is faster.

POP3> STAT
POP3< +OK 10000 ...
POP3> UIDL 5000
POP3> UIDL 7500
...
POP3> UIDL 10000
(10000 is the first unseen)
POP3> LIST 10000

These changes were done after observing that fetchmail had trouble
downloading even one mail over a slow connection, was effectively
going into an infinite loop, and was causing the associated wastage in
bandwidth.

Matthias, you have said that the regular uidl code is faster than the
fastuidl code for 10000 uids. Do you mean 10000 uids in the .fetchids
file or 10000 uids (i.e. 10000 mails) on the server?

The fastuidl code is expected to be faster if there are very few new
mails on the server and fetchmail is running with the keep option on.
As in the above case, there are 10000 mails, but only one new mail.

Please also compare how fast the *first* new mail is delivered with
and without fastuidl.

Of course, to stop the use of fastuidl, you may add 'fastuidl 0' to
the fetchmailrc file.

-- 
Sunil Shetye.