Re: [fetchmail-devel] POP3 UIDL behavior

[Matthias A. <mat...@gm...>]

Seth Mattinen schrieb am 2009-01-19:

> I was working on developing fetchmail for use in a large multiuser 
> environment and came across some issues with the POP3 UID lists. I'm 
> very new to fetchmail's code, so I don't know of these things are 
> intentional or expected behavior, but they seemed "broken" to me.

Hi Seth,

you're scratching a sore spot ;-) Except for minimal streamlining and
minor fixes, the uid.c code is what we inherited from Eric S. Raymond,
who was very loathe to touch anything of it since it broke all too
easily.

A while back, Sunil and I had exchanged patches to save UIDs in one file
per account, but I didn't merge that code into fetchmail, to make
fetchmail 6.3.X as drop-in compatible with 6.2.5 so that nobody would
have excuses for not upgrading from 6.2.X to 6.3.X.

Let me beforehand state that the data structures for storing the UID
lists are very inefficient, and costly at run-time. Several hundred kept
messages bog down old computers, and several thousands bog down modern
computers. We have cascaded linear lists, and we store the file very
inefficiently on-disk (which is less of an issue today).

The whole UID issue is for reconsideration when I really start working
on 6.4.X. At least, we need efficient memory and on-disk structures and
we'd preferably save to one uid file per account, so we can do away with
all the "load unrelated host info to scratchlist" stuff. It may be
useful to store everything in some lightweight/embedded database instead
(such as TokyoCabinet, Berkeley DB, SQLite). No decisions made yet, and
maybe we need to benchmark our options here during development.

> 1) The old UID list is never loaded when the daemon starts or restarts 
> and the whole POP3 box downloaded every time for "keep" sources. My fix 
> was to comment out the break after save_str() in uid.c because as far as 
> I can tell, having that break in there causes it to never load the id 
> file and it'll self-break when the for loop hits a null anyway.

I do not observe such behaviour - which version of fetchmail are you
looking at?

As far as I can see (it's around line 218 in my current uid.c version as
of post-6.3.9 SVN, in initialize_saved_lists()), it just breaks out of
"find the correct list" (there's a for loop three lines above) AFTER it
has saved the UID.

So if commenting out that break; at line 219 fixes anything for you,
your fix is for the symptoms, but not the root cause. We should then
find the latter.

> 2) Everything is duplicated in the scratchlist even for UID entries we 
> should know about so I added a boolean flag to skips duplicates. 
> Otherwise when the UID file is written it'll include oldlist plus 
> duplicates in scratchlist. Is it supposed to duplicate everything into 
> the scratchlist for some reason? This isn't a functional problem, but it 
> does waste memory and disk space if you're thinking about large deployments.

This appears to be a side effect of your commenting out the "break;",
since the for loop you made complete now terminates with uid == NULL.

Make sure the spelling on the command line matches the one in the rcfile
exactly (including case) for a test and see if that helps.

Could you show me your configuration file (remove/mask passwords!) and
your command line? Please do not edit host names, they are crucial here.
If you're concerned about disclosing that in public, send the material
to me GnuPG encrypted off-list (my key is 0x052e7d95), but please again
without passwords.

HTH

-- 
Matthias Andree