Re: [fetchmail-devel] POP3 UIDL behavior

[Matthias A. <mat...@gm...>]

Seth Mattinen schrieb am 2009-01-23:

> Something other than a linear list written to disk on every iteration 
> would obviously be better. Personally I wouldn't worry about backwards 
> compatibility. If it were me, I probably wouldn't bother with loading 
> anything into memory and just do queries against the database. It'll end 
> up in the disk cache anyway.

Yeah, makes sense, although it will make Cygwin users suffer.
Not that I care much.

> > As far as I can see (it's around line 218 in my current uid.c version as
> > of post-6.3.9 SVN, in initialize_saved_lists()), it just breaks out of
> > "find the correct list" (there's a for loop three lines above) AFTER it
> > has saved the UID.
> > 
> > So if commenting out that break; at line 219 fixes anything for you,
> > your fix is for the symptoms, but not the root cause. We should then
> > find the latter.
> 
> I was using the released 6.3.9 version, not the bleeding edge from SVN 
> since I'm not going for a development environment. The break seemed to 
> be causing it to exit the outermost loop though and neither oldlist nor 
> scratchlist were being populated at start time when a uidl file was present.

We have the following structure (roughly):

    while (fgets(buf, POPBUFSIZE, tmpfp) != (char *)NULL) {
    // ...
        /* find proper list and save it */
        for (ctl = hostlist; ctl; ctl = ctl->next) {
            if (strcasecmp(host, ctl->server.queryname) == 0
                    && strcasecmp(user, ctl->remotename) == 0) {
                save_str(&ctl->oldsaved, id, UID_SEEN);
                break;
            }
        }
    }

So if the "break;" breaks out of anything but the for ctl=... list,
something's seriously wrong. The "break;" applies to the for... loop for
as long as I remember C (that's nearly two decades now), not to the
outer while... loop. Anything else is a hardware problem,
compiler/optimizer bug or something is corrupting the heap or stack or
unrelated variables. Can you check that? The latter may be revealed by
running under valgrind (OTOH, OpenSSL reads uninitialized memory to
harvest entropy, so expect a gazillion of complaints unless you have an
ignore file for OpenSSL).

What compiler, version, compilers flags, and operating system are you
using? What platform/hardware does this happen on?

I can provide fetchmail 6.3.9 executables for FreeBSD 7 i386, Linux i386
and Solaris SPARC if we want to see if it's a compiler issue on your
machine.

> >> 2) Everything is duplicated in the scratchlist even for UID entries we 
> >> should know about so I added a boolean flag to skips duplicates. 
> >> Otherwise when the UID file is written it'll include oldlist plus 
> >> duplicates in scratchlist. Is it supposed to duplicate everything into 
> >> the scratchlist for some reason? This isn't a functional problem, but it 
> >> does waste memory and disk space if you're thinking about large deployments.
> > 
> > This appears to be a side effect of your commenting out the "break;",
> > since the for loop you made complete now terminates with uid == NULL.
> 
> It is since according to the debug output oldlist was never populated by 
> initialize_saved_lists(), but without the break they both ran.

fetchmail switches the current list to oldlist somewhere, and picks
either new or old uid lists in other places; check uid_swap_lists() and
expunge_uids(), and "fastuidl" also bears a meaning here WRT which list
fetchmail actually works on.

> This is the specific fetchmailrc file that it's currently running with:
> 
> set daemon 60
> set no syslog
> set logfile /var/lib/fetchmail/test/log
> set idfile /var/lib/fetchmail/test/uidl
> set no bouncemail
> set spambounce

Unrelated, but please don't to the latter. Spambouncing hurts innocent
bystanders.

Otherwise, configuration and command line are sound.

> And one more thing:
> 3) If two servers of the same name were present but one was "skip" and 
> the other "poll", every "skip" would cause the UID file to be written 
> with duplicate data because the server name is in ctl more than once so 
> each loop through ctl during the uidl file write duplicated data. Added 
> cases to check the skip/uidl flags and ignore servers without those set.
> 
> Here's my patch that I ended up with that fixed the issues I observed. 
> I've been running it for several days without any side effects. The only 
> changes are to uid.c.
> 
> http://www.rollernet.us/opensource/patches/fetchmail-rollernet-uidfixes2.patch
> 
> I haven't done extensive testing against it beyond the few cases where I 
> had issues.

I'm not convinced it's actually fixing bugs in fetchmail, rather than
working around them in strange ways. I do not dispute that the patch is
effective, but I'm loathe to apply the patch before I understand why
exactly it's needed. My configuration looks similar to yours as far as
UIDs are concerned, and I don't see problems here.

I'm inclined to believe in compiler bugs for now, but there may be a
fetchmail bug in a different location that corrupts innocent storage and
just happens to hit uid.c.

HTH

-- 
Matthias Andree