Menu
▾
▴
fetchmail-devel
|
From: Seth M. <se...@ro...> - 2009-01-20 00:36:02
|
I was working on developing fetchmail for use in a large multiuser environment and came across some issues with the POP3 UID lists. I'm very new to fetchmail's code, so I don't know of these things are intentional or expected behavior, but they seemed "broken" to me. 1) The old UID list is never loaded when the daemon starts or restarts and the whole POP3 box downloaded every time for "keep" sources. My fix was to comment out the break after save_str() in uid.c because as far as I can tell, having that break in there causes it to never load the id file and it'll self-break when the for loop hits a null anyway. 2) Everything is duplicated in the scratchlist even for UID entries we should know about so I added a boolean flag to skips duplicates. Otherwise when the UID file is written it'll include oldlist plus duplicates in scratchlist. Is it supposed to duplicate everything into the scratchlist for some reason? This isn't a functional problem, but it does waste memory and disk space if you're thinking about large deployments. I'm still muddling through this so I can roll these changes plus some other weirdness I'm seeing into a patch when I'm finished, if interested. -- Seth Mattinen se...@ro... Roller Network LLC |
|
From: Matthias A. <mat...@gm...> - 2009-01-23 09:56:55
|
Seth Mattinen schrieb am 2009-01-19: > I was working on developing fetchmail for use in a large multiuser > environment and came across some issues with the POP3 UID lists. I'm > very new to fetchmail's code, so I don't know of these things are > intentional or expected behavior, but they seemed "broken" to me. Hi Seth, you're scratching a sore spot ;-) Except for minimal streamlining and minor fixes, the uid.c code is what we inherited from Eric S. Raymond, who was very loathe to touch anything of it since it broke all too easily. A while back, Sunil and I had exchanged patches to save UIDs in one file per account, but I didn't merge that code into fetchmail, to make fetchmail 6.3.X as drop-in compatible with 6.2.5 so that nobody would have excuses for not upgrading from 6.2.X to 6.3.X. Let me beforehand state that the data structures for storing the UID lists are very inefficient, and costly at run-time. Several hundred kept messages bog down old computers, and several thousands bog down modern computers. We have cascaded linear lists, and we store the file very inefficiently on-disk (which is less of an issue today). The whole UID issue is for reconsideration when I really start working on 6.4.X. At least, we need efficient memory and on-disk structures and we'd preferably save to one uid file per account, so we can do away with all the "load unrelated host info to scratchlist" stuff. It may be useful to store everything in some lightweight/embedded database instead (such as TokyoCabinet, Berkeley DB, SQLite). No decisions made yet, and maybe we need to benchmark our options here during development. > 1) The old UID list is never loaded when the daemon starts or restarts > and the whole POP3 box downloaded every time for "keep" sources. My fix > was to comment out the break after save_str() in uid.c because as far as > I can tell, having that break in there causes it to never load the id > file and it'll self-break when the for loop hits a null anyway. I do not observe such behaviour - which version of fetchmail are you looking at? As far as I can see (it's around line 218 in my current uid.c version as of post-6.3.9 SVN, in initialize_saved_lists()), it just breaks out of "find the correct list" (there's a for loop three lines above) AFTER it has saved the UID. So if commenting out that break; at line 219 fixes anything for you, your fix is for the symptoms, but not the root cause. We should then find the latter. > 2) Everything is duplicated in the scratchlist even for UID entries we > should know about so I added a boolean flag to skips duplicates. > Otherwise when the UID file is written it'll include oldlist plus > duplicates in scratchlist. Is it supposed to duplicate everything into > the scratchlist for some reason? This isn't a functional problem, but it > does waste memory and disk space if you're thinking about large deployments. This appears to be a side effect of your commenting out the "break;", since the for loop you made complete now terminates with uid == NULL. Make sure the spelling on the command line matches the one in the rcfile exactly (including case) for a test and see if that helps. Could you show me your configuration file (remove/mask passwords!) and your command line? Please do not edit host names, they are crucial here. If you're concerned about disclosing that in public, send the material to me GnuPG encrypted off-list (my key is 0x052e7d95), but please again without passwords. HTH -- Matthias Andree |
|
From: Matthias A. <mat...@gm...> - 2009-01-23 10:01:25
|
Matthias Andree schrieb am 2009-01-23: > > 2) Everything is duplicated in the scratchlist even for UID entries we > > should know about so I added a boolean flag to skips duplicates. > > Otherwise when the UID file is written it'll include oldlist plus > > duplicates in scratchlist. Is it supposed to duplicate everything into > > the scratchlist for some reason? This isn't a functional problem, but it > > does waste memory and disk space if you're thinking about large deployments. > > This appears to be a side effect of your commenting out the "break;", > since the for loop you made complete now terminates with uid == NULL. Read that as "ctl == NULL". -- Matthias Andree |
|
From: Seth M. <se...@ro...> - 2009-01-23 19:17:17
|
Matthias Andree wrote: > Seth Mattinen schrieb am 2009-01-19: > >> I was working on developing fetchmail for use in a large multiuser >> environment and came across some issues with the POP3 UID lists. I'm >> very new to fetchmail's code, so I don't know of these things are >> intentional or expected behavior, but they seemed "broken" to me. > > Hi Seth, > > you're scratching a sore spot ;-) Except for minimal streamlining and > minor fixes, the uid.c code is what we inherited from Eric S. Raymond, > who was very loathe to touch anything of it since it broke all too > easily. > > A while back, Sunil and I had exchanged patches to save UIDs in one file > per account, but I didn't merge that code into fetchmail, to make > fetchmail 6.3.X as drop-in compatible with 6.2.5 so that nobody would > have excuses for not upgrading from 6.2.X to 6.3.X. > > Let me beforehand state that the data structures for storing the UID > lists are very inefficient, and costly at run-time. Several hundred kept > messages bog down old computers, and several thousands bog down modern > computers. We have cascaded linear lists, and we store the file very > inefficiently on-disk (which is less of an issue today). > > The whole UID issue is for reconsideration when I really start working > on 6.4.X. At least, we need efficient memory and on-disk structures and > we'd preferably save to one uid file per account, so we can do away with > all the "load unrelated host info to scratchlist" stuff. It may be > useful to store everything in some lightweight/embedded database instead > (such as TokyoCabinet, Berkeley DB, SQLite). No decisions made yet, and > maybe we need to benchmark our options here during development. Something other than a linear list written to disk on every iteration would obviously be better. Personally I wouldn't worry about backwards compatibility. If it were me, I probably wouldn't bother with loading anything into memory and just do queries against the database. It'll end up in the disk cache anyway. >> 1) The old UID list is never loaded when the daemon starts or restarts >> and the whole POP3 box downloaded every time for "keep" sources. My fix >> was to comment out the break after save_str() in uid.c because as far as >> I can tell, having that break in there causes it to never load the id >> file and it'll self-break when the for loop hits a null anyway. > > I do not observe such behaviour - which version of fetchmail are you > looking at? > > As far as I can see (it's around line 218 in my current uid.c version as > of post-6.3.9 SVN, in initialize_saved_lists()), it just breaks out of > "find the correct list" (there's a for loop three lines above) AFTER it > has saved the UID. > > So if commenting out that break; at line 219 fixes anything for you, > your fix is for the symptoms, but not the root cause. We should then > find the latter. I was using the released 6.3.9 version, not the bleeding edge from SVN since I'm not going for a development environment. The break seemed to be causing it to exit the outermost loop though and neither oldlist nor scratchlist were being populated at start time when a uidl file was present. >> 2) Everything is duplicated in the scratchlist even for UID entries we >> should know about so I added a boolean flag to skips duplicates. >> Otherwise when the UID file is written it'll include oldlist plus >> duplicates in scratchlist. Is it supposed to duplicate everything into >> the scratchlist for some reason? This isn't a functional problem, but it >> does waste memory and disk space if you're thinking about large deployments. > > This appears to be a side effect of your commenting out the "break;", > since the for loop you made complete now terminates with uid == NULL. It is since according to the debug output oldlist was never populated by initialize_saved_lists(), but without the break they both ran. > Make sure the spelling on the command line matches the one in the rcfile > exactly (including case) for a test and see if that helps. > > Could you show me your configuration file (remove/mask passwords!) and > your command line? Please do not edit host names, they are crucial here. > If you're concerned about disclosing that in public, send the material > to me GnuPG encrypted off-list (my key is 0x052e7d95), but please again > without passwords. > This is the specific fetchmailrc file that it's currently running with: set daemon 60 set no syslog set logfile /var/lib/fetchmail/test/log set idfile /var/lib/fetchmail/test/uidl set no bouncemail set spambounce set postmaster "fet...@bo..." defaults pass8bits smtphost mail2.rollernet.us,mail.rollernet.us antispam 554 skip mail.mattinen.org with proto imap and tracepolls user "sethm" pass "xxxxxxxxxxxxx" smtpname "se...@ro..." ssl poll mail.mattinen.org proto pop3 uidl tracepolls user "sethm" pass "xxxxxxxxxxxxx" smtpname "se...@ro..." ssl keep fastuidl 4 su fetchmail -c \ 'fetchmail --quit \ -f /var/lib/fetchmail/test/fetchmailrc \ --pidfile /var/lib/fetchmail/test/fetchmail.pid' And one more thing: 3) If two servers of the same name were present but one was "skip" and the other "poll", every "skip" would cause the UID file to be written with duplicate data because the server name is in ctl more than once so each loop through ctl during the uidl file write duplicated data. Added cases to check the skip/uidl flags and ignore servers without those set. Here's my patch that I ended up with that fixed the issues I observed. I've been running it for several days without any side effects. The only changes are to uid.c. http://www.rollernet.us/opensource/patches/fetchmail-rollernet-uidfixes2.patch I haven't done extensive testing against it beyond the few cases where I had issues. -- Seth Mattinen se...@ro... Roller Network LLC |
|
From: Matthias A. <mat...@gm...> - 2009-01-23 23:41:34
|
Seth Mattinen schrieb am 2009-01-23:
> Something other than a linear list written to disk on every iteration
> would obviously be better. Personally I wouldn't worry about backwards
> compatibility. If it were me, I probably wouldn't bother with loading
> anything into memory and just do queries against the database. It'll end
> up in the disk cache anyway.
Yeah, makes sense, although it will make Cygwin users suffer.
Not that I care much.
> > As far as I can see (it's around line 218 in my current uid.c version as
> > of post-6.3.9 SVN, in initialize_saved_lists()), it just breaks out of
> > "find the correct list" (there's a for loop three lines above) AFTER it
> > has saved the UID.
> >
> > So if commenting out that break; at line 219 fixes anything for you,
> > your fix is for the symptoms, but not the root cause. We should then
> > find the latter.
>
> I was using the released 6.3.9 version, not the bleeding edge from SVN
> since I'm not going for a development environment. The break seemed to
> be causing it to exit the outermost loop though and neither oldlist nor
> scratchlist were being populated at start time when a uidl file was present.
We have the following structure (roughly):
while (fgets(buf, POPBUFSIZE, tmpfp) != (char *)NULL) {
// ...
/* find proper list and save it */
for (ctl = hostlist; ctl; ctl = ctl->next) {
if (strcasecmp(host, ctl->server.queryname) == 0
&& strcasecmp(user, ctl->remotename) == 0) {
save_str(&ctl->oldsaved, id, UID_SEEN);
break;
}
}
}
So if the "break;" breaks out of anything but the for ctl=... list,
something's seriously wrong. The "break;" applies to the for... loop for
as long as I remember C (that's nearly two decades now), not to the
outer while... loop. Anything else is a hardware problem,
compiler/optimizer bug or something is corrupting the heap or stack or
unrelated variables. Can you check that? The latter may be revealed by
running under valgrind (OTOH, OpenSSL reads uninitialized memory to
harvest entropy, so expect a gazillion of complaints unless you have an
ignore file for OpenSSL).
What compiler, version, compilers flags, and operating system are you
using? What platform/hardware does this happen on?
I can provide fetchmail 6.3.9 executables for FreeBSD 7 i386, Linux i386
and Solaris SPARC if we want to see if it's a compiler issue on your
machine.
> >> 2) Everything is duplicated in the scratchlist even for UID entries we
> >> should know about so I added a boolean flag to skips duplicates.
> >> Otherwise when the UID file is written it'll include oldlist plus
> >> duplicates in scratchlist. Is it supposed to duplicate everything into
> >> the scratchlist for some reason? This isn't a functional problem, but it
> >> does waste memory and disk space if you're thinking about large deployments.
> >
> > This appears to be a side effect of your commenting out the "break;",
> > since the for loop you made complete now terminates with uid == NULL.
>
> It is since according to the debug output oldlist was never populated by
> initialize_saved_lists(), but without the break they both ran.
fetchmail switches the current list to oldlist somewhere, and picks
either new or old uid lists in other places; check uid_swap_lists() and
expunge_uids(), and "fastuidl" also bears a meaning here WRT which list
fetchmail actually works on.
> This is the specific fetchmailrc file that it's currently running with:
>
> set daemon 60
> set no syslog
> set logfile /var/lib/fetchmail/test/log
> set idfile /var/lib/fetchmail/test/uidl
> set no bouncemail
> set spambounce
Unrelated, but please don't to the latter. Spambouncing hurts innocent
bystanders.
Otherwise, configuration and command line are sound.
> And one more thing:
> 3) If two servers of the same name were present but one was "skip" and
> the other "poll", every "skip" would cause the UID file to be written
> with duplicate data because the server name is in ctl more than once so
> each loop through ctl during the uidl file write duplicated data. Added
> cases to check the skip/uidl flags and ignore servers without those set.
>
> Here's my patch that I ended up with that fixed the issues I observed.
> I've been running it for several days without any side effects. The only
> changes are to uid.c.
>
> http://www.rollernet.us/opensource/patches/fetchmail-rollernet-uidfixes2.patch
>
> I haven't done extensive testing against it beyond the few cases where I
> had issues.
I'm not convinced it's actually fixing bugs in fetchmail, rather than
working around them in strange ways. I do not dispute that the patch is
effective, but I'm loathe to apply the patch before I understand why
exactly it's needed. My configuration looks similar to yours as far as
UIDs are concerned, and I don't see problems here.
I'm inclined to believe in compiler bugs for now, but there may be a
fetchmail bug in a different location that corrupts innocent storage and
just happens to hit uid.c.
HTH
--
Matthias Andree
|
|
From: Seth M. <se...@ro...> - 2009-01-24 00:06:25
|
Matthias Andree wrote:
> We have the following structure (roughly):
>
> while (fgets(buf, POPBUFSIZE, tmpfp) != (char *)NULL) {
> // ...
> /* find proper list and save it */
> for (ctl = hostlist; ctl; ctl = ctl->next) {
> if (strcasecmp(host, ctl->server.queryname) == 0
> && strcasecmp(user, ctl->remotename) == 0) {
> save_str(&ctl->oldsaved, id, UID_SEEN);
> break;
> }
> }
> }
>
> So if the "break;" breaks out of anything but the for ctl=... list,
> something's seriously wrong. The "break;" applies to the for... loop for
> as long as I remember C (that's nearly two decades now), not to the
> outer while... loop. Anything else is a hardware problem,
> compiler/optimizer bug or something is corrupting the heap or stack or
> unrelated variables. Can you check that? The latter may be revealed by
> running under valgrind (OTOH, OpenSSL reads uninitialized memory to
> harvest entropy, so expect a gazillion of complaints unless you have an
> ignore file for OpenSSL).
I completely agree; something weird is going on.
> What compiler, version, compilers flags, and operating system are you
> using? What platform/hardware does this happen on?
Vanilla Debian "stable" (kernel 2.6.18) on amd64.
fetchmail:/usr/local/src/fetchmail-6.3.9# gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v
--enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr
--enable-shared --with-system-zlib --libexecdir=/usr/lib
--without-included-gettext --enable-threads=posix --enable-nls
--program-suffix=-4.1 --enable-__cxa_atexit --enable-clocale=gnu
--enable-libstdcxx-debug --enable-mpfr --enable-checking=release
x86_64-linux-gnu
Thread model: posix
gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
fetchmail:/usr/local/src/fetchmail-6.3.9-roller# make
make all-recursive
make[1]: Entering directory `/usr/local/src/fetchmail-6.3.9-roller'
Making all in m4
make[2]: Entering directory `/usr/local/src/fetchmail-6.3.9-roller/m4'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/usr/local/src/fetchmail-6.3.9-roller/m4'
Making all in po
make[2]: Entering directory `/usr/local/src/fetchmail-6.3.9-roller/po'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/usr/local/src/fetchmail-6.3.9-roller/po'
make[2]: Entering directory `/usr/local/src/fetchmail-6.3.9-roller'
gcc -DHAVE_CONFIG_H -DLOCALEDIR=\"/usr/local/share/locale\" -I.
-I./libesmtp -g -O2 -I/usr/kerberos/include -MT uid.o -MD -MP -MF
.deps/uid.Tpo -c -o uid.o uid.c
mv -f .deps/uid.Tpo .deps/uid.Po
gcc -g -O2 -I/usr/kerberos/include -L/usr/lib -o fetchmail socket.o
getpass.o fetchmail.o env.o idle.o options.o daemon.o driver.o
transact.o sink.o smtp.o uid.o mxget.o md5ify.o cram.o gssapi.o opie.o
interface.o netrc.o unmime.o conf.o checkalias.o lock.o rcfile_l.o
rcfile_y.o norm_charmap.o pop3.o imap.o etrn.o odmr.o libfm.a
strlcpy.o strlcat.o -lcrypt -lresolv -lssl -lcrypto
make[2]: Leaving directory `/usr/local/src/fetchmail-6.3.9-roller'
make[1]: Leaving directory `/usr/local/src/fetchmail-6.3.9-roller'
>
>> This is the specific fetchmailrc file that it's currently running with:
>>
>> set daemon 60
>> set no syslog
>> set logfile /var/lib/fetchmail/test/log
>> set idfile /var/lib/fetchmail/test/uidl
>> set no bouncemail
>> set spambounce
>
> Unrelated, but please don't to the latter. Spambouncing hurts innocent
> bystanders.
I've made several changes to sink.c so it will only send bounce notices
to the defined postmaster address. ;)
> I'm not convinced it's actually fixing bugs in fetchmail, rather than
> working around them in strange ways. I do not dispute that the patch is
> effective, but I'm loathe to apply the patch before I understand why
> exactly it's needed. My configuration looks similar to yours as far as
> UIDs are concerned, and I don't see problems here.
>
> I'm inclined to believe in compiler bugs for now, but there may be a
> fetchmail bug in a different location that corrupts innocent storage and
> just happens to hit uid.c.
>
When I have time (probably next week) I'll collect the debug output from
before and after the patch to illustrate. The event that started me on
this quest was fetchmail re-downloading the entire contents of a "keep"
POP3 box every time the daemon started even though I had uidl enabled.
--
Seth Mattinen se...@ro...
Roller Network LLC
|
|
From: Seth M. <se...@ro...> - 2009-01-29 07:04:00
|
As promised here's my test run with debugging enabled that prompted me to dig into the uid.c file. This is with stock fetchmail-6.3.9 configured with "./configure --with-ssl": fetchmail:/var/lib/fetchmail/test# cat uidl cat: uidl: No such file or directory fetchmail:/var/lib/fetchmail/test# su fetchmail -c 'fetchmail --quit -f /var/lib/fetchmail/test/fetchmailrc --pidfile /var/lib/fetchmail/test/fetchmail.pid -i /var/lib/fetchmail/test/uidl -vvN' fetchmail: Old UID list from mail.mattinen.org: <empty> fetchmail: Scratch list of UIDs: <empty> fetchmail: starting fetchmail 6.3.9 daemon fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 Jan 2009 09:55:50 PM PST: poll started fetchmail: Trying to connect to 67.118.43.90/995...connected. fetchmail: Issuer Organization: Courier Mail Server fetchmail: Issuer CommonName: mail.mattinen.org fetchmail: Server CommonName: mail.mattinen.org fetchmail: mail.mattinen.org key fingerprint: 0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C fetchmail: Server certificate verification error: self signed certificate fetchmail: Server certificate verification error: certificate has expired fetchmail: POP3< +OK Dovecot ready. fetchmail: POP3> CAPA fetchmail: POP3< +OK fetchmail: POP3< CAPA fetchmail: POP3< TOP fetchmail: POP3< UIDL fetchmail: POP3< RESP-CODES fetchmail: POP3< PIPELINING fetchmail: POP3< USER fetchmail: POP3< SASL PLAIN fetchmail: POP3< . fetchmail: POP3> USER sethm fetchmail: POP3< +OK fetchmail: POP3> PASS * fetchmail: POP3< +OK Logged in. fetchmail: selecting or re-polling default folder fetchmail: POP3> STAT fetchmail: POP3< +OK 4 1792 fetchmail: POP3> UIDL fetchmail: POP3< +OK fetchmail: POP3< 1 0001179e45a1729f fetchmail: 1 is unseen fetchmail: POP3< 2 0001179f45a1729f fetchmail: 2 is unseen fetchmail: POP3< 3 000117a045a1729f fetchmail: 3 is unseen fetchmail: POP3< 4 000117a145a1729f fetchmail: 4 is unseen fetchmail: POP3< . fetchmail: 4 messages for sethm at mail.mattinen.org (1792 octets). ...downloading happens here... fetchmail: POP3> QUIT fetchmail: POP3< +OK Logging out. fetchmail: SMTP> QUIT fetchmail: SMTP< 221 2.0.0 Bye fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 Jan 2009 09:55:55 PM PST: poll completed fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty> fetchmail: swapping UID lists fetchmail: Writing fetchids file. fetchmail: sleeping at Wed 28 Jan 2009 09:55:55 PM PST for 15 seconds fetchmail: awakened at Wed 28 Jan 2009 09:56:10 PM PST fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 Jan 2009 09:56:10 PM PST: poll started fetchmail: Trying to connect to 67.118.43.90/995...connected. fetchmail: Issuer Organization: Courier Mail Server fetchmail: Issuer CommonName: mail.mattinen.org fetchmail: Server CommonName: mail.mattinen.org fetchmail: mail.mattinen.org key fingerprint: 0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C fetchmail: Server certificate verification error: self signed certificate fetchmail: Server certificate verification error: certificate has expired fetchmail: POP3< +OK Dovecot ready. fetchmail: POP3> CAPA fetchmail: POP3< +OK fetchmail: POP3< CAPA fetchmail: POP3< TOP fetchmail: POP3< UIDL fetchmail: POP3< RESP-CODES fetchmail: POP3< PIPELINING fetchmail: POP3< USER fetchmail: POP3< SASL PLAIN fetchmail: POP3< . fetchmail: POP3> USER sethm fetchmail: POP3< +OK fetchmail: POP3> PASS * fetchmail: POP3< +OK Logged in. fetchmail: selecting or re-polling default folder fetchmail: POP3> STAT fetchmail: POP3< +OK 4 1792 fetchmail: POP3> UIDL fetchmail: POP3< +OK fetchmail: POP3< 1 0001179e45a1729f fetchmail: POP3< 2 0001179f45a1729f fetchmail: POP3< 3 000117a045a1729f fetchmail: POP3< 4 000117a145a1729f fetchmail: POP3< . fetchmail: 4 messages (4 seen) for sethm at mail.mattinen.org (1792 octets). fetchmail: skipping message se...@ma...:1 not flushed fetchmail: skipping message se...@ma...:2 not flushed fetchmail: skipping message se...@ma...:3 not flushed fetchmail: skipping message se...@ma...:4 not flushed fetchmail: POP3> QUIT fetchmail: POP3< +OK Logging out. fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 Jan 2009 09:56:10 PM PST: poll completed fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty> fetchmail: swapping UID lists fetchmail: Query status=1 (NOMAIL) fetchmail: Writing fetchids file. fetchmail: sleeping at Wed 28 Jan 2009 09:56:10 PM PST for 15 seconds fetchmail: terminated with signal 2 fetchmail: Writing fetchids file. fetchmail:/var/lib/fetchmail/test# cat uidl se...@ma... 0001179e45a1729f se...@ma... 0001179f45a1729f se...@ma... 000117a045a1729f se...@ma... 000117a145a1729f Everything was normal. The uidl file looks fine too. I'm expecting it to read the uidl file and skip those four messages, correct? So I restart the daemon: fetchmail:/var/lib/fetchmail/test# su fetchmail -c 'fetchmail --quit -f /var/lib/fetchmail/test/fetchmailrc --pidfile /var/lib/fetchmail/test/fetchmail.pid -i /var/lib/fetchmail/test/uidl -vvN' fetchmail: Old UID list from mail.mattinen.org: <empty> fetchmail: Scratch list of UIDs: <empty> fetchmail: starting fetchmail 6.3.9 daemon fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 Jan 2009 09:56:16 PM PST: poll started fetchmail: Trying to connect to 67.118.43.90/995...connected. fetchmail: Issuer Organization: Courier Mail Server fetchmail: Issuer CommonName: mail.mattinen.org fetchmail: Server CommonName: mail.mattinen.org fetchmail: mail.mattinen.org key fingerprint: 0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C fetchmail: Server certificate verification error: self signed certificate fetchmail: Server certificate verification error: certificate has expired fetchmail: POP3< +OK Dovecot ready. fetchmail: POP3> CAPA fetchmail: POP3< +OK fetchmail: POP3< CAPA fetchmail: POP3< TOP fetchmail: POP3< UIDL fetchmail: POP3< RESP-CODES fetchmail: POP3< PIPELINING fetchmail: POP3< USER fetchmail: POP3< SASL PLAIN fetchmail: POP3< . fetchmail: POP3> USER sethm fetchmail: POP3< +OK fetchmail: POP3> PASS * fetchmail: POP3< +OK Logged in. fetchmail: selecting or re-polling default folder fetchmail: POP3> STAT fetchmail: POP3< +OK 4 1792 fetchmail: POP3> UIDL fetchmail: POP3< +OK fetchmail: POP3< 1 0001179e45a1729f fetchmail: 1 is unseen fetchmail: POP3< 2 0001179f45a1729f fetchmail: 2 is unseen fetchmail: POP3< 3 000117a045a1729f fetchmail: 3 is unseen fetchmail: POP3< 4 000117a145a1729f fetchmail: 4 is unseen fetchmail: POP3< . fetchmail: 4 messages for sethm at mail.mattinen.org (1792 octets). ...downloading happens here again even though it shouldn't... fetchmail: POP3> QUIT fetchmail: POP3< +OK Logging out. fetchmail: SMTP> QUIT fetchmail: SMTP< 221 2.0.0 Bye fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 Jan 2009 09:56:21 PM PST: poll completed fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty> fetchmail: swapping UID lists fetchmail: Writing fetchids file. fetchmail: sleeping at Wed 28 Jan 2009 09:56:21 PM PST for 15 seconds fetchmail: terminated with signal 2 fetchmail: Writing fetchids file. fetchmail:/var/lib/fetchmail/test# cat uidl se...@ma... 0001179e45a1729f se...@ma... 0001179f45a1729f se...@ma... 000117a045a1729f se...@ma... 000117a145a1729f se...@ma... 0001179e45a1729f se...@ma... 0001179f45a1729f se...@ma... 000117a045a1729f se...@ma... 000117a145a1729f It re-downloaded everything in the POP3 box and duplicated the uidl file after the daemon was restarted. According to the debug output it didn't bother loading the existing uidl file. Continuing this pattern causes the uidl file to grow out of control. -- Seth Mattinen se...@ro... Roller Network LLC |
|
From: Seth M. <se...@ro...> - 2009-01-29 07:10:48
|
And here's the debug output using my patched version of fetchmail: fetchmail:/var/lib/fetchmail/test# cat uidl se...@ma... 0001179e45a1729f se...@ma... 0001179f45a1729f se...@ma... 000117a045a1729f se...@ma... 000117a145a1729f se...@ma... 0001179e45a1729f se...@ma... 0001179f45a1729f se...@ma... 000117a045a1729f se...@ma... 000117a145a1729f fetchmail:/var/lib/fetchmail/test# su fetchmail -c 'fetchmail --quit -f /var/lib/fetchmail/test/fetchmailrc --pidfile /var/lib/fetchmail/test/fetchmail.pid -i /var/lib/fetchmail/test/uidl -vvN' fetchmail: Old UID list from mail.mattinen.org: 0001179e45a1729f 0001179f45a1729f 000117a045a1729f 000117a145a1729f 0001179e45a1729f 0001179f45a1729f 000117a045a1729f 000117a145a1729f <empty> fetchmail: Scratch list of UIDs: <empty> fetchmail: starting fetchmail 6.3.9 daemon fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 Jan 2009 10:08:20 PM PST: poll started fetchmail: Trying to connect to 67.118.43.90/995...connected. fetchmail: Issuer Organization: Courier Mail Server fetchmail: Issuer CommonName: mail.mattinen.org fetchmail: Server CommonName: mail.mattinen.org fetchmail: mail.mattinen.org key fingerprint: 0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C fetchmail: Server certificate verification error: self signed certificate fetchmail: Server certificate verification error: certificate has expired fetchmail: POP3< +OK Dovecot ready. fetchmail: POP3> CAPA fetchmail: POP3< +OK fetchmail: POP3< CAPA fetchmail: POP3< TOP fetchmail: POP3< UIDL fetchmail: POP3< RESP-CODES fetchmail: POP3< PIPELINING fetchmail: POP3< USER fetchmail: POP3< SASL PLAIN fetchmail: POP3< . fetchmail: POP3> USER sethm fetchmail: POP3< +OK fetchmail: POP3> PASS * fetchmail: POP3< +OK Logged in. fetchmail: selecting or re-polling default folder fetchmail: POP3> STAT fetchmail: POP3< +OK 4 1792 fetchmail: POP3> UIDL fetchmail: POP3< +OK fetchmail: POP3< 1 0001179e45a1729f fetchmail: POP3< 2 0001179f45a1729f fetchmail: POP3< 3 000117a045a1729f fetchmail: POP3< 4 000117a145a1729f fetchmail: POP3< . fetchmail: 4 messages (4 seen) for sethm at mail.mattinen.org (1792 octets). fetchmail: skipping message se...@ma...:1 not flushed fetchmail: skipping message se...@ma...:2 not flushed fetchmail: skipping message se...@ma...:3 not flushed fetchmail: skipping message se...@ma...:4 not flushed fetchmail: POP3> QUIT fetchmail: POP3< +OK Logging out. fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 Jan 2009 10:08:20 PM PST: poll completed fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty> fetchmail: swapping UID lists fetchmail: Query status=1 (NOMAIL) fetchmail: Writing fetchids file. fetchmail: sleeping at Wed 28 Jan 2009 10:08:20 PM PST for 15 seconds fetchmail: terminated with signal 2 fetchmail: Writing fetchids file. fetchmail:/var/lib/fetchmail/test# cat uidl se...@ma... 0001179e45a1729f se...@ma... 0001179f45a1729f se...@ma... 000117a045a1729f se...@ma... 000117a145a1729f Nothing in the configuration was changed and the same uidl file (including the duplicates, which were weeded out) was used. Note the oldlist is now populated at start time whereas it wasn't with the unpatched version. -- Seth Mattinen se...@ro... Roller Network LLC |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X