Re: [fetchmail-devel] POP3 UIDL behavior

[Seth M. <se...@ro...>]

As promised here's my test run with debugging enabled that prompted me 
to dig into the uid.c file. This is with stock fetchmail-6.3.9 
configured with "./configure --with-ssl":


fetchmail:/var/lib/fetchmail/test# cat uidl
cat: uidl: No such file or directory
fetchmail:/var/lib/fetchmail/test# su fetchmail -c 'fetchmail --quit -f 
/var/lib/fetchmail/test/fetchmailrc --pidfile 
/var/lib/fetchmail/test/fetchmail.pid -i /var/lib/fetchmail/test/uidl -vvN'
fetchmail: Old UID list from mail.mattinen.org: <empty>
fetchmail: Scratch list of UIDs: <empty>
fetchmail: starting fetchmail 6.3.9 daemon
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:55:50 PM PST: poll started
fetchmail: Trying to connect to 67.118.43.90/995...connected.
fetchmail: Issuer Organization: Courier Mail Server
fetchmail: Issuer CommonName: mail.mattinen.org
fetchmail: Server CommonName: mail.mattinen.org
fetchmail: mail.mattinen.org key fingerprint: 
0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C
fetchmail: Server certificate verification error: self signed certificate
fetchmail: Server certificate verification error: certificate has expired
fetchmail: POP3< +OK Dovecot ready.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK
fetchmail: POP3< CAPA
fetchmail: POP3< TOP
fetchmail: POP3< UIDL
fetchmail: POP3< RESP-CODES
fetchmail: POP3< PIPELINING
fetchmail: POP3< USER
fetchmail: POP3< SASL PLAIN
fetchmail: POP3< .
fetchmail: POP3> USER sethm
fetchmail: POP3< +OK
fetchmail: POP3> PASS *
fetchmail: POP3< +OK Logged in.
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 4 1792
fetchmail: POP3> UIDL
fetchmail: POP3< +OK
fetchmail: POP3< 1 0001179e45a1729f
fetchmail: 1 is unseen
fetchmail: POP3< 2 0001179f45a1729f
fetchmail: 2 is unseen
fetchmail: POP3< 3 000117a045a1729f
fetchmail: 3 is unseen
fetchmail: POP3< 4 000117a145a1729f
fetchmail: 4 is unseen
fetchmail: POP3< .
fetchmail: 4 messages for sethm at mail.mattinen.org (1792 octets).
...downloading happens here...
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Logging out.
fetchmail: SMTP> QUIT
fetchmail: SMTP< 221 2.0.0 Bye
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:55:55 PM PST: poll completed
fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 
0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty>
fetchmail: swapping UID lists
fetchmail: Writing fetchids file.
fetchmail: sleeping at Wed 28 Jan 2009 09:55:55 PM PST for 15 seconds
fetchmail: awakened at Wed 28 Jan 2009 09:56:10 PM PST
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:56:10 PM PST: poll started
fetchmail: Trying to connect to 67.118.43.90/995...connected.
fetchmail: Issuer Organization: Courier Mail Server
fetchmail: Issuer CommonName: mail.mattinen.org
fetchmail: Server CommonName: mail.mattinen.org
fetchmail: mail.mattinen.org key fingerprint: 
0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C
fetchmail: Server certificate verification error: self signed certificate
fetchmail: Server certificate verification error: certificate has expired
fetchmail: POP3< +OK Dovecot ready.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK
fetchmail: POP3< CAPA
fetchmail: POP3< TOP
fetchmail: POP3< UIDL
fetchmail: POP3< RESP-CODES
fetchmail: POP3< PIPELINING
fetchmail: POP3< USER
fetchmail: POP3< SASL PLAIN
fetchmail: POP3< .
fetchmail: POP3> USER sethm
fetchmail: POP3< +OK
fetchmail: POP3> PASS *
fetchmail: POP3< +OK Logged in.
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 4 1792
fetchmail: POP3> UIDL
fetchmail: POP3< +OK
fetchmail: POP3< 1 0001179e45a1729f
fetchmail: POP3< 2 0001179f45a1729f
fetchmail: POP3< 3 000117a045a1729f
fetchmail: POP3< 4 000117a145a1729f
fetchmail: POP3< .
fetchmail: 4 messages (4 seen) for sethm at mail.mattinen.org (1792 octets).
fetchmail: skipping message se...@ma...:1 not flushed
fetchmail: skipping message se...@ma...:2 not flushed
fetchmail: skipping message se...@ma...:3 not flushed
fetchmail: skipping message se...@ma...:4 not flushed
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Logging out.
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:56:10 PM PST: poll completed
fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 
0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty>
fetchmail: swapping UID lists
fetchmail: Query status=1 (NOMAIL)
fetchmail: Writing fetchids file.
fetchmail: sleeping at Wed 28 Jan 2009 09:56:10 PM PST for 15 seconds
fetchmail: terminated with signal 2
fetchmail: Writing fetchids file.

fetchmail:/var/lib/fetchmail/test# cat uidl
se...@ma... 0001179e45a1729f
se...@ma... 0001179f45a1729f
se...@ma... 000117a045a1729f
se...@ma... 000117a145a1729f


Everything was normal. The uidl file looks fine too. I'm expecting it to 
read the uidl file and skip those four messages, correct? So I restart 
the daemon:


fetchmail:/var/lib/fetchmail/test# su fetchmail -c 'fetchmail --quit -f 
/var/lib/fetchmail/test/fetchmailrc --pidfile 
/var/lib/fetchmail/test/fetchmail.pid -i /var/lib/fetchmail/test/uidl -vvN'
fetchmail: Old UID list from mail.mattinen.org: <empty>
fetchmail: Scratch list of UIDs: <empty>
fetchmail: starting fetchmail 6.3.9 daemon
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:56:16 PM PST: poll started
fetchmail: Trying to connect to 67.118.43.90/995...connected.
fetchmail: Issuer Organization: Courier Mail Server
fetchmail: Issuer CommonName: mail.mattinen.org
fetchmail: Server CommonName: mail.mattinen.org
fetchmail: mail.mattinen.org key fingerprint: 
0D:B9:98:10:A0:C7:B6:4C:70:92:6C:EE:C6:BB:99:1C
fetchmail: Server certificate verification error: self signed certificate
fetchmail: Server certificate verification error: certificate has expired
fetchmail: POP3< +OK Dovecot ready.
fetchmail: POP3> CAPA
fetchmail: POP3< +OK
fetchmail: POP3< CAPA
fetchmail: POP3< TOP
fetchmail: POP3< UIDL
fetchmail: POP3< RESP-CODES
fetchmail: POP3< PIPELINING
fetchmail: POP3< USER
fetchmail: POP3< SASL PLAIN
fetchmail: POP3< .
fetchmail: POP3> USER sethm
fetchmail: POP3< +OK
fetchmail: POP3> PASS *
fetchmail: POP3< +OK Logged in.
fetchmail: selecting or re-polling default folder
fetchmail: POP3> STAT
fetchmail: POP3< +OK 4 1792
fetchmail: POP3> UIDL
fetchmail: POP3< +OK
fetchmail: POP3< 1 0001179e45a1729f
fetchmail: 1 is unseen
fetchmail: POP3< 2 0001179f45a1729f
fetchmail: 2 is unseen
fetchmail: POP3< 3 000117a045a1729f
fetchmail: 3 is unseen
fetchmail: POP3< 4 000117a145a1729f
fetchmail: 4 is unseen
fetchmail: POP3< .
fetchmail: 4 messages for sethm at mail.mattinen.org (1792 octets).
...downloading happens here again even though it shouldn't...
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Logging out.
fetchmail: SMTP> QUIT
fetchmail: SMTP< 221 2.0.0 Bye
fetchmail: 6.3.9 querying mail.mattinen.org (protocol POP3) at Wed 28 
Jan 2009 09:56:21 PM PST: poll completed
fetchmail: New UID list from mail.mattinen.org: 0001179e45a1729f = 1 
0001179f45a1729f = 1 000117a045a1729f = 1 000117a145a1729f = 1 <empty>
fetchmail: swapping UID lists
fetchmail: Writing fetchids file.
fetchmail: sleeping at Wed 28 Jan 2009 09:56:21 PM PST for 15 seconds
fetchmail: terminated with signal 2
fetchmail: Writing fetchids file.

fetchmail:/var/lib/fetchmail/test# cat uidl
se...@ma... 0001179e45a1729f
se...@ma... 0001179f45a1729f
se...@ma... 000117a045a1729f
se...@ma... 000117a145a1729f
se...@ma... 0001179e45a1729f
se...@ma... 0001179f45a1729f
se...@ma... 000117a045a1729f
se...@ma... 000117a145a1729f


It re-downloaded everything in the POP3 box and duplicated the uidl file 
after the daemon was restarted. According to the debug output it didn't 
bother loading the existing uidl file. Continuing this pattern causes 
the uidl file to grow out of control.

-- 
Seth Mattinen		se...@ro...
Roller Network LLC