fetchmail-users — User questions and discussion

Re: [fetchmail-users] self-signed certs + fetchmail nuisance

[Matthias A. <mat...@gm...>]

Volker Kuhlmann <lis...@pa...> writes:

> I just upgraded from SUSE 10.0 to 10.1, and with it to fetchmail 6.3.2.
> Now I see that one of my email providers must have introduced TLS, but
> with a self-signed cert. The first time cron mails me a
>
> fetchmail: Server certificate verification error: self signed
> certificate

> it's informative, but after the 1735th time the novelty value has worn a
> bit.

This issue is fixed in the latest available release, 6.3.4, where
sslfingerprint (on the command line or in the rcfile) should suppress these
warnings unless sslcertck is enabled.

Your options are (pick at least one):

- ask your ISP to provide proper SSL certificates

- list sslfingerpint AND
  ask Novell (SUSE) to update fetchmail to 6.3.4
      or cherrypick(*) these changes from 6.3.4:

  * SSL/TLS: if, for a certain server, an sslfingerprint is specified and
    sslcertck is NOT set, suppress printing SSL certificate mismatch errors.
    (Reported by Hannes Erven.)
  * SSL/TLS: always print if the sslfingerprint mismatches, even in silent
    mode.  (This is for consistency with certificate verification errors.)

  (*) For cherrypicking, the
  repository is: http://mknod.org/svn/fetchmail/BRANCH_6-3
  to pull: svn diff -r4780:4781

- ask your ISP for their home-made CA root certificate that you can
  stuff into your /etc/ssl/certs (or whatever your CApath is).

-- 
Matthias Andree

[fetchmail-users] self-signed certs + fetchmail nuisance

[Volker K. <lis...@pa...>]

I just upgraded from SUSE 10.0 to 10.1, and with it to fetchmail 6.3.2.
Now I see that one of my email providers must have introduced TLS, but
with a self-signed cert. The first time cron mails me a

fetchmail: Server certificate verification error: self signed certificate

it's informative, but after the 1735th time the novelty value has worn a
bit.

How do I get rid of this? The manpage is not helping. I added an
sslfingerprint, no difference. I'm quite sure that mail from and to this
provider is not encrypted, so strong protection between me and the
provider is rather moot, but it does protect my download password. I'm
not concerned with making sure of getting the right fingerprint (there
alternative is using a clear text password). The server is obviously not
mine to fix. Surely there must be a better solution than turning off ssl
altogether, or filtering fetchmail-cron mail to /dev/null.

Please nobody suggest I shouldn't be using cron - I'd be happy to switch
to daemon mode if it allowed me to specify the fetch times for each
mailbox. (Yes I have a somewhat elaborate wrapper script which handels
periodic error mailing and logging.)

Thanks much,

Volker

-- 
Volker Kuhlmann			is list0570 with the domain in header
http://volker.dnsalias.net/	Please do not CC list postings to me.

[fetchmail-users] email not stored ?

[RaFFe <raf...@ya...>]

Hej,

I've configured fetchmail right, it connect right to
my ISP server and successful read email, but I
don't where they was stored...
All file in my /var/spool/mail stay empty and no
log file was created with any potential error messages ...

Does someone have any ideas about my problem ?

Thanks in advance,

Raffe

Re: [fetchmail-users] Simply forward mail?

[Rob F. <rf...@fu...>]

Ken Williams wrote:
> Hi, I'm lost.
>
> I run sendmail on linux.  I have a mail drop (/var/spool/mail/user)
> with some mail in there. I simply want to take this mail and forward it
> to another email address like say us...@ya....
>
> How can I do this?  Seems to be the most difficult thing in the world.

Not exactly a fetchmail question, but.....

formail -ts /usr/sbin/sendmail us...@ya... < /var/spool/mail/user

I have a 141-line shell script to do this robustly, but it all really 
comes down to that one line.

-- 
==============================|   "A microscope locked in on one point
 Rob Funk <rf...@fu...> |Never sees what kind of room that it's in"
 http://www.funknet.net/rfunk |    -- Chris Mars, "Stuck in Rewind"

[fetchmail-users] Simply forward mail?

[Ken W. <ke...@ya...>]

Hi, I'm lost.  

I run sendmail on linux.  I have a mail drop (/var/spool/mail/user) with some mail in there. I
simply want to take this mail and forward it to another email address like say us...@ya....

How can I do this?  Seems to be the most difficult thing in the world.

ke...@ya...
Thanks.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [fetchmail-users] duplicating single remote account into multiple local account

[Matthias A. <mat...@gm...>]

On Wed, 31 May 2006, Filippo Santovito wrote:

> Alle 12:56, mercoledì 31 maggio 2006, Matthias Andree ha scritto:
> > Apparently yes. Note that procmail needs to be setuid-root for this to
> > work (do not run fetchmail as root).
> I'm running fecthmail/procmail with suse 10 standard config:
> 
> $ ls -l $(which procmail)
> -rwxr-xr-x  1 root root 77793 Jan 21 13:45 /usr/bin/procmail
> $ ls -l $(which fetchmail)
> -rwxr-xr-x  1 root root 225988 Sep  9  2005 /usr/bin/fetchmail
> 
> fetchmail is started in deamon mode by init.d script.

SUSE start fetchmail as root, against my advise :-(

So, it works ok, but isn't a recommended configuration security-wise.

-- 
Matthias Andree

Re: [fetchmail-users] duplicating single remote account into multiple local account

[Filippo S. <fil...@em...>]

Alle 12:56, mercoledì 31 maggio 2006, Matthias Andree ha scritto:
> Apparently yes. Note that procmail needs to be setuid-root for this to
> work (do not run fetchmail as root).
I'm running fecthmail/procmail with suse 10 standard config:

$ ls -l $(which procmail)
-rwxr-xr-x  1 root root 77793 Jan 21 13:45 /usr/bin/procmail
$ ls -l $(which fetchmail)
-rwxr-xr-x  1 root root 225988 Sep  9  2005 /usr/bin/fetchmail

fetchmail is started in deamon mode by init.d script.



-- 
Filippo Santovito

Le informazioni contenute nella comunicazione che precede possono essere
riservate e sono, comunque, destinate esclusivamente alla persona o
all'ente sopraindicati. La diffusione, distribuzione e/o copiatura del
documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario
è proibita. La sicurezza e la correttezza dei messaggi di posta elettronica
non possono essere garantite. Se avete ricevuto questo messaggio per
errore, Vi preghiamo di contattarci immediatamente. Grazie.

Confidentiality Notes: The information in this message is intended to be
confidential and for the use of only the individual or entity named above.
The information herein contained may be considered legally privileged
information. If you are not the intended recipient you are notified that
retention, dissemination, distribution other than to the intended
recipient, or copying of this message is strictly prohibited. If you
receive this message in error please notify us immediately by reply e-mail
and delete this message. Thank you.
 

Re: [fetchmail-users] duplicating single remote account into multiple local account

[Matthias A. <mat...@gm...>]

Filippo Santovito <fil...@em...> writes:

> I solved with 
> user 'remote@popserver' there with password 'xxxx'
> options fetchall ssl 
> mda '/usr/bin/procmail -d first_local second_local'
>
> It works but is this a correct solution?

Apparently yes. Note that procmail needs to be setuid-root for this to
work (do not run fetchmail as root).

-- 
Matthias Andree

Re: [fetchmail-users] duplicating single remote account into multiple local account

[Filippo S. <fil...@em...>]

Alle 02:12, mercoledì 31 maggio 2006, Matthias Andree ha scritto:
> poll popserver with proto POP3
> user 'remote@popserver' there with password 'xxxx'
> options fetchall ssl
> mda '/usr/sbin/sendmail -i -f %F -- account1 account2'

I solved with 
user 'remote@popserver' there with password 'xxxx'
options fetchall ssl 
mda '/usr/bin/procmail -d first_local second_local'

It works but is this a correct solution?

-- 

Filippo Santovito

Le informazioni contenute nella comunicazione che precede possono essere
riservate e sono, comunque, destinate esclusivamente alla persona o
all'ente sopraindicati. La diffusione, distribuzione e/o copiatura del
documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario
è proibita. La sicurezza e la correttezza dei messaggi di posta elettronica
non possono essere garantite. Se avete ricevuto questo messaggio per
errore, Vi preghiamo di contattarci immediatamente. Grazie.

Confidentiality Notes: The information in this message is intended to be
confidential and for the use of only the individual or entity named above.
The information herein contained may be considered legally privileged
information. If you are not the intended recipient you are notified that
retention, dissemination, distribution other than to the intended
recipient, or copying of this message is strictly prohibited. If you
receive this message in error please notify us immediately by reply e-mail
and delete this message. Thank you.
 

Re: [fetchmail-users] duplicating single remote account into multiple local account

[Filippo S. <fil...@em...>]

Alle 02:12, mercoledì 31 maggio 2006, Matthias Andree ha scritto:
> If the local machine has a sendmail-compatible sendmail command
> (Postfix, Exim also count here), try:
>
> poll popserver with proto POP3
> user 'remote@popserver' there with password 'xxxx'
> options fetchall ssl
> mda '/usr/sbin/sendmail -i -f %F -- account1 account2'

can this be done with procmail instead of using a full featured MTA?

-- 
Filippo Santovito

Le informazioni contenute nella comunicazione che precede possono essere
riservate e sono, comunque, destinate esclusivamente alla persona o
all'ente sopraindicati. La diffusione, distribuzione e/o copiatura del
documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario
è proibita. La sicurezza e la correttezza dei messaggi di posta elettronica
non possono essere garantite. Se avete ricevuto questo messaggio per
errore, Vi preghiamo di contattarci immediatamente. Grazie.

Confidentiality Notes: The information in this message is intended to be
confidential and for the use of only the individual or entity named above.
The information herein contained may be considered legally privileged
information. If you are not the intended recipient you are notified that
retention, dissemination, distribution other than to the intended
recipient, or copying of this message is strictly prohibited. If you
receive this message in error please notify us immediately by reply e-mail
and delete this message. Thank you.
 

Re: [fetchmail-users] duplicating single remote account into multiple local account

[Matthias A. <mat...@gm...>]

Filippo Santovito <fil...@em...> writes:

> Hi all,
> I have a remote pop account (remote@popserver) that I need to duplicate into 
> two local mail spool (account1@local, account2@local);
> I tried with:
>
> #############################################
> poll popserver with proto POP3
> user 'remote@popserver' there with password 'xxxx' is 'account1' here options 
> fetchall ssl
>
> poll popserver with proto POP3
> user 'remote@popserver' there with password 'xxxx' is 'account2' here options 
> fetchall ssl nokeep
> #############################################
>
> but only account1 spool is created and filled with emails.
> How can I duplicate the remote account?

If the local machine has a sendmail-compatible sendmail command
(Postfix, Exim also count here), try:

poll popserver with proto POP3
user 'remote@popserver' there with password 'xxxx' 
options fetchall ssl
mda '/usr/sbin/sendmail -i -f %F -- account1 account2'

Just be sure none of account1 and account2 forward back onto
remote@popserver. Note that mda has some other implications -- see the
man page.

-- 
Matthias Andree

[fetchmail-users] duplicating single remote account into multiple local account

[Filippo S. <fil...@em...>]

Hi all,
I have a remote pop account (remote@popserver) that I need to duplicate into 
two local mail spool (account1@local, account2@local);
I tried with:

#############################################
poll popserver with proto POP3
user 'remote@popserver' there with password 'xxxx' is 'account1' here options 
fetchall ssl

poll popserver with proto POP3
user 'remote@popserver' there with password 'xxxx' is 'account2' here options 
fetchall ssl nokeep
#############################################

but only account1 spool is created and filled with emails.
How can I duplicate the remote account?

-- 
Filippo Santovito

Le informazioni contenute nella comunicazione che precede possono essere
riservate e sono, comunque, destinate esclusivamente alla persona o
all'ente sopraindicati. La diffusione, distribuzione e/o copiatura del
documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario
è proibita. La sicurezza e la correttezza dei messaggi di posta elettronica
non possono essere garantite. Se avete ricevuto questo messaggio per
errore, Vi preghiamo di contattarci immediatamente. Grazie.

Confidentiality Notes: The information in this message is intended to be
confidential and for the use of only the individual or entity named above.
The information herein contained may be considered legally privileged
information. If you are not the intended recipient you are notified that
retention, dissemination, distribution other than to the intended
recipient, or copying of this message is strictly prohibited. If you
receive this message in error please notify us immediately by reply e-mail
and delete this message. Thank you.
 

[fetchmail-users] Re: Missed messages

[Michelle K. <lin...@fr...>]

Hi Matthias,

Am 2006-05-25 17:44:58, schrieb Matthias Andree:

> As long as procmail is installed, the first recommendation is to check
> *ALL* places that procmail can theoretically deliver to, including the
> $ORGMAIL and $DEFAULT places (whatever they are on the given Gentoo
> system).

Right, I had the same problem for some times and found over
200.000 Messages from different users in /var/mail/<user>.

(procmail has done this without any configurations on my side)

Greetings
    Michelle Konzack


-- 
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
Michelle Konzack   Apt. 917                  ICQ #328449886
                   50, rue de Soultz         MSM LinuxMichi
0033/6/61925193    67100 Strasbourg/France   IRC #Debian (irc.icq.com)

Re: [fetchmail-users] Missed messages

[Matthias A. <mat...@gm...>]

"Rob MacGregor" <rob...@gm...> writes:

> On 5/24/06, j.r...@gm... <j.r...@gm...> wrote:
>> I am running fetchmail as a normal user on my Gentoo Linux box. The
>> fetched messages are delivered to mailboxes by procmail. I am also using
>> postfix.
>
> Version numbers?  Contents of .fetchmailrc?

Rob, could you change the fetchmail-users info page to point users to
<http://www.fetchmail.info/fetchmail-FAQ.html#G3>?

>> I have found out that some expected messages were not delivered to their
>> mailboxes and it seems that I have lost them.
>
> Anything in the postfix log?

As long as procmail is installed, the first recommendation is to check
*ALL* places that procmail can theoretically deliver to, including the
$ORGMAIL and $DEFAULT places (whatever they are on the given Gentoo
system).

procmail is notorious for "losing" mail (actually filing it into the
wrong folder) in even slightly adverse conditions.

> Any vaguely recent version of fetchmail (and probably any version)
> will only tell the remote server to delete the message once the local
> server accepts it.  Of course, if you've bodged some local delivery
> script then all bets are off.  Until we see your .fetchmailrc (and
> really the logs) there's no way of knowing what's going on.

More importantly, /etc/procmailrc, $HOME/.procmailrc and procmail logs.

-- 
Matthias Andree

Re: [fetchmail-users] Missed messages

[Rob M. <rob...@gm...>]

On 5/24/06, j.r...@gm... <j.r...@gm...> wrote:
> I am running fetchmail as a normal user on my Gentoo Linux box. The
> fetched messages are delivered to mailboxes by procmail. I am also using
> postfix.

Version numbers?  Contents of .fetchmailrc?

> fetchmail was logging to a file on my home directory, but the messages
> are delivered to mailboxes on a different partition.
>
> This night it happened that the partition where the fetchmail and the
> procmail log files are saved become full, although there were available
> space in the partition with the mailboxes.
>
> I have found out that some expected messages were not delivered to their
> mailboxes and it seems that I have lost them.

Anything in the postfix log?

> Is this a normal behaviour? Is there anything I can do to get those
> messages again?

Any vaguely recent version of fetchmail (and probably any version)
will only tell the remote server to delete the message once the local
server accepts it.  Of course, if you've bodged some local delivery
script then all bets are off.  Until we see your .fetchmailrc (and
really the logs) there's no way of knowing what's going on.

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> I think, the current code which is assuming that mails have been
> expunged only on a successful quit is ok. It is a straighforward code
> which is not hard to understand. Setting UID_EXPUNGED before sending
> QUIT is probably going to make the code harder to debug, especially in
> the edge cases. Another case I can think of is:
>
> POP3> QUIT
> POP3< -ERR Disk full
>
> What is probably required is to directly delete the mails which have
> been deleted (UID_DELETED), but not yet expunged.

That's what we're trying to achieve. OK, we might hold off EXPUNGE and
look at the return code from the "QUIT" request. I don't have time right
now, but I'll look into this later, to figure out which errors can
happen and what to do with them.

>> Can you elaborate how to handle this separately?
>
> I mean, your patch is doing three things:
>
> - Adding comments to the code, fixing other minor issues
> - Adding the UID_PENDDELE flag
> - Saving and restoring the DELETED flag in .fetchids
>
> The above issues are equally important. But they are separate issues
> and should be addressed separately. I am saying this only to keep the
> size of the patch under discussion small.

Fine. I'll break them out later into a series of patches. I don't think
they can be independent of each other though, but they will have to be
applied in order.

>> BTW, we need the "DELETED" flagging in .fetchids to make your
>> "UID_UNSEEN" patch (re-fetch rather than mark seen) anyways. 
>> Else everything becomes "SEEN vs. UNSEEN" next time fetchmail is
>> (re-)started anyways.
>
> Handling it in .fetchids requires more thought. Just adding the
> DELETED word is not enough. Atleast, an expiration time will have to
> be added to the DELETED flag.

I don't think so, and if the end user's system has a goofed up system
time, it won't help either.

> What if a person runs fetchmail after a week or more? In this case, it
> is probably unsafe to assume that it is the same old mail which should
> be deleted directly.

So we're essentially heading to a flag "assume fetchmail is the only one
accessing said mailbox? yes/no". Not only in this context such a flag
would be useful. I'd rather not contemplate the havoc this flag wreaks
if set to "true" when that isn't true. That scares me away from putting
something like this on my agenda^WTODO.

> Another potential problem: what if the user is running the second
> instance of fetchmail with 'keep'? I think, your patch will still
> delete the mail directly (not checked).

I think it would.

-- 
Matthias Andree

[fetchmail-users] Missed messages

[<j.r...@gm...>]

I am running fetchmail as a normal user on my Gentoo Linux box. The
fetched messages are delivered to mailboxes by procmail. I am also using
postfix.

fetchmail was logging to a file on my home directory, but the messages
are delivered to mailboxes on a different partition.

This night it happened that the partition where the fetchmail and the
procmail log files are saved become full, although there were available
space in the partition with the mailboxes.

I have found out that some expected messages were not delivered to their
mailboxes and it seems that I have lost them.

Is this a normal behaviour? Is there anything I can do to get those
messages again?

Regards,

Romildo

[fetchmail-users] Re: fix lost POP3 deletes

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Wed, May 24, 2006:
> 
> Yes, it will. But how can we reliably tell "server received QUIT" from
> "server did not receive QUIT"? Do we get OS errors because the TCP
> ACK-flagged packets didn't come back, or because we got some ICMP or
> RST? Was the crucial ICMP perhaps dropped by the usual asinine firewalls
> that block all inbound ICMP? What kernel and version uses which codes to
> report issues? We might perhaps try "STAT" before resetting the flags so
> as to check if the connection was broken at the time we reach the logout
> code, but how else can we handle this?

I think, the current code which is assuming that mails have been
expunged only on a successful quit is ok. It is a straighforward code
which is not hard to understand. Setting UID_EXPUNGED before sending
QUIT is probably going to make the code harder to debug, especially in
the edge cases. Another case I can think of is:

POP3> QUIT
POP3< -ERR Disk full

What is probably required is to directly delete the mails which have
been deleted (UID_DELETED), but not yet expunged.

> >> In that case, fetchmail would then safely retry the deletion.
> >
> > Undoubtedly, saving the flag is required. However, to keep things
> > simpler, it would have been better if this issue was fixed separately.
> 
> Can you elaborate how to handle this separately?

I mean, your patch is doing three things:

- Adding comments to the code, fixing other minor issues
- Adding the UID_PENDDELE flag
- Saving and restoring the DELETED flag in .fetchids

The above issues are equally important. But they are separate issues
and should be addressed separately. I am saying this only to keep the
size of the patch under discussion small.

> BTW, we need the "DELETED" flagging in .fetchids to make your
> "UID_UNSEEN" patch (re-fetch rather than mark seen) anyways. 
> Else everything becomes "SEEN vs. UNSEEN" next time fetchmail is
> (re-)started anyways.

Handling it in .fetchids requires more thought. Just adding the
DELETED word is not enough. Atleast, an expiration time will have to
be added to the DELETED flag. What if a person runs fetchmail after a
week or more? In this case, it is probably unsafe to assume that it is
the same old mail which should be deleted directly.

Another potential problem: what if the user is running the second
instance of fetchmail with 'keep'? I think, your patch will still
delete the mail directly (not checked).

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> While debugging the code, I have found a minor problem which causes
> some of the flags not being set properly on the old uid list. Here it
> is:
>
> ================================================================================
> Index: fetchmail-6.3/pop3.c
> ===================================================================
> --- fetchmail-6.3/pop3.c	(revision 4851)
> +++ fetchmail-6.3/pop3.c	(working copy)
> @@ -1019,8 +1019,9 @@
>  			     * the same mail will not be downloaded again.
>  			     */
>  			    old = save_str(&ctl->oldsaved, id, UID_UNSEEN);
> -			    old->val.status.num = unum;
>  			}
> +			/* save the number */
> +			old->val.status.num = unum;
>  		    } else
>  			return PS_ERROR;
>  		} /* multi-line loop for UIDL reply */
> ================================================================================
>
> The above patch is not actually related to any of our patches!

Merged, thanks.

> Quoting from Matthias Andree's mail on Sun, May 21, 2006:
>> Well - fetchmail does not currently save the deleted status to its
>> .fetchids file, so there will be a difference between daemon mode and
>> "fetchmail without daemon option from cron" that I would like to avoid -
>> and my patch is supposed to address this.
>> 
>> I am attaching a revised patch (against UIDL) that drops the UIDs (marks
>> them UID_EXPUNGED) before sending the QUIT command.
>
> I think this assumes that the server necessarily receives the QUIT.
> What if the server does not receive the QUIT at all due to socket
> error? This will cause the mails to be downloaded again.

Yes, it will. But how can we reliably tell "server received QUIT" from
"server did not receive QUIT"? Do we get OS errors because the TCP
ACK-flagged packets didn't come back, or because we got some ICMP or
RST? Was the crucial ICMP perhaps dropped by the usual asinine firewalls
that block all inbound ICMP? What kernel and version uses which codes to
report issues? We might perhaps try "STAT" before resetting the flags so
as to check if the connection was broken at the time we reach the logout
code, but how else can we handle this?

>> In that case, fetchmail would then safely retry the deletion.
>
> Undoubtedly, saving the flag is required. However, to keep things
> simpler, it would have been better if this issue was fixed separately.

Can you elaborate how to handle this separately?

BTW, we need the "DELETED" flagging in .fetchids to make your
"UID_UNSEEN" patch (re-fetch rather than mark seen) anyways. 
Else everything becomes "SEEN vs. UNSEEN" next time fetchmail is
(re-)started anyways.

-- 
Matthias Andree

[fetchmail-users] [OFFTOPIC PATCH] Re: fix lost POP3 deletes

[Sunil S. <sh...@bo...>]

While debugging the code, I have found a minor problem which causes
some of the flags not being set properly on the old uid list. Here it
is:

================================================================================
Index: fetchmail-6.3/pop3.c
===================================================================
--- fetchmail-6.3/pop3.c	(revision 4851)
+++ fetchmail-6.3/pop3.c	(working copy)
@@ -1019,8 +1019,9 @@
 			     * the same mail will not be downloaded again.
 			     */
 			    old = save_str(&ctl->oldsaved, id, UID_UNSEEN);
-			    old->val.status.num = unum;
 			}
+			/* save the number */
+			old->val.status.num = unum;
 		    } else
 			return PS_ERROR;
 		} /* multi-line loop for UIDL reply */
================================================================================

The above patch is not actually related to any of our patches!

Quoting from Matthias Andree's mail on Sun, May 21, 2006:
> Well - fetchmail does not currently save the deleted status to its
> .fetchids file, so there will be a difference between daemon mode and
> "fetchmail without daemon option from cron" that I would like to avoid -
> and my patch is supposed to address this.
> 
> I am attaching a revised patch (against UIDL) that drops the UIDs (marks
> them UID_EXPUNGED) before sending the QUIT command.

I think this assumes that the server necessarily receives the QUIT.
What if the server does not receive the QUIT at all due to socket
error? This will cause the mails to be downloaded again.

> My patch saves the "DELETED" flag, and this should help when the
> connection goes away, for instance, when downloading a message, when
> there is not the slightest chance the server might have seen a "QUIT"
> request, so there is no chance of recycled UIDs.
> 
> In that case, fetchmail would then safely retry the deletion.

Undoubtedly, saving the flag is required. However, to keep things
simpler, it would have been better if this issue was fixed separately.

-- 
Sunil Shetye.

Re: [fetchmail-users] [PATCH 2] Re: only one message gets fetched using IMAP idle

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Lars Tewes's mail on Fri, May 19, 2006:
>> Thanks for the patch, it worked exactly as described!
>
> Here is an updated patch now. There is no change in imap.c. Only the
> test in driver.c has been modified to make fetchmail always enter IDLE
> mode. Also, the comments in driver.c have been updated.
>
> 	with "keep" and "idle" options
>
> Mails   New mails       Enter IDLE mode    Enter IDLE mode
>                         before patch?      after patch?
> ---------------------------------------------------------------
>  0       0               Yes                Yes
>  >= 1    0               No                 Yes
>  >= 1    >= 1            No                 Yes
>
> Please try this patch. As before, this patch has to be applied after
> the patch by Matthias.

Thanks, I've merged this patch. Looks good.

Only we should perhaps complain about --idle and --fetchall, as that
leads to excess mail retrieval.

-- 
Matthias Andree

Re: [fetchmail-users] [PATCH] Re: fix lost POP3 deletes

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

>> Opinions? Let it go in now or wait until 6.4.0?
>
> A far safer option is to just redownload the mail. I think, there are
> issues regarding POP servers recycling UIDs, thereby assigning the
> same UID of a mail which was deleted to a new mail.

> For such servers, there could also be a race condition on a socket
> error. For example, if fetchmail sends a "QUIT" which the POP3 server
> receives and if the POP3 server sends an "+OK" which fetchmail does
> not receive due to socket error. In this situation, the server has
> already expunged the mails but fetchmail does not know that. Now, if
> the server assigns the same UID to a new mail, fetchmail may end up
> deleting that mail.

Well - that problem hurts the existing UIDL code, too, independent of
--flush. When fetchmail gets an error response to QUIT, it will not
forget existing UIDLs, thus it may skip messages with recycled UIDs when
the server has in fact seen the QUIT message.

> This patch should now download the mail when used without 'flush'.
> This will lead to duplicate mails on socket errors.

Well - fetchmail does not currently save the deleted status to its
.fetchids file, so there will be a difference between daemon mode and
"fetchmail without daemon option from cron" that I would like to avoid -
and my patch is supposed to address this.

I am attaching a revised patch (against UIDL) that drops the UIDs (marks
them UID_EXPUNGED) before sending the QUIT command.

My patch saves the "DELETED" flag, and this should help when the
connection goes away, for instance, when downloading a message, when
there is not the slightest chance the server might have seen a "QUIT"
request, so there is no chance of recycled UIDs.

In that case, fetchmail would then safely retry the deletion.

The problem of another client deleting a message and the server
recycling the UID isn't fixed, we cannot fix this in fetchmail.

-- 
Matthias Andree

[fetchmail-users] Re: fix lost POP3 deletes

[Sunil S. <sh...@bo...>]

Quoting from Frederic Marchal's mail on Fri, May 19, 2006:
> Fetchmail should not (1) leave the mails on the server forever, (2) 
> download the mails again and again because it chokes on one mail, (3) 
> drop one mail because it could not distinguish it from a previously 
> deleted mail.

(1) The patch I had sent will do that. It will download the mail
again.

(2) To avoid downloading of the mail again, the 'flush' option should
be enabled along with 'no keep'.

(3) As Matthias has pointed out, many mailservers may have UIDs based
on some strictly monotonically increasing counters, so the actual
chance of duplication of UID should be rare. If your mailserver is
doing this, use the 'flush' option with the 'no keep' option.

If not, do not use the 'flush' option and the mail will get downloaded
again (after the patch). Then, no mails will get dropped.

-- 
Sunil Shetye.

[fetchmail-users] [PATCH 2] Re: only one message gets fetched using IMAP idle

[Sunil S. <sh...@bo...>]

Quoting from Lars Tewes's mail on Fri, May 19, 2006:
> Thanks for the patch, it worked exactly as described!

Here is an updated patch now. There is no change in imap.c. Only the
test in driver.c has been modified to make fetchmail always enter IDLE
mode. Also, the comments in driver.c have been updated.

	with "keep" and "idle" options

Mails   New mails       Enter IDLE mode    Enter IDLE mode
                        before patch?      after patch?
---------------------------------------------------------------
 0       0               Yes                Yes
 >= 1    0               No                 Yes
 >= 1    >= 1            No                 Yes

Please try this patch. As before, this patch has to be applied after
the patch by Matthias.

> So did I, but 1 Hunk failed for me:
> ---------------------------------------------------------------------
> patching file imap.c
> Hunk #8 FAILED at 723.
> 1 out of 9 hunks FAILED -- saving rejects to file imap.c.rej

Are you using a clean version of 6.3.4? I am getting offset messages,
but no failures.

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Matthias A. <ma...@dt...>]

On Fri, 19 May 2006, Frederic Marchal wrote:

> >The refinement of the suggestion would be to hash all but a few
> >non-constant headers with some decent hash function. MD5 would be
> >simple, but we shouldn't hardcode anything here.
> >  
> You are right. This is a better.
> 
> To reduce the load, the hash could be used only when both the UID and 
> the size of a previously deleted message are found again on the next 
> poll. That would not occur too often and, therefore, it should not 
> increase significantly the load on a relatively stable connection with a 
> server that uses good UID and reports valid constant mail size.
> 
> The user could also choose to use the hash instead of the size when the 
> size reported by the server is unreliable.

Just to clarify: a hash is not a short-term solution. I don't think
we'll see this in 6.3.X beyond what's already there.

> We have a pop3 proxy (p3scan) through which fetchmail downloads the 
> mails from the external pop3 server. p3scan get the whole mail and hand 
> it to clamd to scan it for a virus. If everything is ok, the mail is 
> passed to fetchmail. Therefore, fetchmail receives the mail after some 
> delay depending on the size of the mail but it can send it successfully 
> to postfix. Then, fetchmail tries to send the dele command and it fails 
> because the scanning of the mail by clamd took too much time and the 
> pop3 server timed out.

...

> If fetchmail can discover that the mail was delivered but it could not 
> be deleted on the previous poll (remember the dele command was never 
> acknowledged by the server) and delete it without downloading it again, 
> then the problem is solved. In my case, I received a 32MB mails and it 
> took 13 minutes to scan it.
> It is far beyond the patience of the most 
> patient server...

Thanks for providing that information. The actual problem is p3scan's
attempt to scan in real-time, which failed miserably. The proper
solution would be to change the setup:

1. fetchmail downloads the message

2. fetchmail injects either into an MTA which has a after-queueing scanner hooked up
   (for instance Postfix with amavisd-new)

3. the MTA queues the message with a "must inspect content".

4. MTA hands the message to the scanner, which can take as long as it
   desires, and then takes action depending on the result. Either
   defang, pass on, bounce, or erase.

5. MTA then hands the message to the local delivery agent (internal,
   maildrop, procmail, deliver, ...) which stuffs the mail into a
   mailbox

6. a local POP3 or IMAP server offers access to the content-scanned
   message to the client.

Step 3 and 4 are important - this makes for fast acceptance to prevent
timeouts and allows the scanner to take all the time it needs, without
respect to network timeouts.

-- 
Matthias Andree