fetchmail-users — User questions and discussion

[fetchmail-users] Re: Missed messages

[Michelle K. <lin...@fr...>]

Hi Matthias,

Am 2006-05-25 17:44:58, schrieb Matthias Andree:

> As long as procmail is installed, the first recommendation is to check
> *ALL* places that procmail can theoretically deliver to, including the
> $ORGMAIL and $DEFAULT places (whatever they are on the given Gentoo
> system).

Right, I had the same problem for some times and found over
200.000 Messages from different users in /var/mail/<user>.

(procmail has done this without any configurations on my side)

Greetings
    Michelle Konzack


-- 
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
Michelle Konzack   Apt. 917                  ICQ #328449886
                   50, rue de Soultz         MSM LinuxMichi
0033/6/61925193    67100 Strasbourg/France   IRC #Debian (irc.icq.com)

Re: [fetchmail-users] Missed messages

[Matthias A. <mat...@gm...>]

"Rob MacGregor" <rob...@gm...> writes:

> On 5/24/06, j.r...@gm... <j.r...@gm...> wrote:
>> I am running fetchmail as a normal user on my Gentoo Linux box. The
>> fetched messages are delivered to mailboxes by procmail. I am also using
>> postfix.
>
> Version numbers?  Contents of .fetchmailrc?

Rob, could you change the fetchmail-users info page to point users to
<http://www.fetchmail.info/fetchmail-FAQ.html#G3>?

>> I have found out that some expected messages were not delivered to their
>> mailboxes and it seems that I have lost them.
>
> Anything in the postfix log?

As long as procmail is installed, the first recommendation is to check
*ALL* places that procmail can theoretically deliver to, including the
$ORGMAIL and $DEFAULT places (whatever they are on the given Gentoo
system).

procmail is notorious for "losing" mail (actually filing it into the
wrong folder) in even slightly adverse conditions.

> Any vaguely recent version of fetchmail (and probably any version)
> will only tell the remote server to delete the message once the local
> server accepts it.  Of course, if you've bodged some local delivery
> script then all bets are off.  Until we see your .fetchmailrc (and
> really the logs) there's no way of knowing what's going on.

More importantly, /etc/procmailrc, $HOME/.procmailrc and procmail logs.

-- 
Matthias Andree

Re: [fetchmail-users] Missed messages

[Rob M. <rob...@gm...>]

On 5/24/06, j.r...@gm... <j.r...@gm...> wrote:
> I am running fetchmail as a normal user on my Gentoo Linux box. The
> fetched messages are delivered to mailboxes by procmail. I am also using
> postfix.

Version numbers?  Contents of .fetchmailrc?

> fetchmail was logging to a file on my home directory, but the messages
> are delivered to mailboxes on a different partition.
>
> This night it happened that the partition where the fetchmail and the
> procmail log files are saved become full, although there were available
> space in the partition with the mailboxes.
>
> I have found out that some expected messages were not delivered to their
> mailboxes and it seems that I have lost them.

Anything in the postfix log?

> Is this a normal behaviour? Is there anything I can do to get those
> messages again?

Any vaguely recent version of fetchmail (and probably any version)
will only tell the remote server to delete the message once the local
server accepts it.  Of course, if you've bodged some local delivery
script then all bets are off.  Until we see your .fetchmailrc (and
really the logs) there's no way of knowing what's going on.

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> I think, the current code which is assuming that mails have been
> expunged only on a successful quit is ok. It is a straighforward code
> which is not hard to understand. Setting UID_EXPUNGED before sending
> QUIT is probably going to make the code harder to debug, especially in
> the edge cases. Another case I can think of is:
>
> POP3> QUIT
> POP3< -ERR Disk full
>
> What is probably required is to directly delete the mails which have
> been deleted (UID_DELETED), but not yet expunged.

That's what we're trying to achieve. OK, we might hold off EXPUNGE and
look at the return code from the "QUIT" request. I don't have time right
now, but I'll look into this later, to figure out which errors can
happen and what to do with them.

>> Can you elaborate how to handle this separately?
>
> I mean, your patch is doing three things:
>
> - Adding comments to the code, fixing other minor issues
> - Adding the UID_PENDDELE flag
> - Saving and restoring the DELETED flag in .fetchids
>
> The above issues are equally important. But they are separate issues
> and should be addressed separately. I am saying this only to keep the
> size of the patch under discussion small.

Fine. I'll break them out later into a series of patches. I don't think
they can be independent of each other though, but they will have to be
applied in order.

>> BTW, we need the "DELETED" flagging in .fetchids to make your
>> "UID_UNSEEN" patch (re-fetch rather than mark seen) anyways. 
>> Else everything becomes "SEEN vs. UNSEEN" next time fetchmail is
>> (re-)started anyways.
>
> Handling it in .fetchids requires more thought. Just adding the
> DELETED word is not enough. Atleast, an expiration time will have to
> be added to the DELETED flag.

I don't think so, and if the end user's system has a goofed up system
time, it won't help either.

> What if a person runs fetchmail after a week or more? In this case, it
> is probably unsafe to assume that it is the same old mail which should
> be deleted directly.

So we're essentially heading to a flag "assume fetchmail is the only one
accessing said mailbox? yes/no". Not only in this context such a flag
would be useful. I'd rather not contemplate the havoc this flag wreaks
if set to "true" when that isn't true. That scares me away from putting
something like this on my agenda^WTODO.

> Another potential problem: what if the user is running the second
> instance of fetchmail with 'keep'? I think, your patch will still
> delete the mail directly (not checked).

I think it would.

-- 
Matthias Andree

[fetchmail-users] Missed messages

[<j.r...@gm...>]

I am running fetchmail as a normal user on my Gentoo Linux box. The
fetched messages are delivered to mailboxes by procmail. I am also using
postfix.

fetchmail was logging to a file on my home directory, but the messages
are delivered to mailboxes on a different partition.

This night it happened that the partition where the fetchmail and the
procmail log files are saved become full, although there were available
space in the partition with the mailboxes.

I have found out that some expected messages were not delivered to their
mailboxes and it seems that I have lost them.

Is this a normal behaviour? Is there anything I can do to get those
messages again?

Regards,

Romildo

[fetchmail-users] Re: fix lost POP3 deletes

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Wed, May 24, 2006:
> 
> Yes, it will. But how can we reliably tell "server received QUIT" from
> "server did not receive QUIT"? Do we get OS errors because the TCP
> ACK-flagged packets didn't come back, or because we got some ICMP or
> RST? Was the crucial ICMP perhaps dropped by the usual asinine firewalls
> that block all inbound ICMP? What kernel and version uses which codes to
> report issues? We might perhaps try "STAT" before resetting the flags so
> as to check if the connection was broken at the time we reach the logout
> code, but how else can we handle this?

I think, the current code which is assuming that mails have been
expunged only on a successful quit is ok. It is a straighforward code
which is not hard to understand. Setting UID_EXPUNGED before sending
QUIT is probably going to make the code harder to debug, especially in
the edge cases. Another case I can think of is:

POP3> QUIT
POP3< -ERR Disk full

What is probably required is to directly delete the mails which have
been deleted (UID_DELETED), but not yet expunged.

> >> In that case, fetchmail would then safely retry the deletion.
> >
> > Undoubtedly, saving the flag is required. However, to keep things
> > simpler, it would have been better if this issue was fixed separately.
> 
> Can you elaborate how to handle this separately?

I mean, your patch is doing three things:

- Adding comments to the code, fixing other minor issues
- Adding the UID_PENDDELE flag
- Saving and restoring the DELETED flag in .fetchids

The above issues are equally important. But they are separate issues
and should be addressed separately. I am saying this only to keep the
size of the patch under discussion small.

> BTW, we need the "DELETED" flagging in .fetchids to make your
> "UID_UNSEEN" patch (re-fetch rather than mark seen) anyways. 
> Else everything becomes "SEEN vs. UNSEEN" next time fetchmail is
> (re-)started anyways.

Handling it in .fetchids requires more thought. Just adding the
DELETED word is not enough. Atleast, an expiration time will have to
be added to the DELETED flag. What if a person runs fetchmail after a
week or more? In this case, it is probably unsafe to assume that it is
the same old mail which should be deleted directly.

Another potential problem: what if the user is running the second
instance of fetchmail with 'keep'? I think, your patch will still
delete the mail directly (not checked).

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> While debugging the code, I have found a minor problem which causes
> some of the flags not being set properly on the old uid list. Here it
> is:
>
> ================================================================================
> Index: fetchmail-6.3/pop3.c
> ===================================================================
> --- fetchmail-6.3/pop3.c	(revision 4851)
> +++ fetchmail-6.3/pop3.c	(working copy)
> @@ -1019,8 +1019,9 @@
>  			     * the same mail will not be downloaded again.
>  			     */
>  			    old = save_str(&ctl->oldsaved, id, UID_UNSEEN);
> -			    old->val.status.num = unum;
>  			}
> +			/* save the number */
> +			old->val.status.num = unum;
>  		    } else
>  			return PS_ERROR;
>  		} /* multi-line loop for UIDL reply */
> ================================================================================
>
> The above patch is not actually related to any of our patches!

Merged, thanks.

> Quoting from Matthias Andree's mail on Sun, May 21, 2006:
>> Well - fetchmail does not currently save the deleted status to its
>> .fetchids file, so there will be a difference between daemon mode and
>> "fetchmail without daemon option from cron" that I would like to avoid -
>> and my patch is supposed to address this.
>> 
>> I am attaching a revised patch (against UIDL) that drops the UIDs (marks
>> them UID_EXPUNGED) before sending the QUIT command.
>
> I think this assumes that the server necessarily receives the QUIT.
> What if the server does not receive the QUIT at all due to socket
> error? This will cause the mails to be downloaded again.

Yes, it will. But how can we reliably tell "server received QUIT" from
"server did not receive QUIT"? Do we get OS errors because the TCP
ACK-flagged packets didn't come back, or because we got some ICMP or
RST? Was the crucial ICMP perhaps dropped by the usual asinine firewalls
that block all inbound ICMP? What kernel and version uses which codes to
report issues? We might perhaps try "STAT" before resetting the flags so
as to check if the connection was broken at the time we reach the logout
code, but how else can we handle this?

>> In that case, fetchmail would then safely retry the deletion.
>
> Undoubtedly, saving the flag is required. However, to keep things
> simpler, it would have been better if this issue was fixed separately.

Can you elaborate how to handle this separately?

BTW, we need the "DELETED" flagging in .fetchids to make your
"UID_UNSEEN" patch (re-fetch rather than mark seen) anyways. 
Else everything becomes "SEEN vs. UNSEEN" next time fetchmail is
(re-)started anyways.

-- 
Matthias Andree

[fetchmail-users] [OFFTOPIC PATCH] Re: fix lost POP3 deletes

[Sunil S. <sh...@bo...>]

While debugging the code, I have found a minor problem which causes
some of the flags not being set properly on the old uid list. Here it
is:

================================================================================
Index: fetchmail-6.3/pop3.c
===================================================================
--- fetchmail-6.3/pop3.c	(revision 4851)
+++ fetchmail-6.3/pop3.c	(working copy)
@@ -1019,8 +1019,9 @@
 			     * the same mail will not be downloaded again.
 			     */
 			    old = save_str(&ctl->oldsaved, id, UID_UNSEEN);
-			    old->val.status.num = unum;
 			}
+			/* save the number */
+			old->val.status.num = unum;
 		    } else
 			return PS_ERROR;
 		} /* multi-line loop for UIDL reply */
================================================================================

The above patch is not actually related to any of our patches!

Quoting from Matthias Andree's mail on Sun, May 21, 2006:
> Well - fetchmail does not currently save the deleted status to its
> .fetchids file, so there will be a difference between daemon mode and
> "fetchmail without daemon option from cron" that I would like to avoid -
> and my patch is supposed to address this.
> 
> I am attaching a revised patch (against UIDL) that drops the UIDs (marks
> them UID_EXPUNGED) before sending the QUIT command.

I think this assumes that the server necessarily receives the QUIT.
What if the server does not receive the QUIT at all due to socket
error? This will cause the mails to be downloaded again.

> My patch saves the "DELETED" flag, and this should help when the
> connection goes away, for instance, when downloading a message, when
> there is not the slightest chance the server might have seen a "QUIT"
> request, so there is no chance of recycled UIDs.
> 
> In that case, fetchmail would then safely retry the deletion.

Undoubtedly, saving the flag is required. However, to keep things
simpler, it would have been better if this issue was fixed separately.

-- 
Sunil Shetye.

Re: [fetchmail-users] [PATCH 2] Re: only one message gets fetched using IMAP idle

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

> Quoting from Lars Tewes's mail on Fri, May 19, 2006:
>> Thanks for the patch, it worked exactly as described!
>
> Here is an updated patch now. There is no change in imap.c. Only the
> test in driver.c has been modified to make fetchmail always enter IDLE
> mode. Also, the comments in driver.c have been updated.
>
> 	with "keep" and "idle" options
>
> Mails   New mails       Enter IDLE mode    Enter IDLE mode
>                         before patch?      after patch?
> ---------------------------------------------------------------
>  0       0               Yes                Yes
>  >= 1    0               No                 Yes
>  >= 1    >= 1            No                 Yes
>
> Please try this patch. As before, this patch has to be applied after
> the patch by Matthias.

Thanks, I've merged this patch. Looks good.

Only we should perhaps complain about --idle and --fetchall, as that
leads to excess mail retrieval.

-- 
Matthias Andree

Re: [fetchmail-users] [PATCH] Re: fix lost POP3 deletes

[Matthias A. <mat...@gm...>]

Sunil Shetye <sh...@bo...> writes:

>> Opinions? Let it go in now or wait until 6.4.0?
>
> A far safer option is to just redownload the mail. I think, there are
> issues regarding POP servers recycling UIDs, thereby assigning the
> same UID of a mail which was deleted to a new mail.

> For such servers, there could also be a race condition on a socket
> error. For example, if fetchmail sends a "QUIT" which the POP3 server
> receives and if the POP3 server sends an "+OK" which fetchmail does
> not receive due to socket error. In this situation, the server has
> already expunged the mails but fetchmail does not know that. Now, if
> the server assigns the same UID to a new mail, fetchmail may end up
> deleting that mail.

Well - that problem hurts the existing UIDL code, too, independent of
--flush. When fetchmail gets an error response to QUIT, it will not
forget existing UIDLs, thus it may skip messages with recycled UIDs when
the server has in fact seen the QUIT message.

> This patch should now download the mail when used without 'flush'.
> This will lead to duplicate mails on socket errors.

Well - fetchmail does not currently save the deleted status to its
.fetchids file, so there will be a difference between daemon mode and
"fetchmail without daemon option from cron" that I would like to avoid -
and my patch is supposed to address this.

I am attaching a revised patch (against UIDL) that drops the UIDs (marks
them UID_EXPUNGED) before sending the QUIT command.

My patch saves the "DELETED" flag, and this should help when the
connection goes away, for instance, when downloading a message, when
there is not the slightest chance the server might have seen a "QUIT"
request, so there is no chance of recycled UIDs.

In that case, fetchmail would then safely retry the deletion.

The problem of another client deleting a message and the server
recycling the UID isn't fixed, we cannot fix this in fetchmail.

-- 
Matthias Andree

[fetchmail-users] Re: fix lost POP3 deletes

[Sunil S. <sh...@bo...>]

Quoting from Frederic Marchal's mail on Fri, May 19, 2006:
> Fetchmail should not (1) leave the mails on the server forever, (2) 
> download the mails again and again because it chokes on one mail, (3) 
> drop one mail because it could not distinguish it from a previously 
> deleted mail.

(1) The patch I had sent will do that. It will download the mail
again.

(2) To avoid downloading of the mail again, the 'flush' option should
be enabled along with 'no keep'.

(3) As Matthias has pointed out, many mailservers may have UIDs based
on some strictly monotonically increasing counters, so the actual
chance of duplication of UID should be rare. If your mailserver is
doing this, use the 'flush' option with the 'no keep' option.

If not, do not use the 'flush' option and the mail will get downloaded
again (after the patch). Then, no mails will get dropped.

-- 
Sunil Shetye.

[fetchmail-users] [PATCH 2] Re: only one message gets fetched using IMAP idle

[Sunil S. <sh...@bo...>]

Quoting from Lars Tewes's mail on Fri, May 19, 2006:
> Thanks for the patch, it worked exactly as described!

Here is an updated patch now. There is no change in imap.c. Only the
test in driver.c has been modified to make fetchmail always enter IDLE
mode. Also, the comments in driver.c have been updated.

	with "keep" and "idle" options

Mails   New mails       Enter IDLE mode    Enter IDLE mode
                        before patch?      after patch?
---------------------------------------------------------------
 0       0               Yes                Yes
 >= 1    0               No                 Yes
 >= 1    >= 1            No                 Yes

Please try this patch. As before, this patch has to be applied after
the patch by Matthias.

> So did I, but 1 Hunk failed for me:
> ---------------------------------------------------------------------
> patching file imap.c
> Hunk #8 FAILED at 723.
> 1 out of 9 hunks FAILED -- saving rejects to file imap.c.rej

Are you using a clean version of 6.3.4? I am getting offset messages,
but no failures.

-- 
Sunil Shetye.

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Matthias A. <ma...@dt...>]

On Fri, 19 May 2006, Frederic Marchal wrote:

> >The refinement of the suggestion would be to hash all but a few
> >non-constant headers with some decent hash function. MD5 would be
> >simple, but we shouldn't hardcode anything here.
> >  
> You are right. This is a better.
> 
> To reduce the load, the hash could be used only when both the UID and 
> the size of a previously deleted message are found again on the next 
> poll. That would not occur too often and, therefore, it should not 
> increase significantly the load on a relatively stable connection with a 
> server that uses good UID and reports valid constant mail size.
> 
> The user could also choose to use the hash instead of the size when the 
> size reported by the server is unreliable.

Just to clarify: a hash is not a short-term solution. I don't think
we'll see this in 6.3.X beyond what's already there.

> We have a pop3 proxy (p3scan) through which fetchmail downloads the 
> mails from the external pop3 server. p3scan get the whole mail and hand 
> it to clamd to scan it for a virus. If everything is ok, the mail is 
> passed to fetchmail. Therefore, fetchmail receives the mail after some 
> delay depending on the size of the mail but it can send it successfully 
> to postfix. Then, fetchmail tries to send the dele command and it fails 
> because the scanning of the mail by clamd took too much time and the 
> pop3 server timed out.

...

> If fetchmail can discover that the mail was delivered but it could not 
> be deleted on the previous poll (remember the dele command was never 
> acknowledged by the server) and delete it without downloading it again, 
> then the problem is solved. In my case, I received a 32MB mails and it 
> took 13 minutes to scan it.
> It is far beyond the patience of the most 
> patient server...

Thanks for providing that information. The actual problem is p3scan's
attempt to scan in real-time, which failed miserably. The proper
solution would be to change the setup:

1. fetchmail downloads the message

2. fetchmail injects either into an MTA which has a after-queueing scanner hooked up
   (for instance Postfix with amavisd-new)

3. the MTA queues the message with a "must inspect content".

4. MTA hands the message to the scanner, which can take as long as it
   desires, and then takes action depending on the result. Either
   defang, pass on, bounce, or erase.

5. MTA then hands the message to the local delivery agent (internal,
   maildrop, procmail, deliver, ...) which stuffs the mail into a
   mailbox

6. a local POP3 or IMAP server offers access to the content-scanned
   message to the client.

Step 3 and 4 are important - this makes for fast acceptance to prevent
timeouts and allows the scanner to take all the time it needs, without
respect to network timeouts.

-- 
Matthias Andree

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Frederic M. <fre...@wo...>]

Matthias Andree wrote:
> (collecting replies to two postings)
>   
>> - Some mailservers keep the flags of a mail in the mail itself by
>>   adding a header like Status:. So, the size of a mail may actually
>>   change when it turns from 'new' to 'old'. Due to size mismatch,
>>   mails from such mailservers will get downloaded again.
>>     
>
> That is a rather important concern.
>
> The refinement of the suggestion would be to hash all but a few
> non-constant headers with some decent hash function. MD5 would be
> simple, but we shouldn't hardcode anything here.
>   
You are right. This is a better.

To reduce the load, the hash could be used only when both the UID and 
the size of a previously deleted message are found again on the next 
poll. That would not occur too often and, therefore, it should not 
increase significantly the load on a relatively stable connection with a 
server that uses good UID and reports valid constant mail size.

The user could also choose to use the hash instead of the size when the 
size reported by the server is unreliable.

>>> - Some mailservers keep the flags of a mail in the mail itself by
>>>   adding a header like Status:. So, the size of a mail may actually
>>>   change when it turns from 'new' to 'old'. Due to size mismatch,
>>>   mails from such mailservers will get downloaded again.
>>>       
>
>   
>> You got me there :-) In that case, the mail would be downloaded a second
>> time and deleted.
>>     
>
> The "message of same size" (say, automated messages in a fixed format,
> stock reports, particular logs) problem isn't solved though - and they
> might even get the same UID if the server used just the MD5 or a file
> name with a recycled inode number...
>   
I don't think so. If two messages are really different, the date and the 
message ID should at least be different. Even for a badly formated mail 
with a constant date and message ID sent, for instance, by a crontab 
running sendmail us...@ex... < fixed_mail.txt, the received headers 
should be different. If the two mails are completely identical and lead 
to the same hash, then they are the same mail.

The hash of the significant headers looks safe to me if the "significant 
headers" are carefully chosen.

> The question is (1) if it's worth the added effort to track sizes, or
> (2) if we should rather go Sunil's simple "play it safe and redownload
> if in doubt" route; or (3) refine my patch to assume QUIT succeeded the
> moment it is handed off to the write() call. That also has the effect of
> redownloading messages if the QUIT fails, but will retry the DELEte if
> fetchmail doesn't reach the point where it would send QUIT.
>   
Fetchmail should not (1) leave the mails on the server forever, (2) 
download the mails again and again because it chokes on one mail, (3) 
drop one mail because it could not distinguish it from a previously 
deleted mail.

Now, to make things clearer, let me tell you a problem that occurred to 
me. It explain the reason I add point (2) above and is a concrete 
example of something that can go wrong.

We have a pop3 proxy (p3scan) through which fetchmail downloads the 
mails from the external pop3 server. p3scan get the whole mail and hand 
it to clamd to scan it for a virus. If everything is ok, the mail is 
passed to fetchmail. Therefore, fetchmail receives the mail after some 
delay depending on the size of the mail but it can send it successfully 
to postfix. Then, fetchmail tries to send the dele command and it fails 
because the scanning of the mail by clamd took too much time and the 
pop3 server timed out.

If fetchmail can discover that the mail was delivered but it could not 
be deleted on the previous poll (remember the dele command was never 
acknowledged by the server) and delete it without downloading it again, 
then the problem is solved. In my case, I received a 32MB mails and it 
took 13 minutes to scan it. It is far beyond the patience of the most 
patient server... Therefore, the mail kept being downloaded over and 
over and delivered to the two users on the To field... Yes it is rather 
painful when you open you mailbox and discover 15 copies of a 32MB mail :-)

>> And this, only if the connection is dropped after the delete command
>> is sent AND the size of the mail is changed by the server on the next
>> poll.
>>
>> The worst case occurs if the size of the mail keeps changing at every
>> poll. I have no solution here. It's not funny at all to have to deal
>> with broken servers :-(
>>     
>
> I think I might just charge users for adding more workarounds. Then they
> have the cheap route of complaining to the server's operator, or the
> expensive one of having the fix. :-P
>
> I think size isn't necessarily the best complement here to detect
> changes. I like the hash approach better, but this seems quite intrusive
> as well.
>
> A hash that precludes some dynamic headers such as [X-]Status and
> similar may be needed anyways to emulate UIDL for --keep setups on
> servers that don't support UIDL -- but if they lack all UIDL and TOP,
> the network impact of --keep will be rather painful.
> (Perhaps fetchmail should just refuse --keep on such servers.)
>   
Or it might be documented. The user would then have to choose between 
the "cheap route" or the time consuming one :-)

Frederic

[fetchmail-users] Re: [PATCH] Re: only one message gets fetched using IMAP idle

[Lars T. <ls...@gm...>]

Hello Sunil,

sorry for the late answer...

Sunil Shetye wrote:

 > 	with "keep" and "idle" options
 > Mails   New mails       Enter IDLE mode    Enter IDLE mode
 >                         before patch?      after patch?
 > ---------------------------------------------------------------
 >  0       0               Yes                Yes
 >  >= 1    0               No                 No
 >  >= 1    >= 1            No                 Yes

Thanks for the patch, it worked exactly as described!

 > Please try this patch and report. Note that this patch has to be
 > applied after the patch by Matthias Andree.

So did I, but 1 Hunk failed for me:
---------------------------------------------------------------------
patching file driver.c
Hunk #1 succeeded at 1458 (offset -3 lines).
patching file imap.c
Hunk #5 succeeded at 687 (offset 2 lines).
Hunk #6 succeeded at 699 (offset 2 lines).
Hunk #7 succeeded at 714 (offset 2 lines).
Hunk #8 FAILED at 723.
Hunk #9 succeeded at 779 (offset 2 lines).
1 out of 9 hunks FAILED -- saving rejects to file imap.c.rej


imap.c.rej:
***************
*** 719,725 ****
       }
       else
       {
-       count = 0;
         ok = gen_transact(sock,
                           check_only ? "EXAMINE \"%s\"" : "SELECT \"%s\"",
                           folder ? folder : "INBOX");
--- 723,729 ----
       }
       else
       {
+       oldcount = count = 0;
         ok = gen_transact(sock,
                           check_only ? "EXAMINE \"%s\"" : "SELECT \"%s\"",
                           folder ? folder : "INBOX");
---------------------------------------------------------------------

I added it manually - this shouldn't be a problem though.

Nice one - thanks again to both of you!

Best regards,
Lars

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Matthias A. <mat...@gm...>]

(collecting replies to two postings)

Sunil Shetye <sh...@bo...> writes in response to
Frederic Marchal's text (not cited):


[Frederic suggested to save the (UID,size) pair to detect message
recycling.]
> There are a few objections to this:
>
> - fetchmail currently does not store the size in the fetchids file, so
>   doing this will require a modification of the fetchids file format.

fetchmail's uid.c code (just revised a bit so as to be more readable)
does the equivalent of (Perl RE syntax):

      __user__|domain_        ___id____
[ \t]*([^@]\*)@([^ ]+)[ \t]* +([^ \t]*).*

A typical line that can be parsed looks like this:

some user@re...@po... d3b07384.d113edec

And the parser since 6.3.0 is robust, so that we can add a blank or TAB
and anything after a blank after the ID:

some user@re...@po... d3b07384.d113edec foo=bar baz=23|joe=mary

Saving any information such as deleted flags, size, retrieval date is
thus backwards compatible in the sense that older fetchmail versions
would just ignore the extra information and strip it when rewriting the
ID. Exception: scratch lists are saved verbatim, i. e. for unqueried
hosts, the extra information would be kept.

> - fetchmail does not get the size of all mails right now (in the
>   default setup), only of the mails it is going to download. Your
>   suggestion would imply that fetchmail will have to get UIDs as well
>   as sizes of all mails for comparison. This would imply a lot of
>   delay in getting mails, especially when 'keep' is on.

1. POP3 "LIST" is fast, has affordable traffic requirements and part of
   regular fetchmail operation.

2. The size is, as suggested, only needed for messages that are supposed
   to be deleted after retrieval. Discounting --flush, fetchmail marks
   the message for deletion after retrieving it, and at that time, the
   POP3 server's notion of the message size is known. If the size is
   cached in the .fetchids file, that part is covered. For new messages,
   the size will have to be retrieved anyways sooner or later to support
   ESMTP SIZE.

> - Some mailservers keep the flags of a mail in the mail itself by
>   adding a header like Status:. So, the size of a mail may actually
>   change when it turns from 'new' to 'old'. Due to size mismatch,
>   mails from such mailservers will get downloaded again.

That is a rather important concern.

The refinement of the suggestion would be to hash all but a few
non-constant headers with some decent hash function. MD5 would be
simple, but we shouldn't hardcode anything here.

> - Some mailservers misreport sizes! I remember one such server being
>   reported:
>
> <http://lists.ccil.org/pipermail/fetchmail-friends/2002-January/005572.html>

Can we assume that the size is stable even if inaccurate?



Frederic Marchal <fre...@wo...> replied:

> For which a change was proposed if I remember correctly... It was even
> suggested to use a database such as sqlite. I admit I haven't followed
> the status of that proposition. Beside, there is no hurry. It doesn't
> have to be a patch within two days. It may be part of a major version
> change just to make fetchmail even better :-)

sqlite won't happen for fetchmail 6.3.X.

Extending .fetchids as suggested above might.

> processing would remain the same. The fetchids file may even be purged
> of the old UID when the next poll confirms the mails are really gone.

That happens automatically the next time that POP3 chat with the server
completes successfully.

>> - Some mailservers keep the flags of a mail in the mail itself by
>>   adding a header like Status:. So, the size of a mail may actually
>>   change when it turns from 'new' to 'old'. Due to size mismatch,
>>   mails from such mailservers will get downloaded again.

> You got me there :-) In that case, the mail would be downloaded a second
> time and deleted.

The "message of same size" (say, automated messages in a fixed format,
stock reports, particular logs) problem isn't solved though - and they
might even get the same UID if the server used just the MD5 or a file
name with a recycled inode number...

The question is (1) if it's worth the added effort to track sizes, or
(2) if we should rather go Sunil's simple "play it safe and redownload
if in doubt" route; or (3) refine my patch to assume QUIT succeeded the
moment it is handed off to the write() call. That also has the effect of
redownloading messages if the QUIT fails, but will retry the DELEte if
fetchmail doesn't reach the point where it would send QUIT.

> If the deletion fails again, the mail may be downloaded once more if
> its size keeps changing or it may eventually be deleted without any
> further download if the size is not changed any more. If the download
> is the action that causes the connection drop (such as a proxy with a
> virus scanner taking too much time and exceeding the timeout of the
> server), it will be downloaded twice and deleted the third time.

A fetchmail-internal queue might fix this much later on if someone could
document the need for such.

> And this, only if the connection is dropped after the delete command
> is sent AND the size of the mail is changed by the server on the next
> poll.
>
> The worst case occurs if the size of the mail keeps changing at every
> poll. I have no solution here. It's not funny at all to have to deal
> with broken servers :-(

I think I might just charge users for adding more workarounds. Then they
have the cheap route of complaining to the server's operator, or the
expensive one of having the fix. :-P

I think size isn't necessarily the best complement here to detect
changes. I like the hash approach better, but this seems quite intrusive
as well.

A hash that precludes some dynamic headers such as [X-]Status and
similar may be needed anyways to emulate UIDL for --keep setups on
servers that don't support UIDL -- but if they lack all UIDL and TOP,
the network impact of --keep will be rather painful.
(Perhaps fetchmail should just refuse --keep on such servers.)

> Now, you may argue that it is a lot of programming for the sole benefit
> of not deleting unread mails when the connection drop... I have nothing
> to say against that. I'm not likely to be the one to program it although
> I really wish I could.
>
> Note that this is only a solution I propose to solve the problem of the
> mails being either duplicated, deleted without delivery or left forever
> on the server when a connection drop. I'm happy with my current
> configuration which work fine with pop3 + uidl and a server that doesn't
> recycle the UID too quickly and a connection quite reliable enough.

If your server uses a timestamp into the UID and has a strictly
monotonically increasing system lock, the risk is pretty low.

-- 
Matthias Andree

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Frederic M. <fre...@wo...>]

Sunil Shetye wrote:
> Quoting from Frederic Marchal's mail on Thu, May 18, 2006:
>   
>> Using pop3, the size of the mail is one additional information that can 
>> be used to ensure it is not the same message with the same UID. The list 
>> command returns it and fetchmail does use it during the communication. I 
>> expect it would be much less likely that a UID is recycled on a message 
>> with the exact same size.
>>
>> If a message is using the same UID as a message we know should have been 
>> deleted, then list its size and compare it. If it is the same, it can be 
>> deleted again. If not, it is a new message and should be downloaded.
>>
>> Does it make sense ?
>>     
>
> There are a few objections to this:
>
> - fetchmail currently does not store the size in the fetchids file, so
>   doing this will require a modification of the fetchids file format.
>   
For which a change was proposed if I remember correctly... It was even 
suggested to use a database such as sqlite. I admit I haven't followed 
the status of that proposition. Beside, there is no hurry. It doesn't 
have to be a patch within two days. It may be part of a major version 
change just to make fetchmail even better :-)
> - fetchmail does not get the size of all mails right now (in the
>   default setup), only of the mails it is going to download. Your
>   suggestion would imply that fetchmail will have to get UIDs as well
>   as sizes of all mails for comparison. This would imply a lot of
>   delay in getting mails, especially when 'keep' is on.
>   
Only for the mails with a known UID for which fetchmail sent a delete 
command in a previous poll (that should not occur too often because most 
of the time the mails will effectively be deleted on the server when 
fetchmail closes the connection). It would only serve to find out if the 
mail is different and should be deleted or downloaded. For an unknown 
UID or the UID of a mail that was left on the server on purpose, the 
processing would remain the same. The fetchids file may even be purged 
of the old UID when the next poll confirms the mails are really gone. A 
recycled UID coming afterwards would then be seen as a new mail without 
any further comparison.

> - Some mailservers keep the flags of a mail in the mail itself by
>   adding a header like Status:. So, the size of a mail may actually
>   change when it turns from 'new' to 'old'. Due to size mismatch,
>   mails from such mailservers will get downloaded again.
>   
You got me there :-) In that case, the mail would be downloaded a second 
time and deleted. If the deletion fails again, the mail may be 
downloaded once more if its size keeps changing or it may eventually be 
deleted without any further download if the size is not changed any 
more. If the download is the action that causes the connection drop 
(such as a proxy with a virus scanner taking too much time and exceeding 
the timeout of the server), it will be downloaded twice and deleted the 
third time. And this, only if the connection is dropped after the delete 
command is sent AND the size of the mail is changed by the server on the 
next poll.

The worst case occurs if the size of the mail keeps changing at every 
poll. I have no solution here. It's not funny at all to have to deal 
with broken servers :-(

> - Some mailservers misreport sizes! I remember one such server being
>   reported:
>
> <http://lists.ccil.org/pipermail/fetchmail-friends/2002-January/005572.html>
>
>   
It depends on the problem with the size. If the reported size is always 
the same, there is no problem. Now, if the reported size is a random 
function, there is no way it can work. In that case, when the mails fail 
to be deleted, they are downloaded again over and over until the delete 
commands succeed. On the other hand, for all those who have a good 
server reporting the correct sizes, there will be an improvement and the 
mails will not be duplicated when the connection drop.

Now, you may argue that it is a lot of programming for the sole benefit 
of not deleting unread mails when the connection drop... I have nothing 
to say against that. I'm not likely to be the one to program it although 
I really wish I could.

Note that this is only a solution I propose to solve the problem of the 
mails being either duplicated, deleted without delivery or left forever 
on the server when a connection drop. I'm happy with my current 
configuration which work fine with pop3 + uidl and a server that doesn't 
recycle the UID too quickly and a connection quite reliable enough.

Frederic

[fetchmail-users] Re: fix lost POP3 deletes

[Sunil S. <sh...@bo...>]

Quoting from Frederic Marchal's mail on Thu, May 18, 2006:
> Using pop3, the size of the mail is one additional information that can 
> be used to ensure it is not the same message with the same UID. The list 
> command returns it and fetchmail does use it during the communication. I 
> expect it would be much less likely that a UID is recycled on a message 
> with the exact same size.
> 
> If a message is using the same UID as a message we know should have been 
> deleted, then list its size and compare it. If it is the same, it can be 
> deleted again. If not, it is a new message and should be downloaded.
> 
> Does it make sense ?

There are a few objections to this:

- fetchmail currently does not store the size in the fetchids file, so
  doing this will require a modification of the fetchids file format.

- fetchmail does not get the size of all mails right now (in the
  default setup), only of the mails it is going to download. Your
  suggestion would imply that fetchmail will have to get UIDs as well
  as sizes of all mails for comparison. This would imply a lot of
  delay in getting mails, especially when 'keep' is on.

- Some mailservers keep the flags of a mail in the mail itself by
  adding a header like Status:. So, the size of a mail may actually
  change when it turns from 'new' to 'old'. Due to size mismatch,
  mails from such mailservers will get downloaded again.

- Some mailservers misreport sizes! I remember one such server being
  reported:

<http://lists.ccil.org/pipermail/fetchmail-friends/2002-January/005572.html>

-- 
Sunil Shetye.

[fetchmail-users] Re: [PATCH] Re: fix lost POP3 deletes

[Frederic M. <fre...@wo...>]

Sunil Shetye wrote:
> Quoting from Matthias Andree's mail on Wed, May 17, 2006:
>   
>> Frederic Marchal found a problem with POP3 deletes when --flush and
>> --keep are both unset and UIDL is used for the connection:
>>
>> If fetchmail sends a DELE command, but the POP3 connection aborts before
>> the QUIT command can be sent, then fetchmail will skip the messages on
>> the server and leaving them there forever.
>>
>> The patch below is somewhat intrusive but attempts to fix this. It
>> changes the .fetchids file format, adding " DELETED" to UID lines of
>> deleted messages when the QUIT hasn't been acknowledged. This should
>> make fetchmail retry the delete on the next run.
>>
>> Please test.
>>
>> Opinions? Let it go in now or wait until 6.4.0?
>>     
>
> A far safer option is to just redownload the mail. I think, there are
> issues regarding POP servers recycling UIDs, thereby assigning the
> same UID of a mail which was deleted to a new mail.
>
> For such servers, there could also be a race condition on a socket
> error. For example, if fetchmail sends a "QUIT" which the POP3 server
> receives and if the POP3 server sends an "+OK" which fetchmail does
> not receive due to socket error. In this situation, the server has
> already expunged the mails but fetchmail does not know that. Now, if
> the server assigns the same UID to a new mail, fetchmail may end up
> deleting that mail.
>
> This patch should now download the mail when used without 'flush'.
> This will lead to duplicate mails on socket errors.
>
> Frederic, your original query was whether 'flush' is safe or not. It
> is safe to use 'flush' with 'uidl', unless your mailserver is
> reassigning the same UIDs of mails which have been deleted recently to
> new mails received after the deletion.
>   
Using pop3, the size of the mail is one additional information that can 
be used to ensure it is not the same message with the same UID. The list 
command returns it and fetchmail does use it during the communication. I 
expect it would be much less likely that a UID is recycled on a message 
with the exact same size.

If a message is using the same UID as a message we know should have been 
deleted, then list its size and compare it. If it is the same, it can be 
deleted again. If not, it is a new message and should be downloaded.

Does it make sense ?

Frederic

[fetchmail-users] [PATCH] Re: fix lost POP3 deletes

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Wed, May 17, 2006:
> Frederic Marchal found a problem with POP3 deletes when --flush and
> --keep are both unset and UIDL is used for the connection:
> 
> If fetchmail sends a DELE command, but the POP3 connection aborts before
> the QUIT command can be sent, then fetchmail will skip the messages on
> the server and leaving them there forever.
> 
> The patch below is somewhat intrusive but attempts to fix this. It
> changes the .fetchids file format, adding " DELETED" to UID lines of
> deleted messages when the QUIT hasn't been acknowledged. This should
> make fetchmail retry the delete on the next run.
> 
> Please test.
> 
> Opinions? Let it go in now or wait until 6.4.0?

A far safer option is to just redownload the mail. I think, there are
issues regarding POP servers recycling UIDs, thereby assigning the
same UID of a mail which was deleted to a new mail.

For such servers, there could also be a race condition on a socket
error. For example, if fetchmail sends a "QUIT" which the POP3 server
receives and if the POP3 server sends an "+OK" which fetchmail does
not receive due to socket error. In this situation, the server has
already expunged the mails but fetchmail does not know that. Now, if
the server assigns the same UID to a new mail, fetchmail may end up
deleting that mail.

This patch should now download the mail when used without 'flush'.
This will lead to duplicate mails on socket errors.

Frederic, your original query was whether 'flush' is safe or not. It
is safe to use 'flush' with 'uidl', unless your mailserver is
reassigning the same UIDs of mails which have been deleted recently to
new mails received after the deletion.

The patch attached also attempts to fix the manpage regarding the
'flush' option.

-- 
Sunil Shetye.

Re: [fetchmail-users] Mails lost due to --flush option ?

[Frederic M. <fre...@wo...>]

Matthias Andree wrote:
> That's a problem I can confirm looking at the code. POP3 delete attempts
> mark a message seen (rather than deleted), and thus the message is
> skipped.
>
> If you are using ONLY "POP3 + UIDL" configurations, --flush should be safe.
>   
Thank you. I am indeed using pop3 and the uidl.
> Frederic Marchal <fre...@wo...> later answered:
>
>   
>> I tested it on the test server and it worked as expected. Now, it is in
>> production but I may have to wait until next Monday for the flood of
>> mails from the week-end to know if it is ok (and hope the connection
>> drops :-) ).
>>
>> BTW, isn't it another way of doing the flush you said was a scary thing ?
>>     
>
> Well, it is in fact fishing in --flush's pond functionality-wise,
> so I am not going to apply that patch now.
>   
I confirm it is working as expected. I just got a connection drop 
yesterday and fetchmail properly deleted the old mails while downloading 
the unread ones, even those that came in the meantime. It worked even 
though one new mail was inserted in the middle of the UIDL of the 
messages that were downloaded in the previous poll.

Thank you again Matthias.

Frederic

[fetchmail-users] PATCH: EXPERIMENTAL: fix lost POP3 deletes

[Matthias A. <mat...@gm...>]

Greetings,

Frederic Marchal found a problem with POP3 deletes when --flush and
--keep are both unset and UIDL is used for the connection:

If fetchmail sends a DELE command, but the POP3 connection aborts before
the QUIT command can be sent, then fetchmail will skip the messages on
the server and leaving them there forever.

The patch below is somewhat intrusive but attempts to fix this. It
changes the .fetchids file format, adding " DELETED" to UID lines of
deleted messages when the QUIT hasn't been acknowledged. This should
make fetchmail retry the delete on the next run.

Please test.

Opinions? Let it go in now or wait until 6.4.0?

-- 
Matthias Andree

Re: [fetchmail-users] Mails lost due to --flush option ?

[Matthias A. <mat...@gm...>]

Frederic Marchal <fre...@wo...> writes:

> I quote the man page of version 6.3.2:
>
>       -F | --flush
>              POP3/IMAP only.  Delete old (previously retrieved)
>              messages from the mailserver before retrieving new
>              messages.  This option does not work with ETRN or ODMR.
>              Warning: if your local MTA hangs and fetchmail is
>              aborted, the next time you run fetchmail, it will delete
>              mail that was never delivered to you.  What you probably
>              want is the default setting: if you don't specify `-k',
>              then fetchmail will automatically delete messages after
>              successful delivery.
>
> Is it still true that a mail is lost if the MTA hangs  ? It sounds like
> a known bug and may have been fixed in the code but not in the man page.

Well, it indeed appears the bug got fixed. Messages will get delete:

1. if --flush is _en_abled and an "old" message is encountered
   (--flush also disables fastUIDL code in POP3)

2. if --keep is _dis_abled and a message has successfully been shipped

I don't see how a local MTA hang could cause message deletion --
delivery problems inhibit deletion and marking a message as old.  EXCEPT
if the server implicitly marks a message seen (IMAP2 and POP3 without
UIDL)

> I explain my situation: if the connection with the pop3 server drops,
> all the DELE commands are forgotten (as required by the pop3 protocol if
> I understand it right)

correct.

> and the mails reappear on the next poll.

That's a problem I can confirm looking at the code. POP3 delete attempts
mark a message seen (rather than deleted), and thus the message is
skipped.

If you are using ONLY "POP3 + UIDL" configurations, --flush should be safe.



Frederic Marchal <fre...@wo...> later answered:

> I tested it on the test server and it worked as expected. Now, it is in
> production but I may have to wait until next Monday for the flood of
> mails from the week-end to know if it is ok (and hope the connection
> drops :-) ).
>
> BTW, isn't it another way of doing the flush you said was a scary thing ?

Well, it is in fact fishing in --flush's pond functionality-wise,
so I am not going to apply that patch now.

I'll post an experimental test patch to fix the original bug later, but
it's fairly intrusive, so it needs testing and review.

-- 
Matthias Andree

[fetchmail-users] [PATCH] Re: only one message gets fetched using IMAP idle

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Mon, May 15, 2006:
> > But if I try to not to expunge the messages on the server after the 
> > transfer (using "keep" configuration option), fetchmail immediately 
> > terminates after fetching the first message. Don't know if this is intended 
> > behaviour?
> 
> Time to summon Sunil who knows a bit more about the idle code...

Thanks, Matthias.

> Well, --keep and --idle don't appear to be friends, but it doesn't
> appear to be intentional either, at least it's not documented; and the
> interfaces and assumptions in fetchmail aren't documented well, so I
> need to figure out which function expects what input and is expected to
> give what output...
> 
> I need to think about this a bit and understand the IMAP message
> counting stuff a bit better -- and decide if I really want to fix this
> in 6.3.X or only in 6.4.X where I plan to add UID support for IMAP.
> 
> I have committed this stopgap code to SVN however, but I'm not promising
> I can easily fix --keep --idle with reasonable effort.

Here is a patch which calculates recentcount as a difference between
new and old EXISTS count. fetchmail will now no longer use the RECENT
flag.

There was a reason why "keep" and "idle" options could not work
together before. The RECENT flag never gets cleared from a new mail
unless you logout or you delete and expunge the mail. Thus, when
"keep" is enabled, a mail which was \Recent remains so forever(*) for
that session, even if the mail is downloaded and marked as \Seen. Note
that the \Recent flag cannot be cleared by fetchmail (RFC 3501). This
would cause fetchmail to loop forever and not go into IDLE mode.

Now, with this patch, since the \Recent flag will not be used, it is
possible to try supporting "keep" and "idle" together. This patch also
patches driver.c to support this. Note that fetchmail will still not
enter the IDLE mode if you do not have any new mail. This table should
make the picture clearer:

	with "keep" and "idle" options
Mails   New mails       Enter IDLE mode    Enter IDLE mode
                        before patch?      after patch?
---------------------------------------------------------------
 0       0               Yes                Yes
 >= 1    0               No                 No
 >= 1    >= 1            No                 Yes

Please try this patch and report. Note that this patch has to be
applied after the patch by Matthias Andree.

-- 
Sunil Shetye.

(*) Of course, you may find some mailservers who do clear the \Recent
flag automatically when the \Seen flag is set.

Re: [fetchmail-users] only one message gets fetched using IMAP idle

[Matthias A. <mat...@gm...>]

Lars Tewes schrieb am 2006-05-15:

> But if I try to not to expunge the messages on the server after the 
> transfer (using "keep" configuration option), fetchmail immediately 
> terminates after fetching the first message. Don't know if this is intended 
> behaviour?

Time to summon Sunil who knows a bit more about the idle code...

Well, --keep and --idle don't appear to be friends, but it doesn't
appear to be intentional either, at least it's not documented; and the
interfaces and assumptions in fetchmail aren't documented well, so I
need to figure out which function expects what input and is expected to
give what output...

I need to think about this a bit and understand the IMAP message
counting stuff a bit better -- and decide if I really want to fix this
in 6.3.X or only in 6.4.X where I plan to add UID support for IMAP.

I have committed this stopgap code to SVN however, but I'm not promising
I can easily fix --keep --idle with reasonable effort.

HTH,

-- 
Matthias Andree