fetchmail-users — User questions and discussion

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Frederic M. <fre...@wo...>]

Matthias Andree wrote:
> (collecting replies to two postings)
>   
>> - Some mailservers keep the flags of a mail in the mail itself by
>>   adding a header like Status:. So, the size of a mail may actually
>>   change when it turns from 'new' to 'old'. Due to size mismatch,
>>   mails from such mailservers will get downloaded again.
>>     
>
> That is a rather important concern.
>
> The refinement of the suggestion would be to hash all but a few
> non-constant headers with some decent hash function. MD5 would be
> simple, but we shouldn't hardcode anything here.
>   
You are right. This is a better.

To reduce the load, the hash could be used only when both the UID and 
the size of a previously deleted message are found again on the next 
poll. That would not occur too often and, therefore, it should not 
increase significantly the load on a relatively stable connection with a 
server that uses good UID and reports valid constant mail size.

The user could also choose to use the hash instead of the size when the 
size reported by the server is unreliable.

>>> - Some mailservers keep the flags of a mail in the mail itself by
>>>   adding a header like Status:. So, the size of a mail may actually
>>>   change when it turns from 'new' to 'old'. Due to size mismatch,
>>>   mails from such mailservers will get downloaded again.
>>>       
>
>   
>> You got me there :-) In that case, the mail would be downloaded a second
>> time and deleted.
>>     
>
> The "message of same size" (say, automated messages in a fixed format,
> stock reports, particular logs) problem isn't solved though - and they
> might even get the same UID if the server used just the MD5 or a file
> name with a recycled inode number...
>   
I don't think so. If two messages are really different, the date and the 
message ID should at least be different. Even for a badly formated mail 
with a constant date and message ID sent, for instance, by a crontab 
running sendmail us...@ex... < fixed_mail.txt, the received headers 
should be different. If the two mails are completely identical and lead 
to the same hash, then they are the same mail.

The hash of the significant headers looks safe to me if the "significant 
headers" are carefully chosen.

> The question is (1) if it's worth the added effort to track sizes, or
> (2) if we should rather go Sunil's simple "play it safe and redownload
> if in doubt" route; or (3) refine my patch to assume QUIT succeeded the
> moment it is handed off to the write() call. That also has the effect of
> redownloading messages if the QUIT fails, but will retry the DELEte if
> fetchmail doesn't reach the point where it would send QUIT.
>   
Fetchmail should not (1) leave the mails on the server forever, (2) 
download the mails again and again because it chokes on one mail, (3) 
drop one mail because it could not distinguish it from a previously 
deleted mail.

Now, to make things clearer, let me tell you a problem that occurred to 
me. It explain the reason I add point (2) above and is a concrete 
example of something that can go wrong.

We have a pop3 proxy (p3scan) through which fetchmail downloads the 
mails from the external pop3 server. p3scan get the whole mail and hand 
it to clamd to scan it for a virus. If everything is ok, the mail is 
passed to fetchmail. Therefore, fetchmail receives the mail after some 
delay depending on the size of the mail but it can send it successfully 
to postfix. Then, fetchmail tries to send the dele command and it fails 
because the scanning of the mail by clamd took too much time and the 
pop3 server timed out.

If fetchmail can discover that the mail was delivered but it could not 
be deleted on the previous poll (remember the dele command was never 
acknowledged by the server) and delete it without downloading it again, 
then the problem is solved. In my case, I received a 32MB mails and it 
took 13 minutes to scan it. It is far beyond the patience of the most 
patient server... Therefore, the mail kept being downloaded over and 
over and delivered to the two users on the To field... Yes it is rather 
painful when you open you mailbox and discover 15 copies of a 32MB mail :-)

>> And this, only if the connection is dropped after the delete command
>> is sent AND the size of the mail is changed by the server on the next
>> poll.
>>
>> The worst case occurs if the size of the mail keeps changing at every
>> poll. I have no solution here. It's not funny at all to have to deal
>> with broken servers :-(
>>     
>
> I think I might just charge users for adding more workarounds. Then they
> have the cheap route of complaining to the server's operator, or the
> expensive one of having the fix. :-P
>
> I think size isn't necessarily the best complement here to detect
> changes. I like the hash approach better, but this seems quite intrusive
> as well.
>
> A hash that precludes some dynamic headers such as [X-]Status and
> similar may be needed anyways to emulate UIDL for --keep setups on
> servers that don't support UIDL -- but if they lack all UIDL and TOP,
> the network impact of --keep will be rather painful.
> (Perhaps fetchmail should just refuse --keep on such servers.)
>   
Or it might be documented. The user would then have to choose between 
the "cheap route" or the time consuming one :-)

Frederic

[fetchmail-users] Re: [PATCH] Re: only one message gets fetched using IMAP idle

[Lars T. <ls...@gm...>]

Hello Sunil,

sorry for the late answer...

Sunil Shetye wrote:

 > 	with "keep" and "idle" options
 > Mails   New mails       Enter IDLE mode    Enter IDLE mode
 >                         before patch?      after patch?
 > ---------------------------------------------------------------
 >  0       0               Yes                Yes
 >  >= 1    0               No                 No
 >  >= 1    >= 1            No                 Yes

Thanks for the patch, it worked exactly as described!

 > Please try this patch and report. Note that this patch has to be
 > applied after the patch by Matthias Andree.

So did I, but 1 Hunk failed for me:
---------------------------------------------------------------------
patching file driver.c
Hunk #1 succeeded at 1458 (offset -3 lines).
patching file imap.c
Hunk #5 succeeded at 687 (offset 2 lines).
Hunk #6 succeeded at 699 (offset 2 lines).
Hunk #7 succeeded at 714 (offset 2 lines).
Hunk #8 FAILED at 723.
Hunk #9 succeeded at 779 (offset 2 lines).
1 out of 9 hunks FAILED -- saving rejects to file imap.c.rej


imap.c.rej:
***************
*** 719,725 ****
       }
       else
       {
-       count = 0;
         ok = gen_transact(sock,
                           check_only ? "EXAMINE \"%s\"" : "SELECT \"%s\"",
                           folder ? folder : "INBOX");
--- 723,729 ----
       }
       else
       {
+       oldcount = count = 0;
         ok = gen_transact(sock,
                           check_only ? "EXAMINE \"%s\"" : "SELECT \"%s\"",
                           folder ? folder : "INBOX");
---------------------------------------------------------------------

I added it manually - this shouldn't be a problem though.

Nice one - thanks again to both of you!

Best regards,
Lars

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Matthias A. <mat...@gm...>]

(collecting replies to two postings)

Sunil Shetye <sh...@bo...> writes in response to
Frederic Marchal's text (not cited):


[Frederic suggested to save the (UID,size) pair to detect message
recycling.]
> There are a few objections to this:
>
> - fetchmail currently does not store the size in the fetchids file, so
>   doing this will require a modification of the fetchids file format.

fetchmail's uid.c code (just revised a bit so as to be more readable)
does the equivalent of (Perl RE syntax):

      __user__|domain_        ___id____
[ \t]*([^@]\*)@([^ ]+)[ \t]* +([^ \t]*).*

A typical line that can be parsed looks like this:

some user@re...@po... d3b07384.d113edec

And the parser since 6.3.0 is robust, so that we can add a blank or TAB
and anything after a blank after the ID:

some user@re...@po... d3b07384.d113edec foo=bar baz=23|joe=mary

Saving any information such as deleted flags, size, retrieval date is
thus backwards compatible in the sense that older fetchmail versions
would just ignore the extra information and strip it when rewriting the
ID. Exception: scratch lists are saved verbatim, i. e. for unqueried
hosts, the extra information would be kept.

> - fetchmail does not get the size of all mails right now (in the
>   default setup), only of the mails it is going to download. Your
>   suggestion would imply that fetchmail will have to get UIDs as well
>   as sizes of all mails for comparison. This would imply a lot of
>   delay in getting mails, especially when 'keep' is on.

1. POP3 "LIST" is fast, has affordable traffic requirements and part of
   regular fetchmail operation.

2. The size is, as suggested, only needed for messages that are supposed
   to be deleted after retrieval. Discounting --flush, fetchmail marks
   the message for deletion after retrieving it, and at that time, the
   POP3 server's notion of the message size is known. If the size is
   cached in the .fetchids file, that part is covered. For new messages,
   the size will have to be retrieved anyways sooner or later to support
   ESMTP SIZE.

> - Some mailservers keep the flags of a mail in the mail itself by
>   adding a header like Status:. So, the size of a mail may actually
>   change when it turns from 'new' to 'old'. Due to size mismatch,
>   mails from such mailservers will get downloaded again.

That is a rather important concern.

The refinement of the suggestion would be to hash all but a few
non-constant headers with some decent hash function. MD5 would be
simple, but we shouldn't hardcode anything here.

> - Some mailservers misreport sizes! I remember one such server being
>   reported:
>
> <http://lists.ccil.org/pipermail/fetchmail-friends/2002-January/005572.html>

Can we assume that the size is stable even if inaccurate?



Frederic Marchal <fre...@wo...> replied:

> For which a change was proposed if I remember correctly... It was even
> suggested to use a database such as sqlite. I admit I haven't followed
> the status of that proposition. Beside, there is no hurry. It doesn't
> have to be a patch within two days. It may be part of a major version
> change just to make fetchmail even better :-)

sqlite won't happen for fetchmail 6.3.X.

Extending .fetchids as suggested above might.

> processing would remain the same. The fetchids file may even be purged
> of the old UID when the next poll confirms the mails are really gone.

That happens automatically the next time that POP3 chat with the server
completes successfully.

>> - Some mailservers keep the flags of a mail in the mail itself by
>>   adding a header like Status:. So, the size of a mail may actually
>>   change when it turns from 'new' to 'old'. Due to size mismatch,
>>   mails from such mailservers will get downloaded again.

> You got me there :-) In that case, the mail would be downloaded a second
> time and deleted.

The "message of same size" (say, automated messages in a fixed format,
stock reports, particular logs) problem isn't solved though - and they
might even get the same UID if the server used just the MD5 or a file
name with a recycled inode number...

The question is (1) if it's worth the added effort to track sizes, or
(2) if we should rather go Sunil's simple "play it safe and redownload
if in doubt" route; or (3) refine my patch to assume QUIT succeeded the
moment it is handed off to the write() call. That also has the effect of
redownloading messages if the QUIT fails, but will retry the DELEte if
fetchmail doesn't reach the point where it would send QUIT.

> If the deletion fails again, the mail may be downloaded once more if
> its size keeps changing or it may eventually be deleted without any
> further download if the size is not changed any more. If the download
> is the action that causes the connection drop (such as a proxy with a
> virus scanner taking too much time and exceeding the timeout of the
> server), it will be downloaded twice and deleted the third time.

A fetchmail-internal queue might fix this much later on if someone could
document the need for such.

> And this, only if the connection is dropped after the delete command
> is sent AND the size of the mail is changed by the server on the next
> poll.
>
> The worst case occurs if the size of the mail keeps changing at every
> poll. I have no solution here. It's not funny at all to have to deal
> with broken servers :-(

I think I might just charge users for adding more workarounds. Then they
have the cheap route of complaining to the server's operator, or the
expensive one of having the fix. :-P

I think size isn't necessarily the best complement here to detect
changes. I like the hash approach better, but this seems quite intrusive
as well.

A hash that precludes some dynamic headers such as [X-]Status and
similar may be needed anyways to emulate UIDL for --keep setups on
servers that don't support UIDL -- but if they lack all UIDL and TOP,
the network impact of --keep will be rather painful.
(Perhaps fetchmail should just refuse --keep on such servers.)

> Now, you may argue that it is a lot of programming for the sole benefit
> of not deleting unread mails when the connection drop... I have nothing
> to say against that. I'm not likely to be the one to program it although
> I really wish I could.
>
> Note that this is only a solution I propose to solve the problem of the
> mails being either duplicated, deleted without delivery or left forever
> on the server when a connection drop. I'm happy with my current
> configuration which work fine with pop3 + uidl and a server that doesn't
> recycle the UID too quickly and a connection quite reliable enough.

If your server uses a timestamp into the UID and has a strictly
monotonically increasing system lock, the risk is pretty low.

-- 
Matthias Andree

Re: [fetchmail-users] Re: fix lost POP3 deletes

[Frederic M. <fre...@wo...>]

Sunil Shetye wrote:
> Quoting from Frederic Marchal's mail on Thu, May 18, 2006:
>   
>> Using pop3, the size of the mail is one additional information that can 
>> be used to ensure it is not the same message with the same UID. The list 
>> command returns it and fetchmail does use it during the communication. I 
>> expect it would be much less likely that a UID is recycled on a message 
>> with the exact same size.
>>
>> If a message is using the same UID as a message we know should have been 
>> deleted, then list its size and compare it. If it is the same, it can be 
>> deleted again. If not, it is a new message and should be downloaded.
>>
>> Does it make sense ?
>>     
>
> There are a few objections to this:
>
> - fetchmail currently does not store the size in the fetchids file, so
>   doing this will require a modification of the fetchids file format.
>   
For which a change was proposed if I remember correctly... It was even 
suggested to use a database such as sqlite. I admit I haven't followed 
the status of that proposition. Beside, there is no hurry. It doesn't 
have to be a patch within two days. It may be part of a major version 
change just to make fetchmail even better :-)
> - fetchmail does not get the size of all mails right now (in the
>   default setup), only of the mails it is going to download. Your
>   suggestion would imply that fetchmail will have to get UIDs as well
>   as sizes of all mails for comparison. This would imply a lot of
>   delay in getting mails, especially when 'keep' is on.
>   
Only for the mails with a known UID for which fetchmail sent a delete 
command in a previous poll (that should not occur too often because most 
of the time the mails will effectively be deleted on the server when 
fetchmail closes the connection). It would only serve to find out if the 
mail is different and should be deleted or downloaded. For an unknown 
UID or the UID of a mail that was left on the server on purpose, the 
processing would remain the same. The fetchids file may even be purged 
of the old UID when the next poll confirms the mails are really gone. A 
recycled UID coming afterwards would then be seen as a new mail without 
any further comparison.

> - Some mailservers keep the flags of a mail in the mail itself by
>   adding a header like Status:. So, the size of a mail may actually
>   change when it turns from 'new' to 'old'. Due to size mismatch,
>   mails from such mailservers will get downloaded again.
>   
You got me there :-) In that case, the mail would be downloaded a second 
time and deleted. If the deletion fails again, the mail may be 
downloaded once more if its size keeps changing or it may eventually be 
deleted without any further download if the size is not changed any 
more. If the download is the action that causes the connection drop 
(such as a proxy with a virus scanner taking too much time and exceeding 
the timeout of the server), it will be downloaded twice and deleted the 
third time. And this, only if the connection is dropped after the delete 
command is sent AND the size of the mail is changed by the server on the 
next poll.

The worst case occurs if the size of the mail keeps changing at every 
poll. I have no solution here. It's not funny at all to have to deal 
with broken servers :-(

> - Some mailservers misreport sizes! I remember one such server being
>   reported:
>
> <http://lists.ccil.org/pipermail/fetchmail-friends/2002-January/005572.html>
>
>   
It depends on the problem with the size. If the reported size is always 
the same, there is no problem. Now, if the reported size is a random 
function, there is no way it can work. In that case, when the mails fail 
to be deleted, they are downloaded again over and over until the delete 
commands succeed. On the other hand, for all those who have a good 
server reporting the correct sizes, there will be an improvement and the 
mails will not be duplicated when the connection drop.

Now, you may argue that it is a lot of programming for the sole benefit 
of not deleting unread mails when the connection drop... I have nothing 
to say against that. I'm not likely to be the one to program it although 
I really wish I could.

Note that this is only a solution I propose to solve the problem of the 
mails being either duplicated, deleted without delivery or left forever 
on the server when a connection drop. I'm happy with my current 
configuration which work fine with pop3 + uidl and a server that doesn't 
recycle the UID too quickly and a connection quite reliable enough.

Frederic

[fetchmail-users] Re: fix lost POP3 deletes

[Sunil S. <sh...@bo...>]

Quoting from Frederic Marchal's mail on Thu, May 18, 2006:
> Using pop3, the size of the mail is one additional information that can 
> be used to ensure it is not the same message with the same UID. The list 
> command returns it and fetchmail does use it during the communication. I 
> expect it would be much less likely that a UID is recycled on a message 
> with the exact same size.
> 
> If a message is using the same UID as a message we know should have been 
> deleted, then list its size and compare it. If it is the same, it can be 
> deleted again. If not, it is a new message and should be downloaded.
> 
> Does it make sense ?

There are a few objections to this:

- fetchmail currently does not store the size in the fetchids file, so
  doing this will require a modification of the fetchids file format.

- fetchmail does not get the size of all mails right now (in the
  default setup), only of the mails it is going to download. Your
  suggestion would imply that fetchmail will have to get UIDs as well
  as sizes of all mails for comparison. This would imply a lot of
  delay in getting mails, especially when 'keep' is on.

- Some mailservers keep the flags of a mail in the mail itself by
  adding a header like Status:. So, the size of a mail may actually
  change when it turns from 'new' to 'old'. Due to size mismatch,
  mails from such mailservers will get downloaded again.

- Some mailservers misreport sizes! I remember one such server being
  reported:

<http://lists.ccil.org/pipermail/fetchmail-friends/2002-January/005572.html>

-- 
Sunil Shetye.

[fetchmail-users] Re: [PATCH] Re: fix lost POP3 deletes

[Frederic M. <fre...@wo...>]

Sunil Shetye wrote:
> Quoting from Matthias Andree's mail on Wed, May 17, 2006:
>   
>> Frederic Marchal found a problem with POP3 deletes when --flush and
>> --keep are both unset and UIDL is used for the connection:
>>
>> If fetchmail sends a DELE command, but the POP3 connection aborts before
>> the QUIT command can be sent, then fetchmail will skip the messages on
>> the server and leaving them there forever.
>>
>> The patch below is somewhat intrusive but attempts to fix this. It
>> changes the .fetchids file format, adding " DELETED" to UID lines of
>> deleted messages when the QUIT hasn't been acknowledged. This should
>> make fetchmail retry the delete on the next run.
>>
>> Please test.
>>
>> Opinions? Let it go in now or wait until 6.4.0?
>>     
>
> A far safer option is to just redownload the mail. I think, there are
> issues regarding POP servers recycling UIDs, thereby assigning the
> same UID of a mail which was deleted to a new mail.
>
> For such servers, there could also be a race condition on a socket
> error. For example, if fetchmail sends a "QUIT" which the POP3 server
> receives and if the POP3 server sends an "+OK" which fetchmail does
> not receive due to socket error. In this situation, the server has
> already expunged the mails but fetchmail does not know that. Now, if
> the server assigns the same UID to a new mail, fetchmail may end up
> deleting that mail.
>
> This patch should now download the mail when used without 'flush'.
> This will lead to duplicate mails on socket errors.
>
> Frederic, your original query was whether 'flush' is safe or not. It
> is safe to use 'flush' with 'uidl', unless your mailserver is
> reassigning the same UIDs of mails which have been deleted recently to
> new mails received after the deletion.
>   
Using pop3, the size of the mail is one additional information that can 
be used to ensure it is not the same message with the same UID. The list 
command returns it and fetchmail does use it during the communication. I 
expect it would be much less likely that a UID is recycled on a message 
with the exact same size.

If a message is using the same UID as a message we know should have been 
deleted, then list its size and compare it. If it is the same, it can be 
deleted again. If not, it is a new message and should be downloaded.

Does it make sense ?

Frederic

[fetchmail-users] [PATCH] Re: fix lost POP3 deletes

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Wed, May 17, 2006:
> Frederic Marchal found a problem with POP3 deletes when --flush and
> --keep are both unset and UIDL is used for the connection:
> 
> If fetchmail sends a DELE command, but the POP3 connection aborts before
> the QUIT command can be sent, then fetchmail will skip the messages on
> the server and leaving them there forever.
> 
> The patch below is somewhat intrusive but attempts to fix this. It
> changes the .fetchids file format, adding " DELETED" to UID lines of
> deleted messages when the QUIT hasn't been acknowledged. This should
> make fetchmail retry the delete on the next run.
> 
> Please test.
> 
> Opinions? Let it go in now or wait until 6.4.0?

A far safer option is to just redownload the mail. I think, there are
issues regarding POP servers recycling UIDs, thereby assigning the
same UID of a mail which was deleted to a new mail.

For such servers, there could also be a race condition on a socket
error. For example, if fetchmail sends a "QUIT" which the POP3 server
receives and if the POP3 server sends an "+OK" which fetchmail does
not receive due to socket error. In this situation, the server has
already expunged the mails but fetchmail does not know that. Now, if
the server assigns the same UID to a new mail, fetchmail may end up
deleting that mail.

This patch should now download the mail when used without 'flush'.
This will lead to duplicate mails on socket errors.

Frederic, your original query was whether 'flush' is safe or not. It
is safe to use 'flush' with 'uidl', unless your mailserver is
reassigning the same UIDs of mails which have been deleted recently to
new mails received after the deletion.

The patch attached also attempts to fix the manpage regarding the
'flush' option.

-- 
Sunil Shetye.

Re: [fetchmail-users] Mails lost due to --flush option ?

[Frederic M. <fre...@wo...>]

Matthias Andree wrote:
> That's a problem I can confirm looking at the code. POP3 delete attempts
> mark a message seen (rather than deleted), and thus the message is
> skipped.
>
> If you are using ONLY "POP3 + UIDL" configurations, --flush should be safe.
>   
Thank you. I am indeed using pop3 and the uidl.
> Frederic Marchal <fre...@wo...> later answered:
>
>   
>> I tested it on the test server and it worked as expected. Now, it is in
>> production but I may have to wait until next Monday for the flood of
>> mails from the week-end to know if it is ok (and hope the connection
>> drops :-) ).
>>
>> BTW, isn't it another way of doing the flush you said was a scary thing ?
>>     
>
> Well, it is in fact fishing in --flush's pond functionality-wise,
> so I am not going to apply that patch now.
>   
I confirm it is working as expected. I just got a connection drop 
yesterday and fetchmail properly deleted the old mails while downloading 
the unread ones, even those that came in the meantime. It worked even 
though one new mail was inserted in the middle of the UIDL of the 
messages that were downloaded in the previous poll.

Thank you again Matthias.

Frederic

[fetchmail-users] PATCH: EXPERIMENTAL: fix lost POP3 deletes

[Matthias A. <mat...@gm...>]

Greetings,

Frederic Marchal found a problem with POP3 deletes when --flush and
--keep are both unset and UIDL is used for the connection:

If fetchmail sends a DELE command, but the POP3 connection aborts before
the QUIT command can be sent, then fetchmail will skip the messages on
the server and leaving them there forever.

The patch below is somewhat intrusive but attempts to fix this. It
changes the .fetchids file format, adding " DELETED" to UID lines of
deleted messages when the QUIT hasn't been acknowledged. This should
make fetchmail retry the delete on the next run.

Please test.

Opinions? Let it go in now or wait until 6.4.0?

-- 
Matthias Andree

Re: [fetchmail-users] Mails lost due to --flush option ?

[Matthias A. <mat...@gm...>]

Frederic Marchal <fre...@wo...> writes:

> I quote the man page of version 6.3.2:
>
>       -F | --flush
>              POP3/IMAP only.  Delete old (previously retrieved)
>              messages from the mailserver before retrieving new
>              messages.  This option does not work with ETRN or ODMR.
>              Warning: if your local MTA hangs and fetchmail is
>              aborted, the next time you run fetchmail, it will delete
>              mail that was never delivered to you.  What you probably
>              want is the default setting: if you don't specify `-k',
>              then fetchmail will automatically delete messages after
>              successful delivery.
>
> Is it still true that a mail is lost if the MTA hangs  ? It sounds like
> a known bug and may have been fixed in the code but not in the man page.

Well, it indeed appears the bug got fixed. Messages will get delete:

1. if --flush is _en_abled and an "old" message is encountered
   (--flush also disables fastUIDL code in POP3)

2. if --keep is _dis_abled and a message has successfully been shipped

I don't see how a local MTA hang could cause message deletion --
delivery problems inhibit deletion and marking a message as old.  EXCEPT
if the server implicitly marks a message seen (IMAP2 and POP3 without
UIDL)

> I explain my situation: if the connection with the pop3 server drops,
> all the DELE commands are forgotten (as required by the pop3 protocol if
> I understand it right)

correct.

> and the mails reappear on the next poll.

That's a problem I can confirm looking at the code. POP3 delete attempts
mark a message seen (rather than deleted), and thus the message is
skipped.

If you are using ONLY "POP3 + UIDL" configurations, --flush should be safe.



Frederic Marchal <fre...@wo...> later answered:

> I tested it on the test server and it worked as expected. Now, it is in
> production but I may have to wait until next Monday for the flood of
> mails from the week-end to know if it is ok (and hope the connection
> drops :-) ).
>
> BTW, isn't it another way of doing the flush you said was a scary thing ?

Well, it is in fact fishing in --flush's pond functionality-wise,
so I am not going to apply that patch now.

I'll post an experimental test patch to fix the original bug later, but
it's fairly intrusive, so it needs testing and review.

-- 
Matthias Andree

[fetchmail-users] [PATCH] Re: only one message gets fetched using IMAP idle

[Sunil S. <sh...@bo...>]

Quoting from Matthias Andree's mail on Mon, May 15, 2006:
> > But if I try to not to expunge the messages on the server after the 
> > transfer (using "keep" configuration option), fetchmail immediately 
> > terminates after fetching the first message. Don't know if this is intended 
> > behaviour?
> 
> Time to summon Sunil who knows a bit more about the idle code...

Thanks, Matthias.

> Well, --keep and --idle don't appear to be friends, but it doesn't
> appear to be intentional either, at least it's not documented; and the
> interfaces and assumptions in fetchmail aren't documented well, so I
> need to figure out which function expects what input and is expected to
> give what output...
> 
> I need to think about this a bit and understand the IMAP message
> counting stuff a bit better -- and decide if I really want to fix this
> in 6.3.X or only in 6.4.X where I plan to add UID support for IMAP.
> 
> I have committed this stopgap code to SVN however, but I'm not promising
> I can easily fix --keep --idle with reasonable effort.

Here is a patch which calculates recentcount as a difference between
new and old EXISTS count. fetchmail will now no longer use the RECENT
flag.

There was a reason why "keep" and "idle" options could not work
together before. The RECENT flag never gets cleared from a new mail
unless you logout or you delete and expunge the mail. Thus, when
"keep" is enabled, a mail which was \Recent remains so forever(*) for
that session, even if the mail is downloaded and marked as \Seen. Note
that the \Recent flag cannot be cleared by fetchmail (RFC 3501). This
would cause fetchmail to loop forever and not go into IDLE mode.

Now, with this patch, since the \Recent flag will not be used, it is
possible to try supporting "keep" and "idle" together. This patch also
patches driver.c to support this. Note that fetchmail will still not
enter the IDLE mode if you do not have any new mail. This table should
make the picture clearer:

	with "keep" and "idle" options
Mails   New mails       Enter IDLE mode    Enter IDLE mode
                        before patch?      after patch?
---------------------------------------------------------------
 0       0               Yes                Yes
 >= 1    0               No                 No
 >= 1    >= 1            No                 Yes

Please try this patch and report. Note that this patch has to be
applied after the patch by Matthias Andree.

-- 
Sunil Shetye.

(*) Of course, you may find some mailservers who do clear the \Recent
flag automatically when the \Seen flag is set.

Re: [fetchmail-users] only one message gets fetched using IMAP idle

[Matthias A. <mat...@gm...>]

Lars Tewes schrieb am 2006-05-15:

> But if I try to not to expunge the messages on the server after the 
> transfer (using "keep" configuration option), fetchmail immediately 
> terminates after fetching the first message. Don't know if this is intended 
> behaviour?

Time to summon Sunil who knows a bit more about the idle code...

Well, --keep and --idle don't appear to be friends, but it doesn't
appear to be intentional either, at least it's not documented; and the
interfaces and assumptions in fetchmail aren't documented well, so I
need to figure out which function expects what input and is expected to
give what output...

I need to think about this a bit and understand the IMAP message
counting stuff a bit better -- and decide if I really want to fix this
in 6.3.X or only in 6.4.X where I plan to add UID support for IMAP.

I have committed this stopgap code to SVN however, but I'm not promising
I can easily fix --keep --idle with reasonable effort.

HTH,

-- 
Matthias Andree

Re: [fetchmail-users] only one message gets fetched using IMAP idle

[Lars T. <ls...@gm...>]

Hello Matthias,

Matthias Andree wrote:
> Lars Tewes <ls...@gm...> writes:
> 
>> Using fetchmail successfully with polling by interval for several months
>> I just tried to use the IMAP idle-extension to get my mails delivered
>> immediately.
>>
>> Unfortunately only one message gets fetched - after that, fetchmail
>> recognizes the arrival of a new one, but instead of downloading the
>> message, it just triggers another idle.
> 
> Does the attached patch help?

> Index: imap.c
> ===================================================================
> --- imap.c	(Revision 4840)
> +++ imap.c	(Arbeitskopie)
> @@ -679,7 +679,7 @@

I'm quite impressed by this quick solution - it works! Well, mostly.

Working transfer:

---------------------------------------------------------------------

lsr@lsr# /usr/bin/fetchmail -v -v
fetchmail: 6.3.4 querying imap.strato.de (protocol IMAP) at Mon 15 May 2006 
06:28:17 PM CEST: poll started
fetchmail: IMAP< * OK IMAP server ready
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IDLE IMAP4rev1 AUTH=CRAM-MD5 SORT QUOTA NAMESPACE
fetchmail: IMAP< A0001 OK CAPABILITY completed
fetchmail: Protocol identified as IMAP4 rev 1
fetchmail: will idle after poll
fetchmail: IMAP> A0002 AUTHENTICATE CRAM-MD5
fetchmail: IMAP< + PDQ3NzYuMjI1NDUuMTE0NzcxMDQ5NUBwb3N0LndlYm1haWxlci5kZT4=
fetchmail: decoded as <477...@po...>
fetchmail: IMAP> 
bHNyQHZpZXctc291cmNlLmRlIDEwZmJjOGY4ZmMzYjJiNDQ3N2I4YjUzYzE1NzYzZmJi
fetchmail: IMAP< A0002 OK LOGIN completed
fetchmail: selecting or re-polling default folder
fetchmail: IMAP> A0003 SELECT "INBOX"
fetchmail: IMAP< * 0 EXISTS
fetchmail: IMAP< * 0 RECENT
fetchmail: IMAP< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft \Forwarded)
fetchmail: IMAP< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen 
\Draft \Forwarded)] Permanent flags
fetchmail: IMAP< * OK [UIDVALIDITY 1116049368] UIDVALIDITY value
fetchmail: IMAP< * OK [UIDNEXT 3273] may be the next UID value
fetchmail: IMAP< A0003 OK [READ-WRITE] SELECT completed
fetchmail: 0 messages waiting after first poll
fetchmail: IMAP> A0004 IDLE
fetchmail: IMAP< + will notify you of all changes


[sending first mail now...]


fetchmail: IMAP< * 1 EXISTS
fetchmail: IMAP> DONE
fetchmail: IMAP< A0004 OK IDLE completed
fetchmail: 1 message waiting after re-poll
fetchmail: IMAP> A0005 SEARCH UNSEEN NOT DELETED
fetchmail: IMAP< * SEARCH 1
fetchmail: 1 is unseen
fetchmail: IMAP< A0005 OK SEARCH completed
fetchmail: 1 is first unseen
1 message for ls...@vi... at imap.strato.de.
fetchmail: IMAP> A0006 FETCH 1 RFC822.SIZE
fetchmail: IMAP< * 1 FETCH (RFC822.SIZE 900)
fetchmail: IMAP< A0006 OK Fetch complete
fetchmail: IMAP> A0007 FETCH 1 RFC822.HEADER
fetchmail: IMAP< * 1 FETCH (RFC822.HEADER {895}
reading message ls...@vi...@imap.rzone.de:1 of 1 (895 header octets)
About to rewrite From: Lars Tewes <ls...@gm...>
Rewritten version is From: Lars Tewes <ls...@gm...>

About to rewrite To: Lars Tewes <ls...@vi...>
Rewritten version is To: Lars Tewes <ls...@vi...>

fetchmail: about to deliver with: /usr/bin/procmail -d 'lsr'
#
fetchmail: IMAP< )
fetchmail: IMAP< A0007 OK Fetch complete
fetchmail: IMAP> A0008 FETCH 1 BODY.PEEK[TEXT]
fetchmail: IMAP< * 1 FETCH (BODY[TEXT] {5}
  (5 body octets) *
fetchmail: IMAP< )
fetchmail: IMAP< A0008 OK Fetch complete
  flushed
fetchmail: IMAP> A0009 STORE 1 +FLAGS (\Seen \Deleted)
fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen \Recent))
fetchmail: IMAP< A0009 OK STORE completed
fetchmail: IMAP> A0010 EXPUNGE
fetchmail: IMAP< * 1 EXPUNGE
fetchmail: IMAP< A0010 OK EXPUNGE completed
fetchmail: selecting or re-polling default folder
fetchmail: IMAP> A0011 IDLE
fetchmail: IMAP< + will notify you of all changes


[sending second mail now...]


fetchmail: IMAP< * 1 EXISTS
fetchmail: IMAP> DONE
fetchmail: IMAP< A0011 OK IDLE completed
fetchmail: 1 message waiting after re-poll
fetchmail: IMAP> A0012 SEARCH UNSEEN NOT DELETED
fetchmail: IMAP< * SEARCH 1
fetchmail: 1 is unseen
fetchmail: IMAP< A0012 OK SEARCH completed
fetchmail: 1 is first unseen
1 message for ls...@vi... at imap.strato.de.
fetchmail: IMAP> A0013 FETCH 1 RFC822.SIZE
fetchmail: IMAP< * 1 FETCH (RFC822.SIZE 896)
fetchmail: IMAP< A0013 OK Fetch complete
fetchmail: IMAP> A0014 FETCH 1 RFC822.HEADER
fetchmail: IMAP< * 1 FETCH (RFC822.HEADER {896}
reading message ls...@vi...@imap.rzone.de:1 of 1 (896 header octets)
About to rewrite From: Lars Tewes <ls...@gm...>
Rewritten version is From: Lars Tewes <ls...@gm...>

About to rewrite To: Lars Tewes <ls...@vi...>
Rewritten version is To: Lars Tewes <ls...@vi...>

fetchmail: about to deliver with: /usr/bin/procmail -d 'lsr'
#
fetchmail: IMAP< )
fetchmail: IMAP< A0014 OK Fetch complete
fetchmail: IMAP> A0015 FETCH 1 BODY.PEEK[TEXT]
fetchmail: IMAP< * 1 FETCH (BODY[TEXT] {0}
  (0 body octets)
fetchmail: IMAP< )
fetchmail: IMAP< A0015 OK Fetch complete
  flushed
fetchmail: IMAP> A0016 STORE 1 +FLAGS (\Seen \Deleted)
fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen \Recent))
fetchmail: IMAP< A0016 OK STORE completed
fetchmail: IMAP> A0017 EXPUNGE
fetchmail: IMAP< * 1 EXPUNGE
fetchmail: IMAP< A0017 OK EXPUNGE completed
fetchmail: selecting or re-polling default folder
fetchmail: IMAP> A0018 IDLE
fetchmail: IMAP< + will notify you of all changes

[works!]

---------------------------------------------------------------------

But if I try to not to expunge the messages on the server after the transfer 
(using "keep" configuration option), fetchmail immediately terminates after 
fetching the first message. Don't know if this is intended behaviour?

---------------------------------------------------------------------

Logfile with "keep" option set:

lsr@lsr# /usr/bin/fetchmail -v -v
fetchmail: 6.3.4 querying imap.strato.de (protocol IMAP) at Mon 15 May 2006 
06:35:00 PM CEST: poll started
fetchmail: IMAP< * OK IMAP server ready
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IDLE IMAP4rev1 AUTH=CRAM-MD5 SORT QUOTA NAMESPACE
fetchmail: IMAP< A0001 OK CAPABILITY completed
fetchmail: Protocol identified as IMAP4 rev 1
fetchmail: will idle after poll
fetchmail: IMAP> A0002 AUTHENTICATE CRAM-MD5
fetchmail: IMAP< + PDQ4MDIuMjI1NjkuMTE0NzcxMDg5OUBwb3N0LndlYm1haWxlci5kZT4=
fetchmail: decoded as <480...@po...>
fetchmail: IMAP> 
bHNyQHZpZXctc291cmNlLmRlIGQxMGM5ZjMzMDQwNjYxNTRmOTY0OTkxNTk2Nzk3NzJi
fetchmail: IMAP< A0002 OK LOGIN completed
fetchmail: selecting or re-polling default folder
fetchmail: IMAP> A0003 SELECT "INBOX"
fetchmail: IMAP< * 0 EXISTS
fetchmail: IMAP< * 0 RECENT
fetchmail: IMAP< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft \Forwarded)
fetchmail: IMAP< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen 
\Draft \Forwarded)] Permanent flags
fetchmail: IMAP< * OK [UIDVALIDITY 1116049368] UIDVALIDITY value
fetchmail: IMAP< * OK [UIDNEXT 3276] may be the next UID value
fetchmail: IMAP< A0003 OK [READ-WRITE] SELECT completed
fetchmail: 0 messages waiting after first poll
fetchmail: IMAP> A0004 IDLE
fetchmail: IMAP< + will notify you of all changes


[sending first mail now...]


fetchmail: IMAP< * 1 EXISTS
fetchmail: IMAP> DONE
fetchmail: IMAP< A0004 OK IDLE completed
fetchmail: 1 message waiting after re-poll
fetchmail: IMAP> A0005 SEARCH UNSEEN NOT DELETED
fetchmail: IMAP< * SEARCH 1
fetchmail: 1 is unseen
fetchmail: IMAP< A0005 OK SEARCH completed
fetchmail: 1 is first unseen
1 message for ls...@vi... at imap.strato.de.
fetchmail: IMAP> A0006 FETCH 1 RFC822.SIZE
fetchmail: IMAP< * 1 FETCH (RFC822.SIZE 709)
fetchmail: IMAP< A0006 OK Fetch complete
fetchmail: IMAP> A0007 FETCH 1 RFC822.HEADER
fetchmail: IMAP< * 1 FETCH (RFC822.HEADER {704}
reading message ls...@vi...@imap.rzone.de:1 of 1 (704 header octets)
About to rewrite From: Lars Tewes <ls...@gm...>
Rewritten version is From: Lars Tewes <ls...@gm...>

About to rewrite To: Lars Tewes <ls...@vi...>
Rewritten version is To: Lars Tewes <ls...@vi...>

fetchmail: about to deliver with: /usr/bin/procmail -d 'lsr'
#
fetchmail: IMAP< )
fetchmail: IMAP< A0007 OK Fetch complete
fetchmail: IMAP> A0008 FETCH 1 BODY.PEEK[TEXT]
fetchmail: IMAP< * 1 FETCH (BODY[TEXT] {5}
  (5 body octets) *
fetchmail: IMAP< )
fetchmail: IMAP< A0008 OK Fetch complete
  not flushed
fetchmail: IMAP> A0009 STORE 1 +FLAGS (\Seen)
fetchmail: IMAP< * 1 FETCH (FLAGS (\Seen \Recent))
fetchmail: IMAP< A0009 OK STORE completed
fetchmail: IMAP> A0010 LOGOUT
fetchmail: IMAP< * BYE
fetchmail: IMAP< A0010 OK LOGOUT completed
fetchmail: 6.3.4 querying imap.strato.de (protocol IMAP) at Mon 15 May 2006 
06:35:28 PM CEST: poll completed
fetchmail: not swapping UID lists, no UIDs seen this query
fetchmail: Writing fetchids file.
fetchmail: normal termination, status 0
fetchmail: Writing fetchids file.

---------------------------------------------------------------------

Thanks a lot for the patch!

Best,
Lars

Re: [fetchmail-users] only one message gets fetched using IMAP idle

[Matthias A. <mat...@gm...>]

Lars Tewes <ls...@gm...> writes:

> Using fetchmail successfully with polling by interval for several months
> I just tried to use the IMAP idle-extension to get my mails delivered
> immediately.
>
> Unfortunately only one message gets fetched - after that, fetchmail
> recognizes the arrival of a new one, but instead of downloading the
> message, it just triggers another idle.

Does the attached patch help?

-- 
Matthias Andree

Re: [fetchmail-users] only one message gets fetched using IMAP idle

[Matthias A. <mat...@gm...>]

Lars Tewes <ls...@gm...> writes:

> Unfortunately only one message gets fetched - after that, fetchmail
> recognizes the arrival of a new one, but instead of downloading the
> message, it just triggers another idle.

Looks like an incompatibility with fetchmail's IDLE code: when the server
doesn't send "* n RECENT" during idle, fetchmail doesn't actually detect
the new message, but just re-issues the IDLE command.

I'll see how the code works and see to a fix.

-- 
Matthias Andree

Re: [fetchmail-users] Marking messages 'unread' in fetchmail

[Matthias A. <mat...@gm...>]

"Balaji Krishnamoorthy" <ba...@ti...> writes:

>  I have fetchmail setup running to pull messages from the IMAP server
>  using procmail as the mda. I use mozilla as the mail client and use
>  procmail+fetchmail to run some scripts. I wish fetchmail should not mark
>  the messages as "read" after downloading them as my mozilla email client
>  doesn't recognize the new messages as unread.
>
>  Is there a option to specify in .fetchmailrc to mark the messages
>  "unread" in the server as I can view them as usual in the mozilla.

Unfortunately not yet, because fetchmail cannot use IMAP's UID feature
yet (it knows POP3's UIDL however).

So if you can pull with POP3, that might be a workaround.

-- 
Matthias Andree

Re: [fetchmail-users] Marking messages 'unread' in fetchmail

[Rob M. <rob...@gm...>]

On 5/12/06, Balaji Krishnamoorthy <ba...@ti...> wrote:
>
>  I have fetchmail setup running to pull messages from the IMAP server
>  using procmail as the mda. I use mozilla as the mail client and use
>  procmail+fetchmail to run some scripts. I wish fetchmail should not mark
>  the messages as "read" after downloading them as my mozilla email client
>  doesn't recognize the new messages as unread.
>
>  Is there a option to specify in .fetchmailrc to mark the messages
>  "unread" in the server as I can view them as usual in the mozilla.

In short, no.  Fetchmail was written with the assumption that it would
be the only program accessing a mailbox.

You *might* find that if your ISP uses POP3 that you can use that
interface (ensuring you configure fetchmail with the "keep" and "uidl"
options) to achieve what you want.

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

[fetchmail-users] Marking messages 'unread' in fetchmail

[Balaji K. <ba...@ti...>]

 Hi

 I have fetchmail setup running to pull messages from the IMAP server
 using procmail as the mda. I use mozilla as the mail client and use
 procmail+fetchmail to run some scripts. I wish fetchmail should not mark
 the messages as "read" after downloading them as my mozilla email client
 doesn't recognize the new messages as unread.

 Is there a option to specify in .fetchmailrc to mark the messages
 "unread" in the server as I can view them as usual in the mozilla.

 Thanks in advance
 Balaji.k

[fetchmail-users] only one message gets fetched using IMAP idle

[Lars T. <ls...@gm...>]

Hello!

Using fetchmail successfully with polling by interval for several months I 
just tried to use the IMAP idle-extension to get my mails delivered immediately.

Unfortunately only one message gets fetched - after that, fetchmail recognizes 
the arrival of a new one, but instead of downloading the message, it just 
triggers another idle.

---------------------------------------------------------------------

lsr@lsr:# fetchmail -v -v
fetchmail: 6.3.4 querying imap.strato.de (protocol IMAP) at Wed 10 May 2006 
12:05:46 PM CEST: poll started
fetchmail: IMAP< * OK IMAP server ready
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IDLE IMAP4rev1 AUTH=CRAM-MD5 SORT QUOTA NAMESPACE
fetchmail: IMAP< A0001 OK CAPABILITY completed
fetchmail: Protocol identified as IMAP4 rev 1
fetchmail: will idle after poll
fetchmail: IMAP> A0002 AUTHENTICATE CRAM-MD5
fetchmail: IMAP< + PDQ3NzgwMi4yMTY1Ni4xMTQ3MjU1NTMzQHBvc3Qud2VibWFpbGVyLmRlPg==
fetchmail: decoded as <477...@po...>
fetchmail: IMAP> 
bHNyQHZpZXctc291cmNlLmRlIGQ4Nzg1YzVjMzAxZjI4NTBjZTg4ZDA0ZWMxMmU2Y2E5
fetchmail: IMAP< A0002 OK LOGIN completed
fetchmail: selecting or re-polling default folder
fetchmail: IMAP> A0003 SELECT "INBOX"
fetchmail: IMAP< * 0 EXISTS
fetchmail: IMAP< * 0 RECENT
fetchmail: IMAP< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft \Forwarded)
fetchmail: IMAP< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen 
\Draft \Forwarded)] Permanent flags
fetchmail: IMAP< * OK [UIDVALIDITY 1116049368] UIDVALIDITY value
fetchmail: IMAP< * OK [UIDNEXT 3216] may be the next UID value
fetchmail: IMAP< A0003 OK [READ-WRITE] SELECT completed
fetchmail: 0 messages waiting after first poll
fetchmail: IMAP> A0004 IDLE
fetchmail: IMAP< + will notify you of all changes


 > so far, so good: mailbox is empty.
 > sending first mail to this account now!


fetchmail: IMAP< * 1 EXISTS
fetchmail: IMAP> DONE
fetchmail: IMAP< A0004 OK IDLE completed
fetchmail: 1 message waiting after re-poll
fetchmail: IMAP> A0005 SEARCH UNSEEN NOT DELETED
fetchmail: IMAP< * SEARCH 1
fetchmail: 1 is unseen
fetchmail: IMAP< A0005 OK SEARCH completed
fetchmail: 1 is first unseen
fetchmail: 1 message for ls...@vi... at imap.strato.de.
fetchmail: IMAP> A0006 FETCH 1 RFC822.SIZE
fetchmail: IMAP< * 1 FETCH (RFC822.SIZE 906)
fetchmail: IMAP< A0006 OK Fetch complete
fetchmail: IMAP> A0007 FETCH 1 RFC822.HEADER
fetchmail: IMAP< * 1 FETCH (RFC822.HEADER {898}
fetchmail: reading message ls...@vi...@imap.rzone.de:1 of 1 (898 header 
octets)
fetchmail: About to rewrite From: Lars Tewes <ls...@gm...>
Rewritten version is From: Lars Tewes <ls...@gm...>

fetchmail: About to rewrite To: Lars Tewes <ls...@vi...>
Rewritten version is To: Lars Tewes <ls...@vi...>

fetchmail: about to deliver with: /usr/bin/procmail -d 'lsr'
fetchmail: IMAP< )
fetchmail: IMAP< A0007 OK Fetch complete
fetchmail: IMAP> A0008 FETCH 1 BODY.PEEK[TEXT]
fetchmail: IMAP< * 1 FETCH (BODY[TEXT] {8}
fetchmail:  (8 body octets) fetchmail: IMAP< )
fetchmail: IMAP< A0008 OK Fetch complete
fetchmail:  flushed
fetchmail: IMAP> A0009 STORE 1 +FLAGS (\Seen \Deleted)
fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen \Recent))
fetchmail: IMAP< A0009 OK STORE completed
fetchmail: IMAP> A0010 EXPUNGE
fetchmail: IMAP< * 1 EXPUNGE
fetchmail: IMAP< A0010 OK EXPUNGE completed
fetchmail: selecting or re-polling default folder
fetchmail: IMAP> A0011 IDLE
fetchmail: IMAP< + will notify you of all changes


 > well done, fetchmail gets notified, stops "imap idle" and sends "A0005 
SEARCH UNSEEN NOT DELETED", then downloads the message and negotiates idle again
 > sending second mail to this account now...


fetchmail: IMAP< * 1 EXISTS
fetchmail: IMAP> DONE
fetchmail: IMAP< A0011 OK IDLE completed
fetchmail: IMAP> A0012 IDLE
fetchmail: IMAP< + will notify you of all changes


 > hmmm, idle is interuppted by fetchmail as before, but instead of polling 
the new message, it just sends anonther "idle".
 > sending third mail to this account now...

fetchmail: IMAP< * 2 EXISTS
fetchmail: IMAP> DONE
fetchmail: IMAP< A0012 OK IDLE completed
fetchmail: IMAP> A0013 IDLE
fetchmail: IMAP< + will notify you of all changes

etc. etc.

---------------------------------------------------------------------

Who's to blame? My config, a stupid server or a bug in fetchmail?

---------------------------------------------------------------------
.fetchmailrc:

poll imap.strato.de
         protocol imap
         username "ls...@vi..."
         password "xxxxxxxx"
         mda "/usr/bin/procmail -d %s"
         idle
---------------------------------------------------------------------

lsr@lsr:# fetchmail -V
This is fetchmail release 6.3.4+RPA+NTLM+SDPS+SSL+NLS.

Copyright (C) 2002, 2003 Eric S. Raymond
Copyright (C) 2004 Matthias Andree, Eric S. Raymond, Rob F. Funk, Graham Wilson
Copyright (C) 2005-2006 Matthias Andree, Sunil Shetye
Fetchmail comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. For details,
please see the file COPYING in the source or documentation directory.

Fallback MDA: (none)
Linux lsr 2.6.15.6 #4 PREEMPT Mon Apr 24 10:54:39 CEST 2006 i686 AMD 
Athlon(tm) 64 Processor 3400+ AuthenticAMD GNU/Linux
Taking options from command line and /home/lsr/.fetchmailrc
Idfile is /home/lsr/.fetchids
Fetchmail will show progress dots even in logfiles.
Fetchmail will forward misaddressed multidrop messages to lsr.
Options for retrieving from ls...@vi...@imap.strato.de:
   True name of server is imap.strato.de.
   Protocol is IMAP.
   All available authentication methods will be tried.
   Server nonresponse timeout is 300 seconds (default).
   Default mailbox selected.
   Only new messages will be retrieved (--all off).
   Fetched messages will not be kept on the server (--keep off).
   Old messages will not be flushed before message retrieval (--flush off).
   Oversized messages will not be flushed before message retrieval 
(--limitflush off).
   Rewrite of server-local addresses is enabled (--norewrite off).
   Carriage-return stripping is enabled (stripcr on).
   Carriage-return forcing is disabled (forcecr off).
   Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
   MIME decoding is disabled (mimedecode off).
   Idle after poll is enabled (idle on).
   Nonempty Status lines will be kept (dropstatus off)
   Delivered-To lines will be kept (dropdelivered off)
   Fetch message size limit is 100 (--fetchsizelimit 100).
   Do binary search of UIDs during 3 out of 4 polls (--fastuidl 4).
   Messages will be delivered with "/usr/bin/procmail -d %s".
   Single-drop mode: 1 local name recognized.
   No UIDs saved from this host.

---------------------------------------------------------------------

Regards
Lars

Re: [fetchmail-users] Fetchmail not fetching after boot on FC5

[Catalin R. <cat...@gm...>]

Found the culprit: SELinux. When disabled, the problem disappears. I
should have guessed it...

Cata

On 5/9/06, Rob MacGregor <rob...@gm...> wrote:
> On 5/9/06, Catalin Rotaru <cat...@gm...> wrote:
> >
> > Tried. Sadly, that didn't fix it. I will try starting from the source
> > and compiling. Is there any way to view and compare the "boot" and
> > "regular" environments and libraries?
>
> Not easily.  My usual trick is to stick something like "env >
> /tmp/fetchmail-boot-env" in the startup script before fetchmail is
> called.  The other option is to change the start line to include "-x"
> to see what it does (caution, makes the boot screen messy - you'll
> need to have some way of scrolling the console).
>
> --
>                 Please keep list traffic on the list.
>
> Rob MacGregor
>      Whoever fights monsters should see to it that in the process he
>        doesn't become a monster.                  Friedrich Nietzsche
> _______________________________________________
> fetchmail-users mailing list
> fet...@li...
> http://lists.berlios.de/mailman/listinfo/fetchmail-users
>


--
Catalin Rotaru
http://www.rotaru.com

Re: [fetchmail-users] Fetchmail not fetching after boot on FC5

[Rob M. <rob...@gm...>]

On 5/9/06, Catalin Rotaru <cat...@gm...> wrote:
>
> Tried. Sadly, that didn't fix it. I will try starting from the source
> and compiling. Is there any way to view and compare the "boot" and
> "regular" environments and libraries?

Not easily.  My usual trick is to stick something like "env >
/tmp/fetchmail-boot-env" in the startup script before fetchmail is
called.  The other option is to change the start line to include "-x"
to see what it does (caution, makes the boot screen messy - you'll
need to have some way of scrolling the console).

--
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] Mails lost due to --flush option ?

[Frederic M. <fre...@wo...>]

Matthias Andree wrote:
> Frederic Marchal <fre...@wo...> writes:
>
>   
>> Hello,
>>     
> Even if the code behaves better than the documentation suggests,
> describing the "--flush" option in a scary way looks correct to me.
>   
Ok, I can understand that.

> Does this patch help your problem (without your adding --flush, that is)?
>   
I tested it on the test server and it worked as expected. Now, it is in 
production but I may have to wait until next Monday for the flood of 
mails from the week-end to know if it is ok (and hope the connection 
drops :-) ).

BTW, isn't it another way of doing the flush you said was a scary thing ?

> It is against fetchmail 6.3.4 but may also work against 6.3.2:
>   
It does. Thank you.

Frederic

Re: [fetchmail-users] Fetchmail not fetching after boot on FC5

[Catalin R. <cat...@gm...>]

On 5/8/06, Rob MacGregor <rob...@gm...> wrote:
> On 5/8/06, Catalin Rotaru <cat...@gm...> wrote:
> >
> > Oops, sorry about that. I just hit "Reply" without checking. My bad. I
> > notice now that the list doesn't have a "Reply-To" header. Maybe one
> > could be added? I am pretty sure MailMan supports it. Just an idea.
>
> This has been discussed before a number of times.  There are as many
> arguments for as against.  It seems to be generally least harmful to
> not set the Reply-to header - or at least setting it annoys the
> developers :-)

Got it.

> > Aaah, I was afraid you'd say that. I am no Linux (and especially
> > Fedora) expert and my preferred way of installing software is "yum
> > install". I am afraid of breaking even more stuff I don't understand
> > if I try to install from source on my own.
>
> Try re-installing via yum then.  I've a suspicion your problem is
> related to library version mismatches.  It could be that once you've
> logged in interactively enough environment is set so that it finds the
> right libraries.
>

Tried. Sadly, that didn't fix it. I will try starting from the source
and compiling. Is there any way to view and compare the "boot" and
"regular" environments and libraries?

> > Any pointers to where I can learn more about
> > installing/removing/reinstalling fetchmail on Fedora? With all
> > RedHat's idiosyncrasies and stuff?
>
> For that you'll have to look to the RH/Fedora lists/forums.  I'll have
> to admit to having only very limited experience of using RPMs etc.
>

Done - but now answers yet...

Cata

Re: [fetchmail-users] Fetchmail not fetching after boot on FC5

[Rob M. <rob...@gm...>]

On 5/8/06, Catalin Rotaru <cat...@gm...> wrote:
>
> Oops, sorry about that. I just hit "Reply" without checking. My bad. I
> notice now that the list doesn't have a "Reply-To" header. Maybe one
> could be added? I am pretty sure MailMan supports it. Just an idea.

This has been discussed before a number of times.  There are as many
arguments for as against.  It seems to be generally least harmful to
not set the Reply-to header - or at least setting it annoys the
developers :-)

> Aaah, I was afraid you'd say that. I am no Linux (and especially
> Fedora) expert and my preferred way of installing software is "yum
> install". I am afraid of breaking even more stuff I don't understand
> if I try to install from source on my own.

Try re-installing via yum then.  I've a suspicion your problem is
related to library version mismatches.  It could be that once you've
logged in interactively enough environment is set so that it finds the
right libraries.

> Any pointers to where I can learn more about
> installing/removing/reinstalling fetchmail on Fedora? With all
> RedHat's idiosyncrasies and stuff?

For that you'll have to look to the RH/Fedora lists/forums.  I'll have
to admit to having only very limited experience of using RPMs etc.

--
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] Fetchamil is very slow

[Rob M. <rob...@gm...>]

On 5/8/06, Shukla Vipul <vip...@gm...> wrote:
> HI,
>
>      I am using fetchmail for downloading mails from server, but fetchmail
> is very slow i got my mail after 12hours i want to get it fast.

There are only 2 things that can cause fetchmail to run slow (assuming
you're not trying to run it on a 386 or older):

1) Slow network link
2) Slow MTA for delivery

What we really need to see are:

1) Contents of .fetchmailrc (masking passwords etc)
2) Output of "fetchmail -v -v --nosyslog"
3) Contents of your mail log (probably /var/log/maillog or similar)
for the same run as (2)

To help you identify where your problem lies.

--
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche