Re: [fetchmail-devel] Fwd: fetchmail downloading with message number as a parameter

[Matthias A. <mat...@gm...>]

I am sending a copy to fetchmail-users because this is of general
interest, and also a recurring issue with Google Mail.

Am 15.11.2013 16:02, schrieb Joe M:
> Hello Matthias,
> 
> I am missing emails. Is there anyway to correlate the id's from
> .fetchids with the email in the gmail inbox?
> 
> I tried both pop and imap and in both instances, a few emails were not
> downloaded.
> 
> I tried with --uidl and no --startnum.
> 
>>> As an update, I just polled my pop3.live.com accounts and figured that
>>> --uidl works for me on that account.  Note that using --keep without
>>> --uidl is a dangerous endeavor, meaning that if you use LAST or a
>>> locally recorded message number, this setup is prone to mail loss or
>>> skipping, should messages before the designated start number disappear
>>> or should a downloaded message fail to be forwarded.
>>
>> This seems to have happened and I did not get your email.
> 
> Any thoughts, please?

Sorry for the late response, somehow missed your followup.

Up front: gmail is quirky by design.
  Google did not model traditional mailboxes for POP3 or IMAP, but
invented their own stuff around archiving and folders, and to add insult
to injury, some of their help pages defame POP3 as unreliable when the
fault lies with implementations, not the protocol.  POP3 + UIDL can be
made reliable - if you don't abuse it, that is.

Still, I link to Google documentation (apparently the first two links
now redirect to the same contents):
<http://www.fetchmail.info/fetchmail-FAQ.html#I9> has a few links to
Google documentation - check the "recent:" trick described there and see
if that helps. <https://support.google.com/mail/answer/47948?hl=en> in
particular explains that.


The correlation between UIDs and messages is technically possible but
yields only the POP3 message numbers for the .fetchids stored - and
message numbers can change with every login (especially as messages are
removed, or sometimes added, example below), and are therefore not
really useful in your situation.

If you still want to see them, all it takes is running fetchmail with
-vv added on the command line. This logs the transcript of the POP3
exchange, which might look like this (this is from a Dovecot POP3
server, not from Gmail's POP3 interface):

> fetchmail: POP3> USER joe
> fetchmail: POP3< +OK
> fetchmail: POP3> PASS *
> fetchmail: POP3< +OK Logged in.
> fetchmail: POP3> STAT
> fetchmail: POP3< +OK 1314 77621905
> fetchmail: POP3> UIDL
> fetchmail: POP3< +OK
> fetchmail: POP3< 1 000000014eb1c89a
> fetchmail: POP3< 2 000000034eb1c89a
> fetchmail: POP3< 3 000000054eb1c89a
> fetchmail: POP3< 4 000000064eb1c89a
> fetchmail: POP3< 5 000000074eb1c89a
...

That's all there is.  You get a temporary message number (which is a
count that may occasionally skip ahead over deleted messages) and the
UID that you would also find in .fetchids after successful fetch + shipping.

Deleting message 2 would then remove the 000000034eb1c89a, and on the
next login you'd see the output below, so for 000000054eb1c89a the
message number changed from 3 to 2.

> fetchmail: POP3< 1 000000014eb1c89a
> fetchmail: POP3< 2 000000054eb1c89a
> fetchmail: POP3< 3 000000064eb1c89a
> fetchmail: POP3< 4 000000074eb1c89a

As said above, this is not very useful to hunt down missing messages.

The best bet, given Google's documentation, would probably be to try
reconfiguring your user name for Google's recent: mode, and run
fetchmail with a --fetchall --keep --uidl once.

That may cause duplicate message retrieval due to --fetchall, but better
duplicate than none.