fetchmail-users

[fetchmail-users] fetchmail continuously fetching already fetched imap email

[Lee H. <mat...@gm...>]

Hi fetchmail-users,
I'm having some trouble getting fetchmail not to keep fetching the
same IMAP messages over and over and over again. Here is what I have
in my .fetchmailrc:

poll corpusmx10c.corp.emc.com
with proto IMAP
user '**********'
there with password '**********'
is '*********' here
mda "/usr/bin/procmail -d %T"
options keep

Is there an option that I am setting incorrectly that makes it ignore
the messages that have already been downloaded? Is "keep" the right
option? (I don't want it to delete the messages, because I'd like to
read the messages from a different computer also).

Re: [fetchmail-users] fetchmail continuously fetching already fetched imap email

[Rob M. <rob...@gm...>]

On 9/10/07, Lee Hinman <mat...@gm...> wrote:
> Hi fetchmail-users,
> I'm having some trouble getting fetchmail not to keep fetching the
> same IMAP messages over and over and over again. Here is what I have
> in my .fetchmailrc:
>
> poll corpusmx10c.corp.emc.com
> with proto IMAP
> user '**********'
> there with password '**********'
> is '*********' here
> mda "/usr/bin/procmail -d %T"
> options keep
>
> Is there an option that I am setting incorrectly that makes it ignore
> the messages that have already been downloaded? Is "keep" the right
> option? (I don't want it to delete the messages, because I'd like to
> read the messages from a different computer also).

Version of fetchmail?

Transcript as specified in the FAQ
(http://www.fetchmail.info/fetchmail-FAQ.html#G3 - "fetchmail
--nosyslog --nodetach -vvv", along with your usual command line
options).

Also very useful would be the output from adding "--configdump" to
your usual command line options.

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] fetchmail continuously fetching already fetched imap email

[Lee H. <mat...@gm...>]

On 9/10/07, Rob MacGregor <rob...@gm...> wrote:
> On 9/10/07, Lee Hinman <mat...@gm...> wrote:
> > Hi fetchmail-users,
> > I'm having some trouble getting fetchmail not to keep fetching the
> > same IMAP messages over and over and over again. Here is what I have
> > in my .fetchmailrc:
> >
> > poll corpusmx10c.corp.emc.com
> > with proto IMAP
> > user '**********'
> > there with password '**********'
> > is '*********' here
> > mda "/usr/bin/procmail -d %T"
> > options keep
> >
> > Is there an option that I am setting incorrectly that makes it ignore
> > the messages that have already been downloaded? Is "keep" the right
> > option? (I don't want it to delete the messages, because I'd like to
> > read the messages from a different computer also).
>
> Version of fetchmail?
>
> Transcript as specified in the FAQ
> (http://www.fetchmail.info/fetchmail-FAQ.html#G3 - "fetchmail
> --nosyslog --nodetach -vvv", along with your usual command line
> options).
>
> Also very useful would be the output from adding "--configdump" to
> your usual command line options.
>
> --
>                  Please keep list traffic on the list.
>
> Rob MacGregor
>       Whoever fights monsters should see to it that in the process he
>         doesn't become a monster.                  Friedrich Nietzsche
> _______________________________________________
> fetchmail-users mailing list
> fet...@li...
> https://lists.berlios.de/mailman/listinfo/fetchmail-users
>

Here in my configdump:

[hinmanm@Euclid] $ fetchmail --nosyslog --nodetach -vvv --configdump
                                                         ~
TRUE=1; FALSE=0

os_type = 'generic'
feature_options = ('pop3','imap','gssapi','kerberos','etrn','odmr','ssl',)
# Start of configuration initializer
fetchmailrc = {
    'poll_interval':0,
    "logfile":None,
    "idfile":"/Users/hinmanm/.fetchids",
    "postmaster":"hinmanm",
    'bouncemail':TRUE,
    'spambounce':FALSE,
    "properties":None,
    'invisible':FALSE,
    'showdots':TRUE,
    'syslog':FALSE,
    # List of server entries begins here
    'servers': [
     # Entry for site `corpusmx10c.corp.emc.com' begins:
     {
         "pollname":"corpusmx10c.corp.emc.com",
         'active':TRUE,
         "via":None,
         "protocol":"IMAP",
         "service":None,
         'timeout':300,
         'interval':0,
         "envelope":"Received",
         'envskip':0,
         "qvirtual":None,
         "auth":"any",
         'dns':TRUE,
         'uidl':FALSE,
         "aka":[],
         "localdomains":[],
         "plugin":None,
         "plugout":None,
         "principal":None,
         'tracepolls':FALSE,
         'users': [
        {
            "remote":"*********",
            "password":"*********",
            'localnames':["hinmanm"],
            'fetchall':FALSE,
            'keep':TRUE,
            'flush':FALSE,
            'limitflush':FALSE,
            'rewrite':TRUE,
            'stripcr':TRUE,
            'forcecr':FALSE,
            'pass8bits':FALSE,
            'dropstatus':FALSE,
            'dropdelivered':FALSE,
            'mimedecode':FALSE,
            'idle':FALSE,
            "mda":"/usr/bin/procmail -d %T",
            "bsmtp":None,
            'lmtp':FALSE,
            "preconnect":None,
            "postconnect":None,
            'limit':0,
            'warnings':3600,
            'fetchlimit':0,
            'fetchsizelimit':100,
            'fastuidl':4,
            'batchlimit':0,
            'ssl':FALSE,
            "sslkey":None,
            "sslcert":None,
            "sslproto":None,
            'sslcertck':FALSE,
            "sslcertpath":None,
            "sslfingerprint":None,
            'expunge':0,
            "properties":None,
            "smtphunt":["localhost"],
            "fetchdomains":[],
            "smtpaddress":None,
            "smtpname":None,
            'antispam':'',
            "mailboxes":[],
        }
        ,           ]
     }
    ]
}
# End of initializer


And the output of fetchmail --nosyslog --nodetach -vvv:
fetchmail: 6.3.8 querying corpusmx10c.corp.emc.com (protocol IMAP) at
Mon, 10 Sep 2007 13:49:50 -0600 (MDT): poll started
Trying to connect to 128.221.14.108/143...connected.
fetchmail: IMAP< * OK Microsoft Exchange Server 2003 IMAP4rev1 server
version 6.5.7638.1 (CORPUSMX10C.corp.emc.com) ready.
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 IDLE LOGIN-REFERRALS
MAILBOX-REFERRALS NAMESPACE LITERAL+ UIDPLUS CHILDREN AUTH=NTLM
fetchmail: IMAP< A0001 OK CAPABILITY completed.
fetchmail: Protocol identified as IMAP4 rev 1
fetchmail: corpusmx10c.corp.emc.com: opportunistic upgrade to TLS
failed, trying to continue
fetchmail: IMAP> A0002 NOOP
fetchmail: IMAP< A0002 OK NOOP completed.
fetchmail: IMAP> A0003 LOGIN "*********" *
fetchmail: IMAP< A0003 OK LOGIN completed.
fetchmail: selecting or re-polling default folder
fetchmail: IMAP> A0004 SELECT "INBOX"
fetchmail: IMAP< * 299 EXISTS
fetchmail: IMAP< * 0 RECENT
fetchmail: IMAP< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
fetchmail: IMAP< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged
\Deleted \Draft $MDNSent)] Permanent flags
fetchmail: IMAP< * OK [UIDVALIDITY 3890] UIDVALIDITY value
fetchmail: IMAP< A0004 OK [READ-WRITE] SELECT completed.
fetchmail: 299 messages waiting after first poll
fetchmail: IMAP> A0005 SEARCH UNSEEN NOT DELETED
fetchmail: IMAP< * SEARCH
fetchmail: IMAP< A0005 OK SEARCH completed.
299 messages (299 seen) for corp\hinmam at corpusmx10c.corp.emc.com.
skipping message corp\hi...@co...:1 not flushed
skipping message corp\hi...@co...:2 not flushed
... skipped because there are lots of messages ...
skipping message corp\hi...@co...:298 not flushed
skipping message corp\hi...@co...:299 not flushed
fetchmail: IMAP> A0006 LOGOUT
fetchmail: IMAP< * BYE Microsoft Exchange Server 2003 IMAP4rev1 server
version 6.5.7638.1 signing off.
fetchmail: IMAP< A0006 OK LOGOUT completed.
fetchmail: 6.3.8 querying corpusmx10c.corp.emc.com (protocol IMAP) at
Mon, 10 Sep 2007 13:49:52 -0600 (MDT): poll completed
fetchmail: not swapping UID lists, no UIDs seen this query
fetchmail: Query status=1 (NOMAIL)
fetchmail: Deleting fetchids file.
fetchmail: normal termination, status 1
fetchmail: Deleting fetchids file.


And fetchmail is normally called like so:
fetchmail -a $EXTRAARG >/dev/null 2>&1
where $EXTRAARG can be "-v" or not

And the version of fetchmail from fetchmail -V:
This is fetchmail release 6.3.8+GSS+SSL+KRB4+KRB5.

Copyright (C) 2002, 2003 Eric S. Raymond
Copyright (C) 2004 Matthias Andree, Eric S. Raymond, Robert M. Funk,
Graham Wilson
Copyright (C) 2005-2006 Sunil Shetye
Copyright (C) 2005-2007 Matthias Andree
Fetchmail comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. For details,
please see the file COPYING in the source or documentation directory.

Fallback MDA: (none)
Darwin Euclid.local 8.10.1 Darwin Kernel Version 8.10.1: Wed May 23
16:33:00 PDT 2007; root:xnu-792.22.5~1/RELEASE_I386 i386 i386
Taking options from command line and /Users/hinmanm/.fetchmailrc
Idfile is /Users/hinmanm/.fetchids
Fetchmail will show progress dots even in logfiles.
Fetchmail will forward misaddressed multidrop messages to hinmanm.
Options for retrieving from *********@corpusmx10c.corp.emc.com:
  True name of server is corpusmx10c.corp.emc.com.
  Protocol is IMAP.
  All available authentication methods will be tried.
  Server nonresponse timeout is 300 seconds (default).
  Default mailbox selected.
  Only new messages will be retrieved (--all off).
  Fetched messages will be kept on the server (--keep on).
  Old messages will not be flushed before message retrieval (--flush off).
  Oversized messages will not be flushed before message retrieval
(--limitflush off).
  Rewrite of server-local addresses is enabled (--norewrite off).
  Carriage-return stripping is enabled (stripcr on).
  Carriage-return forcing is disabled (forcecr off).
  Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
  MIME decoding is disabled (mimedecode off).
  Idle after poll is disabled (idle off).
  Nonempty Status lines will be kept (dropstatus off)
  Delivered-To lines will be kept (dropdelivered off)
  Fetch message size limit is 100 (--fetchsizelimit 100).
  Do binary search of UIDs during 3 out of 4 polls (--fastuidl 4).
  Messages will be delivered with "/usr/bin/procmail -d %T".
  Single-drop mode: 1 local name recognized.
  No UIDs saved from this host.

I believe this is the information, is there anything else that would
be helpful to you?

Re: [fetchmail-users] fetchmail continuously fetching already fetched imap email

[Rob M. <rob...@gm...>]

On 9/10/07, Lee Hinman <mat...@gm...> wrote:
>
> And the output of fetchmail --nosyslog --nodetach -vvv:
> fetchmail: 6.3.8 querying corpusmx10c.corp.emc.com (protocol IMAP) at
> Mon, 10 Sep 2007 13:49:50 -0600 (MDT): poll started
> Trying to connect to 128.221.14.108/143...connected.
> fetchmail: IMAP< * OK Microsoft Exchange Server 2003 IMAP4rev1 server
> version 6.5.7638.1 (CORPUSMX10C.corp.emc.com) ready.
> fetchmail: IMAP> A0001 CAPABILITY
> fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 IDLE LOGIN-REFERRALS
> MAILBOX-REFERRALS NAMESPACE LITERAL+ UIDPLUS CHILDREN AUTH=NTLM
> fetchmail: IMAP< A0001 OK CAPABILITY completed.
> fetchmail: Protocol identified as IMAP4 rev 1
> fetchmail: corpusmx10c.corp.emc.com: opportunistic upgrade to TLS
> failed, trying to continue
> fetchmail: IMAP> A0002 NOOP
> fetchmail: IMAP< A0002 OK NOOP completed.
> fetchmail: IMAP> A0003 LOGIN "*********" *
> fetchmail: IMAP< A0003 OK LOGIN completed.
> fetchmail: selecting or re-polling default folder
> fetchmail: IMAP> A0004 SELECT "INBOX"
> fetchmail: IMAP< * 299 EXISTS
> fetchmail: IMAP< * 0 RECENT
> fetchmail: IMAP< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
> fetchmail: IMAP< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged
> \Deleted \Draft $MDNSent)] Permanent flags
> fetchmail: IMAP< * OK [UIDVALIDITY 3890] UIDVALIDITY value
> fetchmail: IMAP< A0004 OK [READ-WRITE] SELECT completed.
> fetchmail: 299 messages waiting after first poll
> fetchmail: IMAP> A0005 SEARCH UNSEEN NOT DELETED
> fetchmail: IMAP< * SEARCH
> fetchmail: IMAP< A0005 OK SEARCH completed.
> 299 messages (299 seen) for corp\hinmam at corpusmx10c.corp.emc.com.
> skipping message corp\hi...@co...:1 not flushed
> skipping message corp\hi...@co...:2 not flushed
> ... skipped because there are lots of messages ...
> skipping message corp\hi...@co...:298 not flushed
> skipping message corp\hi...@co...:299 not flushed
> fetchmail: IMAP> A0006 LOGOUT
> fetchmail: IMAP< * BYE Microsoft Exchange Server 2003 IMAP4rev1 server
> version 6.5.7638.1 signing off.
> fetchmail: IMAP< A0006 OK LOGOUT completed.
> fetchmail: 6.3.8 querying corpusmx10c.corp.emc.com (protocol IMAP) at
> Mon, 10 Sep 2007 13:49:52 -0600 (MDT): poll completed
> fetchmail: not swapping UID lists, no UIDs seen this query
> fetchmail: Query status=1 (NOMAIL)
> fetchmail: Deleting fetchids file.
> fetchmail: normal termination, status 1
> fetchmail: Deleting fetchids file.

I don't see the behaviour you describe in your initial email.  Can you
confirm that the problem (of the same message being downloaded at
every run) is ongoing.

Can you also double check that it really is the same message (that is,
it has the same Message-ID and all the other headers are also
identical).

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] fetchmail continuously fetching already fetched imap email

[Rob M. <rob...@gm...>]

Keep it on the list please.

On 9/10/07, Lee Hinman <mat...@gm...> wrote:
>
> It looks like the Message-IDs are the same for both emails

I'd tend to agree.

> Every time
> I run "fetchmail -a" again, it downloads all of the messages again
> (over 300 unfortunately).

You may want to read the man page and see what the "-a" option means:

| Retrieve both old (seen) and new messages
| from the mailserver

Which may just explain the "problem" you're having ;)

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] fetchmail continuously fetching already fetched imap email

[Lee H. <mat...@gm...>]

On 9/10/07, Rob MacGregor <rob...@gm...> wrote:
> Keep it on the list please.
>
> On 9/10/07, Lee Hinman <mat...@gm...> wrote:
> >
> > It looks like the Message-IDs are the same for both emails
>
> I'd tend to agree.
>
> > Every time
> > I run "fetchmail -a" again, it downloads all of the messages again
> > (over 300 unfortunately).
>
> You may want to read the man page and see what the "-a" option means:
>
> | Retrieve both old (seen) and new messages
> | from the mailserver
>
> Which may just explain the "problem" you're having ;)
>
> --
>                  Please keep list traffic on the list.
>
> Rob MacGregor
>       Whoever fights monsters should see to it that in the process he
>         doesn't become a monster.                  Friedrich Nietzsche
> _______________________________________________
> fetchmail-users mailing list
> fet...@li...
> https://lists.berlios.de/mailman/listinfo/fetchmail-users
>

Sorry, accidently hit reply instead of replying to the list.

I see that the -a option is why it is downloading all the old
messages, however, if I get an email on a different machine (a Windows
machine) and I read it, fetchmail doesn't download it for some reason.
It only downloads the "unseen" messages. What I would like is to
download any message that *this* computer hasn't seen, even the ones
I've already read on my Windows machines.

Is there a way to accomplish this?

Thank you very much for the help, very appreciated!

- Lee

Re: [fetchmail-users] fetchmail continuously fetching already fetched imap email

[Rob M. <rob...@gm...>]

On 9/10/07, Lee Hinman <mat...@gm...> wrote:
>
> I see that the -a option is why it is downloading all the old
> messages, however, if I get an email on a different machine (a Windows
> machine) and I read it, fetchmail doesn't download it for some reason.

Ah, now you're getting into territory fetchmail is not designed to handle.

> It only downloads the "unseen" messages. What I would like is to
> download any message that *this* computer hasn't seen, even the ones
> I've already read on my Windows machines.
>
> Is there a way to accomplish this?

No, because right now there is no client side tracking of IMAP emails
(only POP).  If your provider supports POP (with UIDL) then you can
use that as your solution, but there isn't any way to achieve this
with IMAP.

Alternatively, use the fact that you're downloading the emails with
fetchmail and run a local IMAP server, pointing your Windows client at
it.

-- 
                 Please keep list traffic on the list.

Rob MacGregor
      Whoever fights monsters should see to it that in the process he
        doesn't become a monster.                  Friedrich Nietzsche

Re: [fetchmail-users] fetchmail continuously fetching already fetched imap email

[Matthias A. <mat...@gm...>]

Lee Hinman schrieb am 2007-09-10:

> I see that the -a option is why it is downloading all the old
> messages, however, if I get an email on a different machine (a Windows
> machine) and I read it, fetchmail doesn't download it for some reason.

That reason is unfortunately a design flaw, in that the former
maintainer believed it were sufficient to track "I've seen this message"
on the server side. This is based on the assumption that fetchmail were
the only software accessing your mailbox, evidently untrue in your case,
and in many more cases, too. You need client-side tracking, which
fetchmail cannot currently do for IMAP. It's on the TODO list for the
next major release though, but resources are very limited.

> It only downloads the "unseen" messages. What I would like is to
> download any message that *this* computer hasn't seen, even the ones
> I've already read on my Windows machines.

This is not yet implemented in fetchmail unfortunately. The patches that
are available are also not sufficient in that they are agnostic of the
"UIDVALIDITY".

If you're just fetching messages from the INBOX, you may have more luck
with POP3 + UIDL (the latter is important and not the default
unfortunately).

> Is there a way to accomplish this?

Not in fetchmail. Sorry for that.

HTH

Best regards,

-- 
Matthias Andree

Re: [fetchmail-users] fetchmail continuously fetching already fetched imap email

[Matthew L. H. <mat...@gm...>]

* Matthias Andree <mat...@gm...> [2007-09-10 22:51:28 +0200]:

>Lee Hinman schrieb am 2007-09-10:
>
>> I see that the -a option is why it is downloading all the old
>> messages, however, if I get an email on a different machine (a Windows
>> machine) and I read it, fetchmail doesn't download it for some reason.
>
>That reason is unfortunately a design flaw, in that the former
>maintainer believed it were sufficient to track "I've seen this message"
>on the server side. This is based on the assumption that fetchmail were
>the only software accessing your mailbox, evidently untrue in your case,
>and in many more cases, too. You need client-side tracking, which
>fetchmail cannot currently do for IMAP. It's on the TODO list for the
>next major release though, but resources are very limited.
>
>> It only downloads the "unseen" messages. What I would like is to
>> download any message that *this* computer hasn't seen, even the ones
>> I've already read on my Windows machines.
>
>This is not yet implemented in fetchmail unfortunately. The patches that
>are available are also not sufficient in that they are agnostic of the
>"UIDVALIDITY".
>
>If you're just fetching messages from the INBOX, you may have more luck
>with POP3 + UIDL (the latter is important and not the default
>unfortunately).
>
>> Is there a way to accomplish this?
>
>Not in fetchmail. Sorry for that.
>
>HTH
>
>Best regards,
>
>-- 
>Matthias Andree
>_______________________________________________
>fetchmail-users mailing list
>fet...@li...
>https://lists.berlios.de/mailman/listinfo/fetchmail-users

Thank you both for the help, luckily, after some research, I found out
that the exchange server also offers a POP3 method to access the mail,
so I am able to access it through that.

Thanks!
- Lee