fetchmail-users

[Fetchmail-users] fetchmail expunge error

[Leslie R. <les...@si...>]

Hello everyone,

     I have been using fetchmail for many years with no issues, across 
several ISPs.  A few days ago, however one of my users accounts quit 
working.  The issue, actually, was her client.  I had to delete and 
re-add her account in the client.  In the process, her password was 
changed, and then fetchmail could no longer download her mail.  A day 
later, my own account also quit working.  I called the ISP (AT&T) and 
they informed me users could no longer access mail from a client using 
their password. Instead, the user has to create a secure keyword.  I was 
also informed the old server (inbound.att.net) is no longer active. 
Instead, users must download from imap.mail.att.net.  The tech claimed 
POP3 downloads would still work, but I was not able to get fetchmail to 
login to the server with the POP3 protocol.  I would be interested to 
know if anyone is still able to login to the new server using the POP3 
protocol.

     In any case, I can now login using the IMAP protocol, but fetchmail 
can only download a single message at a time, producing a mail expunge 
mismatch error (0 actual != 1 expected) and exiting after each message.  
Eventually, it does manage to get all the messages, but typically only 
after a very long time, whenever several messages come in over a short 
time frame.  While not quite critical, this is definitely not good.

     Is anyone else having this issue with AT&T?  How can I fix it?


fetchmailrc:

set no syslog
set logfile "/var/log/fetchmail.log"
set postmaster "postmaster"
set nobouncemail
set no spambounce
set properties ""
set daemon 60

poll imap.mail.att.net via imap.mail.att.net
     with proto IMAP service 993

user 'xx...@at...' there with password 'xxxxxxxxxx' is 'xxxxxxxx' 
here options fetchall forcecr stripcr ssl
  mda "/usr/bin/procmail -f %F -d %T"


user 'yyy...@at...' there with password 'yyyyyyyyyy' is 'yyyyyyyy' 
here options fetchall forcecr stripcr stripcr ssl
  mda "/usr/bin/procmail -f %F -d %T"

Re: [Fetchmail-users] fetchmail expunge error

[Gene H. <ghe...@sh...>]

On Sunday 25 April 2021 13:24:06 Leslie Rhorer wrote:

> Hello everyone,
>
>      I have been using fetchmail for many years with no issues, across
> several ISPs.  A few days ago, however one of my users accounts quit
> working.  The issue, actually, was her client.  I had to delete and
> re-add her account in the client.  In the process, her password was
> changed, and then fetchmail could no longer download her mail.  A day
> later, my own account also quit working.  I called the ISP (AT&T) and
> they informed me users could no longer access mail from a client using
> their password. Instead, the user has to create a secure keyword.  I
> was also informed the old server (inbound.att.net) is no longer
> active. Instead, users must download from imap.mail.att.net.  The tech
> claimed POP3 downloads would still work, but I was not able to get
> fetchmail to login to the server with the POP3 protocol.  I would be
> interested to know if anyone is still able to login to the new server
> using the POP3 protocol.
>
>      In any case, I can now login using the IMAP protocol, but
> fetchmail can only download a single message at a time, producing a
> mail expunge mismatch error (0 actual != 1 expected) and exiting after
> each message. Eventually, it does manage to get all the messages, but
> typically only after a very long time, whenever several messages come
> in over a short time frame.  While not quite critical, this is
> definitely not good.
>
And ts not anything I have ever had a problem with.
From my .fetchmailrc:
=====================
defaults mda "/usr/bin/procmail -d me "
set postmaster "me"
set no bouncemail
set no spambounce
set daemon 120

poll	imap.shentel.net with proto imap
	user #######@shentel.net with password ########## is ######
	sslcertpath /etc/ssl/certs
	fetchall
	ssl
	pass8bits
	nokeep
=====================
My ISP, the local cable tv folks, who also supply my telephone but no tv 
as I have an off-air antenna, is actually running its mail server on a 
linux box, using dovecot.

And it Just Works.  Leaving nothing in my inbox at their server every 2 
minutes. 

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Re: [Fetchmail-users] fetchmail expunge error

[Leslie R. <les...@si...>]

On 4/25/2021 1:32 PM, Gene Heskett wrote:
> And ts not anything I have ever had a problem with. 


     Well, me either, until a few days ago.

> My ISP, the local cable tv folks, who also supply my telephone but no tv
     I suspect this is limited to AT&T and their new server, or at least 
mosty.
> as I have an off-air antenna, is actually running its mail server on a
> linux box, using dovecot.
     I am also running dovecot as my IMAP server, both it and fetchmail 
on a Linux box.

Re: [Fetchmail-users] fetchmail expunge error

[Gerard E. S. <ger...@ou...>]

On Sun, 25 Apr 2021 12:24:06 -0500, Leslie Rhorer stated:
>     In any case, I can now login using the IMAP protocol, but
> fetchmail 
>can only download a single message at a time, producing a mail expunge 
>mismatch error (0 actual != 1 expected) and exiting after each
>message. Eventually, it does manage to get all the messages, but
>typically only after a very long time, whenever several messages come
>in over a short time frame.  While not quite critical, this is
>definitely not good.
>
>Is anyone else having this issue with AT&T?  How can I fix it?

Both YAHOO and AOL exhibit the same problem. Most MUAs simply ignore the
problem. Fetchmail has taken a more stringent approach.

-- 
Gerard

Re: [Fetchmail-users] fetchmail expunge error

[Leslie R. <les...@si...>]

On 4/25/2021 2:43 PM, Gerard E. Seibert wrote:
> On Sun, 25 Apr 2021 12:24:06 -0500, Leslie Rhorer stated:
>>      In any case, I can now login using the IMAP protocol, but
>> fetchmail
>> can only download a single message at a time, producing a mail expunge
>> mismatch error (0 actual != 1 expected) and exiting after each
>> message. Eventually, it does manage to get all the messages, but
>> typically only after a very long time, whenever several messages come
>> in over a short time frame.  While not quite critical, this is
>> definitely not good.
>>
>> Is anyone else having this issue with AT&T?  How can I fix it?
> Both YAHOO and AOL exhibit the same problem. Most MUAs simply ignore the
> problem. Fetchmail has taken a more stringent approach.

     Yahoo and AT&T are one and the same, and apparently have been for 
quite a while.  Both the old and the new DNS entries redirect to 
addresses on yahoo.com.   The question is, "How do I get fetchmail to 
ignore or work around the issue?"  Regardless of how obtuse the ISP's 
IMAP server may be, I should be able to download messages without any 
issue.  Downloading one message every N seconds is not an answer.

     I am also getting errors like:

fetchmail: Received BYE response from IMAP server:  IMAP4REV1 SERVER 
LOGGING OUTfetchmail: mailbox selection failed
fetchmail: client/server synchronization error while fetching from 
yy...@at...@imap.mail.att.net
fetchmail: Query status=7 (ERROR)

Re: [Fetchmail-users] fetchmail expunge error

[Hans C. <fo...@gm...>]

On Sun, 25 Apr 2021, Gerard E. Seibert wrote:

> On Sun, 25 Apr 2021 12:24:06 -0500, Leslie Rhorer stated:
>
>>     In any case, I can now login using the IMAP protocol, but fetchmail 
>> can only download a single message at a time, producing a mail expunge 
>> mismatch error (0 actual != 1 expected) and exiting after each message. 
>> Eventually, it does manage to get all the messages, but typically only 
>> after a very long time, whenever several messages come in over a short 
>> time frame.  While not quite critical, this is definitely not good.
>>
>> Is anyone else having this issue with AT&T?  How can I fix it?
>
> Both YAHOO and AOL exhibit the same problem. Most MUAs simply ignore the
> problem. Fetchmail has taken a more stringent approach.

Yup... same problem here with YAHOO.  I asked the same question about a 
year ago and Matthias said this is due to a broken server.  See thread 
here:

   https://sourceforge.net/p/fetchmail/mailman/fetchmail-users/thread/alpine.LFD.2.02.2005041620590.1519%40xippy/#msg37003167

Unfortunately, getting someone like YAHOO to fix their server is next to 
impossible.  Calling "support" results in clueless (fictionalized) 
exchanges like the following:

   Me: <DIAL>...<LONG WAIT ON HOLD>

   Support: YAHOO tech support. How can I help?

   Me: It looks like the YAHOO mail server isn't behaving correctly to the IMAP STORE command.

   Support: <PAUSE> Have you tried to reboot Windows?

   Me: I'm not using Windows, I'm using Linux.

   Support: Linux?  I don't know what that is.  We only support Windows.

   Me: It shouldn't matter, the problem is with the mail server.

   Support: <LONG PAUSE> Have you tried to reboot Windows?

   Me: <SIGH> Never Mind.

   Support: Is there anything else I can help you with?

   Me: Nope. Thanks. <CLICK>

To be clear... I don't recall if I actually called YAHOO support last year 
or not.  I may have.  But based on previous calls to support, I suspect 
this was the type of exchange I would have had.  It might be possible to 
eventually get to someone with enough knowledge to at least understand the 
problem, but getting them to fix it would probably be a different story.

Actually, I wonder if it might be the case that YAHOO (et al) is unwilling 
to fix their servers because so many (major) MUAs depend on the current 
broken behavior.  I have no idea if that's the true or not, just 
speculating.

Re: [Fetchmail-users] fetchmail expunge error

[Alan D. S. <sal...@at...>]

Hi Leslie,

On 2021-04-25 12:24:06, Leslie Rhorer <les...@si...> spake thus:
[...]
>I called the ISP (AT&T) and
>they informed me users could no longer access mail from a client using
>their password. Instead, the user has to create a secure keyword.

For other AT&T/Yahoo customers, the specific terminology used by the company
for those new passwords (in case you need to google it) is "secure mail
keys". They seem to have been rolling them out for a while; I needed to make
the change (I think) in fall 2020, but the effort seems to have started at
least as far back as 2019:

   • "Use OAuth or secure mail key for email apps"
     https://www.att.com/support/article/email-support/KM1240462

   • "Create a secure mail key"
     https://www.att.com/support/article/email-support/KM1240308


>I was
>also informed the old server (inbound.att.net) is no longer active.

I think the person who provided that information was misinformed. The
'inbound.att.net' address is still active -- I am successfully using it with
POP3. Also, it is still listed in the AT&T online help articles. E.g.,

   • "Get AT&T email server settings"
     https://www.att.com/support/article/email-support/KM1010523

   • "Set Up or Update AT&T Email - Apple Mail (OS X)"
     https://www.att.com/support/article/email-support/KM1010489

There are relatively frequent availability issues with 'inbound.att.net', but
that is a long-standing problem unrelated to the password changes, AFAICT.


>Instead, users must download from imap.mail.att.net.  The tech claimed
>POP3 downloads would still work, but I was not able to get fetchmail to
>login to the server with the POP3 protocol.  I would be interested to
>know if anyone is still able to login to the new server using the POP3
>protocol.
[...]

Yes, the below fetchmail config line is what I use to pull mail from AT&T via
POP3, but I am doing it from 'inbound.att.net':

     poll inbound.att.net port 995 protocol pop3 timeout 120 uidl username "my-...@at..." ssl sslcertck password "MY-SECURE-MAIL-KEY-VALUE" expunge 10 fetchall

For other AT&T users that have (or had) a working POP3/inbound.att.net
configuration: Note that when switching from legacy passwords to "secure mail
keys", the only value that needed to change was the password. You have to use
their clunky webapp to create the "secure mail key", and then use that
generated value as the password in your fetchmail config.

HTH,
-Al

-- 
a l a n   d.   s a l e w s k i
ads@salewski.email
sal...@at...
https://github.com/salewski

Re: [Fetchmail-users] fetchmail expunge error

[Leslie R. <les...@si...>]

On 4/25/2021 9:15 PM, Alan D. Salewski wrote:
> Hi Leslie,
>
> On 2021-04-25 12:24:06, Leslie Rhorer <les...@si...> 
> spake thus:
> [...]
>
>
>> I was
>> also informed the old server (inbound.att.net) is no longer active.
>
> I think the person who provided that information was misinformed. The
> 'inbound.att.net' address is still active -- I am successfully using 
> it with
> POP3. Also, it is still listed in the AT&T online help articles. E.g.,

     Ha!  That's what I get for believing a corporate tech support 
weenie.  I should have known better.  I made the rather stupid mistake 
of believing him, and made the change without really checking.  Pfft!  I 
went back to the old server (with the new secure mail key, of course), 
and it all seems to be working.

> Yes, the below fetchmail config line is what I use to pull mail from 
> AT&T via
> POP3, but I am doing it from 'inbound.att.net':
>
>     poll inbound.att.net port 995 protocol pop3 timeout 120 uidl 
> username "my-...@at..." ssl sslcertck password 
> "MY-SECURE-MAIL-KEY-VALUE" expunge 10 fetchall
>
     Yep, you bet.  Thanks!  I am not even using the "expunge" verb, and 
it still seems to work so far.

Re: [Fetchmail-users] fetchmail expunge error

[Matthias A. <mat...@gm...>]

Am 26.04.21 um 14:46 schrieb Leslie Rhorer:
>
> On 4/25/2021 9:15 PM, Alan D. Salewski wrote:
>> Hi Leslie,
>>
>> On 2021-04-25 12:24:06, Leslie Rhorer <les...@si...>
>> spake thus:
>> [...]
>>
>>
>>> I was
>>> also informed the old server (inbound.att.net) is no longer active.
>>
>> I think the person who provided that information was misinformed. The
>> 'inbound.att.net' address is still active -- I am successfully using
>> it with
>> POP3. Also, it is still listed in the AT&T online help articles. E.g.,
>
>     Ha!  That's what I get for believing a corporate tech support
> weenie.  I should have known better.  I made the rather stupid mistake
> of believing him, and made the change without really checking.  Pfft! 
> I went back to the old server (with the new secure mail key, of
> course), and it all seems to be working.
>
>> Yes, the below fetchmail config line is what I use to pull mail from
>> AT&T via
>> POP3, but I am doing it from 'inbound.att.net':
>>
>>     poll inbound.att.net port 995 protocol pop3 timeout 120 uidl
>> username "my-...@at..." ssl sslcertck password
>> "MY-SECURE-MAIL-KEY-VALUE" expunge 10 fetchall
>>
>     Yep, you bet.  Thanks!  I am not even using the "expunge" verb,
> and it still seems to work so far.

Just to shed some light on this expunging:

* POP3 and IMAP separate "mark for deletion" and actual erasing of
deleted messages.

For POP3, it's simplistic, the server must remember all the "DELE 1234"
"DELE 1235" etc. These are only to execute when the client (fetchmail
for instance) sends "QUIT". Some servers are known to malfunction and
also execute stored DELE on loss of connection, which is a violation of
the protocol.

For IMAP4rev.1, the client can set \Deleted flags on mesages, and these
take effect by "EXPUNGE" or "CLOSE" commands. Fetchmail only uses
EXPUNGE, unless you either only check for mail, or set the --keep option.


And WRT a similar issue, also see
<https://sourceforge.net/p/fetchmail/mailman/fetchmail-users/thread/alpine.LFD.2.02.2005041620590.1519%40xippy/#msg37004069>
- not sure how much of that holds today, it's nearly a year old.

Re: [Fetchmail-users] fetchmail expunge error

[Alan D. S. <sal...@at...>]

On 2021-04-26 19:31:23, Matthias Andree <mat...@gm...> spake thus:
>Am 26.04.21 um 14:46 schrieb Leslie Rhorer:
>>
>> On 4/25/2021 9:15 PM, Alan D. Salewski wrote:
[...]

>>> Yes, the below fetchmail config line is what I use to pull mail from
>>> AT&T via
>>> POP3, but I am doing it from 'inbound.att.net':
>>>
>>>     poll inbound.att.net port 995 protocol pop3 timeout 120 uidl
>>> username "my-...@at..." ssl sslcertck password
>>> "MY-SECURE-MAIL-KEY-VALUE" expunge 10 fetchall
>>>
>>     Yep, you bet.  Thanks!  I am not even using the "expunge" verb,
>> and it still seems to work so far.
>
>Just to shed some light on this expunging:

[ All good stuff related to protocol-level DELE/QUIT, EXPUNGE/CLOSE. ]

Since we're talking about it, though, I'll note that in the particular config
above, the 'expunge 10' clause is primarily serving as a throttle for message
delivery to the local MTA. It is a minor abuse of the 'expunge' "end the
session" behavior documented in fetchmail(1) for POP2 and POP3:

        -e <count> | --expunge <count>
               (Keyword: expunge)
               Arrange  for  deletions to be made final after a given number of
               messages.  Under POP2 or POP3, fetchmail cannot  make  deletions
               final  without  sending QUIT and ending the session -- with this
               option on, fetchmail will break a long  mail  retrieval  session
               into multiple sub-sessions, sending QUIT after each sub-session.
               ...

I say "minor abuse" because the feature is documented as a tool to help cope
with flakey connections to the maildrop server, which is not the intent of my
use.

The local MTA happens to be configured to only attempt immediate delivery of
up to ten messages from a given SMTP session (the rest get queued for later
delivery whenever the mail queue delivery runner wakes up to do its
thing). The effect of using the 'expunge 10' clause is that fetchmail will
only hand off ten messages at a time to the local MTA, so all will enjoy
immediate delivery (since fetchmail only attempts delivery within the ten
message limit).  However, a (hypothetical) runaway process on the machine
attempting to send out a gazillion email messages all at once would land (
gazillion minus ten ) messages in the mail queue (hopefully with enough time
for somebody to notice and purge them before attempting delivery either
locally or remote).

-Al

-- 
a l a n   d.   s a l e w s k i
ads@salewski.email
sal...@at...
https://github.com/salewski