Menu
▾
▴
Re: [Fetchmail-users] IMAP confusion on unexpected EXPUNGE
|
From: Matthias A. <mat...@gm...> - 2026-06-25 20:24:36
|
Am 25.06.26 um 02:35 schrieb Hans Carlson via Fetchmail-users: > Matthias, > > Could you explain this fix a bit more... or rather could you explain > how this differs from the previous behavior. I ask because the > behavior I see with fetchmail-6.5.7 (latest available on Fedora 43) > seems to be basically the same except for the message. Hans, sure. The crucial change is that fetchmail 6.6.6 will now abort on *ANY* EXPUNGE that's received unless we're in an active "EXPUNGE" request -- the check to count EXPUNGE replies versus the "\Deleted" tags we dole out in the same session remains in place, but happens too late - including on Yahoo, depending on EXPUNGE interval. Let's look at examples. Example #1 Assume you have messages A, B, C. Message numbers (not UIDs) in IMAP are dynamic in a session. The server will always number them 1 2 3 if they are three messages. Assume we have a longer EXPUNGE interval and only expunge every 10 messages because we're on a slow line and want fewer round trips. Satellite link, mobile link in the outskirts, your imagination defines why. mailbox 1A 2B 3C fetch and delete "1". This obtains message A and marks it deleted. So far, so good. Yahoo expunges without our telling them to, and tells us "* 1 EXPUNGE". Meaning message 1 expunged, and messages 2...N now renumbered to 1...N-1. Fetchmail 6.6.5 and older DISREGARDED this * 1 EXPUNGE. <- Bug! 6.6.6 will see the * 1 EXPUNGE and abort the session because something outside our control messed with the mailbox and renumbered messages. Our internal state no longer matches the mailbox state. (Footnote [FN1]). mailbox now has 1B 2C. If fetchmail itself expunges after downloading that message 1A, we've trapped the bug because we request an. fetchmail now downloads message "2" and marks it deleted. But instead of retrieving "B" it obtains C. OK. Yahoo expunges without our telling them to. mailbox now has 1B. fetchmail attempts downloading message "3" and the server complains about a client bug. (Yes, it's right, that's fetchmail's fault.) That's your use case. If we delete messages right away we're skipping every other message and at the end of the session the server complains. Not pretty, but not harmful. Example #2 Other situation, we have messages A, B, C, D, E. Assume C is marked seen. Read in the web interface, read with another client, whatever. mailbox 1A, 2B, 3C (Seen), 4D, 5E. Assume that while fetchmail is downloading 1A, and before that completes, another client or the web interface deletes 1A and expunges it, or it gets auto-expunged through the server. Doesn't matter. Mailbox now is 1B, 2C (Seen), 3D, 4E. fetchmail 6.6.5 and older ignore the * 1 EXPUNGE that arrives here. Now, what might happen is this: In that moment, fetchmail marks message #1 seen and deleted in the assumption it will mark message A. Instead it marks B for deletion. The server auto-expunges. Oops. We've just destroyed message B! The fix in fetchmail 6.6.6 is that in this situation there is an EXPUNGE that we didn't request, so we drop the receiver on the hook immediately, so we don't accidentally lose B. Mailbox now is 1C (Seen), 2D, 3E. Remainder more or less as above. The actual bug is that fetchmail wasn't (and still isn't) counting EXPUNGE replies properly except when we EXPUNGE ourselves because we don't have robust code in place to match our internal state to the mailbox by looking at EXISTS, EXPUNGE, RECENT and thereabouts. I chose not to fix that because I don't believe this would be bullet-proof, ever. I've seen too many servers doing strange and possibly nonconforming things that wouldn't dare take that risk. The proper fix will be accessing messages by UID instead of message numbers, and UIDVALIDITY: the UID numbers must be stable in a session whatever happens. If they change from one session to the next, the server must change the UIDVALIDITY. UIDVALIDITY and UID combined must be globally stable forever. No other message B is ever allowed to have the same UIDVALIDITY:UID combination than another message A. You delete A expunge, new message B arrives, new UID. The other advantage of that future UID change (it's not in 6.6.6 nor any 7.0.0-alpha yet) will be that we need not rely on a \Seen flag that can be changed outside fetchmail. Fetchmail will reliably know "I've had message 61782312:4543" irrespective of any flags. EXPUNGE won't renumber the UIDs so we just need to track if a message we want to download gets expunged so we know to skip it. > With 6.5.7 if I fetch email from yahoo I see this: > > 3 messages for XXXXX at XXXXX. > reading message XXXXX:1 of 3 (6568 header octets) (8 body octets) > flushed > mail expunge mismatch (0 actual != 1 expected) > client/server synchronization error while fetching from XXXXX > Query status=7 (ERROR) > > and then the connection closes. > > With 6.6.6 I see this: > > 3 messages for XXXXX at XXXXX. > reading message XXXXX:1 of 3 (6568 header octets) (8 body octets) > flushed > Unexpected EXPUNGE response from IMAP server: "* 1 EXPUNGE". > fetchmail must abort the session because message numbers are > desynchronized now. > client/server protocol error while fetching from XXXXX > Query status=4 (PROTOCOL) > > and then the connection closes. > > It's definitely a better message, but other than that the behavior > seems to be the same (to me). I'm guessing the unexpected EXPUNGE is > caught at a different stage during the connection? Yes - might be SEARCH, STORE, FETCH, whatever in addition to the EXPUNGE we check now. > > I've lived with this behavior from Yahoo because it's very low volume > (1-3/day if that) and if it takes a bit longer to get the 1-2 other > messages that's fine. But the warning below suggests the wrong message > could be deleted? Not if fetchmail really is the only client accessing the mailbox at the same time. But I won't guarantee what happens if another client access the mailbox as in Example #2, hence my alarming wording. > > In my case, there's only one instance of fetchmail that accesses this > account and it's the only thing that accesses it, so maybe that > minimizes/eliminates the chance of an incorrect deletion? Likely. If you don't fetch from multiple mailboxes, switch to POP3. For Yahoo that means you change from imap.mail.yahoo.com to pop.mail.yahoo.com and add "protocol POP3" somewhere between the "poll pop.mail.yahoo.com" and the "user". HTH. Happy fetching, Matthias > > > On Wed, 24 Jun 2026, Matthias Andree via Fetchmail-users wrote: > >> The 6.6.6 critical bug fix release of fetchmail is now available at the >> usual locations, including >> <https://downloads.sourceforge.net/project/fetchmail/branch_6.6/>. >> >> People who cannot update but use IMAP should set "keep" or use "--keep" >> on the command line immediately to avoid accidentally deleting the wrong >> message in case your upstream IMAP server sees multiple clients on the >> same mailbox OR expunge automatically when a \Deleted flag is STOREd on >> a message. A proper fix for this would be too large and/or risky and/or >> introduce breaking changes, so is unsuitable for 6.6.x. >> >> The source archive is available at: >> <https://downloads.sourceforge.net/project/fetchmail/branch_6.6/fetchmail-6.6.6.tar.xz/download> >> >> >> The detached GnuPG signature is available at: >> <https://downloads.sourceforge.net/project/fetchmail/branch_6.6/fetchmail-6.6.6.tar.xz.asc/download> >> >> >> The SHA256 hashes for the tarballs are: >> SHA2-256(fetchmail-6.6.6.tar.xz)= >> da99f8c573c4d9e63f493c7e24447126aea25b53b4c076ec79266874e29b1975 >> >> >> Here are the release notes: >> -------------------------------------------------------------------------------- >> >> fetchmail-6.6.6 (released 2026-06-24, 32443 LoC): >> >> ## CRITICAL BUGFIX FOR IMAP: >> * The IMAP client, which has always used message indexes for the >> selected >> mailbox, did not abort when receiving an EXPUNGE response - which >> changes >> message numbers inside the mailbox. Unlike UIDs, the message >> numbers are >> not stable and fetchmail does not have internal interfaces to track >> which >> messages are deleted, and adding those to a 6.6.X release would be too >> risky, and switching to UID is also too big a change, so we have no >> choice but to abort the session when seeing an EXPUNGE response without >> our own EXPUNGE request, to avoid marking the wrong message as >> seen/deleted >> or skip the wrong one, or assume the wrong message size. >> Earl Chew reported this versus Yahoo Mail via Gitlab Work Item #91, >> which >> automatically expunges messages that are marked with the \Deleted flag. >> >> This has one new message for which we do not have translations yet, >> it is urgent to get the fix in the field. >> >> ------------------------------------------------------------------------------- >> >> > > > _______________________________________________ > Fetchmail-users mailing list > Fet...@li... > https://lists.sourceforge.net/lists/listinfo/fetchmail-users |
