fetchmail-users — User questions and discussion

[Fetchmail-users] Procmail filtering question

[Richard K. <ric...@bt...>]

I'm running fetchmail -> procmail -> local directories read by Claws on
Ubuntu 20.04

I'm having some problems filtering email messages by Subject using
procmail

"Normal" text Subjects of the form:
Subject: News on Coronavirus
are no problem. But there are a lot of messages nowadays that are
displayed in Claws as:

Subject: Update from GOV.UK – Coronavirus Statutory Sick Pay ... etc

but where the subject in the message itself is actually in the form:

Subject:
=?UTF-8?Q?Update_from_GOV.UK_=E2=80=93_Coronavirus_Statutory_Sick_P?=
=?UTF-8?Q?ay_Rebate_Scheme:_service_availability_and_issues?= 
(the above is all one line)

Does anyone know if there is some generic way of ensuring that messages
are passed to procmail in the "normal" format so that I may reliably and
straightforwardly construct procmail matching rules (i.e. without
having to inspect the message source each time)?

- Richard.
-- 
Richard Kimber

[Fetchmail-users] ANNOUNCE: fetchmail 6.4.5 is now available (regression fix for realpath for SUSv3/SUSv2 systems)

[Matthias A. <mat...@gm...>]

Greetings,

The 6.4.5 release of fetchmail is now available at the usual locations,
including <https://downloads.sourceforge.net/project/fetchmail/branch_6.4/>.

This fixes a regression on older operating systems that can print this error:
fetchmail: Cannot find absolute path for user's home directory

== NOTE ==
I intend this to be the last 6.4.x version with functional changes, and new
6.4.x are not planned, except if regressions, critical fixes, or new
translations or important documentation fixes appear and 6.5.0 is too far out.

Fetchmail 6.4.x is the last branch that will tolerate OpenSSL 1.0.2 and C89
compilers, and support Python 2.7.x (for newer x) for fetchmailconf.

== EXCURSION ---------
The direction for the near future is to do some bugfixing and possibly minor
features on the Git branch 'legacy_6x', for now called 6.5.0.dev*, it is a
branch that will require newer operating systems, toolchains, library, for
instance, OpenSSL 1.1.1 with TLSv1.3 support, C99 (for long long support) or
possibly C11 (TBD) and possibly newer IEEE Std 1003.1 compliance.
== END EXCURSION -----


The source archive is available in two formats at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.5.tar.xz>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.5.tar.lz>

Detached GnuPG signatures for the respective tarballs are at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.5.tar.lz.asc>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.5.tar.xz.asc>

SHA256 hash values:
SHA256(fetchmail-6.4.5.tar.lz)= 3325f468b821377069cd92089a559aa3a4a240621b01bf470646512d15440d88
SHA256(fetchmail-6.4.5.tar.xz)= d30f06920294490141ddcd4a66583bf16a0c88c1d7b4f58a3b9cef8511946b1c

Here are the release notes:

fetchmail-6.4.5 (released 2020-05-07, 27596 LoC):

## REGRESSION FIX:
* fetchmail 6.4.0 and 6.4.1 changed the resolution of the home directory
  in a way that requires SUSv4 semantics of realpath(), which leads to
  'Cannot find absolute path for... directory' error messages followed by aborts
  on systems where realpath() follows strict SUSv2 semantics and returns
  EINVAL if the 2nd argument is NULL.

  On such systems, for instance, Solaris 10, fetchmail requires PATH_MAX to be
  defined, and will then work again.  Regression reported by David Hough.

  On systems that neither provide auto-allocation semantics for realpath(),
  nor PATH_MAX, fetchmail will print this error and abort. Such systems
  are unsupported, see README.

## CHANGES:
* Add a test program fm_realpath, and a t.realpath script, neither to be
  installed. These will test resolution of the current working directory.

## TRANSLATION UPDATES in reverse alphabetical order of language codes,
## with my thanks to the translators:
* zh_CN: Boyuan Yang [Chinese (simplified)]
* sv:    Göran Uddeborg [Swedish]
* sq:    Besnik Bleta [Albanian]
* pl:    Jakub Bogusz [Polish]
* ja:    Takeshi Hamasaki [Japanese]
* fr:    Frédéric Marchal [French]
* cs:    Petr Pisar [Czech]

# KNOWN BUGS AND WORKAROUNDS
  (This section floats upwards through the NEWS file so it stays with the
  current release information)
* Fetchmail does not handle messages without Message-ID header well
  (See sourceforge.net bug #780933)
* Fetchmail currently uses 31-bit signed integers in several places
  where unsigned and/or wider types should have been used, for instance,
  for mailbox sizes, and misreports sizes of 2 GibiB and beyond. 
  Fixing this requires C89 compatibility to be relinquished.
* BSMTP is mostly untested and errors can cause corrupt output.
* Fetchmail does not track pending deletes across crashes.
* The command line interface is sometimes a bit stubborn, for instance,
  fetchmail -s doesn't work with a daemon running.
* Linux systems may return duplicates of an IP address in some circumstances if
  no or no global IPv6 addresses are configured.
  (No workaround. Ubuntu Bug#582585, Novell Bug#606980.)
* Kerberos 5 may be broken, particularly on Heimdal, and provide bogus error
  messages. This will not be fixed, because the maintainer has no Kerberos 5
  server to test against. Use GSSAPI.

--------------------------------------------------------------------------------

Happy fetches,
Matthias

Re: [Fetchmail-users] mail expunge mismatch

[Matthias A. <mat...@gm...>]

[re-send with fixed quotation markers]

Am 06.05.20 um 20:14 schrieb Hans Carlson:

> I'm using fetchmail-6.4.4.
>
> For about a year now I've been seeing the following in my log file for
> mail coming from yahoo (imap.mail.yahoo.com).  This happens regardless
> of the user.

Hans,

Yahoo's mail server is broken, and your trace proves it. Please report
this to Yahoo (but don't ask my how, I don't know their contacts).
Details below.


> I did a debug session and from what I can tell the problem appears to be
> the response from the imap EXPUNGE command.  It seems to be missing some
> information?  (at least based on the response from another server that
> does work - see below).


The Yahoo IMAP server is expunging prematurely, at a time when it is not
allowed to. This is a server behaviour (malfunction) that is EXPLICITLY
FORBIDDEN by the IMAP4r1 standard (RFC3501) page 73 section 7.4.1
EXPUNGE Response:

An EXPUNGE response MUST NOT be sent when no command is in
progress, nor while responding to a FETCH, STORE, or SEARCH
command. This rule is necessary to prevent a loss of
synchronization of message sequence numbers between client and
server.

>   # This is imap.mail.yahoo.com
>   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
>   ...
>   fetchmail: IMAP> A0009 STORE 1 +FLAGS (\Seen \Deleted)
>   fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen $NotJunk))
>   fetchmail: IMAP< * 1 EXPUNGE
>   fetchmail: IMAP< * 0 EXISTS
>   fetchmail: IMAP< A0009 OK STORE completed
>   fetchmail: IMAP> A0010 EXPUNGE
>   fetchmail: IMAP< A0010 OK EXPUNGE completed
>   fetchmail: mail expunge mismatch (0 actual != 1 expected)
>   fetchmail: IMAP> A0011 LOGOUT
>   fetchmail: IMAP< * BYE IMAP4rev1 Server logging out
>   fetchmail: IMAP< A0011 OK LOGOUT completed
>
> If I try an account on a different server that functions correctly
> (imap.gmx.com), the response from EXPUNGE shows what I assume
> fetchmail is expecting to see.  BUT comparing the yahoo response to
> the gmx response it appears yahoo may be sending that information as
> part of the STORE response?
>
>   # This is imap.gmx.com
>   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
>   ...
>   fetchmail: IMAP> A0008 STORE 1 +FLAGS (\Seen \Deleted)
>   fetchmail: IMAP< * 1 FETCH (FLAGS (\Recent \Deleted \Seen))
>   fetchmail: IMAP< A0008 OK STORE completed
>   fetchmail: IMAP> A0009 EXPUNGE
>   fetchmail: IMAP< * 1 EXPUNGE
>   fetchmail: IMAP< * 0 EXISTS
>   fetchmail: IMAP< * 0 RECENT
>   fetchmail: IMAP< A0009 OK EXPUNGE completed
>   fetchmail: selecting or re-polling default folder
>   fetchmail: 0 messages waiting after re-poll
>   fetchmail: IMAP> A0010 LOGOUT
>   fetchmail: IMAP< * BYE Server logging out
>   fetchmail: IMAP< A0010 OK LOGOUT completed
>
> So... is there any way around this issue?

I will not add new code that works around MUST-NOT violations of a server.

You may get away with avoiding to mark messages deleted (--keep), but
then you cannot use --all.

HTH

Regards,
Matthias

Re: [Fetchmail-users] mail expunge mismatch

[Matthias A. <mat...@gm...>]

Am 07.05.20 um 19:54 schrieb Matthias Andree:
> Am 06.05.20 um 20:14 schrieb Hans Carlson:
Sorry - this message has botched citation levels. I'll resend.

Re: [Fetchmail-users] mail expunge mismatch

[Matthias A. <mat...@gm...>]

Am 06.05.20 um 20:14 schrieb Hans Carlson:
> I'm using fetchmail-6.4.4.
>
> For about a year now I've been seeing the following in my log file for
> mail coming from yahoo (imap.mail.yahoo.com).  This happens regardless
> of the user.

Hans,

Yahoo's mail server is broken, and your trace proves it. Please report
this to Yahoo (but don't ask my how, I don't know their contacts).
Details below.


I did a debug session and from what I can tell the problem appears to be
the response from the imap EXPUNGE command.  It seems to be missing some
information?  (at least based on the response from another server that
does work - see below).


The Yahoo IMAP server is expunging prematurely, at a time when it is not
allowed to. This is a server behaviour (malfunction) that is EXPLICITLY
FORBIDDEN by the IMAP4r1 standard (RFC3501) page 73 section 7.4.1
EXPUNGE Response:

      An EXPUNGE response MUST NOT be sent when no command is in
      progress, nor while responding to a FETCH, STORE, or SEARCH
      command.  This rule is necessary to prevent a loss of
      synchronization of message sequence numbers between client and
      server. 


>   # This is imap.mail.yahoo.com
>   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
>   ...
>   fetchmail: IMAP> A0009 STORE 1 +FLAGS (\Seen \Deleted)
>   fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen $NotJunk))
>   fetchmail: IMAP< * 1 EXPUNGE
>   fetchmail: IMAP< * 0 EXISTS
>   fetchmail: IMAP< A0009 OK STORE completed
>   fetchmail: IMAP> A0010 EXPUNGE
>   fetchmail: IMAP< A0010 OK EXPUNGE completed
>   fetchmail: mail expunge mismatch (0 actual != 1 expected)
>   fetchmail: IMAP> A0011 LOGOUT
>   fetchmail: IMAP< * BYE IMAP4rev1 Server logging out
>   fetchmail: IMAP< A0011 OK LOGOUT completed
>
> If I try an account on a different server that functions correctly
> (imap.gmx.com), the response from EXPUNGE shows what I assume
> fetchmail is expecting to see.  BUT comparing the yahoo response to
> the gmx response it appears yahoo may be sending that information as
> part of the STORE response?
>
>   # This is imap.gmx.com
>   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
>   ...
>   fetchmail: IMAP> A0008 STORE 1 +FLAGS (\Seen \Deleted)
>   fetchmail: IMAP< * 1 FETCH (FLAGS (\Recent \Deleted \Seen))
>   fetchmail: IMAP< A0008 OK STORE completed
>   fetchmail: IMAP> A0009 EXPUNGE
>   fetchmail: IMAP< * 1 EXPUNGE
>   fetchmail: IMAP< * 0 EXISTS
>   fetchmail: IMAP< * 0 RECENT
>   fetchmail: IMAP< A0009 OK EXPUNGE completed
>   fetchmail: selecting or re-polling default folder
>   fetchmail: 0 messages waiting after re-poll
>   fetchmail: IMAP> A0010 LOGOUT
>   fetchmail: IMAP< * BYE Server logging out
>   fetchmail: IMAP< A0010 OK LOGOUT completed
>
> So... is there any way around this issue? 

I will not add new code that works around MUST-NOT violations of a server.

You may get away with avoiding to mark messages deleted (--keep), but
then you cannot use --all.

HTH

Regards,
Matthias

[Fetchmail-users] mail expunge mismatch

[Hans C. <fo...@gm...>]

I'm using fetchmail-6.4.4.

For about a year now I've been seeing the following in my log file for 
mail coming from yahoo (imap.mail.yahoo.com).  This happens regardless of 
the user.

   fetchmail: 1 message for USER at SERVERNAME.
   fetchmail: reading message US...@in...:1 of 1 (3733 header octets) (29948 body octets) flushed
   fetchmail: mail expunge mismatch (0 actual != 1 expected)
   fetchmail: client/server synchronization error while fetching from USER@SERVERNAME
   fetchmail: Query status=7 (ERROR)

The main problem with this, is the connection is dropped each time it 
happens and it happens for every message.  That means if there are 10 
messages waiting, fetchmail needs to poll this server 10 times before all 
the messages are retrieved.

Here's my config entry for this user:

   poll SERVERNAME via imap.mail.yahoo.com protocol IMAP service 993
     user 'REMOTE_USER' there password 'PASSWORD' is 'LOCAL_USER' here;

I did a debug session and from what I can tell the problem appears to be 
the response from the imap EXPUNGE command.  It seems to be missing some 
information?  (at least based on the response from another server that 
does work - see below).

   # This is imap.mail.yahoo.com
   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
   ...
   fetchmail: IMAP> A0009 STORE 1 +FLAGS (\Seen \Deleted)
   fetchmail: IMAP< * 1 FETCH (FLAGS (\Deleted \Seen $NotJunk))
   fetchmail: IMAP< * 1 EXPUNGE
   fetchmail: IMAP< * 0 EXISTS
   fetchmail: IMAP< A0009 OK STORE completed
   fetchmail: IMAP> A0010 EXPUNGE
   fetchmail: IMAP< A0010 OK EXPUNGE completed
   fetchmail: mail expunge mismatch (0 actual != 1 expected)
   fetchmail: IMAP> A0011 LOGOUT
   fetchmail: IMAP< * BYE IMAP4rev1 Server logging out
   fetchmail: IMAP< A0011 OK LOGOUT completed

If I try an account on a different server that functions correctly 
(imap.gmx.com), the response from EXPUNGE shows what I assume fetchmail is 
expecting to see.  BUT comparing the yahoo response to the gmx response it 
appears yahoo may be sending that information as part of the STORE 
response?

   # This is imap.gmx.com
   $ fetchmail -vvv -d0 --nodetach --nosyslog SERVERNAME
   ...
   fetchmail: IMAP> A0008 STORE 1 +FLAGS (\Seen \Deleted)
   fetchmail: IMAP< * 1 FETCH (FLAGS (\Recent \Deleted \Seen))
   fetchmail: IMAP< A0008 OK STORE completed
   fetchmail: IMAP> A0009 EXPUNGE
   fetchmail: IMAP< * 1 EXPUNGE
   fetchmail: IMAP< * 0 EXISTS
   fetchmail: IMAP< * 0 RECENT
   fetchmail: IMAP< A0009 OK EXPUNGE completed
   fetchmail: selecting or re-polling default folder
   fetchmail: 0 messages waiting after re-poll
   fetchmail: IMAP> A0010 LOGOUT
   fetchmail: IMAP< * BYE Server logging out
   fetchmail: IMAP< A0010 OK LOGOUT completed

So... is there any way around this issue?

[Fetchmail-users] The 6.4.5-rc2 release candidate of fetchmail is available

[Matthias A. <mat...@gm...>]

Greetings,

The 6.4.5-rc2 release of fetchmail is now available at the usual locations,
including <https://downloads.sourceforge.net/project/fetchmail/branch_6.4/>.

It has been mailed out to the translation project so as to collect the
one new translation, and possibly updated translations for other languages.

The source archive is available at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4//fetchmail-6.4.5-rc2.tar.xz>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4//fetchmail-6.4.5-rc2.tar.lz>

Here are the release notes:

fetchmail-6.4.5 (unreleased):

## REGRESSION FIX:
* fetchmail 6.4.0 and 6.4.1 changed the resolution of the home directory
  in a way that requires SUSv4 semantics of realpath(), which leads to
  'Cannot find absolute path for... directory' error messages followed by aborts
  on systems where realpath() follows strict SUSv2 semantics and returns
  EINVAL if the 2nd argument is NULL.

  On such systems, for instance, Solaris 10, fetchmail requires PATH_MAX to be
  defined, and will then work again.  Regression reported by David Hough.

  On systems that neither provide auto-allocation semantics for realpath(),
  nor PATH_MAX, fetchmail will print this error and abort. Such systems
  are unsupported, see README.

## CHANGES:
* Add a test program fm_realpath, and a t.realpath script, neither to be
  installed. These will test resolution of the current working directory.

Happy fetches,
Matthias

[Fetchmail-users] ANNOUNCE: fetchmail 6.4.4 is now available (translation update for Japanese)

[Matthias A. <mat...@gm...>]

Greetings,

The 6.4.4 release of fetchmail is now available at the usual locations,
including <https://downloads.sourceforge.net/project/fetchmail/branch_6.4/>.

NOTE:
I intend this to be the last 6.4.x version with functional changes, and new
6.4.x are not planned, except if regressions, critical fixes, or new
translations or important documentation fixes appear and 6.5.0 is too far out.

Fetchmail 6.4.x is the last branch that will tolerate OpenSSL 1.0.2 and C89 compilers,
and support Python 2.7.x (for newer x) for fetchmailconf.

== EXCURSION ---------
The direction for the near future is to do some bugfixing and possibly minor
features on the Git branch 'legacy_6x', for now called 6.5.0.dev*, it is a
branch that will require newer operating systems, toolchains, library, for
instance, OpenSSL 1.1.1 with TLSv1.3 support, C99 (for long long support) or
possibly C11 (TBD) and possibly newer IEEE Std 1003.1 compliance.
== END EXCURSION -----

The source archive is available at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4//fetchmail-6.4.4.tar.xz>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4//fetchmail-6.4.4.tar.lz>

Detached GnuPG signatures are at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.4.tar.lz.asc>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.4.tar.xz.asc>

Digests (obtained from openssl dgst -sha256):
SHA256(fetchmail-6.4.4.tar.lz)= 06bead00aacec86b44edcbcdaf3937f667f7962003ab38784563fb9704d6b4bc
SHA256(fetchmail-6.4.4.tar.xz)= 511b60daabf7543a01de06af07c8772290c6807cd53c42a8504960e978f3abea

Here are the release notes:
--------------------------------------------------------------------------------
fetchmail-6.4.4 (released 2020-04-26, 27530 LoC):

## UPDATED TRANSLATION - WITH THANKS TO THE TRANSLATOR:
* ja:    Takeshi Hamasaki [Japanese]
--------------------------------------------------------------------------------

[Fetchmail-users] ANNOUNCE: fetchmail 6.4.3 is now available (bug fixes)

[Matthias A. <mat...@gm...>]

Greetings,

The 6.4.3 release of fetchmail is now available at the usual locations,
including <https://downloads.sourceforge.net/project/fetchmail/branch_6.4/>.

NOTE:
I intend this to be the last 6.4.x version with functional changes, and new
6.4.x are not planned, except if regressions, critical fixes, or new
translations or important documentation fixes appear and 6.5.0 is too far out.

Fetchmail 6.4.x is the last branch that will tolerate OpenSSL 1.0.2 and C89 compilers,
and support Python 2.7.x (for newer x) for fetchmailconf.

== EXCURSION ---------
The direction for the near future is to do some bugfixing and possibly minor
features on the Git branch 'legacy_6x', for now called 6.5.0.dev*, it is a
branch that will require newer operating systems, toolchains, library, for
instance, OpenSSL 1.1.1 with TLSv1.3 support, C99 (for long long support) or
possibly C11 (TBD) and possibly newer IEEE Std 1003.1 compliance.
== END EXCURSION -----

The source archive is available with the same content but different compression format at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.3.tar.lz>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.3.tar.xz>

Detached GnuPG signatures are at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.3.tar.lz.asc>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.3.tar.xz.asc>

Digests (obtained from openssl dgst -sha256):
SHA256(fetchmail-6.4.3.tar.lz)= b22217beb0a1545a84a111baada2008f00887afe395a8514805e5297366385d0
SHA256(fetchmail-6.4.3.tar.xz)= b0360e14b9aa5d065eef8ff99ad0347ef6cbbfc934c8114908295a402a09d3e4

Here are the release notes:

fetchmail-6.4.3 (released 2020-04-05, 27530 LoC):

## BUGFIXES:
* Plug memory leaks when parts of the configuration (defaults, rcfile, command 
  line) override one another.
* fetchmail terminated the placeholder command string too late and included
  garbage from the heap at the end of the string. Workaround: don't use place-
  holders %h or %p in the --plugin string. Bug added in 6.4.0 when merging
  Gitlab merge request !5 in order to fix an input buffer overrun.
  Faulty commit 418cda65f752e367fa663fd13884a45fcbc39ddd.
  Reported by Stefan Thurner, Gitlab issue #16.
* Fetchmail now checks for errors when trying to read the .idfile,
  Gitlab issue #3.
* Fetchmail's error messages that reports that the defaults entry isn't the 
  first was made more precise. It could be misleading if there was a poll or
  skip statement before the defaults.

## CHANGES:
* Fetchmail documentation was updated to require OpenSSL 1.1.1.
  OpenSSL 1.0.2 reached End Of Life status at the end of the year 2019.
  Fetchmail will tolerate, but warn about, 1.0.2 for now on the assumption that 
  distributors backport security fixes as the need arises.
  Fetchmail will also warn if another SSL library that is API-compatible
  with OpenSSL lacks TLS v1.3 support.
* If the trust anchor is missing, fetchmail refers the user to README.SSL.

## INTERNAL CHANGES:
* The AC_DECLS(getenv) check was removed, its only user was broken and not 
  accounting for that AC_DECLS always defines HAVE_DECL_... to 0 or 1, so
  fetchmail never declared a missing getenv() symbol (it was testing with 
  #ifdef).  Remove the backup declaration. getenv is mandated by SUSv2 anyways.

## UPDATED TRANSLATION - WITH THANKS TO THE TRANSLATORS:
* sq:    Besnik Bleta [Albanian]
* zh_CN: Boyuan Yang [Chinese (simplified)]
* pl:    Jakub Bogusz [Polish]
* cs:    Petr Pisar [Czech]
* fr:    Frédéric Marchal [French]
* sv:    Göran Uddeborg [Swedish]
* eo:    Felipe Castro [Esperanto]

--------------------------------------------------------------------------------

[Fetchmail-users] The 6.4.3-rc2 release candidate #2 of fetchmail is available

[Matthias A. <mat...@gm...>]

Greetings,

The 6.4.3-rc2 release of fetchmail is now available at the usual locations,
including <https://downloads.sourceforge.net/project/fetchmail/branch_6.4/>.

| I found that one bug fix caused double-free errors with asan, and
| the TLS1_3 warning rigging wasn't working properly, and that some
| HAVE_DECL_ checks were broken.
| 
| Please test if you use the plugin option, or if you are using a more complex configuration
| with defaults, or where command line overrides rcfile.

The source archive is available at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4//fetchmail-6.4.3-rc2.tar.xz>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4//fetchmail-6.4.3-rc2.tar.lz>

Here are the release notes:

fetchmail-6.4.3 (WIP)

## BUGFIXES:
* Plug memory leaks when parts of the configuration (defaults, rcfile, command 
  line) override one another.
* fetchmail terminated the placeholder command string too late and included
  garbage from the heap at the end of the string. Workaround: don't use place-
  holders %h or %p in the --plugin string. Bug added in 6.4.0 when merging
  Gitlab merge request !5 in order to fix an input buffer overrun.
  Faulty commit 418cda65f752e367fa663fd13884a45fcbc39ddd.
  Reported by Stefan Thurner, Gitlab issue #16.
* Fetchmail now checks for errors when trying to read the .idfile,
  Gitlab issue #3.
* Fetchmail's error messages that reports that the defaults entry isn't the 
  first was made more precise. It could be misleading if there was a poll or
  skip statement before the defaults.

## CHANGES:
* Fetchmail documentation was updated to require OpenSSL 1.1.1.
  OpenSSL 1.0.2 reached End Of Life status at the end of the year 2019.
  Fetchmail will tolerate, but warn about, 1.0.2 for now on the assumption that 
  distributors backport security fixes as the need arises.
  Fetchmail will also warn if another SSL library that is API-compatible
  with OpenSSL lacks TLS v1.3 support.
* If the trust anchor is missing, fetchmail refers the user to README.SSL.

## INTERNAL CHANGES:
* The AC_DECLS(getenv) check was removed, its only user was broken and not 
  accounting for that AC_DECLS always defines HAVE_DECL_... to 0 or 1, so
  fetchmail never declared a missing getenv() symbol (it was testing with 
  #ifdef).  Remove the backup declaration. getenv is mandated by SUSv2 anyways.

And this is the Git history since -rc1:

* aa38c490 2020-03-30 | Record po for 6.4.3-rc2.
* 6f7a83c0 2020-03-30 | Make fetchmail -V print SSL/TLS library warnings...
* 0e590bf4 2020-03-30 | Fix -SSL/+SSL reporting in fetchmail -V output.
* 43b557d5 2020-03-30 | Fix HAVE_DECL_ users to check value, not definition.
* 66a35bd6 2020-03-30 | Remove broken AC_CHECK_DECLS(getenv).
* c9fb6180 2020-03-30 | Properly report if the defaults entry is not the first.
* 5af21c95 2020-03-30 | Bump version, we'll need -rc2.
* 85e5a019 2020-03-30 | fetchmail.c Avoid double-free in optmerge()'s STRING_MERGE macro.

Happy fetches,
Matthias

[Fetchmail-users] The 6.4.3-rc1 release candidate #1 of fetchmail is available

[Matthias A. <mat...@gm...>]

The 6.4.3-rc1 release of fetchmail is now available at the usual locations,
including <https://downloads.sourceforge.net/project/fetchmail/branch_6.4/>.

The source archive is available at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4//fetchmail-6.4.3-rc1.tar.xz>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4//fetchmail-6.4.3-rc1.tar.lz>

Please test if you use the plugin option, or if you are using a more complex configuration
with defaults, or where command line overrides rcfile.

Here are the release notes:

fetchmail-6.4.3 (WIP)

## BUGFIXES:
* Plug memory leaks when parts of the configuration (defaults, rcfile, command 
  line) override one another.
* fetchmail terminated the placeholder command string too late and included
  garbage from the heap at the end of the string. Workaround: don't use place-
  holders %h or %p in the --plugin string. Bug added in 6.4.0 when merging
  Gitlab merge request !5 in order to fix an input buffer overrun.
  Faulty commit 418cda65f752e367fa663fd13884a45fcbc39ddd.
  Reported by Stefan Thurner, Gitlab issue #16.
* Fetchmail now checks for errors when trying to read the .idfile,
  Gitlab issue #3.

## CHANGES:
* Fetchmail documentation was updated to require OpenSSL 1.1.1.
  OpenSSL 1.0.2 reached End Of Life status at the end of the year 2019.
  Fetchmail will tolerate, but warn about, 1.0.2 for now on the assumption that 
  distributors backport security fixes as the need arises.
  Fetchmail will also warn if another SSL library that is API-compatible
  with OpenSSL lacks TLS v1.3 support.
* If the trust anchor is missing, fetchmail refers the user to README.SSL.

By popular demand, diffs from the previous release have been omitted.

[Fetchmail-users] The 6.4.3-beta1 of fetchmail is available [memory leak and --plugin bugfixes]

[Matthias A. <mat...@gm...>]

The 6.4.3-beta1 release of fetchmail is now available at the usual locations,
including <http://sourceforge.net/projects/fetchmail>.

The source archive is available at:
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.3-beta1.tar.lz>
<https://downloads.sourceforge.net/project/fetchmail/branch_6.4/fetchmail-6.4.2-3-beta1ar.xz>

Please test if you use the plugin option, or if you are using a more complex configuration
with defaults, or where command line overrides rcfile.

## BUGFIXES:
* Plug memory leaks when parts of the configuration (defaults, rcfile, command 
  line) override one another.
* fetchmail terminated the placeholder command string too late and included
  garbage from the heap at the end of the string. Workaround: don't use place-
  holders %h or %p in the --plugin string. Bug added in 6.4.0 when merging
  Gitlab merge request !5 in order to fix an input buffer overrun.
  Faulty commit 418cda65f752e367fa663fd13884a45fcbc39ddd.
  Reported by Stefan Thurner.

Re: [Fetchmail-users] Weird Return-Path processing

[Matthias A. <mat...@gm...>]

>> However you seem to be proving that fetchmail is not unfolding the line
>> properly, and is shipping out this broken line via SMTP (RFC5321), which
>> isn't permitted either, the SMTP Domain can't have blanks or line
>> breaks, so this seems to be a bug. Given that RFC5322 Return-Path:
>> header syntax is more permissive than RFC5321 SMTP MAIL FROM:<...>
>> syntax, fetchmail needs to validate input and act accordingly.
> Ok, so what would be fetchmail action then ? Skipping pushing message
> based on malformed field detection ?

Good question. The simplest reaction would be "leave message on server"
and tell local user about it.
It might also wrap the message in an attachment and replace the return
path, or somehow sanitize the return path, but traditionally we haven't
enforced that fetchmail is configured with a reply-capable sender address.

>>
>> FTR, which fetchmail version are you using?c
> It is 6.3.26 but I checked several 6.3.x versions and they all handle
> this case in similar way (except maybe from really old 6.3.8 which did
> removed the message from the server despite SMTP failing).

6.3.26 is also in the "really old" and "no longer supported" categories,
but 6.4.2 doesn't help in this particular case I think, but you'll
likely need to upgrade soon as providers are going to tighten up their
TLS support levels, and then you'll be out of luck with 6.3.x. See
https://gitlab.com/fetchmail/fetchmail/-/blob/legacy_64/NEWS#L68 for
6.4.2 all the way back to 6.4.0 changes.

Re: [Fetchmail-users] Weird Return-Path processing

[Skle.pik <sp...@ma...>]

On 27/03/2020 11:26, Matthias Andree wrote:
> Am 27.03.20 um 09:08 schrieb Skle.pik:
>> I did check POP3 side with Python poplib and clearly the Return-Path
>> is split across lines - result below:
>>
>> (b'+OK',
>> [b'Return-Path:
>> <noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap',
>> b' 11.bnc.salesforce.com>',
>> b'Delivered-To:
>>
>> Does anyone know who is to blame - is this Return-Path formatting
>> violation on POP side, or maybe it is mailer being intolerant ?
> Looking at your Python trace and assuming it's represented properly
> here, the header isn't legit (per RFC5322) because the CFWS after the
> .ap at the end of the first line terminates the first atom or 1*atext
> and before the next atom/atext there would have to be a dot, so you'd
> need to trace back to the originator (which may or may not have been
> Salesforce) where it broke, or if the originator who pretended to have
> been (and may or may not have been) Salesforce sent garbage already.
Thanks for confirming my suspicion. This is enough to work with email 
provided as I suspect upstream POP server to be guilty here.
> However you seem to be proving that fetchmail is not unfolding the line
> properly, and is shipping out this broken line via SMTP (RFC5321), which
> isn't permitted either, the SMTP Domain can't have blanks or line
> breaks, so this seems to be a bug. Given that RFC5322 Return-Path:
> header syntax is more permissive than RFC5321 SMTP MAIL FROM:<...>
> syntax, fetchmail needs to validate input and act accordingly.
Ok, so what would be fetchmail action then ? Skipping pushing message 
based on malformed field detection ?
>
> FTR, which fetchmail version are you using?c
It is 6.3.26 but I checked several 6.3.x versions and they all handle 
this case in similar way (except maybe from really old 6.3.8 which did 
removed the message from the server despite SMTP failing).
>
>
>
>
> _______________________________________________
> Fetchmail-users mailing list
> Fet...@li...
> https://lists.sourceforge.net/lists/listinfo/fetchmail-users

Re: [Fetchmail-users] Weird Return-Path processing

[Matthias A. <mat...@gm...>]

Am 27.03.20 um 09:08 schrieb Skle.pik:
> I did check POP3 side with Python poplib and clearly the Return-Path
> is split across lines - result below:
>
> (b'+OK',
> [b'Return-Path:
> <noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap',
> b' 11.bnc.salesforce.com>',
> b'Delivered-To:
>
> Does anyone know who is to blame - is this Return-Path formatting
> violation on POP side, or maybe it is mailer being intolerant ? 

Looking at your Python trace and assuming it's represented properly
here, the header isn't legit (per RFC5322) because the CFWS after the
.ap at the end of the first line terminates the first atom or 1*atext
and before the next atom/atext there would have to be a dot, so you'd
need to trace back to the originator (which may or may not have been
Salesforce) where it broke, or if the originator who pretended to have
been (and may or may not have been) Salesforce sent garbage already.

However you seem to be proving that fetchmail is not unfolding the line
properly, and is shipping out this broken line via SMTP (RFC5321), which
isn't permitted either, the SMTP Domain can't have blanks or line
breaks, so this seems to be a bug. Given that RFC5322 Return-Path:
header syntax is more permissive than RFC5321 SMTP MAIL FROM:<...>
syntax, fetchmail needs to validate input and act accordingly.

FTR, which fetchmail version are you using?

Re: [Fetchmail-users] Weird Return-Path processing

[Ralph C. <ra...@in...>]

Hi Skle.pik,

> Here is plaintext version of a message with some updates:

Thanks, unfortunately it has wrapped the lines, but we can undo that.

> The investigation of processing for not delivered mails shows
> Return-Path to be split to 2 lines:
> Return-Path: 
> <noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap
> 11.bnc.salesforce.com>
> Delivered-To:

Unwrapped:

    Return-Path: <noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap
    11.bnc.salesforce.com>
    Delivered-To:

> Unfortunately this makes mailer choking:
> SMTP> MAIL 
> FROM:<noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap
> 11.bnc.salesforce.com> SIZE=4070
> SMTP< 553 5.0.0 ....

Unwrapped:

    SMTP> MAIL FROM:<noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap
    11.bnc.salesforce.com> SIZE=4070
    SMTP< 553 5.0.0 ....

> I did check POP3 side with Python poplib and clearly the Return-Path
> is split across lines - result below:
>
> (b'+OK',
> [b'Return-Path: 
> <noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap',
> b' 11.bnc.salesforce.com>',
> b'Delivered-To:
>
> Does anyone know who is to blame - is this Return-Path formatting
> violation on POP side, or maybe it is mailer being intolerant ?

The POP3 server is dishing up the split line as you have used to
separate POP3 clients and both show the problem.

The number of bytes in

    $ wc -c
    MAIL FROM:<noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap
    ^D
    74
    $

is 74, including the linefeed, and that's a typical length for folding
text, though obviously it shouldn't be doing it to protocol lines.

-- 
Cheers, Ralph.

Re: [Fetchmail-users] Weird Return-Path processing

[Skle.pik <sp...@ma...>]

On 27/03/2020 07:41, Skle.pik wrote:
> Hi,  I've been using fetchmail for quite a while without any complain whatsoever, but recently I found some of the messages not to be delivered.  The investigation of processing for not delivered mails shows Return-Path to be split to 2 lines:  Return-Path: <noreply=  abba.com__al4bl7qh36xreia2@35p  11.bnc.salesforce.com>  Delivered-To:   Unfortunately this makes mailer choking:  SMTP> MAIL FROM:<noreply=  abba.com__al4bl7qh36xreia2@35p  11.bnc.salesforce.com> SIZE=4070  SMTP< 553 5.0.0 ....   Does anyone know who is to blame - is this Return-Path formatting violation on POP side, or maybe it is mailer being intolerant ?   Cheers
Sorry for broken formatting, seems like my web MUA can't do plaintext.

Here is plaintext version of a message with some updates:

Hi,
I've been using fetchmail for quite a while without any complain 
whatsoever, but recently I found some of the messages not to be 
delivered.  The investigation of processing for not delivered mails 
shows Return-Path to be split to 2 lines:
Return-Path: 
<noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap
11.bnc.salesforce.com>
Delivered-To:

Unfortunately this makes mailer choking:
SMTP> MAIL 
FROM:<noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap
11.bnc.salesforce.com> SIZE=4070
SMTP< 553 5.0.0 ....

I did check POP3 side with Python poplib and clearly the Return-Path is 
split across lines - result below:

(b'+OK',
[b'Return-Path: 
<noreply=abba.com__al4bl7qh36xreia2@35p9h14cih4s.6f-1gv0suag.ap',
b' 11.bnc.salesforce.com>',
b'Delivered-To:

Does anyone know who is to blame - is this Return-Path formatting 
violation on POP side, or maybe it is mailer being intolerant ?

Cheers

Re: [Fetchmail-users] Weird Return-Path processing

[Matthias A. <mat...@gm...>]

Am 27.03.20 um 07:41 schrieb Skle.pik:
> Hi,  I've been using fetchmail for quite a while without any complain whatsoever, but recently I found some of the messages not to be delivered.  The investigation of processing for not delivered mails shows Return-Path to be split to 2 lines:  Return-Path: <noreply=  abba.com__al4bl7qh36xreia2@35p  11.bnc.salesforce.com>  Delivered-To:   Unfortunately this makes mailer choking:  SMTP> MAIL FROM:<noreply=  abba.com__al4bl7qh36xreia2@35p  11.bnc.salesforce.com> SIZE=4070  SMTP< 553 5.0.0 ....   Does anyone know who is to blame - is this Return-Path formatting violation on POP side, or maybe it is mailer being intolerant ?   Cheers

Hello,

unfortunately, the message has been malformed in transit.

Please re-send in plain text only (the list driver strips html parts)
and feel free to Cc: my sender address so I can see if it's the list
driver corrupting your mail, or your mailer.

Regards,
Matthias

[Fetchmail-users] Weird Return-Path processing

[Skle.pik <sp...@ma...>]

Hi,  I've been using fetchmail for quite a while without any complain whatsoever, but recently I found some of the messages not to be delivered.  The investigation of processing for not delivered mails shows Return-Path to be split to 2 lines:  Return-Path: <noreply=  abba.com__al4bl7qh36xreia2@35p  11.bnc.salesforce.com>  Delivered-To:   Unfortunately this makes mailer choking:  SMTP> MAIL FROM:<noreply=  abba.com__al4bl7qh36xreia2@35p  11.bnc.salesforce.com> SIZE=4070  SMTP< 553 5.0.0 ....   Does anyone know who is to blame - is this Return-Path formatting violation on POP side, or maybe it is mailer being intolerant ?   Cheers

[Fetchmail-users] TLS fingerprints, and option --pinnedpublickey <sha256//b64hash>

[grarpamp <gra...@gm...>]

FYI Re: The former thread with this subject.

One of the main links in it has moved and is now here...

https://owasp.org/www-project-cheat-sheets/cheatsheets/Pinning_Cheat_Sheet
https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Pinning_Cheat_Sheet.md


Added github for the other link...
https://owasp.org/www-community/controls/Certificate_and_Public_Key_Pinning
https://github.com/OWASP/www-community/blob/master/pages/controls/Certificate_and_Public_Key_Pinning.md

Re: [Fetchmail-users] UTF-8 support?

[Matthias A. <mat...@gm...>]

Am 13.03.20 um 05:31 schrieb Sergey Organov:
> Yeah, you are right.
>
> In this particular case it happens to be particular WWW service that was
> misbehaving by putting addresses into generated e-mails in UTF-8
> encoding.
>
> Thanks once again for your help!
Sergey, I would not call such services "misbehaving" - that really
depends: if the service adheres to standing IETF standards, it is not
misbehaving, but I would agree that it is on the edge of overall system
robustness ("be conservative in what you send...") and testing the system...

Re: [Fetchmail-users] UTF-8 support?

[Sergey O. <sor...@gm...>]

Matthias Andree <mat...@gm...> writes:

> Am 06.03.20 um 15:48 schrieb Sergey Organov:
>>> Yup, and client support doesn't seem abundant. Python's poplib appears
>>> to support it in recent versions.
>>>
>>> So, bottom line: I'll put this on the TODO for 6.5, but I will probably
>>> not implement downgrading or conversion code, fetchmail will need to
>>> assume the destination SMTP/LMTP server or MDA is 8bit clean
>>> (8BITMIME/SMTPUTF8).
>>>
>>> This will take a while.
>>> https://gitlab.com/fetchmail/fetchmail/-/issues/14
>> Thanks a lot for taking care of this, Matthias!
>>
>> Right now getting 3-4 such surrogate mails a day is not a big deal, but
>> it might start to affect more people in the future, as these new servers
>> start to spread.
>>
> The question is how much open UTF-8 is really sent - this would not be
> overly compatible and will still often need downconversion to 7-bit
> mail...

Yeah, you are right.

In this particular case it happens to be particular WWW service that was
misbehaving by putting addresses into generated e-mails in UTF-8
encoding.

Thanks once again for your help!

-- Sergey

Re: [Fetchmail-users] Oauth2 support

[Matthias A. <mat...@gm...>]

Am 07.03.20 um 16:10 schrieb Kevin Gehrke:
>
> Hi All,
>
>
> I believe this year Gmail and MS Office 365 will no longer support
> basic authentication  (UserID and PW) for Pop3 or IMAP.
>
> Any plans to support Oauth2 in Fetchmail? 

Then check if you can have API or application specific passwords, and
use those, or keep the switch on to enable what Google/Microsoft believe
to be "less secure apps" (which is also nonsense).

Beyond that, the "master" branch has initial bits in place that were
contributed, but has fallen a bit behind recent changes to 6.4.2, but
you might want to test the master branch and share your findings.
<https://gitlab.com/search?utf8=%E2%9C%93&search=oauth2&group_id=&project_id=188557&search_code=true&repository_ref=master&nav_source=navbar>

[Fetchmail-users] Oauth2 support

[Kevin G. <kg...@co...>]

Hi All,


I believe this year Gmail and MS Office 365 will no longer support basic 
authentication  (UserID and PW) for Pop3 or IMAP.

Any plans to support Oauth2 in Fetchmail?

Re: [Fetchmail-users] fetchmail vs divecot server setup for pop3 and imap

[Ralph C. <ra...@in...>]

Hi Gene,

> > > fetchmail: POP3> DELE 9
> > > fetchmail: POP3< +OK Marked to be deleted.
> > > fetchmail: POP3> QUIT
> > > fetchmail: POP3< +OK Logging out, messages deleted.
> >
> > if their Dovecot claims the messages deleted and the same messages
> > remain, something needs fixing on their end.
>
> And their response has always been that they have clients using both
> interfaces who would be upset to discover that mail fetched by a
> mailfetcher and then deleted was missing when they next came in with a
> #$^(&%* browser.

Then move away from that #$^(&%* POP3 provider.  :-)

DELE is the mechanism POP3 provides to delete messages.  They have to
honour it.  With *every* other email provider I've heard about on lists,
etc., over the years, if the user wants to retrieve a *copy* of the
email and leave the original there for web-mail then they have to use
IMAP, not POP3.  That is a key difference between the user's choice.

So much of a key difference that some ISPs force users into POP3 so
their inbox storage can't grow over time.

Given what you now know, I'm not sure what you expect fetchmail to be
able to do when the ISP is violating the protocol.

-- 
Cheers, Ralph.