fetchmail-users

[Fetchmail-users] Procmail filtering question

[Richard K. <ric...@bt...>]

I'm running fetchmail -> procmail -> local directories read by Claws on
Ubuntu 20.04

I'm having some problems filtering email messages by Subject using
procmail

"Normal" text Subjects of the form:
Subject: News on Coronavirus
are no problem. But there are a lot of messages nowadays that are
displayed in Claws as:

Subject: Update from GOV.UK – Coronavirus Statutory Sick Pay ... etc

but where the subject in the message itself is actually in the form:

Subject:
=?UTF-8?Q?Update_from_GOV.UK_=E2=80=93_Coronavirus_Statutory_Sick_P?=
=?UTF-8?Q?ay_Rebate_Scheme:_service_availability_and_issues?= 
(the above is all one line)

Does anyone know if there is some generic way of ensuring that messages
are passed to procmail in the "normal" format so that I may reliably and
straightforwardly construct procmail matching rules (i.e. without
having to inspect the message source each time)?

- Richard.
-- 
Richard Kimber

Re: [Fetchmail-users] Procmail filtering question

[Peter P. <ro...@ri...>]

On Tue, May 26, 2020 at 11:36:59AM +0100, Richard Kimber via Fetchmail-users wrote:
> I'm running fetchmail -> procmail -> local directories read by Claws on
> Ubuntu 20.04
> 
> I'm having some problems filtering email messages by Subject using
> procmail
> 
> "Normal" text Subjects of the form:
> Subject: News on Coronavirus
> are no problem. But there are a lot of messages nowadays that are
> displayed in Claws as:
> 
> Subject: Update from GOV.UK – Coronavirus Statutory Sick Pay ... etc
> 
> but where the subject in the message itself is actually in the form:
> 
> Subject:
> =?UTF-8?Q?Update_from_GOV.UK_=E2=80=93_Coronavirus_Statutory_Sick_P?=
> =?UTF-8?Q?ay_Rebate_Scheme:_service_availability_and_issues?= 
> (the above is all one line)
> 
> Does anyone know if there is some generic way of ensuring that messages
> are passed to procmail in the "normal" format so that I may reliably and
> straightforwardly construct procmail matching rules (i.e. without
> having to inspect the message source each time)?

Unfortunately, this is the most "normal" of the formats that you are
going to get with e-mail. Since none of the software that generates or
reformats e-mail messages along the way has any idea what other software
will be used to accept, transfer, or process the message, none of it can
rely on anything else being able to process anything other than "simple"
plain text with all characters being in the US-ASCII character set.
Thus, if somebody wants to include, say, a pretty long dash or chevrons,
the only way to do that is to use a special format for encoding them,
a special format for saying "this here is a character that is outside
the US-ASCII character set, I'll tell you how it is represented in
UTF-8, but since its UTF-8 representation itself depends on bytes that
are outside the US-ASCII character set, here's a marker that says that
the next couple of characters are base64-encoded representations of
UTF-8 encoded characters".

Anything that wants to process e-mail messages the way they are to be
displayed to the end-user should be able to decode MIME-encoded content,
including MIME-encoded header fields. And... unfortunately, here we come
once again to the "procmail is kind of outdated, it does not really
support a couple of things that are sort of essential in the current
world, is there any way you might switch to other message-filtering
software?" For example, it looks like at least courier-maildrop may be
configured to parse MIME-encoded headers:
http://courier-mail-server.10983.n7.nabble.com/Filtering-mails-with-UTF-8-headers-td18177.html
(and yes, I know that switching the mail filtering program is not
trivial at all, but procmail also has other problems, some of which
people have become accustomed to working around for decades...)

G'luck,
Peter

-- 
Peter Pentchev  ro...@ri... ro...@de... pp...@st...
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

Re: [Fetchmail-users] Procmail filtering question

[Richard K. <ric...@bt...>]

On Tue, 26 May 2020 16:39:04 +0300
Peter Pentchev <ro...@ri...> wrote:

> Unfortunately, this is the most "normal" of the formats that you are
> going to get with e-mail. Since none of the software that generates or
> reformats e-mail messages along the way has any idea what other
> software will be used to accept, transfer, or process the message,
> none of it can rely on anything else being able to process anything
> other than "simple" plain text with all characters being in the
> US-ASCII character set. Thus, if somebody wants to include, say, a
> pretty long dash or chevrons, the only way to do that is to use a
> special format for encoding them, a special format for saying "this
> here is a character that is outside the US-ASCII character set, I'll
> tell you how it is represented in UTF-8, but since its UTF-8
> representation itself depends on bytes that are outside the US-ASCII
> character set, here's a marker that says that the next couple of
> characters are base64-encoded representations of UTF-8 encoded
> characters".
> 
> Anything that wants to process e-mail messages the way they are to be
> displayed to the end-user should be able to decode MIME-encoded
> content, including MIME-encoded header fields. And... unfortunately,
> here we come once again to the "procmail is kind of outdated, it does
> not really support a couple of things that are sort of essential in
> the current world, is there any way you might switch to other
> message-filtering software?" For example, it looks like at least
> courier-maildrop may be configured to parse MIME-encoded headers:
> http://courier-mail-server.10983.n7.nabble.com/Filtering-mails-with-UTF-8-headers-td18177.html
> (and yes, I know that switching the mail filtering program is not
> trivial at all, but procmail also has other problems, some of which
> people have become accustomed to working around for decades...)

Thanks.

I was wondering, though, whether one could do something like
mda '/usr/bin/formail -s <program that converts to ascii> | procmail'
in the .fetchmailrc file.

- Richard.
-- 
Richard Kimber

Re: [Fetchmail-users] Procmail filtering question

[Carlos E. R. <rob...@te...>]

On 27/05/2020 13.15, Richard Kimber via Fetchmail-users wrote:
> On Tue, 26 May 2020 16:39:04 +0300
> Peter Pentchev <ro...@ri...> wrote:
> 
>> Unfortunately, this is the most "normal" of the formats that you are
>> going to get with e-mail. Since none of the software that generates or
>> reformats e-mail messages along the way has any idea what other
>> software will be used to accept, transfer, or process the message,
>> none of it can rely on anything else being able to process anything
>> other than "simple" plain text with all characters being in the
>> US-ASCII character set. Thus, if somebody wants to include, say, a
>> pretty long dash or chevrons, the only way to do that is to use a
>> special format for encoding them, a special format for saying "this
>> here is a character that is outside the US-ASCII character set, I'll
>> tell you how it is represented in UTF-8, but since its UTF-8
>> representation itself depends on bytes that are outside the US-ASCII
>> character set, here's a marker that says that the next couple of
>> characters are base64-encoded representations of UTF-8 encoded
>> characters".
>>
>> Anything that wants to process e-mail messages the way they are to be
>> displayed to the end-user should be able to decode MIME-encoded
>> content, including MIME-encoded header fields. And... unfortunately,
>> here we come once again to the "procmail is kind of outdated, it does
>> not really support a couple of things that are sort of essential in
>> the current world, is there any way you might switch to other
>> message-filtering software?" For example, it looks like at least
>> courier-maildrop may be configured to parse MIME-encoded headers:
>> http://courier-mail-server.10983.n7.nabble.com/Filtering-mails-with-UTF-8-headers-td18177.html
>> (and yes, I know that switching the mail filtering program is not
>> trivial at all, but procmail also has other problems, some of which
>> people have become accustomed to working around for decades...)
> 
> Thanks.
> 
> I was wondering, though, whether one could do something like
> mda '/usr/bin/formail -s <program that converts to ascii> | procmail'
> in the .fetchmailrc file.

It is not that simple. The mail itself is already ascii, you would have 
to convert to utf8 to see a complete rendering of the headers. But 
procmail can't handle that.

If you force conversion to ascii of the headers, so to speak, how do you 
handle the characters in the headers that do not have an ascii 
equivalent? You have to replace with '*'.


The only thing would be grep patterns in procmail that would handle the 
headers as they come without conversion. I don't remember if procmail 
handles grep.


-- 
Cheers / Saludos,

		Carlos E. R.
		(from 15.1 x86_64 at Telcontar)

Re: [Fetchmail-users] Procmail filtering question

[Peter P. <ro...@ri...>]

On Wed, May 27, 2020 at 11:53:07AM +0100, Richard Kimber wrote:
> On Tue, 26 May 2020 16:39:04 +0300
> Peter Pentchev <ro...@ri...> wrote:
> 
> > Unfortunately, this is the most "normal" of the formats that you are
> > going to get with e-mail. Since none of the software that generates or
> > reformats e-mail messages along the way has any idea what other
> > software will be used to accept, transfer, or process the message,
> > none of it can rely on anything else being able to process anything
> > other than "simple" plain text with all characters being in the
> > US-ASCII character set. Thus, if somebody wants to include, say, a
> > pretty long dash or chevrons, the only way to do that is to use a
> > special format for encoding them, a special format for saying "this
> > here is a character that is outside the US-ASCII character set, I'll
> > tell you how it is represented in UTF-8, but since its UTF-8
> > representation itself depends on bytes that are outside the US-ASCII
> > character set, here's a marker that says that the next couple of
> > characters are base64-encoded representations of UTF-8 encoded
> > characters".
> > 
> > Anything that wants to process e-mail messages the way they are to be
> > displayed to the end-user should be able to decode MIME-encoded
> > content, including MIME-encoded header fields. And... unfortunately,
> > here we come once again to the "procmail is kind of outdated, it does
> > not really support a couple of things that are sort of essential in
> > the current world, is there any way you might switch to other
> > message-filtering software?" For example, it looks like at least
> > courier-maildrop may be configured to parse MIME-encoded headers:
> > http://courier-mail-server.10983.n7.nabble.com/Filtering-mails-with-UTF-8-headers-td18177.html
> > (and yes, I know that switching the mail filtering program is not
> > trivial at all, but procmail also has other problems, some of which
> > people have become accustomed to working around for decades...)
> 
> Thanks.
> 
> I was wondering, though, whether one could do something like
> mda '/usr/bin/formail -s <program that converts to ascii> | procmail'
> in the .fetchmailrc file.

...but the whole point is that this encoding means that the subject
contains non-ASCII characters. There is no way to convert them to
ASCII.

G'luck,
Peter

-- 
Peter Pentchev  ro...@ri... ro...@de... pp...@st...
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

Re: [Fetchmail-users] Procmail filtering question

[Richard K. <ric...@bt...>]

On Wed, 27 May 2020 14:33:40 +0300
Peter Pentchev <ro...@ri...> wrote:

> ...but the whole point is that this encoding means that the subject
> contains non-ASCII characters. There is no way to convert them to
> ASCII.

But if it's utf-8 can't you use something like:
konwert utf8-ascii

- Richard
-- 
Richard Kimber

Re: [Fetchmail-users] Procmail filtering question

[Peter P. <ro...@ri...>]

On Wed, May 27, 2020 at 12:39:57PM +0100, Richard Kimber via Fetchmail-users wrote:
> On Wed, 27 May 2020 14:33:40 +0300
> Peter Pentchev <ro...@ri...> wrote:
> 
> > ...but the whole point is that this encoding means that the subject
> > contains non-ASCII characters. There is no way to convert them to
> > ASCII.
> 
> But if it's utf-8 can't you use something like:
> konwert utf8-ascii

Even if you do something like this, even if you manage to figure out
a way to deal with characters that *do not exist* in the US-ASCII
character set (yeah, I can see that konwert may have some limited
capabilities of approximating them, but that sounds quite weird to me);
even if you manage to do that, you will not have the original message.
If you feed the modified message to procmail and it stashes it into some
mailbox/folder/whatever or passes it to another program, you will *not*
have the message that the sender wanted you to have. In some cases this
might not be so bad ("I don't care if a spam message has an extra quote
or not"), but in others, like verifying OpenPGP or S/MIME signatures,
it might be, how do I put it, bad :)

G'luck,
Peter

-- 
Peter Pentchev  ro...@ri... ro...@de... pp...@st...
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

Re: [Fetchmail-users] Procmail filtering question

[Chris P. <cpo...@em...>]

On Tue, 2020-05-26 at 11:36 +0100, Richard Kimber via Fetchmail-users
wrote:
> I'm running fetchmail -> procmail -> local directories read by Claws
> on
> Ubuntu 20.04
> 
> I'm having some problems filtering email messages by Subject using
> procmail
> 
> "Normal" text Subjects of the form:
> Subject: News on Coronavirus
> are no problem. But there are a lot of messages nowadays that are
> displayed in Claws as:
> 
> Subject: Update from GOV.UK – Coronavirus Statutory Sick Pay ... etc
> 
> but where the subject in the message itself is actually in the form:
> 
> Subject:
> =?UTF-8?Q?Update_from_GOV.UK_=E2=80=93_Coronavirus_Statutory_Sick_P?=
> =?UTF-8?Q?ay_Rebate_Scheme:_service_availability_and_issues?= 
> (the above is all one line)
> 
> Does anyone know if there is some generic way of ensuring that
> messages
> are passed to procmail in the "normal" format so that I may reliably
> and
> straightforwardly construct procmail matching rules (i.e. without
> having to inspect the message source each time)?
> 
> - Richard.

Are any of these helpful?


https://stackoverflow.com/questions/29715013/decode-the-utf8-to-iso-8859-1-mail-subject-to-text-in-procmailrc-file

https://www.mhonarc.org/archive/html/procmail/2015-10/msg00003.html


-- 
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
09:37:29 up 22:36, 1 user, load average: 0.83, 0.72, 0.74
Description:	Ubuntu 18.04.4 LTS, kernel 5.3.0-53-generic

Re: [Fetchmail-users] Procmail filtering question

[Matthias A. <mat...@gm...>]

Am 26.05.20 um 12:36 schrieb Richard Kimber via Fetchmail-users:
> I'm running fetchmail -> procmail -> local directories read by Claws on
> Ubuntu 20.04
>
> I'm having some problems filtering email messages by Subject using
> procmail
>
> "Normal" text Subjects of the form:
> Subject: News on Coronavirus
> are no problem. But there are a lot of messages nowadays that are
> displayed in Claws as:
>
> Subject: Update from GOV.UK – Coronavirus Statutory Sick Pay ... etc
>
> but where the subject in the message itself is actually in the form:
>
> Subject:
> =?UTF-8?Q?Update_from_GOV.UK_=E2=80=93_Coronavirus_Statutory_Sick_P?=
> =?UTF-8?Q?ay_Rebate_Scheme:_service_availability_and_issues?= 
> (the above is all one line)
>
> Does anyone know if there is some generic way of ensuring that messages
> are passed to procmail in the "normal" format so that I may reliably and
> straightforwardly construct procmail matching rules (i.e. without
> having to inspect the message source each time)?

This has been a normal format since 1993 or so (RFC 1522, now RFC 2047).

Everyone just stop using that unmaintained (for almost 20 years) pile of
design garbage called procmail.
Procmail wasn't palatable in 1990, and isn't in 2020.

reformime (part of maildrop) can help with decoding, but I do not know
if it would do character set conversions into the current locale.

Also someone tell Ubuntu to update fetchmail to at least 6.4.4 everywhere...

Re: [Fetchmail-users] Procmail filtering question

[Carlos E. R. <rob...@te...>]

On 28/05/2020 16.57, Matthias Andree wrote:
> Am 26.05.20 um 12:36 schrieb Richard Kimber via Fetchmail-users:

...

> 
> This has been a normal format since 1993 or so (RFC 1522, now RFC 2047).
> 
> Everyone just stop using that unmaintained (for almost 20 years) pile of
> design garbage called procmail.

And use what, then?


-- 
Cheers / Saludos,

		Carlos E. R.
		(from 15.1 x86_64 at Telcontar)

Re: [Fetchmail-users] Procmail filtering question

[Gerard E. S. <ger...@ou...>]

On Thu, 28 May 2020 19:59:37 +0200, Carlos E. R. stated:
>On 28/05/2020 16.57, Matthias Andree wrote:
>> Am 26.05.20 um 12:36 schrieb Richard Kimber via Fetchmail-users:  
>
>...
>
>> 
>> This has been a normal format since 1993 or so (RFC 1522, now RFC
>> 2047).
>> 
>> Everyone just stop using that unmaintained (for almost 20 years)
>> pile of design garbage called procmail.  
>
>And use what, then?

I use 'sieve' with dovecot. It is way more powerful than procmail ever
hoped to be, and in my opinion, easier to configure.

-- 
Gerard

Re: [Fetchmail-users] Procmail filtering question

[Peter P. <ro...@ri...>]

On Thu, May 28, 2020 at 07:59:37PM +0200, Carlos E. R. wrote:
> On 28/05/2020 16.57, Matthias Andree wrote:
> > Am 26.05.20 um 12:36 schrieb Richard Kimber via Fetchmail-users:
> 
> ...
> 
> > 
> > This has been a normal format since 1993 or so (RFC 1522, now RFC 2047).
> > 
> > Everyone just stop using that unmaintained (for almost 20 years) pile of
> > design garbage called procmail.
> 
> And use what, then?

maildrop? (possibly courier-maildrop)

G'luck,
Peter

-- 
Peter Pentchev  ro...@ri... ro...@de... pp...@st...
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

Re: [Fetchmail-users] Procmail filtering question

[Carlos E. R. <rob...@te...>]

On 28/05/2020 21.19, Peter Pentchev wrote:
> On Thu, May 28, 2020 at 07:59:37PM +0200, Carlos E. R. wrote:
>> On 28/05/2020 16.57, Matthias Andree wrote:
>>> Am 26.05.20 um 12:36 schrieb Richard Kimber via Fetchmail-users:
>>
>> ...
>>
>>>
>>> This has been a normal format since 1993 or so (RFC 1522, now RFC 2047).
>>>
>>> Everyone just stop using that unmaintained (for almost 20 years) pile of
>>> design garbage called procmail.
>>
>> And use what, then?
> 
> maildrop? (possibly courier-maildrop)

I considered that once, but in my distribution of choice it meant 
installing the entire courier thing, and that was problematic (I use 
dovecot). I did not find out how to replace procmail with maildrop.


-- 
Cheers / Saludos,

		Carlos E. R.
		(from 15.1 x86_64 at Telcontar)

Re: [Fetchmail-users] Procmail filtering question

[Matthias A. <mat...@gm...>]

Am 28.05.20 um 21:32 schrieb Carlos E. R.:
> On 28/05/2020 21.19, Peter Pentchev wrote:
>> On Thu, May 28, 2020 at 07:59:37PM +0200, Carlos E. R. wrote:
>>> On 28/05/2020 16.57, Matthias Andree wrote:
>>>> Am 26.05.20 um 12:36 schrieb Richard Kimber via Fetchmail-users:
>>>
>>> ...
>>>
>>>>
>>>> This has been a normal format since 1993 or so (RFC 1522, now RFC
>>>> 2047).
>>>>
>>>> Everyone just stop using that unmaintained (for almost 20 years)
>>>> pile of
>>>> design garbage called procmail.
>>>
>>> And use what, then?
>>
>> maildrop? (possibly courier-maildrop)
>
> I considered that once, but in my distribution of choice it meant
> installing the entire courier thing, and that was problematic (I use
> dovecot). I did not find out how to replace procmail with maildrop.

Dovecot has Sieve and dovecot-lda or however it's called these days, and
on my distros, maildrop is also available without the full courier rig -
and maildrop has been my recommendation for a while.

There may be other options, and I wonder if Claws doesn't have built-in
filters, but I haven't used it in over a decade.

Re: [Fetchmail-users] Procmail filtering question

[Carlos E. R. <rob...@te...>]

On 28/05/2020 22.54, Matthias Andree wrote:
> Am 28.05.20 um 21:32 schrieb Carlos E. R.:
>> On 28/05/2020 21.19, Peter Pentchev wrote:
>>> On Thu, May 28, 2020 at 07:59:37PM +0200, Carlos E. R. wrote:
>>>> On 28/05/2020 16.57, Matthias Andree wrote:
>>>>> Am 26.05.20 um 12:36 schrieb Richard Kimber via Fetchmail-users:
>>>>
>>>> ...
>>>>
>>>>>
>>>>> This has been a normal format since 1993 or so (RFC 1522, now RFC
>>>>> 2047).
>>>>>
>>>>> Everyone just stop using that unmaintained (for almost 20 years)
>>>>> pile of
>>>>> design garbage called procmail.
>>>>
>>>> And use what, then?
>>>
>>> maildrop? (possibly courier-maildrop)
>>
>> I considered that once, but in my distribution of choice it meant
>> installing the entire courier thing, and that was problematic (I use
>> dovecot). I did not find out how to replace procmail with maildrop.
> 
> Dovecot has Sieve and dovecot-lda or however it's called these days,

The man page of dovecot-lda doesn't explain how to do filtering. I have 
used dovecot-lda called from inside procmail, to do the delivery, but 
after some years I reverted it, because it deletes flags such as "read" 
or "replied".

Sieve I have not tried.


-- 
Cheers / Saludos,

		Carlos E. R.
		(from 15.1 x86_64 at Telcontar)