Re: [Fetchmail-users] Duplicate Emails

[Julian B. <jul...@bt...>]

Good morning Mattias and all,

I have managed to suppress the X-Original-To header by setting 
"*enable_original_recipient = no*" in the postfix configuration file. 
But still don't get duplicate emails suppressed, though the 
X-Original-To header is. (Matthias - I can send you the mailbox file if 
you want).

I have cut out two emails and verified they still have differences in 
the headers (some de-personalization done):

    1,3c1,3
    < From XXX...@co... Fri Jan 29 07:38:31 2021
    < Return-Path: <XXX...@co...>
    < Delivered-To: up...@my...
    ---
     > From jul...@co... Fri Jan 29 07:38:29 2021
     > Return-Path: <XXX...@co...>
     > Delivered-To: do...@my...
    38c38
    < X-UID: 24
    ---
     > X-UID: 23

It's hard to see how you could ever get two identical sets of headers 
for de-duplication to work.

Not wishing to either make trouble or make work for others but could the 
documentation be reworded to reflect how the de-duplication works. 
Currently it says "/A piece of mail is considered duplicate if it has 
the same message-ID as the message immediately preceding and more than 
one addressee/".

Many thanks and have a good weekend,

Julian

On 29/01/2021 00:04, Matthias Andree wrote:
> Am 28.01.21 um 09:00 schrieb Julian Bane via Fetchmail-users:
>> I have exactly the same results on fetchmail 6.4.15 - openSUSE
>> Tumbleweed current release which reports:
>>
>>      "This is fetchmail release
>> 6.4.15+POP2+GSS+RPA+NTLM+SDPS+SSL-SSLv2-SSLv3+OPIE+NLS+KRB5."
>>
>>
>> On 27/01/2021 13:41, Julian Bane via Fetchmail-users wrote:
>>> I have been running a test using fetchmail (6.3.26) on an openSUSE
>>> Leap 15.2 server to retrieve email from a local email account. That
>>> account has been sent 5 seperate emails. Three of the emails are to
>>> single addresses, one is to two addresses and another is to three
>>> addresses. As expected there are 8 separate emails in the local mail
>>> spool file from the 5 I sent. As a test I am using the fetchmailrc file:
>>>
>>>      set postmaster root
>>>      poll cumulus protocol POP3 local mydomain.net envelope
>>> "X-Original-To:"
>>>              username "user" password "pass" to * here fetchall keep
>>>              mda "echo 'Hello World - %T - %F'"
>>>
>>> The mda is just a dummy to see what is going on when I run it from
>>> the command line.
>>>
>>> It retrieves all 8 emails correctly identifying the origonal
>>> recipient and %T is substituted correctly for the different email
>>> adresses.
>>> Doing "fetchmail -V -f fetchmailrc" confirms that it is running in
>>> multidrop mode.
>>>
>>> If I substitute 'myname' for the '*' (asterisk) in fetchmailrc,
>>> fetchmail goes into singledrop mode as you would expect (checked with
>>> the -V option) but also retrieves all 8 emails.
>>>
>>> I am not sure which way round it should be, multidrop or singledrop,
>>> but I do want to remove duplicates and so want to retrieve just 5
>>> emails from the spool file (have been reading "THE USE AND ABUSE OF
>>> MULTIDROP MAILBOXES" on the man page). I have checked that the
>>> message-IDs are the same and there are multiple recipients on the
>>> 'To:' header of the relevant emails. I also have a copy of the spool
>>> file if it would help.
>>>
>>> Any advice or insight would be most welcome.
> Greetings,
>
> So after reviewing 6.4.15 logs that Julian sent to me off-list, some
> remarks:
>
> * The server in the test scenario contains messages with these
> Message-ID shown below. Meaning that the duplication already happened on
> the server-side mailbox:
>
> Message-ID: <16116718
> Message-ID: <6ec00e74
> Message-ID: <bbe93702
> Message-ID: <f1711007
> Message-ID: <f1711007
> Message-ID: <f1711007
> Message-ID: <f30f6378
> Message-ID: <7c6c3256
> Message-ID: <7c6c3256
>
> * fetchmail works as I believe it should: the server's mailbox in
> Julian's test scenario contains eight messages and fetchmail obtains and
> delivers eight. So fetchmail is not duplicating messages. Neither is it
> deduplicating, see next:
>
> * fetchmail has some deduplication code, but it is for a very specific
> and historic purpose: if the most-dangerous "To:/Cc:" guessing is being
> applied for lack of a header that carries a copy of the envelope's (SMTP
> dialogue's) destination address (Julian is using X-Original-To), then
> deduplication is supposed to kick in. The dedup'ing code compares the
> ENTIRE header (by way of calculating its MD5 hash value) in a run of
> message and suppresses all copies but the first.
>
> This code however is not supposed to work and also technically can not
> work in Julian's scenario: the headers of the server-side messages are
> *not exactly* the same, and differ in the X-Original-To: headers.
> Enhancing this deduplication code would mean changing behaviour and
> hence should not happen before fetchmail 7.0.0. An idea is to filter out
> a certain list of headers (among which the envelope's destination
> address recording headers) and only then hashing what is left of the
> header. But given the proposal below I'd say we don't actually need to
> change fetchmail:
>
> * Possible solution: install the standalone "maildrop" package from the
> Courier mailserver suite and use it, or a wrapper script for it, as mda.
> Maildrop ships with a "reformail" tool that can track Message-IDs of
> seen messages in a database and its exit status can be used to suppress
> these duplicates by message ID (it's option -D). The maildropex(7)
> manual page contains an example, just look for the first occurrence of:
> reformail -D
>
> https://www.courier-mta.org/maildrop/maildrop.html
>
>
> Note I strongly discourage using procmail. It has been unmaintained for
> like two decades and is inherently unsafe: All error handling and
> locking needs to be put in your recipes explicitly to achieve
> deterministic behavior in adverse conditions (such as under load).
> maildrop is safer to use and its .mailfilter files are easier to
> understand and write, and it is well-behaved by default, unlike procmail.
>
> HTH
>
> Happy fetching & deduplicating,
>
> Matthias
>
>
>
> _______________________________________________
> Fetchmail-users mailing list
> Fet...@li...
> https://lists.sourceforge.net/lists/listinfo/fetchmail-users