fetchmail-devel

[fetchmail-devel] Add feature: add header to messages?

[Jonathan K. <ji...@ka...>]

Hello everyone,

We've run into a fetchmail use case which doesn't seem to be easily 
solved with existing functionality, and I'm writing to ask (a) are we 
missing something and (b) if not, would the fetchmail maintainers 
consider accepting a patch to solve the problem?

In a nutshell, our custom MDA wants to see the recipient addresses of 
incoming messages; it's not there anywhere; and we can't figure out any 
scalable way to get fetchmail to provide it to the MDA.

Our .fetchmailrc fetches messages from multiple mailboxes with different 
addresses. These messages are fed into our proprietary application using 
a custom MDA script that we wrote. Our application does different things 
based on which mailbox each message is from.

The mailboxes receiving these messages are often BCC'd, which means that 
the recipient address doesn't appear in To or CC. Furthermore, the mail 
servers we're downloading from unfortunately don't insert 
X-Delivered-To, or Received lines showing the recipient address, or 
anything like that. In short, there's no way to tell from looking at the 
message fed into the MDA what address it was sent to.

We could hard-code a command-line option for the custom MDA command 
identifying the mailbox, but as far as I can tell, that fails if you've 
got more than one account in the .fetchmailrc file, because I don't 
believe it's possible to specify different MDA settings for different 
accounts (am I wrong about that?).

The idea I've come up with is to add a new "id" per-mailbox 
configuration setting to allow the user to specify an arbitrary 
identifier for the mailbox, and a "%I" command-line escape for the MDA 
settings to allow the identifier for the account from which the current 
message originated to be passed into the MDA.

So, my questions are (a) is there a way I'm missing to solve this 
problem, (b) if not, and we implemented the functionality I described, 
would the maintainers take a patch, and (c) if not, is there some other 
way to solve this problem that the maintainers would suggest we 
implement instead?

Thanks,

Jonathan Kamens

Re: [fetchmail-devel] Add feature: add header to messages?

[Matthias A. <mat...@gm...>]

Am 15.11.2011 14:29, schrieb Jonathan Kamens:

> We've run into a fetchmail use case which doesn't seem to be easily
> solved with existing functionality, and I'm writing to ask (a) are we
> missing something and (b) if not, would the fetchmail maintainers
> consider accepting a patch to solve the problem?

Hi Jonathan,

if I understand you correctly, (a) would apply, but as a second line of
defence, I'm willing to consider (b).

> The mailboxes receiving these messages are often BCC'd, which means that
> the recipient address doesn't appear in To or CC. Furthermore, the mail
> servers we're downloading from unfortunately don't insert
> X-Delivered-To, or Received lines showing the recipient address, or
> anything like that. In short, there's no way to tell from looking at the
> message fed into the MDA what address it was sent to.

And configuring a filter on the upstream server isn't possible either?
Sounds like a rather limited service to me.

> We could hard-code a command-line option for the custom MDA command
> identifying the mailbox, but as far as I can tell, that fails if you've
> got more than one account in the .fetchmailrc file, because I don't
> believe it's possible to specify different MDA settings for different
> accounts (am I wrong about that?).

Yes, you're wrong - luckily ;-) The MDA setting is a user-specific
setting, so you can generate a modified rcfile that uses a distinct mda
script (or arguments) for each user.  Now, that's for the easy case
where you have one destination address per upstream account.  It won't
work with --mda on the command line though.

For more complex cases that can't be solved that way, I'd be willing to
consider patches against fetchmail's 7 alpha versions (the "master"
branch in Git), such as the proposed one.

> So, my questions are (a) is there a way I'm missing to solve this
> problem, (b) if not, and we implemented the functionality I described,
> would the maintainers take a patch, and (c) if not, is there some other
> way to solve this problem that the maintainers would suggest we
> implement instead?

I'd have to ponder a bit about (c) if the (a) doesn't help.

Hope that helps.

Best regards,
Matthias Andree

Re: [fetchmail-devel] Add feature: add header to messages?

[Jonathan K. <ji...@ka...>]

Hi Matthias,

Thanks for the response. Comments and questions below.

On 11/21/2011 08:10 PM, Matthias Andree wrote:
>> The mailboxes receiving these messages are often BCC'd, which means that
>> the recipient address doesn't appear in To or CC. Furthermore, the mail
>> servers we're downloading from unfortunately don't insert
>> X-Delivered-To, or Received lines showing the recipient address, or
>> anything like that. In short, there's no way to tell from looking at the
>> message fed into the MDA what address it was sent to.
> And configuring a filter on the upstream server isn't possible either?
> Sounds like a rather limited service to me.
We cannot ask our users to muck around with the settings on the mail 
server. They are often technically naive or inexperienced, and we need 
to be able to make things work within our application without getting 
anything from them but the mail server's host name, protocol (IMAP or 
POP3), username, and password.

>> We could hard-code a command-line option for the custom MDA command
>> identifying the mailbox, but as far as I can tell, that fails if you've
>> got more than one account in the .fetchmailrc file, because I don't
>> believe it's possible to specify different MDA settings for different
>> accounts (am I wrong about that?).
> Yes, you're wrong - luckily ;-) The MDA setting is a user-specific
> setting, so you can generate a modified rcfile that uses a distinct mda
> script (or arguments) for each user.  Now, that's for the easy case
> where you have one destination address per upstream account.  It won't
> work with --mda on the command line though.
The problem with this approach, if I am understanding what you are 
saying correctly, is that we have multiple remote mailboxes /being 
delivered to the same local user account./ If the MDA setting were 
specific to the remote account being queried that would work fine, but 
if it's specific to the local account, it's problematic.

I suppose we could theoretically create dummy local users for the 
various remote accounts being polled, and write the RC file to "deliver" 
each remote account to a separate dummy user, but that seems awfully 
kludgy. It seems like it would be a lot cleaner to add the identifier + 
%-escape functionality I proposed. What do you think?

Thanks again,

Jonathan Kamens

P.S. I apologize for the misleading Subject line I used to start this 
thread. When I started writing my message to the list, I was thinking in 
terms of modifying fetchmail to allow arbitrary headers to be added to 
messages before delivery, but my thinking evolved as I wrote the message 
and I obviously ended up proposing something different, but I forgot to 
go back and update the Subject. I'm not going to change the Subject line 
now, though, because that would just confuse everybody's threading. :-)

Re: [fetchmail-devel] Add feature: add header to messages?

[Jonathan K. <ji...@ka...>]

A few minutes ago, I wrote:

> The problem with this approach, if I am understanding what you are 
> saying correctly, is that we have multiple remote mailboxes /being 
> delivered to the same local user account./ If the MDA setting were 
> specific to the remote account being queried that would work fine, but 
> if it's specific to the local account, it's problematic.
Never mind, I am an idiot. I understand now what you are saying, and 
yes, I think what you proposed will solve our problem.

Thanks again for your response.

   jik

Re: [fetchmail-devel] Add feature: add header to messages?

[Matthias A. <mat...@gm...>]

Am 22.11.2011 03:42, schrieb Jonathan Kamens:

>> Yes, you're wrong - luckily ;-) The MDA setting is a user-specific
>> setting, so you can generate a modified rcfile that uses a distinct mda
>> script (or arguments) for each user.  Now, that's for the easy case
>> where you have one destination address per upstream account.  It won't
>> work with --mda on the command line though.

> The problem with this approach, if I am understanding what you are
> saying correctly, is that we have multiple remote mailboxes /being
> delivered to the same local user account./ If the MDA setting were
> specific to the remote account being queried that would work fine, but
> if it's specific to the local account, it's problematic.

Well, "user" is as opposed to "server", and was referring to fetchmail
rcfile syntax.  What I actually mean is that it's specific to the
upstream server's user, or account.

You should be able to go about something like (untested fragment!):

#------------------------------------------
# one server with two accounts to be polled
poll someserver.example.org proto pop3 uidl
user john pass 123 is joe.sixpack mda "/my/fancy/script john" ssl
user jack pass 456 is joe.sixpack here mda "/my/fancy/script jack" ssl
#...

poll otherimaptoaster.example proto imap
user joe pass "really@ strange!password" is joe.sixpack here
mda "/my/fancy/script joe@other"
#------------------------------------------

Would that work for you?  All three accounts from two servers map to the
same local user, but have a distinct mda option, so I hope this meets
your needs.

> P.S. I apologize for the misleading Subject line I used to start this
> thread. When I started writing my message to the list, I was thinking in
> terms of modifying fetchmail to allow arbitrary headers to be added to
> messages before delivery, but my thinking evolved as I wrote the message
> and I obviously ended up proposing something different, but I forgot to
> go back and update the Subject. I'm not going to change the Subject line
> now, though, because that would just confuse everybody's threading. :-)

Would it really?  I rely on In-Reply-To: and References: headers for
threading...

But generally I'm open to provide filtering through an external command
should the need arise, possibly combined with decisions as to the
transient/permanent error or deliver or drop-silently.  I'm not sure
that it needs to be more specific like that because there are tools like
maildrop's reformail that can modify message headers.