sqlgrey-users

[Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Sascha L. <sas...@ru...>]

Hi list,

I have configured sqlgrey-1.6.5 to PREPEND the X-Greylist: header. What I 
noticed is that the header was prepended multiple times (for each RCPT TO) 
in the mail. The logfile shows also "sqlgrey: grey: from awl match: 
updating" for each RCPT TO.

My question is: can the instance= in the postfix policy protocol be 
used, to prevent multiple action=PREPEND .... ? From postfix's 
SMTPD_POLICY_READM the instance=... can be used to correlate different 
requests regarding the same message delivery.

THX,

Sascha.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Lionel B. <lio...@bo...>]

Sascha Lucas wrote the following on 12.08.2005 14:58 :

> Hi list,
>
> I have configured sqlgrey-1.6.5 to PREPEND the X-Greylist: header. 
> What I noticed is that the header was prepended multiple times (for 
> each RCPT TO) in the mail. The logfile shows also "sqlgrey: grey: from 
> awl match: updating" for each RCPT TO.
>
> My question is: can the instance= in the postfix policy protocol be 
> used, to prevent multiple action=PREPEND .... ? From postfix's 
> SMTPD_POLICY_READM the instance=... can be used to correlate different 
> requests regarding the same message delivery.


There's a problem with that: nothing prevents two recipients for the 
same message from being treated differently (one can match a connect 
entry and the following matches the AWL entry just created:  two 
different headers). The problem is that we don't add the header on 
delivery but way before that. We have no way of adding a header per RCPT 
TO: this is a limitation of the policy protocol. We could have more 
meaningful headers with the "RCPT TO" listed for each header but I'm 
afraid it's the best we can do.

Lionel.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Sascha L. <sas...@ru...>]

> There's a problem with that: nothing prevents two recipients for the same 
> message from being treated differently (one can match a connect entry and the 
> following matches the AWL entry just created:  two different headers). The 
> problem is that we don't add the header on delivery but way before that. We 
> have no way of adding a header per RCPT TO: this is a limitation of the 
> policy protocol. We could have more meaningful headers with the "RCPT TO" 
> listed for each header but I'm afraid it's the best we can do.

I dont understand. If the 1st RCPT matches the connect entry and it is an 
"reconnect ok", then you add the X-Greylist: delayed ... header. The other 
RCPTs would result in a match of the from_awl table which would result in 
a X-Greylist: from auto-whitelisted ... header. But you know this 
instance=... allready gots it's header so sqlgrey don't say 
action=PREPEND....

Please tell me what point I misunderstand.

Thanks,

Sascha.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Lionel B. <lio...@bo...>]

Sascha Lucas wrote the following on 12.08.2005 17:13 :

>> There's a problem with that: nothing prevents two recipients for the 
>> same message from being treated differently (one can match a connect 
>> entry and the following matches the AWL entry just created:  two 
>> different headers). The problem is that we don't add the header on 
>> delivery but way before that. We have no way of adding a header per 
>> RCPT TO: this is a limitation of the policy protocol. We could have 
>> more meaningful headers with the "RCPT TO" listed for each header but 
>> I'm afraid it's the best we can do.
>
>
> I dont understand. If the 1st RCPT matches the connect entry and it is 
> an "reconnect ok", then you add the X-Greylist: delayed ... header. 
> The other RCPTs would result in a match of the from_awl table which 
> would result in a X-Greylist: from auto-whitelisted ... header. But 
> you know this instance=... allready gots it's header so sqlgrey don't 
> say action=PREPEND....


Ok, adding only the first header would indeed being meaningful in the 
example above. I was thinking of the future evolutions of SQLgrey and 
didn't explained myself well.

The above works only because we don't use AWLs based on recipients... 
*yet*. In the future, SQLgrey will have an intermediate connect_awl 
table (with src, sender and recipient) and a rcpt_awl (with only src, 
recipients) in addition to the current from_awl and domain_awl (this 
will solve some problems - false positive and negatives that could be 
avoided - seen in the wild and discussed earlier on this list). They 
will both introduce more complex cases that won't be resolvable by 
allowing only a single prepend to the same instance (simply because a 
single mail to different recipients would be accepted for totally 
different reasons in some cases).
Even if awls using "rcpt to"s wouldn't be used, maintaining a backlog of 
past instances is not that simple (nothing tells us that an instance is 
gone, so we will have to use heuristics to clean this backlog).
There is the case of optin/optout too: the prepended header for one 
"optout" address has nothing to do with the awl match of another.

Currently the multiple X-Greylist headers looks like a bug at first 
glance (I've already had reports on this). I believe adding the "rcpt 
to" matching each header to make them less confusing is the right thing 
to do. In a mail sent to 3 recipients, one of which happens to get a 
match in connect (another unrelated mail created the connect entry for 
example), another already being auto-whitelisted for this source (future 
rcpt_awl) and a last one which doesn't want to use greylisting at all 
(current optout mechanism):

X-Greylist: delayed 01:21:26.853989 by SQLgrey-1.6.5 for ad...@ex...
X-Greylist: recipient auto-whitelisted by SQLgrey-1.6.5 for 
for...@ex...
X-Greylist: greylisting inactive for cri...@ex... in 
SQLgrey-1.6.5

On a purely theoritical level, this doesn't really make sense to try to 
consolidate all headers in a single one. All decisions are currently 
completely independent from each other and only happen to be losely 
related because of some choices in the current AWLs design that *will* 
evolve.

Lionel.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Michael S. <Mic...@lr...>]

On Fri, 12 Aug 2005, Lionel Bouton wrote:

>
> X-Greylist: delayed 01:21:26.853989 by SQLgrey-1.6.5 for ad...@ex...
> X-Greylist: recipient auto-whitelisted by SQLgrey-1.6.5 for
> for...@ex...
> X-Greylist: greylisting inactive for cri...@ex... in
> SQLgrey-1.6.5
>

This will mean, that in some rare situations BCCed recipients will see
each other :-(

Michael Storz
-------------------------------------------------
Leibniz-Rechenzentrum   !   <mailto:St...@lr...>
Barer Str. 21           !   Fax: +49 89 2809460
80333 Muenchen, Germany !   Tel: +49 89 289-28840

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Lionel B. <lio...@bo...>]

Michael Storz wrote the following on 13.08.2005 07:04 :

>On Fri, 12 Aug 2005, Lionel Bouton wrote:
>
>  
>
>>X-Greylist: delayed 01:21:26.853989 by SQLgrey-1.6.5 for ad...@ex...
>>X-Greylist: recipient auto-whitelisted by SQLgrey-1.6.5 for
>>for...@ex...
>>X-Greylist: greylisting inactive for cri...@ex... in
>>SQLgrey-1.6.5
>>
>>    
>>
>
>This will mean, that in some rare situations BCCed recipients will see
>each other :-(
>  
>

They might already do : optout email addresses are already added to the 
headers.

But this is irrelevant if the sender or one of the mail gateway along 
the path follows the security recommendation of RFC2822 : send a 
separate copy of the message to the "To: and CC:" recipient and another 
to the "Bcc:" recipients.
Anyway, I'll make the header configurable with some macros (already in 
my TODO), so that the local admin can make the decision.

Lionel.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Sascha L. <sas...@ru...>]

> They might already do : optout email addresses are already added to the 
> headers.
>
> But this is irrelevant if the sender or one of the mail gateway along the 
> path follows the security recommendation of RFC2822 : send a separate copy of 
> the message to the "To: and CC:" recipient and another to the "Bcc:" 
> recipients.
> Anyway, I'll make the header configurable with some macros (already in my 
> TODO), so that the local admin can make the decision.

Does this mean that Bcc: a@b.c and Bcc: d@e.f can see each other? Well I 
don't know.

But here in germany the people of data privacy (not me :-)), wouldn't even 
be glad about RCPT-specific headers regardless of To:, CC: and Bcc:.

BTW: does anybody know how the postfix-users mailinglist works? This is 
the list, where I first saw multiple X-Greylist: headers. So I know 
someone else is getting this mail. For now I had to look in the
postfix-logfile to see who the others are, but with RCPT-specific headers 
every one can see.

If the header is customizable (in future versions), there should be a big 
warning in the config file.

Thanks,

Sascha.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Lionel B. <lio...@bo...>]

Sascha Lucas wrote the following on 15.08.2005 10:28 :

>> They might already do : optout email addresses are already added to 
>> the headers.
>>
>> But this is irrelevant if the sender or one of the mail gateway along 
>> the path follows the security recommendation of RFC2822 : send a 
>> separate copy of the message to the "To: and CC:" recipient and 
>> another to the "Bcc:" recipients.
>> Anyway, I'll make the header configurable with some macros (already 
>> in my TODO), so that the local admin can make the decision.
>
>
> Does this mean that Bcc: a@b.c and Bcc: d@e.f can see each other? Well 
> I don't know.


As stated in the RFC2822, it is up to the implementation to decide. This 
said I'm not aware of any implementation showing each BCC to each other.

>
> But here in germany the people of data privacy (not me :-)), wouldn't 
> even be glad about RCPT-specific headers regardless of To:, CC: and Bcc:.
>
> BTW: does anybody know how the postfix-users mailinglist works? This 
> is the list, where I first saw multiple X-Greylist: headers. So I know 
> someone else is getting this mail. For now I had to look in the
> postfix-logfile to see who the others are, but with RCPT-specific 
> headers every one can see.
>
> If the header is customizable (in future versions), there should be a 
> big warning in the config file.
>

This is the plan. The defaults won't show any "RCPT TO" value to be on 
the safe side.

Lionel.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Sascha L. <sas...@ru...>]

> Ok, adding only the first header would indeed being meaningful in the example 
> above. I was thinking of the future evolutions of SQLgrey and didn't 
> explained myself well.
>
> The above works only because we don't use AWLs based on recipients... *yet*. 
> In the future, SQLgrey will have an intermediate connect_awl table (with src, 
> sender and recipient) and a rcpt_awl (with only src, recipients) in addition

Ok. Thats what I didn't know :-).

Lionel: Thanks a lot for your long explanation. Can you tell me roughly 
when SQLgrey will have rcpt_awl?

BTW: I just want to say that I (and many mail-users) are happy with 
sqlgrey.

Thanks,

Sascha.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Lionel B. <lio...@bo...>]

Sascha Lucas wrote the following on 15.08.2005 10:27 :

>> Ok, adding only the first header would indeed being meaningful in the 
>> example above. I was thinking of the future evolutions of SQLgrey and 
>> didn't explained myself well.
>>
>> The above works only because we don't use AWLs based on recipients... 
>> *yet*. In the future, SQLgrey will have an intermediate connect_awl 
>> table (with src, sender and recipient) and a rcpt_awl (with only src, 
>> recipients) in addition
>
>
> Ok. Thats what I didn't know :-).
>
> Lionel: Thanks a lot for your long explanation. Can you tell me 
> roughly when SQLgrey will have rcpt_awl?
>

Depends on my free time... There's a lot to do in the 1.7.x branch and I 
may very well start by splitting the code into more manageable modules 
which will delay new functionnalities but then speed the development.

> BTW: I just want to say that I (and many mail-users) are happy with 
> sqlgrey.
>

Knowing that people benefit from my work is a good motivation for me :-)

Thanks!

Lionel.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Michael S. <Mic...@lr...>]

On Mon, 15 Aug 2005, Lionel Bouton wrote:

> Depends on my free time... There's a lot to do in the 1.7.x branch and I
> may very well start by splitting the code into more manageable modules
> which will delay new functionnalities but then speed the development.
>

Hi Lionel,

don't begin with splitting the code in different modules. This is nothing
for the 1.7-branch. I think for the new structure we should start a
2.0-branch, in parallel to the 1.X-branches, because it needs a lot of
time and testing to get the new structure back into production.

I already began with this restructuring to an OO-structure and put many
hours of my free time into it. But it is still far away from a prototype.

I think there is still a lot to do for the 1.X-branches.

Regards,
Michael Storz
-------------------------------------------------
Leibniz-Rechenzentrum   !   <mailto:St...@lr...>
Barer Str. 21           !   Fax: +49 89 2809460
80333 Muenchen, Germany !   Tel: +49 89 289-28840