sqlgrey-users

[Sqlgrey-users] Excessive X-Greylist headers

[Alex <mys...@gm...>]

Hi,

It seems when a message is sent to multiple recipients, multiple
X-Greylist headers are added. When there are dozens of these, it
appears some systems are rejecting it due to the header being too
large.

How can I configure it to only include one copy of the X-Greylist header?

I'm using postfix. Is it possible to strip off all but one copy of the header?

X-Greylist: whitelisted by SQLgrey-1.8.0

Thanks,
Alex

Re: [Sqlgrey-users] Excessive X-Greylist headers

[Karl O. P. <ko...@me...>]

On Mon, 8 Jun 2015 17:12:22 -0400
Alex <mys...@gm...> wrote:

> Hi,
> 
> It seems when a message is sent to multiple recipients, multiple
> X-Greylist headers are added. When there are dozens of these, it
> appears some systems are rejecting it due to the header being too
> large.
> 
> How can I configure it to only include one copy of the X-Greylist
> header?
> 
> I'm using postfix. Is it possible to strip off all but one copy of
> the header?
> 
> X-Greylist: whitelisted by SQLgrey-1.8.0

Is this inbound, outbound, or relayed mail?



Karl <ko...@me...>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

Re: [Sqlgrey-users] Excessive X-Greylist headers

[Alex R. <mys...@gm...>]

Hi Karl,

>> It seems when a message is sent to multiple recipients, multiple
>> X-Greylist headers are added. When there are dozens of these, it
>> appears some systems are rejecting it due to the header being too
>> large.
>>
>> How can I configure it to only include one copy of the X-Greylist
>> header?
>>
>> I'm using postfix. Is it possible to strip off all but one copy of
>> the header?
>>
>> X-Greylist: whitelisted by SQLgrey-1.8.0
>
> Is this inbound, outbound, or relayed mail?

This is from the headers on the destination system. It is received by 
our mail relay, where the header is added, then sent to the destination 
system where the user reads the mail.

The mail is apparently sent to dozens of recipients via a mail expander 
at the corporate office to recipients on this destination system because 
there is no indication in the email that it was sent to dozens of 
recipients. Apparently the relay is adding an X-Greylist header for each 
of the recipients in the email.

I've included a copy of the email here:

http://pastebin.com/1reiWEM3

Any ideas would be greatly appreciated.
Thanks,
Alex

Re: [Sqlgrey-users] Excessive X-Greylist headers

[Karl O. P. <ko...@me...>]

On Tue, 09 Jun 2015 09:51:53 -0400
Alex Regan <mys...@gm...> wrote:

> Hi Karl,
> 
> >> It seems when a message is sent to multiple recipients, multiple
> >> X-Greylist headers are added. When there are dozens of these, it
> >> appears some systems are rejecting it due to the header being too
> >> large.
> >>
> >> How can I configure it to only include one copy of the X-Greylist
> >> header?
> >>
> >> I'm using postfix. Is it possible to strip off all but one copy of
> >> the header?
> >> 
> >> X-Greylist: whitelisted by SQLgrey-1.8.0
> >
> > Is this inbound, outbound, or relayed mail?
> 
> This is from the headers on the destination system. It is received by 
> our mail relay, where the header is added, then sent to the
> destination system where the user reads the mail.

You shouldn't be relaying except from trusted systems,
so your smtpd_relay_restrictions should include
something like "permit_mynetworks", in addition
to your check_policy_service that runs sqlgrey.
Then relayed mail won't go through sqlgrey.
(Use smtpd_recipient_restrictions on older
versions of postfix.)

Whitelisting via sqlgrey is the wong approach.

But it looks to me like you've got something
else wrong since I wouldn't think, without
having really investigated, that any piece of
mail should go through sqlgrey more than once.
So you should only get one header no matter what.


Karl <ko...@me...>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

Re: [Sqlgrey-users] Excessive X-Greylist headers

[Alex R. <mys...@gm...>]

Hi,

>>>> X-Greylist: whitelisted by SQLgrey-1.8.0
>>>
>>> Is this inbound, outbound, or relayed mail?
>>
>> This is from the headers on the destination system. It is received by
>> our mail relay, where the header is added, then sent to the
>> destination system where the user reads the mail.
>
> You shouldn't be relaying except from trusted systems,
> so your smtpd_relay_restrictions should include
> something like "permit_mynetworks", in addition
> to your check_policy_service that runs sqlgrey.
> Then relayed mail won't go through sqlgrey.
> (Use smtpd_recipient_restrictions on older
> versions of postfix.)

The check_policy_service is in smtpd_recipient_restrictions. Is that not 
correct?

The smtpd_relay_restrictions is as follows:

smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, 
defer_unauth_destination

This is on postfix-2.10.

> Whitelisting via sqlgrey is the wong approach.

It looks like this was auto-whitelisted, because it has seen this 
IP/email combination previously, no?

> But it looks to me like you've got something
> else wrong since I wouldn't think, without
> having really investigated, that any piece of
> mail should go through sqlgrey more than once.
> So you should only get one header no matter what.

Yes, I agree. It would be great to figure out how to configure it so 
trusted networks weren't greylisted.

Sure appreciate any ideas you might have.
Thanks,
Alex

Re: [Sqlgrey-users] Excessive X-Greylist headers

[Karl O. P. <ko...@me...>]

On Tue, 09 Jun 2015 10:49:32 -0400
Alex Regan <mys...@gm...> wrote:

> Hi,
> 
> >>>> X-Greylist: whitelisted by SQLgrey-1.8.0
> >>>
> >>> Is this inbound, outbound, or relayed mail?
> >>
> >> This is from the headers on the destination system. It is received
> >> by our mail relay, where the header is added, then sent to the
> >> destination system where the user reads the mail.
> >
> > You shouldn't be relaying except from trusted systems,
> > so your smtpd_relay_restrictions should include
> > something like "permit_mynetworks", in addition
> > to your check_policy_service that runs sqlgrey.
> > Then relayed mail won't go through sqlgrey.
> > (Use smtpd_recipient_restrictions on older
> > versions of postfix.)
> 
> The check_policy_service is in smtpd_recipient_restrictions. Is that
> not correct?

Yes, that's right.

> 
> The smtpd_relay_restrictions is as follows:
> 
> smtpd_relay_restrictions = permit_mynetworks,
> permit_sasl_authenticated, defer_unauth_destination

I greylist here too.  Keeps more spam out.

> 
> This is on postfix-2.10.
> 
> > Whitelisting via sqlgrey is the wong approach.
> 
> It looks like this was auto-whitelisted, because it has seen this 
> IP/email combination previously, no?

Then it must be that the email is exiting postfix and
passing through amavis and being re-injected back
into postfix.  You'll need to make sure that amavis
is either on a trusted network (127.0.0.1) or
uses lmtp not smtp or has some other way of getting the mail back
into postfix that does not get greylisted.

I'm not really focusing here so I could be off
on the wrong track.  But this is what it looks
like offhand.  (And I don't remember anything about amavis
any more.)


Karl <ko...@me...>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

Re: [Sqlgrey-users] Excessive X-Greylist headers

[Alex R. <mys...@gm...>]

Hi,

>> The smtpd_relay_restrictions is as follows:
>>
>> smtpd_relay_restrictions = permit_mynetworks,
>> permit_sasl_authenticated, defer_unauth_destination
>
> I greylist here too.  Keeps more spam out.

Okay, interesting. I'll try that too.

>> This is on postfix-2.10.
>>
>>> Whitelisting via sqlgrey is the wong approach.
>>
>> It looks like this was auto-whitelisted, because it has seen this
>> IP/email combination previously, no?
>
> Then it must be that the email is exiting postfix and
> passing through amavis and being re-injected back
> into postfix.  You'll need to make sure that amavis
> is either on a trusted network (127.0.0.1) or
> uses lmtp not smtp or has some other way of getting the mail back
> into postfix that does not get greylisted.

All of the networks involved are on the amavis "TRUSTED" or in @mynetworks.

I'm not doubting what you're saying, but here's a thread that seems to 
indicate it's the expected behavior:

http://lists.policyd.org/pipermail/users_lists.policyd.org/2007-April/001426.html

Here's another one from Lionel discussing:

http://sourceforge.net/p/sqlgrey/mailman/sqlgrey-users/thread/42F...@bo.../

Could that be the solution here as well?

I'm going to have to use postfix to strip them because I believe it's 
causing some mail to be rejected because the headers are too large.

Thanks again,
Alex

Re: [Sqlgrey-users] Excessive X-Greylist headers

[Karl O. P. <ko...@me...>]

On Tue, 09 Jun 2015 18:18:29 -0400
Alex Regan <mys...@gm...> wrote

> >> The smtpd_relay_restrictions is as follows:
> >>
> >> smtpd_relay_restrictions = permit_mynetworks,
> >> permit_sasl_authenticated, defer_unauth_destination
> >
> > I greylist here too.  Keeps more spam out.
> 
> Okay, interesting. I'll try that too.

On reflection, no, this is not a good idea
in the general case.  You need to be careful
with this and if you're not using it now
I'd be leery about adding it.





Karl <ko...@me...>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

Re: [Sqlgrey-users] Excessive X-Greylist headers

[Karl O. P. <ko...@me...>]

On Tue, 09 Jun 2015 18:18:29 -0400
Alex Regan <
> >> This is on postfix-2.10.
> >>
> >>> Whitelisting via sqlgrey is the wong approach.
> >>
> >> It looks like this was auto-whitelisted, because it has seen this
> >> IP/email combination previously, no?
> >
> > Then it must be that the email is exiting postfix and
> > passing through amavis and being re-injected back
> > into postfix.  You'll need to make sure that amavis
> > is either on a trusted network (127.0.0.1) or
> > uses lmtp not smtp or has some other way of getting the mail back
> > into postfix that does not get greylisted.
> 
> All of the networks involved are on the amavis "TRUSTED" or in
> @mynetworks.
> 
> I'm not doubting what you're saying, but here's a thread that seems
> to indicate it's the expected behavior:

Looks like what I'm saying is wrong.  The policy
daemon is called for each destination address.
See the postfix SMTPD_POLICY_README.

> Here's another one from Lionel discussing:
> 
> http://sourceforge.net/p/sqlgrey/mailman/sqlgrey-users/thread/42F...@bo.../
> 
> Could that be the solution here as well?

"That" being rcpt_awl?  Dunno.

Looks like as of postfix 2.1 there is an "instance"
variable passed to the policy daemon that could be used
to keep from adding multiple header lines.  It's not
clear how this would work and it's not implemented
in sqlgrey.

Something for somebody's todo list.

> 
> I'm going to have to use postfix to strip them because I believe it's 
> causing some mail to be rejected because the headers are too large.

That sounds like a sensible approach.

Since the policy check is smtpd you ought to be able
to use header_checks = pcre:/etc/postfix/header_checks
with file content of:

/^X-Greylist: whitelisted by SQLgrey/ IGNORE

I'd bet there's an even more clever way to
"collapse" all the identical entries but
haven't investigated.

Regards,



Karl <ko...@me...>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

Re: [Sqlgrey-users] Excessive X-Greylist headers

[Alex R. <mys...@gm...>]

Hi Karl,

>> I'm not doubting what you're saying, but here's a thread that seems
>> to indicate it's the expected behavior:
>
> Looks like what I'm saying is wrong.  The policy
> daemon is called for each destination address.
> See the postfix SMTPD_POLICY_README.
>
>> Here's another one from Lionel discussing:
>>
>> http://sourceforge.net/p/sqlgrey/mailman/sqlgrey-users/thread/42F...@bo.../
>>
>> Could that be the solution here as well?
>
> "That" being rcpt_awl?  Dunno.
>
> Looks like as of postfix 2.1 there is an "instance"
> variable passed to the policy daemon that could be used
> to keep from adding multiple header lines.  It's not
> clear how this would work and it's not implemented
> in sqlgrey.
>
> Something for somebody's todo list.

Do you think sqlgrey is still the best option for doing greylisting with 
postfix, or are is there something else we should be looking it?

I originally thought policyd was an alternative, but it looks to be more 
of a front-end or management system that just uses apps like sqlgrey.

>> I'm going to have to use postfix to strip them because I believe it's
>> causing some mail to be rejected because the headers are too large.
>
> That sounds like a sensible approach.
>
> Since the policy check is smtpd you ought to be able
> to use header_checks = pcre:/etc/postfix/header_checks
> with file content of:
>
> /^X-Greylist: whitelisted by SQLgrey/ IGNORE
>
> I'd bet there's an even more clever way to
> "collapse" all the identical entries but
> haven't investigated.

Yes, that's exactly how I'm currently doing it.

Thanks again,
Alex

Re: [Sqlgrey-users] Excessive X-Greylist headers

[Karl O. P. <ko...@me...>]

On Wed, 10 Jun 2015 13:18:49 -0400
Alex Regan <mys...@gm...> wrote:

> Hi Karl,

> > Looks like as of postfix 2.1 there is an "instance"
> > variable passed to the policy daemon that could be used
> > to keep from adding multiple header lines.  It's not
> > clear how this would work and it's not implemented
> > in sqlgrey.
> >
> > Something for somebody's todo list.
> 
> Do you think sqlgrey is still the best option for doing greylisting
> with postfix, or are is there something else we should be looking it?
> 
> I originally thought policyd was an alternative, but it looks to be
> more of a front-end or management system that just uses apps like
> sqlgrey.
> 

I don't have an informed option, not really knowing what
else is out there.  My guess is that if you have something
working it's not really worth the time spent to switch.

I like sqlgrey because it works with Postgres.  Not only
do I think Postgres technically superior to MySQL but
MySQL, in my opinion, is not truly Open Source.  The
MySQL documentation (and as far as I can tell the
MariaDB documentation) is _not_ open source.  A database
is no good without documentation, which makes forking
a db product with no documentation a huge problem.

It took MariaDB _years_ to get decent documentation.
(And near as I can tell they still don't have
a document for each release, making it hard to tell
just how the version you happen to be running
actually works.)

If it can't be forked it's not open source.


Karl <ko...@me...>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein