sqlgrey-users

[Sqlgrey-users] Comments and features requests

[Michel B. <mi...@bo...>]

Hi there,

I've started using SQLgrey a couple of days ago, and I'm rather happy with it.

I have however a couple of comments and feature requests :

1/ In the provided "clients_fqdn_whitelist", the following line:
scd.yahoo.com
is wrong, does not work, and should be replaced with
*.scd.yahoo.com

2/ The latest version of Postgrey allows specifying the result (both code and 
text) that Postgrey should give when a connection is greylisted. It would be 
nice if SQLgrey could do the same.
Currently SQLgrey returns "defer_if_permit" and this cannot be modified 
(except by fiddling in the code).
However, some may prefer to reject immediately with a 450 temporary failure, 
rather than carry on with further potentially expensive Postfix tests (such 
as checking external DNSBLs or verifying sender existence), where the 
connection will in the end be rejected anyway...

3/ It would be nice to have a whitelist of senders (using regexps) for whom 
greylisting should never be done. This list should by default include the 
addresses that some MTA use to perform "sender address verification", as we 
should avoid greylisting those verifications as much as possible, the 
combination of greylisting and sender address verification introducing 
problems and rather long delays. These verification origin addresses are, 
among the most common:
<> (null sender)
MAILER-DAEMON@*
postmaster@*


Thanks for this excellent piece of software.

Regards.

-- 
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] Comments and features requests

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 01/10/2005 09:26 AM :

>Hi there,
>
>I've started using SQLgrey a couple of days ago, and I'm rather happy with it.
>
>I have however a couple of comments and feature requests :
>
>1/ In the provided "clients_fqdn_whitelist", the following line:
>scd.yahoo.com
>is wrong, does not work, and should be replaced with
>*.scd.yahoo.com
>  
>

Thanks. Commited in CVS.

>2/ The latest version of Postgrey allows specifying the result (both code and 
>text) that Postgrey should give when a connection is greylisted. It would be 
>nice if SQLgrey could do the same.
>Currently SQLgrey returns "defer_if_permit" and this cannot be modified 
>(except by fiddling in the code).
>However, some may prefer to reject immediately with a 450 temporary failure, 
>rather than carry on with further potentially expensive Postfix tests (such 
>as checking external DNSBLs or verifying sender existence), where the 
>connection will in the end be rejected anyway...
>
>  
>

I suppose you use a 550 rejection at least with DNSBLs. In this case 
you'll lose time by forcing a temporary failure instead of rejecting on 
the first connection.
With sender verification it can be more complex, I'm not sure of the 
cases where 450 and 550 are used. If the sender verification use 550, 
you end up in the same case than the one described above : you lose time 
and performance if a 450 is used then it will save one 450 but not the 
subsequent ones.

So in the end, I'm not sure you will gain much from it or if in fact you 
won't lose something...
But it's a simple thing to add, so if you find a case where it clearly 
is beneficial, I'll code it.

>3/ It would be nice to have a whitelist of senders (using regexps) for whom 
>greylisting should never be done. This list should by default include the 
>addresses that some MTA use to perform "sender address verification", as we 
>should avoid greylisting those verifications as much as possible, the 
>combination of greylisting and sender address verification introducing 
>problems and rather long delays. These verification origin addresses are, 
>among the most common:
><> (null sender)
>MAILER-DAEMON@*
>postmaster@*
>  
>

I don't think greylisting and sender address verification clash with 
each other. The common problem is the following :
Server A uses greylisting, Server B uses sender address verification.
- Server A formard a mail to Server B,
- Server B wants to check that the sender exists and Server A is the MX 
for the corresponding domain, Server A is not in the auto-whitelists of 
Server B
- Server A don't trust Server B yet, so Server B verification fails and 
(temporarily) refuses the mail,
- Server B will retry (around 20 minutes later with default Postfix 
configuration under light load), then Server A will be able to verify 
(and will be for 60 days and more with default SQLgrey config), the mail 
gets through.

In the end what happens is that you more or less apply greylisting on 
the "outgoing" mails where the destination uses sender verification as 
well as the "incoming" mails. With auto-whitelisting, you shouldn't 
notice much of a difference.

The problem with adding hard whitelists for origin addressses is that 
you will have many SPAMs coming through (postmaster, MAILER-DAEMON and 
<> are used rather often).

In conclusion, I think auto-whitelisting should take care of the problem 
efficiently and adding sender whitelists will bring more harm than good.
If you find odd installations where sender verification is poorly 
implemented (for example using a 550 insted of a 450) at the point of 
breaking greylisting, I believe you should add them to the whitelists 
server by server (these would be broken SMTP configurations, so it 
should be the right place), warn their postmasters and post the cases on 
this mailing-list.

>
>Thanks for this excellent piece of software.
>
>  
>

You are welcomed. Thanks for the comments.

Lionel.

Re: [Sqlgrey-users] Comments and features requests

[Michel B. <mi...@bo...>]

Le Lundi 10 Janvier 2005 15:08, Lionel Bouton a =E9crit :
>
> >2/ The latest version of Postgrey allows specifying the result (both code
> > and text) that Postgrey should give when a connection is greylisted. It
> > would be nice if SQLgrey could do the same.
> >Currently SQLgrey returns "defer_if_permit" and this cannot be modified
> >(except by fiddling in the code).
> >However, some may prefer to reject immediately with a 450 temporary
> > failure, rather than carry on with further potentially expensive Postfix
> > tests (such as checking external DNSBLs or verifying sender existence),
> > where the connection will in the end be rejected anyway...
>
> I suppose you use a 550 rejection at least with DNSBLs. In this case
> you'll lose time by forcing a temporary failure instead of rejecting on
> the first connection.

Let me explain my view : The huge majority of trash that my mailserver=20
currently receives comes from zombie virus-sending Windows machines, or=20
spambots that don't have a normal retry pattern. The whole point in=20
greylisting is precisely to eliminate them, because they will *NOT* retry. =
So=20
rejecting their mail attempt with any code, either 450 or 550 results in th=
e=20
same effect : They don't come back (at least using the same from/to/IP).

I personally prefer to put first in my Postfix series of tests all the test=
s=20
that can be "local and quick", and only for messages that passes these,=20
perform tests that may be "longer or network or resources consuming", such =
as=20
sender address verification.

That's why I'd like that a "Greylisted" event could immediately reject the=
=20
mail on first attempt (I wouldn't mind a "defer_if_permit" on an "early=20
reconnection" BTW, because it would allow me to check sender now that it is=
=20
probable that the same mail will come back again).

=46urthermore, if Greylisting caused an immediate reject, it would permit m=
e not=20
to uselessly bother remote servers with address verification for addresses=
=20
that are dummy, and messages that won't come back.

=46urthermore, I prefer not to "pollute" my address verification DB where i=
t can=20
be avoided, to prevent it from growing too much. SQLgrey uses a nice=20
PostgreSQL database on my system, it's size is no problem and it can be=20
easily maintained (vacuum analyze etc...), which is not the case of the=20
Postfix address verification daemon DB, which uses a simpler BerkeleyDB.

Last but not least, I don't mind rejecting first with a 450 a spam that wil=
l=20
actually come back and that I will reject with a 554 on 2nd attempt (origin=
=20
blacklisted, non-existent sender...). By rejecting the message twice, I the=
n=20
put a double load on the spammer's server, and I don't hate the idea of=20
giving them more work to get rejected twice ;-)

> With sender verification it can be more complex, I'm not sure of the
> cases where 450 and 550 are used.

Basically the Postfix verification daemon gives the same family of error th=
at=20
the remote server gave when performing verification. If the verification=20
failed with a 5xx, Postfix will issue a 5xx. If the verification failed wit=
h=20
a 4xx (or could not be done), Postfix will issue a 4xx.

> If the sender verification use 550,=20
> you end up in the same case than the one described above : you lose time
> and performance if a 450 is used then it will save one 450 but not the
> subsequent ones.
>
> So in the end, I'm not sure you will gain much from it or if in fact you
> won't lose something...
> But it's a simple thing to add, so if you find a case where it clearly
> is beneficial, I'll code it.

Thanks :-)


> >3/ It would be nice to have a whitelist of senders (using regexps) for
> > whom greylisting should never be done. This list should by default
> > include the addresses that some MTA use to perform "sender address
> > verification", as we should avoid greylisting those verifications as mu=
ch
> > as possible, the combination of greylisting and sender address
> > verification introducing problems and rather long delays. These
> > verification origin addresses are, among the most common:
> ><> (null sender)
> >MAILER-DAEMON@*
> >postmaster@*
>
> I don't think greylisting and sender address verification clash with
> each other.

=46rom my experience they actually do.

> The common problem is the following :=20
> Server A uses greylisting, Server B uses sender address verification.
> - Server A formard a mail to Server B,
> - Server B wants to check that the sender exists and Server A is the MX
> for the corresponding domain, Server A is not in the auto-whitelists of
> Server B
> - Server A don't trust Server B yet, so Server B verification fails and
> (temporarily) refuses the mail,
> - Server B will retry (around 20 minutes later with default Postfix
> configuration under light load), then Server A will be able to verify
> (and will be for 60 days and more with default SQLgrey config), the mail
> gets through.

No, because server B will probably *NOT* verify again so quickly. Server B=
=20
will probably have a "negative cache TTL" telling it not to retry a failed=
=20
verification before a given delay (I personally use 3 hours delay) to avoid=
=20
repeteadly bothering remote servers with possible repeated requests for the=
=20
same non-existent address.

If this scheme, what happens is as follows:
=2D Server A wants to send mail to server B
=2D Server B tries to verify sender address on server A, but gets greyliste=
d,=20
and its verification receives a "450".
=2D Server B keeps this 450 in cache for its "negative TTL time" (let's say=
 3=20
hours).
=2D Server A will retry several times to send its mail to server B, and wil=
l be=20
refused with 450 immediately as long as the sender address verification=20
negative TTL isn't expired.
=2D After at least 3 hours (but some sending servers may have retry delays =
that=20
grows exponentially, such as qmail, so it could well be 8 or 10 hours),=20
server A will retry, server B will retry its verification, and, provided=20
server A satisfied the verification fast enough, the mail will in the end b=
e=20
accepted. If server A is a bit long to answer, the verification will timeou=
t=20
and be finished asychronously, the mail will be refused once more, and serv=
er=20
A will have to retry once more to send it.

With the result that on average, the mail from A to B will not have been=20
delayed by "5 minutes or so" by the greylisting made on A, but rather by 4=
=20
hours or more (after a dozen of attempts), by the combination of greylistin=
g=20
on A, verification on B, and negative TTL on B's verification.

Well, 4 hours or so, that's becoming a delay long enough to cause trouble t=
o=20
many sysadmins ;-))

> The problem with adding hard whitelists for origin addressses is that
> you will have many SPAMs coming through (postmaster, MAILER-DAEMON and
> <> are used rather often).

Not that often on my experience. "postmaster" has actually been used by rec=
ent=20
worm-viruses (as well as hostmaster and webmaster), but I have seen very fe=
w=20
spams or viruses "From: <>" and even less "From: MAILER-DAEMON@" (talking=20
about SMTP enveloppe Froms).

Thanks again for SQLgrey, I like it ;-)

=2D-=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Les taxis de la Marne ont fait la guerre.
Alors moi je rigole quand je vois un taxi qui a peur d'aller en banlieue.

Re: [Sqlgrey-users] Comments and features requests

[Lionel B. <lio...@bo...>]

Postfix-users is CCed as I propose a change in the verify daemon later 
on in this mail, please skip to 2/ if you don't care about SQLgrey 
specific questions but only for greylisting + address verification 
behaviour.

# 1/ greylister return value

Michel Bouissou wrote the following on 01/10/2005 03:46 PM :

>
>Let me explain my view : The huge majority of trash that my mailserver 
>currently receives comes from zombie virus-sending Windows machines, or 
>spambots that don't have a normal retry pattern. The whole point in 
>greylisting is precisely to eliminate them, because they will *NOT* retry. So 
>rejecting their mail attempt with any code, either 450 or 550 results in the 
>same effect : They don't come back (at least using the same from/to/IP).
>
>I personally prefer to put first in my Postfix series of tests all the tests 
>that can be "local and quick", and only for messages that passes these, 
>perform tests that may be "longer or network or resources consuming", such as 
>sender address verification.
>  
>

Makes sense to me.

>That's why I'd like that a "Greylisted" event could immediately reject the 
>mail on first attempt (I wouldn't mind a "defer_if_permit" on an "early 
>reconnection" BTW, because it would allow me to check sender now that it is 
>probable that the same mail will come back again).
>
>  
>

This last one is a little more complex but I agree it can be helpful.

>Furthermore, if Greylisting caused an immediate reject, it would permit me not 
>to uselessly bother remote servers with address verification for addresses 
>that are dummy, and messages that won't come back.
>
>Furthermore, I prefer not to "pollute" my address verification DB where it can 
>be avoided, to prevent it from growing too much. SQLgrey uses a nice 
>PostgreSQL database on my system, it's size is no problem and it can be 
>easily maintained (vacuum analyze etc...), which is not the case of the 
>Postfix address verification daemon DB, which uses a simpler BerkeleyDB.
>
>Last but not least, I don't mind rejecting first with a 450 a spam that will 
>actually come back and that I will reject with a 554 on 2nd attempt (origin 
>blacklisted, non-existent sender...). By rejecting the message twice, I then 
>put a double load on the spammer's server, and I don't hate the idea of 
>giving them more work to get rejected twice ;-)
>  
>

Ok you convinced me. Added to my TODO list for 1.4.x.

# 2/ address verification

>
>  
>
>>The common problem is the following : 
>>Server A uses greylisting, Server B uses sender address verification.
>>- Server A formard a mail to Server B,
>>- Server B wants to check that the sender exists and Server A is the MX
>>for the corresponding domain, Server A is not in the auto-whitelists of
>>Server B
>>- Server A don't trust Server B yet, so Server B verification fails and
>>(temporarily) refuses the mail,
>>- Server B will retry (around 20 minutes later with default Postfix
>>configuration under light load), then Server A will be able to verify
>>(and will be for 60 days and more with default SQLgrey config), the mail
>>gets through.
>>    
>>
>
>No, because server B will probably *NOT* verify again so quickly. Server B 
>will probably have a "negative cache TTL" telling it not to retry a failed 
>verification before a given delay (I personally use 3 hours delay) to avoid 
>repeteadly bothering remote servers with possible repeated requests for the 
>same non-existent address.
>  
>

Negative caches should not be this high if the return code is 45x when 
validating senders. I can understand high TTL for 55x, but remember that 
greylisting only simulates a temporary unavailability of the MX, if the 
mail is delayed for several hours when a server is unavailable for a 
couple of minutes there is a defect in the configuration IMHO.

The problem is more with sender verification configurability than with 
greylisting in my opinion.

sender verification is a risky thing to do btw, you should only call 
verify on known-to-be forged domain where you are pretty sure you can 
verify the Return-Path of each and every valid mail.

Could Postfix's verify daemon be updated to have a separate (lower) 
negative cache ttl for 45x replies ? Seems the right thing to do as the 
informations stored in cache have very different meanings in the 45x and 
55x cases.

>If this scheme, what happens is as follows:
>- Server A wants to send mail to server B
>- Server B tries to verify sender address on server A, but gets greylisted, 
>and its verification receives a "450".
>- Server B keeps this 450 in cache for its "negative TTL time" (let's say 3 
>hours).
>- Server A will retry several times to send its mail to server B, and will be 
>refused with 450 immediately as long as the sender address verification 
>negative TTL isn't expired.
>- After at least 3 hours (but some sending servers may have retry delays that 
>grows exponentially, such as qmail, so it could well be 8 or 10 hours),
>

in qmail's case, it shouldn't grow much since the 2nd verification 
attempt will most probably succeed.

Best regards,

Lionel.

Re: [Sqlgrey-users] Comments and features requests

[Michel B. <mi...@bo...>]

Le Lundi 10 Janvier 2005 17:19, Lionel Bouton a =E9crit :
>
> Negative caches should not be this high if the return code is 45x when
> validating senders. I can understand high TTL for 55x, but remember tha=
t
> greylisting only simulates a temporary unavailability of the MX, if the
> mail is delayed for several hours when a server is unavailable for a
> couple of minutes there is a defect in the configuration IMHO.

I fully agree with this.

> The problem is more with sender verification configurability than with
> greylisting in my opinion.

Yes, but we also have to live with things "as they are" rather than "as t=
hey=20
should be", until of course these things are fixed ;-)

Many Linux distributions come with Postfix packages and many users aren't=
=20
going to install a possibly updated Postfix until the updated package is =
in=20
their favourite distro. Not everybody is of the patch-and-compile-myself=20
sort ;-)

OTOH, when one adds a separate greylisting daemon to his existing Postifx=
=20
setup, he'd like it to integrate as well as possible with the verificatio=
n=20
daemon, not "as it should be", but "as it is" ;-)

Furthermore, the problem is not how our greylisting system may affect *ou=
r*=20
own verification daemon, but how it will affect remote server's verificat=
ion=20
daemons when they try to verify an address from our site. And we have no=20
control upon how these daemons, if any, work. We don't know how the remot=
e=20
sites are configured. So the best thing we can do it to keep as much as=20
possible our greylisting system out of the way of these remote verificati=
on=20
daemons.

> Could Postfix's verify daemon be updated to have a separate (lower)
> negative cache ttl for 45x replies ? Seems the right thing to do as the
> informations stored in cache have very different meanings in the 45x an=
d
> 55x cases.

True, and that would be great. Meanwhile, we have to work around this=20
feature ;-)

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] Comments and features requests

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 01/10/05 19:32 :

>[...]
>
>Furthermore, the problem is not how our greylisting system may affect *our* 
>own verification daemon, but how it will affect remote server's verification 
>daemons when they try to verify an address from our site. And we have no 
>control upon how these daemons, if any, work. We don't know how the remote 
>sites are configured.
>

If they want to receive/send mails in a timely manner, they should be 
configured to cope well with temporary failures, after all *they* decide 
to refuse the mail.
sender address verification isn't something people should use blindly, 
if you do so you block some mails the recipient are willing to receive. 
I'm not willing to poke holes in the greylisting process just to cope 
with what I still think is a (small) defect in the verify daemon. In 
fact if you really want to, you can configure Postfix itself to not 
greylist mails coming from the verification addresses.

If there really is a need, I can write how to configure Postfix to do so 
in SQLgrey's HOWTO.

Unless there are some new hard arguments for adding address-based 
whitelists to SQLgrey and not modifying the verify daemon, let us keep 
this on sqlgrey-users in the next messages.

Best regards,

Lionel.

Re: [Sqlgrey-users] Comments and features requests

[Michel B. <mi...@bo...>]

Le Lundi 10 Janvier 2005 21:22, Lionel Bouton a =E9crit :
>
> sender address verification isn't something people should use blindly,
> if you do so you block some mails the recipient are willing to receive.

Well, I've actually been using sender address verification on several ser=
vers=20
in different environments for months, and it works rather good. The sende=
r=20
addresses that cannot be verified but yet are "legitimate" are very rare,=
 and=20
in any case it's a sender system misconfiguration issue, generally for so=
me=20
automated mail systems such as newsletter or website generated mail.

My own opinion is that any mail should come with an envelope-sender which=
 is=20
replyable (bounceable), and, if not, the mail shouldn't be accepted. For =
this=20
reason sender address verification makes much sense for me.

> I'm not willing to poke holes in the greylisting process just to cope
> with what I still think is a (small) defect in the verify daemon. In
> fact if you really want to, you can configure Postfix itself to not
> greylist mails coming from the verification addresses.

Yes, I have already configured a workaround in Postfix. But I don't belie=
ve=20
that adding configuration flexibility in a tool is "poking holes" into it=
. I=20
believe that a tool should be as configurable and flexible as possible,=20
according to users demands, and provided it is feasible and the developpe=
rs=20
have time to develop the requested features ;-)

I'd rather have an sqlgrey that manages sender whitelisting by itself rat=
her=20
than having to have to configure a workaround in Postfix, because the=20
workaround in Postfix "pokes bigger holes" in the filtering than what it=20
would if done only at sqlgrey level.

I believe that each filter / policy server should be as configurable as=20
possible,  leaving the the sysadmin the choice of what should be processe=
d or=20
not by any given filter.

Cheers.

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E