sqlgrey-users

[Sqlgrey-users] Patch: Configurable rejection

[Michel B. <mi...@bo...>]

Hi there,

As it was on the to-do list and I was waiting for it, I wrote it ;-)

Please find attached a patch that adds to SQLgrey the choice of rejecting a 
message immediately (with 450) or to delay rejection (defer_if_permit).

The choices can be different for a "first time rejection" or an "early 
reconnection".

It seems to work plain good here, feedback welcome.

Cheers.

-- 
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] Patch: Configurable rejection

[Michel B. <mi...@bo...>]

Le Mercredi 16 F=E9vrier 2005 16:15, Michel Bouissou a =E9crit :
>
> [...] a patch that adds to SQLgrey the choice of rejecting a
> message immediately (with 450) or to delay rejection (defer_if_permit).
>
> The choices can be different for a "first time rejection" or an "early
> reconnection".
>
> It seems to work plain good here, feedback welcome.

It shows very efficient on an overall MTA performance standpoint : It all=
ows=20
me to organize my Postfix restrictions in the following order (partial=20
example):

- Local validation checks, such as:
reject_non_fqdn_recipient, reject_multi_recipient_bounce,
reject_non_fqdn_sender, etc.

- Local tables checks, such as:
check_client_access hash:/etc/postfix/combined_blacklist,
check_helo_access hash:/etc/postfix/combined_blacklist,
check_sender_access hash:/etc/postfix/combined_blacklist,
check_sender_access hash:/etc/postfix/sender_checks,

- # SQLgrey
check_policy_service inet:127.0.0.1:2501,
	=3D> With
		reject_first_attempt =3D immed
		reject_early_reconnect =3D delay

- "Slower" network checks, such as external DNSBL blacklists, SPF...
reject_unknown_sender_domain,
reject_rbl_client sbl.spamhaus.org,
reject_rbl_client xbl.spamhaus.org,
reject_rbl_client relays.ordb.org,
reject_spf_invalid_sender,

- Sender existence callback check:
reject_unverified_sender,

[...]

The fastest and less expensive checks being performed first, we try to re=
ject=20
unwanted messages at the lowest "cost".

SQLgrey is called _before_ performing slow network checks, which by the w=
ay=20
saves load onto external DNSBLs, which is nice for them.

SQLgrey is configured to reject immediately first connection attempts wit=
hout=20
going further, so most viruses / non-retrying spams will be rejected ther=
e=20
without bothering querying blacklists or checking sender existence.

In case of "early reconnections", we assume that if the same message came=
 back=20
once, it is probable that it will come back again. So SQLgrey now rejects=
=20
with delayed rejection, which allows checking blacklists (and refusing th=
e=20
message with a permanent 5xx code if the sending host is blacklisted), an=
d=20
performing sender existence callback check.

Now that we have this information (in DNS cache and Postfix address verif=
y=20
DB), when the message comes back once more after greylisting time is expi=
red,=20
it can be quickly and efficiently accepted.
     =20
Comments ? =20

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] Patch: Configurable rejection

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 17.02.2005 09:51 :

>[...]
>Comments ?  
>
>  
>

I just need to dump the details you just sent into the HOWTO, and test 
the integration of your code on one server.
I'll release a 1.4.5 with this and the fixes we discussed since 1.4.4. 
Then I think I'll move on 1.5.x beginning with database changes instead 
of releasing them for the stable 1.4 series.

Lionel.

Re: [Sqlgrey-users] Patch: Configurable rejection

[Michel B. <mi...@bo...>]

Le Jeudi 17 F=E9vrier 2005 10:52, Lionel Bouton a =E9crit :
>
> I'll release a 1.4.5 with this and the fixes we discussed since 1.4.4.
> Then I think I'll move on 1.5.x beginning with database changes instead
> of releasing them for the stable 1.4 series.

Would you consider integrating the index order changes in next 1.4x ? It =
is=20
important for sites having big SQLgrey DBs...

I suggest that you could put 2 entries in the "config" tables : The curre=
nt=20
one for the tables level, and maybe an "indexes" one that would only take=
=20
care of indexes level.

Because indexes can easily be dropped / recreated without having to touch=
 the=20
rest of the DB structure...

Anyway, if the indexes changes are not in 2.4.x, anybody who wishes to ch=
ange=20
them can do it manually directly on his SQL DB...

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] Patch: Configurable rejection

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 17.02.2005 11:04 :

>Le Jeudi 17 F=E9vrier 2005 10:52, Lionel Bouton a =E9crit :
> =20
>
>>I'll release a 1.4.5 with this and the fixes we discussed since 1.4.4.
>>Then I think I'll move on 1.5.x beginning with database changes instead
>>of releasing them for the stable 1.4 series.
>>   =20
>>
>
>Would you consider integrating the index order changes in next 1.4x ? It=
 is=20
>important for sites having big SQLgrey DBs...
>
>I suggest that you could put 2 entries in the "config" tables : The curr=
ent=20
>one for the tables level, and maybe an "indexes" one that would only tak=
e=20
>care of indexes level.
>
> =20
>

What problem are you trying to solve with this separate index level entry=
 ?
If SQLgrey must take into account every possible combination of these=20
entries, this will become awfully complex.

>Because indexes can easily be dropped / recreated without having to touc=
h the=20
>rest of the DB structure...
>
>Anyway, if the indexes changes are not in 2.4.x, anybody who wishes to c=
hange=20
>them can do it manually directly on his SQL DB...
>
> =20
>
Yep.

Let me explain how I plan to make the changes. As renaming columns=20
(which will be done) is platform specific and creating indexes could=20
conflict with the ones created by the DBA, I'll simply *copy* the tables=20
into new ones and drop the original ones.
On big databases where the migration could take several seconds or even=20
minutes I'd suggest disabling the policy service until the convertion is=20
complete (will be logged). DBAs will have to recreate additional indexes=20
they might have created (if they are not created by the new db version).

At first I wanted to make this migration in 1.4.x but the number of=20
changes needed rised significantly. As I must make sure the three=20
supported RDBMS will migrate smoothly, I prefer to use 1.5.x for this.

Lionel.

Re: [Sqlgrey-users] Patch: Configurable rejection

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 16.02.2005 16:15 :

>Hi there,
>
>As it was on the to-do list and I was waiting for it, I wrote it ;-)
>
>Please find attached a patch that adds to SQLgrey the choice of rejecting a 
>message immediately (with 450) or to delay rejection (defer_if_permit).
>
>The choices can be different for a "first time rejection" or an "early 
>reconnection".
>
>It seems to work plain good here, feedback welcome.
>
>  
>

Cool, I'm nearly finished including this in my tree. I took out the new
command-line parameters as nobody seemed to use them when I asked and
the code is too large and intrusive for this (I modified the pod doc to
mark the deprecated cmdline parameters).

Thanks a lot,

Lionel.