sqlgrey-users

[Sqlgrey-users] Using SQLgrey in an ISP environment

[Josh E. <jo...@en...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

I'm wondering if anyone out there uses sqlgrey (or any greylisting
policies) in a large/ISP environment, and what success/problems
they've had. I've been thinking about deploying sqlgrey (due to the
feature of one backend and multiple nodes), but in my testing on my
private server I found that the delays were often annoying and
didn't seem to stop. Maybe I didn't set it up correctly or
something... :/ I'll test it again sometime.

Anyway, the main concern I have is that users will not see email
immediately as most are accustomed to. Unfortunately this seems to
be a greylisting downfall, not sqlgrey's, and I'm just curious if
anyone has deployed this on a large scale and if they've run into
problems or if people are complaining, etc., or any ideas on the
matter. I realize it could be bad for businesses, but it's effect on
spam is great. :)

Thanks,
Josh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBvxd4V/+PyAj2L+IRAqGKAJ9P1LJ6M1P2WSArxck4uQQxsmbBLACgnWDS
wUpRDLUqqt79pWxWa6PwrfQ=
=02o1
-----END PGP SIGNATURE-----

Re: [Sqlgrey-users] Using SQLgrey in an ISP environment

[Lionel B. <lio...@bo...>]

Josh Endries wrote the following on 12/14/04 17:40 :

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello,
>
> I'm wondering if anyone out there uses sqlgrey (or any greylisting
> policies) in a large/ISP environment, and what success/problems
> they've had. I've been thinking about deploying sqlgrey (due to the
> feature of one backend and multiple nodes), but in my testing on my
> private server I found that the delays were often annoying and
> didn't seem to stop. Maybe I didn't set it up correctly or
> something... :/ I'll test it again sometime.


For one isolated user it takes a long time for auto-whitelist to kick in 
(especially SQLgrey's domain-based one) because the greylister can't 
learn from trafic to other users. You'll have a much better 
auto-whitelist usage in an ISP environment.

>
> Anyway, the main concern I have is that users will not see email
> immediately as most are accustomed to. Unfortunately this seems to
> be a greylisting downfall, not sqlgrey's, and I'm just curious if
> anyone has deployed this on a large scale and if they've run into
> problems or if people are complaining, etc., or any ideas on the
> matter. I realize it could be bad for businesses, but it's effect on
> spam is great. :)


The best way is to let the user decides if it wants to use greylisting 
(or even make them pay for it :-)).
I think you can already use postfix to selectively use greylisting, see 
the postfix online documentation, especially the chapter where it is 
configured to greylist only specific source domains.

I can add opt-in and opt-out support if needed. This could work like this :
- default : current behaviour,
- --opt-in : the RCPT TO: must be in a "optin" table for greylisting to 
be used,
- --opt-out : the RCPT TO: must *not* be in a "optout" table for 
greylisting to be used.

Caveat : IIRC the policy daemon is called *before* alias expansion. If 
you have mailing-lists and/or several aliases for the same users, you'll 
have to take this into consideration when populating optin or optout tables.

Then an ISP can launch sqlgrey in "optin" or "optout" mode and add to 
its web interfaces some configuration pages that will allow its users to 
subscribe to the service by adding/removing entries in the correct 
sqlgrey tables.

Best regards,

Lionel.

Re: [Sqlgrey-users] Using SQLgrey in an ISP environment

[Josh E. <jo...@en...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lionel Bouton wrote:
| For one isolated user it takes a long time for auto-whitelist to
kick in
| (especially SQLgrey's domain-based one) because the greylister can't
| learn from trafic to other users. You'll have a much better
| auto-whitelist usage in an ISP environment.

I actually have a number of live/active users on my test server, but
I see your point. Of course, everyone gets different email, so I'm
wondering how large the difference really is.

| I think you can already use postfix to selectively use
greylisting, see
| the postfix online documentation, especially the chapter where it is
| configured to greylist only specific source domains.

Do you mean greylist for specific users or specific incoming email
domains? This is what I was considering, a per-user opt-in approach.
I just have to look into how to get Postfix to look up the policies
to use. I can insert it before or after alias expansion with my
setup (Postfix is so flexible :)), but I guess that's beside the point.

| Then an ISP can launch sqlgrey in "optin" or "optout" mode and add to
| its web interfaces some configuration pages that will allow its
users to
| subscribe to the service by adding/removing entries in the correct
| sqlgrey tables.

This is what I'll probably do, make a web interface, but I was
thinking about using an SQL lookup in Postfix to get the policies
(not sure if that is possible) and/or putting it in amavisd-new or
something so you can have "trickle-down" organization-based
policies. If this isn't possible, adding it to sqlgrey may be the
only option, but I think it "belongs" in Postfix, personally.

Anyway thanks for the response!

Josh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBvyvwV/+PyAj2L+IRAt0HAJwIBPbZdok+fWfsp1Vkk7UqVbehXgCdHYaZ
4RuBdrSSxR9IC8P2lZmD02Y=
=uaJ2
-----END PGP SIGNATURE-----

Re: [Sqlgrey-users] Using SQLgrey in an ISP environment

[Lionel B. <lio...@bo...>]

Josh Endries wrote the following on 12/14/04 19:07 :

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Lionel Bouton wrote:
> | For one isolated user it takes a long time for auto-whitelist to
> kick in
> | (especially SQLgrey's domain-based one) because the greylister can't
> | learn from trafic to other users. You'll have a much better
> | auto-whitelist usage in an ISP environment.
>
> I actually have a number of live/active users on my test server, but
> I see your point. Of course, everyone gets different email, so I'm
> wondering how large the difference really is.


It depends on the sender repartition accross the recipients.
If multiple recipients see the same sender on different occasions they 
will benefit from the common auto-whitelist.
If multiple recipients see different senders but these are from the same 
domains, they will benefit too.

>
> | I think you can already use postfix to selectively use
> greylisting, see
> | the postfix online documentation, especially the chapter where it is
> | configured to greylist only specific source domains.
>
> Do you mean greylist for specific users or specific incoming email
> domains? This is what I was considering, a per-user opt-in approach.
> I just have to look into how to get Postfix to look up the policies
> to use. I can insert it before or after alias expansion with my
> setup (Postfix is so flexible :)), but I guess that's beside the point.
>

I didn't realise you could make Postfix use the greylisting policy 
daemon after alias expansion. How do you do that ?

> | Then an ISP can launch sqlgrey in "optin" or "optout" mode and add to
> | its web interfaces some configuration pages that will allow its
> users to
> | subscribe to the service by adding/removing entries in the correct
> | sqlgrey tables.
>
> This is what I'll probably do, make a web interface, but I was
> thinking about using an SQL lookup in Postfix to get the policies
> (not sure if that is possible) and/or putting it in amavisd-new or
> something so you can have "trickle-down" organization-based
> policies. If this isn't possible, adding it to sqlgrey may be the
> only option, but I think it "belongs" in Postfix, personally.


If it can be done there, that's good please explain to the list how you 
do it and I'll make it an HOWTO. I saw at least another greylisting 
implementation providing optin optout, so I'm wondering why they had to 
do it.

I have two orthogonal goals :
- make the software easy to use (if optin/optout is painful in Postfix, 
then I'll add it),
- don't bloat it (no, another piece of code in a 50k perl script !).

Lionel.

Re: [Sqlgrey-users] Using SQLgrey in an ISP environment

[Josh E. <jo...@en...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lionel Bouton wrote:
| I didn't realise you could make Postfix use the greylisting policy
| daemon after alias expansion. How do you do that ?

Well, I lied (kinda). I do it via multiple instances (actually,
multiple physical servers).

| If it can be done there, that's good please explain to the list
how you
| do it and I'll make it an HOWTO. I saw at least another greylisting
| implementation providing optin optout, so I'm wondering why they
had to
| do it.

I'm pretty confident I'll get something worked out as this would be
a great thing to offer. I need to finish my Horde module first,
though. I'll be sure to share my findings. :)

Thanks,
Josh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD4DBQFBv9x4V/+PyAj2L+IRAjVyAJ9eCHWGZYuDzCxGhP/K+iPpS0iDuwCYrSod
hXOTt2h8FO2LHtj3FgUbYQ==
=4Vqm
-----END PGP SIGNATURE-----

Re: [Sqlgrey-users] Using SQLgrey in an ISP environment

[Lionel B. <lio...@bo...>]

Josh Endries wrote the following on 12/15/04 07:40 :

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Lionel Bouton wrote:
> | I didn't realise you could make Postfix use the greylisting policy
> | daemon after alias expansion. How do you do that ?
>
> Well, I lied (kinda). I do it via multiple instances (actually,
> multiple physical servers).
>

I'm not sure I see how you do it. Here's what I imagine (probably 
because this was the process I thought of when trying to) :
- a first pool accepts the messages, processes alias expansion and 
forwards the messages to a second pool
- the second pool greylists.
But it doesn't work : the first pool did already accept the message when 
the second wants to greylist.

I just realised that in fact it shouldn't be possible to do greylisting 
after alias expansion. Let me explain :
- Postfix handles the domain example.com
- there'a an alias "adm...@ex..." expanding to the final 
recipients "pos...@ex..." and "ro...@ex...".
- Postfix wants messages to root being greylisted and not messages to 
postmaster.
- The greylister doesn't know yet that "se...@ot..." on 
123.48.12.58 is a valid couple.
- se...@ot... sends an e-mail from 123.48.12.58 to 
adm...@do...
What should Postfix do ? It can't refuse the mail because postmaster 
doesn't want its incoming messages to be greylisted but at the same time 
root doesn't want to receive messages that haven't been greylisted so it 
can't accept it either.

Conclusion : no greylisting before alias expansion

Lionel.

Re: [Sqlgrey-users] Using SQLgrey in an ISP environment

[Josh E. <jo...@en...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lionel Bouton wrote:
| But it doesn't work : the first pool did already accept the
message when
| the second wants to greylist.

Ahh I see now, this is what I was thinking also. I guess it will be
most effective if done on the first machine.

| I just realised that in fact it shouldn't be possible to do
greylisting
| after alias expansion. Let me explain :
| - Postfix handles the domain example.com
| - there'a an alias "adm...@ex..." expanding to the final
| recipients "pos...@ex..." and "ro...@ex...".
| - Postfix wants messages to root being greylisted and not messages to
| postmaster.
| - The greylister doesn't know yet that "se...@ot..." on
| 123.48.12.58 is a valid couple.
| - se...@ot... sends an e-mail from 123.48.12.58 to
| adm...@do...
| What should Postfix do ? It can't refuse the mail because postmaster
| doesn't want its incoming messages to be greylisted but at the
same time
| root doesn't want to receive messages that haven't been greylisted
so it
| can't accept it either.
|
| Conclusion : no greylisting before alias expansion

I'm confused. First you said it shouldn't be possible to greylist
after expansion, then you said no greylisting before expansion. I'm
guessing Postfix will do the alias resolution before policy, as
different "real" users may have different policies, but that's just
a hunch. I can test this to find out what happens.

Josh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBv/YDV/+PyAj2L+IRAuzPAJ41QAQkBOznOOPx4hMMrh+q6Brt6QCgrHY7
WcbailgdvWjnqiwHSAPoMoY=
=AVRV
-----END PGP SIGNATURE-----

Re: [Sqlgrey-users] Using SQLgrey in an ISP environment

[Lionel B. <lio...@bo...>]

Josh Endries wrote the following on 12/15/04 09:29 :

> [...]
> | Conclusion : no greylisting before alias expansion


Oups, typo. I meant *after* of course.

>
> I'm confused. First you said it shouldn't be possible to greylist
> after expansion, then you said no greylisting before expansion. I'm
> guessing Postfix will do the alias resolution before policy, as
> different "real" users may have different policies, but that's just
> a hunch. I can test this to find out what happens.

Re: [Sqlgrey-users] Using SQLgrey in an ISP environment

[HaJo S. <ha...@ha...>]

On Wed, December 15, 2004 1:38, Lionel Bouton said:
> I can add opt-in and opt-out support if needed.

I think this is an excellent idea! One more for your little TODO ;-)

-- 
HaJo Schatz <ha...@ha...>
http://www.HaJo.Net

PGP-Key:  http://www.hajo.net/hajonet/keys/pgpkey_hajo.txt