Re: [Sqlgrey-users] warn_if_reject to populate the db

[Michael S. <Mic...@lr...>]

On Thu, 30 Jun 2005, Michel Bouissou wrote:

>
> The major objections that I see to integrating any kind of SPF check in
> sqlgrey is that :
>
> 1/ SPF not having anything to do with greylisting, it isn't pure greylisting
> at all anymore.

This is just a statement without any reasons behind it. Therefore I just
state the opposite: it has to do with greylisting.

>
> 2/ SPF being a complex algorithm, it necessitates a complex implementation in
> SQLgrey, which is both heavy and bug-prone. And as the SPF standard might
> evolve, it would need SQLgrey to follow its evolutions -- we're adding a lot
> of foreseeable trouble there.

Wrong. Here is the relevant code out of my program:

use Mail::SPF::Query;
my %query_options = (   ipv4    => $conn_ip,
                        sender  => $sender_domain,
                        helo    => $sender_domain, # should be helo-domain, but helo-domain not yet avail.
my $query = new Mail::SPF::Query(%query_options);
my ($result, $smtp_comment, $header_comment) = $query->result;
if ($result eq 'pass') {
...

I think these few lines of code cannot be called a complex implementation.

> Furthermore there are other proposed standards competing with SPF, so if you
> consider adding one, then you'll have to consider adding others, and this is
> endless.

As long as the relevant information is outside the evelope, e.g.
information in the header, these standards cannot be used directly by
greylisting. If, however, other standards exist, which operate on the
envelope, we will see, if they could be used by greylisting. But as long
as I do not know them, I cannot say anything about them.

>
> 3/ SPF cannot be done using only the information that Postfix provides to the
> SQLgrey policy server. SPF needs at least one, possibly several, DNS calls,
> which will have a very noticeable negative impact on SQLgrey performance. I
> object to the idea that SQLgrey would need to perform any kind of network
> request to be able to make a decision. One could say that DNS calls are fast,
> but it's not always the case. The remote DNS server you call can sometimes be
> very slow, unreachable or unresponsive...

This is true, the actual implementation of sqlgrey does not allow any
algorithm which uses DNS request which will go out into the Internet. If
we want to use such algorithms, then either sqlgrey must go from multiplex
to prefork mode, or the propagation algorithms must use a sideline
process, similar to the cleanup process. I am aware of this problem since
the beginning.

>
> > The same is true for the rcpt_awl, we need for forwarded emails. We have a
> > lot of employees, which separate their email addresses for business and
> > private use. For their privat emails they use an address with a freemail
> > provider. All emails arriving at the freemail provider are forwarded to
> > their business address, which is allowed in our university environment.
> > This way they have separated their addresses, but still have all emails
> > together in one mailbox, where they can be accessed immediately.
> >
> > The other case is, a lot of employees of the universities come from other
> > universities. When they moved, they set up a forward to an email address
> > here.
> >
> > In both cases we want to accept such forwarded emails immediately without
> > any delay. But with from_awl and domain_awl alone, it is not possible. The
> > sending MTA may have an entry in domain_awl, but normally only for the
> > domains it is responsible. But often the forwarded emails have an
> > originator with a different domain. Therefore every new originator will go
> > through greylisting, which makes no sense, because often we now that the
> > sending MTA is a well behaved MTA and we trust this MTA.
> >
> > With rcpt_awl this is different. In this table we will include the ip
> > address of the MTA and the recipient address, but no information about the
> > originator. That means we need only one entry per forward and all emails
> > coming this way will be accepted immediately. And this means greylisting
> > is strengthened again.
>
> Regarding email forwarded from other universities, I believe the number of
> such universities isn't that high and doesn't grow that fast. You'd probably
> be better adding the know MTAs of those universities to an existing manual
> whitelist, as you know those servers are true MTAs, well behaved.
>
> Same could be done rather easily for the 4 or 5 major "free email providers
> and forwarders" you're talking about. That makes what ? Hotmail, MSN, Yahoo,
> a couple of biggest ISPs in your country, and you'd probably cover more than
> 80% of your forwarding greylisting issues (if one believes the 80/20 law ;-)
>
> For the remaining 20%, it would concern mainly "private forwarded mail", and
> in any case, the from_awl would take care that only the first mail from A to
> B thru "forwarder" would be greylisted. And if "forwarder" forwards a
> noticeable amount of email from any given domain, it would end in domain_awl
> for this domain anyway...
>
> So I'm really not sure that your forwarding issue necessitates the added
> complexity and performance cost that adding new tables to SQLgrey would
> necessitate.
>
> For example, rcpt_awl seems plain useless to me, as if you manually put an
> entry in rcpt_awl for a couple MTA_IP / recipient, then you mean that MTA_IP
> is well behaved. If it's well behaved for "recipient", it's also well behaved
> for any other recipient, and MTA_IP can perfectly be added to the existing
> manual whitelists without the need of adding a supplementary table, isn't
> it ?
>

The most expensive thing at our computer centre ist person power, whereas
computers are cheap. Therefore manual maintenance must be avoided at all
costs. Instead automatic maintenance of whitelists must be done. This is
the reason why we choose sqlgrey, as sqlgrey implements AWLs! rcpt_awl is
an automatic whitelist as the name suggests. The manual whitelist for ip
addresses must only be used as a last resort.

Michael Storz
-------------------------------------------------
Leibniz-Rechenzentrum   !   <mailto:St...@lr...>
Barer Str. 21           !   Fax: +49 89 2809460
80333 Muenchen, Germany !   Tel: +49 89 289-28840