Re: FULL versus SMART (was Re: [Sqlgrey-users] Spammer alex@hotmail.com benefits from from_awl)

[Michael S. <Mic...@lr...>]

On Sat, 30 Apr 2005, Lionel Bouton wrote:

> Michael Storz wrote the following on 29.04.2005 11:28 :
>
> >- Higher delay FULL can have compared to SMART:
> >  The right medizine against that are good whitelists!
> >
>
> whitelists are good for delays I agree. But until you see a new IP, you
> don't put it in your whitelists...
>
> > Good means the
> >  number of whitelists and the content of whitelists. In addition to the
> >  standard sqlgrey algorithms for filling the tables via traffic analysis,
> >  we have implemented our own algorithms :-)
> >
> >  * fast propagation (fills from_awl): This algorithm is based on the
> >    trust we have about a sending MTA. If we trust it, we accept the
> >    email, even if there is no entry about this triple in the whitelists.
> >  * MX-check (fills domain_awl): if outgoing and incoming MTAs are the
> >    same, put an entry in domain_awl.
> >  * A-check (fills domain_awl): if sending MTA sends emails for its
> >    hostname only, put it in domain_awl.
> >
> >  These additional algorithms give us a lot of entries in our from_awl and
> >  domain_awl and therefore reduce the delay significantly. And the last 2
> >  algorithms only work with FULL, not with SMART, with the current design
> >  of sqlgrey. Sorry for you, guys :-)
> >
> >
> >
>
> How do you implement them ? One way to do it would be to modify
> SQLgrey's log to log the whole IP address of each new connection (in
> fact I was sorry to not have them in log, so I'll have the log format
> changed to do this) and have a little daemon tailing the log file
> looking for
> "^<syslog_header> sqlgrey: grey: new: (<ip>) .*" lines
> doing DNS requests and adding entries in the database.

Since we are using FULL instead of SMART, the full ip address is included
in table from_awl. cron starts my script every 5 minutes. At the moment it
selects all entries used in the last 6 minutes (last_seen) and in the
future all new entries in the last 6 minutes. For these entries DNS
lookups are made and relevant records inserted in domain_awl.

>
> >  About additional whitelists, forward_awl/rcpt_awl is one of them. At the
> >  moment fast propagation replaces this table, because most of the time we
> >  accept immidiately all the spam mails from forwarding where the remote
> >  MTA does not use greylisting, but at the cost of many unnecessary
> >  entries in from_awl.
> >
> >  Another one would be prevalidation as implented in other greylisting
> >  software. here you put the tuple originator/recipient without an ip
> >  address in a table for every email you send out.
> >
> >
>
> Interesting. How is it done? The policy service is called before
> permit_mynetworks and the policy service is taught the value of
> "mynetworks" ?
> This can be problematic :
> Here is part of my smtpd_recipient_restrictions :
> (...)
>         permit_mynetworks
>         reject_unknown_recipient_domain
>         reject_unauth_destination
>         check_helo_access hash:/etc/postfix/smtp_helo_blacklist
>         reject_unlisted_recipient
>         check_policy_service inet:127.0.0.1:2501
>
> If I move check_policy_service before permit_my_networks, I'll have lots
> of crap passing through the policy service which will be rejected later.
> One solution would be to have two separate policy services, one called
> before permit_mynetworks which would only process "mynetworks" sources
> in order to whitelist some tuples and the classic greylister later.
> That's for the distant future of SQLgrey :-)
>

I can't say how it could be done with postfix. In our case we have
different queues for incomming and outgoing emails. We can programm our
own daemon for outgoing emails, which call an interface, which put the
relevant records in the prevalidation table.

> >  Actually, what I want is to strengthen the trust in the sending MTA,
> >  e.g. to use the domain from HELO/ELHO
> >
>
> how do use this domain (isn't it an hostname btw)? I'm thinking about
> MXs/relays for several domains.
>

I would use it as string value with no special meaning. Instead of
checking for the ip address only, I would check for the tuple (ip address,
helo domain string) in the various tables. Normmaly there should be no
difference checking the ip adress or this tuple. I assume the following
differences could exist, where multiple entries exist for one ip address
but different helo domain strings:

- virtual outgoing MTAs are running on the same ip address, each of them
  announcing a different virtual mailserver via the helo domain. Another
  evidence for this would be several PTR records for this ip address which
  show the different virtual servers
- dynamic ip addresses: every time a dynamic ip address is assigned to a
  new MTA, this MTA will use a different helo domain
- spammer, who tries to get more randomness into the header of an email
  via the logged hel domain in received lines. He would use a different
  helo domain for every sent email.

If using the helo domain would indeed strenghten the trust in the sending
MTA or if it is just useful for traffic analysis and better understanding
of what different MTAs are doing, we would see, if we are using this
field.


Michael Storz
-------------------------------------------------
Leibniz-Rechenzentrum   !   <mailto:St...@lr...>
Barer Str. 21           !   Fax: +49 89 2809460
80333 Muenchen, Germany !   Tel: +49 89 289-28840