sqlgrey-users — Users of SQLgrey

Re: [Sqlgrey-users] A smarter smart

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 15.02.2005 15:51 :

>Le Mardi 15 F=E9vrier 2005 15:38, Michel Bouissou a =E9crit :
> =20
>
>>And here comes the Big Hideous Ugly Regexp ;-)
>>
>>The attached patch makes SQLgrey's smart decisions much smarter in deci=
ding
>>if C-Class or complete IP should be used for a given client.
>>   =20
>>
>
>Oops, sorry, there was a little issue with the way the regexp were split=
 into=20
>several lines.
>The attached patch replaces the previous one, and fixes the issue.
> =20
>

Thanks, I'm worried about the size of the regexp though. There are two=20
things on my mind :
- is it maintainable ?
- how much processing time is needed for these regexp ?

I'd like to add this as a separate algorithm and put the regexp in=20
external files that can be reloaded (ie: like the whitelists, being=20
updated from a repository). Being a switch, the second potential problem=20
wouldn't actually be one. I've not access on enough maillog to test=20
these regexps on each update. Are you willing to maintain them ? This=20
will solve the first potential problem for me :-)

I'll probably start the 1.5.x branch for this new algorithm.

Lionel.

Re: [Sqlgrey-users] A smarter smart

[Michel B. <mi...@bo...>]

Le Mardi 15 F=E9vrier 2005 15:38, Michel Bouissou a =E9crit :
> And here comes the Big Hideous Ugly Regexp ;-)
>
> The attached patch makes SQLgrey's smart decisions much smarter in deci=
ding
> if C-Class or complete IP should be used for a given client.

Oops, sorry, there was a little issue with the way the regexp were split =
into=20
several lines.
The attached patch replaces the previous one, and fixes the issue.

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

[Sqlgrey-users] A smarter smart

[Michel B. <mi...@bo...>]

And here comes the Big Hideous Ugly Regexp ;-)

The attached patch makes SQLgrey's smart decisions much smarter in deciding if 
C-Class or complete IP should be used for a given client.

It's heuristic... So imperfect, but my tests show it gives much more accurate 
results compared to the original simpler algorithm.

(Using a "debug" loglevel will show the decision rules that trigger)

Give it a try ;-)

Cheers.

-- 
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Michel B. <mi...@bo...>]

Le Mardi 15 F=E9vrier 2005 10:50, Lionel Bouton a =E9crit :
>
> Shouldn't be a problem (MYTMP can't be '/' or something approaching) ,
> but cd ~sqlgrey is better.

Yep, it shouldn't. But better safe than sorry ;-)

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 15.02.2005 09:52 :

>Le Lundi 14 F=E9vrier 2005 23:40, Lionel Bouton a =E9crit :
> =20
>
>>Some distributions might not have the latest mktemp, so SQLgrey doesn't
>>depend on it anymore :
>> - update_sqlgrey_whitelists doesn't rely on mktemp's '-t' parameter
>>anymore.
>>   =20
>>
>
>Seems there's a booog in the statement, which is
>MYTMP=3D`mktemp -d ${TMPDIR:/-tmp}/sqlgrey.XXXXXX`
>where it should be
>MYTMP=3D`mktemp -d ${TMPDIR:-/tmp}/sqlgrey.XXXXXX`
> =20
>

Right. What I wonder now is : how in the hell did this work on my server =
?
I just tested this part of the code and I understand : the script=20
creates its temporary directory in '/' instead of the temp dir...
Not harmful but definitely not what I intended :-(

All your fixes will be in 1.4.5.

Thanks again,

Lionel.

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 15.02.2005 07:57 :

>Le Lundi 14 F=E9vrier 2005 23:40, Lionel Bouton a =E9crit :
> =20
>
>> - update_sqlgrey_whitelists doesn't rely on mktemp's '-t' parameter
>>anymore.
>>   =20
>>
>
>By the way, I find 2 issues in update_sqlgrey_whitelists :
>
>1/ The result code or variable existence for mktemp is not tested :
>
># Go into a temp directory
>MYTMP=3D`mktemp -dt sqlgrey.XXXXXX`
>cd $MYTMP
>
>I would prefer :
>
># Go into a temp directory
>MYTMP=3D`mktemp -dt sqlgrey.XXXXXX`
>[ -n "$MYTMP" -a -d "$MYTMP" ] && cd $MYTMP || {
>   echo "Error creating temporary directory;
>   exit 1
>}
>
> =20
>

Took it.

>2/ This frightens me (afraid of cd /; rm -rf * ;-) especially because of=
 (1)=20
>above:
>
># Setup a clean exit
>clean_exit() {
>    cd /
>    rm -rf $MYTMP
>    exit $1
>}
>
>I would very much prefer something like:
>
># Setup a clean exit
>clean_exit() {
>    cd ~sqlgrey
>    [ -n "$MYTMP" -a -d "$MYTMP" ] && rm -rf $MYTMP
>    exit $1
>}
>
> =20
>

Shouldn't be a problem (MYTMP can't be '/' or something approaching) ,=20
but cd ~sqlgrey is better.
The test before rm -rf shouldn't be needed now but might come handy if=20
clean_exit can be called before we make sure MYTMP is a real directory.
Took it too.

Thanks.

Lionel

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Michel B. <mi...@bo...>]

Le Lundi 14 F=E9vrier 2005 23:40, Lionel Bouton a =E9crit :
>
> Some distributions might not have the latest mktemp, so SQLgrey doesn't
> depend on it anymore :
>  - update_sqlgrey_whitelists doesn't rely on mktemp's '-t' parameter
> anymore.

Seems there's a booog in the statement, which is
MYTMP=3D`mktemp -d ${TMPDIR:/-tmp}/sqlgrey.XXXXXX`
where it should be
MYTMP=3D`mktemp -d ${TMPDIR:-/tmp}/sqlgrey.XXXXXX`

Please find a patch attached, that also includes the fixes I suggested in=
 my=20
previous message.

I've built 1.4.4 RPMs (including this patch), that you can get from :
http://www.bouissou.net/sqlgrey/

Cheers.

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Michel B. <mi...@bo...>]

Le Lundi 14 F=E9vrier 2005 23:40, Lionel Bouton a =E9crit :
>
>  - update_sqlgrey_whitelists doesn't rely on mktemp's '-t' parameter
> anymore.

By the way, I find 2 issues in update_sqlgrey_whitelists :

1/ The result code or variable existence for mktemp is not tested :

# Go into a temp directory
MYTMP=3D`mktemp -dt sqlgrey.XXXXXX`
cd $MYTMP

I would prefer :

# Go into a temp directory
MYTMP=3D`mktemp -dt sqlgrey.XXXXXX`
[ -n "$MYTMP" -a -d "$MYTMP" ] && cd $MYTMP || {
   echo "Error creating temporary directory;
   exit 1
}


2/ This frightens me (afraid of cd /; rm -rf * ;-) especially because of =
(1)=20
above:

# Setup a clean exit
clean_exit() {
    cd /
    rm -rf $MYTMP
    exit $1
}

I would very much prefer something like:

# Setup a clean exit
clean_exit() {
    cd ~sqlgrey
    [ -n "$MYTMP" -a -d "$MYTMP" ] && rm -rf $MYTMP
    exit $1
}


--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

[Sqlgrey-users] [RELEASE] 1.4.4 on SF (and artwork)

[Lionel B. <lio...@bo...>]

Hi everybody,

1.4.4 is out on sourceforge (and running on one of my server for 24 hours) :

Commented Changelog :

As requested by Michel :
 - Autowhitelists now understand SRS
 - more VERP support for autowhitelists

This I find useful :
 - SQLgrey can warn by mail when the database is unavailable (this is 
rate limited to prevent mail floods in case of "flip-flop")

Some distributions might not have the latest mktemp, so SQLgrey doesn't 
depend on it anymore :
 - update_sqlgrey_whitelists doesn't rely on mktemp's '-t' parameter 
anymore.

As a side note, there's Hajo's artwork on the website now !

The next version (1.4.5) will focus on database indexes.

Lionel.

Re: [Sqlgrey-users] update_sqlgrey_whitelists error

[Michel B. <mi...@bo...>]

Le Mercredi 09 F=E9vrier 2005 14:15, Lionel Bouton a =E9crit :
>
> >Seems that Mandrake's mktemp is severely outdated, doesn't manage the =
"-t"
> >flag, and uses an old mktemp from BSD instead of using the much more
> > recent one from mktemp.org...
> >
> >Some other distros may possibly have the same problem ?
>
> I'll emulate the '-t' behaviour in 1.4.4 then (look for $TMPDIR,
> fallback on /tmp for initial location).

Following the bug report I sent them, Mandrake has updated their mktemp=20
package in Cooker, with mktemp-1.5-12mdk.i586.rpm which is now built from=
=20
http://www.mktemp.org and manages the -t flag.

It works OK here with update_sqlgrey_whitelists .

However, I'm not sure whether this will be included in the "MandrakeUpdat=
e"=20
system, or if Mandrake users will have to get it from Cooker or wait for =
the=20
next Mandrake version...

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] Weird entries: C-Class or full IP

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 02/09/05 23:02 :

>Hi there,
>
>Taking a look at my from_awl, I can see that I have 2 entries with same domain 
>and same sender, one with the IP recorded as "C-Class", one recorded with the 
>full IP.
>
>Both entries identify the same sender and host.
>
>Taking a look at Postfix logs, I notice that once Postfix got the remote 
>hostname, and once got it as "unknown" (probably a temporary rDNS failure 
>somewhere ?).
>
>  
>

Most probably.

>Seems that the unknown/complete IP got there first, before the 
>hostname/C-Class one.
>
>  
>

No guarantee, but it should be the case because mail servers should have 
a DNS cache sitting right next to them...

>But it raises the issues:
>
>1/ When a "complete IP" is looked up if an AWL, what if the same "C-Class" 
>entry is already there ? Will it be checked first ?
>  
>

No. If the DNS is unavailable SQLgrey doesn't trust this IP enough to 
greylist basing its decisions on the C-class. It can be argued it 
should, each time checking both for class-C and IP in the awl and 
connect table. This way it would handle temporary DNS problems more 
gracefully. The problem is that you can't verify in any way that the 
C-class entry you match isn't from a neighbour *with* a valid rDNS and 
the one you let in is in fact an evil spam-zombie.

If you want to get rid of the problem, you can switch to the classc 
algorithm which uses the class C everytime.

>2/ When a "C-Class" is going to be added to an AWL, shouldn't the possible 
>corresponding "complete IPs" be removed ?
>  
>

I considered doing so, but the benefits aren't clear. If all goes well, 
you remove unused entries, which is good for the overall DB performance. 
But if the DNS at the other end is flaky and your users definitely want 
mails from this domain you can end up with lots of headaches (SQLgrey 
will have to rebuild individual entries, defeating the AWL purpose, 
slowing mail down for everybody at the same time...).

>3/ If an entry is still at "connect" stage, and the message comes back with a 
>different (present/absent) hostname, won't it probably cause the message to 
>get rejected once more, unless a new connection "completely matches" ?
>  
>

Yes.

>4/ Should this kind of problem be considered so marginal that we might want to 
>keep it unadressed, and let the "old age" get rid of "complete IPs" that 
>become useless because the corresponding complete "C-Class" has been added 
>since ?
>
>  
>

Unless there are lots of duplicate entries, I wouldn't bother, I'll let 
the entries being cleaned up automatically.

Lionel.

[Sqlgrey-users] Weird entries: C-Class or full IP

[Michel B. <mi...@bo...>]

Hi there,

Taking a look at my from_awl, I can see that I have 2 entries with same domain 
and same sender, one with the IP recorded as "C-Class", one recorded with the 
full IP.

Both entries identify the same sender and host.

Taking a look at Postfix logs, I notice that once Postfix got the remote 
hostname, and once got it as "unknown" (probably a temporary rDNS failure 
somewhere ?).

Seems that the unknown/complete IP got there first, before the 
hostname/C-Class one.

But it raises the issues:

1/ When a "complete IP" is looked up if an AWL, what if the same "C-Class" 
entry is already there ? Will it be checked first ?

2/ When a "C-Class" is going to be added to an AWL, shouldn't the possible 
corresponding "complete IPs" be removed ?

3/ If an entry is still at "connect" stage, and the message comes back with a 
different (present/absent) hostname, won't it probably cause the message to 
get rejected once more, unless a new connection "completely matches" ?

4/ Should this kind of problem be considered so marginal that we might want to 
keep it unadressed, and let the "old age" get rid of "complete IPs" that 
become useless because the corresponding complete "C-Class" has been added 
since ?

Cheers.

-- 
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] update_sqlgrey_whitelists error

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 02/09/05 11:17 :

>Le Mercredi 09 F=E9vrier 2005 10:19, Michel Bouissou a =E9crit :
> =20
>
>>(On Mandrake 10.1)
>>
>>[root@totor sqlgrey]# /usr/sbin/update_sqlgrey_whitelists
>>mktemp: invalid option -- t
>>Usage: mktemp [-d] [-q] [-u] template
>>   =20
>>
>
>Seems that Mandrake's mktemp is severely outdated, doesn't manage the "-=
t"=20
>flag, and uses an old mktemp from BSD instead of using the much more rec=
ent=20
>one from mktemp.org...
>
>Some other distros may possibly have the same problem ?
>
> =20
>

I'll emulate the '-t' behaviour in 1.4.4 then (look for $TMPDIR,=20
fallback on /tmp for initial location).

Lionel.

Re: [Sqlgrey-users] update_sqlgrey_whitelists error

[Michel B. <mi...@bo...>]

Le Mercredi 09 F=E9vrier 2005 10:19, Michel Bouissou a =E9crit :
> (On Mandrake 10.1)
>
> [root@totor sqlgrey]# /usr/sbin/update_sqlgrey_whitelists
> mktemp: invalid option -- t
> Usage: mktemp [-d] [-q] [-u] template

Seems that Mandrake's mktemp is severely outdated, doesn't manage the "-t=
"=20
flag, and uses an old mktemp from BSD instead of using the much more rece=
nt=20
one from mktemp.org...

Some other distros may possibly have the same problem ?

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

[Sqlgrey-users] update_sqlgrey_whitelists error

[Michel B. <mi...@bo...>]

(On Mandrake 10.1)

[root@totor sqlgrey]# /usr/sbin/update_sqlgrey_whitelists
mktemp: invalid option -- t
Usage: mktemp [-d] [-q] [-u] template

-- 
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] [RELEASE] 1.4.3

[Michel B. <mi...@bo...>]

Le Lundi 07 F=E9vrier 2005 17:20, Michel Bouissou a =E9crit :
>
> I'll give a shot at the .spec when I have time, maybe tomorrow. If I
> produce a working RPM, I'll give an URL here.

Here it is : sqlgrey-1.4.3 RPMs produced on Mandrake 10.1. They should=20
probably work as well on Mandrake 10.0, RedHat or Fedora:

http://www.bouissou.net/sqlgrey/

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] [RELEASE] 1.4.3

[HaJo S. <ha...@ha...>]

On Mon, 2005-02-07 at 18:50 +0100, Lionel Bouton wrote:
> HaJo Schatz wrote the following on 02/07/2005 04:59 PM :

> >/var/tmp/rpm-tmp.95341: line 33: /usr/lib/rpm/check-rpaths: No such file
> >or directory
> >error: Bad exit status from /var/tmp/rpm-tmp.95341 (%install)


>  From the errors, this seems to be a problem on your configuration.

Yes it was, non-installation of fedora-rpmdevtools*.rpm to be precise...

HaJo
-- 
HaJo Schatz <ha...@ha...>
http://www.HaJo.Net

PGP-Key:  http://www.hajo.net/hajonet/keys/pgpkey_hajo.txt

Re: [Sqlgrey-users] [RELEASE] 1.4.3 & artwork

[Lionel B. <lio...@bo...>]

HaJo Schatz wrote the following on 02/08/05 07:05 :

>On Mon, 2005-02-07 at 18:50 +0100, Lionel Bouton wrote:
>  
>
>>HaJo Schatz wrote the following on 02/07/2005 04:59 PM :
>>    
>>
>
>  
>
>> From the errors, this seems to be a problem on your configuration.
>>    
>>
>
>Yes, quite possibly. I'm not usually doing rpmbuilds. Will have to dig
>into this one deeper...
>
>Reading about your Gimped piano a quick thought crossed my mind --
>you're collecting "3rd party animals" on your page, Ie the Postfix mouse
>and the GNU gnu. Shouldn't sqlgrey's logo be an animal as well? I'm
>thinking of the ostrich here, it's head being stuck into sand;
>symbolizing sqlgrey sticking it's head into the sand and not accepting
>delivery?
>

I didn't thought much about that. An ostrich with the head in sand seems 
good. Feel free to submit a pic (without copyrights) :-)

Lionel.

Re: [Sqlgrey-users] [RELEASE] 1.4.3 & artwork

[HaJo S. <ha...@ha...>]

On Tue, 2005-02-08 at 14:05 +0800, HaJo Schatz wrote:
> I'm
> thinking of the ostrich here, it's head being stuck into sand;

OK, here's what happens when I'm on vacation trying to avoid to go out
and wash the car: 

http://hajo.net:8081/test/sqlgrey-button.xcf

Take it or leave it, up to you...

HaJo

-- 
HaJo Schatz <ha...@ha...>
http://www.HaJo.Net

PGP-Key:  http://www.hajo.net/hajonet/keys/pgpkey_hajo.txt

Re: [Sqlgrey-users] [RELEASE] 1.4.3 & artwork

[HaJo S. <ha...@ha...>]

On Mon, 2005-02-07 at 18:50 +0100, Lionel Bouton wrote:
> HaJo Schatz wrote the following on 02/07/2005 04:59 PM :

>  From the errors, this seems to be a problem on your configuration.

Yes, quite possibly. I'm not usually doing rpmbuilds. Will have to dig
into this one deeper...

Reading about your Gimped piano a quick thought crossed my mind --
you're collecting "3rd party animals" on your page, Ie the Postfix mouse
and the GNU gnu. Shouldn't sqlgrey's logo be an animal as well? I'm
thinking of the ostrich here, it's head being stuck into sand;
symbolizing sqlgrey sticking it's head into the sand and not accepting
delivery? How you interpret the "plugin" nature with the bird is up to
your interpretation ;-)

OK, just a thought...

HaJo

-- 
HaJo Schatz <ha...@ha...>
http://www.HaJo.Net

PGP-Key:  http://www.hajo.net/hajonet/keys/pgpkey_hajo.txt

Re: [Sqlgrey-users] Re: Sender whitelisting

[Michel B. <mi...@bo...>]

Le Lundi 07 F=E9vrier 2005 18:42, Lionel Bouton a =E9crit :
>
> You hit below the belt :-)

Sorry :-}

> At least I did release SQLgrey under the GPL=20

And for sure everybody here duly appreciate this !

> Resistance to new features is a very important part of project
> management.

That's right.

> (but patches are welcomed...).

If I were a decent Perl coder, surely I would code myself what I need and=
=20
submit the patches. Unfortunately I'm not. But I may take a look at it=20
someday ;-)

Cheers.

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] [RELEASE] 1.4.3

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 02/07/2005 05:20 PM :

>Le Lundi 07 F=E9vrier 2005 16:59, HaJo Schatz a =E9crit :
> =20
>
>>error: Bad exit status from /var/tmp/rpm-tmp.95341 (%install)
>>   =20
>>
>
>No better luck here :
>
>Erreur de construction de RPM:
>    Fichier non=20
>trouv=E9: /var/tmp/sqlgrey-1.4.3-build/usr/share/man/man1/sqlgrey.1.gz
>
>But this one is easy : Mandrake builds manpages as .bz2 and not .gz, so =
it's=20
>normal the .gz file can't be found.
>
>I'll give a shot at the .spec when I have time, maybe tomorrow. If I pro=
duce a=20
>working RPM, I'll give an URL here.
>
>For the moment I stick with 1.4.2 ;-)
>
> =20
>

This is a mistake from me : the spec file should ref the "sqlgrey.1"=20
file instead of the 1.gz.
I believe it is bitting us because the rpmbuild workdir wasn't cleaned=20
up after each build and I switched from gzip to no compression (which=20
means old version of the manpage were distributed).

Could you test modifying the spec file to look for "sqlgrey.1" ?

Lionel.

Re: [Sqlgrey-users] [RELEASE] 1.4.3

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 02/07/2005 05:18 PM :

>Le Lundi 07 F=E9vrier 2005 17:12, Rene Joergensen a =E9crit :
> =20
>
>>On Mon, Feb 07, 2005 at 11:59:18PM +0800, HaJo Schatz wrote:
>>   =20
>>
>>>PS: Just recogn'd -- what does that blurry piano labeled "sqlgrey" at
>>>the top-left of sqlgrey.sf.net mean?
>>>     =20
>>>
>>I think that the piano is blurred because it's falling on "SPAM" :-)
>>   =20
>>
>
>Is it a piano or some kind of plugin / connector with 3 stings ?
>
> =20
>

At least before the Gimp abuse it was a piano...

Re: [Sqlgrey-users] [RELEASE] 1.4.3

[Lionel B. <lio...@bo...>]

Rene Joergensen wrote the following on 02/07/2005 05:12 PM :

>On Mon, Feb 07, 2005 at 11:59:18PM +0800, HaJo Schatz wrote:
>
>  
>
>>PS: Just recogn'd -- what does that blurry piano labeled "sqlgrey" at
>>the top-left of sqlgrey.sf.net mean?
>>    
>>
>
>I think that the piano is blurred because it's falling on "SPAM" :-)
>
>  
>

You got it. But my Gimp skills aren't so great...

Lionel

Re: [Sqlgrey-users] [RELEASE] 1.4.3

[Lionel B. <lio...@bo...>]

HaJo Schatz wrote the following on 02/07/2005 04:59 PM :

>Hmm, not really:
>
>[...]
>install -m 644 sqlgrey.1 /var/tmp/sqlgrey-1.4.3-build/usr/share/man/man1
>install init/sqlgrey /var/tmp/sqlgrey-1.4.3-build/etc/init.d
>+ /usr/lib/rpm/find-
>debuginfo.sh /home/users/hajo/install/rpmbuild/BUILD/sqlgrey-1.4.3
>0 blocks
>find: /var/tmp/sqlgrey-1.4.3-build/usr/lib/debug: No such file or
>directory
>+ /usr/lib/rpm/check-rpaths /usr/lib/rpm/check-buildroot
>/var/tmp/rpm-tmp.95341: line 33: /usr/lib/rpm/check-rpaths: No such file
>or directory
>error: Bad exit status from /var/tmp/rpm-tmp.95341 (%install)
>
>  
>

 From the errors, this seems to be a problem on your configuration.