sqlgrey-users

[Sqlgrey-users] [RELEASE] 1.4.4 on SF (and artwork)

[Lionel B. <lio...@bo...>]

Hi everybody,

1.4.4 is out on sourceforge (and running on one of my server for 24 hours) :

Commented Changelog :

As requested by Michel :
 - Autowhitelists now understand SRS
 - more VERP support for autowhitelists

This I find useful :
 - SQLgrey can warn by mail when the database is unavailable (this is 
rate limited to prevent mail floods in case of "flip-flop")

Some distributions might not have the latest mktemp, so SQLgrey doesn't 
depend on it anymore :
 - update_sqlgrey_whitelists doesn't rely on mktemp's '-t' parameter 
anymore.

As a side note, there's Hajo's artwork on the website now !

The next version (1.4.5) will focus on database indexes.

Lionel.

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Michel B. <mi...@bo...>]

Le Lundi 14 F=E9vrier 2005 23:40, Lionel Bouton a =E9crit :
>
>  - update_sqlgrey_whitelists doesn't rely on mktemp's '-t' parameter
> anymore.

By the way, I find 2 issues in update_sqlgrey_whitelists :

1/ The result code or variable existence for mktemp is not tested :

# Go into a temp directory
MYTMP=3D`mktemp -dt sqlgrey.XXXXXX`
cd $MYTMP

I would prefer :

# Go into a temp directory
MYTMP=3D`mktemp -dt sqlgrey.XXXXXX`
[ -n "$MYTMP" -a -d "$MYTMP" ] && cd $MYTMP || {
   echo "Error creating temporary directory;
   exit 1
}


2/ This frightens me (afraid of cd /; rm -rf * ;-) especially because of =
(1)=20
above:

# Setup a clean exit
clean_exit() {
    cd /
    rm -rf $MYTMP
    exit $1
}

I would very much prefer something like:

# Setup a clean exit
clean_exit() {
    cd ~sqlgrey
    [ -n "$MYTMP" -a -d "$MYTMP" ] && rm -rf $MYTMP
    exit $1
}


--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 15.02.2005 07:57 :

>Le Lundi 14 F=E9vrier 2005 23:40, Lionel Bouton a =E9crit :
> =20
>
>> - update_sqlgrey_whitelists doesn't rely on mktemp's '-t' parameter
>>anymore.
>>   =20
>>
>
>By the way, I find 2 issues in update_sqlgrey_whitelists :
>
>1/ The result code or variable existence for mktemp is not tested :
>
># Go into a temp directory
>MYTMP=3D`mktemp -dt sqlgrey.XXXXXX`
>cd $MYTMP
>
>I would prefer :
>
># Go into a temp directory
>MYTMP=3D`mktemp -dt sqlgrey.XXXXXX`
>[ -n "$MYTMP" -a -d "$MYTMP" ] && cd $MYTMP || {
>   echo "Error creating temporary directory;
>   exit 1
>}
>
> =20
>

Took it.

>2/ This frightens me (afraid of cd /; rm -rf * ;-) especially because of=
 (1)=20
>above:
>
># Setup a clean exit
>clean_exit() {
>    cd /
>    rm -rf $MYTMP
>    exit $1
>}
>
>I would very much prefer something like:
>
># Setup a clean exit
>clean_exit() {
>    cd ~sqlgrey
>    [ -n "$MYTMP" -a -d "$MYTMP" ] && rm -rf $MYTMP
>    exit $1
>}
>
> =20
>

Shouldn't be a problem (MYTMP can't be '/' or something approaching) ,=20
but cd ~sqlgrey is better.
The test before rm -rf shouldn't be needed now but might come handy if=20
clean_exit can be called before we make sure MYTMP is a real directory.
Took it too.

Thanks.

Lionel

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Michel B. <mi...@bo...>]

Le Mardi 15 F=E9vrier 2005 10:50, Lionel Bouton a =E9crit :
>
> Shouldn't be a problem (MYTMP can't be '/' or something approaching) ,
> but cd ~sqlgrey is better.

Yep, it shouldn't. But better safe than sorry ;-)

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Michel B. <mi...@bo...>]

Le Lundi 14 F=E9vrier 2005 23:40, Lionel Bouton a =E9crit :
>
> Some distributions might not have the latest mktemp, so SQLgrey doesn't
> depend on it anymore :
>  - update_sqlgrey_whitelists doesn't rely on mktemp's '-t' parameter
> anymore.

Seems there's a booog in the statement, which is
MYTMP=3D`mktemp -d ${TMPDIR:/-tmp}/sqlgrey.XXXXXX`
where it should be
MYTMP=3D`mktemp -d ${TMPDIR:-/tmp}/sqlgrey.XXXXXX`

Please find a patch attached, that also includes the fixes I suggested in=
 my=20
previous message.

I've built 1.4.4 RPMs (including this patch), that you can get from :
http://www.bouissou.net/sqlgrey/

Cheers.

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 15.02.2005 09:52 :

>Le Lundi 14 F=E9vrier 2005 23:40, Lionel Bouton a =E9crit :
> =20
>
>>Some distributions might not have the latest mktemp, so SQLgrey doesn't
>>depend on it anymore :
>> - update_sqlgrey_whitelists doesn't rely on mktemp's '-t' parameter
>>anymore.
>>   =20
>>
>
>Seems there's a booog in the statement, which is
>MYTMP=3D`mktemp -d ${TMPDIR:/-tmp}/sqlgrey.XXXXXX`
>where it should be
>MYTMP=3D`mktemp -d ${TMPDIR:-/tmp}/sqlgrey.XXXXXX`
> =20
>

Right. What I wonder now is : how in the hell did this work on my server =
?
I just tested this part of the code and I understand : the script=20
creates its temporary directory in '/' instead of the temp dir...
Not harmful but definitely not what I intended :-(

All your fixes will be in 1.4.5.

Thanks again,

Lionel.

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Michel B. <mi...@bo...>]

Le Lundi 14 F=E9vrier 2005 23:40, Lionel Bouton a =E9crit :
>
> 1.4.4 is out on sourceforge (and running on one of my server for 24 hou=
rs)
>
> Commented Changelog :
> As requested by Michel :
> =A0- Autowhitelists now understand SRS
> =A0- more VERP support for autowhitelists

Hi, me again ;-)

I find that SQLgrey 1.4.4 generates in its awl some localparts that are:

#.#                   | calva.glou.org=20

#.#.#                 | aol.com

Hmmm... Maybe the VERP-like address management has gone a bit too far ;-)=
 and=20
may produce localparts that are too generic...

I suggest that the beginning of the localpart (before the first [._-]=20
separator that we find) should probably not be replaced with "#", so=20
all-numeric email addresses don't find themselves reduced to "#".

Maybe we should "#" only what we find after the first [._-] separator, if=
 any.

Cheers.

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

L'=E9volution de la pens=E9e pr=E9-situationniste entre l'=E9cole h=E9g=E9=
lienne
et le n=E9gativisme de l'infrastructure n=E9o-nietzsch=E9enne a-t-elle,
inconsciemment ou non, influenc=E9 la carri=E8re de Raymond Poulidor ?
	-- Pierre Desproges.

Re: [Sqlgrey-users] [RELEASE] 1.4.4 on SF

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 15.02.2005 23:56 :

>Le Lundi 14 F=E9vrier 2005 23:40, Lionel Bouton a =E9crit :
> =20
>
>>1.4.4 is out on sourceforge (and running on one of my server for 24 hou=
rs)
>>
>>Commented Changelog :
>>As requested by Michel :
>> - Autowhitelists now understand SRS
>> - more VERP support for autowhitelists
>>   =20
>>
>
>Hi, me again ;-)
>
>I find that SQLgrey 1.4.4 generates in its awl some localparts that are:
>
>#.#                   | calva.glou.org=20
>
>#.#.#                 | aol.com
>
>Hmmm... Maybe the VERP-like address management has gone a bit too far ;-=
) and=20
>may produce localparts that are too generic...
>
>I suggest that the beginning of the localpart (before the first [._-]=20
>separator that we find) should probably not be replaced with "#", so=20
>all-numeric email addresses don't find themselves reduced to "#".
> =20
>

I see the problem :-)

Side note : today I realized that I could fine tune the VERP handling.=20
Now SQLgrey use the "#" replacements in the connect table. I think it=20
could be wise to not put the modified adresses into the connect table=20
but wait them to be greylisted and then use the "#" replaced adresses=20
only in the awl tables.

>Maybe we should "#" only what we find after the first [._-] separator, i=
f any.
>
> =20
>

I can do that. In fact it will remove one regexp to match on.

Thanks,

Lionel.

Re: [Sqlgrey-users] VERP

[Michel B. <mi...@bo...>]

Le Mercredi 16 F=E9vrier 2005 00:50, Lionel Bouton a =E9crit :
>
> Side note : today I realized that I could fine tune the VERP handling.
> Now SQLgrey use the "#" replacements in the connect table. I think it
> could be wise to not put the modified adresses into the connect table
> but wait them to be greylisted and then use the "#" replaced adresses
> only in the awl tables.

Looks like a good idea even though I'm not sure it will make a big differ=
ence=20
on a practical standpoint.

But you need to make sure that you trim the localpart <=3D 64 chars anywa=
y,=20
because some VERP or SRS localparts may be longer (actually the RFC limit=
 to=20
64 chars is not enforced by any modern MTA, only by some old broken ones,=
 and=20
you can find some legitimate mail with longer localparts).

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] VERP

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 16.02.2005 10:11 :

>Le Mercredi 16 F=E9vrier 2005 00:50, Lionel Bouton a =E9crit :
> =20
>
>>Side note : today I realized that I could fine tune the VERP handling.
>>Now SQLgrey use the "#" replacements in the connect table. I think it
>>could be wise to not put the modified adresses into the connect table
>>but wait them to be greylisted and then use the "#" replaced adresses
>>only in the awl tables.
>>   =20
>>
>
>Looks like a good idea even though I'm not sure it will make a big diffe=
rence=20
>on a practical standpoint.
>
>But you need to make sure that you trim the localpart <=3D 64 chars anyw=
ay,=20
>because some VERP or SRS localparts may be longer (actually the RFC limi=
t to=20
>64 chars is not enforced by any modern MTA, only by some old broken ones=
, and=20
>you can find some legitimate mail with longer localparts).
> =20
>

Already coded in CVS like that :-)
I removed deVERP/SRS code from the recipient processing too as it never=20
enters any AWL.

Re: [Sqlgrey-users] VERP patch

[Michel B. <mi...@bo...>]

Le Mardi 15 F=E9vrier 2005 23:56, Michel Bouissou a =E9crit :
> I find that SQLgrey 1.4.4 generates in its awl some localparts that are=
:
> #.# =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 | calva.glou.org
> #.#.# =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 | aol.com
>
> Hmmm... Maybe the VERP-like address management has gone a bit too far ;=
-)
> and may produce localparts that are too generic...
>
> I suggest that the beginning of the localpart (before the first [._-]
> separator that we find) should probably not be replaced with "#", so
> all-numeric email addresses don't find themselves reduced to "#".

The attached patch should do it. It also fixes a little booog with SRS1

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

k=E9rabovidopr=E9hension : d=E9termination

Re: [Sqlgrey-users] VERP patch

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 16.02.2005 00:32 :

>Le Mardi 15 F=E9vrier 2005 23:56, Michel Bouissou a =E9crit :
> =20
>
>>I find that SQLgrey 1.4.4 generates in its awl some localparts that are=
:
>>#.#                   | calva.glou.org
>>#.#.#                 | aol.com
>>
>>Hmmm... Maybe the VERP-like address management has gone a bit too far ;=
-)
>>and may produce localparts that are too generic...
>>
>>I suggest that the beginning of the localpart (before the first [._-]
>>separator that we find) should probably not be replaced with "#", so
>>all-numeric email addresses don't find themselves reduced to "#".
>>   =20
>>
>
>The attached patch should do it. It also fixes a little booog with SRS1
>
> =20
>
>------------------------------------------------------------------------
>
>diff -aurN sqlgrey-1.4.4-interm/sqlgrey sqlgrey-1.4.4/sqlgrey
>--- sqlgrey-1.4.4-interm/sqlgrey	2005-02-15 15:44:15.000000000 +0100
>+++ sqlgrey-1.4.4/sqlgrey	2005-02-16 00:23:36.000000000 +0100
>@@ -579,12 +579,11 @@
>     ## Try to match single-use addresses
>     # SRS (first and subsequent levels of forwarding)
>     $user =3D~ s/^SRS0=3D[^=3D]+=3D[^=3D]+=3D([^=3D]+)=3D([^=3D]+)^/SRS=
0=3D#=3D#=3D$1=3D$2/;
>-    $user =3D~ s/^SRS1=3D[^=3D]+=3D([^=3D]+)=3D[^=3D]+=3D([^=3D]+)=3D([=
^=3D]+)^/SRS0=3D#=3D$1=3D#=3D#=3D$2=3D$3/;
>+    $user =3D~ s/^SRS1=3D[^=3D]+=3D([^=3D]+)=3D[^=3D]+=3D([^=3D]+)=3D([=
^=3D]+)^/SRS1=3D#=3D$1=3D#=3D#=3D$2=3D$3/;
> =20
>

The original was harmless, but your fix point to an obsious flaw in the=20
regexp : '^' is used at the end of the regexp instead of '$' !!!

Obviously not much useful awl entries were created with SRS1 adresses...

Fixed in my tree.

>     # strip extension, used sometimes for mailing-list VERP
>     $user =3D~ s/\+.*//;
>     # strip hexadecimal sequences (doable in one regexp ?)
>     $user =3D~ s/([\._-])[0-9A-Fa-f]+([\._-])/$1#$2/g;
>-    $user =3D~ s/^[0-9A-Fa-f]+([\._-])/#$1/g;
>     $user =3D~ s/([\._-])[0-9A-Fa-f]+$/$1#/g;
>     # Simple VERP substitution : replace numbers with '#'
>     # will match VERP mailing-list message retransmissions
> =20
>

Re: [Sqlgrey-users] VERP patch

[Michel B. <mi...@bo...>]

Le Mercredi 16 F=E9vrier 2005 00:52, Lionel Bouton a =E9crit :
>
> The original was harmless, but your fix point to an obsious flaw in the
> regexp : '^' is used at the end of the regexp instead of '$' !!!
>
> Fixed in my tree.

Another patch attached, because it looks like the SRS1 regexp was still f=
lawed=20
(I checked it with true SRS1 addresses; there's also a place where the "=3D=
"=20
separator can be doubled "=3D=3D").

This patch also removes the "$user =3D~ s/\b\d+\b/#/g;" substitution, as =
it is=20
useless because the hex substitution just before it already does the job.

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] VERP patch

[Lionel B. <lio...@bo...>]

Michel Bouissou wrote the following on 16.02.2005 10:15 :

>Le Mercredi 16 F=E9vrier 2005 00:52, Lionel Bouton a =E9crit :
> =20
>
>>The original was harmless, but your fix point to an obsious flaw in the
>>regexp : '^' is used at the end of the regexp instead of '$' !!!
>>
>>Fixed in my tree.
>>   =20
>>
>
>Another patch attached, because it looks like the SRS1 regexp was still =
flawed=20
>(I checked it with true SRS1 addresses; there's also a place where the "=
=3D"=20
>separator can be doubled "=3D=3D").
> =20
>

Thanks, good to know that : I had no example to check against :-/

>This patch also removes the "$user =3D~ s/\b\d+\b/#/g;" substitution, as=
 it is=20
>useless because the hex substitution just before it already does the job=
.
>
> =20
>

I wonder: this regexp is straight from postgrey and I'm not sure what=20
the '\b' word-boundary will match. I did a quick google search and found=20
no detailed information. I don't even know if it is locale-dependent.=20
Anyone on the list knows the details ?

Thanks again,

Lionel.

Re: [Sqlgrey-users] VERP patch

[Michel B. <mi...@bo...>]

Le Mercredi 16 F=E9vrier 2005 10:37, Lionel Bouton a =E9crit :
>
> >This patch also removes the "$user =3D~ s/\b\d+\b/#/g;" substitution, =
as it
> > is useless because the hex substitution just before it already does t=
he
> > job.
>
> I wonder: this regexp is straight from postgrey and I'm not sure what
> the '\b' word-boundary will match. I did a quick google search and foun=
d
> no detailed information. I don't even know if it is locale-dependent.
> Anyone on the list knows the details ?

Check "man perlre". It seems it would match beginning or end of expressio=
n=20
[^$] and "word boundaries", which usually means the position between alnu=
m=20
and non-alnum chars.

I usually avoid using those "backslashed shortcuts" in writing regexps, a=
s I=20
find that it makes them harder to read and understand...

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

Re: [Sqlgrey-users] VERP patch

[Michel B. <mi...@bo...>]

Le Mercredi 16 F=E9vrier 2005 10:37, Lionel Bouton a =E9crit :
>
> Thanks, good to know that : I had no example to check against :-/

You can install libsrs2 on your system and test manually using the provid=
ed=20
"srs" program that lets you transform from the command line any email add=
ress=20
back and forth into SRS

--=20
Michel Bouissou <mi...@bo...> OpenPGP ID 0xDDE8AC6E

J'aime le travail : il me fascine. Je peux rester des heures =E0 le regar=
der.
	-- Jerome K. Jerome