sqlgrey-users — Users of SQLgrey

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Michael S. <Mic...@lr...>]

On Fri, 12 Aug 2005, Lionel Bouton wrote:

>
> X-Greylist: delayed 01:21:26.853989 by SQLgrey-1.6.5 for ad...@ex...
> X-Greylist: recipient auto-whitelisted by SQLgrey-1.6.5 for
> for...@ex...
> X-Greylist: greylisting inactive for cri...@ex... in
> SQLgrey-1.6.5
>

This will mean, that in some rare situations BCCed recipients will see
each other :-(

Michael Storz
-------------------------------------------------
Leibniz-Rechenzentrum   !   <mailto:St...@lr...>
Barer Str. 21           !   Fax: +49 89 2809460
80333 Muenchen, Germany !   Tel: +49 89 289-28840

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Lionel B. <lio...@bo...>]

Sascha Lucas wrote the following on 12.08.2005 17:13 :

>> There's a problem with that: nothing prevents two recipients for the 
>> same message from being treated differently (one can match a connect 
>> entry and the following matches the AWL entry just created:  two 
>> different headers). The problem is that we don't add the header on 
>> delivery but way before that. We have no way of adding a header per 
>> RCPT TO: this is a limitation of the policy protocol. We could have 
>> more meaningful headers with the "RCPT TO" listed for each header but 
>> I'm afraid it's the best we can do.
>
>
> I dont understand. If the 1st RCPT matches the connect entry and it is 
> an "reconnect ok", then you add the X-Greylist: delayed ... header. 
> The other RCPTs would result in a match of the from_awl table which 
> would result in a X-Greylist: from auto-whitelisted ... header. But 
> you know this instance=... allready gots it's header so sqlgrey don't 
> say action=PREPEND....


Ok, adding only the first header would indeed being meaningful in the 
example above. I was thinking of the future evolutions of SQLgrey and 
didn't explained myself well.

The above works only because we don't use AWLs based on recipients... 
*yet*. In the future, SQLgrey will have an intermediate connect_awl 
table (with src, sender and recipient) and a rcpt_awl (with only src, 
recipients) in addition to the current from_awl and domain_awl (this 
will solve some problems - false positive and negatives that could be 
avoided - seen in the wild and discussed earlier on this list). They 
will both introduce more complex cases that won't be resolvable by 
allowing only a single prepend to the same instance (simply because a 
single mail to different recipients would be accepted for totally 
different reasons in some cases).
Even if awls using "rcpt to"s wouldn't be used, maintaining a backlog of 
past instances is not that simple (nothing tells us that an instance is 
gone, so we will have to use heuristics to clean this backlog).
There is the case of optin/optout too: the prepended header for one 
"optout" address has nothing to do with the awl match of another.

Currently the multiple X-Greylist headers looks like a bug at first 
glance (I've already had reports on this). I believe adding the "rcpt 
to" matching each header to make them less confusing is the right thing 
to do. In a mail sent to 3 recipients, one of which happens to get a 
match in connect (another unrelated mail created the connect entry for 
example), another already being auto-whitelisted for this source (future 
rcpt_awl) and a last one which doesn't want to use greylisting at all 
(current optout mechanism):

X-Greylist: delayed 01:21:26.853989 by SQLgrey-1.6.5 for ad...@ex...
X-Greylist: recipient auto-whitelisted by SQLgrey-1.6.5 for 
for...@ex...
X-Greylist: greylisting inactive for cri...@ex... in 
SQLgrey-1.6.5

On a purely theoritical level, this doesn't really make sense to try to 
consolidate all headers in a single one. All decisions are currently 
completely independent from each other and only happen to be losely 
related because of some choices in the current AWLs design that *will* 
evolve.

Lionel.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Sascha L. <sas...@ru...>]

> There's a problem with that: nothing prevents two recipients for the same 
> message from being treated differently (one can match a connect entry and the 
> following matches the AWL entry just created:  two different headers). The 
> problem is that we don't add the header on delivery but way before that. We 
> have no way of adding a header per RCPT TO: this is a limitation of the 
> policy protocol. We could have more meaningful headers with the "RCPT TO" 
> listed for each header but I'm afraid it's the best we can do.

I dont understand. If the 1st RCPT matches the connect entry and it is an 
"reconnect ok", then you add the X-Greylist: delayed ... header. The other 
RCPTs would result in a match of the from_awl table which would result in 
a X-Greylist: from auto-whitelisted ... header. But you know this 
instance=... allready gots it's header so sqlgrey don't say 
action=PREPEND....

Please tell me what point I misunderstand.

Thanks,

Sascha.

Re: [Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Lionel B. <lio...@bo...>]

Sascha Lucas wrote the following on 12.08.2005 14:58 :

> Hi list,
>
> I have configured sqlgrey-1.6.5 to PREPEND the X-Greylist: header. 
> What I noticed is that the header was prepended multiple times (for 
> each RCPT TO) in the mail. The logfile shows also "sqlgrey: grey: from 
> awl match: updating" for each RCPT TO.
>
> My question is: can the instance= in the postfix policy protocol be 
> used, to prevent multiple action=PREPEND .... ? From postfix's 
> SMTPD_POLICY_READM the instance=... can be used to correlate different 
> requests regarding the same message delivery.


There's a problem with that: nothing prevents two recipients for the 
same message from being treated differently (one can match a connect 
entry and the following matches the AWL entry just created:  two 
different headers). The problem is that we don't add the header on 
delivery but way before that. We have no way of adding a header per RCPT 
TO: this is a limitation of the policy protocol. We could have more 
meaningful headers with the "RCPT TO" listed for each header but I'm 
afraid it's the best we can do.

Lionel.

[Sqlgrey-users] prevent multiple headers prepended for each RCPT

[Sascha L. <sas...@ru...>]

Hi list,

I have configured sqlgrey-1.6.5 to PREPEND the X-Greylist: header. What I 
noticed is that the header was prepended multiple times (for each RCPT TO) 
in the mail. The logfile shows also "sqlgrey: grey: from awl match: 
updating" for each RCPT TO.

My question is: can the instance= in the postfix policy protocol be 
used, to prevent multiple action=PREPEND .... ? From postfix's 
SMTPD_POLICY_READM the instance=... can be used to correlate different 
requests regarding the same message delivery.

THX,

Sascha.

Re: [Sqlgrey-users] optout question

[Lionel B. <lio...@bo...>]

Beast wrote the following on 12.08.2005 09:29 :

>
> What if we can edit file instead of direct modification to database?
> Just a suggestion.
>
That was a possibility... but I wanted to allow mail admins to easily 
integrate optout/optin into their admin infrastructure. An ISP should be 
able to add a switch somewhere in its user interface to allow its user 
to decide themselves if they want the greylisting. This is far easier if 
optin/optout decision is stored in database.

Checking files for modification is time-consuming too (especially when 
they become large). SQLgrey only does this for the "local" whitelists 
which should:
- change rarely,
- remain quite small as their content is fed to the main whitelists upon 
notification on this list.

If you don't want to develop a web-based interface, editing database 
entries is rather straightforward if you install a generic web-based 
database admin tool like phpPgAdmin (PostgreSQL) or phpMyAdmin (MySQL) 
(I'm not sure for SQLite but I believe there is an admin package out 
there too).

Lionel.

Re: [Sqlgrey-users] optout question

[Beast <be...@i6...>]

Lionel Bouton wrote:
> Beast wrote the following on 11.08.2005 13:36 :
> 
>>
>> Some of my user did not want their email delayed, so I have to exclude 
>> them from sqlgrey.
>>
>> These step is required:
>> 1. edit sqlgrey.conf
>>
>> optmethod = optout
>>
>> 2. add "sp...@my..." in optout_email table.
>>
>> 3. restart sqlgrey.
>>
>> Is that correct?
>>
> 
> Yes it is.
> 

What if we can edit file instead of direct modification to database?
Just a suggestion.

-- 

--beast

Re: [Sqlgrey-users] AWL

[Lionel B. <lio...@bo...>]

Beast wrote the following on 11.08.2005 14:35 :

>
> AWL is based on domain or per email address?


Both. The domain AWL is used when enough entries for the same domain/src 
couple exist in the mail AWL.

>
> What happen when sender using more than one mail server (with same 
> return path address)?
>

 From the example sqlgrey.conf:

## Greylisting method:
# - full   : greylist by IP address
# - classc : greylist by class C network. eg:
#            2.3.4.6 connection accepted if 2.3.4.145 did connect earlier
# - smart  : greylist by class C network unless there is no reverse lookup
#            or it looks like a home-user address
# Default is smart
# greymethod = smart

So if the 2 servers are in the same class C network, the AWL entry is 
reused in 'classc' and may be in 'smart' depending on the reverse lookup.
If the 2 servers aren't in the same class C network, SQLgrey greylists 
and create a new AWL entry.

> In sqlgrey.conf:
>
> group_domain_level = 2
>
> 2 means 2 different email address or 2 different email (with same 
> address)?
>

2 different email addresses in the same domain.

Lionel.

Re: [Sqlgrey-users] Status of sqlgrey development

[Lionel B. <lio...@bo...>]

Michael Storz wrote the following on 11.08.2005 14:12 :

>Hi Lionel,
>
>since your answers are coming so fast, you do not seem to be in vacation ;-)
>
>  
>

I'm not but I've not had much free time to work on SQLgrey either.

>What is the status of the sqlgrey development? On which parts are you
>working now?
>
>  
>

As you know I've began looking at your patches but didn't work much on 
it since our last mail exchanges.

>I got the book "Programming the Perl DBI" yesterday and now I try to
>understand all this database stuff in sqlgrey.
>
>And I am still trying to figure out, how sqlgrey could be broken up into
>objects, classes, modules ... It seems to be more work than I thought.
>  
>

I'll take vacations from August 22 to 29, after that I'll have my 
batteries fully charged and probably more free time. I expect 1.7.x 
development to resume in September.

Lionel

[Sqlgrey-users] AWL

[Beast <be...@i6...>]

AWL is based on domain or per email address?

What happen when sender using more than one mail server (with same 
return path address)?

In sqlgrey.conf:

group_domain_level = 2

2 means 2 different email address or 2 different email (with same address)?

Tks.




-- 

--beast

[Sqlgrey-users] Status of sqlgrey development

[Michael S. <Mic...@lr...>]

Hi Lionel,

since your answers are coming so fast, you do not seem to be in vacation ;-)

What is the status of the sqlgrey development? On which parts are you
working now?

I got the book "Programming the Perl DBI" yesterday and now I try to
understand all this database stuff in sqlgrey.

And I am still trying to figure out, how sqlgrey could be broken up into
objects, classes, modules ... It seems to be more work than I thought.

Michael Storz
-------------------------------------------------
Leibniz-Rechenzentrum   !   <mailto:St...@lr...>
Barer Str. 21           !   Fax: +49 89 2809460
80333 Muenchen, Germany !   Tel: +49 89 289-28840

Re: [Sqlgrey-users] optout question

[Michael S. <Mic...@lr...>]

On Thu, 11 Aug 2005, Beast wrote:

>
> Some of my user did not want their email delayed, so I have to exclude
> them from sqlgrey.
>
> These step is required:
> 1. edit sqlgrey.conf
>
> optmethod = optout
>
> 2. add "sp...@my..." in optout_email table.
>
> 3. restart sqlgrey.
>
> Is that correct?
>

Yes, this is correct. After a change of the config file, you have to
restart sqlgrey. After that you can add entries to the optin/out tables at
any time without restarting sqlgrey.

Michael Storz
-------------------------------------------------
Leibniz-Rechenzentrum   !   <mailto:St...@lr...>
Barer Str. 21           !   Fax: +49 89 2809460
80333 Muenchen, Germany !   Tel: +49 89 289-28840

Re: [Sqlgrey-users] sqlgrey dies

[Michael S. <Mic...@lr...>]

On Thu, 11 Aug 2005, Beast wrote:

> Lionel Bouton wrote:
> > Beast wrote the following on 11.08.2005 08:45 :
> >
> >> [...]
> >> It is because of mail from using "%". I have tried sending mail with
> >> this kind of address (from any host) and sqlgrey dies, probably
> >> because of unquote '%'.
> >>
> >> MAIL FROM: <LNGNOTES/LNGJAPAN%LNG...@ln...>
> >>
> >>
> >
> > This problem was fixed with the 1.6.2 release. Just install latest
> > stable SQLgrey (1.6.5) and the problem will go away.
>
> Do I need to recreate mysql table after upgrading?
> Tks.
>
You don't have to, sqlgrey handles necessary upgrades of the tables
automatically.

Michael Storz
-------------------------------------------------
Leibniz-Rechenzentrum   !   <mailto:St...@lr...>
Barer Str. 21           !   Fax: +49 89 2809460
80333 Muenchen, Germany !   Tel: +49 89 289-28840

Re: [Sqlgrey-users] optout question

[Lionel B. <lio...@bo...>]

Beast wrote the following on 11.08.2005 13:36 :

>
> Some of my user did not want their email delayed, so I have to exclude 
> them from sqlgrey.
>
> These step is required:
> 1. edit sqlgrey.conf
>
> optmethod = optout
>
> 2. add "sp...@my..." in optout_email table.
>
> 3. restart sqlgrey.
>
> Is that correct?
>

Yes it is.

Re: [Sqlgrey-users] sqlgrey dies

[Lionel B. <lio...@bo...>]

Beast wrote the following on 11.08.2005 13:40 :

> Lionel Bouton wrote:
>
>> [...]
>> This problem was fixed with the 1.6.2 release. Just install latest 
>> stable SQLgrey (1.6.5) and the problem will go away.
>
>
> Do I need to recreate mysql table after upgrading?
> Tks.
>
No. SQLgrey upgrades never need database administration: tables are 
automatically converted to new formats and new tables are created by 
SQLgrey itself when needed.

Lionel.

Re: [Sqlgrey-users] sqlgrey dies

[Beast <be...@i6...>]

Lionel Bouton wrote:
> Beast wrote the following on 11.08.2005 08:45 :
> 
>> [...]
>> It is because of mail from using "%". I have tried sending mail with 
>> this kind of address (from any host) and sqlgrey dies, probably 
>> because of unquote '%'.
>>
>> MAIL FROM: <LNGNOTES/LNGJAPAN%LNG...@ln...>
>>
>>
> 
> This problem was fixed with the 1.6.2 release. Just install latest 
> stable SQLgrey (1.6.5) and the problem will go away.

Do I need to recreate mysql table after upgrading?
Tks.

-- 

--beast

[Sqlgrey-users] optout question

[Beast <be...@i6...>]

Some of my user did not want their email delayed, so I have to exclude 
them from sqlgrey.

These step is required:
1. edit sqlgrey.conf

optmethod = optout

2. add "sp...@my..." in optout_email table.

3. restart sqlgrey.

Is that correct?



-- 

--beast

Re: [Sqlgrey-users] sqlgrey dies

[Beast <be...@i6...>]

Lionel Bouton wrote:
> Beast wrote the following on 11.08.2005 08:45 :
> 
>> [...]
>> It is because of mail from using "%". I have tried sending mail with 
>> this kind of address (from any host) and sqlgrey dies, probably 
>> because of unquote '%'.
>>
>> MAIL FROM: <LNGNOTES/LNGJAPAN%LNG...@ln...>
>>
>>
> 
> This problem was fixed with the 1.6.2 release. Just install latest 
> stable SQLgrey (1.6.5) and the problem will go away.
> 

Tks. I've reject it on sender_access.

-- 

--beast

Re: [Sqlgrey-users] sqlgrey dies

[Lionel B. <lio...@bo...>]

Beast wrote the following on 11.08.2005 08:45 :

> [...]
> It is because of mail from using "%". I have tried sending mail with 
> this kind of address (from any host) and sqlgrey dies, probably 
> because of unquote '%'.
>
> MAIL FROM: <LNGNOTES/LNGJAPAN%LNG...@ln...>
>
>

This problem was fixed with the 1.6.2 release. Just install latest 
stable SQLgrey (1.6.5) and the problem will go away.

Lionel

Re: [Sqlgrey-users] sqlgrey dies

[Beast <be...@i6...>]

Beast wrote:
> 
> I think I've found the problem, it always dies whenever 
> 061196164024.cidr.odn.ne.jp send an email.
> 
> ---
> Aug 11 12:55:58 blowfish sqlgrey: optin: greylisting active for 
> xx...@my...
> Aug 11 12:55:58 blowfish sqlgrey: grey: identified dynamic pattern (last 
> IP byte): 061196164024.cidr.odn.ne.jp, 61.196.164.24: Using full IP.
> Aug 11 12:55:58 blowfish sqlgrey: fatal: Modification of a read-only 
> value attempted at 
> /usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.
> Aug 11 12:55:58 blowfish postfix/smtpd[5359]: warning: premature 
> end-of-input on 127.0.0.1:2501 while reading input attribute name
> 

It is because of mail from using "%". I have tried sending mail with 
this kind of address (from any host) and sqlgrey dies, probably because 
of unquote '%'.

MAIL FROM: <LNGNOTES/LNGJAPAN%LNG...@ln...>





-- 

--beast

Re: [Sqlgrey-users] sqlgrey dies

[Beast <be...@i6...>]

Beast wrote:
> Beast wrote:
> 
>>
>> This morning I found that sqlgrey daemon dies.
>>
>> Aug 11 07:35:22 blowfish sqlgrey: fatal: Modification of a read-only 
>> value attempted at 
>> /usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.
>> Aug 11 07:35:22 blowfish postfix/smtpd[4367]: warning: premature 
>> end-of-input on 127.0.0.1:2501 while reading input attribute name
>> Aug 11 07:35:23 blowfish postfix/smtpd[4367]: warning: connect to 
>> 127.0.0.1:2501: Connection refused
>>
>> What could be the reason?
>>
>> sqlgrey ver. 1.5.7
>>
>>
>>
> 
> This is the first time after running smoothly for more than 2 months, 
> today it dies 3 times. I think some mail could trigger it, but found 
> nothing on the log file.
> 
> [root@blowfish log]# grep fatal maillog
> Aug 11 07:35:22 blowfish sqlgrey: fatal: Modification of a read-only 
> value attempted at 
> /usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.
> Aug 11 10:38:12 blowfish sqlgrey: fatal: Modification of a read-only 
> value attempted at 
> /usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.
> Aug 11 12:09:51 blowfish sqlgrey: fatal: Modification of a read-only 
> value attempted at 
> /usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.
> 
> I'm setting loglevel to 4 now and see what happen in the next few hours :-(
> 

I think I've found the problem, it always dies whenever 
061196164024.cidr.odn.ne.jp send an email.

---
Aug 11 12:55:58 blowfish sqlgrey: optin: greylisting active for 
xx...@my...
Aug 11 12:55:58 blowfish sqlgrey: grey: identified dynamic pattern (last 
IP byte): 061196164024.cidr.odn.ne.jp, 61.196.164.24: Using full IP.
Aug 11 12:55:58 blowfish sqlgrey: fatal: Modification of a read-only 
value attempted at 
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.
Aug 11 12:55:58 blowfish postfix/smtpd[5359]: warning: premature 
end-of-input on 127.0.0.1:2501 while reading input attribute name



-- 

--beast

Re: [Sqlgrey-users] sqlgrey dies

[Beast <be...@i6...>]

Beast wrote:
> 
> This morning I found that sqlgrey daemon dies.
> 
> Aug 11 07:35:22 blowfish sqlgrey: fatal: Modification of a read-only 
> value attempted at 
> /usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.
> Aug 11 07:35:22 blowfish postfix/smtpd[4367]: warning: premature 
> end-of-input on 127.0.0.1:2501 while reading input attribute name
> Aug 11 07:35:23 blowfish postfix/smtpd[4367]: warning: connect to 
> 127.0.0.1:2501: Connection refused
> 
> What could be the reason?
> 
> sqlgrey ver. 1.5.7
> 
> 
> 

This is the first time after running smoothly for more than 2 months, 
today it dies 3 times. I think some mail could trigger it, but found 
nothing on the log file.

[root@blowfish log]# grep fatal maillog
Aug 11 07:35:22 blowfish sqlgrey: fatal: Modification of a read-only 
value attempted at 
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.
Aug 11 10:38:12 blowfish sqlgrey: fatal: Modification of a read-only 
value attempted at 
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.
Aug 11 12:09:51 blowfish sqlgrey: fatal: Modification of a read-only 
value attempted at 
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.

I'm setting loglevel to 4 now and see what happen in the next few hours :-(




-- 

--beast

[Sqlgrey-users] sqlgrey dies

[Beast <be...@i6...>]

This morning I found that sqlgrey daemon dies.

Aug 11 07:35:22 blowfish sqlgrey: fatal: Modification of a read-only 
value attempted at 
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/Sys/Syslog.pm line 312.
Aug 11 07:35:22 blowfish postfix/smtpd[4367]: warning: premature 
end-of-input on 127.0.0.1:2501 while reading input attribute name
Aug 11 07:35:23 blowfish postfix/smtpd[4367]: warning: connect to 
127.0.0.1:2501: Connection refused

What could be the reason?

sqlgrey ver. 1.5.7



-- 

--beast

Re: [Sqlgrey-users] 1.6.x and 1.7.x freezes

[Rupa S. (lists) <rup...@ru...>]

On 7/26/2005 10:46 AM, Lionel Bouton wrote:
 > Good catch again :-)
> This start/fork_cleanup mixup is odd, I remember having checked an
> intermediate version for such cases (I was initialy unsure on how to
> name these methods) and corrected them. I may have mixed up borked and
> cleaned version at some point before commiting...

Glad to be of help -- I know I've done similar things during the
'cleanup' phase after doing some coding.

 > I'll drop support for cleanup_method and only leave support for
> clean_method (TODO: make the config parser aware of supported options
> and complain when unknown are found). This was the only config entry
> advertised in the etc/sqlgrey.conf example.
> 
> Preparing a 1.6.5 version now.

Sounds good to me.  I've been running this way now for a couple of days
and have not had a lockup.

> Lionel
> 
> PS: your strace following the two processes might help understand what's
> going on when using tha async method, I'll try to give it a shot later.
> 

Great!  The lockup with async came on suddenly so I don't know if it'll
magically fix itself.  When you have something to test I'll give it a go.

-- 
 -Rupa

RE: [Sqlgrey-users] Postgresql + lost connection

[Daniel V. P. <dv...@cs...>]

> -----Original Message-----
> From: sql...@li... [mailto:sqlgrey-users-
> ad...@li...] On Behalf Of Lionel Bouton
> Sent: 27. juli 2005 18:54
> 
> All versions before 1.6.5 didn't use sync when told to do so. Some
> configurations have database related problems with async...

Yup, upgraded to 1.6.5 and so far no errors (been running about an hour)
So that seems to have solved it, cheers.

/Dan.