Menu
▾
▴
sqlgrey-users
|
From: Dave S. <dst...@ma...> - 2007-01-21 16:03:07
|
I am looking for some configuration advice from the list.=20 =20 Currently we're using SQLGrey v1.6.7, and we are getting a lot of = "warning: connect to 127.0.0.1:2501: Connection timed out". They seem to = come in clumps, as if the SQLGrey code was very busy, and couldn't answer = the requests. This gives a "450 Server configuration problem" to the = Sender, and makes the sender think we have done something very bad. =20 Now, I will say that we run the code on many servers, and each of them get = about a million inbound emails per day. Of course, most of that is junk, = but SQLGrey needs to process it anyway. So, SQLGrey is working very hard. = However, even on off-hours (nights & weekends), we still get these. =20 We're running Postgres for our server, and the DB is not being hit = excessively when we get these, and all the hardware we run is top-notch. =20 My questions are: =20 1. Does anyone else get these and is there a fix ? =20 2. What's the "preferred" method of setting up your main.cf and master.cf = files for SQLGrey ? =20 Thanks, =20 -- Dave This message has been certified virus-free by MailWise Filter - The real-ti= me, intelligent, e-mail firewall used to scan inbound and outbound messages = for SPAM, Viruses and Content. =0A=0A For more information, please visit: http:= //www.mailwise.com=0A |
|
From: Dan F. <da...@ha...> - 2007-01-21 17:37:15
|
Dave Strickler wrote:
> Currently we're using SQLGrey v1.6.7, and we are getting a lot of "warning: connect to 127.0.0.1:2501: Connection timed out". They seem to come in clumps, as if the SQLGrey code was very busy, and couldn't answer the requests. This gives a "450 Server configuration problem" to the Sender, and makes the sender think we have done something very bad.
>
We have never had the problem with sqlgrey you describe.
I can't solve your problem, but i can tell you what we have made our setup:
In our cluster of mailservers, each server is scaled to handle about
600.000 mail-requests daily. (some a little more or a little less based
on hardware). The same cluster nodes run 2 other policy-deamons, our own
"malware-sandbox", spamassasin and virusscanning using 3 scanners. On
each node, that is.
In peak-hour they sometimes have a load of 4.0-8.0, but they still run
great.
In master.cf we have raised the limit of simultaneous smtpd to 500,
based on real-life measurements (debian default is 50 i think). So
thats maximum 500 simultaneous request for sqlgrey as well.
We use sqlgrey's db_clustering. All writes go to a central MySQL, read
request go to a MySQL on 127.0.0.1.
The MySQL clutster also handles postfix'es Access, Virtual & transport
tables and Spam/Virus quarantine. So SQL is pretty busy as well.
Sqlgrey is configured in Postfix main.cf as such:
---------------------
smtpd_restriction_classes = sqlgrey
sqlgrey = check_policy_service inet:127.0.0.1:2501
smtpd_sender_restrictions =
sqlgrey
<other restrictions go here>
---------------------
Most hardware is:
- Dual Xeon 3.00GHz
- 2 x 60gig scsi discs, running as a raid MIRROR (not for speed
obviously, but for safety)
- 1 gig ram.
Dont know if this helps you in any way :)
- Dan
|
|
From: Dave S. <dst...@ma...> - 2007-01-21 20:40:56
|
Actually it does help - thanks!
=20
It gives me a sense that we're not using SQLGrey in a strange way, or have =
it configured wrong. We have made several mods to the code, but we have =
had these "450 Server Config Problems" long before the mods.
=20
Question: Do you run the cleanup in Async or Sync mode? We have been =
running Async, and I have always feared this was a problem, even though we =
have tested Sync mode and gotten the same errors.
=20
I also noted that you run MySQL instead of Postgres. It would be possible =
for us to switch, but certainly difficult to try just for testing.
=20
Also, we almost always get these errors at the top of the hour. The only =
cronjob that runs at the top of the hour is a process that rolls our log =
files, and gzips them for us. During the gzip process the load spikes a =
little, but not much, even through the gzip process may take 90% of the =
CPU for 30-60 seconds.
=20
Thanks,
=20
-- Dave
>>> Dan Faerch <da...@ha...> 12:37 PM Sunday, January 21, 2007 >>>
Dave Strickler wrote:
> Currently we're using SQLGrey v1.6.7, and we are getting a lot of =
"warning: connect to 127.0.0.1:2501: Connection timed out". They seem to =
come in clumps, as if the SQLGrey code was very busy, and couldn't answer =
the requests. This gives a "450 Server configuration problem" to the =
Sender, and makes the sender think we have done something very bad.
> =20
We have never had the problem with sqlgrey you describe.
I can't solve your problem, but i can tell you what we have made our =
setup:
In our cluster of mailservers, each server is scaled to handle about=20
600.000 mail-requests daily. (some a little more or a little less based=20
on hardware). The same cluster nodes run 2 other policy-deamons, our =
own=20
"malware-sandbox", spamassasin and virusscanning using 3 scanners. On=20
each node, that is.
In peak-hour they sometimes have a load of 4.0-8.0, but they still run=20
great.
In master.cf we have raised the limit of simultaneous smtpd to 500,=20
based on real-life measurements (debian default is 50 i think). So=20
thats maximum 500 simultaneous request for sqlgrey as well.
We use sqlgrey's db_clustering. All writes go to a central MySQL, read=20
request go to a MySQL on 127.0.0.1.
The MySQL clutster also handles postfix'es Access, Virtual & transport=20
tables and Spam/Virus quarantine. So SQL is pretty busy as well.
Sqlgrey is configured in Postfix main.cf as such:
---------------------
smtpd_restriction_classes =3D sqlgrey
sqlgrey =3D check_policy_service inet:127.0.0.1:2501
smtpd_sender_restrictions =3D
sqlgrey
<other restrictions go here>
---------------------
Most hardware is:
- Dual Xeon 3.00GHz
- 2 x 60gig scsi discs, running as a raid MIRROR (not for speed=20
obviously, but for safety)
- 1 gig ram.
Dont know if this helps you in any way :)
- Dan
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share =
your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=3Djoin.php&p=3Dsourceforge&CID=3DDE=
VDEV=20
_______________________________________________
Sqlgrey-users mailing list
Sql...@li...=20
https://lists.sourceforge.net/lists/listinfo/sqlgrey-users
This message has been certified virus-free by MailWise Filter - The real-ti=
me, intelligent, e-mail firewall used to scan inbound and outbound messages =
for SPAM, Viruses and Content. =0A=0A For more information, please visit: http:=
//www.mailwise.com=0A
|
|
From: Dan F. <da...@ha...> - 2007-01-21 21:34:04
|
Dave Strickler wrote:
> had these "450 Server Config Problems" long before the mods.
>
Well.. Ive had that error plenty of times on our own, self-coded
policy-deamons some years ago ;). I didnt mention it because your error
is "connection timeout", while mine is "connection refused". The latter
indicates that the port is no longer listening (eg. the deamon crashed
and halted), where your error seems to indicate very slow response.
> Question: Do you run the cleanup in Async or Sync mode? We have been running Async, and I have always feared this was a problem, even though we have tested Sync mode and gotten the same errors.
>
Due to the warning in sqlgrey.conf ("BEWARE: lockups have been
reported"), we do NOT use async. I havent even tested async.
> I also noted that you run MySQL instead of Postgres. It would be possible for us to switch, but certainly difficult to try just for testing.
>
Hmm. Dunno.. This day and age, MySQL and Postgres should be pretty equal
in perfomance. However i never REALLY used postgres, so i can say for sure.
> Also, we almost always get these errors at the top of the hour. The only cronjob that runs at the top of the hour is a process that rolls our log files, and gzips them for us. During the gzip process the load spikes a little, but not much, even through the gzip process may take 90% of the CPU for 30-60 seconds.
>
Well.. You could change the cron time 30 minutes and see if it still
coincides with the error.
I remember hunting alot of errors like this..
With postgrey i got "450 Server configuration error" if the cleanup
didnt finish quickly enough.. So i disabled cleaup, and had it run at 5
am from a cronjob, then restart postfix to make sure.
One of our other policy-daemons had a tendency of hanging, not every
day, but always around 6am when it happend. Turns out logrotate ran at
6am. And logrotate stops syslog while rotating.
We have an outstanding bug in sqlgrey (due to be fixed very soon) that
kills sqlgrey if syslog isnt running and sqlgrey is trying to write a
log entry. If you rotate your logs every hour, that may include
restarting syslog. If a mail transaction is initiated in the time it
takes to rotate your logs, sqlgrey will probably die. That would explain
your problem, however logic dictates that it would result in "Connection
refused", not a timeout.. But i may be wrong :)
- Dan
|
|
From: Dan F. <da...@ha...> - 2007-01-24 19:05:31
|
Patch to change db_cleanup timekeeping for both normal and clustered setup. To fix item from tracker: [ 1574884 ] db_clean_host name and HOSTNAME Feel free to check it and make comments. Notes on patch: Old HOSTNAME codes is not removed yet to keep this diff simple to read. Added new value into "config" table. "last_dbclean". Contains unix timestampor zero. Changed/corrected updateconfig() to return number of affected rows Changed updateconfig() to allow 1 extra arg. If given, the arg will be added to sql's where statement: "AND value='$arg'" (used for "old_timestamp") Removed "next_maint" everywhere. Removed dbclean on startup - Dan |
|
From: Dan F. <da...@ha...> - 2007-01-24 19:07:13
Attachments:
sqlgrey-last_dbclean.patch
|
Forgot to attach patch.. Dan Faerch wrote: > Patch to change db_cleanup timekeeping for both normal and clustered setup. > > To fix item from tracker: > [ 1574884 ] db_clean_host name and HOSTNAME > > Feel free to check it and make comments. > > Notes on patch: > Old HOSTNAME codes is not removed yet to keep this diff simple to read. > > Added new value into "config" table. "last_dbclean". Contains unix > timestampor zero. > Changed/corrected updateconfig() to return number of affected rows > Changed updateconfig() to allow 1 extra arg. If given, the arg will be > added to sql's where statement: "AND value='$arg'" (used for > "old_timestamp") > Removed "next_maint" everywhere. > Removed dbclean on startup > > > > - Dan > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys - and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Sqlgrey-users mailing list > Sql...@li... > https://lists.sourceforge.net/lists/listinfo/sqlgrey-users > > |
|
From: Lionel B. <lio...@bo...> - 2007-01-29 18:40:39
|
Hi, I'm back from my winter holidays. I initially liked your patch to updateconfig because it seemed cleaner than what we discussed (at least less complex) but then I realized there are 2 bugs. update_config doesn't try to check for an $old_value if it is 0 which makes it impossible to check for a successful update from a value expected to be 0. This is a corner case, but you can avoid it by using (defined $old_value) ? ... instead of ($old_value) ? ... This is only a design problem with no actual consequence now but it's safer to avoid any complication... Anyway the second bug is quite serious and you'll have to remove the patch to update_config unless you find another need for it... Given that you only check for equality and not the timestamp being expired at most only one SQLgrey instance will ever update the value and then clean up (looking at the details, this will be the last instance being launched). If this instance is shut down no other instance will try to cleanup. You could even end with no cleanup occuring if the two successive calls to time() give back two different values (your local copy won't match the database), this could be fixed, but you can't avoid the "cleanup" instance shuting down stops cleanup if you check for equality. Conclusion : you can't use the config table, given that it only stores varchars (you can't compare timestamps stored as varchars). You need to create another table with only one stored value in a timestamp column and check for expiration not equality. Lionel. |
|
From: Dan F. <da...@ha...> - 2007-01-29 23:03:08
|
Welcome back. Hope you enjoyed you trip :)
Bug 1:
>(defined $old_value) ? ...
>instead of
>($old_value) ? .
Correct. Nicely spotted.
Bug 2:
>If this instance is shut down no other instance will
>try to cleanup. You could even end with no cleanup occuring if the two
>successive calls to time() give back two different values (your local
>copy won't match the database),
If i understand you correctly, i disagree with your analysis. I believe the patch handles this perfectly.
The patch makes it like this:
- If an UPDATE fails (because of timestamp mismatch) then $last_dbclean = 0;
And just before the cleanup time comparision, there is an:
- If $last_dbclean == 0 then reload value from db.
Which basically means:
if UPDATE affected_rows = 0 then SELECT last_dbclean from config.
Thus, if db_cleantime = 30mins, all sqlgrey's, except the one who did the cleaning, reloads the timestamp from DB every 30mins. If no-one did the cleaning, everyone will reload and the cycle will continue.
Relevant code:
#If last_dbclean == 0, reload value from DB
if ($self->{sqlgrey}{last_dbclean} == 0) {
$self->{sqlgrey}{last_dbclean} = $self->getconfig('last_dbclean');
.......
# updateconfig() returns affected_rows
if ($self->updateconfig('last_dbclean',time(),$self->{sqlgrey}{last_dbclean})) {
...cut...
} else {
#If affected_rows == 0, then someone already cleaned db
$self->{sqlgrey}{last_dbclean} = 0; #make sqlgrey reload time from db on next pass
}
- Dan
Lionel Bouton wrote:
> Hi,
>
> I'm back from my winter holidays. I initially liked your patch to
> updateconfig because it seemed cleaner than what we discussed (at least
> less complex) but then I realized there are 2 bugs. update_config
> doesn't try to check for an $old_value if it is 0 which makes it
> impossible to check for a successful update from a value expected to be
> 0. This is a corner case, but you can avoid it by using
> (defined $old_value) ? ...
> instead of
> ($old_value) ? ...
> This is only a design problem with no actual consequence now but it's
> safer to avoid any complication... Anyway the second bug is quite
> serious and you'll have to remove the patch to update_config unless you
> find another need for it...
>
> Given that you only check for equality and not the timestamp being
> expired at most only one SQLgrey instance will ever update the value and
> then clean up (looking at the details, this will be the last instance
> being launched). If this instance is shut down no other instance will
> try to cleanup. You could even end with no cleanup occuring if the two
> successive calls to time() give back two different values (your local
> copy won't match the database), this could be fixed, but you can't avoid
> the "cleanup" instance shuting down stops cleanup if you check for equality.
>
> Conclusion : you can't use the config table, given that it only stores
> varchars (you can't compare timestamps stored as varchars). You need to
> create another table with only one stored value in a timestamp column
> and check for expiration not equality.
>
> Lionel.
>
>
|
|
From: Lionel B. <lio...@bo...> - 2007-01-31 08:42:25
Attachments:
db_clean.patch
|
Dan Faerch wrote the following on 30.01.2007 00:02 :
> Welcome back. Hope you enjoyed you trip :)
>
>
> Bug 1:
>
>> (defined $old_value) ? ...
>> instead of
>> ($old_value) ? .
>>
> Correct. Nicely spotted.
>
> Bug 2:
>
>> If this instance is shut down no other instance will
>> try to cleanup. You could even end with no cleanup occuring if the two
>> successive calls to time() give back two different values (your local
>> copy won't match the database),
>>
>
> If i understand you correctly, i disagree with your analysis. I believe the patch handles this perfectly.
>
> The patch makes it like this:
> - If an UPDATE fails (because of timestamp mismatch) then $last_dbclean = 0;
> And just before the cleanup time comparision, there is an:
> - If $last_dbclean == 0 then reload value from db.
>
> Which basically means:
> if UPDATE affected_rows = 0 then SELECT last_dbclean from config.
>
> Thus, if db_cleantime = 30mins, all sqlgrey's, except the one who did the cleaning, reloads the timestamp from DB every 30mins. If no-one did the cleaning, everyone will reload and the cycle will continue.
>
>
> Relevant code:
>
> #If last_dbclean == 0, reload value from DB
> if ($self->{sqlgrey}{last_dbclean} == 0) {
> $self->{sqlgrey}{last_dbclean} = $self->getconfig('last_dbclean');
>
I see, I misread the patch and thought this code was in insertconfig. I
just looked at the whole patched script and things makes more sense to
me now. So the only things left are the (defined $old_value) and
the 2 time() calls that could introduce an inefficiency.
In fact I don't really like last_dbclean == 0 being a special case. I'm
patching the code to use undef, it makes things clearer to me.
The patch is attached and commited to CVS. Not yet tested (I've only
spent some time on it before going to work). Please tell me what you
think of it.
Lionel
|
|
From: David L. <da...@la...> - 2007-01-31 08:52:35
|
Lionel Bouton wrote:
> Dan Faerch wrote the following on 30.01.2007 00:02 :
>> Welcome back. Hope you enjoyed you trip :)
>>
>>
>> Bug 1:
>>
>>> (defined $old_value) ? ...
>>> instead of
>>> ($old_value) ? .
>>>
>> Correct. Nicely spotted.
>>
>> Bug 2:
>>
>>> If this instance is shut down no other instance will
>>> try to cleanup. You could even end with no cleanup occuring if the two
>>> successive calls to time() give back two different values (your local
>>> copy won't match the database),
>>>
>> If i understand you correctly, i disagree with your analysis. I believe the patch handles this perfectly.
>>
>> The patch makes it like this:
>> - If an UPDATE fails (because of timestamp mismatch) then $last_dbclean = 0;
>> And just before the cleanup time comparision, there is an:
>> - If $last_dbclean == 0 then reload value from db.
>>
>> Which basically means:
>> if UPDATE affected_rows = 0 then SELECT last_dbclean from config.
>>
>> Thus, if db_cleantime = 30mins, all sqlgrey's, except the one who did the cleaning, reloads the timestamp from DB every 30mins. If no-one did the cleaning, everyone will reload and the cycle will continue.
>>
>>
>> Relevant code:
>>
>> #If last_dbclean == 0, reload value from DB
>> if ($self->{sqlgrey}{last_dbclean} == 0) {
>> $self->{sqlgrey}{last_dbclean} = $self->getconfig('last_dbclean');
>>
>
> I see, I misread the patch and thought this code was in insertconfig. I
> just looked at the whole patched script and things makes more sense to
> me now. So the only things left are the (defined $old_value) and
> the 2 time() calls that could introduce an inefficiency.
I wouldn't be worried about inefficiency, it's a cheap system call. I
would be far more worried about internal inconsistencies. You'll wind up
with cases where time() rolls over to the next second in the two places
where it's called and depending on what the rest of the code does, this
can lead to bugs that are very hard to track down. And if the
update_config() takes a significant amount of time, the two values could
be very far apart.
Taking a snapshot of of time() and holding it in a variable is the right
thing to do.
> In fact I don't really like last_dbclean == 0 being a special case. I'm
> patching the code to use undef, it makes things clearer to me.
>
> The patch is attached and commited to CVS. Not yet tested (I've only
> spent some time on it before going to work). Please tell me what you
> think of it.
> # Is it time for cleanups ?
> - if (time() > ($self->{sqlgrey}{last_dbclean} + $self->{sqlgrey}{db_cleandelay})) {
> -
> + my $current_time = time();
> + if ($current_time() > ($self->{sqlgrey}{last_dbclean} + $self->{sqlgrey}{db_cleandelay})) {
^^^^^^^^^^^^^^^
That won't compile as is. I think you mean $current_time.
Later,
David
> # updateconfig() returns affected_rows
> - if ($self->updateconfig('last_dbclean',time(),$self->{sqlgrey}{last_dbclean})) {
> + if ($self->updateconfig('last_dbclean',$current_time,$self->{sqlgrey}{last_dbclean})) {
> # If affected_rows > 0, its my job to clean the db
> - $self->{sqlgrey}{last_dbclean} = time();
> + $self->{sqlgrey}{last_dbclean} = $current_time;
|
|
From: Lionel B. <lio...@bo...> - 2007-01-31 09:22:09
|
David Landgren wrote the following on 31.01.2007 09:52 :
>
>> # Is it time for cleanups ?
>> - if (time() > ($self->{sqlgrey}{last_dbclean} + $self->{sqlgrey}{db_cleandelay})) {
>> -
>> + my $current_time = time();
>> + if ($current_time() > ($self->{sqlgrey}{last_dbclean} + $self->{sqlgrey}{db_cleandelay})) {
>>
> ^^^^^^^^^^^^^^^
>
> That won't compile as is. I think you mean $current_time.
>
Yes, thanks for the typocheck, the fix has just been commited.
Lionel
|
|
From: Dan F. <da...@ha...> - 2007-01-31 19:00:42
|
> You'll wind up > with cases where time() rolls over to the next second in the two places > where it's called and depending on what the rest of the code does, this > can lead to bugs that are very hard to track down. Well I see your point and it is better the way Lionel did it. However the only thing that would happen is that "UPDATE ... last_dbtime" would fail, thus sqlgrey would reload value from DB and do cleanup 1 second later.. |
|
From: Dan F. <da...@ha...> - 2007-01-31 19:11:36
|
Lionel Bouton wrote: > The patch is attached and commited to CVS. Not yet tested (I've only > spent some time on it before going to work). Please tell me what you > think of it. > > Looks fine.. > + # 0 will force a cleanup (unless db_cleandelay is really huge) You mean, 0 will force a cleanup unless db_cleandelay > +30 years :) - Dan |
|
From: Dave S. <dst...@ma...> - 2007-01-21 16:08:05
|
I have got this up and running for the AWL code, and it seems to be = helping a little, but not a lot. Certainly the DB reads are faster, but = they are so fast to start with that I am only saving 100ths of seconds on = each one. =20 My main goal is lighten the load on Postgres during the day by reducing = the amount of SELECTs that need to be done. I will keep the listed posted = on the results I have seen, and certainly share my sloppy Perl code with = Lionel if it helps improve SQLGrey. =20 =20 Dave Strickler MailWise LLC 617-933-5810 (direct) www.mailwise.com ( http://www.mailwise.com/ ) "Intelligent E-mail Protection" This message has been certified virus-free by MailWise Filter - The real-ti= me, intelligent, e-mail firewall used to scan inbound and outbound messages = for SPAM, Viruses and Content. =0A=0A For more information, please visit: http:= //www.mailwise.com=0A |
|
From: Paul B. <pa...@hy...> - 2007-01-23 11:48:19
|
Just to confirm this has gotten rid of the error for me as well. Thanks a lot. Paul -----Original Message----- From: sql...@li... [mailto:sql...@li...] On Behalf Of Dan Faerch Sent: Saturday, January 20, 2007 1:12 PM To: sql...@li... Subject: [Sqlgrey-users] 1.7.4 warnings I need a helping hand.. >> Name "DBIx::DBCluster::DEBUG" used only once: possible typo at >> /usr/sbin/sqlgrey line 2413. >> Name "DBIx::DBCluster::WRITE_HOSTS_NEVER_READ" used only once: possible >> typo at /usr/sbin/sqlgrey line 827. >> Name "DBIx::DBCluster::CLUSTERS" used only once: possible typo at >> /usr/sbin/sqlgrey line 818.=20 I cant get these warnings on to appear any of my computers (no idea why), so i cant verify that my patch makes them go away. Could someone who actually gets these warnings apply the attached patch and check that they disappear? $ patch /path/to/your/sqlgrey < sqlgrey-trivial.patch - Dan |
|
From: Micha S. <mi...@ar...> - 2007-01-19 18:26:13
|
Dan Faerch wrote: > I agree with Lionel. > Its best letting sqlgrey do all this since you really cant tell who is > valid senders if you dont temporally reject first, and then sees who > comes back. (which is what greylisting does) > > If you are worried that delaying all mail for 5 minutes at the same day > you enable sqlgrey, consider using the "discrimination" feature from > 1.7.4+. That will let you, via regular expressions, select what to > greylist and what to just let through . Good way of slowly introducing > greylisting by making the discrimination rules more and more restrictive. > > Where can I see more details on this 1.7.4 version, and the "discrimination" feature? We've been using greylisting for several months, and while spam has indeed decreased dramatically, we have to spend time every day or two adding new entries to the client_fqdn_addr.local file because of false postivies. I think the problem is mostly mail servers that resend almost immediately and get marked as spam. Thanks, Micha |
|
From: Lionel B. <lio...@bo...> - 2007-01-19 18:33:06
|
Micha Silver wrote the following on 19.01.2007 19:25 : > Dan Faerch wrote: > > >> I agree with Lionel. >> Its best letting sqlgrey do all this since you really cant tell who is >> valid senders if you dont temporally reject first, and then sees who >> comes back. (which is what greylisting does) >> >> If you are worried that delaying all mail for 5 minutes at the same day >> you enable sqlgrey, consider using the "discrimination" feature from >> 1.7.4+. That will let you, via regular expressions, select what to >> greylist and what to just let through . Good way of slowly introducing >> greylisting by making the discrimination rules more and more restrictive. >> >> >> > Where can I see more details on this 1.7.4 version, and the > "discrimination" feature? > We've been using greylisting for several months, and while spam has > indeed decreased dramatically, we have to spend time every day or two > adding new entries to the client_fqdn_addr.local file because of false > postivies. I think the problem is mostly mail servers that resend almost > immediately and get marked as spam. > What do you mean by "marked as spam"? mails should either be rejected or received in the end never "marked as spam" (at least by SQLgrey). If you need to add entries to client_fqdn_addr.local in order to receive mail from them, I'd like to know them. Lionel. |
|
From: Micha S. <mi...@ar...> - 2007-01-19 18:59:33
|
Lionel Bouton wrote: > Micha Silver wrote the following on 19.01.2007 19:25 : > >> Where can I see more details on this 1.7.4 version, and the >> "discrimination" feature? >> We've been using greylisting for several months, and while spam has >> indeed decreased dramatically, we have to spend time every day or two >> adding new entries to the client_fqdn_addr.local file because of false >> postivies. I think the problem is mostly mail servers that resend almost >> immediately and get marked as spam. >> >> > > What do you mean by "marked as spam"? mails should either be rejected or > received in the end never "marked as spam" (at least by SQLgrey). > Sorry. I meant rejected. > If you need to add entries to client_fqdn_addr.local in order to receive > mail from them, I'd like to know them. > > Here's the current list: http://www.arava.co.il/~micha/clients_fqdn_whitelist.local They're mostly domains hosted here in Israel. Regards, Micha > Lionel. > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys - and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Sqlgrey-users mailing list > Sql...@li... > https://lists.sourceforge.net/lists/listinfo/sqlgrey-users > |
|
From: Lionel B. <lio...@bo...> - 2007-01-19 19:26:09
|
Micha Silver wrote the following on 19.01.2007 19:58 : > Lionel Bouton wrote: > > >> Micha Silver wrote the following on 19.01.2007 19:25 : >> >> >>> Where can I see more details on this 1.7.4 version, and the >>> "discrimination" feature? >>> We've been using greylisting for several months, and while spam has >>> indeed decreased dramatically, we have to spend time every day or two >>> adding new entries to the client_fqdn_addr.local file because of false >>> postivies. I think the problem is mostly mail servers that resend almost >>> immediately and get marked as spam. >>> >>> >>> >> What do you mean by "marked as spam"? mails should either be rejected or >> received in the end never "marked as spam" (at least by SQLgrey). >> >> > Sorry. I meant rejected. > >> If you need to add entries to client_fqdn_addr.local in order to receive >> mail from them, I'd like to know them. >> >> >> > Here's the current list: > http://www.arava.co.il/~micha/clients_fqdn_whitelist.local > They're mostly domains hosted here in Israel. > I've seen many entries that should work without whitelisting them. microsoft.com, yahoo.com, wanadoo.fr, gmail.com at least work on my servers and they are others so big that I doubt other SQLgrey users never had mail from them. Did you setup SQLgrey with a low max_connect_age, the default is 24 (24 hours)? It could explain so many false positives. Lionel |
|
From: Micha S. <mi...@ar...> - 2007-01-20 09:28:01
|
Lionel Bouton wrote: > I've seen many entries that should work without whitelisting them. > microsoft.com, yahoo.com, wanadoo.fr, gmail.com at least work on my > servers and they are others so big that I doubt other SQLgrey users > never had mail from them. > Did you setup SQLgrey with a low max_connect_age, the default is 24 (24 > hours)? It could explain so many false positives. > > Lionel > > I set: reconnect_delay = 2 max_connect_age = 24 Do you recommend something different? Another question I've been trying to work out: Our ISP serves as a backup MX for our mail domains. How should I handle this as far as greylisting? When sqlgrey sends a 450 response to a new message, the backup MX queues it and does it own retry after a short delay. Then the message is blocked as spam since the sender addr and IP are the same - I think. (I looked thru the logs today, but couldn't find a good example, but cases like this did come up) . Thanks, Micha |
|
From: Lionel B. <lio...@bo...> - 2007-01-20 16:12:28
|
Micha Silver wrote the following on 20.01.2007 10:27 : > I set: > reconnect_delay = 2 > max_connect_age = 24 > Do you recommend something different? > No, it looks good to me. > Another question I've been trying to work out: Our ISP serves as a > backup MX for our mail domains. How should I handle this as far as > greylisting? Don't use a backup MX if you can't control its own anti-SPAM settings. With greylisting, this means make them access your greylisting database. In your case, you'll be better of dropping the backup MX (more on this later). > When sqlgrey sends a 450 response to a new message, the > backup MX queues it and does it own retry after a short delay. Then the > message is blocked as spam since the sender addr and IP are the same - I > think. It is blocked by SQLgrey once because the backup MX didn't sent the message originally (unless there is an awl entry for your backup already). Then the mail passes on the second try because the backup should comply with RFCs as a full-fledge MTA. If it is blocked as SPAM, this is for a reason outside SQLgrey's scope (did you forget to whitelist your backup MX for some anti-SPAM measure?). Anyway, just dropping the backup MX should help, its only purpose is to buffer mails when your server is down (which would happen at the origin anyway). This (only if configured properly, which means you can trigger the delivery yourself or your server is constantly monitored by the backup) can help deliver mails waiting for your server to come back faster. You are trading a non-measurable and dubious speedup in case of a crash for more SPAMs in the common case. This even hides the problem to the sender when your server is down for an extended period of time (some server are configured to warn the sender when a mail couldn't reach its destination after a configured period of time). So the clients/partners/... of your mail user may discover the problem by phone too late if you are struggling to put systems back online... I occasionaly use backup MXs, but only if I'm in control of all of them and they use a more robust common backend (ie: several cheap pizzaboxes with good CPUs to handle the computations of clamav and spamassassin, with a fat system for the storage with big redondant disks, redondant power and high quality components). Lionel. |
|
From: Dan F. <da...@ha...> - 2007-01-20 14:06:52
|
Micha Silver wrote: > Where can I see more details on this 1.7.4 version, and the > "discrimination" feature? In 1.7.4+ theres a README.DISCRIMINATION It allows you to write regex's for any of the fields passed by postfix to sqlgrey. Such as HELO string, client ip, client hostname sender and recipient and so forth. If i remember correctly, theres even a bunch of examples. By enabling discrimination, those who do NOT match any expression defined, skip greylisting entirely. ill give you two very small examples, and for example purpose, we assume theres is only 1 expression enabled: client_name =~ ^unknown$ - Hosts without reverse-dns gets greylisted. client_name !~ \.il$ - All hosts that do NOT end on ".il" gets greylisted. We use 14 expressions that we have tweaked over time to catch quite a lot of spammersm while bothering only few costumers. |
|
From: Michael S. <Mic...@lr...> - 2007-01-19 19:53:00
|
On Fri, 19 Jan 2007, Lionel Bouton wrote: > Micha Silver wrote the following on 19.01.2007 19:58 : > > Lionel Bouton wrote: > > > > > >> Micha Silver wrote the following on 19.01.2007 19:25 : > >> > >> > >>> Where can I see more details on this 1.7.4 version, and the > >>> "discrimination" feature? > >>> We've been using greylisting for several months, and while spam has > >>> indeed decreased dramatically, we have to spend time every day or two > >>> adding new entries to the client_fqdn_addr.local file because of false > >>> postivies. I think the problem is mostly mail servers that resend almost > >>> immediately and get marked as spam. > >>> > >>> > >>> > >> What do you mean by "marked as spam"? mails should either be rejected or > >> received in the end never "marked as spam" (at least by SQLgrey). > >> > >> > > Sorry. I meant rejected. > > > >> If you need to add entries to client_fqdn_addr.local in order to receive > >> mail from them, I'd like to know them. > >> > >> > >> > > Here's the current list: > > http://www.arava.co.il/~micha/clients_fqdn_whitelist.local > > They're mostly domains hosted here in Israel. > > > > I've seen many entries that should work without whitelisting them. > microsoft.com, yahoo.com, wanadoo.fr, gmail.com at least work on my > servers and they are others so big that I doubt other SQLgrey users > never had mail from them. > Did you setup SQLgrey with a low max_connect_age, the default is 24 (24 > hours)? It could explain so many false positives. > > Lionel > And the following domains are in my domain_awl, which means they work correctly: bgu.ac.il cc.huji.ac.il inter.net.il mail.biu.ac.il netvision.net.il technion.ac.il zahav.net.il and severeral other have at least an entry in from_awl. And several of the entries are entered more than once: < fcradio.net < flowersdirect.co.il < ganshmuel.com < ganshmuel.com < gmail.com < iroads.co.il < mindspring.com < nana.co.il < taligrapes.co.il Michael Storz -- ====================================================== Leibniz-Rechenzentrum | <mailto:St...@lr...> Boltzmannstr. 1 | Fax: +49 89 35831-9700 85748 Garching / Germany | Tel: +49 89 35831-8840 ====================================================== |
|
From: Micha S. <mi...@ar...> - 2007-01-20 09:44:39
|
Michael Storz wrote: > And the following domains are in my domain_awl, which means they work > correctly: > > bgu.ac.il > cc.huji.ac.il > inter.net.il > mail.biu.ac.il > netvision.net.il > technion.ac.il > zahav.net.il > > and severeral other have at least an entry in from_awl. > > And several of the entries are entered more than once: > > < fcradio.net > < flowersdirect.co.il > < ganshmuel.com > < ganshmuel.com > < gmail.com > < iroads.co.il > < mindspring.com > < nana.co.il > < taligrapes.co.il > > Michael Storz > -- > Michael: Thanks for the insight. I'll double check our local list. And I appreciate your time. SOme of the "...ac.il" domains have multiple mail servers and I think that certain servers behave with greylisting as expected, while others do not. Warm regards, Micha |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X