pyzor-users

Is pyzor active?

[John H. <joh...@pl...>]

Hello,

I have just installed pyzor in conjunction with SpamAssassin. However,
as the list archives seem to indicate, the only (?) pyzor server seems
to no longer be available. The archives seem to indicate no changes to
the software for quite some time.

Does anyone know if pyzor is still active at all? Any contact from the
pyzor maintainer? I suspect that even if a new server was found then
wouldn't the database be somewhat out of date if nothing new had been
added to it for some time?


Thanks,

John.

-- 
---------------------------------------------------------------
John Horne, University of Plymouth, UK  Tel: +44 (0)1752 233914
E-mail: Joh...@pl...       Fax: +44 (0)1752 233839

Re: Is pyzor active?

[Geert N. <gee...@am...>]

John Horne wrote:
> Hello,
> 
> I have just installed pyzor in conjunction with SpamAssassin. However,
> as the list archives seem to indicate, the only (?) pyzor server seems
> to no longer be available. The archives seem to indicate no changes to
> the software for quite some time.
>

I don't know about the official status of the project. However, the
debian pyzor package seems to have some patches in it to resolve several
problems.

When doing a 'pyzor ping', I get a reply from the pyzor server returned
when doing 'pyzor discover'.
# pyzor ping
66.250.40.33:24441      (200, 'OK')

> Does anyone know if pyzor is still active at all? Any contact from the
> pyzor maintainer? I suspect that even if a new server was found then
> wouldn't the database be somewhat out of date if nothing new had been
> added to it for some time?
> 

Today, pyzor hit on 880 out of 2843 mails for me so far. A while ago,
Milton Cyrus setup a new server located at 82.94.255.100:24441. I'm not
using that server and don't know how it performs.

Regards,

Geert

Re: Is pyzor active?

[John H. <joh...@pl...>]

On Mon, 2006-06-26 at 13:48 +0200, Geert Nijpels wrote:
> John Horne wrote:
> > Hello,
> > 
> > I have just installed pyzor in conjunction with SpamAssassin. However,
> > as the list archives seem to indicate, the only (?) pyzor server seems
> > to no longer be available. The archives seem to indicate no changes to
> > the software for quite some time.
> >
> 
> I don't know about the official status of the project. However, the
> debian pyzor package seems to have some patches in it to resolve several
> problems.
> 
Interesting. Do you know if these patches have been submitted for
inclusion in the official pyzor (i.e. on sourceforge)?

> When doing a 'pyzor ping', I get a reply from the pyzor server returned
> when doing 'pyzor discover'.
> # pyzor ping
> 66.250.40.33:24441      (200, 'OK')
> 
Oh rats. It must be a firewall issue then (I get timeouts).


Thanks,

John.

-- 
---------------------------------------------------------------
John Horne, University of Plymouth, UK  Tel: +44 (0)1752 233914
E-mail: Joh...@pl...       Fax: +44 (0)1752 233839

Re: Is pyzor active?

[Geert N. <gee...@am...>]

John Horne wrote:
> On Mon, 2006-06-26 at 13:48 +0200, Geert Nijpels wrote:
>> John Horne wrote:
>>> Hello,
>>>
>>> I have just installed pyzor in conjunction with SpamAssassin. However,
>>> as the list archives seem to indicate, the only (?) pyzor server seems
>>> to no longer be available. The archives seem to indicate no changes to
>>> the software for quite some time.
>>>
>> I don't know about the official status of the project. However, the
>> debian pyzor package seems to have some patches in it to resolve several
>> problems.
>>
> Interesting. Do you know if these patches have been submitted for
> inclusion in the official pyzor (i.e. on sourceforge)?

I'm not sure about that. I think most of the patches are already in the
SF.net patch/bug tracker.

> 
>> When doing a 'pyzor ping', I get a reply from the pyzor server returned
>> when doing 'pyzor discover'.
>> # pyzor ping
>> 66.250.40.33:24441      (200, 'OK')
>>
> Oh rats. It must be a firewall issue then (I get timeouts).
> 

I tried some more pings and out of 8 tries, I get 4 timeouts.

Geert

Re: Is pyzor active?

[Jake V. <ja...@v2...>]

John Horne wrote:
> On Mon, 2006-06-26 at 13:48 +0200, Geert Nijpels wrote:
>   
> Interesting. Do you know if these patches have been submitted for
> inclusion in the official pyzor (i.e. on sourceforge)?
>
>   
>> When doing a 'pyzor ping', I get a reply from the pyzor server returned
>> when doing 'pyzor discover'.
>> # pyzor ping
>> 66.250.40.33:24441      (200, 'OK')
>>
>>     
> Oh rats. It must be a firewall issue then (I get timeouts).
>
>
>   
I was given the impression that the project was stagnant as well. I also 
receive errors, which I assume is because the server (as I was told) is 
no longer online.
Can anyone confirm or deny?

Re: Is pyzor active?

[John H. <joh...@pl...>]

On Mon, 2006-06-26 at 10:02 -0400, Jake Vickers wrote:
> John Horne wrote: 
> > On Mon, 2006-06-26 at 13:48 +0200, Geert Nijpels wrote:
> >   
> > Interesting. Do you know if these patches have been submitted for
> > inclusion in the official pyzor (i.e. on sourceforge)?
> > 
> >   
> > > When doing a 'pyzor ping', I get a reply from the pyzor server returned
> > > when doing 'pyzor discover'.
> > > # pyzor ping
> > > 66.250.40.33:24441      (200, 'OK')
> > > 
> > >     
> > Oh rats. It must be a firewall issue then (I get timeouts).
> > 
> > 
> >   
> I was given the impression that the project was stagnant as well. I
> also receive errors, which I assume is because the server (as I was
> told) is no longer online.
> Can anyone confirm or deny?
>
If 'pyzor ping' works for some users then I assume that the server is
still active. It certainly replies to ICMP pings, so it is still on the
'net at least.



John.

-- 
---------------------------------------------------------------
John Horne, University of Plymouth, UK  Tel: +44 (0)1752 233914
E-mail: Joh...@pl...       Fax: +44 (0)1752 233839

Re: Is pyzor active?

[Graham M. <gr...@gm...>]

John Horne <joh...@pl...> writes:

> If 'pyzor ping' works for some users then I assume that the server is
> still active. It certainly replies to ICMP pings, so it is still on the
> 'net at least.

For me, pyzor is very intermittent. Using SpamAssassin most emails
show a pyzor timeout/failure but some show a score.

Re: Is pyzor active?

[<pyz...@nr...>]

Ok for the last 10 people who asked, and the next 10 people who will ask,
pyzor is not dead. It may simply appear dead because it has worked ok for=
 a
long time and there have been minimal changes.

This is the server that I am using: 66.250.40.33:24441

So far this month, 4500 pyzor filtered spams, and 8849 spamassassin =
filtered
spams. It used to be 50/50 but I have improved my spamassassin filters to
catch more lower scored spams.

We definitely should get those patches onto sourceforge so that new =
people
can start using pyzor. I expect many would give up on it before finding =
the
patches.

My pyzor reporting thru spamassassin has been broken for a while and my
recent tests show pyzor timing out quite often. I should get on that.

Re: Is pyzor active?

[John H. <joh...@pl...>]

On Mon, 2006-06-26 at 11:47 -0400, pyz...@nr... wrote:
> Ok for the last 10 people who asked, and the next 10 people who will ask,
> pyzor is not dead. It may simply appear dead because it has worked ok for a
> long time and there have been minimal changes.
> 
Er, are you Frank Tobin, the pyzor maintainer? Your message has no
signature, and email address no real name with it, but you seem to be
stating that pyzor is not dead as a fact.

As one of the 'new people' (see below) it certainly does appear dead,
and I don't think it has been working ok for a while. My first check
when things weren't working, was the mailing lists and bug list.

Users on the mailing list appear to have been having problems with the
server back to July/August last year. So that's almost a year with a
server problem. (Why is there only one server, it would obviously cause
problems if the server failed. I gather from the list that another
server has been offered, although that one too seems to timeout now.)

The software itself is dated from 2002, so no changes since then? There
are bugs listed dating from 2005 back to 2002. There are two patches
listed dating from 2002, and 3 feature requests dating from 2004. Hence,
the 'appearance' is that nothing is happening with the software - no
bugs fixed, no patches applied, and no new release in over 3 years.

>
> We definitely should get those patches onto sourceforge so that new people
> can start using pyzor. I expect many would give up on it before finding the
> patches.
> 
What patches? The Debian ones? Do you already know what these patches
do? If a patch is at least available on SF then we, as users, can apply
it if we wish and hence test it.

> My pyzor reporting thru spamassassin has been broken for a while and my
> recent tests show pyzor timing out quite often. I should get on that.
> 
"Broken for a while" - so it hasn't been working ok then? If the server
times out then we have no chance of seeing if the software works (or
rather if it works well). Definitely needs sorting out.

Don't misunderstand me, I appreciate that these things take a lot of
time to maintain. But the 'impression' is that it isn't being
maintained. Unfortunately I'm more a perl-person, only having looked at
python for a short while, otherwise I may have been able to help out
with the code. However, the server issue can only be resolved by those
in charge of the server.



John.

-- 
---------------------------------------------------------------
John Horne, University of Plymouth, UK  Tel: +44 (0)1752 233914
E-mail: Joh...@pl...       Fax: +44 (0)1752 233839

Re: Is pyzor active?

[<pyz...@nr...>]

Hi. No, I'm not Frank. Didn't mean to give that impression at all. It
appears that Frank last posted to this list in september. I just don't
expect Frank to reply right away with "its not dead" every time someone
asks.

It's a fact that pyzor isn't dead because I have been using it for three
years with great results, using Frank's server (or whatever server Frank
kindly arranged for us).=20

There's simply no code to manage distributed servers. Someone would have =
to
write it. I suppose that if Frank thought it was a big problem he might =
have
done it already. I can only speculate based on his posts to the mailing
list.

People reporting server problems doesn't necessarily mean there is a =
server
problem. I see those posts pop up now and then, but I still see plenty of
spam going into my pyzor folder. I'm not impressed when people report =
server
problems and don't know or even try to do a traceroute, use other =
computers,
or internet connections. I don't have handy stats but it seems that I =
have
less than 2% timeouts or other errors. Probably pretty good for UDP over =
the
net.

As far as Milton's server goes I've never needed to use it. Has he posted
anything to indicated that it is overloaded by clients?
(Milton Cyrus <mi...@i-...> 82.94.255.100:24441)

That's right there hasn't really been any changes in the software since
2002, IMO because it is simple and it works. There are a few patches that
you can get all rolled into one:

>To: us...@sp...
>Subject: Re: Pyzor timing out
>From: Chris <cpo...@ea...>
>Date: Sat, 10 Dec 2005 21:35:44 -0600
...
>To make it simple, I have combined those 3 patches into 1...
>http://www.engelken.net/download/pyzor.patch
...

I don't really know the origin of these patches and I'm not aware of any
others. Did anyone even bother to send these patches to Frank to to be
included in the sourceforge release? I have no idea.

My pyzor reporting problem is not pyzor's fault. Manual reporting works =
ok.
I'm running a cvs spamassassin and who knows what horrible things they =
have
done to it. It could be that my spamassassin configuration is way out of
date. I haven't bothered to dig into it yet but I will. Obviously it =
works
for a lot of other people.

My recent manual attempts to check pyzor spams seemed to time out but I
checked my spamassassin logs and pyzor doesn't seem to be timing out that
often at all. Maybe it retries.

Anywhoo, the major point was that its not dead, it is working, and it's =
just
really low maintenance. Until we see a deluge of posts about server =
timeouts
I don't expect to hear from Frank about it. Even then all it might take =
is
someone to step up and provide another server, or hack the client code to
failover to multiple servers.

-Sean


On Mon, 26 Jun 2006 23:53:00 +0100, you John Horne wrote:

>On Mon, 2006-06-26 at 11:47 -0400, pyz...@nr... wrote:
>> Ok for the last 10 people who asked, and the next 10 people who will =
ask,
>> pyzor is not dead. It may simply appear dead because it has worked ok =
for a
>> long time and there have been minimal changes.
>>=20
>Er, are you Frank Tobin, the pyzor maintainer? Your message has no
>signature, and email address no real name with it, but you seem to be
>stating that pyzor is not dead as a fact.
>
>As one of the 'new people' (see below) it certainly does appear dead,
>and I don't think it has been working ok for a while. My first check
>when things weren't working, was the mailing lists and bug list.
>
>Users on the mailing list appear to have been having problems with the
>server back to July/August last year. So that's almost a year with a
>server problem. (Why is there only one server, it would obviously cause
>problems if the server failed. I gather from the list that another
>server has been offered, although that one too seems to timeout now.)
>
>The software itself is dated from 2002, so no changes since then? There
>are bugs listed dating from 2005 back to 2002. There are two patches
>listed dating from 2002, and 3 feature requests dating from 2004. Hence,
>the 'appearance' is that nothing is happening with the software - no
>bugs fixed, no patches applied, and no new release in over 3 years.
>
>>
>> We definitely should get those patches onto sourceforge so that new =
people
>> can start using pyzor. I expect many would give up on it before =
finding the
>> patches.
>>=20
>What patches? The Debian ones? Do you already know what these patches
>do? If a patch is at least available on SF then we, as users, can apply
>it if we wish and hence test it.
>
>> My pyzor reporting thru spamassassin has been broken for a while and =
my
>> recent tests show pyzor timing out quite often. I should get on that.
>>=20
>"Broken for a while" - so it hasn't been working ok then? If the server
>times out then we have no chance of seeing if the software works (or
>rather if it works well). Definitely needs sorting out.
>
>Don't misunderstand me, I appreciate that these things take a lot of
>time to maintain. But the 'impression' is that it isn't being
>maintained. Unfortunately I'm more a perl-person, only having looked at
>python for a short while, otherwise I may have been able to help out
>with the code. However, the server issue can only be resolved by those
>in charge of the server.
>
>
>
>John.

Re: Is pyzor active?

[John H. <joh...@pl...>]

On Tue, 2006-06-27 at 02:17 -0400, pyz...@nr... wrote:
>
> That's right there hasn't really been any changes in the software since
> 2002, IMO because it is simple and it works. There are a few patches that
> you can get all rolled into one:
> 
> >To: us...@sp...
> >Subject: Re: Pyzor timing out
> >From: Chris <cpo...@ea...>
> >Date: Sat, 10 Dec 2005 21:35:44 -0600
> ...
> >To make it simple, I have combined those 3 patches into 1...
> >http://www.engelken.net/download/pyzor.patch
> ...
Hello,

It seems that the Debian maintainer has been working on pyzor to fix
some reported problems. The latest patch (i.e. their version of pyzor),
taken from the testing distribution, was released this year. In total
are 7 patches I think (I run Fedora Core myself so I've simply been
looking at the diff file Debian released). Details can be found at 
http://packages.debian.org/testing/mail/pyzor

Debian also has some man pages for pyzor/pyzord, which might be nice to
release generally. What I should do is compare the SF source code
against the released FC5 version, and then see how that compares to the
Debian version. From that it should be possible to extract the patches
which could be put up on to SF for the maintainer to check and others to
test if they want.



John.

-- 
---------------------------------------------------------------
John Horne, University of Plymouth, UK  Tel: +44 (0)1752 233914
E-mail: Joh...@pl...       Fax: +44 (0)1752 233839

Re: Is pyzor active?

[John H. <joh...@pl...>]

On Tue, 2006-06-27 at 02:17 -0400, pyz...@nr... wrote:
> Hi. No, I'm not Frank. Didn't mean to give that impression at all.
>
Okay, thanks for that. I really couldn't tell so I thought it best to
ask :-)

> It appears that Frank last posted to this list in september. I just don't
> expect Frank to reply right away with "its not dead" every time someone
> asks.
> 
Ah, but people will only ask if it is dead because it 'appears' to be
dead. I asked because of our timeout problem, and checking the list
archives, the bugs etc, it 'appears' that nothing has really happened
for some time despite others having problems as well (as reported on the
list). The overall impression is of it not being worked on - i.e. dead.

> There's simply no code to manage distributed servers. Someone would have to
> write it. I suppose that if Frank thought it was a big problem he might have
> done it already. I can only speculate based on his posts to the mailing
> list.
> 
But there is a feature request on SF for multiple servers dating from
2003/2004, so this idea has been around for 2 years. Using one server to
support all of the pyzor requests is surely risky. What if the server
fails for good or the owners decide to withdraw their support? Pyzor
then would be dead. Using multiple servers surely must be a good thing.

(In terms of coding it, it seems that 'pyzor ping' already supports
multiple servers, so what is required is for each server to synchronise
the local DB from the main server. I wouldn't have thought that actually
requires any coding, but just using something like rsync or whatever run
regularly via cron. How big is the DB though? Catering for DB updates
would require that these are sent to the main server. Perhaps the
'servers' file could simply list each server as either master or slave.
DB updates then go to the listed 'master'. Rotating the slave server
used each time would be nice too.)

> People reporting server problems doesn't necessarily mean there is a server
> problem.
>
Agreed, especially when UDP is used.

> I see those posts pop up now and then, but I still see plenty of
> spam going into my pyzor folder. I'm not impressed when people report server
> problems and don't know or even try to do a traceroute, use other computers,
> or internet connections. I don't have handy stats but it seems that I have
> less than 2% timeouts or other errors. Probably pretty good for UDP over the
> net.
> 
Again I tend to agree. In our case, continuous timeouts, I suspect is a
local firewall issue and was my first assumption. However, then going on
to check the mailing list archive to see if others have the same problem
(and what port to open up), I see they do. Checking the patches to see
if there may have already have been some code fix (if it was the code),
shows that there have been no patches for a very long time. Overall an
impression of not much happening, hence why I initially asked if the
project was in fact dead.

> That's right there hasn't really been any changes in the software since
> 2002, IMO because it is simple and it works.
>
Yes, but relying on just one server is risky. Multiple servers would
help.

>  There are a few patches that you can get all rolled into one:
> 
Ah, thanks for that I'll take a look at them.

> 
> My recent manual attempts to check pyzor spams seemed to time out but I
> checked my spamassassin logs and pyzor doesn't seem to be timing out that
> often at all. Maybe it retries.
> 
No it doesn't. Tcpdump shows that just one UDP packet is sent out to
each server.

> Anywhoo, the major point was that its not dead, it is working, and it's just
> really low maintenance. Until we see a deluge of posts about server timeouts
> I don't expect to hear from Frank about it.
>
I don't expect a deluge. How would someone know if the pyzor server was
timing out or not? Their spam would probably be filtered by other
methods (SARE rules, DCC etc). Only if the sysadmin actually looked, or
monitored, the log file would they see that no spam mail is being
flagged as hitting a pyzor check (in SA). They would then need to
investigate further if there was a problem with pyzor. For these reasons
I suspect others may be experiencing a timeout problem but perhaps don't
know it.

>  Even then all it might take is someone to step up and provide another
> server, or hack the client code to failover to multiple servers.
> 
True, but why not do it now before using a single server does become a
problem, and pyzor stops for everyone?


John.

-- 
---------------------------------------------------------------
John Horne, University of Plymouth, UK  Tel: +44 (0)1752 233914
E-mail: Joh...@pl...       Fax: +44 (0)1752 233839

Re: Is pyzor active?

[Jake V. <ja...@v2...>]

John Horne wrote:
>   
> Ah, but people will only ask if it is dead because it 'appears' to be
> dead. I asked because of our timeout problem, and checking the list
> archives, the bugs etc, it 'appears' that nothing has really happened
> for some time despite others having problems as well (as reported on the
> list). The overall impression is of it not being worked on - i.e. dead.
>
>   
True. I've searched for a few days trying to find specific answers (and 
see if there was life in the project).
>> There's simply no code to manage distributed servers. Someone would have to
>> write it. I suppose that if Frank thought it was a big problem he might have
>> done it already. I can only speculate based on his posts to the mailing
>> list.
>>
>>     
> Yes, but relying on just one server is risky. Multiple servers would
> help.
>
>   
Maybe distribute the incoming requests, with something akin to NFS? 
Multiple "faces" could answer requests then, and the next bottle neck to 
look at would be the NFS machine. I for one would be willing to donate 
some bandwidth to spreading this out a little and making the project 
better overall. Just my 2-cents.

Re: Is pyzor active?

[John E H. <jh...@ti...>]

John Horne wrote at 13:05 +0100 on Jun 27, 2006:
 > I don't expect a deluge. How would someone know if the pyzor server was
 > timing out or not? Their spam would probably be filtered by other
 > methods (SARE rules, DCC etc). Only if the sysadmin actually looked, or
 > monitored, the log file would they see that no spam mail is being
 > flagged as hitting a pyzor check (in SA). They would then need to
 > investigate further if there was a problem with pyzor. For these reasons
 > I suspect others may be experiencing a timeout problem but perhaps don't
 > know it.

Yes, you _can_ look in the logs, but you don't have to.
I put this in my ~/.spamassassin/user_prefs

add_header all Pyzor _PYZOR_

This puts the X-Spam-Pyzor header in each mail.

You see headers like this:

X-Spam-Pyzor: 

X-Spam-Pyzor: Reported 0 times.

X-Spam-Pyzor: Reported 26 times.

The first one happens when pyzor times out (there are lots of those).
The second one happens when pyzor responds that this mail has not
been reported to pyzor.
The third one... well you get the picture.
There is a threshold (5 reports by default) where SA fires the
PYZOR_CHECK rule and awards spam points to the email.

FWIW, 

pyzor ping shows that both servers I know about are alive.

66.250.40.33:24441      (200, 'OK')
82.94.255.100:24441     (200, 'OK')

But responsiveness is typically spotty...

sh -c 'i=10;while [ $i -gt 0 ]; do pyzor ping; i=$(expr $i - 1); done'
66.250.40.33:24441      TimeoutError: 
66.250.40.33:24441      TimeoutError: 
66.250.40.33:24441      (200, 'OK')
66.250.40.33:24441      TimeoutError: 
66.250.40.33:24441      TimeoutError: 
66.250.40.33:24441      (200, 'OK')
66.250.40.33:24441      TimeoutError: 
66.250.40.33:24441      TimeoutError: 
66.250.40.33:24441      (200, 'OK')
66.250.40.33:24441      TimeoutError: 

My spambox shows that quite a few spams (say 30-40%) are getting
awarded the PYZOR_CHECK score.  If there was work done to make pyzor
less prone to timeouts, we'd probably see more.

It'd be nice if it was better, but I still consider a tool worth
including in my spam checks.

In addition to the timeout issue, another thing that is missing is a
way to revoke false positives.


Hmmm... actually, it looks like only SA 2.63 is catching 30-40%.
Another test using SA 3.1.1 showed far fewer hits (< 1%).

Perhaps the older SA code used to do retries or some other change in
the SA code is affecting this.

This might be a coincidence in my sample set or an OS difference (two
different OS versions used for the two tests), but at first glance it
looks like a change in SA is the difference.  It might be a question
for the SA lists or a quick look at the source.

Okay, after looking through the SA source... 3.1.1 uses a 5 seconds
timeout by default.  2.63 uses 10.  I'm going to try bumping up the 5
second timeout to see how it affects my hit percentage.  Other than
the timeout, the pyzor_lookup code looks similar on the surface.

Re: Is pyzor active?

[John H. <joh...@pl...>]

On Tue, 2006-06-27 at 09:43 -0600, John E Hein wrote:
>
> Yes, you _can_ look in the logs, but you don't have to.
> I put this in my ~/.spamassassin/user_prefs
> 
> add_header all Pyzor _PYZOR_
> 
> This puts the X-Spam-Pyzor header in each mail.
> 
> You see headers like this:
> 
> X-Spam-Pyzor: 
> 
> X-Spam-Pyzor: Reported 0 times.
> 
> X-Spam-Pyzor: Reported 26 times.
> 
> The first one happens when pyzor times out (there are lots of those).
> The second one happens when pyzor responds that this mail has not
> been reported to pyzor.
> The third one... well you get the picture.
> There is a threshold (5 reports by default) where SA fires the
> PYZOR_CHECK rule and awards spam points to the email.
> 
Not sure I follow this. So how does that help a sysadmin know if pyzor
is generally working or not? He/she would have to manually look through
a mail message's headers to see what was happening. Not something that I
would do very often.

To generally monitor pyzor it would probably be easier to run in cron
something like 'pyzor ping|egrep -i timeout', and let cron mail that to
the sysadmin. 

> But responsiveness is typically spotty...
>
> sh -c 'i=10;while [ $i -gt 0 ]; do pyzor ping; i=$(expr $i - 1); done'
> 66.250.40.33:24441      TimeoutError: 
> 66.250.40.33:24441      TimeoutError: 
>
Why isn't this showing the second server? For me 'pyzor ping' tries both
servers:

pyzor ping
66.250.40.33:24441      TimeoutError:
82.94.255.100:24441     (200, 'OK')


John.

-- 
---------------------------------------------------------------
John Horne, University of Plymouth, UK  Tel: +44 (0)1752 233914
E-mail: Joh...@pl...       Fax: +44 (0)1752 233839

Re: Is pyzor active?

[John E H. <jh...@ti...>]

John Horne wrote at 17:06 +0100 on Jun 27, 2006:
 > Not sure I follow this. So how does that help a sysadmin know if pyzor
 > is generally working or not? He/she would have to manually look through
 > a mail message's headers to see what was happening. Not something that I
 > would do very often.

You said a sys admin had to be used to determine if pyzor was working.
I supplied a way for joe user to figure it out if using SA.

Since I keep a couple weeks of filed spam around, a quick grep is an
easy way to see how the pyzor server has been responding lately -
easier than logging in to the server and looking at logs (particularly
since our SA debug is not set high enough to log pyzor timeouts).  You
can also just occasionally look at the headers of a random email in
your inbox.


 > To generally monitor pyzor it would probably be easier to run in cron
 > something like 'pyzor ping|egrep -i timeout', and let cron mail that to
 > the sysadmin. 

Sure you could do that... or grep the logs if your debug is set high
enough to catch pyzor timeouts.  Yep, there are a number of ways.  I
don't need to be instantly notified.  If I were the pyzor server
admin, however, I might want to be notified of trouble promptly.


 > > But responsiveness is typically spotty...
 > >
 > > sh -c 'i=10;while [ $i -gt 0 ]; do pyzor ping; i=$(expr $i - 1); done'
 > > 66.250.40.33:24441      TimeoutError: 
 > > 66.250.40.33:24441      TimeoutError: 
 > >
 > Why isn't this showing the second server? For me 'pyzor ping' tries both
 > servers:
 > 
 > pyzor ping
 > 66.250.40.33:24441      TimeoutError:
 > 82.94.255.100:24441     (200, 'OK')

Because I only put the 66.* server in my file.  When I report, I
report to both (with a custom script using pyzor --homedir pointing to
an alternate .pyzor config), but having SA check both is less easy, so
I just use the "official" one.  I think that would require a change to
SA (or, of course, pyzor).  If someone does the work, post it here.

Here is basically what I do (where tmpfile is the spam message):

 tmpd=$(mktemp -d -t altpy)
 cp -p ~/.pyzor/servers.alt $tmpd/servers
 spamassassin -d < $tmpfile | pyzor -d --homedir $tmpd report
 rm -rf $tmpd

Re: Is pyzor active?

[John H. <joh...@pl...>]

On Tue, 2006-06-27 at 10:49 -0600, John E Hein wrote:
>
> You said a sys admin had to be used to determine if pyzor was working.
> I supplied a way for joe user to figure it out if using SA.
> 
Okay, that's fair enough :-)

>
> Because I only put the 66.* server in my file.  When I report, I
> report to both (with a custom script using pyzor --homedir pointing to
> an alternate .pyzor config), but having SA check both is less easy, so
> I just use the "official" one.  I think that would require a change to
> SA (or, of course, pyzor).  If someone does the work, post it here.
>
That shouldn't be required though. If a second official server is
provided, then both will contain the same DB information. SA only then
needs to check one. The current "unofficial" server, whilst indeed
providing a second server that can be used for checks/reports, is in
effect autonomous of the official one. In terms of working with SA, yes
it causes a problem.


John.

-- 
---------------------------------------------------------------
John Horne, University of Plymouth, UK  Tel: +44 (0)1752 233914
E-mail: Joh...@pl...       Fax: +44 (0)1752 233839

Re: Is pyzor active?

[John E H. <jh...@ti...>]

Bill McCormick wrote at 11:02 -0500 on Jun 27, 2006:
 > >Okay, after looking through the SA source... 3.1.1 uses a 5 seconds
 > >timeout by default.  2.63 uses 10.  I'm going to try bumping up the 5
 > >second timeout to see how it affects my hit percentage.  Other than
 > >the timeout, the pyzor_lookup code looks similar on the surface.
 > >  
 > Where/How do you set the time out?

man Mail::SpamAssassin::Plugin::Pyzor

But in retrospect, I don't think this will help.  SA just invokes
'pyzor check' and sets a timeout around that.  Since pyzor has its own
notion of a timeout, waiting longer than pyzor's timeout (plus a
little overhead for forking) isn't going to buy you anything.

It looks like pyzor's timeout is 5 seconds:

sh -xc 'time pyzor -d check < some_email'
+ time pyzor -d check
calculated digest: dac867e642c5abc187f4cdd7cebc86d2d959fda9
sending: 'User: anonymous\nTime: 1151427741\nSig: b0373e9cfd8b57db9f8e74af370038609a0bd92e\n\nOp: check\nOp-Digest: dac867e642c5abc187f4cdd7cebc86d2d959fda9\nThread: 45631\nPV: 2.0\n\n'
66.250.40.33:24441      TimeoutError: 
        5.13 real         0.07 user         0.01 sys


I don't see a way to adjust that in the docs, but it's probably easy
to change in the source code.  From my timing tests, however, I don't
think it will help.  It looks like the success cases always happen in
less than 1 second.

So now I am back to not knowing why SA 3.1.1 seems to hit PYZOR_CHECK
far less than SA 2.63.

Re: Is pyzor active?

[John H. <joh...@pl...>]

On Tue, 2006-06-27 at 11:43 -0600, John E Hein wrote:
> Bill McCormick wrote at 11:02 -0500 on Jun 27, 2006:
>  > >Okay, after looking through the SA source... 3.1.1 uses a 5 seconds
>  > >timeout by default.  2.63 uses 10.  I'm going to try bumping up the 5
>  > >second timeout to see how it affects my hit percentage.  Other than
>  > >the timeout, the pyzor_lookup code looks similar on the surface.
>  > >  
>  > Where/How do you set the time out?
> 
> man Mail::SpamAssassin::Plugin::Pyzor
> 
> But in retrospect, I don't think this will help.  SA just invokes
> 'pyzor check' and sets a timeout around that.  Since pyzor has its own
> notion of a timeout, waiting longer than pyzor's timeout (plus a
> little overhead for forking) isn't going to buy you anything.
> 
> It looks like pyzor's timeout is 5 seconds:
> 
You might want to try creating in your ~/.pyzor directory a 'config'
file. In that file put this:

    [client]
    ServersFile = servers
    Timeout = 5

There is, as far as I can tell, a '[server]' section too which can take
a timeout (I suspect used for reporting rather than checking). However,
depending on your setup, you need to make sure you create the file in
the directory of the user that SA uses to run the 'pyzor check' command.

Note, I haven't checked any of this, other than by using 'pyzor ping'.
It works for that. (I don't have SA set up on this PC to use pyzor.)



John.

-- 
---------------------------------------------------------------
John Horne, University of Plymouth, UK  Tel: +44 (0)1752 233914
E-mail: Joh...@pl...       Fax: +44 (0)1752 233839

Re: Is pyzor active?

[John E H. <jh...@ti...>]

John Horne wrote at 19:13 +0100 on Jun 27, 2006:
 > You might want to try creating in your ~/.pyzor directory a 'config'
 > file. In that file put this:
 > 
 >     [client]
 >     ServersFile = servers
 >     Timeout = 5

5 is hard-coded in client.py (unless one of the patches floating around
changes that).

But again, bumping it up doesn't help.  It seems that if it doesn't
respond in a second or so, it won't, and some other strategy (like a
retry) would be needed.

Re: Is pyzor active?

[John H. <joh...@pl...>]

On Tue, 2006-06-27 at 15:22 -0600, John E Hein wrote:
> John Horne wrote at 19:13 +0100 on Jun 27, 2006:
>  > You might want to try creating in your ~/.pyzor directory a 'config'
>  > file. In that file put this:
>  > 
>  >     [client]
>  >     ServersFile = servers
>  >     Timeout = 5
> 
> 5 is hard-coded in client.py (unless one of the patches floating around
> changes that).
> 
Ah yes. Sorry I had forgotten that I had already applied the patch
bundle previously posted to the list. So, yes making the timeout
configurable was a patch.


John.

-- 
---------------------------------------------------------------
John Horne, University of Plymouth, UK  Tel: +44 (0)1752 233914
E-mail: Joh...@pl...       Fax: +44 (0)1752 233839

Pyzor Usage statistics for 82.94.255.100

[MIlton C. <cy...@i-...>]

Hi,

On Tue, 2006-06-27 at 02:17 -0400, pyz...@nr... wrote:

> As far as Milton's server goes I've never needed to use it. Has he posted
> anything to indicated that it is overloaded by clients?
> (Milton Cyrus <mi...@i-...> 82.94.255.100:24441)

http://www.i-block.com/php/stats.php

shows current and past statistics.. ... "I was wondering where the
system-load came from :-) "


best,
Milton

Re: Pyzor Usage statistics for 82.94.255.100

[MIlton C. <cy...@i-...>]

sorry.. should be ..
http://www.i-block.com/php/stats.php?host=&span=daily&refresh=300

Milt.


On Thu, 2006-07-06 at 14:49 +0200, MIlton Cyrus wrote:
> Hi,
> 
> On Tue, 2006-06-27 at 02:17 -0400, pyz...@nr... wrote:
> 
> > As far as Milton's server goes I've never needed to use it. Has he posted
> > anything to indicated that it is overloaded by clients?
> > (Milton Cyrus <mi...@i-...> 82.94.255.100:24441)
> 
> http://www.i-block.com/php/stats.php
> 
> shows current and past statistics.. ... "I was wondering where the
> system-load came from :-) "
> 
> 
> best,
> Milton
> 
> 
> 
> 
> 
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> pyzor-users mailing list
> pyz...@li...
> https://lists.sourceforge.net/lists/listinfo/pyzor-users