pyzor-users — general discussion of Pyzor and Pyzor-related topics

Re: failure to parse response

[SCHUH K. BFI-B. <k....@bf...>]

Hi Tony,
thank You for the quick reply!
 
> Message: 3
> Date: Thu, 16 Jul 2009 08:40:23 +1200
> From: Tony Meyer <to...@sp...>
> Subject: Re: failure to parse response
> To: pyz...@li...
> Message-ID:
> 	<da3...@ma...>
> Content-Type: text/plain; charset=ISO-8859-1
> 
> > I am running pyzor on a debian-lenny-box with spamassassin and
> exim4.
> 
> What version of Pyzor, and what version of SpamAssassin?
>
I use the lenny-versions
spamassassin     3.2.5-2
pyzor            1:0.4.0+cv

> > Wed Jul 15 13:16:40 2009 [20697] dbg: pyzor: opening pipe:
> /usr/bin/pyzor -d --homedir /etc/mail/spamassassin check <
> /tmp/.spamassassin20697w1Tbnqtmp
> 
> You are running pyzor in debug mode (the '-d' flag).  Is there a
> reason for that?  I expect that SA is unable to handle the debug
> output.

we had some othe trouble - it is in normal mode now. Now I cant see any messages from pyzor in the log. 
When I look at the mail-headers it looks like everything is working now :-))

> 
> > Wed Jul 15 13:16:40 2009 [20697] dbg: pyzor: listed: COUNT=43/5
> WHITELIST=0
> > Wed Jul 15 13:16:40 2009 [20697] dbg: rules: ran eval rule
> PYZOR_CHECK ======> got hit (1)
> 
> However, it appears that SA is able to ignore the debug output and
> still get the response.
> 
> Cheers,
> Tony
> 

Thanks a lot.

Cheers
Karl

Re: public.pyzor.org timeouts ... Re: interested in mirror server

[Benny P. <me...@ju...>]

On Thu, July 16, 2009 11:00, Andreas Schamanek wrote:

> $ grep -ci "reporting to pyzor services" $LOGSPAMJULY
> 2409
> $ grep -ci "^public.pyzor.org:24441.*TimeoutError" $LOGSPAMJULY
> 226
>
> That's a 9 % rate in July.
> Is it just me here in Vienna, Austria?

time to make pyzor client code with cache, or make server to server digest working, and help create more mirrors, acl should be so
whitelist/report still is not public submits, unless its submitted to localhost or own server

timeout can be adjusted in sa pyzor to wait more, or if you have local dns server this would help

i see timeouts also but not that much

-- 
xpoint

Re: public.pyzor.org timeouts ... Re: interested in mirror server

[Tony M. <to...@sp...>]

> No, I have seen it give timeouts up to a day at a time.

Do you mean that the server will time out any request for a whole day?
 Have you seen this recently?  Nothing like this should ever happen,
and we need to address the monitoring that is done to pick it up (I've
certainly not seen anything like that) if it reoccurs.

> I don't really complain, as my mail server supports just me and two other people.

Please do feel free to post a note on the list here if timeouts are
common.  If we don't know about the issues (e.g. if the server
monitoring isn't showing anything, and none of our servers are having
trouble using the public server) then we don't know to fix them.  You
(or anyone else) can email me (to...@sp...) offlist if you
prefer.

Thanks,
Tony

Re: public.pyzor.org timeouts ... Re: interested in mirror server

[Dreas v. D. <dr...@sp...>]

Robert Hajime Lanning wrote:
> No, I have seen it give timeouts up to a day at a time.  I don't really
> complain, as my mail server supports just me and two other people.
>   
Ok we have to look into that and fix it. Please send me a traceroute if 
possible offlist.

Dreas

Re: public.pyzor.org timeouts ... Re: interested in mirror server

[Robert H. L. <la...@la...>]

Andreas Schamanek wrote:
> On Thu, 16 Jul 2009, at 20:43, Tony Meyer wrote:
> 
>> Is there anything about the existing public.pyzor.org that makes it
>> unsuitable for use?  i.e. is there anything that could be improved
>> that would mean that you didn't see the need to have your own mirror?
> 
> My answer is no. Indeed, no matter what I think about it it's no 
> because until today I managed to run my mail servers without a local 
> pyzor server (though I did think about installing one quite often ;) 
> during Pyzor's more troubled times of past years).
> 
> Nevertheless, we do see occasional timeouts which I think a 
> disrupting the whole idea of Pyzor:
> 
> $ grep -ci "reporting to pyzor services" $LOGSPAMJULY          
> 2409
> $ grep -ci "^public.pyzor.org:24441.*TimeoutError" $LOGSPAMJULY
> 226
> 
> That's a 9 % rate in July.
> Is it just me here in Vienna, Austria?
> 

No, I have seen it give timeouts up to a day at a time.  I don't really
complain, as my mail server supports just me and two other people.

-- 
END OF LINE
       --MCP

Re: public.pyzor.org timeouts ... Re: interested in mirror server

[Dreas v. D. <dr...@sp...>]

Andreas Schamanek wrote:
> Nevertheless, we do see occasional timeouts which I think a 
> disrupting the whole idea of Pyzor:
>
> $ grep -ci "reporting to pyzor services" $LOGSPAMJULY          
> 2409
> $ grep -ci "^public.pyzor.org:24441.*TimeoutError" $LOGSPAMJULY
> 226
>
> That's a 9 % rate in July.
> Is it just me here in Vienna, Austria?
>   
That's not good. Can you contact me off list so we can try and figure 
out where these timeouts come from and how we can resolve it?

Thanks :)

Dreas

public.pyzor.org timeouts ... Re: interested in mirror server

[Andreas S. <sch...@fa...>]

On Thu, 16 Jul 2009, at 20:43, Tony Meyer wrote:

> Is there anything about the existing public.pyzor.org that makes it
> unsuitable for use?  i.e. is there anything that could be improved
> that would mean that you didn't see the need to have your own mirror?

My answer is no. Indeed, no matter what I think about it it's no 
because until today I managed to run my mail servers without a local 
pyzor server (though I did think about installing one quite often ;) 
during Pyzor's more troubled times of past years).

Nevertheless, we do see occasional timeouts which I think a 
disrupting the whole idea of Pyzor:

$ grep -ci "reporting to pyzor services" $LOGSPAMJULY          
2409
$ grep -ci "^public.pyzor.org:24441.*TimeoutError" $LOGSPAMJULY
226

That's a 9 % rate in July.
Is it just me here in Vienna, Austria?

-- 
-- Andreas


   Re-Alpine: https://sourceforge.net/projects/re-alpine/
   Reborn Alpine continues UW's Alpine/Pine email client

Re: interested in mirror server

[Andreas S. <sch...@fa...>]

On Thu, 16 Jul 2009, at 09:22, Dreas van Donselaar wrote:

> Luis Daniel Lucio Quiroz wrote:
> > we are interested on being a mirror of public.pyzor.org.
> You are able to run a local Pyzor server with your own data, but the 
> public server data is currently not available for mirroring.

Though nobody keeps you from making your local server public.
And I'd certainly welcome a secondary public pyzor server.
HTH,

-- 
-- Andreas

Re: interested in mirror server

[Tony M. <to...@sp...>]

> After working on fine tunning anti-spam, I've realize that statistics are
> primordial to identify high-fuzzy spam.  Because of that, we are interested on
> being a mirror of public.pyzor.org.

Is there anything about the existing public.pyzor.org that makes it
unsuitable for use?  i.e. is there anything that could be improved
that would mean that you didn't see the need to have your own mirror?

Cheers,
Tony

Re: interested in mirror server

[Dreas v. D. <dr...@sp...>]

Luis Daniel Lucio Quiroz wrote:
> Hi all,
>
> After working on fine tunning anti-spam, I've realize that statistics are 
> primordial to identify high-fuzzy spam.  Because of that, we are interested on 
> being a mirror of public.pyzor.org.
>
> Is here anyone that could guide on this?
>   
Hi!

You are able to run a local Pyzor server with your own data, but the 
public server data is currently not available for mirroring.

Kind regards,

Dreas

interested in mirror server

[Luis D. L. Q. <lui...@gm...>]

Hi all,

After working on fine tunning anti-spam, I've realize that statistics are 
primordial to identify high-fuzzy spam.  Because of that, we are interested on 
being a mirror of public.pyzor.org.

Is here anyone that could guide on this?

kind regards,

LD

Re: failure to parse response

[Tony M. <to...@sp...>]

> I am running pyzor on a debian-lenny-box with spamassassin and exim4.

What version of Pyzor, and what version of SpamAssassin?

> Wed Jul 15 13:16:40 2009 [20697] dbg: pyzor: opening pipe: /usr/bin/pyzor -d --homedir /etc/mail/spamassassin check < /tmp/.spamassassin20697w1Tbnqtmp

You are running pyzor in debug mode (the '-d' flag).  Is there a
reason for that?  I expect that SA is unable to handle the debug
output.

> Wed Jul 15 13:16:40 2009 [20697] dbg: pyzor: listed: COUNT=43/5 WHITELIST=0
> Wed Jul 15 13:16:40 2009 [20697] dbg: rules: ran eval rule PYZOR_CHECK ======> got hit (1)

However, it appears that SA is able to ignore the debug output and
still get the response.

Cheers,
Tony

failure to parse response

[SCHUH K. BFI-B. <k....@bf...>]

Hi, 

I am running pyzor on a debian-lenny-box with spamassassin and exim4. I have set up a user and a group "spam" to run spamassassin and set the rights on the file /etc/mail/spamassassin/servers so thaz pyzor can read it. Looking into the spamd.log I find the following:

Wed Jul 15 13:16:40 2009 [20697] dbg: pyzor: pyzor is available: /usr/bin/pyzor
Wed Jul 15 13:16:40 2009 [20697] dbg: info: entering helper-app run mode
Wed Jul 15 13:16:40 2009 [20697] dbg: pyzor: opening pipe: /usr/bin/pyzor -d --homedir /etc/mail/spamassassin check < /tmp/.spamassassin20697w1Tbnqtmp
Wed Jul 15 13:16:40 2009 [20741] dbg: util: setuid: ruid=1001 euid=1001
Wed Jul 15 13:16:40 2009 [20697] dbg: pyzor: got response: sending: 'User: anonymous\nTime: 1247656600\nSig: b384d1e2cefabd421852d9c1cf579cd0c0e0026f\n\nOp: check\nOp-Digest: 591dfccc427937976fcf2458962d1fb7eefa18f8\nThread: 63688\nPV: 2.0\n\n'\nreceived: 'Thread: 63688\nCount: 43\nWL-Count: 0\nCode: 200\nDiag: OK\nPV: 2.0\n\n'\npublic.pyzor.org:24441 (200, 'OK') 43 0
Wed Jul 15 13:16:40 2009 [20697] dbg: info: leaving helper-app run mode
Wed Jul 15 13:16:40 2009 [20697] dbg: pyzor: c "sending: 'User: anonymous\nTime: 1247656600\nSig: b384d1e2cefabd421852d9c1cf579cd0c0e0026f\n\nOp: check\nOp-Digest: 591dfccc427937976fcf2458962d1fb7eefa18f8\nThread: 63688\nPV: 2.0\n\n'"
Wed Jul 15 13:16:40 2009 [20697] dbg: pyzor: failure to parse response "received: 'Thread: 63688\nCount: 43\nWL-Count: 0\nCode: 200\nDiag: OK\nPV: 2.0\n\n'"
Wed Jul 15 13:16:40 2009 [20697] dbg: pyzor: listed: COUNT=43/5 WHITELIST=0
Wed Jul 15 13:16:40 2009 [20697] dbg: rules: ran eval rule PYZOR_CHECK ======> got hit (1)
Wed Jul 15 13:16:40 2009 [20697] dbg: rules: running meta tests; score so far=18.102
Wed Jul 15 13:16:40 2009 [20697] dbg: check: running tests for priority: 500
Wed Jul 15 13:16:40 2009 [20697] dbg: dns: harvest_dnsbl_queries
Wed Jul 15 13:16:40 2009 [20697] dbg: async: select found 5 responses ready (t.o.=0.0)
Wed Jul 15 13:16:40 2009 [20697] dbg: async: completed in 0.157 s: URI-DNSBL, DNSBL:sbl.spamhaus.org.:6.144.75.218

As I see it, pyzor connects to his server and gets a response, but it cannot understand it correctly. Could this be relatetd to the part 'User: anonymous\nTime: 1247656...' ???

Any help appreciated, tia

Karl


-- 
mit freundlichen Grüßen

Ing. Karl SCHUH
EDV-Leiter, ITSM, QM-Auditor
BFI-Burgenland

Grazerstrasse 86
A-7400 Oberwart
Tel. +43(3352)38980-2125
Mob. +43(664)3806107   0664-80045225
web  www.bfi-burgenland.at

UID: ATU 39163301
ZVR: 02 44 86 285
Vereinsregister-Nr.: 294/90 SiD Bgld., 11/01-530 (BH Oberwart)

Re: gtube.txt (aka sample-spam.txt) with headers isn't recognized

[Tom <mai...@ii...>]

Hi Tony,

First thank you for your detailed explanations.

> We could use the sample message provided by SA
> (http://spamassassin.apache.org/gtube/gtube.txt) as the 'high hit
> count' example - that would somewhat address both concerns (and in
> fact it is the case now, since there have been 151 reports of the
> sample message).  Any other message containing the GTUBE string
> wouldn't (necessarily) have a high hit count, but the example message
> would.  OTOH, maybe that would just be confusing, as here.

Just pinging the service (pyzor ping) doesn't really mean it's well 
configured in spamassassin and amavis and that it will score SPAM.

Maybe because I'm Thomas, I only believe what I see :) So I will open a 
ticket in TRAC to ask this "pong" feature which will allow us 
(pessimists and newbies) to test/see if pyzor is correctly enabled in 
the whole chain.

I agree, pyzor worked well out of the box and it was mainly a Pebkac 
error [1] than a pyzor issue. Now with your explanations I have a far 
better understanding than before. Thanks.

Thomas.

[1] : Problem exists between keyboard and chair

Re: SMTP-Interface for pyzor

[Guido <lis...@gu...>]

Thanks for your reply.

On (09-06-26 00:13), Benny Pedersen wrote:
> On Thu, June 25, 2009 19:12, Guido wrote:
> > As Amavis/SpamAssassin is very resource hungry I would like to reject
> > mails if pyzor returns a number of reports of lets say 20.
> pyzord is imho very fuzzy in there sigs so use it as a mta reject is imho a bit dangerous
> > Of course the mails should be rejected before entering Amavis and SpamAssassin.
> no problem for me in this way
Sorry, but I can't understand what you mean. I mean rejected by Pyzor.
How can this work? As far as I can understand this, your statements are
conflicting? :-)

> >  - Therefore I am wondering if there is any solution to use pyzor
> >    as an  SMTP proxy. Functionality like dccifd[1] provides for DCC[2].
> i would like it to be used as a spamtrapper, learn all as spam if sent to newer existsing recipient
> or extend it with spf in the proxy mode if that would be coded, or with spf checking ?, learn as whitelist if spf pass ?, all else
> learn as spam ?
Currently our Pyzor server learns mails from a specific mailbox. But we
plan to set up a spam trap server. I think it should work by simply
adding an entry to /etc/aliases like this one:
catchall: |"/usr/bin/pyzor report"
Where the server accepts all mails for a spam trap domain.

Sure, a dedicated pyzor MTA may be faster doing this job.

> > Are there any other suggestions how to use pyzor as a before queue
> > filter? On the homepage I could only find how to call pyzor by
> > SpamAssassin or by procmail.
> 
> procmail is fine for when recipient is not virtual user

Procmail does not work for us because the mail server must relay to
other servers. Beside that procmail usually comes after
Amavis/SpamAssassin.

Guido

Re: gtube.txt (aka sample-spam.txt) with headers isn't recognized

[Tony M. <to...@sp...>]

> I have setup an postfix + amavis + spamassassin + pyzor all from the
> debian lenny packages [1]. Everything works fine. But I discovered that
> the sample spam [2] send through the whole system is not scored with
> pyzor (but is with razor). Pyzor returns an exit code of 1.

The SA debugging output you included show that Pyzor was checked:

> /usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: pyzor: got response:
> public.pyzor.org:24441 (200, 'OK') 0 0

This shows that Pyzor was checked, and that the response was 0 hits
and 0 whitelist counts.

> Indeed :
>   # pyzor check < /tmp/gtube.txt
>     public.pyzor.org:24441  (200, 'OK')     151     0
>   # echo $?
>     0
> and
>   # pyzor check /tmp/.spamassassin21828A1Yhoatmp
>     public.pyzor.org:24441  (200, 'OK')     0       0
>   # echo $?
>     1
>
> So what's the differences ?

Your diff shows that it's not additional headers being added - it is
replacements headers.  The key is the last line, which is blank.  The
/tmp/.spamassassin21828A1Yhoatmp message has only the headers that are
shown in the diff, not those in the original GTUBE sample message.
Then the body of the /tmp/.spamassassin21828A1Yhoatmp message is all
of the gtube.txt file (i.e. the headers in that file are part of the
body as well).  That means that the messages are substantially
different, so there are different pyzor digests, and therefore
different responses.

The exit codes simply reflect the results - 0 means "found hits and no
whitelist count" and 1 means "found no hits, or a positive whitelist
count".

If you're asking the larger question about whether GTUBE should always
trigger a pyzor hit, I'm not certain, but I lean towards "no".  Pyzor
is about creating unique hashes for essentially identical messages,
and checking how often those have been seen by others.  My feeling is
that GTUBE checking is therefore not appropriate here (because it's
part of a larger message).  GTUBE isn't meant to be detected by every
anti-spam solution (e.g. DNSBL systems generally provide a 127.0.0.2
checking address for the same purpose), and it is simple to add GTUBE
checking to any system that also uses Pyzor (the dominant system being
SA, which, of course, already does a GTUBE check).

The purpose of GTUBE is to check that the filter is working correctly.
 It does seem reasonable to provide a similar function in pyzor - but
I think this would be best done by providing (on the Pyzor wiki) a
couple of complete emails that can be checked - one that is known to
have a high hit count, one that is known to have a high whitelist
count (and ensure that these results stay constant).  If anyone would
find that useful, then please open a ticket on the issue tracker, and
I'll happily add such functionality (but if no-one needs it, then it's
not really worth doing).

We could use the sample message provided by SA
(http://spamassassin.apache.org/gtube/gtube.txt) as the 'high hit
count' example - that would somewhat address both concerns (and in
fact it is the case now, since there have been 151 reports of the
sample message).  Any other message containing the GTUBE string
wouldn't (necessarily) have a high hit count, but the example message
would.  OTOH, maybe that would just be confusing, as here.

Cheers,
Tony

Re: SMTP-Interface for pyzor

[Benny P. <me...@ju...>]

On Thu, June 25, 2009 19:12, Guido wrote:
> I set up a mail server (postfix) with Amavis/SpamAssassin to make use of
> pyzor. Runs very fine so far.

lets keep it

> As Amavis/SpamAssassin is very resource hungry I would like to reject
> mails if pyzor returns a number of reports of lets say 20.

pyzord is imho very fuzzy in there sigs so use it as a mta reject is imho a bit dangerous

> Of course the mails should be rejected before entering Amavis and SpamAssassin.

no problem for me in this way

>  - Therefore I am wondering if there is any solution to use pyzor
>    as an  SMTP proxy. Functionality like dccifd[1] provides for DCC[2].

i would like it to be used as a spamtrapper, learn all as spam if sent to newer existsing recipient

or extend it with spf in the proxy mode if that would be coded, or with spf checking ?, learn as whitelist if spf pass ?, all else
learn as spam ?

> Are there any other suggestions how to use pyzor as a before queue
> filter? On the homepage I could only find how to call pyzor by
> SpamAssassin or by procmail.

procmail is fine for when recipient is not virtual user

-- 
xpoint

SMTP-Interface for pyzor

[Guido <lis...@gu...>]

Hi there!

I set up a mail server (postfix) with Amavis/SpamAssassin to make use of
pyzor. Runs very fine so far.

As Amavis/SpamAssassin is very resource hungry I would like to reject
mails if pyzor returns a number of reports of lets say 20. Of course the
mails should be rejected before entering Amavis and SpamAssassin. 

 - Therefore I am wondering if there is any solution to use pyzor 
   as an  SMTP proxy. Functionality like dccifd[1] provides for DCC[2].

Are there any other suggestions how to use pyzor as a before queue
filter? On the homepage I could only find how to call pyzor by
SpamAssassin or by procmail.

Thanks a lot

Guido

[1] http://www.dcc-servers.net/dcc/dccifd.html
[2] http://www.dcc-servers.net/dcc/

gtube.txt (aka sample-spam.txt) with headers isn't recognized

[Tom <mai...@ii...>]

Hi,

I have setup an postfix + amavis + spamassassin + pyzor all from the 
debian lenny packages [1]. Everything works fine. But I discovered that 
the sample spam [2] send through the whole system is not scored with 
pyzor (but is with razor). Pyzor returns an exit code of 1.

[..]
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: razor2: results: spam? 1
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: razor2: results: engine 
8, highest cf score: 0
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: razor2: results: engine 
4, highest cf score: 100
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: rules: ran eval rule 
RAZOR2_CHECK ======> got hit (1)
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: rules: ran eval rule 
RAZOR2_CF_RANGE_E4_51_100 ======> got hit (1)
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: rules: ran eval rule 
RAZOR2_CF_RANGE_51_100 ======> got hit (1)
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: pyzor: pyzor is 
available: /usr/bin/pyzor
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: info: entering 
helper-app run mode
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: pyzor: opening pipe: 
/usr/bin/pyzor check < /var/lib/amavis/tmp/.spamassassin21828A1Yhoatmp
/usr/sbin/amavisd-new[21838]: (21828-01) SA dbg: util: setuid: ruid=108 
euid=108
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: pyzor: [21838] 
finished: exit=0x0100
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: pyzor: got response: 
public.pyzor.org:24441 (200, 'OK') 0 0
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: info: leaving 
helper-app run mode
/usr/sbin/amavisd-new[21828]: (21828-01) SA dbg: async: select found 1 
responses ready (t.o.=0.0)
[..]

Indeed :
   # pyzor check < /tmp/gtube.txt
     public.pyzor.org:24441  (200, 'OK')     151     0
   # echo $?
     0
and
   # pyzor check /tmp/.spamassassin21828A1Yhoatmp
     public.pyzor.org:24441  (200, 'OK')     0       0
   # echo $?
     1

So what's the differences ?
The _headers_ added by postfix/amavis [3]. So is it normal or a well 
known error in my configuration or worst a bug ? Can someone put my on 
the right way how to fix that.

Cheers,
Thomas

--
[1] List of packages installed on a Debian 5.0.1 (stable/lenny)
ii  pyzor          1:0.4.0+cvs20030201-8    spam-catcher using a 
collaborative filtering network
ii  amavisd-new    1:2.6.1.dfsg-1           Interface between MTA and 
virus scanner/content filters
ii  spamassassin   3.2.5-2                  Perl-based spam filter using 
text analysis

[2] : http://spamassassin.apache.org/gtube/gtube.txt

[3] : diff /tmp/gtube.txt /tmp/.spamassassin21828A1Yhoatmp
0a1,12
 > Return-Path: <ro...@bl...>
 > X-Envelope-To: <to...@fo...>,
 >  <to...@an...>
 > X-Amavis-PolicyBank:
 > Received: by mailhub.anotherfoo.com (Postfix, from userid 0)
 >       id AF9E43D52F; Thu, 25 Jun 2009 08:53:54 +0200 (CEST)
 > To: to...@an...
 > Subject: hop
 > Message-Id: <200...@ma...>
 > Date: Thu, 25 Jun 2009 08:53:54 +0200 (CEST)
 > From: ro...@bl... (root)
 >

Re: 0.5 Can't handle some multipart emails

[Tony M. <to...@sp...>]

> But oh mime - this bug was around for almost 5 years!

Pyzor wasn't really under any sort of active development from roughly
2003 to late 2008.  Unfortunately, that meant that a lot of things
didn't get fixed that should have been.  In the last six months, I've
been getting up to speed, and will try and sort out these remaining
issues ASAP.

> If it's not going to get resolved in the near future,

My hope is to get a 0.6 release out in a couple of weeks (0.5 was
released nearly a month ago).

> wouldn't it be nice if
> there's at least an error message for novice users like me? :-)

I don't see what would cause the error message - exactly what would
you say is an error here?

> It's kind of confusing to get nothing as output especially when we were
> used to 0.4 behaviors, and it probably might break some old scripts
> depending on pyzor's output.

Are you thinking that pyzor 0.4 would have output something if there
was no digestible content?  That is not the case.  The difference with
your example message was whether there was any digestible content or
not.

> Or how about an informational message in debug mode if there's nothing
> to digest or check?

I don't see what that would tell you that just running "predigest"
wouldn't?  However, please feel free to open a feature request
(http://p.sf.net/pyzor/newticket).

Cheers,
Tony

Pyzor website and bug/feature tracker changes

[Tony M. <to...@sp...>]

SourceForge is in the process of discontinuing their documentation
manager and tracker system, in favour of 'hosted applications' that do
the same task.  As a result, we've had to make some changes to the way
that you report and monitor bugs, patches, and feature requests for
Pyzor.  (We've done this a little in advance of the SourceForge
shut-down, so that we're not rushed into it).

The changes are:

 * Bug reports, patches, and feature requests will be managed via a
new system.  You can report a new bug, or open a new feature request,
by going to http://p.sf.net/pyzor/newticket and you can look at the
existing tickets by going to http://p.sf.net/pyzor/report/ .  You
should find this system as easy to use as the old SourceForge one, if
not easier.

 * Documentation has been moved to a wiki system - the big advantage
here is that anyone (with a SourceForge account) is able to contribute
by editing and creating pages.  You can find the documentation at
http://p.sf.net/pyzor/wiki

 * The http://pyzor.org address will take you to the front page of the
wiki, where you can easily find links to tickets, downloads, and other
documentation.

All of the old content has been migrated to this system, including all
closed bug reports, patches, and feature requests (dates of ticket
changes and submitter information has not been preserved, but can be
corrected once SourceForge officially drops the old system, if
required).  We'll continue to add more information (particularly
documentation) as time allows.

The new system uses the Trac management software, which is fairly
widely used, especially among projects implemented in Python, so
hopefully many of you are already familiar with the system.

We'd like to emphasise that the website, documentation, and tickets
are still hosted by SourceForge (just a different part), and that the
project is still completely open and welcomes contributions from the
community.  In fact, we hope that these changes make it easier to take
part.

Please let us know if you have any difficulties with the new site, or
have any suggestions!

--
Tony Meyer, on behalf of the Pyzor team.

Re: Whitelist count = 10000?

[Patrick Yu <ipa...@gm...>]

On Wed, Jun 3, 2009 at 5:37 AM, Tony Meyer <to...@sp...> wrote:
> This digest (778941d994b5281bf5652cd293a2761421cc109d) is a special
> case.  Ticket #1037314 deals with this case.  Basically, the only
> content that Pyzor finds to use for digesting for some types of
> message is:

Thanks again Tony. That explains it.

-PY

Re: 0.5 Can't handle some multipart emails

[Patrick Yu <ipa...@gm...>]

On Wed, Jun 3, 2009 at 6:22 PM, Patrick Yu <ipa...@gm...> wrote:
> On Wed, Jun 3, 2009 at 6:19 AM, Tony Meyer <to...@sp...> wrote:
>> This change was made to fix ticket #2542509.  In 0.4, if you have a
>
> Thanks for your detailed explanation.
>
> But oh mime - this bug was around for almost 5 years! If it's not

Oops - sorry that I mixed up with the other ticket #1037314 you
mentioned in the other post. But my suggestion should still be valid.

-PY

Re: 0.5 Can't handle some multipart emails

[Patrick Yu <ipa...@gm...>]

On Wed, Jun 3, 2009 at 6:19 AM, Tony Meyer <to...@sp...> wrote:
> This change was made to fix ticket #2542509.  In 0.4, if you have a

Thanks for your detailed explanation.

But oh mime - this bug was around for almost 5 years! If it's not
going to get resolved in the near future, wouldn't it be nice if
there's at least an error message for novice users like me? :-) It's
kind of confusing to get nothing as output especially when we were
used to 0.4 behaviors, and it probably might break some old scripts
depending on pyzor's output.

Or how about an informational message in debug mode if there's nothing
to digest or check?

-PY

Re: 0.5 Can't handle some multipart emails

[Tony M. <to...@sp...>]

> I paste an example email here:
> http://pastebin.com/f18a30da9

With Pyzor 0.4, I get:

~$ pyzor predigest < pyzor_test.txt
<Ahref="src="
~$ pyzor digest < pyzor_test.txt
33e160414f372cd14e2414f8f92045e9e0cdea63
~$ pyzor check < pyzor_test.txt
public.pyzor.org:24441	(200, 'OK')	486	0

Although a digest is produced, clearly this is not a valid result, as
the text being used for the digest is insufficiently unique to
identify a message as spam or ham.  0.5 finds no text to digest
("predigest" returns nothing), so there is no digest, and consequently
no check.  IMO that makes 0.5 more correct than 0.4 here.

Note that your example message is not a multipart message (there is no
Content-Type header in the topmost message).

The change here occurs as a result of the change of order of the
normalisation.  In 0.4, the transformation to remove email addresses
and URLs are done before the transformation to remove long strings.
In 0.5, the long string transformation is done first.  In the example
message, the URL is a long string, and so removed in 0.5.  In 0.4, the
URL transformation removes the URLs from the href and src parameters,
but leaves the "href=" and "src=" text in place.  These aren't long
strings, so the long string transformation doesn't touch them.

This change was made to fix ticket #2542509.  In 0.4, if you have a
message with an extremely long sequence of non-whitespace characters
(e.g. binary data), that will be passed through the email address and
URL transformations.  Doing so effectively hangs the regex engine, and
so pyzor hangs for these messages.  In 0.5, those excessively long
sequences are removed first, so the email address and URL
transformations only have to deal with smaller strings.

In this case, I believe 0.5 is more correct in the choice of
normalised text.  In 0.4, the URLs are removed, and the predigest text
is generic text that could be found anywhere.  In 0.5, all generic
text is removed.  That doesn't leave any text (so the message can't be
digested), but if you look at the message, there isn't any unique text
once you exclude the URLs and content (like the boundary identifiers)
that would typically be found in headers rather than the message body.

You are, of course, welcome to stay at 0.4 (or just change the order
of normalisation in your client code), but you will find that Pyzor
hangs on some messages because of this problem, and you get false
positives because of the generic predigest text.

Cheers,
Tony