pyzor-users — general discussion of Pyzor and Pyzor-related topics

Re: Both public servers are down :(

[Dreas v. D. <dr...@sp...>]

Hi guys,

On 03/03/2011 07:32 PM, Andreas Schamanek wrote:
> At 128.130.51.1 the spam reporting log shows timeouts from around 12
> until 16 o'clock (these entries have no timestamps or its own).
>
> HTH,
We've changed the automatic monitoring. Please do let me know (directly 
or via the list) if you experience problems from certain locations.

Greets,

-- 
Dreas van Donselaar

Re: Both public servers are down :(

[Andreas S. <sch...@fa...>]

On Thu, 3 Mar 2011, at 18:18, John Horne wrote:

> On Thu, 2011-03-03 at 18:26 +0100, Dreas van Donselaar wrote:
> > I'm seeing both servers processing data, they were not restarted. 
> > Are you still experiencing problems here? If so, from which IPs 
> > are you testing?
> We saw the same problem earlier today (from 141.163.66.134) - both 
> servers timing out to pings - but now they seem to be working fine.

Same here. When I wrote the original message I could not reach any of 
the pyzor servers from both my locations (128.130.51.1 & 88.198.22.1). 
Now both answer from both locations.

At 88.198.22.1 I have a crude logging (1 ping/h). It shows timeouts 
from 9:20 UTC until 16:20, it got through, though, once at 12:20.

At 128.130.51.1 the spam reporting log shows timeouts from around 12 
until 16 o'clock (these entries have no timestamps or its own).

HTH,

-- 
-- Andreas

Re: Both public servers are down :(

[John H. <joh...@pl...>]

On Thu, 2011-03-03 at 18:26 +0100, Dreas van Donselaar wrote:
> Hi Andreas,
> 
> I'm seeing both servers processing data, they were not restarted. Are 
> you still experiencing problems here? If so, from which IPs are you testing?
> 
Hello,

We saw the same problem earlier today (from 141.163.66.134) - both
servers timing out to pings - but now they seem to be working fine.



John.

> 
> On 03/03/2011 04:36 PM, Andreas Schamanek wrote:
> > Apparently, since several hours both servers serving public.pyzor.org
> > show Timeouts! Checked from 2 distinct locations.
> >
> > HTH,
> >
> 
> ------------------------------------------------------------------------------
> Free Software Download: Index, Search & Analyze Logs and other IT data in 
> Real-Time with Splunk. Collect, index and harness all the fast moving IT data 
> generated by your applications, servers and devices whether physical, virtual
> or in the cloud. Deliver compliance at lower cost and gain new business 
> insights. http://p.sf.net/sfu/splunk-dev2dev 
> _______________________________________________
> pyzor-users mailing list
> pyz...@li...
> https://lists.sourceforge.net/lists/listinfo/pyzor-users

-- 
John Horne, University of Plymouth, UK
Tel: +44 (0)1752 587287    Fax: +44 (0)1752 587001

Re: Both public servers are down :(

[Dreas v. D. <dr...@sp...>]

Hi Andreas,

I'm seeing both servers processing data, they were not restarted. Are 
you still experiencing problems here? If so, from which IPs are you testing?

Regards,

Dreas

On 03/03/2011 04:36 PM, Andreas Schamanek wrote:
> Apparently, since several hours both servers serving public.pyzor.org
> show Timeouts! Checked from 2 distinct locations.
>
> HTH,
>

Both public servers are down :(

[Andreas S. <sch...@fa...>]

Apparently, since several hours both servers serving public.pyzor.org 
show Timeouts! Checked from 2 distinct locations.

HTH,

-- 
-- Andreas

Re: TimeOuts of 79.99.133.40

[Dreas v. D. <dr...@sp...>]

Hi guys,

That second server has been restarted and should work fine again. I'll 
add some more automatic monitoring to prevent this from happening. Sorry 
for the inconvenience!

Regards,

Dreas

On 02/27/2011 07:17 PM, Tom Armistead wrote:
>
> I've been seeing the same thing for about 3 or 4 weeks.
>
> Tom
> --
> Tom Armistead
> to...@ar...
> 214-233-5167 cell
> Sent from my Android phone
>
> -----Original Message-----
> From: Andreas Schamanek <sch...@fa...>
> To: pyz...@li...
> Sent: Sat, 26 Feb 2011 8:31 AM
> Subject: TimeOuts of 79.99.133.40
>
>
> Hi there,
>
> One of the 2 servers serving public.pyzor.org is apparently not
> answering since several hours, maybe longer:
>
> $ pyzor ping
> 109.72.80.227:24441     (200, 'OK')
> $ pyzor ping
> 109.72.80.227:24441     (200, 'OK')
> $ pyzor ping
> 109.72.80.227:24441     (200, 'OK')
> $ pyzor ping
> 79.99.133.40:24441      TimeoutError:
> $ pyzor ping
> 79.99.133.40:24441      TimeoutError:
> $ pyzor ping
> 79.99.133.40:24441      TimeoutError:
>
> HTH,
>
> -- 
> -- Andreas
>
>
>
> ------------------------------------------------------------------------------
> Free Software Download: Index, Search & Analyze Logs and other IT data in
> Real-Time with Splunk. Collect, index and harness all the fast moving 
> IT data
> generated by your applications, servers and devices whether physical, 
> virtual
> or in the cloud. Deliver compliance at lower cost and gain new business
> insights. http://p.sf.net/sfu/splunk-dev2dev
> _______________________________________________
> pyzor-users mailing list
> pyz...@li...
> https://lists.sourceforge.net/lists/listinfo/pyzor-users
>
>
> ------------------------------------------------------------------------------
> Free Software Download: Index, Search&  Analyze Logs and other IT data in
> Real-Time with Splunk. Collect, index and harness all the fast moving IT data
> generated by your applications, servers and devices whether physical, virtual
> or in the cloud. Deliver compliance at lower cost and gain new business
> insights. http://p.sf.net/sfu/splunk-dev2dev
>
>
> _______________________________________________
> pyzor-users mailing list
> pyz...@li...
> https://lists.sourceforge.net/lists/listinfo/pyzor-users

Re: TimeOuts of 79.99.133.40

[Tom A. <to...@ar...>]

I've been seeing the same thing for about 3 or 4 weeks.

Tom
--
Tom Armistead
to...@ar...
214-233-5167 cell
Sent from my Android phone

-----Original Message-----
From: Andreas Schamanek <sch...@fa...>
To: pyz...@li...
Sent: Sat, 26 Feb 2011 8:31 AM
Subject: TimeOuts of 79.99.133.40


Hi there,

One of the 2 servers serving public.pyzor.org is apparently not 
answering since several hours, maybe longer:

$ pyzor ping
109.72.80.227:24441     (200, 'OK')
$ pyzor ping
109.72.80.227:24441     (200, 'OK')
$ pyzor ping
109.72.80.227:24441     (200, 'OK')
$ pyzor ping
79.99.133.40:24441      TimeoutError:
$ pyzor ping
79.99.133.40:24441      TimeoutError:
$ pyzor ping
79.99.133.40:24441      TimeoutError:

HTH,

-- 
-- Andreas



------------------------------------------------------------------------------
Free Software Download: Index, Search & Analyze Logs and other IT data in 
Real-Time with Splunk. Collect, index and harness all the fast moving IT data 
generated by your applications, servers and devices whether physical, virtual
or in the cloud. Deliver compliance at lower cost and gain new business 
insights. http://p.sf.net/sfu/splunk-dev2dev 
_______________________________________________
pyzor-users mailing list
pyz...@li...
https://lists.sourceforge.net/lists/listinfo/pyzor-users

TimeOuts of 79.99.133.40

[Andreas S. <sch...@fa...>]

Hi there,

One of the 2 servers serving public.pyzor.org is apparently not 
answering since several hours, maybe longer:

$ pyzor ping
109.72.80.227:24441     (200, 'OK')
$ pyzor ping
109.72.80.227:24441     (200, 'OK')
$ pyzor ping
109.72.80.227:24441     (200, 'OK')
$ pyzor ping
79.99.133.40:24441      TimeoutError:
$ pyzor ping
79.99.133.40:24441      TimeoutError:
$ pyzor ping
79.99.133.40:24441      TimeoutError:

HTH,

-- 
-- Andreas

Re: Pyzor server 79.99.133.40 not responding

[Andreas S. <sch...@fa...>]

On Thu, 23 Dec 2010, at 18:22, Dreas van Donselaar wrote:

> There was a (software) crash of the Pyzor server. It's back online. 
> We haven't found the source of the issue yet unfortunately, but are 
> investigating it further.

Well, there is always soft errors, and there is 
worse [ http://www.cs.toronto.edu/~bianca/#news ] ;-)

Thanks for fixing it!

-- 
-- Andreas

Re: Pyzor server 79.99.133.40 not responding

[Dreas v. D. <dr...@sp...>]

Hi,

There was a (software) crash of the Pyzor server. It's back online. We 
haven't found the source of the issue yet unfortunately, but are 
investigating it further. Thanks!

Kind regards,

-- 
Dreas van Donselaar
CTO SpamExperts B.V.

Email: dr...@sp...
Phone: +31 (0)20 8200 004
Fax:   +31 (0)43 7113 801
Web:   www.spamexperts.com

SpamExperts B.V.
Nieuwezijds Voorburgwal 340
1012 RX Amsterdam
The Netherlands



On 12/23/2010 04:00 PM, Andreas Schamanek wrote:
> Hi fellow pyzors,
>
> Apparently, public.pyzor.org now resolves to 2 IPs:
>
>    $ host public.pyzor.org
>    public.pyzor.org has address 109.72.80.227
>    public.pyzor.org has address 79.99.133.40
>
> Unfortunately, 79.99.133.40 is not responding:

>    $ echo 109.72.80.227:24441>.pyzor/servers
>    $ pyzor ping
>    109.72.80.227:24441     (200, 'OK')
>    $ echo 79.99.133.40:24441>.pyzor/servers
>    $ pyzor ping
>    79.99.133.40:24441      TimeoutError:
>
> Looks like about half of the pyzor requests are bound to fail, right
> now.
>

Pyzor server 79.99.133.40 not responding

[Andreas S. <sch...@fa...>]

Hi fellow pyzors,

Apparently, public.pyzor.org now resolves to 2 IPs:

  $ host public.pyzor.org
  public.pyzor.org has address 109.72.80.227
  public.pyzor.org has address 79.99.133.40

Unfortunately, 79.99.133.40 is not responding:

  $ echo 109.72.80.227:24441 >.pyzor/servers
  $ pyzor ping
  109.72.80.227:24441     (200, 'OK')
  $ echo 79.99.133.40:24441 >.pyzor/servers
  $ pyzor ping
  79.99.133.40:24441      TimeoutError:

Looks like about half of the pyzor requests are bound to fail, right 
now.

-- 
-- Andreas

Re: pyzor distributed server ?

[Tony M. <to...@sp...>]

> I can read on pyzor debian package description
> <http://packages.debian.org/squeeze/pyzor> that "[pyzor] establishes a
> distributed database of spam digests".
>
> Is that true ? From what I got, pyzor use only one server of your own
> choice and no distributed database mechanism.

I'm not sure whose wording that is, but it's not particularly clear.
Pyzor is distributed in the sense that reports come from any location;
rather than being a centrally controlled definition, it's a
collaborative definition.

You're correct that the database mechanism itself is centralised.
(Although the pyzor client can check as many servers as you like, not
just one).

Cheers,
Tony

pyzor distributed server ?

[Jocelyn D. <joc...@cr...>]

Hi,

I can read on pyzor debian package description 
<http://packages.debian.org/squeeze/pyzor> that "[pyzor] establishes a 
distributed database of spam digests".

Is that true ? From what I got, pyzor use only one server of your own 
choice and no distributed database mechanism.
Unfortunately, I can't find any document explaining the behavior of 
pyzor server, so I can confirm/infirm my doubts.

Hopping someone here will be able to enlighten me,

Thanks by advance,

--
Jocelyn

Re: Timeouts

[Dreas v. D. <dr...@sp...>]

Hi Clay,

There do not appear to be problems currently. Can you provide some 
date/timestamps when the timeouts occur?

Kind regards,

Dreas

On 09/15/2010 09:46 PM, Clayton Keller wrote:
> Over the course of the week I have been seeing debug information
> indicating Pyzor related checks timing out quite frequently.
>
> I have the following configured in my local.cf:
>
> 	# Pyzor Settings
> 	#
> 	use_pyzor       1
> 	pyzor_timeout   5
> 	#
>
> My server file contains the following:
>
> 	public.pyzor.org:24441
>
> And public.pyzor.org resolves to the following:
>
> 	;; ANSWER SECTION:
> 	public.pyzor.org.       124     IN      A       109.72.80.227
> 	public.pyzor.org.       124     IN      A       79.99.133.40
>
> I am not sure if this is isolated, or if there is any known issues or
> possible issues that could be causing the timeouts.
>
> I wanted to post my findings in case others are seeing this as well.
>
> Clay
>
>
> ------------------------------------------------------------------------------
> Start uncovering the many advantages of virtual appliances
> and start using them to simplify application deployment and
> accelerate your shift to cloud computing.
> http://p.sf.net/sfu/novell-sfdev2dev
> _______________________________________________
> pyzor-users mailing list
> pyz...@li...
> https://lists.sourceforge.net/lists/listinfo/pyzor-users
>    

Timeouts

[Clayton K. <ine...@ru...>]

Over the course of the week I have been seeing debug information 
indicating Pyzor related checks timing out quite frequently.

I have the following configured in my local.cf:

	# Pyzor Settings
	#
	use_pyzor       1
	pyzor_timeout   5
	#

My server file contains the following:

	public.pyzor.org:24441

And public.pyzor.org resolves to the following:

	;; ANSWER SECTION:
	public.pyzor.org.       124     IN      A       109.72.80.227
	public.pyzor.org.       124     IN      A       79.99.133.40

I am not sure if this is isolated, or if there is any known issues or 
possible issues that could be causing the timeouts.

I wanted to post my findings in case others are seeing this as well.

Clay

Re: BUG with reproducer: pyzor silently fails to check certain e-mails.

[tlhackque <tlh...@ya...>]

>>There's a minimum amount of data required.  This is by design -
hash-based systems are risky when there is little data to hash,
because the risk of collisions significantly increases.

 
I understand - but nonetheless am surprised that there is no response - even
-d doesn't say "too small, not processed".

>>The best way to combat spam is with a combination of techniques.  For
the messages you are describing, a tool like SURBL would probably be
ideal.

Well, these folks seem to be clever (or lucky).  Despite my best efforts so far, 
a lot of
this is finding its way through. As you can see below, they're slipping by just
under the radar; I had hoped that pyzor would help..

X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on localhost
X-Spam-Level: ****
X-Spam-Status: No, score=4.9 required=5.0 tests=BAYES_99,
    RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,RCVD_IN_SORBS_WEB,T_RP_MATCHES_RCVD
    autolearn=no version=3.3.1
X-Spam-Date-Scanned: Tue, 14 Sep 2010 00:59:55 -0400
X-Spam-Bayes: score=1.0000 summary=Tokens: new, 31; hammy, 1; neutral, 91;
    spammy, 12.
X-Spam-Pyzor: 
Subject: Hot Discounts on Boner Pills

---------------------------------------------------------
This communication may not represent my employer's views,
if any, on the matters discussed.



Message: 4
Date: Mon, 13 Sep 2010 09:04:00 +1200
From: Tony Meyer <to...@sp...>
Subject: Re: BUG with reproducer: pyzor silently fails to check
    certain    e-mails.
To: pyz...@li...
Message-ID:
    <AAN...@ma...>
Content-Type: text/plain; charset=ISO-8859-1

> What seems to be happening is that the message text consists of a single data
> line - a url.

There's a minimum amount of data required.  This is by design -
hash-based systems are risky when there is little data to hash,
because the risk of collisions significantly increases.

> I'm actually seeing quite a bit of spam like this, where the subject is some
> illicit offer, and the text is just a url.

The best way to combat spam is with a combination of techniques.  For
the messages you are describing, a tool like SURBL would probably be
ideal.

Cheers,
Tony


      

Re: whitelist request

[Tony M. <to...@sp...>]

> I appreciate the feedback, would it be appropriate / possible to delist the problem string in the digest I sent?

As I said originally, this digest is not in the database (at least, it
is not in the public.pyzor.org database).  Are you actually having a
problem, or is this just trying to whitelist a message that you are
planning to send?

Cheers,
Tony

Re: whitelist request

[Submit <su...@pa...>]

I appreciate the feedback, would it be appropriate / possible to delist the problem string in the digest I sent?

Sent from an iPhone.

On 14/09/2010, at 3:55 PM, Tony Meyer <to...@sp...> wrote:

>>>> Please consider the following:
> [...]
>>> I don't see any data about this digest in the database.  Is this a
>>> pre-emptive whitelisting?  If so, can I ask why?
>> 
>> I meant to say please delist my domain or mail server from your bad sources list. No spam has ever been sent from it...
> 
> You are misunderstanding how Pyzor works.  Pyzor creates a unique
> string based on the content of email messages.  Users of pyzor share
> information about such strings that were created from spam messages,
> which allows everyone else to recognise when a message is spam.
> 
> This has absolutely nothing to do with the sender of a message.  Only
> the content of the message is important.
> 
> You can read more about Pyzor here: http://pyzor.org
> 
> Cheers,
> Tony

Re: whitelist request

[Tony M. <to...@sp...>]

>>> Please consider the following:
[...]
>> I don't see any data about this digest in the database.  Is this a
>> pre-emptive whitelisting?  If so, can I ask why?
>
> I meant to say please delist my domain or mail server from your bad sources list. No spam has ever been sent from it...

You are misunderstanding how Pyzor works.  Pyzor creates a unique
string based on the content of email messages.  Users of pyzor share
information about such strings that were created from spam messages,
which allows everyone else to recognise when a message is spam.

This has absolutely nothing to do with the sender of a message.  Only
the content of the message is important.

You can read more about Pyzor here: http://pyzor.org

Cheers,
Tony

Re: whitelist request

[Tony M. <to...@sp...>]

Hi,

> Please consider the following:
>
> pyzor digest --mbox < ../pyzor-falsepositive
>
> 523e7558440979cb7e7cdc77af18241f1c665092

I don't see any data about this digest in the database.  Is this a
pre-emptive whitelisting?  If so, can I ask why?

Cheers,
Tony

whitelist request

[PartyVibe.com <su...@pa...>]

Hello,

Please consider the following:

pyzor digest --mbox < ../pyzor-falsepositive

523e7558440979cb7e7cdc77af18241f1c665092

 

Thanks.

Hugh

 

Re: BUG with reproducer: pyzor silently fails to check certain e-mails.

[Tony M. <to...@sp...>]

> What seems to be happening is that the message text consists of a single data
> line - a url.

There's a minimum amount of data required.  This is by design -
hash-based systems are risky when there is little data to hash,
because the risk of collisions significantly increases.

> I'm actually seeing quite a bit of spam like this, where the subject is some
> illicit offer, and the text is just a url.

The best way to combat spam is with a combination of techniques.  For
the messages you are describing, a tool like SURBL would probably be
ideal.

Cheers,
Tony

BUG with reproducer: pyzor silently fails to check certain e-mails.

[tlhackque <tlh...@ya...>]

With certain e-mails, pyzor does not even ask the server for status, instead 
exiting.

What seems to be happening is that the message text consists of a single data 
line - a url.

As I read the code (and I'm not a python person), it looks like the digester 
removes the url, discovers that there is no other text, and gives up.

No message or status is produced, even with -d.  This is bad.  When used with 
Spamassassin, the status line is blank.  


I'm actually seeing quite a bit of spam like this, where the subject is some 
illicit offer, and the text is just a url.

If you run strace on the client, you can see where this happens...

The digester needs to produce something in this case - one thing to do would be 
to use the Subject line as the basis for the hash in this case.   Of course, if 
there's no subject, then I guess the URL.  


But in any case, there needs to be a response, and the digest should be 
something useful.  Pyzor shold be able to register these as spam.

Here are two test cases which I've contrived to demonstrate the behavior.  Note 
that they are trivial to run from the command line.

pyzor 0.5.0-3.fc12 - but it also reproduces on earlier versions.

It would be great if someone who understands the subtleties of how the digests 
are supposed to work would produce a patch.

Failing case:
====
$ pyzor -d check
From: bad@sales
Subject: Elephants

http://example.com
<Ctrl/D>
$ echo $?
1
====
Passing case:
=====
$  pyzor -d check
From: bad@sales
Subject: Elephants

http://example.com is the place to buy them.
<Ctrl/D>sending: 'User: anonymous\nTime: 1284122607\nSig: 
8f8f80896dab02c811fdcc8b9920e40318d67046\n\nOp: check\nOp-Digest: 
e3fb6c7c8102274611bd7edd5c1aa81804226a73\nThread: 33336\nPV: 2.0\n\n'
received: 'Thread: 33336\nCount: 0\nWL-Count: 0\nCode: 200\nDiag: OK\nPV: 
2.0\n\n'
public.pyzor.org:24441  (200, 'OK')     0       0
$
===


Thanks.

 ---------------------------------------------------------
This communication may not represent my employer's views,
if any, on the matters discussed.



      

Re: Whitelist Request

[Tony M. <to...@sp...>]

Hi Clay,

Can you send me (offlist if you like) a copy of the message (ideally)
or the digest?  I can take care of doing that.  You can get the digest
like:

# pyzor digest < ~/test.msg
63ff03ef7e45aac98101c6d0320d80ef508ed1be

Thanks,
Tony

On Thu, Sep 2, 2010 at 4:17 AM, Clayton Keller <ine...@ru...> wrote:
> What is the proper procedure to request a whitelist on a message we see
> hitting as positive?
>
> I am unable to submit any "pyzor whitelist < msg" requests in with the
> sample message because of authorization limitations.
>
> Clay
>
> ------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> _______________________________________________
> pyzor-users mailing list
> pyz...@li...
> https://lists.sourceforge.net/lists/listinfo/pyzor-users
>

Whitelist Request

[Clayton K. <ine...@ru...>]

What is the proper procedure to request a whitelist on a message we see 
hitting as positive?

I am unable to submit any "pyzor whitelist < msg" requests in with the 
sample message because of authorization limitations.

Clay