pyzor-users — general discussion of Pyzor and Pyzor-related topics

Re: Server down?

[Milton <mi...@i-...>]

Hi ,

We had "some" problems on the iblock pyzor server. that caused a service
interrupt on sat/sun. Problems should be solved now . but if you run
into issues pls. feel free to email me..

regards 

Milton


On Sun, 2006-12-17 at 08:13 -0800, Ed Kasky wrote:
> I have not been able to report or ping for two days now.  Anyone else 
> having the same problem?
> 
> I double checked my router config and port 24441 is open for both ip's...
> 
> $ pyzor ping
> 82.94.255.100:24441     TimeoutError:
> 66.250.40.33:24441      TimeoutError:
> 
> Ed
> 
> . . . . . . . . . . . . . . . . . .
> Randomly Generated Quote (345 of 1122):
> A+ B + C = Success if, A = Hard Work, B = Hard Play,
> C = Keeping your mouth shut.  - Albert Einstein
> 
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> pyzor-users mailing list
> pyz...@li...
> https://lists.sourceforge.net/lists/listinfo/pyzor-users

Re: Server down?

[Andreas S. <sch...@fa...>]

Hi Ed,

On Sun, 17 Dec 2006, Ed Kasky wrote:

> $ pyzor ping
> 82.94.255.100:24441     TimeoutError:
> 66.250.40.33:24441      TimeoutError:

Same here @ 88.198.22.135
66.250.40.33:24441      TimeoutError:
82.94.255.100:24441     TimeoutError:

And here @ 128.130.51.99
82.94.255.100:24441     TimeoutError:

HTH,

-- 
-- Andreas

Server down?

[Ed K. <ed...@es...>]

I have not been able to report or ping for two days now.  Anyone else 
having the same problem?

I double checked my router config and port 24441 is open for both ip's...

$ pyzor ping
82.94.255.100:24441     TimeoutError:
66.250.40.33:24441      TimeoutError:

Ed

. . . . . . . . . . . . . . . . . .
Randomly Generated Quote (345 of 1122):
A+ B + C = Success if, A = Hard Work, B = Hard Play,
C = Keeping your mouth shut.  - Albert Einstein

Re: Help me feed pyzor

[Chris <cpo...@ea...>]

On Saturday 02 December 2006 6:59 am, Matija Nalis wrote:

> It is going to be more complicated. not much -- if you really want "perl"
> which imitates a "sh", then you need:
>
> system ('for f in *; do pyzor report < $f; done')
>
> (single quotes are important)
>
> > My Perl-foo is very weak, so I ask you, pyzor hive mind, for help. :D
>
> BTW, why don't you use "spamassassin --report" ? It is intented to do
> both bayesian sa-learn, and report to all services you have enabled
> (pyzor, DCC, spamcop, ....) which seems to be exactly what you want

Maybe this isn't exactly what you're looking for, but on my home box here I=
=20
have a perl script that I run nightly that does it all, runs sa-learn,=20
reports to razor, pyzor and DCC. It will even report to spamcop if the=20
variables are set. I've been using it for years. It may or may not be what=
=20
you're looking for. I didn't write it but I'd be glad to share it if you=20
want to give it a try.

=2D-=20
Chris

Re: Help me feed pyzor

[Matija N. <mna...@vo...>]

On Tue, Nov 28, 2006 at 10:10:29PM -0800, Jonathan Nichols wrote:
> sometimes there are quite a few emails in the "cur" folder. Can I do 
> something like "pyzor report < *" or is it going to be more complicated 
> than that?

It is going to be more complicated. not much -- if you really want "perl"
which imitates a "sh", then you need:

system ('for f in *; do pyzor report < $f; done')

(single quotes are important)

> My Perl-foo is very weak, so I ask you, pyzor hive mind, for help. :D

BTW, why don't you use "spamassassin --report" ? It is intented to do
both bayesian sa-learn, and report to all services you have enabled (pyzor,
DCC, spamcop, ....) which seems to be exactly what you want

-- 
Opinions above are GNU-copylefted.

Re: Pyzor server(s) ...

[Jesus C. <jc...@ar...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Praetorius wrote:
>      I'm not a python programmer, either, but

Is the pyzor programmer still working on it, or is this project
"abandoned"?. In the second case the first step would be to regenerate
the project with new blood.

I remember, a few years ago, trying to integrate pyzor in my antispam
service (written in python), but the code was not easily reusable as a
service "object". You need to call an external process. I did a few
suggestions with no luck.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jc...@ar... http://www.argo.es/~jcea/ _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jc...@ja...         _/_/    _/_/          _/_/_/_/_/
                               _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBRXBVl5lgi5GaxT1NAQJ8EgP/Y+3ibzfkagBIU1iV06HkMpPlysQdEdNg
ycKkrfslBhj+Vsy1eccas6/rBcOFZjhzjoIJqq8fAWX5W758eCd4UqDpN3tnUIU+
FD0m8E53KNiL0H4nbgcYEct5/SM2pqLB/s7ARkNgbMbyrkaXjpk+ekf4C2OrE8/w
qy7iOdKCWok=
=MlWv
-----END PGP SIGNATURE-----

RE: The Protocol [2nd Try]

[Giampaolo T. <Gia...@To...>]

From: pyz...@li...
[mailto:pyz...@li...]On Behalf Of Jerry Wilborn

  Talk about a can of worms!

  Does it really need a full client/server implementation? I'm new here, so
I'm ignorant to any discussion of why that method may have been selected to
start.  I can say that the only things I've seen that can stand up to the
hundreds of queries/second that worldwide lists punish servers with are
'rbldnsd' and 'rbldns'.  With a message hash it feels a bit like 'square
peg/round hole' (hashes and daemons built to do reverse DNS).  You guys have
experience stress testing tinydns? =)

  To ask the shorter question, can we setup 'hash.pyzor.something.ext' as
the host, and then have a TXT record with the response?
Pyzor cames with its own client/server protocol. It doesn't mean that the
query interface cannot be changed sometime in the future, but actually FWIK
this is not regarded as a main priority.

Just to le you know, the Pyzor's main priority is actually server's load
balancing and multi-master asynchronous replication. The purpouse is to
allow many, distributed public servers serving queries to the pyzor
database.

Please note that similar, proprietary implementations (notably, Razor) do
adopt a non-rfc client/server protocol, but also supply a net of distributed
fontends to their database.

A DNS-lookup interface to pyzor would shurely simplify quering the db, but
would not make any functional improvement in spam reporting.

Giampaolo

  Here's what I can tell from just kinda 10,000' overviewing it, please
interject any information you may have.  The program 'pyzor' connects to the
server/port listed in ~/.pyzor/servers (can be updated with 'pyzor
discover').  If you tell it to checck ('pyzor check < /tmp/spamfile') it
will compute a hash of the message (how?) and then send a HTTPish type
command to the server via tcp/cleartext on the other end. A command looks
like this:

  User: anonymous [ static ]
  Time: 1164688654  [ seconds since the epoch ]
  Sig: 45aed8ad7d2bd59f339c7e88c0d4e533c1c68b9c   [ dunno how it's computed,
but it changes based on message content ]


  Op: check [ specified via command line ]
  Op-Digest: 0dfe4745346676d78cc6bc11ff71d444546c0544 [ don't now how it's
computed, 'echo -n check | md5sum' doesn't come back with this hash... it's
static ]
  Thread: 26283 [ pid ?]
  PV: 2.0 [ version number i guess; static, ends with \n\n]

  I haven't been able to get a response, and with my python being
rudementary at best, it's going to take me a while to figure out what one
looks like.  It looks like the Spamassassin parsing code
[.../Mail/SpamAssassin/Plugin/Pyzor.pm] may be more helpful than the
server.py.


  On 11/27/06, Marc G. Fournier < sc...@hu...> wrote:
    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1



    - --On Monday, November 27, 2006 22:04:30 -0600 Jerry Wilborn
    < jer...@gm...> wrote:

    > Please forgive my ignorance of the protocol. Is there a place where I
can
    > read about how the actual question/answer is transmitted? Are we
re-inventing
    > the wheel? Is it possible to hook this into DNS like the other .*lists
do?

    What do you mean by 'hook this into DNS'?  Do you mean like SPF and
such?  Or
    are you thinking of something different ...

    As for Pyzor protocol  ... no idea on whether or not this was ever
published by
    the original developer ;(



  --
  Jerry Wilborn
  jer...@gm...

The Protocol [2nd Try]

[Jerry W. <jer...@gm...>]

Talk about a can of worms!

Does it really need a full client/server implementation? I'm new here, so
I'm ignorant to any discussion of why that method may have been selected to
start.  I can say that the only things I've seen that can stand up to the
hundreds of queries/second that worldwide lists punish servers with are
'rbldnsd' and 'rbldns'.  With a message hash it feels a bit like 'square
peg/round hole' (hashes and daemons built to do reverse DNS).  You guys have
experience stress testing tinydns? =)

To ask the shorter question, can we setup 'hash.pyzor.something.ext' as the
host, and then have a TXT record with the response?

Here's what I can tell from just kinda 10,000' overviewing it, please
interject any information you may have.  The program 'pyzor' connects to the
server/port listed in ~/.pyzor/servers (can be updated with 'pyzor
discover').  If you tell it to checck ('pyzor check < /tmp/spamfile') it
will compute a hash of the message (how?) and then send a HTTPish type
command to the server via tcp/cleartext on the other end. A command looks
like this:

User: anonymous [ static ]
Time: 1164688654  [ seconds since the epoch ]
Sig: 45aed8ad7d2bd59f339c7e88c0d4e533c1c68b9c   [ dunno how it's computed,
but it changes based on message content ]

Op: check [ specified via command line ]
Op-Digest: 0dfe4745346676d78cc6bc11ff71d444546c0544 [ don't now how it's
computed, 'echo -n check | md5sum' doesn't come back with this hash... it's
static ]
Thread: 26283 [ pid ?]
PV: 2.0 [ version number i guess; static, ends with \n\n]

I haven't been able to get a response, and with my python being rudementary
at best, it's going to take me a while to figure out what one looks like.
It looks like the Spamassassin parsing code
[.../Mail/SpamAssassin/Plugin/Pyzor.pm] may be more helpful than the
server.py.
On 11/27/06, Marc G. Fournier <sc...@hu...> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> - --On Monday, November 27, 2006 22:04:30 -0600 Jerry Wilborn
> <jer...@gm...> wrote:
>
> > Please forgive my ignorance of the protocol. Is there a place where I
> can
> > read about how the actual question/answer is transmitted? Are we
> re-inventing
> > the wheel? Is it possible to hook this into DNS like the other .*lists
> do?
>
> What do you mean by 'hook this into DNS'?  Do you mean like SPF and
> such?  Or
> are you thinking of something different ...
>
> As for Pyzor protocol  ... no idea on whether or not this was ever
> published by
> the original developer ;(
>


-- 
Jerry Wilborn
jer...@gm...

Help me feed pyzor

[Jonathan N. <jni...@pb...>]

I currently have a setup where users drop spam into a "Learn as Spam" 
folder. That folder is tarred up and SCPd to a mail gateway host, where 
the Worlds Crappiest Perl Script unpacks it, runs the mails through 
sa-learn, and gets rid of the spam tarball that was left over. Every 15 
minutes, this process repeats itself, gobbling up whatever spam users 
have added to the "learn as spam" folder.

Here's a bit of the script:

system("mv home spam");
system("sa-learn --showdots --spam 
spam/vmail/shared-maildirs/Spamdrop/.Incoming/cur");
system("/usr/sbin/amavisd reload");
system("rm -rf /home/food/spam");

sometimes there are quite a few emails in the "cur" folder. Can I do 
something like "pyzor report < *" or is it going to be more complicated 
than that?

My Perl-foo is very weak, so I ask you, pyzor hive mind, for help. :D

--
Jonathan

Re: Pyzor server(s)

[Andreas S. <sch...@fa...>]

On Tue, 28 Nov 2006, Ugo Bellavance wrote:

> Anyone willing to donate some money to help the process.  I would 
> put 20$.

Count me in! With 21$!

-- 
-- Andreas

RE: Pyzor server(s)

[Ugo B. <ug...@ca...>]

>=20
>      I'm not a python programmer, either, but
>=20
>         I've picked up lotsa languages in my life, would look
>         forward to writing some python and don't think the
>         learning curve looks too steep (given experience with
>         Perl, Common Lisp and Haskell in particular)
>=20
>         I'm currently unemployed
>=20
>         I have a home network with several machines capable
>         of running python (and could test in a heterogenous
>         environment in terms of OS/distro/arch)
>=20
>         I've also done some thinking about building a
>         distributed pyzor and agree that it's the obvious
>         next step but don't have "ideas on how to accomplish
>         a distributed pyzor relatively easily"*
>=20
> Perhaps I could act as your amanuensis.
>=20
>                                                best regards,
>=20
>                                                    Robt. P.
>=20

Anyone willing to donate some money to help the process.  I would put
20$.

Ugo

Re: Pyzor server(s) ...

[Jerry W. <jer...@gm...>]

Talk about a can of worms!

Does it really need a full client/server implementation? I'm new here, so
I'm ignorant to any discussion of why that method may have been selected to
start.  I can say that the only things I've seen that can stand up to the
hundreds of queries/second that worldwide lists punish servers with are
'rbldnsd' and 'rbldns'.  With a message hash it feels a bit like 'square
peg/round hole' (hashes and daemons built to do reverse DNS).  You guys have
experience stress testing tinydns? =)

To ask the shorter question, can we setup 'hash.pyzor.something.ext' as a
host, and then have a TXT record with the response?

Here's what I can tell from just kinda 10,000' overviewing it, please
interject any information you may have.  The program 'pyzor' connects to the
server/port listed in ~/.pyzor/servers (can be updated with 'pyzor
discover').  If you tell it to checck ('pyzor check < /tmp/spamfile') it
will compute a hash of the message (how?) and then send a HTTPish type
command to the server via tcp/cleartext on the other end. A command looks
like this:

User: anonymous [ static ]
Time: 1164688654  [ seconds since the epoch ]
Sig: 45aed8ad7d2bd59f339c7e88c0d4e533c1c68b9c   [ dunno how it's computed,
but it changes based on message content ]

Op: check [ specified via command line ]
Op-Digest: 0dfe4745346676d78cc6bc11ff71d444546c0544 [ don't now how it's
computed, 'echo -n check | md5sum' doesn't come back with this hash... it's
static ]
Thread: 26283 [ pid ?]
PV: 2.0 [ version number i guess; static, ends with \n\n]

I haven't been able to get a response, and with my python being rudementary
at best, it's going to take me a while to figure out what one looks like.
It looks like the Spamassassin parsing code
[.../Mail/SpamAssassin/Plugin/Pyzor.pm] may be more helpful than the
server.py.

On 11/27/06, Marc G. Fournier <sc...@hu...> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> - --On Monday, November 27, 2006 22:04:30 -0600 Jerry Wilborn
> <jer...@gm...> wrote:
>
> > Please forgive my ignorance of the protocol. Is there a place where I
> can
> > read about how the actual question/answer is transmitted? Are we
> re-inventing
> > the wheel? Is it possible to hook this into DNS like the other .*lists
> do?
>
> What do you mean by 'hook this into DNS'?  Do you mean like SPF and
> such?  Or
> are you thinking of something different ...
>
> As for Pyzor protocol  ... no idea on whether or not this was ever
> published by
> the original developer ;(
>
> >
> >
> > On 11/27/06, Marc G. Fournier <sc...@hu...> wrote:
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> >
> >
> > - --On Monday, November 27, 2006 21:31:53 -0500 Robert Praetorius
> > <rm...@ii...>
> > wrote:
> >
> >>         I've also done some thinking about building a
> >>         distributed pyzor and agree that it's the obvious
> >>         next step but don't have "ideas on how to accomplish
> >>         a distributed pyzor relatively easily"*
> >
> > Actually, its very simple "in theory" ... all you need is to change the
> > communications protocol slightly to distinguish between a server or
> client
> > packet ... basically, if a client reports to the server, then that
> server
> > needs
> > to then report the same thing to the other servers, but "as a server"
> ...
> > when
> > a server reports to a server, that is the end of the line ...
> >
> > I was just thinking about it, and the best paradigm, I think, for this
> is
> > Usenet news ... each server would be a 'registered peer' off of another
> one
> > ...
> >
> > Let's say, for instance, that we have a server in NA at Site A ... if I
> wanted
> > to add a pyzor server onto that system, I would talk to the admin @ Site
> A
> > about connecting to him ... connection would be a simple as a
> servers.txt file
> > that contains the hostname of the remote server ...
> >
> > When Site A receives a report, it auto gets forwarded down to its
> 'children'
> > ... if my site receives a report, I would forward it up to Site A ...
> >
> > Now, Site A knows that it received the report from me, so wouldn't send
> it
> > back
> > to me ...
> >
> > Now, in theory, each server would only need max two connections ... an
> > upstream
> > and a downstream, but for redundancy, and speed, more then one would be
> > preferred ...
> >
> > So, as an example, let's say we have a single server in: US, Panama (my
> > servers), EU, Australia ... at a minimum, you'd want something like:
> >
> > Australia <-> US <-> EU
> >                ^
> >                |
> >                v
> >             Panama
> >
> > To improve redundancy, you could add links between Australia <-> EU, AU
> <-> PA
> > and PA <-> EU, but they wouldn't be required ...
> >
> > EU could then get a link from Africa, while Australia maybe from Korean,
> PA
> > from Brazil, etc ...
> >
> > Within US, you could break things down add add regional servers, etc,
> etc ...
> >
> > A large ISP could then run their own Pyzor server, but as part of the
> > distributed network ...
> >
> > Its theory, and simple one at that
> >
> > - ----
> > Marc G. Fournier           Hub.Org Networking Services (
> http://www.hub.org)
> > Email . sc...@hu...                              MSN .
> sc...@hu...
> > Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.5 (FreeBSD)
> >
> > iD8DBQFFa6li4QvfyHIvDvMRAmVNAKCB3VXTKuWA8a15XxpKHr9VmjB11wCfWbls
> > qDm9JHFClR8UBhg7p/CeIkM=
> > =EnYO
> > -----END PGP SIGNATURE-----
> >
> >
> >
> >
> >
> > --
> > Jerry Wilborn
> > jer...@gm...
>
>
>
> - ----
> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org
> )
> Email . sc...@hu...                              MSN . sc...@hu...
> Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (FreeBSD)
>
> iD8DBQFFa7sn4QvfyHIvDvMRAo/rAJ46wPrlPAO17t9GTDlI37tanYJdZQCgg2gN
> EXtsj+9PTlIYIhydrzF8iyQ=
> =Z7rK
> -----END PGP SIGNATURE-----
>
>


-- 
Jerry Wilborn
jer...@gm...

Re: Pyzor server(s) ...

[Marc G. F. <sc...@hu...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



- --On Monday, November 27, 2006 22:04:30 -0600 Jerry Wilborn 
<jer...@gm...> wrote:

> Please forgive my ignorance of the protocol. Is there a place where I can
> read about how the actual question/answer is transmitted? Are we re-inventing
> the wheel? Is it possible to hook this into DNS like the other .*lists do?

What do you mean by 'hook this into DNS'?  Do you mean like SPF and such?  Or 
are you thinking of something different ...

As for Pyzor protocol  ... no idea on whether or not this was ever published by 
the original developer ;(

>
>
> On 11/27/06, Marc G. Fournier <sc...@hu...> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> - --On Monday, November 27, 2006 21:31:53 -0500 Robert Praetorius
> <rm...@ii...>
> wrote:
>
>>         I've also done some thinking about building a
>>         distributed pyzor and agree that it's the obvious
>>         next step but don't have "ideas on how to accomplish
>>         a distributed pyzor relatively easily"*
>
> Actually, its very simple "in theory" ... all you need is to change the
> communications protocol slightly to distinguish between a server or client
> packet ... basically, if a client reports to the server, then that server
> needs
> to then report the same thing to the other servers, but "as a server" ...
> when
> a server reports to a server, that is the end of the line ...
>
> I was just thinking about it, and the best paradigm, I think, for this is
> Usenet news ... each server would be a 'registered peer' off of another one
> ...
>
> Let's say, for instance, that we have a server in NA at Site A ... if I wanted
> to add a pyzor server onto that system, I would talk to the admin @ Site A
> about connecting to him ... connection would be a simple as a servers.txt file
> that contains the hostname of the remote server ...
>
> When Site A receives a report, it auto gets forwarded down to its 'children'
> ... if my site receives a report, I would forward it up to Site A ...
>
> Now, Site A knows that it received the report from me, so wouldn't send it
> back
> to me ...
>
> Now, in theory, each server would only need max two connections ... an
> upstream
> and a downstream, but for redundancy, and speed, more then one would be
> preferred ...
>
> So, as an example, let's say we have a single server in: US, Panama (my
> servers), EU, Australia ... at a minimum, you'd want something like:
>
> Australia <-> US <-> EU
>                ^
>                |
>                v
>             Panama
>
> To improve redundancy, you could add links between Australia <-> EU, AU <-> PA
> and PA <-> EU, but they wouldn't be required ...
>
> EU could then get a link from Africa, while Australia maybe from Korean, PA
> from Brazil, etc ...
>
> Within US, you could break things down add add regional servers, etc, etc ...
>
> A large ISP could then run their own Pyzor server, but as part of the
> distributed network ...
>
> Its theory, and simple one at that
>
> - ----
> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
> Email . sc...@hu...                              MSN . sc...@hu...
> Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (FreeBSD)
>
> iD8DBQFFa6li4QvfyHIvDvMRAmVNAKCB3VXTKuWA8a15XxpKHr9VmjB11wCfWbls
> qDm9JHFClR8UBhg7p/CeIkM=
> =EnYO
> -----END PGP SIGNATURE-----
>
>
>
>
>
> --
> Jerry Wilborn
> jer...@gm...



- ----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . sc...@hu...                              MSN . sc...@hu...
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)

iD8DBQFFa7sn4QvfyHIvDvMRAo/rAJ46wPrlPAO17t9GTDlI37tanYJdZQCgg2gN
EXtsj+9PTlIYIhydrzF8iyQ=
=Z7rK
-----END PGP SIGNATURE-----

Re: Pyzor server(s) ...

[Jerry W. <jer...@gm...>]

Please forgive my ignorance of the protocol. Is there a place where I can
read about how the actual question/answer is transmitted? Are we
re-inventing the wheel? Is it possible to hook this into DNS like the other
.*lists do?

On 11/27/06, Marc G. Fournier <sc...@hu...> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> - --On Monday, November 27, 2006 21:31:53 -0500 Robert Praetorius <
> rm...@ii...>
> wrote:
>
> >         I've also done some thinking about building a
> >         distributed pyzor and agree that it's the obvious
> >         next step but don't have "ideas on how to accomplish
> >         a distributed pyzor relatively easily"*
>
> Actually, its very simple "in theory" ... all you need is to change the
> communications protocol slightly to distinguish between a server or client
> packet ... basically, if a client reports to the server, then that server
> needs
> to then report the same thing to the other servers, but "as a server" ...
> when
> a server reports to a server, that is the end of the line ...
>
> I was just thinking about it, and the best paradigm, I think, for this is
> Usenet news ... each server would be a 'registered peer' off of another
> one ...
>
> Let's say, for instance, that we have a server in NA at Site A ... if I
> wanted
> to add a pyzor server onto that system, I would talk to the admin @ Site A
> about connecting to him ... connection would be a simple as a servers.txtfile
> that contains the hostname of the remote server ...
>
> When Site A receives a report, it auto gets forwarded down to its
> 'children'
> ... if my site receives a report, I would forward it up to Site A ...
>
> Now, Site A knows that it received the report from me, so wouldn't send it
> back
> to me ...
>
> Now, in theory, each server would only need max two connections ... an
> upstream
> and a downstream, but for redundancy, and speed, more then one would be
> preferred ...
>
> So, as an example, let's say we have a single server in: US, Panama (my
> servers), EU, Australia ... at a minimum, you'd want something like:
>
> Australia <-> US <-> EU
>                ^
>                |
>                v
>             Panama
>
> To improve redundancy, you could add links between Australia <-> EU, AU
> <-> PA
> and PA <-> EU, but they wouldn't be required ...
>
> EU could then get a link from Africa, while Australia maybe from Korean,
> PA
> from Brazil, etc ...
>
> Within US, you could break things down add add regional servers, etc, etc
> ...
>
> A large ISP could then run their own Pyzor server, but as part of the
> distributed network ...
>
> Its theory, and simple one at that
>
> - ----
> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org
> )
> Email . sc...@hu...                              MSN . sc...@hu...
> Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (FreeBSD)
>
> iD8DBQFFa6li4QvfyHIvDvMRAmVNAKCB3VXTKuWA8a15XxpKHr9VmjB11wCfWbls
> qDm9JHFClR8UBhg7p/CeIkM=
> =EnYO
> -----END PGP SIGNATURE-----
>
>


-- 
Jerry Wilborn
jer...@gm...

Re: Pyzor server(s) ...

[Marc G. F. <sc...@hu...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



- --On Monday, November 27, 2006 21:31:53 -0500 Robert Praetorius <rm...@ii...> 
wrote:

>         I've also done some thinking about building a
>         distributed pyzor and agree that it's the obvious
>         next step but don't have "ideas on how to accomplish
>         a distributed pyzor relatively easily"*

Actually, its very simple "in theory" ... all you need is to change the 
communications protocol slightly to distinguish between a server or client 
packet ... basically, if a client reports to the server, then that server needs 
to then report the same thing to the other servers, but "as a server" ... when 
a server reports to a server, that is the end of the line ...

I was just thinking about it, and the best paradigm, I think, for this is 
Usenet news ... each server would be a 'registered peer' off of another one ...

Let's say, for instance, that we have a server in NA at Site A ... if I wanted 
to add a pyzor server onto that system, I would talk to the admin @ Site A 
about connecting to him ... connection would be a simple as a servers.txt file 
that contains the hostname of the remote server ...

When Site A receives a report, it auto gets forwarded down to its 'children' 
... if my site receives a report, I would forward it up to Site A ...

Now, Site A knows that it received the report from me, so wouldn't send it back 
to me ...

Now, in theory, each server would only need max two connections ... an upstream 
and a downstream, but for redundancy, and speed, more then one would be 
preferred ...

So, as an example, let's say we have a single server in: US, Panama (my 
servers), EU, Australia ... at a minimum, you'd want something like:

Australia <-> US <-> EU
               ^
               |
               v
            Panama

To improve redundancy, you could add links between Australia <-> EU, AU <-> PA 
and PA <-> EU, but they wouldn't be required ...

EU could then get a link from Africa, while Australia maybe from Korean, PA 
from Brazil, etc ...

Within US, you could break things down add add regional servers, etc, etc ...

A large ISP could then run their own Pyzor server, but as part of the 
distributed network ...

Its theory, and simple one at that

- ----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . sc...@hu...                              MSN . sc...@hu...
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)

iD8DBQFFa6li4QvfyHIvDvMRAmVNAKCB3VXTKuWA8a15XxpKHr9VmjB11wCfWbls
qDm9JHFClR8UBhg7p/CeIkM=
=EnYO
-----END PGP SIGNATURE-----

Re: Pyzor server(s) ...

[Robert P. <rm...@ii...>]

> the problem isn't so much mirror servers, but that pyzor itself
> needs to be extended to allow for a distributed environment, which
> means someone needs to know how to program python :(

> I don't ... I'm a perl/php programmer ... I have ideas on how to
> accomplish a distributed pyzor relatively easily, if someone wants
> to step up to the programming plate? :)

     I'm not a python programmer, either, but

        I've picked up lotsa languages in my life, would look
        forward to writing some python and don't think the
        learning curve looks too steep (given experience with
        Perl, Common Lisp and Haskell in particular)

        I'm currently unemployed

        I have a home network with several machines capable
        of running python (and could test in a heterogenous
        environment in terms of OS/distro/arch)

        I've also done some thinking about building a
        distributed pyzor and agree that it's the obvious
        next step but don't have "ideas on how to accomplish
        a distributed pyzor relatively easily"*

Perhaps I could act as your amanuensis.

                                               best regards,

                                                   Robt. P.

*maybe 'cuz my ideas run towards radical distribution, which is
 somewhat more complicated than a smaller scale distribution

RE: Pyzor server(s) ...

[Marc G. F. <sc...@hu...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


the problem isn't so much mirror servers, but that pyzor itself needs to be 
extended to allow for a distributed environment, which means someone needs to 
know how to program python :(

I don't ... I'm a perl/php programmer ... I have ideas on how to accomplish a 
distributed pyzor relatively easily, if someone wants to step up to the 
programming plate? :)

- --On Monday, November 27, 2006 19:39:51 -0600 Jerry Wilborn 
<jer...@gm...> wrote:

> Hello,
>
> The company I work for (an ISP/CLEC) presently provides public mirror
> services for two of the more popular blacklisting services; we have several
> years long working relationships with both maintainers. I'd like to also
> extend the offer of a pyzor mirror.  It would reside on a server in one of
> our datacenters in Eastern PA.
>
> I'd just need a little data on usage statistics.  Root access (if necessary)
> could also be provided.
>
> Thoughts?
>
>
>
>
> Pyzor server(s) ... From: Marc G. Fournier <scrappy@hu...> - 2006-02-06 10:59
>
>  Hi ...
>
>     Since pyzor is pretty much the only viable 'clearinghouse' still, with
>  everyone else starting to limit who can make use of it, I'm wondering what
>  it would take to improve Pyzor ...
>
>
>     The big thing I've seen on the lists is the timeouts issue, and, from
>  what I can tell, the problem is that its all running on a single server
>  right now ...
>
>     Unfortunately, python isn't my language, so doing any coding on that
>
>  side I'll be useless for, but I've been looking at how clamav is dealing
>  with the virus signature updates and am wondering if its somehow possible
>  to apply what they are doing to pyzor itself ...
>
>     Now, granted, I have no idea how large the
> pyzord.db file is/gets, so,
>  of course, this becomes a very big limiting factor, but with the clamav
>  folks, all they do is use rsync to pass around the database itself to
>  'slave servers' ...
>
>     The idea is to have one master server (
> db.pyzor.org) that all reports
>  are submitted to, and several slave servers (db.xy.pyzor.org) that handle
>  the check requests ... where xy would be a country code ... right off the
>
>  bat, I could easily provide a db.us.pyzor.org and db.pa.pyzor.org server
>  ...
>
>     The only thing that would (I think) have to change in the pyzor code
>
>  itself is allow for a master vs slave scenario in the config files, so
>  what when discover is done, they get set appropriately, but I can't think
>  that that would be too hard ...
>
>     Frank, about what size is the
> pyzord.db file?  How small does it shrink
>  if compressed?
>
>     Just my two cents, and an offer of server/network resources ...
>
>
>  ----
>  Marc G. Fournier           Hub.Org Networking Services (
> http://www.hub.org)
>  Email: scrappy@hu...           Yahoo!: yscrappy              ICQ: 7615664
>
>
>
>
> --
> Jerry Wilborn
> jer...@gm...



- ----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . sc...@hu...                              MSN . sc...@hu...
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)

iD8DBQFFa5XI4QvfyHIvDvMRAg5WAKDOiYZCiNjaX2MDf1L8lDaYVG91AwCgg+/Q
Xx5zh7/5ZXnUIKCTn52Kkfo=
=xNLw
-----END PGP SIGNATURE-----

RE: Pyzor server(s) ...

[Jerry W. <jer...@gm...>]

Hello,

The company I work for (an ISP/CLEC) presently provides public mirror
services for two of the more popular blacklisting services; we have several
years long working relationships with both maintainers. I'd like to also
extend the offer of a pyzor mirror.  It would reside on a server in one of
our datacenters in Eastern PA.

I'd just need a little data on usage statistics.  Root access (if necessary)
could also be provided.

Thoughts?


*Pyzor server(s) ...*
From: Marc G. Fournier <scrappy@hu...> - 2006-02-06 10:59

 Hi ...

    Since pyzor is pretty much the only viable 'clearinghouse' still, with
 everyone else starting to limit who can make use of it, I'm wondering what
 it would take to improve Pyzor ...

    The big thing I've seen on the lists is the timeouts issue, and, from
 what I can tell, the problem is that its all running on a single server
 right now ...

    Unfortunately, python isn't my language, so doing any coding on that
 side I'll be useless for, but I've been looking at how clamav is dealing
 with the virus signature updates and am wondering if its somehow possible
 to apply what they are doing to pyzor itself ...

    Now, granted, I have no idea how large the pyzord.db file is/gets, so,
 of course, this becomes a very big limiting factor, but with the clamav
 folks, all they do is use rsync to pass around the database itself to
 'slave servers' ...

    The idea is to have one master server (db.pyzor.org) that all reports
 are submitted to, and several slave servers (db.xy.pyzor.org) that handle
 the check requests ... where xy would be a country code ... right off the
 bat, I could easily provide a db.us.pyzor.org and db.pa.pyzor.org server
 ...

    The only thing that would (I think) have to change in the pyzor code
 itself is allow for a master vs slave scenario in the config files, so
 what when discover is done, they get set appropriately, but I can't think
 that that would be too hard ...

    Frank, about what size is the pyzord.db file?  How small does it shrink
 if compressed?

    Just my two cents, and an offer of server/network resources ...


 ----
 Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
 Email: scrappy@hu...           Yahoo!: yscrappy              ICQ: 7615664



-- 
Jerry Wilborn
jer...@gm...

Re[2]: ping

[Dmitry <py...@vi...>]

Yes, peereing is the bets solution here. But AFAIK currently it is not
implemented.
I have site with huge mail traffic and it less expensive to
download copy of DB every hour then make request to remote server for every
message.



BP> On Tue, November 14, 2006 17:13, Dmitry wrote:

>> What is URL for DB copy?
>> Is it available for maintaining local pyzor server?

BP> this is not the best way to solve it :(

BP> pyzord should have server 2 server exchange of digest, so pyzor works transperant




-- 
Regards,
 Dmitry

Re: ping

[Benny P. <me...@ju...>]

On Tue, November 14, 2006 17:13, Dmitry wrote:

> What is URL for DB copy?
> Is it available for maintaining local pyzor server?

this is not the best way to solve it :(

pyzord should have server 2 server exchange of digest, so pyzor works tra=
nsperant

--=20
This message was sent using 100% recycled spam mails.

Re: ping

[Dmitry <py...@vi...>]

Greetings!
> Just sending a ping to the list to check if we're still alive.
>  after looking at the stats and the load on the machine('s) I can  
>  conclude that:
>       3. we need to get the database transparent and  as distributed as
>  possible asap.
>  
>  re. point 3 I will make a copy of the DB available via http (.tgz) on  
>  an hourly bases starting 2morrow. this means still there will still  
>  be a central point to report but the "check" load will be distributed.

What is URL for DB copy?
Is it available for maintaining local pyzor server?

>  best
>  Milton Cyrus



-- 
Regards,
 Dmitry

Re: Server out?

[Benny P. <me...@ju...>]

On Tue, October 24, 2006 21:34, Marc G. Fournier wrote:

> Thoughts?  I'd be willing to put up a well connected server to act as
> one leg of this if someone had the skills to design/develop such an
> extension ...

i have last night got my own pyzord up, next problem is how to configure
spamassassin pyzor plugin to both make check / report to it :(

it could be nice this was easy :-)

i have a dream about it could report if spamassassin finds score highter
then x and just check as normal, but the plugin does imho not support it =
:(

so to speak, i need more info on how to make pyzor client run on a sitewi=
de
server that uses ones own pyzord server

pyzor 0.4.0-r2 amd64 dual gentoo here
let me know if i need to provide more info --

--=20
"This message was sent using 100% recycled spam mails."

R: R: R: Reporting

[Giampaolo T. <Gia...@To...>]

> Note you can put this in your SA configuration:
>=20
> add_header all Pyzor _PYZOR_
>=20
> That will show the number of hits reported on the spam
> (or nothing in case of a timeout).

Thanks for this: it seems a very good hint to me.


>  > Are you shure that reports made on 82.94.255.100 are somehow
>  > synchronized with the ones made on 66.250.40.33? The latter,
>  > infact, was the server I was using when pyzor worked to me...
>=20
> There's no synchronization between servers.
> It's a project that no one has taken on (see the list archives).

Ah, that's why: scores are, in the best case, halved among two servers. =
In the worst, I'm using the server with much less reporters...


> You can put both servers in your pyzor config.  Look in the list
> archives for a patch to SA to handle that.
> (Subject: [patch] multiple pyzor servers for SpamAssassin
>  or http://issues.apache.org/SpamAssassin/show_bug.cgi?id=3D5148)

Ok. I'll check it out.

Thanks,

-----------------------------------
Giampaolo Tomassoni - IT Consultant
Piazza VIII Aprile 1948, 4
I-53044 Chiusi (SI) - Italy
Ph: +39-0578-21100

MAI inviare una e-mail a:
NEVER send an e-mail to:
 rai...@to...

Re: R: R: Reporting

[Milton C. <mi...@i-...>]

>
> There's no synchronization between servers.
> It's a project that no one has taken on (see the list archives).
>


I can confirm there's no sync between the current servers, I  
encourage people to,for now , report to to both.

Re. the sync. project, we are in the process of completing a version  
that will enable sync. between connected servers.
I will keep the list updated on our developments and will post a link  
if we have a version ready that can go-out.

best,
Milton

Re: R: R: Reporting

[John E H. <jh...@ti...>]

Giampaolo Tomassoni wrote at 18:52 +0200 on Oct 27, 2006:
 > Ok, thanks. However, some weeks ago (when I could see 'PYZOR'
 > scores as soon as I reported a spam), it was consistently working
 > the way I told you. I mean, is it possible I was always the fifth
 > reporter of the spam? Weird...
 > 
 > You shure nothing changed?

Just the servers coming and going.

I've found that a few retries with a shorter timeout are more
effective than a single longer timeout (which is what SpamAssassin
uses, at least in all versions I've seen up to 3.1.6).

Note you can put this in your SA configuration:

add_header all Pyzor _PYZOR_

That will show the number of hits reported on the spam
(or nothing in case of a timeout).


 > Are you shure that reports made on 82.94.255.100 are somehow
 > synchronized with the ones made on 66.250.40.33? The latter,
 > infact, was the server I was using when pyzor worked to me...

There's no synchronization between servers.
It's a project that no one has taken on (see the list archives).

You can put both servers in your pyzor config.  Look in the list
archives for a patch to SA to handle that.
(Subject: [patch] multiple pyzor servers for SpamAssassin
 or http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5148)