Menu
▾
▴
pyzor-users
|
From: Jerry W. <jer...@gm...> - 2006-11-28 01:40:03
|
Hello,
The company I work for (an ISP/CLEC) presently provides public mirror
services for two of the more popular blacklisting services; we have several
years long working relationships with both maintainers. I'd like to also
extend the offer of a pyzor mirror. It would reside on a server in one of
our datacenters in Eastern PA.
I'd just need a little data on usage statistics. Root access (if necessary)
could also be provided.
Thoughts?
*Pyzor server(s) ...*
From: Marc G. Fournier <scrappy@hu...> - 2006-02-06 10:59
Hi ...
Since pyzor is pretty much the only viable 'clearinghouse' still, with
everyone else starting to limit who can make use of it, I'm wondering what
it would take to improve Pyzor ...
The big thing I've seen on the lists is the timeouts issue, and, from
what I can tell, the problem is that its all running on a single server
right now ...
Unfortunately, python isn't my language, so doing any coding on that
side I'll be useless for, but I've been looking at how clamav is dealing
with the virus signature updates and am wondering if its somehow possible
to apply what they are doing to pyzor itself ...
Now, granted, I have no idea how large the pyzord.db file is/gets, so,
of course, this becomes a very big limiting factor, but with the clamav
folks, all they do is use rsync to pass around the database itself to
'slave servers' ...
The idea is to have one master server (db.pyzor.org) that all reports
are submitted to, and several slave servers (db.xy.pyzor.org) that handle
the check requests ... where xy would be a country code ... right off the
bat, I could easily provide a db.us.pyzor.org and db.pa.pyzor.org server
...
The only thing that would (I think) have to change in the pyzor code
itself is allow for a master vs slave scenario in the config files, so
what when discover is done, they get set appropriately, but I can't think
that that would be too hard ...
Frank, about what size is the pyzord.db file? How small does it shrink
if compressed?
Just my two cents, and an offer of server/network resources ...
----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hu... Yahoo!: yscrappy ICQ: 7615664
--
Jerry Wilborn
jer...@gm...
|
|
From: Marc G. F. <sc...@hu...> - 2006-11-28 01:50:03
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 the problem isn't so much mirror servers, but that pyzor itself needs to be extended to allow for a distributed environment, which means someone needs to know how to program python :( I don't ... I'm a perl/php programmer ... I have ideas on how to accomplish a distributed pyzor relatively easily, if someone wants to step up to the programming plate? :) - --On Monday, November 27, 2006 19:39:51 -0600 Jerry Wilborn <jer...@gm...> wrote: > Hello, > > The company I work for (an ISP/CLEC) presently provides public mirror > services for two of the more popular blacklisting services; we have several > years long working relationships with both maintainers. I'd like to also > extend the offer of a pyzor mirror. It would reside on a server in one of > our datacenters in Eastern PA. > > I'd just need a little data on usage statistics. Root access (if necessary) > could also be provided. > > Thoughts? > > > > > Pyzor server(s) ... From: Marc G. Fournier <scrappy@hu...> - 2006-02-06 10:59 > > Hi ... > > Since pyzor is pretty much the only viable 'clearinghouse' still, with > everyone else starting to limit who can make use of it, I'm wondering what > it would take to improve Pyzor ... > > > The big thing I've seen on the lists is the timeouts issue, and, from > what I can tell, the problem is that its all running on a single server > right now ... > > Unfortunately, python isn't my language, so doing any coding on that > > side I'll be useless for, but I've been looking at how clamav is dealing > with the virus signature updates and am wondering if its somehow possible > to apply what they are doing to pyzor itself ... > > Now, granted, I have no idea how large the > pyzord.db file is/gets, so, > of course, this becomes a very big limiting factor, but with the clamav > folks, all they do is use rsync to pass around the database itself to > 'slave servers' ... > > The idea is to have one master server ( > db.pyzor.org) that all reports > are submitted to, and several slave servers (db.xy.pyzor.org) that handle > the check requests ... where xy would be a country code ... right off the > > bat, I could easily provide a db.us.pyzor.org and db.pa.pyzor.org server > ... > > The only thing that would (I think) have to change in the pyzor code > > itself is allow for a master vs slave scenario in the config files, so > what when discover is done, they get set appropriately, but I can't think > that that would be too hard ... > > Frank, about what size is the > pyzord.db file? How small does it shrink > if compressed? > > Just my two cents, and an offer of server/network resources ... > > > ---- > Marc G. Fournier Hub.Org Networking Services ( > http://www.hub.org) > Email: scrappy@hu... Yahoo!: yscrappy ICQ: 7615664 > > > > > -- > Jerry Wilborn > jer...@gm... - ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . sc...@hu... MSN . sc...@hu... Yahoo . yscrappy Skype: hub.org ICQ . 7615664 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (FreeBSD) iD8DBQFFa5XI4QvfyHIvDvMRAg5WAKDOiYZCiNjaX2MDf1L8lDaYVG91AwCgg+/Q Xx5zh7/5ZXnUIKCTn52Kkfo= =xNLw -----END PGP SIGNATURE----- |
|
From: Robert P. <rm...@ii...> - 2006-11-28 02:32:24
|
> the problem isn't so much mirror servers, but that pyzor itself
> needs to be extended to allow for a distributed environment, which
> means someone needs to know how to program python :(
> I don't ... I'm a perl/php programmer ... I have ideas on how to
> accomplish a distributed pyzor relatively easily, if someone wants
> to step up to the programming plate? :)
I'm not a python programmer, either, but
I've picked up lotsa languages in my life, would look
forward to writing some python and don't think the
learning curve looks too steep (given experience with
Perl, Common Lisp and Haskell in particular)
I'm currently unemployed
I have a home network with several machines capable
of running python (and could test in a heterogenous
environment in terms of OS/distro/arch)
I've also done some thinking about building a
distributed pyzor and agree that it's the obvious
next step but don't have "ideas on how to accomplish
a distributed pyzor relatively easily"*
Perhaps I could act as your amanuensis.
best regards,
Robt. P.
*maybe 'cuz my ideas run towards radical distribution, which is
somewhat more complicated than a smaller scale distribution
|
|
From: Jesus C. <jc...@ar...> - 2006-12-01 16:17:41
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Robert Praetorius wrote: > I'm not a python programmer, either, but Is the pyzor programmer still working on it, or is this project "abandoned"?. In the second case the first step would be to regenerate the project with new blood. I remember, a few years ago, trying to integrate pyzor in my antispam service (written in python), but the code was not easily reusable as a service "object". You need to call an external process. I did a few suggestions with no luck. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jc...@ar... http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jc...@ja... _/_/ _/_/ _/_/_/_/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBRXBVl5lgi5GaxT1NAQJ8EgP/Y+3ibzfkagBIU1iV06HkMpPlysQdEdNg ycKkrfslBhj+Vsy1eccas6/rBcOFZjhzjoIJqq8fAWX5W758eCd4UqDpN3tnUIU+ FD0m8E53KNiL0H4nbgcYEct5/SM2pqLB/s7ARkNgbMbyrkaXjpk+ekf4C2OrE8/w qy7iOdKCWok= =MlWv -----END PGP SIGNATURE----- |
|
From: Marc G. F. <sc...@hu...> - 2006-11-28 03:13:34
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
- --On Monday, November 27, 2006 21:31:53 -0500 Robert Praetorius <rm...@ii...>
wrote:
> I've also done some thinking about building a
> distributed pyzor and agree that it's the obvious
> next step but don't have "ideas on how to accomplish
> a distributed pyzor relatively easily"*
Actually, its very simple "in theory" ... all you need is to change the
communications protocol slightly to distinguish between a server or client
packet ... basically, if a client reports to the server, then that server needs
to then report the same thing to the other servers, but "as a server" ... when
a server reports to a server, that is the end of the line ...
I was just thinking about it, and the best paradigm, I think, for this is
Usenet news ... each server would be a 'registered peer' off of another one ...
Let's say, for instance, that we have a server in NA at Site A ... if I wanted
to add a pyzor server onto that system, I would talk to the admin @ Site A
about connecting to him ... connection would be a simple as a servers.txt file
that contains the hostname of the remote server ...
When Site A receives a report, it auto gets forwarded down to its 'children'
... if my site receives a report, I would forward it up to Site A ...
Now, Site A knows that it received the report from me, so wouldn't send it back
to me ...
Now, in theory, each server would only need max two connections ... an upstream
and a downstream, but for redundancy, and speed, more then one would be
preferred ...
So, as an example, let's say we have a single server in: US, Panama (my
servers), EU, Australia ... at a minimum, you'd want something like:
Australia <-> US <-> EU
^
|
v
Panama
To improve redundancy, you could add links between Australia <-> EU, AU <-> PA
and PA <-> EU, but they wouldn't be required ...
EU could then get a link from Africa, while Australia maybe from Korean, PA
from Brazil, etc ...
Within US, you could break things down add add regional servers, etc, etc ...
A large ISP could then run their own Pyzor server, but as part of the
distributed network ...
Its theory, and simple one at that
- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . sc...@hu... MSN . sc...@hu...
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)
iD8DBQFFa6li4QvfyHIvDvMRAmVNAKCB3VXTKuWA8a15XxpKHr9VmjB11wCfWbls
qDm9JHFClR8UBhg7p/CeIkM=
=EnYO
-----END PGP SIGNATURE-----
|
|
From: Jerry W. <jer...@gm...> - 2006-11-28 04:04:32
|
Please forgive my ignorance of the protocol. Is there a place where I can read about how the actual question/answer is transmitted? Are we re-inventing the wheel? Is it possible to hook this into DNS like the other .*lists do? On 11/27/06, Marc G. Fournier <sc...@hu...> wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > > - --On Monday, November 27, 2006 21:31:53 -0500 Robert Praetorius < > rm...@ii...> > wrote: > > > I've also done some thinking about building a > > distributed pyzor and agree that it's the obvious > > next step but don't have "ideas on how to accomplish > > a distributed pyzor relatively easily"* > > Actually, its very simple "in theory" ... all you need is to change the > communications protocol slightly to distinguish between a server or client > packet ... basically, if a client reports to the server, then that server > needs > to then report the same thing to the other servers, but "as a server" ... > when > a server reports to a server, that is the end of the line ... > > I was just thinking about it, and the best paradigm, I think, for this is > Usenet news ... each server would be a 'registered peer' off of another > one ... > > Let's say, for instance, that we have a server in NA at Site A ... if I > wanted > to add a pyzor server onto that system, I would talk to the admin @ Site A > about connecting to him ... connection would be a simple as a servers.txtfile > that contains the hostname of the remote server ... > > When Site A receives a report, it auto gets forwarded down to its > 'children' > ... if my site receives a report, I would forward it up to Site A ... > > Now, Site A knows that it received the report from me, so wouldn't send it > back > to me ... > > Now, in theory, each server would only need max two connections ... an > upstream > and a downstream, but for redundancy, and speed, more then one would be > preferred ... > > So, as an example, let's say we have a single server in: US, Panama (my > servers), EU, Australia ... at a minimum, you'd want something like: > > Australia <-> US <-> EU > ^ > | > v > Panama > > To improve redundancy, you could add links between Australia <-> EU, AU > <-> PA > and PA <-> EU, but they wouldn't be required ... > > EU could then get a link from Africa, while Australia maybe from Korean, > PA > from Brazil, etc ... > > Within US, you could break things down add add regional servers, etc, etc > ... > > A large ISP could then run their own Pyzor server, but as part of the > distributed network ... > > Its theory, and simple one at that > > - ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org > ) > Email . sc...@hu... MSN . sc...@hu... > Yahoo . yscrappy Skype: hub.org ICQ . 7615664 > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.5 (FreeBSD) > > iD8DBQFFa6li4QvfyHIvDvMRAmVNAKCB3VXTKuWA8a15XxpKHr9VmjB11wCfWbls > qDm9JHFClR8UBhg7p/CeIkM= > =EnYO > -----END PGP SIGNATURE----- > > -- Jerry Wilborn jer...@gm... |
|
From: Marc G. F. <sc...@hu...> - 2006-11-28 04:29:34
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - --On Monday, November 27, 2006 22:04:30 -0600 Jerry Wilborn <jer...@gm...> wrote: > Please forgive my ignorance of the protocol. Is there a place where I can > read about how the actual question/answer is transmitted? Are we re-inventing > the wheel? Is it possible to hook this into DNS like the other .*lists do? What do you mean by 'hook this into DNS'? Do you mean like SPF and such? Or are you thinking of something different ... As for Pyzor protocol ... no idea on whether or not this was ever published by the original developer ;( > > > On 11/27/06, Marc G. Fournier <sc...@hu...> wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > > - --On Monday, November 27, 2006 21:31:53 -0500 Robert Praetorius > <rm...@ii...> > wrote: > >> I've also done some thinking about building a >> distributed pyzor and agree that it's the obvious >> next step but don't have "ideas on how to accomplish >> a distributed pyzor relatively easily"* > > Actually, its very simple "in theory" ... all you need is to change the > communications protocol slightly to distinguish between a server or client > packet ... basically, if a client reports to the server, then that server > needs > to then report the same thing to the other servers, but "as a server" ... > when > a server reports to a server, that is the end of the line ... > > I was just thinking about it, and the best paradigm, I think, for this is > Usenet news ... each server would be a 'registered peer' off of another one > ... > > Let's say, for instance, that we have a server in NA at Site A ... if I wanted > to add a pyzor server onto that system, I would talk to the admin @ Site A > about connecting to him ... connection would be a simple as a servers.txt file > that contains the hostname of the remote server ... > > When Site A receives a report, it auto gets forwarded down to its 'children' > ... if my site receives a report, I would forward it up to Site A ... > > Now, Site A knows that it received the report from me, so wouldn't send it > back > to me ... > > Now, in theory, each server would only need max two connections ... an > upstream > and a downstream, but for redundancy, and speed, more then one would be > preferred ... > > So, as an example, let's say we have a single server in: US, Panama (my > servers), EU, Australia ... at a minimum, you'd want something like: > > Australia <-> US <-> EU > ^ > | > v > Panama > > To improve redundancy, you could add links between Australia <-> EU, AU <-> PA > and PA <-> EU, but they wouldn't be required ... > > EU could then get a link from Africa, while Australia maybe from Korean, PA > from Brazil, etc ... > > Within US, you could break things down add add regional servers, etc, etc ... > > A large ISP could then run their own Pyzor server, but as part of the > distributed network ... > > Its theory, and simple one at that > > - ---- > Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) > Email . sc...@hu... MSN . sc...@hu... > Yahoo . yscrappy Skype: hub.org ICQ . 7615664 > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.5 (FreeBSD) > > iD8DBQFFa6li4QvfyHIvDvMRAmVNAKCB3VXTKuWA8a15XxpKHr9VmjB11wCfWbls > qDm9JHFClR8UBhg7p/CeIkM= > =EnYO > -----END PGP SIGNATURE----- > > > > > > -- > Jerry Wilborn > jer...@gm... - ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email . sc...@hu... MSN . sc...@hu... Yahoo . yscrappy Skype: hub.org ICQ . 7615664 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (FreeBSD) iD8DBQFFa7sn4QvfyHIvDvMRAo/rAJ46wPrlPAO17t9GTDlI37tanYJdZQCgg2gN EXtsj+9PTlIYIhydrzF8iyQ= =Z7rK -----END PGP SIGNATURE----- |
|
From: Jerry W. <jer...@gm...> - 2006-11-28 05:10:46
|
Talk about a can of worms!
Does it really need a full client/server implementation? I'm new here, so
I'm ignorant to any discussion of why that method may have been selected to
start. I can say that the only things I've seen that can stand up to the
hundreds of queries/second that worldwide lists punish servers with are
'rbldnsd' and 'rbldns'. With a message hash it feels a bit like 'square
peg/round hole' (hashes and daemons built to do reverse DNS). You guys have
experience stress testing tinydns? =)
To ask the shorter question, can we setup 'hash.pyzor.something.ext' as a
host, and then have a TXT record with the response?
Here's what I can tell from just kinda 10,000' overviewing it, please
interject any information you may have. The program 'pyzor' connects to the
server/port listed in ~/.pyzor/servers (can be updated with 'pyzor
discover'). If you tell it to checck ('pyzor check < /tmp/spamfile') it
will compute a hash of the message (how?) and then send a HTTPish type
command to the server via tcp/cleartext on the other end. A command looks
like this:
User: anonymous [ static ]
Time: 1164688654 [ seconds since the epoch ]
Sig: 45aed8ad7d2bd59f339c7e88c0d4e533c1c68b9c [ dunno how it's computed,
but it changes based on message content ]
Op: check [ specified via command line ]
Op-Digest: 0dfe4745346676d78cc6bc11ff71d444546c0544 [ don't now how it's
computed, 'echo -n check | md5sum' doesn't come back with this hash... it's
static ]
Thread: 26283 [ pid ?]
PV: 2.0 [ version number i guess; static, ends with \n\n]
I haven't been able to get a response, and with my python being rudementary
at best, it's going to take me a while to figure out what one looks like.
It looks like the Spamassassin parsing code
[.../Mail/SpamAssassin/Plugin/Pyzor.pm] may be more helpful than the
server.py.
On 11/27/06, Marc G. Fournier <sc...@hu...> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> - --On Monday, November 27, 2006 22:04:30 -0600 Jerry Wilborn
> <jer...@gm...> wrote:
>
> > Please forgive my ignorance of the protocol. Is there a place where I
> can
> > read about how the actual question/answer is transmitted? Are we
> re-inventing
> > the wheel? Is it possible to hook this into DNS like the other .*lists
> do?
>
> What do you mean by 'hook this into DNS'? Do you mean like SPF and
> such? Or
> are you thinking of something different ...
>
> As for Pyzor protocol ... no idea on whether or not this was ever
> published by
> the original developer ;(
>
> >
> >
> > On 11/27/06, Marc G. Fournier <sc...@hu...> wrote:
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> >
> >
> > - --On Monday, November 27, 2006 21:31:53 -0500 Robert Praetorius
> > <rm...@ii...>
> > wrote:
> >
> >> I've also done some thinking about building a
> >> distributed pyzor and agree that it's the obvious
> >> next step but don't have "ideas on how to accomplish
> >> a distributed pyzor relatively easily"*
> >
> > Actually, its very simple "in theory" ... all you need is to change the
> > communications protocol slightly to distinguish between a server or
> client
> > packet ... basically, if a client reports to the server, then that
> server
> > needs
> > to then report the same thing to the other servers, but "as a server"
> ...
> > when
> > a server reports to a server, that is the end of the line ...
> >
> > I was just thinking about it, and the best paradigm, I think, for this
> is
> > Usenet news ... each server would be a 'registered peer' off of another
> one
> > ...
> >
> > Let's say, for instance, that we have a server in NA at Site A ... if I
> wanted
> > to add a pyzor server onto that system, I would talk to the admin @ Site
> A
> > about connecting to him ... connection would be a simple as a
> servers.txt file
> > that contains the hostname of the remote server ...
> >
> > When Site A receives a report, it auto gets forwarded down to its
> 'children'
> > ... if my site receives a report, I would forward it up to Site A ...
> >
> > Now, Site A knows that it received the report from me, so wouldn't send
> it
> > back
> > to me ...
> >
> > Now, in theory, each server would only need max two connections ... an
> > upstream
> > and a downstream, but for redundancy, and speed, more then one would be
> > preferred ...
> >
> > So, as an example, let's say we have a single server in: US, Panama (my
> > servers), EU, Australia ... at a minimum, you'd want something like:
> >
> > Australia <-> US <-> EU
> > ^
> > |
> > v
> > Panama
> >
> > To improve redundancy, you could add links between Australia <-> EU, AU
> <-> PA
> > and PA <-> EU, but they wouldn't be required ...
> >
> > EU could then get a link from Africa, while Australia maybe from Korean,
> PA
> > from Brazil, etc ...
> >
> > Within US, you could break things down add add regional servers, etc,
> etc ...
> >
> > A large ISP could then run their own Pyzor server, but as part of the
> > distributed network ...
> >
> > Its theory, and simple one at that
> >
> > - ----
> > Marc G. Fournier Hub.Org Networking Services (
> http://www.hub.org)
> > Email . sc...@hu... MSN .
> sc...@hu...
> > Yahoo . yscrappy Skype: hub.org ICQ . 7615664
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.5 (FreeBSD)
> >
> > iD8DBQFFa6li4QvfyHIvDvMRAmVNAKCB3VXTKuWA8a15XxpKHr9VmjB11wCfWbls
> > qDm9JHFClR8UBhg7p/CeIkM=
> > =EnYO
> > -----END PGP SIGNATURE-----
> >
> >
> >
> >
> >
> > --
> > Jerry Wilborn
> > jer...@gm...
>
>
>
> - ----
> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org
> )
> Email . sc...@hu... MSN . sc...@hu...
> Yahoo . yscrappy Skype: hub.org ICQ . 7615664
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (FreeBSD)
>
> iD8DBQFFa7sn4QvfyHIvDvMRAo/rAJ46wPrlPAO17t9GTDlI37tanYJdZQCgg2gN
> EXtsj+9PTlIYIhydrzF8iyQ=
> =Z7rK
> -----END PGP SIGNATURE-----
>
>
--
Jerry Wilborn
jer...@gm...
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X