[Mod-pubsub-developer] New release (0.5). Plus, /kn, sitewatch, pubsub.py

[Adam R. <Ad...@Kn...>]

I just uploaded the 0.5 tarball to:
   =
https://sourceforge.net/project/showfiles.php?group_id=3D66293&release_id=
=3D139123

Changes:

   1. Made /kn style work with all of the kn_apps
   2. Added "sitewatch" application in kn_apps, including sensor
   3. Some smaller modifications


Also, I wrote:
> There's no KNN anymore.  We should get modpubsub.com up and
> running soon, to serve as the *new* KN Network.  (Yes, with a weblog,
> too.  Joyce just loves it when I volunteer her for things. :)

Mike responded:
> Would you like to use xmlrouter.net? I also have topiczero.com &
> some others...

Nifty!  I don't see a reason why we shouldn't run multiple sites.
Perhaps even bridge them... thoughts?

Also, I wrote:
> For example, do_method=3Dwhoami was added to pubsub.py ... of course,
> when running pubsub.py I see that [pubsub.py] does not, at present,
> work.  Will need to go through it carefully to determine why.

Kragen responded:
> Last time I tried pubsub.py, it didn't work with any apps because the
> apps expect to find the event router at "../../cgi-bin/pubsub.cgi",
> while pubsub.py puts it at "/kn", the same place KnowNow's product
> does.  I've begun making the necessary changes to the apps and the
> Perl server to use "/kn" instead, but haven't finished.

By "haven't finished", you mean that apps can't find the auxiliary cgi's
and docs when they are set up to view /kn ?

Yesterday we modified all the kn_apps to use the /kn convention instead
of the ../../cgi-bin/pubsub.cgi convention, and they seem to work fine
with the (ever)"unfinished" Perl server, so I checked the changes into =
CVS.

I also checked in a new app called "sitewatch" which basically is just
a web server monitoring app.  The UI is neat (lifted from a previous =
dhtml
app we wrote internally for Xander's traffic monitor), but the sensor
is the reason we embarked on a new app.

The goal for the sitewatch app was to look into what it would take to
write an all-asynchronous Python networking app.  I found another =
project
on Sourceforge that hadn't been touched for years called asynchttp, =
which
provides some nice facilities for making asynchronous HTTP/1.1 calls.
Nice little library.

We wrote some other helper Classes -- asyncgeturl (asynchronous URL
GET), scheduler (asynchronous scheduler), and wakeup (a wakeup timer) --
and put them together to create sitewatch_sensor.py .  All of these
programs, along with the presentation logic in index.html and the
asynchttp.py code [it's Python-licensed], have been checked into CVS
in the mod_pubsub/kn_apps/sitewatch directory.  Feel free to check it =
out.

(Side note: sitewatch could also offer the beginning of a "webmaster
dashboard", especially if one were to add Fred's Apache log sensor
and feed it into one of Scott's ActivePanel front ends (available
in kn_apps/jscomponents ), as a second part of sitewatch.)

Now that we have a little experience building asynchronous networked
event loops in Python, what's next?  Well, we have a scheduler, an
async http library, and a wakeup timer, so the steps to building a
Python pubsub client library are:

      1. Implement a tunnel packet parser.
      2. Implement a simple response format decoder.
      3. Implement asynchronous http post, which includes request queue.
      4. Implement publish.
      5. Implement subscribe, which includes dispatcher.
      6. Implement unsubscribe, which is a recast of subscribe.
      7. Implement connect and disconnect, to robustify subscribe.

If we do it right, rather than struggle with asynchronous networking
Python primitives, we can focus on building apps using publish,
subscribe, and unsubscribe.  For example, the "sitewatch" app would
have been much easier to write if we had that.  (Then we could have
sitewatch "subscribe" to PubSub servers instead of pinging them with
a noop, to determine their status by publishing to the same topic
sitewatch would be subscribing to...)

The goal is a simple Python client for use in event-driven programs
that can publish and subscribe in (more or less) the same way the
PubSub JavaScript Library does.

Phil Harris' kn_python was written 18 months ago, and has much of the
right interface -- it's available at http://cvs.developer.knownow.com/ .

Two ways in which a newly written Python client library would differ
from kn_python:

       1. The interface is URL based.=20
       2. All operations are in a single thread and nonblocking,
          using the already-modified asyncore in the python_pubsub
          distribution.

Kragen also wrote:
> IIRC, pubsub_test.cgi showed that pubsub.py more or less worked,
> except for some features without which nearly all apps should still
> work (off-host routes, content-transform, content-filter, and event
> replay, IIRC), but it doesn't test the all-important JavaScript tunnel
> format at all.  It appears to me that the sending code for that format
> is in &PubSub::Topic::send, and the receiving code is split between
> the JavaScript parser in a browser and _kn_sendCallback in
> pubsub_raw.js.  Inspection of the code doesn't suggest that the
> formats have changed much --- except for nameU and valueU --- since I
> wrote pubsub.py in early 2001.  I'll try it out soon, but not tonight.

Ah, this explains why I thought it wasn't working -- I can't get it to
vend the (dhtml) kn_apps, so I figured pubsub.py was broken.

You looked in the right places for the JavaScript tunnel format.  Those
are, AFAIK, the reference implementation.

Ben says that actually, nameU and valueU are optional for ASCII data,
so yeah, formats have not changed really for more than two years.
This is actually a good thing; rather than things having gotten worse,
they stayed the same.  "Better" is a nice-to-have.  :)

   Adam