[Mod-pubsub-developer] RE: [pushlet] Digest Number 267

[Adam R. <Ad...@Kn...>]

Two weeks ago I wrote:
> > Just, thanks for showing us the way with HTTP/JavaScript streaming.
> > You've done the world a great service by educating people as to how
> > this technique is accomplished, and we've always admired Pushlets
> > (the project and the person) from afar.
> >=20
> > An interesting piece of history: We only discovered your earlier =
work
> > in Pushlets after we had already solidifed our API and programming
> > model.  It's an amazing coincidence that the mechanisms Pushlets and
> > mod_pubsub use are so similar -- maybe proof that great (humble) =
minds
> > think alike.  :)

Just replied:
> Well, the Karma ;-)  may have been spread even earlier (1996) when I =
was=20
> on the FoRR and later dist-obj mailing lists where great minds were=20
> diving deep into plumbing in a distributed world (from CORBA to REST =
and=20
> everything inbetween). In more earthly wording: at least I have been=20
> deeply influenced by these lists. (Sorry folks, this is some inside=20
> info). Ok, the Dutch and the Chinese both invented book-printing at =
the=20
> same time I read somewhere :-).

I thought your name sounded familiar from before Pushlets!
It's a small world after all... I spent several years in my
youth corrupting (and being corrupted by) FoRR and dist-obj.

> > And yes, we do deal with permanent HTTP connections as well.
> > As a result we can get some pretty good scale (see below).
> Ok. I guess Python will have some kind of NIO/select() mechanism.

There are two major options here.  One is the built-in asyncore class.
The other is part of the Twisted Internet package.

> > A key difference is that in mod_pubsub, a single push connection can
> > be shared by several open browser windows in the same browser =
process.
> Yes that is cool. Multiplexing is on my TODO.

Multiplexing is simple and can even be done from a completely =
single-threaded
application; take a look at the pubsub.py server in mod_pubsub for an
example of how to do this.

> > We also support a variety of non-JavaScript clients, from Python to
> > Perl to PHP, and Java and Ruby and C.  For these clients we have a
> > simpler data format that doesn't require JavaScript or HTML parsing.
> Since 1.0 Pushlets support the "format" parameter where on =
subscription=20
> the client can indicate the data encoding to be received. Supported=20
> formats are "js" (JavaScript, default), "xml" and "ser" (serialized =
Java=20
> objects). So a client may even use telnet to receive XML events.

That's really cool.  Ours is called kn_response_format and the formats
we support presently are "js" (JavaScript, default), "simple" (header
value pairs in RFC822-like format), and "flash" (Flash, only supported
in the Python version of our PubSub Server).

> > On the scalability question, the mod_pubsub Python server is
> > theoretically quite scalable, since it doesn't use separate threads
> > for different connections.  However, this has not been tested beyond
> > a few dozen. (We do note that http://www.mod-pubsub.org/blog/ now
> > regularly has between 3 and 30 sessions connected to it at a time,
> > and the server seems to handle that condition fairly well.)
> > We would love to figure out if there's any way the Pushlets and
> > mod_pubsub projects could help each other.  Any suggestions?
> I will further study mod_pubsub. A quick think: there are several =
areas=20
> where the projects encounter similar challenges, to name a few:
> - Safari (someone from Apple posted me and is now pushing the Safari=20
> developers to support JS streaming)

If Safari would support JS streaming we would be in heaven.
This one has been a pain in our sides for almost a year now.

> - WebServices (the new MS WS-event spec seems to use callbacks rather=20
> than streaming)

This is true, and it seems to be supported by Tibco and BEA as well.

> - Native Flash integration (i.e. not through JS and setParameter())
> Just like there are so many Wiki and blog implementations I've seen =
many
> variants of Pushlets (e.g. www.xmlblaster.org). What they share (at=20
> least for the publishing side) is that they operate (hack) at =
unforeseen=20
> use of HTTP, servers and browsers. This also makes it fun!

Lots of fun!!  :)

> This discussion started with permanent HTTP connections. Like I stated =

> any real-time notification using HTTP over TCP will have to deal with=20
> Sockets and Threads. Most webservers are not optimized for this type =
of=20
> usage. One can optimize thread-usage at the server side using a=20
> dedicated server like Adam described.  Multiplexing multiple=20
> subscriptions for a single client may further optimize resource usage.
> What I have implemented for a commercial project=20
> (www.rabotreasuryweb.com hence not OSS) is a combination of
> multiplexing (in applets) and stateful semi-permanent connections=20
> (Pullet mode). I observed much improved scalability in a regular=20
> Apache/Tomcat setup by playing with the waiting time on the client. My =

> plan is to merge Pushlet and Pullet into using a single (leasing)=20
> protocol that may dynamically adapt to server load.

Sounds really cool and useful, Just.

    Adam