[Mod-pubsub-developer] More about the mod-pubsub server rewrite...

[<mod...@li...>]

Some more notes...


1. It will incorporate Twisted 2.0 as well as the architectural
changes we envision, and will fix all memory leaks. It should fly and
scale just fine.


2. Based on feedback, we decided there's no real point in changing the
protocol in major ways.  The big architectural changes involve moving
ancillary functionality out to hooks -- like persistence and replay
and auth -- but those things have never worked properly in
Mod-pubsub before now anyway.


3. We will provide beta drops in early May to any company that
contributes to paying for the development. Please see me if you're
interested.


4. One of our users wrote:

> We rely on ordered delivery for our system. Ideally, we would like our
> system to handle each published event. But sometimes our publishers
> publish more data than we can process. Our solution has been to
> have the listener publish a "self-addressed stamped envelope" on
> the topic. The subscriber listens on the topic until the "self-addressed
> stamped envelope" is received by the listener. The listener keeps the
> most recently received payload about a particular network node.
> We would have to come up with some other solution if we could
> not rely on ordering.

To this we respond: Apparently can keep ordering -- turns out to be
less expensive to implement than we thought.


5. One of our users wrote:

> I'd like to explore a use case to see if the new server architecture might
> be able to support it. 
>  
> We have a large number of devices in the field. We need to know when a
> device goes down or loses network connectivity.  We currently do this using
> an HTTP heartbeat mechanism where the device calls a heartbeat page using
> HTTP GET on a regular interval and passes in its device id.  If a device
> doesn't check in on the expected interval (or 2) then we suspect that the
> device is down or has lost network connectivity.  This is a simple mechanism
> and works fine, but is a headache when you need to scale up to lots of
> devices. You end up scaling up the server farm or stretching out the
> heartbeat interval (or both). 
>  
> What if we could use the pubsub server to manage connectivity monitoring... 
> We are going to have a connection to the pubsub server from every device
> anyway. If we could write ConnectionOpen and ConnectionClose hooks
> to notice when devices come and go then we could piggyback on
> mod-pubsub (or repubsub or rerepubsub or whatever it is going to be
> called ;-) to get our connectivity monitoring for free. On the surface,
> based on what I know, it seems reasonable; the only possible catch
> I can see is how do we identify the particular device/client on
> connection open and close. Journal path? 
>  
> What do you think? Should this be doable with the new server?   

To which we responded:The ConnectionOpen and ConnectionClose
hooks are already in the plan.

As for identifying the particular device... you can't just send the
device ID?

Actually, this is a brilliant use-case for Mod-pubsub and would
be helpful to a lot of people.  You can imagine the cool things
you could do... realtime monitoring, but also easily sending an
SMS or email or IM to notify people that a connection had
gone down.  W00t!

To which our user responded:

> I could... and that would work on connect, but not on disconnect.
> If a device is crashing or suddenly loses network connectivity, it
> doesn't have a chance to send a message. We need to somehow
> associate a connection with the device id so that when the
> connection breaks we know which device it was. It seems that
> passing a device id with the connection establishment would be
> the cleanest and most symmetrical. When the connection is
> opened, we could make the association between the device id
> and the connection. When the connection closes we would need
> to identify which connection closed and determine which device
> was associated with that connection. The trick is somehow
> injecting the device id in the tunnel establishment. 

To which we respond: it will be done, as requested, as part of the new
architecture. This just keeps getting better...

Can I get another amen,
  Adam


----
ifindkarma (ahem) at gmail yadda yadda yadda dot booya com w00t w00t 

Gary, this is a dangerous mission. If you happen to get captured,
suicide may be the more humane option. Here take this.
[hands Gary a hammer] 
   -- Team America: World Police