[Mod-pubsub-developer] Status report

[Kragen S. <kra...@ai...>]

We have mod_pubsub's pubsub.py deployed at around 40 customer sites now,
running some of the core functions of our wireless network management
system.  The next release (due out this week) is slated actually to say
"mod_pubsub message server" on the server's status page, where it lists
which services are up and which are down.

We eventually ended up adopting the 'send out a ping message, then
don't do any slow processing until it comes back' strategy to coalesce
duplicate messages and keep from falling behind, but that in itself
wasn't enough --- we still had three subscribers that were basically
runnable all the time.  Linux doesn't cope real well with a load of 3,
and we had some real-timish processes on the same machine (the Web UI,
running in Apache), so we ended up coalescing the subscribing daemons
into one daemon.

We only send out the ping messages when there's some data available on
the tunnel socket, in order to avoid wasting lots of CPU deciding when
to process no messages.  This works OK.

We're serializing data with Data::Dumper when we send it across
mod_pubsub, because it was fast enough.  YAML wasn't; it was amazingly
slow.  Dumper means, though, that having post access to the
association-table topic gives you full control of the data-analysis
daemons, because you use Perl's 'eval STRING' to deserialize.  Our other
reasonably-fast choice was Storable, but I didn't want to go so far away
from human-readability.

After the initial integration pain, it has proved to be quite stable
and reliable.

We're thinking it might be nice to configure the logging to be a
little less verbose in pubsub.py.  The automatic log-file rotation is
really nice --- much nicer than any of the other services we depend on.
Thanks, Joyce!  (I think it was you who put that in.)

I'd love to get together with folks and talk about mod_pubsub and what
to do next.