mod-pubsub-developer — Developer list for mod_pubsub

[Mod-pubsub-developer] mod_pubsub v0.993 is available

[Adam R. <Ad...@Kn...>]

We put together this tarball to include several
recent patches to parts of the distribution, and
add a useful app for webdevs, including:                     =20

A. Joyce's addition of topicsubtool.php found in=20
the directory mod_pubsub/webdev_toolkit .

B. A patch to the ServerSaver to hammer the CPU
less in mod_pubsub/python_pubsub/pubsub.py .

C. Patches to mod_pubsub/python_pubsub/apps/repeater.py         =20
to make it more useful.

Download v0.993 of mod_pubsub at:=20

http://sourceforge.net/project/showfiles.php?group_id=3D66293&release_id=3D=
192963

Re: [Mod-pubsub-developer] using mod_pubsub in the middle of a production app

[Kragen S. <kra...@ai...>]

On Wed, Oct 22, 2003 at 05:42:01PM -0700, Joyce Park wrote:
> Wow Kragen!  Great doc!  Can we put it in the wiki?  :-)  JP

Sure!  Hope it's helpful.

Re: [Mod-pubsub-developer] using mod_pubsub in the middle of a production app

[Joyce P. <tru...@ya...>]

Wow Kragen!  Great doc!  Can we put it in the wiki?  :-)  JP


--- Kragen Sitaker <kra...@ai...> wrote:
> So we've been slowly increasing our use of mod_pubsub in our product,
> and I thought I'd share my experience with all of you.
> 
> First, the good news: we're mostly pretty happy with the improvements
> in performance and maintainability that are coming from restructuring
> most of our IPC to use mod_pubsub.  It's easy to integrate.
> 
> A note about YAML: we tried using YAML to serialize Perl data structures,
> but it was amazingly slow.  A small hash of 1000 entries took 0.1 seconds
> to serialize and deserialize.  Now we're using Data::Dumper to serialize,
> and Perl eval to deserialize, which is about 100 times as fast.
> 
> We're using the Perl microserver.
> 
> One of the first problems we hit was that the microserver's 'subscribe'
> call wasn't persistent enough for us.  Normally we start pubsub.py
> (the python_pubsub server) at the same time as all the other daemons.
> Sometimes it takes longer to initialize than the other daemons, so
> they start trying to subscribe too early, and get 'connection refused'.
> So we added a 'subscribe_blocking' method that retries every few seconds
> for a couple of minutes.
> 
> Another one is that the microserver's PubSub::Tunnel::parse_events
> function is deeply recursive, if you're processing a large batch of
> events, and at least Perl 5.8 seems to use a lot of memory in this case.
> (Probably because each stack frame has its own local copy of the remaining
> data in the network buffer.)  We fixed this by breaking it apart into
> parse_event, which returns zero or one events, and parse_events, which
> is a loop around parse_event.
> 
> A minor problem is that PubSub::Client::handle_events doesn't have any way
> to block until there are events, so we settle for calling it frequently.
> It's pretty efficient when there aren't any events.
> 
> One of the reasons for wanting to run things in an event-driven fashion
> is to decouple the different parts of your software.  For example,
> our software monitors network nodes and periodically collects some data
> from them, which several programs analyze as they get it.  It's nice for
> the data collector to have a separate set of resources, and not share
> failure modes, with the data analysis programs.  So the data collector
> posts its data to a topic as it reads it, and the data analyzer subscribes
> to the topic and analyzes data as it can, which is often slower than
> the collector can collect it.
> 
> This causes some big problems with pubsub.py.  As soon as an event is
> posted, it appends an encoded copy of the event to the outbound network
> buffers of all the subscribing tunnels, then tries to flush them.
> When the client sucking events down the tunnel isn't keeping up with
> the event stream, then eventually the network buffers associated with
> the client's socket and the socket on pubsub.py's side fill up, and
> pubsub.py starts getting EAGAIN errors.
> 
> It handles these temporary failures "correctly" in that it keeps
> the data buffered and retries later when it can, but this creates a
> secondary problem.  The outgoing data buffer can get very large; I've
> observed over 100MB.  Events in the outgoing data buffer don't expire,
> and don't get updated by more recent events; this happens by design,
> although I can't remember why.  But eventually the unbounded growth of
> this buffer starts paging everything else out to disk.
> 
> It's even worse than that, actually, because the buffer is just a Python
> string.  So when we append to it, we create a new string, copy the old
> contents into it, and append the new data.  This happens frequently
> enough that it becomes a performance bottleneck.
> 
> A further problem is that falling behind like this means that you're
> always processing outdated data.  This is a problem for a real-time
> network monitoring system, where late answers are often wrong answers.
> 
> We've adopted the solution of closing connections that get too much
> outgoing data buffered for them, by the simple expedient of raising an
> error in Connection.send if there's more than 50M of data already in
> the outbuffer before we append to it (by calling our superclass's send).
> (This could, of course, cause problems with large events, but as mentioned
> before, we don't handle those very well anyway.)  Our subscribers see the
> EOF on the tunnel and die, and then get restarted externally.  The error
> gets logged in pubsub.err.log and also in the individual client's log.
> 
> This solution prevents the slow client from rendering the entire machine
> unusably slow, but it still produces late answers.  So we have adopted a
> client-side solution to improve matters further.  Instead of analyzing
> each event as it arrives, we simply deserialize the events and store
> them in a hash (indexed by network node ID), and periodically analyze the
> entire hash --- the most recent data for each network node.  Events still
> pile up while we're analyzing, but the analyze time is bounded by the
> number of network nodes, so we keep up if we analyze rarely enough.
> 
> So the question remains: how do we analyze rarely enough?  We initially
> tried calling PubSub::Client::handle_events, then analyzing, and
> that worked up to a point.  It would read all the available events,
> then analyze them.  The trouble was, we could read the events into the
> hash fast enough that we could empty out our 128K of socket buffers
> and return from get_events before pubsub.py had a chance to get out of
> its select() and send us the other 4MB of data it had queued for us.
> So now we publish an event to a 'wake-up' topic, and analyze our hash
> of events when we receive that event back.  We don't publish that event
> again until we've gotten it back, so we know that when we receive it,
> we've received all the events that got sent to us during our previous
> analysis phase.
> 
> This seems to work reasonably well, but needs some kind of throttle to
> keep us from spending all of our time sending and handling the 'wake-up'
> event.  Right now, we sleep 0.3 seconds, which puts us in the
> uncomfortable position of adjusting a polling interval to overcome
> problems in our event system.  Perhaps instead we should wait for some
> other event to show up.
> 
> I'm curious who else is doing stuff like this, what problems you've
> run into, and what solutions you've come up with.  And when is the next
> mod-pubsub user's group?

[Mod-pubsub-developer] using mod_pubsub in the middle of a production app

[Kragen S. <kra...@ai...>]

So we've been slowly increasing our use of mod_pubsub in our product,
and I thought I'd share my experience with all of you.

First, the good news: we're mostly pretty happy with the improvements
in performance and maintainability that are coming from restructuring
most of our IPC to use mod_pubsub.  It's easy to integrate.

A note about YAML: we tried using YAML to serialize Perl data structures,
but it was amazingly slow.  A small hash of 1000 entries took 0.1 seconds
to serialize and deserialize.  Now we're using Data::Dumper to serialize,
and Perl eval to deserialize, which is about 100 times as fast.

We're using the Perl microserver.

One of the first problems we hit was that the microserver's 'subscribe'
call wasn't persistent enough for us.  Normally we start pubsub.py
(the python_pubsub server) at the same time as all the other daemons.
Sometimes it takes longer to initialize than the other daemons, so
they start trying to subscribe too early, and get 'connection refused'.
So we added a 'subscribe_blocking' method that retries every few seconds
for a couple of minutes.

Another one is that the microserver's PubSub::Tunnel::parse_events
function is deeply recursive, if you're processing a large batch of
events, and at least Perl 5.8 seems to use a lot of memory in this case.
(Probably because each stack frame has its own local copy of the remaining
data in the network buffer.)  We fixed this by breaking it apart into
parse_event, which returns zero or one events, and parse_events, which
is a loop around parse_event.

A minor problem is that PubSub::Client::handle_events doesn't have any way
to block until there are events, so we settle for calling it frequently.
It's pretty efficient when there aren't any events.

One of the reasons for wanting to run things in an event-driven fashion
is to decouple the different parts of your software.  For example,
our software monitors network nodes and periodically collects some data
from them, which several programs analyze as they get it.  It's nice for
the data collector to have a separate set of resources, and not share
failure modes, with the data analysis programs.  So the data collector
posts its data to a topic as it reads it, and the data analyzer subscribes
to the topic and analyzes data as it can, which is often slower than
the collector can collect it.

This causes some big problems with pubsub.py.  As soon as an event is
posted, it appends an encoded copy of the event to the outbound network
buffers of all the subscribing tunnels, then tries to flush them.
When the client sucking events down the tunnel isn't keeping up with
the event stream, then eventually the network buffers associated with
the client's socket and the socket on pubsub.py's side fill up, and
pubsub.py starts getting EAGAIN errors.

It handles these temporary failures "correctly" in that it keeps
the data buffered and retries later when it can, but this creates a
secondary problem.  The outgoing data buffer can get very large; I've
observed over 100MB.  Events in the outgoing data buffer don't expire,
and don't get updated by more recent events; this happens by design,
although I can't remember why.  But eventually the unbounded growth of
this buffer starts paging everything else out to disk.

It's even worse than that, actually, because the buffer is just a Python
string.  So when we append to it, we create a new string, copy the old
contents into it, and append the new data.  This happens frequently
enough that it becomes a performance bottleneck.

A further problem is that falling behind like this means that you're
always processing outdated data.  This is a problem for a real-time
network monitoring system, where late answers are often wrong answers.

We've adopted the solution of closing connections that get too much
outgoing data buffered for them, by the simple expedient of raising an
error in Connection.send if there's more than 50M of data already in
the outbuffer before we append to it (by calling our superclass's send).
(This could, of course, cause problems with large events, but as mentioned
before, we don't handle those very well anyway.)  Our subscribers see the
EOF on the tunnel and die, and then get restarted externally.  The error
gets logged in pubsub.err.log and also in the individual client's log.

This solution prevents the slow client from rendering the entire machine
unusably slow, but it still produces late answers.  So we have adopted a
client-side solution to improve matters further.  Instead of analyzing
each event as it arrives, we simply deserialize the events and store
them in a hash (indexed by network node ID), and periodically analyze the
entire hash --- the most recent data for each network node.  Events still
pile up while we're analyzing, but the analyze time is bounded by the
number of network nodes, so we keep up if we analyze rarely enough.

So the question remains: how do we analyze rarely enough?  We initially
tried calling PubSub::Client::handle_events, then analyzing, and
that worked up to a point.  It would read all the available events,
then analyze them.  The trouble was, we could read the events into the
hash fast enough that we could empty out our 128K of socket buffers
and return from get_events before pubsub.py had a chance to get out of
its select() and send us the other 4MB of data it had queued for us.
So now we publish an event to a 'wake-up' topic, and analyze our hash
of events when we receive that event back.  We don't publish that event
again until we've gotten it back, so we know that when we receive it,
we've received all the events that got sent to us during our previous
analysis phase.

This seems to work reasonably well, but needs some kind of throttle to
keep us from spending all of our time sending and handling the 'wake-up'
event.  Right now, we sleep 0.3 seconds, which puts us in the
uncomfortable position of adjusting a polling interval to overcome
problems in our event system.  Perhaps instead we should wait for some
other event to show up.

I'm curious who else is doing stuff like this, what problems you've
run into, and what solutions you've come up with.  And when is the next
mod-pubsub user's group?

RE: [Mod-pubsub-developer] Notification events for google searches - searchalert.net

[Adam R. <Ad...@Kn...>]

I believe the

content-type

header does the trick... this looks like nice work, Mike!

   Adam


-----Original Message-----
From: Asynch Messaging [mailto:asy...@ho...]
Sent: Saturday, October 18, 2003 11:51 AM
To: mod...@li...
Subject: Re: [Mod-pubsub-developer] Notification events for google
searches - searchalert.net


I added support for BloggerAPI last night. So www.searchalert.net =
supports
sending MetaweblogAPI, BloggerAPI, plain text and RSS 0.91.
Weird thing - BloggerAPI doesn't have a slot for 'title' of the post. =
Lots
of inane excuses from Pyra why they don't support MetaweblogAPI. Dude - =
its
just text....

Next step - point a bunch of search results at a mod-pubsub hosted topic =
&
figure out what text format to send for a list of links with titles. NOt
sure if XML is appropriate for the level of clients typically connected =
- is
there a standard header to indicate content-type of a message?


----- Original Message -----=20
From: "Asynch Messaging" <asy...@ho...>
To: "Joyce Park" <tru...@ya...>;
<mod...@li...>
Sent: Friday, October 17, 2003 10:48 PM
Subject: Re: [Mod-pubsub-developer] Notification events for google
searches - searchalert.net


> The Web UI captures search strings into MySQL.
> A background task processes each search and does a request to Google =
(for
> both web searches and news searches). The response is scanned for =
links.
> Each link is added to the DB (if it doesn't already exist) and =
associated
> with the search - each link is part of a 'collection' of results.
>
> Each user is 'subscribed' to the results of the search. If the results
> change (links are added), then an event is sent to the destination =
that is
> subscribed (either email or web URL). The event is formatted first, =
via a
> hacked up template language.
>
> Essentially, searchalert.net manages subscriptions between two Web
resources
> & provides formatting in between.
> For example, the following resource is 'results for the search for
> mod-pubsub'
> http://www.searchalert.net/searchalert/results.jsp?queryId=3D685
>
> I have routed changes for this resource (such as a link being added) =
to
the
> following resource (which is a blog)
> http://www.topiczero.com/mt/mt-xmlrpc.cgi
>
> (The annoying thing is that the destination (the blog) cannot be =
specified
> via a URL. I had to hack in a 'blogId' into the content of the message
when
> all I really wanted was to append a list of links to a known blog. =
Fookin
> xml-rpc dunderheads.)
>
> The message format uses this template:
> <?xml version=3D"1.0"?>
> <methodCall>
>  <methodName>metaWeblog.newPost</methodName>
>  <params>
>   <param><value><sa.destination.extra1></value></param>
>   <param><value><sa.destination.username></value></param>
>   <param><value><sa.destination.password></value></param>
>   <param>
>    <value>
>     <struct>
>      <member><name>title</name><value>New links for
> <sa.query></value></member>
>      <member><name>description</name><value>
> <sa.results>
> <a href=3D'<result.link>'><result.title></a><br />
> </sa.results>
> </value></member>
>     </struct>
>    </value>
>   </param>
>   <param><value>true</value></param>
>  </params>
> </methodCall>
>
>
>
>
>
>
> ----- Original Message -----=20
> From: "Joyce Park" <tru...@ya...>
> To: "Asynch Messaging" <asy...@ho...>;
> <mod...@li...>
> Sent: Friday, October 17, 2003 9:49 PM
> Subject: Re: [Mod-pubsub-developer] Notification events for google
> searches - searchalert.net
>
>
> > Cool Mike!  It craps out if you neglect the "www" in
www.searchalert.net,
> but
> > otherwise it looks good.  How does it work?  JP
> >
> >
> > --- Asynch Messaging <asy...@ho...> wrote:
> > > I've updated my searchalert.net app to support posting to a Web
> destination
> > > in addition to an email destination.
> > > Currently it supports three formats, but adding another one is =
pretty
> > > trivial - I just need to create a template.
> > > The format supported are WeblogAPI, RSS0.91 and plain text.
> > >
> > > If you want to hook up an event source for results from google
searches,
> you
> > > can use www.searchalert.net and use the 'advanced' subscribe =
button,
> then
> > > create a new destination & subscribe to a search.
> > >
> > > Here's an example blog with search results for 'amazon web =
service'
(it
> also
> > > receives results for 'knownow')
> > > http://www.topiczero.com/blogs/
> > >
> > > Let me know if you try to use it & have any problems.
> >
>
>
> -------------------------------------------------------
> This SF.net email sponsored by: Enterprise Linux Forum Conference & =
Expo
> The Event For Linux Datacenter Solutions & Strategies in The =
Enterprise
> Linux in the Boardroom; in the Front Office; & in the Server Room
> http://www.enterpriselinuxforum.com
> _______________________________________________
> Mod-pubsub-developer mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-pubsub-developer
>


-------------------------------------------------------
This SF.net email sponsored by: Enterprise Linux Forum Conference & Expo
The Event For Linux Datacenter Solutions & Strategies in The Enterprise=20
Linux in the Boardroom; in the Front Office; & in the Server Room=20
http://www.enterpriselinuxforum.com
_______________________________________________
Mod-pubsub-developer mailing list
Mod...@li...
https://lists.sourceforge.net/lists/listinfo/mod-pubsub-developer

Re: [Mod-pubsub-developer] Notification events for google searches - searchalert.net

[Asynch M. <asy...@ho...>]

I added support for BloggerAPI last night. So www.searchalert.net supports
sending MetaweblogAPI, BloggerAPI, plain text and RSS 0.91.
Weird thing - BloggerAPI doesn't have a slot for 'title' of the post. Lots
of inane excuses from Pyra why they don't support MetaweblogAPI. Dude - its
just text....

Next step - point a bunch of search results at a mod-pubsub hosted topic &
figure out what text format to send for a list of links with titles. NOt
sure if XML is appropriate for the level of clients typically connected - is
there a standard header to indicate content-type of a message?


----- Original Message ----- 
From: "Asynch Messaging" <asy...@ho...>
To: "Joyce Park" <tru...@ya...>;
<mod...@li...>
Sent: Friday, October 17, 2003 10:48 PM
Subject: Re: [Mod-pubsub-developer] Notification events for google
searches - searchalert.net


> The Web UI captures search strings into MySQL.
> A background task processes each search and does a request to Google (for
> both web searches and news searches). The response is scanned for links.
> Each link is added to the DB (if it doesn't already exist) and associated
> with the search - each link is part of a 'collection' of results.
>
> Each user is 'subscribed' to the results of the search. If the results
> change (links are added), then an event is sent to the destination that is
> subscribed (either email or web URL). The event is formatted first, via a
> hacked up template language.
>
> Essentially, searchalert.net manages subscriptions between two Web
resources
> & provides formatting in between.
> For example, the following resource is 'results for the search for
> mod-pubsub'
> http://www.searchalert.net/searchalert/results.jsp?queryId=685
>
> I have routed changes for this resource (such as a link being added) to
the
> following resource (which is a blog)
> http://www.topiczero.com/mt/mt-xmlrpc.cgi
>
> (The annoying thing is that the destination (the blog) cannot be specified
> via a URL. I had to hack in a 'blogId' into the content of the message
when
> all I really wanted was to append a list of links to a known blog. Fookin
> xml-rpc dunderheads.)
>
> The message format uses this template:
> <?xml version="1.0"?>
> <methodCall>
>  <methodName>metaWeblog.newPost</methodName>
>  <params>
>   <param><value><sa.destination.extra1></value></param>
>   <param><value><sa.destination.username></value></param>
>   <param><value><sa.destination.password></value></param>
>   <param>
>    <value>
>     <struct>
>      <member><name>title</name><value>New links for
> <sa.query></value></member>
>      <member><name>description</name><value>
> <sa.results>
> <a href='<result.link>'><result.title></a><br />
> </sa.results>
> </value></member>
>     </struct>
>    </value>
>   </param>
>   <param><value>true</value></param>
>  </params>
> </methodCall>
>
>
>
>
>
>
> ----- Original Message ----- 
> From: "Joyce Park" <tru...@ya...>
> To: "Asynch Messaging" <asy...@ho...>;
> <mod...@li...>
> Sent: Friday, October 17, 2003 9:49 PM
> Subject: Re: [Mod-pubsub-developer] Notification events for google
> searches - searchalert.net
>
>
> > Cool Mike!  It craps out if you neglect the "www" in
www.searchalert.net,
> but
> > otherwise it looks good.  How does it work?  JP
> >
> >
> > --- Asynch Messaging <asy...@ho...> wrote:
> > > I've updated my searchalert.net app to support posting to a Web
> destination
> > > in addition to an email destination.
> > > Currently it supports three formats, but adding another one is pretty
> > > trivial - I just need to create a template.
> > > The format supported are WeblogAPI, RSS0.91 and plain text.
> > >
> > > If you want to hook up an event source for results from google
searches,
> you
> > > can use www.searchalert.net and use the 'advanced' subscribe button,
> then
> > > create a new destination & subscribe to a search.
> > >
> > > Here's an example blog with search results for 'amazon web service'
(it
> also
> > > receives results for 'knownow')
> > > http://www.topiczero.com/blogs/
> > >
> > > Let me know if you try to use it & have any problems.
> >
>
>
> -------------------------------------------------------
> This SF.net email sponsored by: Enterprise Linux Forum Conference & Expo
> The Event For Linux Datacenter Solutions & Strategies in The Enterprise
> Linux in the Boardroom; in the Front Office; & in the Server Room
> http://www.enterpriselinuxforum.com
> _______________________________________________
> Mod-pubsub-developer mailing list
> Mod...@li...
> https://lists.sourceforge.net/lists/listinfo/mod-pubsub-developer
>

Re: [Mod-pubsub-developer] Notification events for google searches - searchalert.net

[Asynch M. <asy...@ho...>]

The Web UI captures search strings into MySQL.
A background task processes each search and does a request to Google (for
both web searches and news searches). The response is scanned for links.
Each link is added to the DB (if it doesn't already exist) and associated
with the search - each link is part of a 'collection' of results.

Each user is 'subscribed' to the results of the search. If the results
change (links are added), then an event is sent to the destination that is
subscribed (either email or web URL). The event is formatted first, via a
hacked up template language.

Essentially, searchalert.net manages subscriptions between two Web resources
& provides formatting in between.
For example, the following resource is 'results for the search for
mod-pubsub'
http://www.searchalert.net/searchalert/results.jsp?queryId=685

I have routed changes for this resource (such as a link being added) to the
following resource (which is a blog)
http://www.topiczero.com/mt/mt-xmlrpc.cgi

(The annoying thing is that the destination (the blog) cannot be specified
via a URL. I had to hack in a 'blogId' into the content of the message when
all I really wanted was to append a list of links to a known blog. Fookin
xml-rpc dunderheads.)

The message format uses this template:
<?xml version="1.0"?>
<methodCall>
 <methodName>metaWeblog.newPost</methodName>
 <params>
  <param><value><sa.destination.extra1></value></param>
  <param><value><sa.destination.username></value></param>
  <param><value><sa.destination.password></value></param>
  <param>
   <value>
    <struct>
     <member><name>title</name><value>New links for
<sa.query></value></member>
     <member><name>description</name><value>
<sa.results>
<a href='<result.link>'><result.title></a><br />
</sa.results>
</value></member>
    </struct>
   </value>
  </param>
  <param><value>true</value></param>
 </params>
</methodCall>






----- Original Message ----- 
From: "Joyce Park" <tru...@ya...>
To: "Asynch Messaging" <asy...@ho...>;
<mod...@li...>
Sent: Friday, October 17, 2003 9:49 PM
Subject: Re: [Mod-pubsub-developer] Notification events for google
searches - searchalert.net


> Cool Mike!  It craps out if you neglect the "www" in www.searchalert.net,
but
> otherwise it looks good.  How does it work?  JP
>
>
> --- Asynch Messaging <asy...@ho...> wrote:
> > I've updated my searchalert.net app to support posting to a Web
destination
> > in addition to an email destination.
> > Currently it supports three formats, but adding another one is pretty
> > trivial - I just need to create a template.
> > The format supported are WeblogAPI, RSS0.91 and plain text.
> >
> > If you want to hook up an event source for results from google searches,
you
> > can use www.searchalert.net and use the 'advanced' subscribe button,
then
> > create a new destination & subscribe to a search.
> >
> > Here's an example blog with search results for 'amazon web service' (it
also
> > receives results for 'knownow')
> > http://www.topiczero.com/blogs/
> >
> > Let me know if you try to use it & have any problems.
>

Re: [Mod-pubsub-developer] Notification events for google searches - searchalert.net

[Joyce P. <tru...@ya...>]

Cool Mike!  It craps out if you neglect the "www" in www.searchalert.net, but
otherwise it looks good.  How does it work?  JP


--- Asynch Messaging <asy...@ho...> wrote:
> I've updated my searchalert.net app to support posting to a Web destination
> in addition to an email destination.
> Currently it supports three formats, but adding another one is pretty
> trivial - I just need to create a template.
> The format supported are WeblogAPI, RSS0.91 and plain text.
> 
> If you want to hook up an event source for results from google searches, you
> can use www.searchalert.net and use the 'advanced' subscribe button, then
> create a new destination & subscribe to a search.
> 
> Here's an example blog with search results for 'amazon web service' (it also
> receives results for 'knownow')
> http://www.topiczero.com/blogs/
> 
> Let me know if you try to use it & have any problems.

[Mod-pubsub-developer] Notification events for google searches - searchalert.net

[Asynch M. <asy...@ho...>]

I've updated my searchalert.net app to support posting to a Web destination
in addition to an email destination.
Currently it supports three formats, but adding another one is pretty
trivial - I just need to create a template.
The format supported are WeblogAPI, RSS0.91 and plain text.

If you want to hook up an event source for results from google searches, you
can use www.searchalert.net and use the 'advanced' subscribe button, then
create a new destination & subscribe to a search.

Here's an example blog with search results for 'amazon web service' (it also
receives results for 'knownow')
http://www.topiczero.com/blogs/

Let me know if you try to use it & have any problems.

Re: [Mod-pubsub-developer] Getting started?

[Joyce P. <tru...@ya...>]

Hi Jens!  Sorry, we don't have a lot of stuff on architecture in the sense I
think you mean it -- although maybe we should.  The main thing to remember is
that a mod-pubsub client needs to live next to any application that will either
publish or subscribe to the mod-pubsub server -- sometimes this confuses people
who are used to a certain client-server relationship.  What else is there that
would be helpful to know?

Numerous individuals have in fact written demo apps that do not work in the
browser.  Look in the directories for Java, Perl, and Python in particular.
  
  -- Joyce Park


--- Jens Alfke <je...@ma...> wrote:
> I'm trying to learn my way around the pubsub protocol, and having some 
> difficulty. I'm interested in the protocol itself, and in using it in 
> an application, i.e. not in a web browser. The main protocol document
> 	http://www.mod-pubsub.org/kn_docs/pubsub_protocol.html
> seems to assume the reader already knows how the architecture works, 
> and launches directly into the minutae of syntax and schema. Other 
> documents seem to be aimed only at using the JavaScript library from a 
> web application.
> 
> Is there some kind of document that describes the overall design of the 
> protocol, that I could peruse first?
> 
> --Jens

[Mod-pubsub-developer] RE: Differences between KnowNow commercial and mod-pubsub

[Adam R. <Ad...@Kn...>]

Yeah, I did mean to say that.  Thanks, Joyce.

I think I'll add this to my Wiki page so I can cut-n-paste it
whenever anyone asks me this question again.  :)

   http://www.mod-pubsub.org/moinmoin/AdamRifkin

   Adam


-----Original Message-----
From: Joyce Park [mailto:tru...@ya...]
Sent: Thursday, October 16, 2003 3:42 PM
To: Adam Rifkin; Jens Alfke
Cc: mod...@li...
Subject: RE: Differences between KnowNow commercial and mod-pubsub

Jens,

What Adam actually meant to say is:

I am the co-founder of KnowNow, Inc., and one of only two Mod-pubsub =
developers
who is still employed by that entity.  Due to the conditions of my =
continued
employment, there are things I can't/must say and do in public -- even =
in
inappropriate settings like the mod-pubsub mailing list.

KnowNow, Inc. realizes revenue by selling software to customers with =
enterprise
needs.  These needs are not necessarily technical in nature -- but =
rather may
have to do with perceived or theoretical increases in scalability, =
performance,
ease of integration, etc..  Design decisions affecting KnowNow products =
may
have been made for a host of commercial and personal reasons having =
little to
do with the technical needs of most users.  In any case, it may be in =
KnowNow's
commercial interest to sell more servers and ancillary goods and =
services.

Therefore, anything that I -- Adam Rifkin -- say about KnowNow on the
mod-pubsub mailing list should be filtered through your knowledge of =
these
pertinent facts.  Nothing anyone else says on this mailing list should =
be
construed as constituting an endorsement or critique of any commercial =
product.

  Yours truly,

  Adam Rifkin's personal editor


--- Adam Rifkin <Ad...@Kn...> wrote:
> Suggestion heard, loud and clear.  (By the way, welcome!)
>=20
> I think it's fair to say KnowNow is the industrial-strength commercial
> implementation; for it you can buy support, clustering, failover,
> higher-performance, and an Excel connector, plus it has documentation
> (which, as we've noted many times, we're lacking), decent installers,
> and it has been tested and used by many paying customers.
>=20
> Most of KnowNow's customers do buy it for commercial line-of-business
> applications, whereas mod-pubsub does tend to appeal more to the web
> architects and developers of things like email and news apps.
>=20
> Note that all of the mod-pubsub client libraries work with the =
commercial
> server as connectors, so if you ever need to switch over, migration
> is fairly straightforward.

RE: [Mod-pubsub-developer] Persistent storage / forwarding of events

[Adam R. <Ad...@Kn...>]

Oh, we like Ruby.  Greg Burd wrote a Ruby pubsub client library
which we distribute with mod_pubsub.

Two possible solutions to the O(n^2) problem...

1. Perhaps one solution to the explosion would be to remove events
from a topic once you've consumed them?

2. Another solution would be to keep the timestamp of the last received
event, add a fudge factor of extra time to that, and use that as the
do_max_age of the subscription.  Reason you need a fudge factor is
because time-based replay is against the event publication time,
and there is some latency in delivering events.

Do either of those work for you?

   Adam


-----Original Message-----
From: Jens Alfke [mailto:je...@ma...]
Sent: Thursday, October 16, 2003 4:33 PM
To: Adam Rifkin
Cc: mod...@li...; Ben Sittler
Subject: Re: [Mod-pubsub-developer] Persistent storage / forwarding of
events

On Oct 16, 2003, at 4:04 PM, Adam Rifkin wrote:

> I'll take a moment to put in a plug for Python.  Most of the pubsub
> apps I write that don't use JavaScript use Python.

I've been getting into Ruby lately, which is very nice, and I don't=20
want to confuse myself by learning too many languages at the same time=20
:)

> So we'd write code to do this as part of the app that is able to =
squash
> duplicates received before the last connection dropped.  Note that
> do_max_age of infinity on the subscription lets a client replay all
> the events in the topic so the app should be able to ascertain where
> it last left off.

Hm. This seems inefficient =97 I've run into the same issue with the=20
Blogger API. I have to do something like:
	"OK server, give me the last 10 events."
	(Hmm, I don't recognize any of those, better ask for more...)
	"OK server, give me the last _20_ events."
	(Ah, I already had #16-20, so now I'm caught up.)
So the server had to send me 30 events (10 of them duplicates!) just so=20
I could get the 15 new ones. And this is an O(n^2) algorithm, so it=20
gets even worse if there are even more messages to sync up.

--Jens

Re: [Mod-pubsub-developer] Persistent storage / forwarding of events

[Jens A. <je...@ma...>]

On Oct 16, 2003, at 4:04 PM, Adam Rifkin wrote:

> I'll take a moment to put in a plug for Python.  Most of the pubsub
> apps I write that don't use JavaScript use Python.

I've been getting into Ruby lately, which is very nice, and I don't=20
want to confuse myself by learning too many languages at the same time=20=

:)

> So we'd write code to do this as part of the app that is able to =
squash
> duplicates received before the last connection dropped.  Note that
> do_max_age of infinity on the subscription lets a client replay all
> the events in the topic so the app should be able to ascertain where
> it last left off.

Hm. This seems inefficient =97 I've run into the same issue with the=20
Blogger API. I have to do something like:
	"OK server, give me the last 10 events."
	(Hmm, I don't recognize any of those, better ask for more...)
	"OK server, give me the last _20_ events."
	(Ah, I already had #16-20, so now I'm caught up.)
So the server had to send me 30 events (10 of them duplicates!) just so=20=

I could get the 15 new ones. And this is an O(n^2) algorithm, so it=20
gets even worse if there are even more messages to sync up.

--Jens=

RE: [Mod-pubsub-developer] Persistent storage / forwarding of events

[Adam R. <Ad...@Kn...>]

I agree, it's nice to know the source is out there in case any of us
need to muck with it.

I'll take a moment to put in a plug for Python.  Most of the pubsub
apps I write that don't use JavaScript use Python.  It's compilable
into Java bytecodes or Windows dll's/exe's, plus it runs natively
on more than a hundred platforms.  And Python is fun.  And readable.
And... I'll shut up now.

> It sounds like I can get similar functionality using a topic=20
> that stores its events persistently. Especially since topics chain=20
> together: could I create a persistent "mailbox" topic that, then route =

> the topics I want to receive into that, and then connect my client to=20
> the mailbox periodically and slurp up the new events?

Yes.  In fact, this is what most applications we have do.

> Thanks for the examples of retrieving events =97 is there a way a =
client=20
> on reconnect can say "send me all the events since #_____, which is =
the=20
> last one I have"?

Unfortunately, no.  Because events can expire, and because the PubSub
Server does not guarantee order of delivery, such a reconnection would
have to be done in client library logic that is usually =
application-specific.
So we'd write code to do this as part of the app that is able to squash
duplicates received before the last connection dropped.  Note that
do_max_age of infinity on the subscription lets a client replay all
the events in the topic so the app should be able to ascertain where
it last left off.

   Adam


-----Original Message-----
From: Jens Alfke [mailto:je...@ma...]
Sent: Thursday, October 16, 2003 2:56 PM
To: mod...@li...
Subject: Re: [Mod-pubsub-developer] Persistent storage / forwarding of
events



On Oct 16, 2003, at 2:39 PM, Adam Rifkin wrote:

> multihops could use a store-and-forward paradigm at each of the client =

> hops,
> with servers storing as little as possible so they act more like=20
> routers and
> less like databases.  If this is something you're interested in=20
> working on,
> I say go for it.

I'm actually trying to focus on applications and not get into the=20
plumbing too much. (And, uh, I don't speak Python.) But it's nice to=20
know the source is out there to muck with if I have to :)

However, it sounds like I can get similar functionality using a topic=20
that stores its events persistently. Especially since topics chain=20
together: could I create a persistent "mailbox" topic that, then route=20
the topics I want to receive into that, and then connect my client to=20
the mailbox periodically and slurp up the new events?

Thanks for the examples of retrieving events =97 is there a way a client =

on reconnect can say "send me all the events since #_____, which is the=20
last one I have"?

--Jens


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Mod-pubsub-developer mailing list
Mod...@li...
https://lists.sourceforge.net/lists/listinfo/mod-pubsub-developer

[Mod-pubsub-developer] RE: Differences between KnowNow commercial and mod-pubsub

[Joyce P. <tru...@ya...>]

Jens,

What Adam actually meant to say is:

I am the co-founder of KnowNow, Inc., and one of only two Mod-pubsub developers
who is still employed by that entity.  Due to the conditions of my continued
employment, there are things I can't/must say and do in public -- even in
inappropriate settings like the mod-pubsub mailing list.

KnowNow, Inc. realizes revenue by selling software to customers with enterprise
needs.  These needs are not necessarily technical in nature -- but rather may
have to do with perceived or theoretical increases in scalability, performance,
ease of integration, etc..  Design decisions affecting KnowNow products may
have been made for a host of commercial and personal reasons having little to
do with the technical needs of most users.  In any case, it may be in KnowNow's
commercial interest to sell more servers and ancillary goods and services.

Therefore, anything that I -- Adam Rifkin -- say about KnowNow on the
mod-pubsub mailing list should be filtered through your knowledge of these
pertinent facts.  Nothing anyone else says on this mailing list should be
construed as constituting an endorsement or critique of any commercial product.

  Yours truly,

  Adam Rifkin's personal editor


--- Adam Rifkin <Ad...@Kn...> wrote:
> Suggestion heard, loud and clear.  (By the way, welcome!)
> 
> I think it's fair to say KnowNow is the industrial-strength commercial
> implementation; for it you can buy support, clustering, failover,
> higher-performance, and an Excel connector, plus it has documentation
> (which, as we've noted many times, we're lacking), decent installers,
> and it has been tested and used by many paying customers.
> 
> Most of KnowNow's customers do buy it for commercial line-of-business
> applications, whereas mod-pubsub does tend to appeal more to the web
> architects and developers of things like email and news apps.
> 
> Note that all of the mod-pubsub client libraries work with the commercial
> server as connectors, so if you ever need to switch over, migration
> is fairly straightforward.

Re: [Mod-pubsub-developer] Persistent storage / forwarding of events

[Jens A. <je...@ma...>]

On Oct 16, 2003, at 2:39 PM, Adam Rifkin wrote:

> multihops could use a store-and-forward paradigm at each of the client=20=

> hops,
> with servers storing as little as possible so they act more like=20
> routers and
> less like databases.  If this is something you're interested in=20
> working on,
> I say go for it.

I'm actually trying to focus on applications and not get into the=20
plumbing too much. (And, uh, I don't speak Python.) But it's nice to=20
know the source is out there to muck with if I have to :)

However, it sounds like I can get similar functionality using a topic=20
that stores its events persistently. Especially since topics chain=20
together: could I create a persistent "mailbox" topic that, then route=20=

the topics I want to receive into that, and then connect my client to=20
the mailbox periodically and slurp up the new events?

Thanks for the examples of retrieving events =97 is there a way a client=20=

on reconnect can say "send me all the events since #_____, which is the=20=

last one I have"?

--Jens=

RE: [Mod-pubsub-developer] Persistent storage / forwarding of events

[Adam R. <Ad...@Kn...>]

Hi Jens,

Your investigation sounds like you're looking at mod-pubsub for the
right kinds of things.

Answers to your questions:

> Do topics have the capability to store-and-forward events?

Not really.  It is one of the features we were thinking of adding to
the client libraries so that, for example,

  pubsub client -> pubsub server -> pubsub client -> pubsub server -> =
pubsub client

multihops could use a store-and-forward paradigm at each of the client =
hops,
with servers storing as little as possible so they act more like routers =
and
less like databases.  If this is something you're interested in working =
on,
I say go for it.=20

> If I set up a route from a source topic to my 'journal' topic, and =
then=20
> disconnect, will the journal queue up the incoming events as a POP=20
> server would, letting me reconnect later and receive the events I've=20
> received in the interim?

No.  There is partial support for this but it doesn't quite work at
the moment.  Want to work on this feature?  I hear there's plenty of
room to run for anyone who wants to work on the server... :)

> Can topics store their events persistently so they can be retrieved=20
> later by clients that connect to them?

Yes.

> For example, a topic might implement a newsgroup, and a new subscriber
> would want to be able to download the existing posts, or at least the
> last hundred or so.

Yes, this is a subscribe option using do_max_age or do_max_n; for =
example,

  kn_subscribe(conf_topic, onConfigure, {do_max_age: "infinity"});

would get all unexpired events coming out of conf_topic and send them to
the onConfigure() callback function.  Another example:

  kn.subscribe(rss_topic, rss_handleEvent, { do_max_n: 10 });

would get the last 10 events coming out of the rss_topic and send them =
to
the rss_handleEvent() callback function.

> If this isn't built-in, would it be straightforward to build a =
solution=20
> on top of mod-pubsub, or would this be a bad design choice, attempting =

> to force it to do something it just wasn't meant to?

I think it's one of the features we use in apps a lot.  I'm not sure how
good of a design choice it was, though, since it forces the PubSub =
Server
to have some database-like functionality.  There are some among us who =
would
advocate using the PubSub Server as just a router and let the database =
store
any data you might later want to go back and query for.

   Adam


-----Original Message-----
From: Jens Alfke [mailto:je...@ma...]
Sent: Thursday, October 16, 2003 9:11 AM
To: mod...@li...
Subject: [Mod-pubsub-developer] Persistent storage / forwarding of
events


First questions. I should preface this by saying that I'm mostly=20
investigating using mod-pubsub for asynchronous messaging, things like=20
email or newsgroups or blogs where a recipient may not be online at the=20
time a message is sent. So:

(1) Do topics have the capability to store-and-forward events? If I set=20
up a route from a source topic to my 'journal' topic, and then=20
disconnect, will the journal queue up the incoming events as a POP=20
server would, letting me reconnect later and receive the events I've=20
received in the interim?

(2) Can topics store their events persistently so they can be retrieved=20
later by clients that connect to them? For example, a topic might=20
implement a newsgroup, and a new subscriber would want to be able to=20
download the existing posts, or at least the last hundred or so.

If this isn't built-in, would it be straightforward to build a solution=20
on top of mod-pubsub, or would this be a bad design choice, attempting=20
to force it to do something it just wasn't meant to?

--Jens



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Mod-pubsub-developer mailing list
Mod...@li...
https://lists.sourceforge.net/lists/listinfo/mod-pubsub-developer

RE: [Mod-pubsub-developer] Getting started?

[Adam R. <Ad...@Kn...>]

Please do keep asking questions, it's the best way we know to keep =
refining
how we explain how our pieces work.  (It's not an understatement to say =
that
we're not the world's greatest documenters...)

   Adam


-----Original Message-----
From: Jens Alfke [mailto:je...@ma...]
Sent: Wednesday, October 15, 2003 9:52 PM
To: mod...@li...
Subject: [Mod-pubsub-developer] Getting started?


I'm trying to learn my way around the pubsub protocol, and having some =
difficulty. I'm interested in the protocol itself, and in using it in an =
application, i.e. not in a web browser. The main protocol document=20
http://www.mod-pubsub.org/kn_docs/pubsub_protocol.html=20
seems to assume the reader already knows how the architecture works, and =
launches directly into the minutae of syntax and schema. Other documents =
seem to be aimed only at using the JavaScript library from a web =
application.=20


Is there some kind of document that describes the overall design of the =
protocol, that I could peruse first?=20


--Jens

[Mod-pubsub-developer] RE: Differences between KnowNow commercial and mod-pubsub

[Adam R. <Ad...@Kn...>]

Suggestion heard, loud and clear.  (By the way, welcome!)

I think it's fair to say KnowNow is the industrial-strength commercial
implementation; for it you can buy support, clustering, failover,
higher-performance, and an Excel connector, plus it has documentation
(which, as we've noted many times, we're lacking), decent installers,
and it has been tested and used by many paying customers.

Most of KnowNow's customers do buy it for commercial line-of-business
applications, whereas mod-pubsub does tend to appeal more to the web
architects and developers of things like email and news apps.

Note that all of the mod-pubsub client libraries work with the =
commercial
server as connectors, so if you ever need to switch over, migration
is fairly straightforward.

   Adam



-----Original Message-----
From: Jens Alfke [mailto:je...@ma...]
Sent: Thursday, October 16, 2003 2:10 PM
To: Adam Rifkin
Cc: Joyce Park; mod...@li...
Subject: Re: [Mod-pubsub-developer] Getting started?

On Oct 16, 2003, at 2:00 PM, Adam Rifkin wrote:

> Perhaps Jens was suggesting that we should do something like that=20
> introduction?

Yes, please!

Actually now I'm curious what the differences are between KnowNow and=20
mod-pubsub; I had been assuming that the former was 'merely' the=20
industrial-strength commercial implementation of the latter, as with=20
jabber.com and jabber.org.

--Jens

Re: [Mod-pubsub-developer] Getting started?

[Jens A. <je...@ma...>]

On Oct 16, 2003, at 2:00 PM, Adam Rifkin wrote:

> Perhaps Jens was suggesting that we should do something like that 
> introduction?

Yes, please!

Actually now I'm curious what the differences are between KnowNow and 
mod-pubsub; I had been assuming that the former was 'merely' the 
industrial-strength commercial implementation of the latter, as with 
jabber.com and jabber.org.

--Jens

RE: [Mod-pubsub-developer] Getting started?

[Adam R. <Ad...@Kn...>]

Perhaps Jens was suggesting that we should do something like that =
introduction?
Which is a good idea, actually.  I'll put it on my ever-expanding to-do =
list.

  :) Adam


-----Original Message-----
From: Joyce Park [mailto:tru...@ya...]
Sent: Thursday, October 16, 2003 12:24 PM
To: Jens Alfke
Cc: mod...@li...
Subject: Re: [Mod-pubsub-developer] Getting started?

--- Jens Alfke <je...@ma...> wrote:
>
> Actually, later last night I came across
> 	http://developer.knownow.com/devguide/intro_knownow/kn_intro.html
> which is pretty much exactly what I wanted. You should put a link to =
it=20
> on the home page of mod-pubsub.org!

Um, I don't think so.

Don't assume mod-pubsub is a workalike for KN in every respect.  JP
_______________________________________________
Mod-pubsub-developer mailing list
Mod...@li...
https://lists.sourceforge.net/lists/listinfo/mod-pubsub-developer

Re: [Mod-pubsub-developer] Getting started?

[Joyce P. <tru...@ya...>]

--- Jens Alfke <je...@ma...> wrote:
>
> Actually, later last night I came across
> 	http://developer.knownow.com/devguide/intro_knownow/kn_intro.html
> which is pretty much exactly what I wanted. You should put a link to it 
> on the home page of mod-pubsub.org!

Um, I don't think so.

Don't assume mod-pubsub is a workalike for KN in every respect.  JP

[Mod-pubsub-developer] Persistent storage / forwarding of events

[Jens A. <je...@ma...>]

First questions. I should preface this by saying that I'm mostly 
investigating using mod-pubsub for asynchronous messaging, things like 
email or newsgroups or blogs where a recipient may not be online at the 
time a message is sent. So:

(1) Do topics have the capability to store-and-forward events? If I set 
up a route from a source topic to my 'journal' topic, and then 
disconnect, will the journal queue up the incoming events as a POP 
server would, letting me reconnect later and receive the events I've 
received in the interim?

(2) Can topics store their events persistently so they can be retrieved 
later by clients that connect to them? For example, a topic might 
implement a newsgroup, and a new subscriber would want to be able to 
download the existing posts, or at least the last hundred or so.

If this isn't built-in, would it be straightforward to build a solution 
on top of mod-pubsub, or would this be a bad design choice, attempting 
to force it to do something it just wasn't meant to?

--Jens

Re: [Mod-pubsub-developer] Getting started?

[Jens A. <je...@ma...>]

On Oct 16, 2003, at 8:31 AM, Joyce Park wrote:

> Hi Jens!  Sorry, we don't have a lot of stuff on architecture in the 
> sense I
> think you mean it -- although maybe we should.

Actually, later last night I came across
	http://developer.knownow.com/devguide/intro_knownow/kn_intro.html
which is pretty much exactly what I wanted. You should put a link to it 
on the home page of mod-pubsub.org!

I do have other questions but I'll start new threads for those.

--Jens

[Mod-pubsub-developer] Getting started?

[Jens A. <je...@ma...>]

I'm trying to learn my way around the pubsub protocol, and having some 
difficulty. I'm interested in the protocol itself, and in using it in 
an application, i.e. not in a web browser. The main protocol document
	http://www.mod-pubsub.org/kn_docs/pubsub_protocol.html
seems to assume the reader already knows how the architecture works, 
and launches directly into the minutae of syntax and schema. Other 
documents seem to be aimed only at using the JavaScript library from a 
web application.

Is there some kind of document that describes the overall design of the 
protocol, that I could peruse first?

--Jens