Re: [Krang-devel] Krang Queue System

[Christopher W. <Chr...@ny...>]

Thanks again Pete this was really helpful. As far as deadlines, it's not
too bad the worst that could happen is I miss the deadline.  At the very
least i'd still be alive :-) Gotta look at the positive and all that.

Anyway thanks i'm gonna get up to speed on JMS/AMQ.

-C

On Fri, 2007-05-11 at 09:36 -0400, Peter Leonard wrote:

> Chris,
> 
> Take a look at AMQ - it works well out of the box (we're running 4.1.0
> plus a couple of patches found in 4.1.1).  The only advice I'd give is
> to pay attention to memory allocation, as it will hang if the JVM runs
> out of memory (though that might have been fixed in 4.1.1).  We got
> around the issue by setting memory limits high & monitoring the system,
> restarting in maintenance windows as needed.  In an environment like
> Krang, this will be no big deal.
> 
> AMQ provides the STOMP interface for non-native client implementations,
> which the publicly-available Perl client uses.  The issue with the
> client is really more one of immaturity than anything else - it's a
> proof-of-concept, and not really production-ready.  That being said,
> rewriting the prototype would be easier than implementing a pub/sub
> system.  
> 
> As for your deadlines, I'm sorry to hear that you've had deadlines
> imposed on you before you know the scope of the work - not an easy task
> to undertake.  ;)
> 
> Good luck,
> 
> Peter
> 
> 
> 
> On Fri, 2007-05-11 at 09:16 -0400, Christopher Warner wrote:
> > Thanks much for this reply. Originally, Albert Lee had the very same
> > idea in regards to using a Broker to manage the publish process. It
> > would add an external piece I wasn't fully sure about; since I don't
> > know of any decent Corba perl stuff, unless we used Java (or some
> > other orb stuff) in which case that would involve another layer, the
> > only other idea was to sadly write my own solution. At least if there
> > were problems, i'd be able to manage them myself. If ActiveMQ works
> > well it'll save lots of work as the only major piece would be AMQ
> > itself, everything else is obviously plugging krang into it. What
> > sorts of problems have you been having with the client though and is
> > it enough that I should be even bothered. The whole thing is that I
> > need to employ some solution in August. Which is not a lot of time to
> > bang all this out; but with just AMQ it'll save lots of time.
> > 
> > Thanks again!
> > -C
> > 
> > 
> > On Thu, 2007-05-10 at 22:13 -0400, Peter Leonard wrote: 
> > > Chris,
> > > 
> > > I've been removed from Krang development for a while, but I'd like to
> > > offer this thought to the mix:
> > > 
> > > What you're really looking for is a message framework.  Krang could
> > > definitely benefit from a distributed-publishing model when it comes to
> > > larger publishing jobs, but reinventing the wheel doesn't make a lot of
> > > sense.
> > > 
> > > What I would suggest is that those people more involved in the Krang
> > > development process take a look the various messaging frameworks.  In
> > > the OSS world, the best available one (and under the Apache Foundation)
> > > is ActiveMQ (http://activemq.org).  
> > > 
> > > If Krang was to integrate with ActiveMQ, it would simplify the entire
> > > process of event dissemination & queue handling.  Admittedly, AMQ is a
> > > large piece of software in its' own right, and it takes some time to
> > > understand what can be done with it, but I think that it could be used
> > > to really leverage Krang in a number of different directions.
> > > 
> > > The only downside at the moment is that the Perl client for AMQ is not
> > > very good - more of a proof-of-concept than anything else.  At my
> > > current employer, we've developed a more robust Perl client, but I've
> > > hit some resistance in the OSS process.  Still working on it, this might
> > > be enough to get it to happen.
> > > 
> > > 
> > > Pete
> > > 
> > > 
> > > 
> > > On Wed, 2007-05-02 at 11:20 -0400, Christopher Warner wrote:
> > > > It's been a while since discussion on Parallelism :-)
> > > > 
> > > > I've begun charting a course to add a Queue system to Krang where any
> > > > object that can be published may be added to the queue and then a
> > > > Publisher Daemon would simply publish the stories in the queue in
> > > > threaded manner to the file-system. This would separate the process of
> > > > publishing a bit. I've already been over the idea and some details
> > > > here at the mag and am presenting the idea to the list in hopes that
> > > > someone may or may not have good ideas or words of caution. The
> > > > primary reason I feel it is needed is because it will help with some
> > > > of the performance problems larger installations of krang will run
> > > > into while dealing with a large amount of stories. I've run into this
> > > > problem at Primedia but addressing it there was problematic (very old
> > > > versions of krang in some cases) and stop-energy was high and overall
> > > > management didn't seem to care. At NYM it's a little different as I'm
> > > > only dealing with one installation but it's an extremely large one
> > > > with lots of data. 
> > > > 
> > > > I'd like for this to eventually see it's way back into the trunk if
> > > > possible; we are currently using Krang v 2.001 with modification
> > > > primarily in our add-ons which have extended some functionality in
> > > > regards to publishing for listings etc but it shouldn't affect the
> > > > overall idea of the above. This is currently one of the things I think
> > > > will be a bit painful to decouple after finishing the work. I haven't
> > > > done a direct comparison of 2.001 and 2.1/SVN so changes might already
> > > > be significant enough that this is actually not possible without much
> > > > effort but hopefully someone can guide me in that respect.
> > > > 
> > > > I realize that there are other things to take into consideration as
> > > > far as performance goes but I think when the discussion of parallel
> > > > publishing came up before much of the list agreed it was probably the
> > > > best way to go. 
> > > > 
> > > > Thanks much! 
> > > > -- 
> > > > Christopher Warner
> > > > New York Magazine
> > > > 444 Madison Ave, 4th Floor
> > > > New York, NY 10022-6955
> > > > 212-508-0542
> > > > -------------------------------------------------------------------------
> > > > This SF.net email is sponsored by DB2 Express
> > > > Download DB2 Express C - the FREE version of DB2 express and take
> > > > control of your XML. No limits. Just data. Click to get it now.
> > > > http://sourceforge.net/powerbar/db2/
> > > > _______________________________________________ Krang-devel mailing list Kra...@li... https://lists.sourceforge.net/lists/listinfo/krang-devel
> > > 
> > -- 
> > Christopher Warner
> > New York Magazine
> > 444 Madison Ave, 4th Floor
> > New York, NY 10022-6955
> > 212-508-0542
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by DB2 Express
> > Download DB2 Express C - the FREE version of DB2 express and take
> > control of your XML. No limits. Just data. Click to get it now.
> > http://sourceforge.net/powerbar/db2/
> > _______________________________________________ Krang-devel mailing list Kra...@li... https://lists.sourceforge.net/lists/listinfo/krang-devel
> 

-- 
Christopher Warner
New York Magazine
444 Madison Ave, 4th Floor
New York, NY 10022-6955
212-508-0542