From: Christopher W. <Chr...@ny...> - 2007-05-11 13:58:42
|
Right, essentially that is what I was going to do but after actually thinking about it and with this new information it's probably a better way to go. There are other things we want to do such as notification via jabber when a publish process was complete. Still it didn't seem problematic but then there are a lot of events I began to take into account that AMQ would abstract away for me. Making sure there are no race conditions, holding the queue on failure, communication between one daemon and the next, priorities, all sorts of possbile error event etc etc. All that and I was pretty deep down the rabbit hole of all the things I was beginning to have to take into account just for the daemon itself; at least if I wanted it to operate properly. So i'm going to pickup a book asap on JMS and start playing with this. It might even make things alot quicker, we'll see what I can come up with. Thanks again for this Pete, sincerely. On Fri, 2007-05-11 at 09:41 -0400, Peter Leonard wrote: > One other note - > > If there is desire to do this as a 'pure-Krang' play, and not tie into > external systems, I really don't think it would be all that difficult to > write a publish daemon that would essentially do the same thing. I've > kicked around the idea with a number of people on this list previously, > and the general consensus is that a simple implementation of the idea is > likely a short project. > > AMQ opens up a realm of possibilities, but it becomes a real question as > to how far down the rabbit hole you want to go with it - I really think > that distributed publishing is really the tip of the iceberg on the > matter, and fairly easy to work with. Far more interesting is what can > be done with events related to the publish process (e.g. notifications & > the like), or other intensive processes in Krang where being able to > hand off messages in an asynchronous fashion for other systems to work > against is desirable. > > Peter > > > On Fri, 2007-05-11 at 09:16 -0400, Christopher Warner wrote: > > Thanks much for this reply. Originally, Albert Lee had the very same > > idea in regards to using a Broker to manage the publish process. It > > would add an external piece I wasn't fully sure about; since I don't > > know of any decent Corba perl stuff, unless we used Java (or some > > other orb stuff) in which case that would involve another layer, the > > only other idea was to sadly write my own solution. At least if there > > were problems, i'd be able to manage them myself. If ActiveMQ works > > well it'll save lots of work as the only major piece would be AMQ > > itself, everything else is obviously plugging krang into it. What > > sorts of problems have you been having with the client though and is > > it enough that I should be even bothered. The whole thing is that I > > need to employ some solution in August. Which is not a lot of time to > > bang all this out; but with just AMQ it'll save lots of time. > > > > Thanks again! > > -C > > > > > > On Thu, 2007-05-10 at 22:13 -0400, Peter Leonard wrote: > > > Chris, > > > > > > I've been removed from Krang development for a while, but I'd like to > > > offer this thought to the mix: > > > > > > What you're really looking for is a message framework. Krang could > > > definitely benefit from a distributed-publishing model when it comes to > > > larger publishing jobs, but reinventing the wheel doesn't make a lot of > > > sense. > > > > > > What I would suggest is that those people more involved in the Krang > > > development process take a look the various messaging frameworks. In > > > the OSS world, the best available one (and under the Apache Foundation) > > > is ActiveMQ (http://activemq.org). > > > > > > If Krang was to integrate with ActiveMQ, it would simplify the entire > > > process of event dissemination & queue handling. Admittedly, AMQ is a > > > large piece of software in its' own right, and it takes some time to > > > understand what can be done with it, but I think that it could be used > > > to really leverage Krang in a number of different directions. > > > > > > The only downside at the moment is that the Perl client for AMQ is not > > > very good - more of a proof-of-concept than anything else. At my > > > current employer, we've developed a more robust Perl client, but I've > > > hit some resistance in the OSS process. Still working on it, this might > > > be enough to get it to happen. > > > > > > > > > Pete > > > > > > > > > > > > On Wed, 2007-05-02 at 11:20 -0400, Christopher Warner wrote: > > > > It's been a while since discussion on Parallelism :-) > > > > > > > > I've begun charting a course to add a Queue system to Krang where any > > > > object that can be published may be added to the queue and then a > > > > Publisher Daemon would simply publish the stories in the queue in > > > > threaded manner to the file-system. This would separate the process of > > > > publishing a bit. I've already been over the idea and some details > > > > here at the mag and am presenting the idea to the list in hopes that > > > > someone may or may not have good ideas or words of caution. The > > > > primary reason I feel it is needed is because it will help with some > > > > of the performance problems larger installations of krang will run > > > > into while dealing with a large amount of stories. I've run into this > > > > problem at Primedia but addressing it there was problematic (very old > > > > versions of krang in some cases) and stop-energy was high and overall > > > > management didn't seem to care. At NYM it's a little different as I'm > > > > only dealing with one installation but it's an extremely large one > > > > with lots of data. > > > > > > > > I'd like for this to eventually see it's way back into the trunk if > > > > possible; we are currently using Krang v 2.001 with modification > > > > primarily in our add-ons which have extended some functionality in > > > > regards to publishing for listings etc but it shouldn't affect the > > > > overall idea of the above. This is currently one of the things I think > > > > will be a bit painful to decouple after finishing the work. I haven't > > > > done a direct comparison of 2.001 and 2.1/SVN so changes might already > > > > be significant enough that this is actually not possible without much > > > > effort but hopefully someone can guide me in that respect. > > > > > > > > I realize that there are other things to take into consideration as > > > > far as performance goes but I think when the discussion of parallel > > > > publishing came up before much of the list agreed it was probably the > > > > best way to go. > > > > > > > > Thanks much! > > > > -- > > > > Christopher Warner > > > > New York Magazine > > > > 444 Madison Ave, 4th Floor > > > > New York, NY 10022-6955 > > > > 212-508-0542 > > > > ------------------------------------------------------------------------- > > > > This SF.net email is sponsored by DB2 Express > > > > Download DB2 Express C - the FREE version of DB2 express and take > > > > control of your XML. No limits. Just data. Click to get it now. > > > > http://sourceforge.net/powerbar/db2/ > > > > _______________________________________________ Krang-devel mailing list Kra...@li... https://lists.sourceforge.net/lists/listinfo/krang-devel > > > > > -- > > Christopher Warner > > New York Magazine > > 444 Madison Ave, 4th Floor > > New York, NY 10022-6955 > > 212-508-0542 > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by DB2 Express > > Download DB2 Express C - the FREE version of DB2 express and take > > control of your XML. No limits. Just data. Click to get it now. > > http://sourceforge.net/powerbar/db2/ > > _______________________________________________ Krang-devel mailing list Kra...@li... https://lists.sourceforge.net/lists/listinfo/krang-devel > -- Christopher Warner New York Magazine 444 Madison Ave, 4th Floor New York, NY 10022-6955 212-508-0542 |