Menu
▾
▴
RE: [Babeldoc-devel] Late night musings on parallel execution ofp ipeline stage results
|
From: Leech, J. <jl...@vi...> - 2003-07-17 17:28:26
|
OK. That makes sense. I have another question: Is the other maxThreads -- the one for the pipeline -- an overall maxThreads, or a default for stages that don't specify maxThreads? -Jonathan -----Original Message----- From: McDonald, Bruce [mailto:Bru...@ba...] Sent: Thursday, July 17, 2003 11:23 AM To: Leech, Jonathan; Bruce McDonald; bab...@li... Subject: RE: [Babeldoc-devel] Late night musings on parallel execution ofpipeline stage results No. then in those stages that need to not run multithreaded, put maxThreads=1 -----Original Message----- From: Leech, Jonathan [mailto:jl...@vi...] Sent: Thursday, July 17, 2003 1:17 PM To: 'Bruce McDonald'; bab...@li... Subject: RE: [Babeldoc-devel] Late night musings on parallel execution ofpipeline stage results Bruce, I need to be able to mix parallel and serial pipeline processing, as I have stages that depend on other stages. To accomplish this, in my pipeline I have some non-threaded, ordered router stages that start threaded stages. With your changes, it looks like I can't have the non-threaded stages, unless I want the whole pipeline to be non-threaded. Am I missing something? -Jonathan -----Original Message----- From: Bruce McDonald [mailto:br...@mc...] Sent: Wednesday, July 16, 2003 4:55 PM To: Leech, Jonathan; bab...@li... Subject: Re: [Babeldoc-devel] Late night musings on parallel execution of pipeline stage results Ok all, I have been kinda busy today working on this. I have checked in code that will make the threading in the pipelines a policy of the pipeline stage factory and configurable from that point. Please review the affected files. To get a good idea of how to use this - please take a look at the core/examples/threads example. This contains three pipelines (with a single definition file) which implements all of the three threading processors. 1. SyncPipelineStageProcessor (this is the current implementation and is the default use unless the pipeline is configured otherwise.) 2. AsyncPipelineStageProcessor (this is basically Jonathans code extracted from the the PipelineStageFactory. This creates threads as necessary and then waits for them to complete) 3. ThreadPooledPipelineStageProcessor (this is a thread pooled implementation that I whipped into shape today - it still leaves something to be desired) Here are some notes: 1. The standard behavior is retained unless you configure it differently. 2. All the configuration of the threading for pipelines is done in the pipeline/config.properties file. 3. The Async processor accepts hints from the pipeline definition file (maxThreads) 4. I have added three required properties to the services/query.properties file to look up the following service names: sync; async; threadpool regards, Bruce. On Wednesday 16 July 2003 10:45 am, Leech, Jonathan wrote: > Bruce, > > I haven't looked at the refactoring yet but that sounds excellent. I had > thought about using a proper ThreadPool but didn't have one and didn't > think to look for one. The problem with not letting a PipelineStage > specify maxThreads is that you can end up with one PipelineStage using all > the available threads, and starving other stages that you want to run > concurrently. Also, if you are using up a limited resource in the stage > e.g. database connections, it doesn't make sense to have more threads > running than connections. But once you go with a thread pool, there should > also be a maxThreads there as well. > > -Jonathan > > -----Original Message----- > From: Bruce McDonald [mailto:br...@mc...] > Sent: Tuesday, July 15, 2003 11:29 PM > To: bab...@li... > Subject: [Babeldoc-devel] Late night musings on parallel execution of > pipeline stage results > > > Jonathan, All: > > I have refactored the PipelineStageFactory so that methods: > > processPipelineStageResults (non-threaded) > processPipelineStageResultsParallel (threaded) > > are as similar as possible and all possible code overlaps are extracted to > other methods. At the core, the difference between the two of them is the > maxThreads argument. This argument indicates the number of threads to be > spawned to handle the processing of the pipeline stage results. I contend > that this is an attribute of the pipeline stage factory and should be > managed > by a threadpool. In fact, in direct contradiction to my prior email on > this > > matter, an even simpler implementation would be to subclass the > non-threaded > > implementation and override the processPipelineStageResults method to > handle > > all the threaded complexity. > > thoughts? > > regards, > Bruce. > > PS. My thanks to Jonathan for stinging my lazy butt out of a destructive > cycle of Rise Of Nations multiplayer games :) > > PPS. I would seriously advise everyone to look at this code - it looks > solid. > That ThreadLocal thing was new to me. > > > ------------------------------------------------------- > This SF.net email is sponsored by: VM Ware > With VMware you can run multiple operating systems on a single machine. > WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the > same time. Free trial click here: http://www.vmware.com/wl/offer/345/0 > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel > > > ------------------------------------------------------- > This SF.net email is sponsored by: VM Ware > With VMware you can run multiple operating systems on a single machine. > WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the > same time. Free trial click here: http://www.vmware.com/wl/offer/345/0 > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel ------------------------------------------------------- This SF.net email is sponsored by: VM Ware With VMware you can run multiple operating systems on a single machine. WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the same time. Free trial click here: http://www.vmware.com/wl/offer/345/0 _______________________________________________ Babeldoc-devel mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-devel ------------------------------------------------------- This SF.net email is sponsored by: VM Ware With VMware you can run multiple operating systems on a single machine. WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the same time. Free trial click here: http://www.vmware.com/wl/offer/345/0 _______________________________________________ Babeldoc-devel mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-devel |
