Menu
▾
▴
RE: [Babeldoc-devel] Late night musings on parallel execution ofpipeline stage results
|
From: McDonald, B. <Bru...@ba...> - 2003-07-17 17:18:50
|
Take a look at that example in the core module - in the pipeline/config.properties: #pipeline/config configuration #Thu Jun 05 10:31:49 EDT 2003 pipeline.type=simple pipeline.configFile=pipeline/pipeline asyncpipeline.type=simple asyncpipeline.configFile=pipeline/pipeline asyncpipeline.processor.type=async asyncpipeline.processor.maxThreads=4 pooledpipeline.type=simple pooledpipeline.configFile=pipeline/pipeline pooledpipeline.processor.type=threadpool pooledpipeline.processor.poolSize=10 This defines three pipelines using the three different pipelinestage processors. The first pipeline does not actually define a processor - it gets the synchronous processor. This means that existing pipelines work as before. The other two pipelines: asyncpipeline and pooledpipeline used the async processor and the pooled processor respectively. regards, Bruce. -----Original Message----- From: Leech, Jonathan [mailto:jl...@vi...] Sent: Thursday, July 17, 2003 12:56 PM To: 'Bruce McDonald'; bab...@li... Subject: RE: [Babeldoc-devel] Late night musings on parallel execution ofpipeline stage results Quick question -- how do I choose which processor to use? -----Original Message----- From: Bruce McDonald [mailto:br...@mc...] Sent: Wednesday, July 16, 2003 4:55 PM To: Leech, Jonathan; bab...@li... Subject: Re: [Babeldoc-devel] Late night musings on parallel execution of pipeline stage results Ok all, I have been kinda busy today working on this. I have checked in code that will make the threading in the pipelines a policy of the pipeline stage factory and configurable from that point. Please review the affected files. To get a good idea of how to use this - please take a look at the core/examples/threads example. This contains three pipelines (with a single definition file) which implements all of the three threading processors. 1. SyncPipelineStageProcessor (this is the current implementation and is the default use unless the pipeline is configured otherwise.) 2. AsyncPipelineStageProcessor (this is basically Jonathans code extracted from the the PipelineStageFactory. This creates threads as necessary and then waits for them to complete) 3. ThreadPooledPipelineStageProcessor (this is a thread pooled implementation that I whipped into shape today - it still leaves something to be desired) Here are some notes: 1. The standard behavior is retained unless you configure it differently. 2. All the configuration of the threading for pipelines is done in the pipeline/config.properties file. 3. The Async processor accepts hints from the pipeline definition file (maxThreads) 4. I have added three required properties to the services/query.properties file to look up the following service names: sync; async; threadpool regards, Bruce. On Wednesday 16 July 2003 10:45 am, Leech, Jonathan wrote: > Bruce, > > I haven't looked at the refactoring yet but that sounds excellent. I had > thought about using a proper ThreadPool but didn't have one and didn't > think to look for one. The problem with not letting a PipelineStage > specify maxThreads is that you can end up with one PipelineStage using all > the available threads, and starving other stages that you want to run > concurrently. Also, if you are using up a limited resource in the stage > e.g. database connections, it doesn't make sense to have more threads > running than connections. But once you go with a thread pool, there should > also be a maxThreads there as well. > > -Jonathan > > -----Original Message----- > From: Bruce McDonald [mailto:br...@mc...] > Sent: Tuesday, July 15, 2003 11:29 PM > To: bab...@li... > Subject: [Babeldoc-devel] Late night musings on parallel execution of > pipeline stage results > > > Jonathan, All: > > I have refactored the PipelineStageFactory so that methods: > > processPipelineStageResults (non-threaded) > processPipelineStageResultsParallel (threaded) > > are as similar as possible and all possible code overlaps are extracted to > other methods. At the core, the difference between the two of them is the > maxThreads argument. This argument indicates the number of threads to be > spawned to handle the processing of the pipeline stage results. I contend > that this is an attribute of the pipeline stage factory and should be > managed > by a threadpool. In fact, in direct contradiction to my prior email on > this > > matter, an even simpler implementation would be to subclass the > non-threaded > > implementation and override the processPipelineStageResults method to > handle > > all the threaded complexity. > > thoughts? > > regards, > Bruce. > > PS. My thanks to Jonathan for stinging my lazy butt out of a destructive > cycle of Rise Of Nations multiplayer games :) > > PPS. I would seriously advise everyone to look at this code - it looks > solid. > That ThreadLocal thing was new to me. > > > ------------------------------------------------------- > This SF.net email is sponsored by: VM Ware > With VMware you can run multiple operating systems on a single machine. > WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the > same time. Free trial click here: http://www.vmware.com/wl/offer/345/0 > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel > > > ------------------------------------------------------- > This SF.net email is sponsored by: VM Ware > With VMware you can run multiple operating systems on a single machine. > WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the > same time. Free trial click here: http://www.vmware.com/wl/offer/345/0 > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel ------------------------------------------------------- This SF.net email is sponsored by: VM Ware With VMware you can run multiple operating systems on a single machine. WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the same time. Free trial click here: http://www.vmware.com/wl/offer/345/0 _______________________________________________ Babeldoc-devel mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-devel ------------------------------------------------------- This SF.net email is sponsored by: VM Ware With VMware you can run multiple operating systems on a single machine. WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the same time. Free trial click here: http://www.vmware.com/wl/offer/345/0 _______________________________________________ Babeldoc-devel mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-devel |
