Menu
▾
▴
babeldoc-devel — Babeldoc developer list
You can subscribe to this list here.
| 2003 |
Jan
|
Feb
(14) |
Mar
(107) |
Apr
(211) |
May
(93) |
Jun
(158) |
Jul
(159) |
Aug
(368) |
Sep
(188) |
Oct
(151) |
Nov
(115) |
Dec
(98) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2004 |
Jan
(25) |
Feb
|
Mar
(33) |
Apr
(28) |
May
(116) |
Jun
(2) |
Jul
(117) |
Aug
(19) |
Sep
(9) |
Oct
(2) |
Nov
|
Dec
(4) |
| 2005 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
(9) |
Dec
|
| 2006 |
Jan
|
Feb
|
Mar
(22) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2007 |
Jan
|
Feb
|
Mar
(6) |
Apr
|
May
|
Jun
|
Jul
|
Aug
(267) |
Sep
|
Oct
|
Nov
(6) |
Dec
(512) |
| 2008 |
Jan
(187) |
Feb
|
Mar
|
Apr
|
May
|
Jun
(3) |
Jul
(6) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
|
Dec
|
| 2012 |
Jan
|
Feb
|
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: McDonald, B. <Bru...@ba...> - 2003-09-30 19:15:59
|
Please submit. On the subject of JTidy - is it possible to use that stage? -----Original Message----- From: Leech, Jonathan [mailto:jl...@vi...] Sent: Tuesday, September 30, 2003 12:36 PM To: bab...@li... Subject: [Babeldoc-devel] HttpClientPipelineStage If anyone is interested in some modifications I've made to the HttpClientPipelineStage, I'll commit them sooner rather than later. In short, they make it easier for babeldoc to screen-scrape websites and do other useful things. I've added the following: - an option to use JTidy on the response to convert html to xhtml - support for BASIC authentication - binary mode for downloading binary files -Jonathan |
|
From: David G. <dg...@co...> - 2003-09-30 19:05:44
|
Hi Qin, Your dump below shows you starting in the c:\babeldoc-bin-1.2.0-RC1\bin directory. For the template that I sent you, you need to start from the c:\qin directory (assuming that's where you installed the zip file I sent). Also, I think you mentioned that you run on Windows NT. You may want to try changing the long pathnames to mangled paths to see if that makes any difference. I haven't run NT in years, but I thought that NT provided two command shells: command.com and cmd.exe. As I recall, cmd.exe allows long path names and command.com prefers mangled names. Just a thought... David On Tuesday 30 September 2003 8:44 am, Qin Ding wrote: > Bruce and David: > > I downloaded the latest the version (babeldoc-bin-1.2.0-RC1) and modified > the env BABELDOC_HOME to point to c:\babeldoc-bin-1.2.0-RC1. Started a new > command shell and re-ran the test: > C:\babeldoc-bin-1.2.0-RC1\bin>babeldoc process -p test-pipeline -f c: > \qin\dummy.txt > > I got the same error: > <2003-09-30 10:35:19,670> ERROR [main] : [PipelineFeeder.process] > com.babeldoc.core.pipeline.PipelineException: PipelineStage: entryStage not > found > at > com.babeldoc.core.pipeline.PipelineStageFactory.getEntryStageName(Unknown > Source) > at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source) > at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown > Source) > at > com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source) > at com.babeldoc.core.pipeline.feeder.FeederFactory.process(Unknown > Source) > at com.babeldoc.core.pipeline.command.PipelineFeeder.process(Unknown > Source) > at com.babeldoc.core.pipeline.command.PipelineFeeder.execute(Unknown > Source) > at com.babeldoc.core.BabeldocCommand.executeCommand(Unknown Source) > at com.babeldoc.core.BabeldocCommand.<init>(Unknown Source) > at com.babeldoc.core.pipeline.command.PipelineFeeder.<init>(Unknown > Source) > at com.babeldoc.core.pipeline.command.PipelineFeeder.main(Unknown > Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3 >9) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp >l.java:25) at java.lang.reflect.Method.invoke(Method.java:324) > at com.babeldoc.core.Main.handleCommand(Unknown Source) > at com.babeldoc.core.Main.main(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3 >9) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp >l.java:25) at java.lang.reflect.Method.invoke(Method.java:324) > at com.babeldoc.init.Main.runBabeldocMain(Unknown Source) > at com.babeldoc.init.Main.main(Unknown Source) > > > Here is the modified setenv.bat file: > set BABELDOC_HOME=c:\babeldoc-bin-1.2.0-RC1 > set BABELDOC_USER=c:\qin\config > set JAVA_HOME=D:\j2sdk1.4.1_03 > > set PATH=%BABELDOC_HOME%\bin;%JAVA_HOME%\bin;%PATH% > > Please see the attached the qin.zip file.(See attached file: qin.zip) > > This is a PRIVATE message. If you are not the intended recipient, please > delete without copying and kindly advise us by e-mail of the mistake in > delivery. NOTE: Regardless of content, this e-mail shall not operate to > bind CSC to any order or other contract unless pursuant to explicit written > agreement or government initiative expressly permitting the use of e-mail > for such purpose. > --------------------------------------------------------------------------- >------------- > > > > > > "McDonald, > Bruce" To: Qin > Ding/ASD/CSC@CSC <Bruce.McDonald cc: Babeldoc Developer List > @bankofamerica.c <bab...@li...>, David > Glick om> <dg...@co...> > Subject: RE: > [Babeldoc-devel] Re: [Babeldoc-user] 09/29/03 10:31 Questions. > Please Help AM > > > > > > > Download the latest version and try again - I suspect your installation is > broken. > > -----Original Message----- > From: Qin Ding [mailto:qd...@cs...] > Sent: Friday, September 26, 2003 1:10 PM > To: McDonald, Bruce > Cc: Babeldoc Developer List; David Glick > Subject: RE: [Babeldoc-devel] Re: [Babeldoc-user] Questions. Please Help > > > > Bruce: > > This is the test-pipeline.properties: > > entryStage=initialize > > initialize.stageType=Scripting > initialize.nextStage=null > initialize.failOnError=true > initialize.language=javascript > initialize.script=\ > java.io.File file = new java.io.File(document.get("file_name"));\ > TestManager.validate(file); > > > Here is the error: > > <2003-09-26 11:05:11,960> INFO [Thread-1] : Processing document... > <2003-09-26 11:05:11,960> ERROR [Thread-1] : [AsynchronousFeeder$1.run] > com.babeldoc.core.pipeline.PipelineException: PipelineStage: entryStage not > foun > d > at > com.babeldoc.core.pipeline.PipelineStageFactory.getEntryStageName(Unk > nown Source) > at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown > Source) > at > com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Sou > rce) > at > com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown S > ource) > at > com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess( > Unknown Source) > at > com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown So > urce) > at > EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown So > urce) > at java.lang.Thread.run(Thread.java:536) > > > Qin > --------------------------------------------------------------------------- >------------- > > > This is a PRIVATE message. If you are not the intended recipient, please > delete without copying and kindly advise us by e-mail of the mistake in > delivery. NOTE: Regardless of content, this e-mail shall not operate to > bind CSC to any order or other contract unless pursuant to explicit written > agreement or government initiative expressly permitting the use of e-mail > for such purpose. > --------------------------------------------------------------------------- >------------- > > > > > > > "McDonald, > > Bruce" To: Qin Ding > <qd...@cs...>, David Glick > <Bruce.McDonald <dg...@co...> > > @bankofamerica.c cc: Babeldoc Developer > List > om> > <bab...@li...> > Subject: RE: > [Babeldoc-devel] Re: [Babeldoc-user] > 09/26/03 11:39 Questions. Please Help > > AM > > > > > > > > Huh?!??! > > I dont understand the issue here. > > Somewhere in your pipeline definition file, place the line: > > entryStage=<<some pipeline stage name>> > > Where the entryStage is pointing to the first pipeline stage in your > pipeline. > > regards, > Bruce. > > > -----Original Message----- > From: Qin Ding [mailto:qd...@cs...] > Sent: Friday, September 26, 2003 12:14 PM > To: David Glick > Cc: Babeldoc Developer List > Subject: [Babeldoc-devel] Re: [Babeldoc-user] Questions. Please Help > > > > David: > > I don't have the Unix-to-Windows converter program; but I retyped the file > into wordpad and saved as text document. However, it is no possitive > effect. It still gives the same error, that is, entryStage not found. I > then copied the first line from src > example/scripting/pipeline/scannerxml.properties. Again, no use. The same > error occured no matter what I did. Seems like the application could not > recognize the "entryStage". By the way, I am using version bin-1.1.9-dev. > > Any idea? > > Qin > > > --------------------------------------------------------------------------- >------------- > > > > This is a PRIVATE message. If you are not the intended recipient, please > delete without copying and kindly advise us by e-mail of the mistake in > delivery. NOTE: Regardless of content, this e-mail shall not operate to > bind CSC to any order or other contract unless pursuant to explicit written > agreement or government initiative expressly permitting the use of e-mail > for such purpose. > --------------------------------------------------------------------------- >------------- > > > > > > > > > ------------------------------------------------------- > This sf.net email is sponsored by:ThinkGeek > Welcome to geek heaven. > http://thinkgeek.com/sf > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel > > > > > > > > ------------------------------------------------------- > This sf.net email is sponsored by:ThinkGeek > Welcome to geek heaven. > http://thinkgeek.com/sf > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel -- David Glick Transmit Consulting, Inc 619-475-4052 dg...@tr... |
|
From: McDonald, B. <Bru...@ba...> - 2003-09-30 19:00:22
|
Wow - that sounds like a great plan - you are the master of things multithreaded!! -----Original Message----- From: Leech, Jonathan [mailto:jl...@vi...] Sent: Tuesday, September 30, 2003 12:26 PM To: McDonald, Bruce; David Kinnvall; bab...@li... Subject: RE: [Babeldoc-devel] Multithreading problems... Hmmm. My copy of ConfigInfo doesn't have that method -- I haven't done an update in a long while -- and cvs.sourceforge.net is refusing connections from me at the moment. At some point in the not-too-distant future I will update everything and put it through the multi-threaded babeldoc pressure-cooker that I've got going. -Jonathan -----Original Message----- From: McDonald, Bruce [mailto:Bru...@ba...] Sent: Tuesday, September 30, 2003 10:17 AM To: Leech, Jonathan; David Kinnvall; bab...@li... Subject: RE: [Babeldoc-devel] Multithreading problems... Jonathan, The pipeline stage gets created in the pipelinestagefactory code. But... There is a place in the configdata/configinfo code that actually creates suboptions when data is found that does not have a corresponding config option. I suspect that this is involved because the pipeline stage type that we are dealing with here are those with suboptions - just the kind that will be doing this kind of creating. The entry method for this is ConfigInfo.applyConfigData. This method takes the configuration data and applies it to the configuration options. It will create options if necessary. It will be necessary to synchronize either this method or on the data being fed to this method. Please experiment with this and report back. regards, Bruce. -----Original Message----- From: Leech, Jonathan [mailto:jl...@vi...] Sent: Tuesday, September 30, 2003 11:51 AM To: 'David Kinnvall'; bab...@li... Subject: RE: [Babeldoc-devel] Multithreading problems... I poked around the code a little bit. I didn't see where the PipelineStage gets created, or the config options get set, but its possible that more than one thread is setting the config options (suboptions) at the same time in the HashMap. None of the access to the ConfigOption.suboptions HashMap is synchronized (at least in the version of code I'm looking at, haven't done an update in a while). That's where I would start. -Jonathan -----Original Message----- From: David Kinnvall [mailto:dav...@al...] Sent: Tuesday, September 30, 2003 8:27 AM To: bab...@li... Subject: [Babeldoc-devel] Multithreading problems... Hi guys, I am still struggling with getting my pipeline(s) going together with multiple threads. I have changed my approach from using the threadpool pipeline processor to simply use the asynchronous scanner feeder with a poolSize=x, where x > 1, config as follows: feeder/config.properties: # Scanner feeder implementation scanner.type=asynchronous scanner.queue=disk scanner.queueDir=scanner/queue scanner.queueName=scanner scanner.poolSize=3 which gets loaded and configured by Babeldoc, as expected. Now, to trigger the problem I just have to supply the scanner with two or more documents to scan and submit to the pipeline for parallel processing and the processing dies _almost_ every time, with an NPE in VariableProcessor.mustExpand, for some reason. When it doesn't die, it does strange things further down the pipeline, indicating corrupted data payload in the document, that messes things up, albeit not causing an NPE this time. My particular processing consists of three pipelines, where two of them scan documents from separate sources, applies source-specific initial processing and then call a common main pipeline for the remaining processing tasks. I have had the processing fail in both the initial pipelines as well as in the later, common, one. The processing fails most commonly in an XpathExtractPipelineStage, but now and then it also fails in stages of other types. Example stacktrace (sorry for the formatting): (Oh, and the extra text after extract_fid: is just a little debug printout I added *after* observing the problem, to aid in my searching for the cause. It's not part of the problem, that is.) <2003-09-30 16:05:27,895> INFO [Thread-3] : extract_fid:processStage(ticket:1064930727756,document:null) java.lang.NullPointerException at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source) at com.babeldoc.core.VariableProcessor.expandString(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source) at com.babeldoc.core.pipeline.stage.XpathExtractPipelineStage.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source) at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source) at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source) at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Thread.java:536) As you see, the PooledExecutor gets going, calling the Async Feeder, which calls the underlying Sync Feeder and of the Pipeline goes. Up to the NPE, that is... Several stages have already been successfully executed, in parallel, up to this point. And, as I said, it doesn't *always* fail, and when it does, it isn't *always* in the extract_fid stage of type XpathExtractPipelineStage. Hmm...I managed to catch one of the other ones as well. Here: <2003-09-30 16:25:10,800> INFO [Thread-2] : dl_router:processStage(ticket:1064931905930,document:null) java.lang.NullPointerException at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source) at com.babeldoc.core.VariableProcessor.expandString(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source) at com.babeldoc.core.pipeline.stage.RouterPipelineStage.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source) at com.babeldoc.core.pipeline.stage.CallStagePipelineStage.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source) at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source) at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source) at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Thread.java:536) Any ideas? I can provide almost any file from my configuration setup, if it can be of any aid in tracking this down. I am currently stumped. The reason I need the threading support to work is that in a few of the later pipeline stages there can be substantial delays, in case of which it would certainly be nice if the documents that don't cause any delays can be happily processed in parallel, but that's kinda obvious, I know. :-) Would be nice, though. My personal guess at this time (I have done quite some digging in the code, but obviously not yet enough) is that there seems to be some kind of threading race in the code supporting the options. Then again, that might be totally off, since I don't understand it fully, yet. Help? Regards, David Kinnvall ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Babeldoc-devel mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-devel ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Babeldoc-devel mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-devel |
|
From: McDonald, B. <Bru...@ba...> - 2003-09-30 18:53:44
|
Jonathan,
The pipeline stage gets created in the pipelinestagefactory code. But...
There is a place in the configdata/configinfo code that actually creates suboptions when data is found that does not have a corresponding config option. I suspect that this is involved because the pipeline stage type that we are dealing with here are those with suboptions - just the kind that will be doing this kind of creating.
The entry method for this is ConfigInfo.applyConfigData. This method takes the configuration data and applies it to the configuration options. It will create options if necessary. It will be necessary to synchronize either this method or on the data being fed to this method. Please experiment with this and report back.
regards,
Bruce.
-----Original Message-----
From: Leech, Jonathan [mailto:jl...@vi...]
Sent: Tuesday, September 30, 2003 11:51 AM
To: 'David Kinnvall'; bab...@li...
Subject: RE: [Babeldoc-devel] Multithreading problems...
I poked around the code a little bit. I didn't see where the PipelineStage
gets created, or the config options get set, but its possible that more than
one thread is setting the config options (suboptions) at the same time in
the HashMap. None of the access to the ConfigOption.suboptions HashMap is
synchronized (at least in the version of code I'm looking at, haven't done
an update in a while). That's where I would start.
-Jonathan
-----Original Message-----
From: David Kinnvall [mailto:dav...@al...]
Sent: Tuesday, September 30, 2003 8:27 AM
To: bab...@li...
Subject: [Babeldoc-devel] Multithreading problems...
Hi guys,
I am still struggling with getting my pipeline(s) going together
with multiple threads. I have changed my approach from using the
threadpool pipeline processor to simply use the asynchronous
scanner feeder with a poolSize=x, where x > 1, config as follows:
feeder/config.properties:
# Scanner feeder implementation
scanner.type=asynchronous
scanner.queue=disk
scanner.queueDir=scanner/queue
scanner.queueName=scanner
scanner.poolSize=3
which gets loaded and configured by Babeldoc, as expected.
Now, to trigger the problem I just have to supply the scanner
with two or more documents to scan and submit to the pipeline
for parallel processing and the processing dies _almost_ every
time, with an NPE in VariableProcessor.mustExpand, for some
reason.
When it doesn't die, it does strange things further down the
pipeline, indicating corrupted data payload in the document,
that messes things up, albeit not causing an NPE this time.
My particular processing consists of three pipelines, where
two of them scan documents from separate sources, applies
source-specific initial processing and then call a common
main pipeline for the remaining processing tasks.
I have had the processing fail in both the initial pipelines
as well as in the later, common, one. The processing fails
most commonly in an XpathExtractPipelineStage, but now and
then it also fails in stages of other types.
Example stacktrace (sorry for the formatting):
(Oh, and the extra text after extract_fid: is just a little
debug printout I added *after* observing the problem, to
aid in my searching for the cause. It's not part of the
problem, that is.)
<2003-09-30 16:05:27,895> INFO [Thread-3] :
extract_fid:processStage(ticket:1064930727756,document:null)
java.lang.NullPointerException
at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source)
at com.babeldoc.core.VariableProcessor.expandString(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown
Source)
at
com.babeldoc.core.pipeline.stage.XpathExtractPipelineStage.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown
Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:536)
As you see, the PooledExecutor gets going, calling the Async Feeder,
which calls the underlying Sync Feeder and of the Pipeline goes. Up
to the NPE, that is... Several stages have already been successfully
executed, in parallel, up to this point. And, as I said, it doesn't
*always* fail, and when it does, it isn't *always* in the extract_fid
stage of type XpathExtractPipelineStage.
Hmm...I managed to catch one of the other ones as well. Here:
<2003-09-30 16:25:10,800> INFO [Thread-2] :
dl_router:processStage(ticket:1064931905930,document:null)
java.lang.NullPointerException
at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source)
at com.babeldoc.core.VariableProcessor.expandString(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown
Source)
at
com.babeldoc.core.pipeline.stage.RouterPipelineStage.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown
Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown
Source)
at
com.babeldoc.core.pipeline.stage.CallStagePipelineStage.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown
Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:536)
Any ideas?
I can provide almost any file from my configuration setup, if it
can be of any aid in tracking this down. I am currently stumped.
The reason I need the threading support to work is that in a few
of the later pipeline stages there can be substantial delays, in
case of which it would certainly be nice if the documents that
don't cause any delays can be happily processed in parallel, but
that's kinda obvious, I know. :-) Would be nice, though.
My personal guess at this time (I have done quite some digging in
the code, but obviously not yet enough) is that there seems to be
some kind of threading race in the code supporting the options.
Then again, that might be totally off, since I don't understand
it fully, yet.
Help?
Regards,
David Kinnvall
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
|
|
From: McDonald, B. <Bru...@ba...> - 2003-09-30 18:50:57
|
Bill, At the moment the user stuff is handled by the "simple" user factory etc. This code is to be found in package: com.babeldoc.core.user.simple. The "heart" of this is in the class: SimpleUserFactory. And the authentication code is in method: authenticate. This code is HARDCODED to accept passwords of name "password". All usernames are acceptable except usernames starting with "z" - go figure. There is a SQL implementation of this code that allows for usernames/passwords/roles to be stored in a db. regards, Bruce. -----Original Message----- From: Bill Harrelson [mailto:Bil...@Ac...] Sent: Tuesday, September 30, 2003 11:40 AM To: bab...@li...; Bab...@li... Subject: [Babeldoc-devel] Can anyone help? Setting up babeldoc-web Hi, I sent this a week ago, and nobody has answered yet, so I'm sure that it has dropped off the bottom of the list. Can anyone help? I'm trying to connect Babeldoc to a web application and I can't seem to get it to work. I'm getting a little worried, as I have a tight deadline. I found the directions in modules/web/readme.txt for building and setting up babeldoc with my web server and it mostly worked, but there seems to be some conflict between my basic jakarta-tomcat- 4.1.24 setup in the commons-digester.jar. When I start with the tomcat one, it starts fine, when I start with the one copied over by babeldoc it fails pretty quickly with a class not found error. Anyway, I'm able to get the babeldoc console up, but when I try to access the journal, modules or pipelines it asks for a username and password and I don't know how to set those up. My tomcat currently uses database-based access control, so I assume that I just need a role defined, but don't know what that is. Can someone tell me? Anybody willing to spend 15 minutes on the phone with me? Thanks for any assistance, Bill ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Babeldoc-user mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-user ------- End of forwarded message ------- |
|
From: McDonald, B. <Bru...@ba...> - 2003-09-30 18:29:54
|
Bill I am looking into this one. You are most of the way there. Bruce. -----Original Message----- From: Bill Harrelson [mailto:Bil...@Ac...] Sent: Tuesday, September 30, 2003 11:40 AM To: bab...@li...; Bab...@li... Subject: [Babeldoc-devel] Can anyone help? Setting up babeldoc-web Hi, I sent this a week ago, and nobody has answered yet, so I'm sure that it has dropped off the bottom of the list. Can anyone help? I'm trying to connect Babeldoc to a web application and I can't seem to get it to work. I'm getting a little worried, as I have a tight deadline. I found the directions in modules/web/readme.txt for building and setting up babeldoc with my web server and it mostly worked, but there seems to be some conflict between my basic jakarta-tomcat- 4.1.24 setup in the commons-digester.jar. When I start with the tomcat one, it starts fine, when I start with the one copied over by babeldoc it fails pretty quickly with a class not found error. Anyway, I'm able to get the babeldoc console up, but when I try to access the journal, modules or pipelines it asks for a username and password and I don't know how to set those up. My tomcat currently uses database-based access control, so I assume that I just need a role defined, but don't know what that is. Can someone tell me? Anybody willing to spend 15 minutes on the phone with me? Thanks for any assistance, Bill ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Babeldoc-user mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-user ------- End of forwarded message ------- |
|
From: McDonald, B. <Bru...@ba...> - 2003-09-30 18:22:13
|
Jonathan is onto something here... Let me respond to his email. -----Original Message----- From: David Kinnvall [mailto:dav...@al...] Sent: Tuesday, September 30, 2003 12:05 PM To: McDonald, Bruce; bab...@li... Subject: Re: [Babeldoc-devel] Multithreading problems... McDonald, Bruce wrote: > David, > > Ok, Lets find out exactly what the issue is - is it always the same configuration option that is failing or is some random option. Can you surround the getOptionList in XPathExtract with a try/catch and then dump the configuration data from the pipelinestage stage and the attributes on the document. Hmm...let's see...I managed to trigger a failure twice in the same stage with the same two input documents. The stage configuration is: # XPath Extract - get the file_id attribute extract_fid.stageType=XpathExtract extract_fid.nextStage=file_writer extract_fid.failOnError=false extract_fid.errorStage=error extract_fid.XPath.orig_file_id=/newsitem/@file_id And the (horribly unformatted) dump of doc attributes and pipeline stage configuration are attached for both runs. Initial observation (I think): - failOnError is null, as is - errorStage, as well as - XPath.orig_file_id in both cases. Correct? Any clues from that? > regards, > Bruce. David |
|
From: David K. <dav...@al...> - 2003-09-30 17:41:57
|
Jonathan, Bruce, list, Leech, Jonathan wrote: > Hmmm. My copy of ConfigInfo doesn't have that method -- I haven't done an > update in a long while -- and cvs.sourceforge.net is refusing connections > from me at the moment. At some point in the not-too-distant future I will > update everything and put it through the multi-threaded babeldoc > pressure-cooker that I've got going. My copy has it, based on 1.2.0-RC1 (but no different behavior on 1.1.9 - haven't tested older versions). Synchronizing the ConfigInfo.applyConfigData method does unfortunately not help. Same result, same error. Synchronizing *all* (yeah...silly) methods in ConfigInfo makes no difference. Been working too long now - my brain hurts, can't think clearly. If anyone comes up with any idea(s) or things for me to try with my config, please yell. I'll try to get a better understanding of the thread usage of Babeldoc in the coming days. I find it a tiny bit strange, though, that noone else has experienced the same or similar problems in this area. I can't be the only one running multithreaded, right? Could there be something really *wrong* with my setup? Strange, since it works *very* well single-threaded and has done so for several days now, processing live data in test- mode without any problems. Oh, well, it's solvable, it will just need some additional thinking/testing/tinkering. ;-P > -Jonathan /David |
|
From: McDonald, B. <Bru...@ba...> - 2003-09-30 17:06:03
|
David,
It seems that the document is always null. Is this correct from your understanding of the pipeline/data?
this might be issue here.
regards,
Bruce.
-----Original Message-----
From: David Kinnvall [mailto:dav...@al...]
Sent: Tuesday, September 30, 2003 10:27 AM
To: bab...@li...
Subject: [Babeldoc-devel] Multithreading problems...
Hi guys,
I am still struggling with getting my pipeline(s) going together
with multiple threads. I have changed my approach from using the
threadpool pipeline processor to simply use the asynchronous
scanner feeder with a poolSize=x, where x > 1, config as follows:
feeder/config.properties:
# Scanner feeder implementation
scanner.type=asynchronous
scanner.queue=disk
scanner.queueDir=scanner/queue
scanner.queueName=scanner
scanner.poolSize=3
which gets loaded and configured by Babeldoc, as expected.
Now, to trigger the problem I just have to supply the scanner
with two or more documents to scan and submit to the pipeline
for parallel processing and the processing dies _almost_ every
time, with an NPE in VariableProcessor.mustExpand, for some
reason.
When it doesn't die, it does strange things further down the
pipeline, indicating corrupted data payload in the document,
that messes things up, albeit not causing an NPE this time.
My particular processing consists of three pipelines, where
two of them scan documents from separate sources, applies
source-specific initial processing and then call a common
main pipeline for the remaining processing tasks.
I have had the processing fail in both the initial pipelines
as well as in the later, common, one. The processing fails
most commonly in an XpathExtractPipelineStage, but now and
then it also fails in stages of other types.
Example stacktrace (sorry for the formatting):
(Oh, and the extra text after extract_fid: is just a little
debug printout I added *after* observing the problem, to
aid in my searching for the cause. It's not part of the
problem, that is.)
<2003-09-30 16:05:27,895> INFO [Thread-3] : extract_fid:processStage(ticket:1064930727756,document:null)
java.lang.NullPointerException
at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source)
at com.babeldoc.core.VariableProcessor.expandString(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source)
at com.babeldoc.core.pipeline.stage.XpathExtractPipelineStage.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source)
at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:536)
As you see, the PooledExecutor gets going, calling the Async Feeder,
which calls the underlying Sync Feeder and of the Pipeline goes. Up
to the NPE, that is... Several stages have already been successfully
executed, in parallel, up to this point. And, as I said, it doesn't
*always* fail, and when it does, it isn't *always* in the extract_fid
stage of type XpathExtractPipelineStage.
Hmm...I managed to catch one of the other ones as well. Here:
<2003-09-30 16:25:10,800> INFO [Thread-2] : dl_router:processStage(ticket:1064931905930,document:null)
java.lang.NullPointerException
at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source)
at com.babeldoc.core.VariableProcessor.expandString(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source)
at com.babeldoc.core.pipeline.stage.RouterPipelineStage.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.stage.CallStagePipelineStage.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source)
at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:536)
Any ideas?
I can provide almost any file from my configuration setup, if it
can be of any aid in tracking this down. I am currently stumped.
The reason I need the threading support to work is that in a few
of the later pipeline stages there can be substantial delays, in
case of which it would certainly be nice if the documents that
don't cause any delays can be happily processed in parallel, but
that's kinda obvious, I know. :-) Would be nice, though.
My personal guess at this time (I have done quite some digging in
the code, but obviously not yet enough) is that there seems to be
some kind of threading race in the code supporting the options.
Then again, that might be totally off, since I don't understand
it fully, yet.
Help?
Regards,
David Kinnvall
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
|
|
From: McDonald, B. <Bru...@ba...> - 2003-09-30 16:57:18
|
David, Ok, Lets find out exactly what the issue is - is it always the same configuration option that is failing or is some random option. Can you surround the getOptionList in XPathExtract with a try/catch and then dump the configuration data from the pipelinestage stage and the attributes on the document. regards, Bruce. -----Original Message----- From: David Kinnvall [mailto:dav...@al...] Sent: Tuesday, September 30, 2003 10:50 AM To: McDonald, Bruce; bab...@li... Subject: Re: [Babeldoc-devel] Multithreading problems... McDonald, Bruce wrote: > David, Hi Bruce! > It seems that the document is always null. Is this correct from your understanding of the pipeline/data? Heh...the debug output is junk, really...it's supposed to echo the *name* of the document, which isn't set, so outputs "null". (Note to self: Strip own debugging stuff before submitting mail next time. Less confusing. ;-) ) > this might be issue here. Nope, that's not it. The document is set alright. It's parsed and goes through several stages before bombing with the NPE. > regards, > Bruce. David |
|
From: Leech, J. <jl...@vi...> - 2003-09-30 16:38:41
|
If anyone is interested in some modifications I've made to the HttpClientPipelineStage, I'll commit them sooner rather than later. In short, they make it easier for babeldoc to screen-scrape websites and do other useful things. I've added the following: - an option to use JTidy on the response to convert html to xhtml - support for BASIC authentication - binary mode for downloading binary files -Jonathan |
|
From: Leech, J. <jl...@vi...> - 2003-09-30 16:28:27
|
Hmmm. My copy of ConfigInfo doesn't have that method -- I haven't done an update in a long while -- and cvs.sourceforge.net is refusing connections from me at the moment. At some point in the not-too-distant future I will update everything and put it through the multi-threaded babeldoc pressure-cooker that I've got going. -Jonathan -----Original Message----- From: McDonald, Bruce [mailto:Bru...@ba...] Sent: Tuesday, September 30, 2003 10:17 AM To: Leech, Jonathan; David Kinnvall; bab...@li... Subject: RE: [Babeldoc-devel] Multithreading problems... Jonathan, The pipeline stage gets created in the pipelinestagefactory code. But... There is a place in the configdata/configinfo code that actually creates suboptions when data is found that does not have a corresponding config option. I suspect that this is involved because the pipeline stage type that we are dealing with here are those with suboptions - just the kind that will be doing this kind of creating. The entry method for this is ConfigInfo.applyConfigData. This method takes the configuration data and applies it to the configuration options. It will create options if necessary. It will be necessary to synchronize either this method or on the data being fed to this method. Please experiment with this and report back. regards, Bruce. -----Original Message----- From: Leech, Jonathan [mailto:jl...@vi...] Sent: Tuesday, September 30, 2003 11:51 AM To: 'David Kinnvall'; bab...@li... Subject: RE: [Babeldoc-devel] Multithreading problems... I poked around the code a little bit. I didn't see where the PipelineStage gets created, or the config options get set, but its possible that more than one thread is setting the config options (suboptions) at the same time in the HashMap. None of the access to the ConfigOption.suboptions HashMap is synchronized (at least in the version of code I'm looking at, haven't done an update in a while). That's where I would start. -Jonathan -----Original Message----- From: David Kinnvall [mailto:dav...@al...] Sent: Tuesday, September 30, 2003 8:27 AM To: bab...@li... Subject: [Babeldoc-devel] Multithreading problems... Hi guys, I am still struggling with getting my pipeline(s) going together with multiple threads. I have changed my approach from using the threadpool pipeline processor to simply use the asynchronous scanner feeder with a poolSize=x, where x > 1, config as follows: feeder/config.properties: # Scanner feeder implementation scanner.type=asynchronous scanner.queue=disk scanner.queueDir=scanner/queue scanner.queueName=scanner scanner.poolSize=3 which gets loaded and configured by Babeldoc, as expected. Now, to trigger the problem I just have to supply the scanner with two or more documents to scan and submit to the pipeline for parallel processing and the processing dies _almost_ every time, with an NPE in VariableProcessor.mustExpand, for some reason. When it doesn't die, it does strange things further down the pipeline, indicating corrupted data payload in the document, that messes things up, albeit not causing an NPE this time. My particular processing consists of three pipelines, where two of them scan documents from separate sources, applies source-specific initial processing and then call a common main pipeline for the remaining processing tasks. I have had the processing fail in both the initial pipelines as well as in the later, common, one. The processing fails most commonly in an XpathExtractPipelineStage, but now and then it also fails in stages of other types. Example stacktrace (sorry for the formatting): (Oh, and the extra text after extract_fid: is just a little debug printout I added *after* observing the problem, to aid in my searching for the cause. It's not part of the problem, that is.) <2003-09-30 16:05:27,895> INFO [Thread-3] : extract_fid:processStage(ticket:1064930727756,document:null) java.lang.NullPointerException at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source) at com.babeldoc.core.VariableProcessor.expandString(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source) at com.babeldoc.core.pipeline.stage.XpathExtractPipelineStage.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source) at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source) at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source) at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Thread.java:536) As you see, the PooledExecutor gets going, calling the Async Feeder, which calls the underlying Sync Feeder and of the Pipeline goes. Up to the NPE, that is... Several stages have already been successfully executed, in parallel, up to this point. And, as I said, it doesn't *always* fail, and when it does, it isn't *always* in the extract_fid stage of type XpathExtractPipelineStage. Hmm...I managed to catch one of the other ones as well. Here: <2003-09-30 16:25:10,800> INFO [Thread-2] : dl_router:processStage(ticket:1064931905930,document:null) java.lang.NullPointerException at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source) at com.babeldoc.core.VariableProcessor.expandString(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source) at com.babeldoc.core.pipeline.stage.RouterPipelineStage.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source) at com.babeldoc.core.pipeline.stage.CallStagePipelineStage.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResult(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStageResults(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel ineStage(Unknown Source) at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn own Source) at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source) at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source) at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source) at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source) at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Thread.java:536) Any ideas? I can provide almost any file from my configuration setup, if it can be of any aid in tracking this down. I am currently stumped. The reason I need the threading support to work is that in a few of the later pipeline stages there can be substantial delays, in case of which it would certainly be nice if the documents that don't cause any delays can be happily processed in parallel, but that's kinda obvious, I know. :-) Would be nice, though. My personal guess at this time (I have done quite some digging in the code, but obviously not yet enough) is that there seems to be some kind of threading race in the code supporting the options. Then again, that might be totally off, since I don't understand it fully, yet. Help? Regards, David Kinnvall ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Babeldoc-devel mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-devel ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Babeldoc-devel mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-devel |
|
From: David K. <dav...@al...> - 2003-09-30 16:06:11
|
McDonald, Bruce wrote: > David, > > Ok, Lets find out exactly what the issue is - is it always the same configuration option that is failing or is some random option. Can you surround the getOptionList in XPathExtract with a try/catch and then dump the configuration data from the pipelinestage stage and the attributes on the document. Hmm...let's see...I managed to trigger a failure twice in the same stage with the same two input documents. The stage configuration is: # XPath Extract - get the file_id attribute extract_fid.stageType=XpathExtract extract_fid.nextStage=file_writer extract_fid.failOnError=false extract_fid.errorStage=error extract_fid.XPath.orig_file_id=/newsitem/@file_id And the (horribly unformatted) dump of doc attributes and pipeline stage configuration are attached for both runs. Initial observation (I think): - failOnError is null, as is - errorStage, as well as - XPath.orig_file_id in both cases. Correct? Any clues from that? > regards, > Bruce. David |
|
From: Leech, J. <jl...@vi...> - 2003-09-30 15:53:52
|
I poked around the code a little bit. I didn't see where the PipelineStage
gets created, or the config options get set, but its possible that more than
one thread is setting the config options (suboptions) at the same time in
the HashMap. None of the access to the ConfigOption.suboptions HashMap is
synchronized (at least in the version of code I'm looking at, haven't done
an update in a while). That's where I would start.
-Jonathan
-----Original Message-----
From: David Kinnvall [mailto:dav...@al...]
Sent: Tuesday, September 30, 2003 8:27 AM
To: bab...@li...
Subject: [Babeldoc-devel] Multithreading problems...
Hi guys,
I am still struggling with getting my pipeline(s) going together
with multiple threads. I have changed my approach from using the
threadpool pipeline processor to simply use the asynchronous
scanner feeder with a poolSize=x, where x > 1, config as follows:
feeder/config.properties:
# Scanner feeder implementation
scanner.type=asynchronous
scanner.queue=disk
scanner.queueDir=scanner/queue
scanner.queueName=scanner
scanner.poolSize=3
which gets loaded and configured by Babeldoc, as expected.
Now, to trigger the problem I just have to supply the scanner
with two or more documents to scan and submit to the pipeline
for parallel processing and the processing dies _almost_ every
time, with an NPE in VariableProcessor.mustExpand, for some
reason.
When it doesn't die, it does strange things further down the
pipeline, indicating corrupted data payload in the document,
that messes things up, albeit not causing an NPE this time.
My particular processing consists of three pipelines, where
two of them scan documents from separate sources, applies
source-specific initial processing and then call a common
main pipeline for the remaining processing tasks.
I have had the processing fail in both the initial pipelines
as well as in the later, common, one. The processing fails
most commonly in an XpathExtractPipelineStage, but now and
then it also fails in stages of other types.
Example stacktrace (sorry for the formatting):
(Oh, and the extra text after extract_fid: is just a little
debug printout I added *after* observing the problem, to
aid in my searching for the cause. It's not part of the
problem, that is.)
<2003-09-30 16:05:27,895> INFO [Thread-3] :
extract_fid:processStage(ticket:1064930727756,document:null)
java.lang.NullPointerException
at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source)
at com.babeldoc.core.VariableProcessor.expandString(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown
Source)
at
com.babeldoc.core.pipeline.stage.XpathExtractPipelineStage.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown
Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:536)
As you see, the PooledExecutor gets going, calling the Async Feeder,
which calls the underlying Sync Feeder and of the Pipeline goes. Up
to the NPE, that is... Several stages have already been successfully
executed, in parallel, up to this point. And, as I said, it doesn't
*always* fail, and when it does, it isn't *always* in the extract_fid
stage of type XpathExtractPipelineStage.
Hmm...I managed to catch one of the other ones as well. Here:
<2003-09-30 16:25:10,800> INFO [Thread-2] :
dl_router:processStage(ticket:1064931905930,document:null)
java.lang.NullPointerException
at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source)
at com.babeldoc.core.VariableProcessor.expandString(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown
Source)
at
com.babeldoc.core.pipeline.stage.RouterPipelineStage.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown
Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown
Source)
at
com.babeldoc.core.pipeline.stage.CallStagePipelineStage.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown
Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResult(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStageResults(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipel
ineStage(Unknown Source)
at
com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unkn
own Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:536)
Any ideas?
I can provide almost any file from my configuration setup, if it
can be of any aid in tracking this down. I am currently stumped.
The reason I need the threading support to work is that in a few
of the later pipeline stages there can be substantial delays, in
case of which it would certainly be nice if the documents that
don't cause any delays can be happily processed in parallel, but
that's kinda obvious, I know. :-) Would be nice, though.
My personal guess at this time (I have done quite some digging in
the code, but obviously not yet enough) is that there seems to be
some kind of threading race in the code supporting the options.
Then again, that might be totally off, since I don't understand
it fully, yet.
Help?
Regards,
David Kinnvall
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
|
|
From: Qin D. <qd...@cs...> - 2003-09-30 15:45:58
|
Bruce and David:
I downloaded the latest the version (babeldoc-bin-1.2.0-RC1) and modified
the env BABELDOC_HOME to point to c:\babeldoc-bin-1.2.0-RC1. Started a new
command shell and re-ran the test:
C:\babeldoc-bin-1.2.0-RC1\bin>babeldoc process -p test-pipeline -f c:
\qin\dummy.txt
I got the same error:
<2003-09-30 10:35:19,670> ERROR [main] : [PipelineFeeder.process]
com.babeldoc.core.pipeline.PipelineException: PipelineStage: entryStage not
found
at
com.babeldoc.core.pipeline.PipelineStageFactory.getEntryStageName(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
at com.babeldoc.core.pipeline.feeder.FeederFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.command.PipelineFeeder.process(Unknown
Source)
at com.babeldoc.core.pipeline.command.PipelineFeeder.execute(Unknown
Source)
at com.babeldoc.core.BabeldocCommand.executeCommand(Unknown Source)
at com.babeldoc.core.BabeldocCommand.<init>(Unknown Source)
at com.babeldoc.core.pipeline.command.PipelineFeeder.<init>(Unknown
Source)
at com.babeldoc.core.pipeline.command.PipelineFeeder.main(Unknown
Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at com.babeldoc.core.Main.handleCommand(Unknown Source)
at com.babeldoc.core.Main.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at com.babeldoc.init.Main.runBabeldocMain(Unknown Source)
at com.babeldoc.init.Main.main(Unknown Source)
Here is the modified setenv.bat file:
set BABELDOC_HOME=c:\babeldoc-bin-1.2.0-RC1
set BABELDOC_USER=c:\qin\config
set JAVA_HOME=D:\j2sdk1.4.1_03
set PATH=%BABELDOC_HOME%\bin;%JAVA_HOME%\bin;%PATH%
Please see the attached the qin.zip file.(See attached file: qin.zip)
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. NOTE: Regardless of content, this e-mail shall not operate to
bind CSC to any order or other contract unless pursuant to explicit written
agreement or government initiative expressly permitting the use of e-mail
for such purpose.
----------------------------------------------------------------------------------------
"McDonald,
Bruce" To: Qin Ding/ASD/CSC@CSC
<Bruce.McDonald cc: Babeldoc Developer List
@bankofamerica.c <bab...@li...>, David Glick
om> <dg...@co...>
Subject: RE: [Babeldoc-devel] Re: [Babeldoc-user]
09/29/03 10:31 Questions. Please Help
AM
Download the latest version and try again - I suspect your installation is
broken.
-----Original Message-----
From: Qin Ding [mailto:qd...@cs...]
Sent: Friday, September 26, 2003 1:10 PM
To: McDonald, Bruce
Cc: Babeldoc Developer List; David Glick
Subject: RE: [Babeldoc-devel] Re: [Babeldoc-user] Questions. Please Help
Bruce:
This is the test-pipeline.properties:
entryStage=initialize
initialize.stageType=Scripting
initialize.nextStage=null
initialize.failOnError=true
initialize.language=javascript
initialize.script=\
java.io.File file = new java.io.File(document.get("file_name"));\
TestManager.validate(file);
Here is the error:
<2003-09-26 11:05:11,960> INFO [Thread-1] : Processing document...
<2003-09-26 11:05:11,960> ERROR [Thread-1] : [AsynchronousFeeder$1.run]
com.babeldoc.core.pipeline.PipelineException: PipelineStage: entryStage not
foun
d
at
com.babeldoc.core.pipeline.PipelineStageFactory.getEntryStageName(Unk
nown Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown
Source)
at
com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Sou
rce)
at
com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown S
ource)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(
Unknown Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown So
urce)
at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown So
urce)
at java.lang.Thread.run(Thread.java:536)
Qin
----------------------------------------------------------------------------------------
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. NOTE: Regardless of content, this e-mail shall not operate to
bind CSC to any order or other contract unless pursuant to explicit written
agreement or government initiative expressly permitting the use of e-mail
for such purpose.
----------------------------------------------------------------------------------------
"McDonald,
Bruce" To: Qin Ding
<qd...@cs...>, David Glick
<Bruce.McDonald <dg...@co...>
@bankofamerica.c cc: Babeldoc Developer
List
om>
<bab...@li...>
Subject: RE:
[Babeldoc-devel] Re: [Babeldoc-user]
09/26/03 11:39 Questions. Please Help
AM
Huh?!??!
I dont understand the issue here.
Somewhere in your pipeline definition file, place the line:
entryStage=<<some pipeline stage name>>
Where the entryStage is pointing to the first pipeline stage in your
pipeline.
regards,
Bruce.
-----Original Message-----
From: Qin Ding [mailto:qd...@cs...]
Sent: Friday, September 26, 2003 12:14 PM
To: David Glick
Cc: Babeldoc Developer List
Subject: [Babeldoc-devel] Re: [Babeldoc-user] Questions. Please Help
David:
I don't have the Unix-to-Windows converter program; but I retyped the file
into wordpad and saved as text document. However, it is no possitive
effect. It still gives the same error, that is, entryStage not found. I
then copied the first line from src
example/scripting/pipeline/scannerxml.properties. Again, no use. The same
error occured no matter what I did. Seems like the application could not
recognize the "entryStage". By the way, I am using version bin-1.1.9-dev.
Any idea?
Qin
----------------------------------------------------------------------------------------
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. NOTE: Regardless of content, this e-mail shall not operate to
bind CSC to any order or other contract unless pursuant to explicit written
agreement or government initiative expressly permitting the use of e-mail
for such purpose.
----------------------------------------------------------------------------------------
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
|
|
From: Bill H. <Bil...@Ac...> - 2003-09-30 15:45:54
|
Hi, I sent this a week ago, and nobody has answered yet, so I'm sure that it has dropped off the bottom of the list. Can anyone help? I'm trying to connect Babeldoc to a web application and I can't seem to get it to work. I'm getting a little worried, as I have a tight deadline. I found the directions in modules/web/readme.txt for building and setting up babeldoc with my web server and it mostly worked, but there seems to be some conflict between my basic jakarta-tomcat- 4.1.24 setup in the commons-digester.jar. When I start with the tomcat one, it starts fine, when I start with the one copied over by babeldoc it fails pretty quickly with a class not found error. Anyway, I'm able to get the babeldoc console up, but when I try to access the journal, modules or pipelines it asks for a username and password and I don't know how to set those up. My tomcat currently uses database-based access control, so I assume that I just need a role defined, but don't know what that is. Can someone tell me? Anybody willing to spend 15 minutes on the phone with me? Thanks for any assistance, Bill ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Babeldoc-user mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-user ------- End of forwarded message ------- |
|
From: Qin D. <qd...@cs...> - 2003-09-30 15:08:42
|
David and Bruce:
Sorry, I was sick in the past a few days. Now I am well and back. Just a
moment ago, I retested the command line like this:
C:\babeldoc-bin-1.1.9-dev\bin>babeldoc process -p test-pipeline -f c:
\babeltest\
test\dummy.txt >. I got our old error again. So, the manual command line
and scanner run both received the same error: entryStage not found.
<2003-09-30 10:01:05,747> ERROR [main] : [PipelineFeeder.process]
com.babeldoc.core.pipeline.PipelineException: PipelineStage: entryStage not
found
at
com.babeldoc.core.pipeline.PipelineStageFactory.getEntryStageName(Unknown
Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
at com.babeldoc.core.pipeline.feeder.FeederFactory.process(Unknown
Source)
at com.babeldoc.core.pipeline.command.PipelineFeeder.process(Unknown
Source)
at com.babeldoc.core.pipeline.command.PipelineFeeder.execute(Unknown
Source)
at com.babeldoc.core.BabeldocCommand.executeCommand(Unknown Source)
at com.babeldoc.core.BabeldocCommand.<init>(Unknown Source)
at com.babeldoc.core.pipeline.command.PipelineFeeder.<init>(Unknown
Source)
at com.babeldoc.core.pipeline.command.PipelineFeeder.main(Unknown
Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at com.babeldoc.core.Main.handleCommand(Unknown Source)
at com.babeldoc.core.Main.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at com.babeldoc.init.Main.runBabeldocMain(Unknown Source)
at com.babeldoc.init.Main.main(Unknown Source)
Bruce: my directory structure is the same as yours because I am using
yours. So, let me download the babeldoc binary again. I'll then report
back to you. Thanks. Qin
----------------------------------------------------------------------------------------
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. NOTE: Regardless of content, this e-mail shall not operate to
bind CSC to any order or other contract unless pursuant to explicit written
agreement or government initiative expressly permitting the use of e-mail
for such purpose.
----------------------------------------------------------------------------------------
David Glick
<dglick To: "Qin Ding" <qd...@cs...>
@cox.net> cc: Babeldoc Developer List
<bab...@li...>
09/26/03 06:19 Subject: Re: [Babeldoc-devel] Re: [Babeldoc-user]
PM Questions. Please Help
Hi Qin,
The '-f dummy.txt' portion is to tell Babeldoc where to find the test file.
Did you cd to the directory before you ran the command? If not, you'll
need
to change the -f parameter to point to the dummy.txt file.
David
On Friday 26 September 2003 3:36 pm, Qin Ding wrote:
> David: To execute from command line, I got this error:
>
> <2003-09-26 17:35:07,411> ERROR [main] : [PipelineFeeder.execute]
Pipeline
> Error
> com.babeldoc.core.pipeline.PipelineException: Could not access the file:
> dummy.txt
> at
com.babeldoc.core.pipeline.command.PipelineFeeder.process(Unknown
> Source)
> at
com.babeldoc.core.pipeline.command.PipelineFeeder.execute(Unknown
> Source)
> at com.babeldoc.core.BabeldocCommand.executeCommand(Unknown Source)
> at com.babeldoc.core.BabeldocCommand.<init>(Unknown Source)
> at com.babeldoc.core.pipeline.command.PipelineFeeder.<init>(Unknown
> Source)
> at com.babeldoc.core.pipeline.command.PipelineFeeder.main(Unknown
> Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3
>9) at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
>l.java:25) at java.lang.reflect.Method.invoke(Method.java:324)
> at com.babeldoc.core.Main.handleCommand(Unknown Source)
> at com.babeldoc.core.Main.main(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3
>9) at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
>l.java:25) at java.lang.reflect.Method.invoke(Method.java:324)
> at com.babeldoc.init.Main.runBabeldocMain(Unknown Source)
> at com.babeldoc.init.Main.main(Unknown Source)
>
>
>
---------------------------------------------------------------------------
>-------------
>
> This is a PRIVATE message. If you are not the intended recipient, please
> delete without copying and kindly advise us by e-mail of the mistake in
> delivery. NOTE: Regardless of content, this e-mail shall not operate to
> bind CSC to any order or other contract unless pursuant to explicit
written
> agreement or government initiative expressly permitting the use of e-mail
> for such purpose.
>
---------------------------------------------------------------------------
>-------------
>
>
>
>
>
> David Glick
> <dglick To: "Qin Ding"
> <qd...@cs...> @cox.net> cc: Babeldoc Developer List
> <bab...@li...> 09/26/03 05:18 Subject:
> Re: [Babeldoc-devel] Re: [Babeldoc-user] PM
> Questions. Please Help
>
>
>
>
>
>
> Hi Qin,
>
> Please try testing submitting the file directly from the command line as
> follows:
>
> babeldoc process -p test-pipeline -f dummy.txt
>
> and see what error you get.
>
>
> David
>
> On Friday 26 September 2003 12:28 pm, Qin Ding wrote:
> > Bruce:
> >
> > I unzipped your qin-test.zip. And then I changed the BABELDOC_USER to
> > point to C:\qin\config. Not change anything else. Re-start babeldoc
in
> > command line: babeldoc scanner which started successfully:
> >
> > C:\babeldoc-bin-1.1.9-dev\bin>babeldoc scanner
> > <2003-09-26 14:16:29,374> INFO [main] : Initializing workers:
> > <2003-09-26 14:16:29,444> INFO [main] : dirscan (directory)
>
> configured...
>
> > <2003-09-26 14:16:29,444> INFO [main] : Starting workers...
> > <2003-09-26 14:16:29,444> INFO [main] : dirscan started...
> >
> > Then I drop the dummy.txt file into the inventory folder. Alas, I still
>
> got
>
> > the entryStage not found error!!!!
> > Is there anything that you did but I omitted?
> >
> >
> > <2003-09-26 14:16:58,937> INFO [Thread-1] : Processing document...
> > <2003-09-26 14:16:59,107> ERROR [Thread-1] :
[AsynchronousFeeder$1.run]
> > com.babeldoc.core.pipeline.PipelineException: PipelineStage: entryStage
>
> not
>
> > foun
> > d
> > at
> > com.babeldoc.core.pipeline.PipelineStageFactory.getEntryStageName(Unk
> > nown Source)
> > at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown
> > Source)
> > at
> > com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Sou
> > rce)
> > at
> > com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown S
> > ource)
> > at
> > com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(
> > Unknown Source)
> > at
> > com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown So
> > urce)
> > at
> > EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown So
> > urce)
> > at java.lang.Thread.run(Thread.java:536)
>
>
---------------------------------------------------------------------------
>
> >-------------
> >
> > This is a PRIVATE message. If you are not the intended recipient,
please
> > delete without copying and kindly advise us by e-mail of the mistake in
> > delivery. NOTE: Regardless of content, this e-mail shall not operate to
> > bind CSC to any order or other contract unless pursuant to explicit
>
> written
>
> > agreement or government initiative expressly permitting the use of
e-mail
> > for such purpose.
>
>
---------------------------------------------------------------------------
>
> >-------------
> >
> >
> >
> >
> >
> > "McDonald,
> > Bruce" To: Qin Ding
> > <qd...@cs...> <Bruce.McDonald cc: Babeldoc Developer
List
> > @bankofamerica.c <bab...@li...>, David
> > Glick om> <dg...@co...>
> > Subject: RE:
> > [Babeldoc-devel] Re: [Babeldoc-user] 09/26/03 01:28
Questions.
> > Please Help PM
> >
> >
> >
> >
> >
> >
> > Scanner works fine.
> >
> > I have included a zip of my recreation of your environment.
> >
> > Please investigate and report back to the list.
> >
> > Bruce.
> >
> >
> > -----Original Message-----
> > From: Qin Ding [mailto:qd...@cs...]
> > Sent: Friday, September 26, 2003 1:50 PM
> > To: McDonald, Bruce
> > Cc: Babeldoc Developer List; David Glick
> > Subject: RE: [Babeldoc-devel] Re: [Babeldoc-user] Questions. Please
Help
> >
> >
> >
> > Bruce: Thank you for your help.
> >
> > I use scanner.
> > Here is the config/pipeline/config.properties file:
> >
> > test-pipeline.type=simple
> > test-pipeline.configFile=test/test-pipeline
> >
> > Here is the config/scanner/config.properties file:
> >
> > dirscan.type=directory
> > dirscan.period=10000
> > dirscan.inDirectory=c:/babeltest/test/inventory
> > dirscan.doneDirectory=c:/babeltest/test/inventory/processed
> > dirscan.pipeline=test-pipeline
> >
> > Here is the test/test-pipeline.properties file:
> >
> > entryStage=initialize
> >
> > initialize.stageType=Scripting
> > initialize.nextStage=null
> > initialize.failOnError=true
> > initialize.language=javascript
> > initialize.script=\
> > java.io.File file = new java.io.File(document.get("file_name"));\
> > TestManager.validate(file);
> >
> > Qin
>
>
---------------------------------------------------------------------------
>
> >-------------
> >
> >
> > This is a PRIVATE message. If you are not the intended recipient,
please
> > delete without copying and kindly advise us by e-mail of the mistake in
> > delivery. NOTE: Regardless of content, this e-mail shall not operate to
> > bind CSC to any order or other contract unless pursuant to explicit
>
> written
>
> > agreement or government initiative expressly permitting the use of
e-mail
> > for such purpose.
>
>
---------------------------------------------------------------------------
>
> >-------------
> >
> >
> >
> >
> >
> >
> > "McDonald,
> > Bruce" To: Qin
> > Ding/ASD/CSC@CSC
> > <Bruce.McDonald cc: Babeldoc
>
> Developer
>
> > List
> > @bankofamerica.c
> > <bab...@li...>, David Glick
> > om> <dg...@co...>
> > Subject: RE:
> > [Babeldoc-devel] Re: [Babeldoc-user]
> > 09/26/03 12:26 Questions. Please Help
> > PM
> >
> >
> >
> >
> >
> >
> > Qin,
> >
> > Please show me the following:
> >
> > 1. How do you invoke babeldoc (the command line)
> > 2. Please show me the contents of the
config/pipeline/config.properties
> > file
> >
> > regards,
> > Bruce.
> >
> > -----Original Message-----
> > From: Qin Ding [mailto:qd...@cs...]
> > Sent: Friday, September 26, 2003 1:10 PM
> > To: McDonald, Bruce
> > Cc: Babeldoc Developer List; David Glick
> > Subject: RE: [Babeldoc-devel] Re: [Babeldoc-user] Questions. Please
Help
> >
> >
> >
> > Bruce:
> >
> > This is the test-pipeline.properties:
> >
> > entryStage=initialize
> >
> > initialize.stageType=Scripting
> > initialize.nextStage=null
> > initialize.failOnError=true
> > initialize.language=javascript
> > initialize.script=\
> > java.io.File file = new java.io.File(document.get("file_name"));\
> > TestManager.validate(file);
> >
> >
> > Here is the error:
> >
> > <2003-09-26 11:05:11,960> INFO [Thread-1] : Processing document...
> > <2003-09-26 11:05:11,960> ERROR [Thread-1] :
[AsynchronousFeeder$1.run]
> > com.babeldoc.core.pipeline.PipelineException: PipelineStage: entryStage
>
> not
>
> > foun
> > d
> > at
> > com.babeldoc.core.pipeline.PipelineStageFactory.getEntryStageName(Unk
> > nown Source)
> > at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown
> > Source)
> > at
> > com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Sou
> > rce)
> > at
> > com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown S
> > ource)
> > at
> > com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(
> > Unknown Source)
> > at
> > com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown So
> > urce)
> > at
> > EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown So
> > urce)
> > at java.lang.Thread.run(Thread.java:536)
> >
> >
> > Qin
>
>
---------------------------------------------------------------------------
>
> >-------------
> >
> >
> >
> > This is a PRIVATE message. If you are not the intended recipient,
please
> > delete without copying and kindly advise us by e-mail of the mistake in
> > delivery. NOTE: Regardless of content, this e-mail shall not operate to
> > bind CSC to any order or other contract unless pursuant to explicit
>
> written
>
> > agreement or government initiative expressly permitting the use of
e-mail
> > for such purpose.
>
>
---------------------------------------------------------------------------
>
> >-------------
> >
> >
> >
> >
> >
> >
> >
> > "McDonald,
> >
> > Bruce" To: Qin Ding
> > <qd...@cs...>, David Glick
> > <Bruce.McDonald <dg...@co...>
> >
> > @bankofamerica.c cc: Babeldoc
>
> Developer
>
> > List
> > om>
> > <bab...@li...>
> > Subject: RE:
> > [Babeldoc-devel] Re: [Babeldoc-user]
> > 09/26/03 11:39 Questions. Please Help
> >
> > AM
> >
> >
> >
> >
> >
> >
> >
> > Huh?!??!
> >
> > I dont understand the issue here.
> >
> > Somewhere in your pipeline definition file, place the line:
> >
> > entryStage=<<some pipeline stage name>>
> >
> > Where the entryStage is pointing to the first pipeline stage in your
> > pipeline.
> >
> > regards,
> > Bruce.
> >
> >
> > -----Original Message-----
> > From: Qin Ding [mailto:qd...@cs...]
> > Sent: Friday, September 26, 2003 12:14 PM
> > To: David Glick
> > Cc: Babeldoc Developer List
> > Subject: [Babeldoc-devel] Re: [Babeldoc-user] Questions. Please Help
> >
> >
> >
> > David:
> >
> > I don't have the Unix-to-Windows converter program; but I retyped the
>
> file
>
> > into wordpad and saved as text document. However, it is no possitive
> > effect. It still gives the same error, that is, entryStage not found.
>
> I
>
> > then copied the first line from src
> > example/scripting/pipeline/scannerxml.properties. Again, no use. The
same
> > error occured no matter what I did. Seems like the application could
not
> > recognize the "entryStage". By the way, I am using version
>
> bin-1.1.9-dev.
>
> > Any idea?
> >
> > Qin
>
>
---------------------------------------------------------------------------
>
> >-------------
> >
> >
> >
> >
> > This is a PRIVATE message. If you are not the intended recipient,
please
> > delete without copying and kindly advise us by e-mail of the mistake in
> > delivery. NOTE: Regardless of content, this e-mail shall not operate to
> > bind CSC to any order or other contract unless pursuant to explicit
>
> written
>
> > agreement or government initiative expressly permitting the use of
e-mail
> > for such purpose.
>
>
---------------------------------------------------------------------------
>
> >-------------
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > -------------------------------------------------------
> > This sf.net email is sponsored by:ThinkGeek
> > Welcome to geek heaven.
> > http://thinkgeek.com/sf
> > _______________________________________________
> > Babeldoc-devel mailing list
> > Bab...@li...
> > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > << Attachment Removed : qin-test.zip >>
--
David Glick
Transmit Consulting, Inc
619-475-4052
dg...@tr...
|
|
From: David K. <dav...@al...> - 2003-09-30 14:50:10
|
McDonald, Bruce wrote: > David, Hi Bruce! > It seems that the document is always null. Is this correct from your understanding of the pipeline/data? Heh...the debug output is junk, really...it's supposed to echo the *name* of the document, which isn't set, so outputs "null". (Note to self: Strip own debugging stuff before submitting mail next time. Less confusing. ;-) ) > this might be issue here. Nope, that's not it. The document is set alright. It's parsed and goes through several stages before bombing with the NPE. > regards, > Bruce. David |
|
From: <de...@us...> - 2003-09-30 14:37:27
|
Update of /cvsroot/babeldoc/babeldoc/modules/scanner/src/com/babeldoc/scanner
In directory sc8-pr-cvs1:/tmp/cvs-serv5783/modules/scanner/src/com/babeldoc/scanner
Modified Files:
ScannerWorker.java
Log Message:
Added log info message to inform about new document
Index: ScannerWorker.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/scanner/src/com/babeldoc/scanner/ScannerWorker.java,v
retrieving revision 1.26
retrieving revision 1.27
diff -C2 -d -r1.26 -r1.27
*** ScannerWorker.java 27 Sep 2003 00:49:34 -0000 1.26
--- ScannerWorker.java 30 Sep 2003 14:37:21 -0000 1.27
***************
*** 379,382 ****
--- 379,384 ----
try {
+ log.logInfo("Document " + attr.get(FILE_NAME_KEY) +
+ " enqueued for pipeline " + this.getPipelineName());
getFeeder().process(feed);
} catch (GeneralException e) {
|
|
From: <de...@us...> - 2003-09-30 14:36:41
|
Update of /cvsroot/babeldoc/babeldoc/modules/scanner/src/com/babeldoc/scanner/worker
In directory sc8-pr-cvs1:/tmp/cvs-serv5522/modules/scanner/src/com/babeldoc/scanner/worker
Modified Files:
FtpScanner.java
Log Message:
file_name should not contain path
Index: FtpScanner.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/scanner/src/com/babeldoc/scanner/worker/FtpScanner.java,v
retrieving revision 1.21
retrieving revision 1.22
diff -C2 -d -r1.21 -r1.22
*** FtpScanner.java 12 Sep 2003 01:09:17 -0000 1.21
--- FtpScanner.java 30 Sep 2003 14:36:37 -0000 1.22
***************
*** 309,313 ****
boolean success = ftpClient.retrieveFile(fileName, baos);
LogService.getInstance().logDebug("Retrieved file " + fileName);
!
//if file was retrieved without errors feed it!
if (success) {
--- 309,313 ----
boolean success = ftpClient.retrieveFile(fileName, baos);
LogService.getInstance().logDebug("Retrieved file " + fileName);
! String scanFileName = fileName.substring(ftpClient.printWorkingDirectory().length()+1);
//if file was retrieved without errors feed it!
if (success) {
***************
*** 318,322 ****
SCAN_PATH_KEY,
"ftp://" + ftpHost + fileName),
! new NameValuePair("file_name", fileName),
new NameValuePair(
SCAN_MIMETYPE_KEY,
--- 318,322 ----
SCAN_PATH_KEY,
"ftp://" + ftpHost + fileName),
! new NameValuePair("file_name", scanFileName),
new NameValuePair(
SCAN_MIMETYPE_KEY,
|
|
From: <de...@us...> - 2003-09-30 14:30:51
|
Update of /cvsroot/babeldoc/babeldoc/modules/scanner/src/com/babeldoc/scanner
In directory sc8-pr-cvs1:/tmp/cvs-serv4443/modules/scanner/src/com/babeldoc/scanner
Modified Files:
Scanner.java
Log Message:
Do not start scanner if no configuration found
Index: Scanner.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/scanner/src/com/babeldoc/scanner/Scanner.java,v
retrieving revision 1.29
retrieving revision 1.30
diff -C2 -d -r1.29 -r1.30
*** Scanner.java 17 Sep 2003 21:11:57 -0000 1.29
--- Scanner.java 30 Sep 2003 14:30:47 -0000 1.30
***************
*** 82,86 ****
import com.babeldoc.scanner.rmi.RemoteScannerBridge;
-
/**
* The main scanner entry point and babeldoc command implementer. This class
--- 82,85 ----
***************
*** 91,323 ****
*/
public class Scanner extends BabeldocCommand {
! private static LogService log = LogService.getInstance(Scanner.class.getName());
! private IFeeder feeder;
! private ScannerFactory factory;
! private ScannerWorkerScheduler scheduler;
!
! public static final String SCANNER_FEEDER = "scanner";
! public String scannerConfig;
! /**
! * Constructor with all the command line arguments
! *
! * @param args command line arguments
! */
! public Scanner(String[] args) {
! super("scanner", I18n.get("scanner.Scanner.toolDescription"), args);
! }
! /**
! * Handle the scanner command line
! *
! * @param commandLine
! */
! public void execute(CommandLine commandLine) {
! //super.execute(commandLine);
! // otherwise, continue with normal execution path
! if (commandLine.hasOption('s')) {
! scannerConfig = commandLine.getOptionValue('s');
! }
! try {
! start();
! } catch (ScannerException se) {
! this.stop();
! }
! }
! /**
! * Finish up processing - stop threads, etc
! *
! */
! public void finishUp() {
! this.stop();
! }
! /**
! * Main entry point
! *
! * @param args DOCUMENT ME!
! */
! public static void main(String[] args) {
! new Scanner(args);
! }
! /**
! * Get the list of workers as value objects
! *
! * @param workerName DOCUMENT ME!
! *
! * @return DOCUMENT ME!
! *
! * @throws ScannerException DOCUMENT ME!
! */
! public ScannerWorkerVO listWorker(String workerName)
! throws ScannerException {
! return factory.getWorker(workerName).getValueObject();
! }
! /**
! * List all workers in scanner
! *
! * @return Array of scanner workers
! */
! public ScannerWorkerVO[] listWorkers() {
! ScannerWorker[] workers = factory.getWorkers();
! ScannerWorkerVO[] result = new ScannerWorkerVO[workers.length];
! for (int i = 0; i < workers.length; i++) {
! result[i] = workers[i].getValueObject();
! }
! return result;
! }
! /**
! *
! */
! public void saveConfiguration() {
! // TODO Auto-generated method stub
! }
! /* (non-Javadoc)
! * @see com.babeldoc.core.BabeldocCommand#setupCommandLine(org.apache.commons.cli.Options)
! */
! public void setupCommandLine(Options options) {
! super.setupCommandLine(options);
! options.addOption(OptionBuilder.isRequired(false).hasArg(true)
! .withDescription(I18n.get(
! "scanner.Scanner.option.scanner")).withLongOpt("scanner").create('s'));
! options.addOption(OptionBuilder.isRequired(false).hasArg(false)
! .withDescription(I18n.get(
! "scanner.Scanner.option.rmi")).withLongOpt("rmi").create('r'));
! options.addOption(OptionBuilder.isRequired(false).hasArg(true)
! .withDescription(I18n.get(
! "scanner.Scanner.option.port")).withLongOpt("port").create('p'));
! }
! /**
! * Start the scanner workers
! *
! * @throws ScannerException
! */
! public void start() throws ScannerException {
! factory = ScannerFactory.getInstance(scannerConfig);
! // log.logInfo("Starting feeder...");
! // feeder = new AsynchronousFeeder(queue);
! try {
! setFeeder(FeederFactory.getInstance().getFeeder(SCANNER_FEEDER));
! } catch (GeneralException e) {
! getLog().logError(e);
! }
! //handleRemoteServer(-1);
! getLog().logInfo("Initializing workers:");
! ScannerWorker[] sw = factory.getWorkers();
! scheduler = new ScannerWorkerScheduler();
! getLog().logInfo("Starting workers...");
! for (int i = 0; i < sw.length; i++) {
! try {
! sw[i].setFeeder(getFeeder());
! scheduler.startWorker(sw[i]);
! } catch (Exception e) {
! getLog().logError("Error starting scanner worker " + sw[i].getName(), e);
! sw[i].stop();
! //stop();
! }
! }
! }
! /**
! * Start worker with the given name
! *
! * @param workerName name of the worker that should be started
! *
! * @throws ScannerException if worker does not exist or some other error
! * occures
! */
! public void startWorker(String workerName) throws ScannerException {
! factory.getWorker(workerName).start();
! }
! /**
! *
! */
! public void stop() {
! // if there is no scheduler, then we haven't started anything that needs to be
! // stopped...
! if (this.scheduler == null) {
! return;
! }
! getLog().logInfo("Shutting down...");
! getLog().logInfo("Stopping workers...");
! this.scheduler.stopAll();
! this.getFeeder().terminate();
! getLog().logInfo("Stopping feeder");
! }
! /**
! * Stop worker with the given name.
! *
! * @param workerName name of worker to be stopped
! *
! * @throws ScannerException if worker with given name does not exist or some
! * other error occures
! */
! public void stopWorker(String workerName) throws ScannerException {
! factory.getWorker(workerName).stop();
! }
! /**
! * Setup the remote server configuration.
! *
! * @param port DOCUMENT ME!
! */
! private void handleRemoteServer(int port) {
! if (System.getSecurityManager() == null) {
! System.setSecurityManager(new RMISecurityManager());
! }
! try {
! RemoteScannerBridge remoteScanner = null;
! if (port > 0) {
! remoteScanner = new RemoteScannerBridge(port);
! } else {
! remoteScanner = new RemoteScannerBridge();
! }
! remoteScanner.setScanner(this);
! if (port > 0) {
! RmiRegistry.rmiPort = port;
! }
! RmiRegistry.rebind(IRemoteScanner.RMI_NAME, remoteScanner);
! } catch (Exception e) {
! LogService.getInstance().logError(I18n.get("scanner.Scanner.error.rmi"), e);
! }
! }
! public static LogService getLog() {
! return log;
! }
! public static void setLog(LogService log) {
! Scanner.log = log;
! }
! public IFeeder getFeeder() {
! return feeder;
! }
! public void setFeeder(IFeeder feeder) {
! this.feeder = feeder;
! }
}
--- 90,343 ----
*/
public class Scanner extends BabeldocCommand {
! private static LogService log =
! LogService.getInstance(Scanner.class.getName());
! private IFeeder feeder;
! private ScannerFactory factory;
! private ScannerWorkerScheduler scheduler;
! public static final String SCANNER_FEEDER = "scanner";
! public String scannerConfig;
! /**
! * Constructor with all the command line arguments
! *
! * @param args command line arguments
! */
! public Scanner(String[] args) {
! super("scanner", I18n.get("scanner.Scanner.toolDescription"), args);
! }
! /**
! * Handle the scanner command line
! *
! * @param commandLine
! */
! public void execute(CommandLine commandLine) {
! //super.execute(commandLine);
! // otherwise, continue with normal execution path
! if (commandLine.hasOption('s')) {
! scannerConfig = commandLine.getOptionValue('s');
! }
! try {
! start();
! } catch (ScannerException se) {
! this.stop();
! }
! }
! /**
! * Finish up processing - stop threads, etc
! *
! */
! public void finishUp() {
! this.stop();
! }
! /**
! * Main entry point
! *
! * @param args DOCUMENT ME!
! */
! public static void main(String[] args) {
! new Scanner(args);
! }
! /**
! * Get the list of workers as value objects
! *
! * @param workerName DOCUMENT ME!
! *
! * @return DOCUMENT ME!
! *
! * @throws ScannerException DOCUMENT ME!
! */
! public ScannerWorkerVO listWorker(String workerName)
! throws ScannerException {
! return factory.getWorker(workerName).getValueObject();
! }
! /**
! * List all workers in scanner
! *
! * @return Array of scanner workers
! */
! public ScannerWorkerVO[] listWorkers() {
! ScannerWorker[] workers = factory.getWorkers();
! ScannerWorkerVO[] result = new ScannerWorkerVO[workers.length];
! for (int i = 0; i < workers.length; i++) {
! result[i] = workers[i].getValueObject();
! }
! return result;
! }
! /**
! *
! */
! public void saveConfiguration() {
! // TODO Auto-generated method stub
! }
! /* (non-Javadoc)
! * @see com.babeldoc.core.BabeldocCommand#setupCommandLine(org.apache.commons.cli.Options)
! */
! public void setupCommandLine(Options options) {
! super.setupCommandLine(options);
! options.addOption(
! OptionBuilder
! .isRequired(false)
! .hasArg(true)
! .withDescription(I18n.get("scanner.Scanner.option.scanner"))
! .withLongOpt("scanner")
! .create('s'));
! options.addOption(
! OptionBuilder
! .isRequired(false)
! .hasArg(false)
! .withDescription(I18n.get("scanner.Scanner.option.rmi"))
! .withLongOpt("rmi")
! .create('r'));
! options.addOption(
! OptionBuilder
! .isRequired(false)
! .hasArg(true)
! .withDescription(I18n.get("scanner.Scanner.option.port"))
! .withLongOpt("port")
! .create('p'));
! }
! /**
! * Start the scanner workers
! *
! * @throws ScannerException
! */
! public void start() throws ScannerException {
! factory = ScannerFactory.getInstance(scannerConfig);
! // log.logInfo("Starting feeder...");
! // feeder = new AsynchronousFeeder(queue);
! try {
! setFeeder(FeederFactory.getInstance().getFeeder(SCANNER_FEEDER));
! } catch (GeneralException e) {
! getLog().logError(e);
! }
! //handleRemoteServer(-1);
! getLog().logInfo("Initializing workers:");
! ScannerWorker[] sw = factory.getWorkers();
! if (sw.length > 0) {
! scheduler = new ScannerWorkerScheduler();
! getLog().logInfo("Starting workers...");
! for (int i = 0; i < sw.length; i++) {
! try {
! sw[i].setFeeder(getFeeder());
! scheduler.startWorker(sw[i]);
! } catch (Exception e) {
! getLog().logError(
! "Error starting scanner worker " + sw[i].getName(),
! e);
! sw[i].stop();
! //stop();
! }
! }
! } else {
! log.logInfo("No configured workers found!");
! stop();
! }
! }
! /**
! * Start worker with the given name
! *
! * @param workerName name of the worker that should be started
! *
! * @throws ScannerException if worker does not exist or some other error
! * occures
! */
! public void startWorker(String workerName) throws ScannerException {
! factory.getWorker(workerName).start();
! }
! /**
! *
! */
! public void stop() {
! if (this.scheduler != null) {
! getLog().logInfo("Shutting down...");
! getLog().logInfo("Stopping workers...");
! this.scheduler.stopAll();
! }
! if (this.getFeeder()!=null) {
! this.getFeeder().terminate();
! getLog().logInfo("Stopping feeder");
! }
!
! }
! /**
! * Stop worker with the given name.
! *
! * @param workerName name of worker to be stopped
! *
! * @throws ScannerException if worker with given name does not exist or some
! * other error occures
! */
! public void stopWorker(String workerName) throws ScannerException {
! factory.getWorker(workerName).stop();
! }
! /**
! * Setup the remote server configuration.
! *
! * @param port DOCUMENT ME!
! */
! private void handleRemoteServer(int port) {
! if (System.getSecurityManager() == null) {
! System.setSecurityManager(new RMISecurityManager());
! }
! try {
! RemoteScannerBridge remoteScanner = null;
! if (port > 0) {
! remoteScanner = new RemoteScannerBridge(port);
! } else {
! remoteScanner = new RemoteScannerBridge();
! }
! remoteScanner.setScanner(this);
! if (port > 0) {
! RmiRegistry.rmiPort = port;
! }
! RmiRegistry.rebind(IRemoteScanner.RMI_NAME, remoteScanner);
! } catch (Exception e) {
! LogService.getInstance().logError(
! I18n.get("scanner.Scanner.error.rmi"),
! e);
! }
! }
! public static LogService getLog() {
! return log;
! }
! public static void setLog(LogService log) {
! Scanner.log = log;
! }
! public IFeeder getFeeder() {
! return feeder;
! }
! public void setFeeder(IFeeder feeder) {
! this.feeder = feeder;
! }
}
|
|
From: David K. <dav...@al...> - 2003-09-30 14:27:04
|
Hi guys,
I am still struggling with getting my pipeline(s) going together
with multiple threads. I have changed my approach from using the
threadpool pipeline processor to simply use the asynchronous
scanner feeder with a poolSize=x, where x > 1, config as follows:
feeder/config.properties:
# Scanner feeder implementation
scanner.type=asynchronous
scanner.queue=disk
scanner.queueDir=scanner/queue
scanner.queueName=scanner
scanner.poolSize=3
which gets loaded and configured by Babeldoc, as expected.
Now, to trigger the problem I just have to supply the scanner
with two or more documents to scan and submit to the pipeline
for parallel processing and the processing dies _almost_ every
time, with an NPE in VariableProcessor.mustExpand, for some
reason.
When it doesn't die, it does strange things further down the
pipeline, indicating corrupted data payload in the document,
that messes things up, albeit not causing an NPE this time.
My particular processing consists of three pipelines, where
two of them scan documents from separate sources, applies
source-specific initial processing and then call a common
main pipeline for the remaining processing tasks.
I have had the processing fail in both the initial pipelines
as well as in the later, common, one. The processing fails
most commonly in an XpathExtractPipelineStage, but now and
then it also fails in stages of other types.
Example stacktrace (sorry for the formatting):
(Oh, and the extra text after extract_fid: is just a little
debug printout I added *after* observing the problem, to
aid in my searching for the cause. It's not part of the
problem, that is.)
<2003-09-30 16:05:27,895> INFO [Thread-3] : extract_fid:processStage(ticket:1064930727756,document:null)
java.lang.NullPointerException
at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source)
at com.babeldoc.core.VariableProcessor.expandString(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source)
at com.babeldoc.core.pipeline.stage.XpathExtractPipelineStage.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source)
at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:536)
As you see, the PooledExecutor gets going, calling the Async Feeder,
which calls the underlying Sync Feeder and of the Pipeline goes. Up
to the NPE, that is... Several stages have already been successfully
executed, in parallel, up to this point. And, as I said, it doesn't
*always* fail, and when it does, it isn't *always* in the extract_fid
stage of type XpathExtractPipelineStage.
Hmm...I managed to catch one of the other ones as well. Here:
<2003-09-30 16:25:10,800> INFO [Thread-2] : dl_router:processStage(ticket:1064931905930,document:null)
java.lang.NullPointerException
at com.babeldoc.core.VariableProcessor.mustExpand(Unknown Source)
at com.babeldoc.core.VariableProcessor.expandString(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.templatize(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.getOptionList(Unknown Source)
at com.babeldoc.core.pipeline.stage.RouterPipelineStage.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.stage.CallStagePipelineStage.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStage.processStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResult(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStageResults(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.processPipelineStage(Unknown Source)
at com.babeldoc.core.pipeline.processor.SyncPipelineStageProcessor.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineStageFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Source)
at com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown Source)
at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(Unknown Source)
at com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:536)
Any ideas?
I can provide almost any file from my configuration setup, if it
can be of any aid in tracking this down. I am currently stumped.
The reason I need the threading support to work is that in a few
of the later pipeline stages there can be substantial delays, in
case of which it would certainly be nice if the documents that
don't cause any delays can be happily processed in parallel, but
that's kinda obvious, I know. :-) Would be nice, though.
My personal guess at this time (I have done quite some digging in
the code, but obviously not yet enough) is that there seems to be
some kind of threading race in the code supporting the options.
Then again, that might be totally off, since I don't understand
it fully, yet.
Help?
Regards,
David Kinnvall
|
|
From: <de...@us...> - 2003-09-30 13:47:58
|
Update of /cvsroot/babeldoc/babeldoc/modules/scanner/src/com/babeldoc/scanner/worker
In directory sc8-pr-cvs1:/tmp/cvs-serv28582/modules/scanner/src/com/babeldoc/scanner/worker
Modified Files:
MailboxScanner.java
Log Message:
Check if connection still allive prior to retrieving e-mail
Index: MailboxScanner.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/scanner/src/com/babeldoc/scanner/worker/MailboxScanner.java,v
retrieving revision 1.24
retrieving revision 1.25
diff -C2 -d -r1.24 -r1.25
*** MailboxScanner.java 5 Sep 2003 13:28:29 -0000 1.24
--- MailboxScanner.java 30 Sep 2003 13:47:54 -0000 1.25
***************
*** 147,150 ****
--- 147,154 ----
//Folder folder = null;
try {
+ if (!store.isConnected()) {
+ getLog().logDebug("Connection closed. Reconnecting");
+ store.connect(host, username, password);
+ }
folder = store.getFolder(folder_);
folder.open(Folder.READ_WRITE);
|
|
From: <de...@us...> - 2003-09-30 13:46:29
|
Update of /cvsroot/babeldoc/babeldoc/modules/core/src/com/babeldoc/core/pipeline/stage
In directory sc8-pr-cvs1:/tmp/cvs-serv28240/modules/core/src/com/babeldoc/core/pipeline/stage
Modified Files:
FtpWriterPipelineStage.java
Log Message:
Changed exception message thrown when error occurs while uploading file
Index: FtpWriterPipelineStage.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/core/src/com/babeldoc/core/pipeline/stage/FtpWriterPipelineStage.java,v
retrieving revision 1.6
retrieving revision 1.7
diff -C2 -d -r1.6 -r1.7
*** FtpWriterPipelineStage.java 27 Jun 2003 02:19:59 -0000 1.6
--- FtpWriterPipelineStage.java 30 Sep 2003 13:46:21 -0000 1.7
***************
*** 188,192 ****
return super.processHelper();
} catch (Exception except) {
! throw new PipelineException("[FlatToXmlPipelineStage.process] Exception",
except);
}
--- 188,192 ----
return super.processHelper();
} catch (Exception except) {
! throw new PipelineException("Error while uploading file to ftp server",
except);
}
|
|
From: McDonald, B. <Bru...@ba...> - 2003-09-29 19:38:12
|
Download the latest version and try again - I suspect your installation is broken.
-----Original Message-----
From: Qin Ding [mailto:qd...@cs...]
Sent: Friday, September 26, 2003 1:10 PM
To: McDonald, Bruce
Cc: Babeldoc Developer List; David Glick
Subject: RE: [Babeldoc-devel] Re: [Babeldoc-user] Questions. Please Help
Bruce:
This is the test-pipeline.properties:
entryStage=initialize
initialize.stageType=Scripting
initialize.nextStage=null
initialize.failOnError=true
initialize.language=javascript
initialize.script=\
java.io.File file = new java.io.File(document.get("file_name"));\
TestManager.validate(file);
Here is the error:
<2003-09-26 11:05:11,960> INFO [Thread-1] : Processing document...
<2003-09-26 11:05:11,960> ERROR [Thread-1] : [AsynchronousFeeder$1.run]
com.babeldoc.core.pipeline.PipelineException: PipelineStage: entryStage not
foun
d
at
com.babeldoc.core.pipeline.PipelineStageFactory.getEntryStageName(Unk
nown Source)
at com.babeldoc.core.pipeline.PipelineFactory.process(Unknown
Source)
at
com.babeldoc.core.pipeline.PipelineFactoryFactory.process(Unknown Sou
rce)
at
com.babeldoc.core.pipeline.feeder.SynchronousFeeder.process(Unknown S
ource)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder.actuallyProcess(
Unknown Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown So
urce)
at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown So
urce)
at java.lang.Thread.run(Thread.java:536)
Qin
----------------------------------------------------------------------------------------
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. NOTE: Regardless of content, this e-mail shall not operate to
bind CSC to any order or other contract unless pursuant to explicit written
agreement or government initiative expressly permitting the use of e-mail
for such purpose.
----------------------------------------------------------------------------------------
"McDonald,
Bruce" To: Qin Ding <qd...@cs...>, David Glick
<Bruce.McDonald <dg...@co...>
@bankofamerica.c cc: Babeldoc Developer List
om> <bab...@li...>
Subject: RE: [Babeldoc-devel] Re: [Babeldoc-user]
09/26/03 11:39 Questions. Please Help
AM
Huh?!??!
I dont understand the issue here.
Somewhere in your pipeline definition file, place the line:
entryStage=<<some pipeline stage name>>
Where the entryStage is pointing to the first pipeline stage in your
pipeline.
regards,
Bruce.
-----Original Message-----
From: Qin Ding [mailto:qd...@cs...]
Sent: Friday, September 26, 2003 12:14 PM
To: David Glick
Cc: Babeldoc Developer List
Subject: [Babeldoc-devel] Re: [Babeldoc-user] Questions. Please Help
David:
I don't have the Unix-to-Windows converter program; but I retyped the file
into wordpad and saved as text document. However, it is no possitive
effect. It still gives the same error, that is, entryStage not found. I
then copied the first line from src
example/scripting/pipeline/scannerxml.properties. Again, no use. The same
error occured no matter what I did. Seems like the application could not
recognize the "entryStage". By the way, I am using version bin-1.1.9-dev.
Any idea?
Qin
----------------------------------------------------------------------------------------
This is a PRIVATE message. If you are not the intended recipient, please
delete without copying and kindly advise us by e-mail of the mistake in
delivery. NOTE: Regardless of content, this e-mail shall not operate to
bind CSC to any order or other contract unless pursuant to explicit written
agreement or government initiative expressly permitting the use of e-mail
for such purpose.
----------------------------------------------------------------------------------------
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
|
2 messages has been excluded from this view by a project administrator.
