Menu
▾
▴
babeldoc-devel — Babeldoc developer list
You can subscribe to this list here.
| 2003 |
Jan
|
Feb
(14) |
Mar
(107) |
Apr
(211) |
May
(93) |
Jun
(158) |
Jul
(159) |
Aug
(368) |
Sep
(188) |
Oct
(151) |
Nov
(115) |
Dec
(98) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2004 |
Jan
(25) |
Feb
|
Mar
(33) |
Apr
(28) |
May
(116) |
Jun
(2) |
Jul
(117) |
Aug
(19) |
Sep
(9) |
Oct
(2) |
Nov
|
Dec
(4) |
| 2005 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
(9) |
Dec
|
| 2006 |
Jan
|
Feb
|
Mar
(22) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2007 |
Jan
|
Feb
|
Mar
(6) |
Apr
|
May
|
Jun
|
Jul
|
Aug
(267) |
Sep
|
Oct
|
Nov
(6) |
Dec
(512) |
| 2008 |
Jan
(187) |
Feb
|
Mar
|
Apr
|
May
|
Jun
(3) |
Jul
(6) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
|
Dec
|
| 2012 |
Jan
|
Feb
|
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Michael A. <mic...@ze...> - 2003-11-05 17:45:34
|
Does this mean that we're going to branch 1.2 maintenance soon? Cheers... On Wed, 2003-11-05 at 15:06, David Kinnvall wrote: > McDonald, Bruce wrote: > > > Good. Great news about the bug. This is/was a big one. > > Yeah, it feels much better now. This allows me to get around > the delay issues I have in some of the subsequent pipeline > stage processing, which is very nice indeed. > > > Logging. There is a method called getLog which is peppered all over the code which gets a particular logger for the code. This is structured largely as a hierarchy based on the babeldoc package layout. Now, here is the thing: > > > > The base getInstance() actually tries to get a logger based on the name of the class that is calling it. It does this by inspecting the call stack until it can get the Fully Qualified Class Name (FQCN). (see LogService line 161). This and the log4j properies file can specify the exactly logging requirements for your class - you just need to add the appender (named either as the FQCN or a parent package) with the particular pattern that you want to log with. > > Hmm. Seems I haven't really understood how Babeldoc logs > I will take a second, closer, look and try to fix it the > configuration way. Thanks for the pointer(s). > > > Bruce. > > /David > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel |
|
From: David K. <dav...@al...> - 2003-11-05 15:07:12
|
McDonald, Bruce wrote: > Good. Great news about the bug. This is/was a big one. Yeah, it feels much better now. This allows me to get around the delay issues I have in some of the subsequent pipeline stage processing, which is very nice indeed. > Logging. There is a method called getLog which is peppered all over the code which gets a particular logger for the code. This is structured largely as a hierarchy based on the babeldoc package layout. Now, here is the thing: > > The base getInstance() actually tries to get a logger based on the name of the class that is calling it. It does this by inspecting the call stack until it can get the Fully Qualified Class Name (FQCN). (see LogService line 161). This and the log4j properies file can specify the exactly logging requirements for your class - you just need to add the appender (named either as the FQCN or a parent package) with the particular pattern that you want to log with. Hmm. Seems I haven't really understood how Babeldoc logs I will take a second, closer, look and try to fix it the configuration way. Thanks for the pointer(s). > Bruce. /David |
|
From: McDonald, B. <Bru...@ba...> - 2003-11-05 14:37:59
|
Good. Great news about the bug. This is/was a big one. Logging. There is a method called getLog which is peppered all over the code which gets a particular logger for the code. This is structured largely as a hierarchy based on the babeldoc package layout. Now, here is the thing: The base getInstance() actually tries to get a logger based on the name of the class that is calling it. It does this by inspecting the call stack until it can get the Fully Qualified Class Name (FQCN). (see LogService line 161). This and the log4j properies file can specify the exactly logging requirements for your class - you just need to add the appender (named either as the FQCN or a parent package) with the particular pattern that you want to log with. Bruce. -----Original Message----- From: David Kinnvall [mailto:dav...@al...] Sent: Wednesday, November 05, 2003 9:25 AM To: bab...@li... Cc: McDonald, Bruce Subject: Re: [Babeldoc-devel] Multithreading problems... McDonald, Bruce wrote: > You got it. It was two problems - one from the unsynchronized methods and another from the subdata configuration not getting set. I have run some initial test on my internal production setup, with no ill effects so far, I am very happy to say. The fix you made looks to be *it* as far as I am concerned. Great! For a completely different question/issue: I have noticed, partly when trying to aid in debugging this particular issue, that not very many of the logging points in Babeldoc use the possibility to provide a name through the LogService. This leads to the log output being sometimes hard to follow, as in finding out who exactly logged a particular message. I think it could be a useful task to review the use of the LogService and in particular the instances that look like this LogService.getInstance().logFoo(bar); and try to invent, say "baz", for each LogService user, and instead of the above, do this LogService.getInstance(baz).logFoo(bar); Some kind of naming hierarchy perhaps? E.g for pipelines and their respective stages a name like "PipelineName"."PipelineStageName" would lead to log output resembling <time> INFO [PipelineName.PipelineStageName] : <message> instead of the current <time> INFO [Thread-x] : <message> and so on, for the different parts of Babeldoc. To make it easy to use, I think direct LogService.getInstance() calls should be discouraged in most of the code and instead there could be made available a utility method getLogService() in which the proper setup for each class and instance is made. (getLogService would preferably be responsible for selecting a suitable log-name, and whatever else might be necessary, and could likely be made into a base-class method in many cases, e.g for PipelineStage implementations to use.) What do you think? Overkill, maybe? /David |
|
From: David K. <dav...@al...> - 2003-11-05 14:27:14
|
McDonald, Bruce wrote: > You got it. It was two problems - one from the unsynchronized methods and another from the subdata configuration not getting set. I have run some initial test on my internal production setup, with no ill effects so far, I am very happy to say. The fix you made looks to be *it* as far as I am concerned. Great! For a completely different question/issue: I have noticed, partly when trying to aid in debugging this particular issue, that not very many of the logging points in Babeldoc use the possibility to provide a name through the LogService. This leads to the log output being sometimes hard to follow, as in finding out who exactly logged a particular message. I think it could be a useful task to review the use of the LogService and in particular the instances that look like this LogService.getInstance().logFoo(bar); and try to invent, say "baz", for each LogService user, and instead of the above, do this LogService.getInstance(baz).logFoo(bar); Some kind of naming hierarchy perhaps? E.g for pipelines and their respective stages a name like "PipelineName"."PipelineStageName" would lead to log output resembling <time> INFO [PipelineName.PipelineStageName] : <message> instead of the current <time> INFO [Thread-x] : <message> and so on, for the different parts of Babeldoc. To make it easy to use, I think direct LogService.getInstance() calls should be discouraged in most of the code and instead there could be made available a utility method getLogService() in which the proper setup for each class and instance is made. (getLogService would preferably be responsible for selecting a suitable log-name, and whatever else might be necessary, and could likely be made into a base-class method in many cases, e.g for PipelineStage implementations to use.) What do you think? Overkill, maybe? /David |
|
From: McDonald, B. <Bru...@ba...> - 2003-11-05 13:53:48
|
You got it. It was two problems - one from the unsynchronized methods and another from the subdata configuration not getting set. -----Original Message----- From: David Kinnvall [mailto:dav...@al...] Sent: Wednesday, November 05, 2003 7:59 AM To: bab...@li... Cc: McDonald, Bruce Subject: Re: [Babeldoc-devel] Multithreading problems... Bruce, list, McDonald, Bruce wrote: > It looks like I might have a fix here. I am inclined to agree! See below. > It is: > > 1. Make the following two methods synchronized (at the method level): > > SimplePipelineStageFactory.getResolver > PipelineFactory.getPipelineStageFactory Done. > 2. Add the line to: ConfigInfo.applyConfigValue(IConfigData data, ConfigOption configoption): > > After the line: subconfigoption = createDynamicOption(subdata, key); > add: subconfigoption.setConfigData(subdata); Done. (But changed "subdata" to "option" in my local copy.) > Please confirm! Success! I am unable to reproduce the aformentioned threading problems with the above changes in place. Great work, Bruce!! (My tests so far have been with the isolated test-case code. I will now continue with my "real" setup with multiple pipes and call-stages etc. and verify that this also works, but I feel confident that it will. Will report back if it doesn't.) > regards, > Bruce. Regards, David |
|
From: David K. <dav...@al...> - 2003-11-05 12:59:12
|
Bruce, list, McDonald, Bruce wrote: > It looks like I might have a fix here. I am inclined to agree! See below. > It is: > > 1. Make the following two methods synchronized (at the method level): > > SimplePipelineStageFactory.getResolver > PipelineFactory.getPipelineStageFactory Done. > 2. Add the line to: ConfigInfo.applyConfigValue(IConfigData data, ConfigOption configoption): > > After the line: subconfigoption = createDynamicOption(subdata, key); > add: subconfigoption.setConfigData(subdata); Done. (But changed "subdata" to "option" in my local copy.) > Please confirm! Success! I am unable to reproduce the aformentioned threading problems with the above changes in place. Great work, Bruce!! (My tests so far have been with the isolated test-case code. I will now continue with my "real" setup with multiple pipes and call-stages etc. and verify that this also works, but I feel confident that it will. Will report back if it doesn't.) > regards, > Bruce. Regards, David |
|
From: McDonald, B. <Bru...@ba...> - 2003-11-04 21:59:56
|
It looks like I might have a fix here.
It is:
1. Make the following two methods synchronized (at the method level):
SimplePipelineStageFactory.getResolver
PipelineFactory.getPipelineStageFactory
2. Add the line to: ConfigInfo.applyConfigValue(IConfigData data, ConfigOption configoption):
After the line: subconfigoption = createDynamicOption(subdata, key);
add: subconfigoption.setConfigData(subdata);
Please confirm!
regards,
Bruce.
-----Original Message-----
From: McDonald, Bruce
Sent: Tuesday, November 04, 2003 11:01 AM
To: McDonald, Bruce; David Kinnvall;
bab...@li...
Subject: RE: [Babeldoc-devel] Multithreading problems...
Another data point...
If I synchronize the getConfig methods on the ConfigService class (as opposed to the internal synchronization) then I can run the scanner
with only the first and second group (each group has 4 files) of data files causing an exception. All subsequent runs are perfect.
go figure...
Bruce.
-----Original Message-----
From: McDonald, Bruce
Sent: Tuesday, November 04, 2003 10:19 AM
To: David Kinnvall; bab...@li...
Subject: RE: [Babeldoc-devel] Multithreading problems...
Here is complete example - ready to test.
Start the scanner (babeldoc scanner)
and then copy in the data?.xml files to the input directory and watch it
break.
We need some help here - Lets just say some giving back from however babeldoc might have helped you in the past.
-----Original Message-----
From: McDonald, Bruce
Sent: Tuesday, November 04, 2003 9:38 AM
To: David Kinnvall; bab...@li...
Subject: RE: [Babeldoc-devel] Multithreading problems...
David,
Thank you - I will be working on this. I would appreciate any help that we can get here.
THIS IS THE LAST BIG ISSUE BEFORE BABELDOC 1.2 GOES FINAL.
PLEASE HELP.
regards,
Bruce.
-----Original Message-----
From: David Kinnvall [mailto:dav...@al...]
Sent: Tuesday, November 04, 2003 9:25 AM
To: bab...@li...
Subject: Re: [Babeldoc-devel] Multithreading problems...
Hi everybody!
David Kinnvall (me) wrote at a previous time:
[stuff about my multithreading problems and the lack of datapoints]
I now have a very small setup that triggers the problem, suggesting
that there is nothing really particularly strange about my setup,
but rather that it should be reproducible (debuggable) for others.
Interesting datapoint: The *first* thread seems to always survive.
It is the remaining one(s) that crashes, sooner or later.
Relevant config snippets to (hopefully) be able to reproduce:
Setup a pipeline called whatever and a directory scanner that
feeds it, using the configs outlined below. Adjust as necessary
for your environment(s).
feeder/config.properties:
=========================
# Scanner feeder implementation
scanner.type=asynchronous
#scanner.type=synchronous
scanner.queue=disk
scanner.queueDir=scanner/queue
scanner.queueName=scanner
scanner.poolSize=3
pipeline/whatever/whatever.properties:
======================================
entryStage=entry
entry.stageType=Null
entry.nextStage=enricher
entry.tracked=false
enricher.stageType=XpathExtract
enricher.nextStage=writer
enricher.XPath.foo=/foo/@name
writer.stageType=FileWriter
writer.nextStage=null
writer.outputFile=output/whatever/${ticket.getValue()}_${document.get("foo")}.xml
Input document(s) - create a couple of these, named <anything>.xml:
===================================================================
<?xml version="1.0" encoding="iso-8859-1" standalone="no" ?>
<foo name="bar1"/>
Now, armed with an active Babeldoc instance, using config as
above, give a bunch (say 4) of input xml documents of the above
form to the scanner. For me it (almost) every time bombs for the
second and third threads in the enricher stage. You might have to
retry a few times, but for me it is very consistent, with an NPE
in VariableProcessor.mustExpand()
Important: You need to give multiple input documents to the
scanner at once, to trigger the problem *quickly* ...but...
However: It is certainly possible to trigger it with one single
document at a time as well! Just keep feeding the scanner+pipeline
one of the documents, watch it process it (successfully at least
once), and keep going, until a different thread from the first
gets to work on it. Boom.
The first thread seems to correctly get its configuration for
pipeline stages et al, but the subsequent configurations do
have a tendency to lack values, particularly, as Bruce noticed,
the ones of "complex" nature. A dump of the config data for an
XpathExtract stage, as configured in the snippet above, during
exception handling of one of the NPEs shows this (formatted)
com.babeldoc.core.option.ConfigData@11c2b67[
name=enricher,value=<null>,children={
stageType=com.babeldoc.core.option.ConfigData@659db7[
name=stageType,value=XpathExtract,children=<null>
],
XPath=com.babeldoc.core.option.ConfigData@1556d12[
name=XPath,value=<null>,children={
foo=com.babeldoc.core.option.ConfigData@edf389[
name=foo,value=<null>,children=<null>
]
}
],
nextStage=com.babeldoc.core.option.ConfigData@16be68f[
name=nextStage,value=writer,children=<null>
]
}
]
from which we learn that what should have been
enricher.XPath.foo=/foo/@name
has instead become
enricher.XPath.foo=<null>
which wasn't expected by the pipeline stage. Hence the NPE.
As before, I need help in resolving this. I have so far been
utterly unable to find a solution on my own. I hope that the
additional data in the above text might be of some assistance
for others to have a go at the problem as well, or at least
verify that the problem exists outside of my particular setup.
Best regards,
David
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
Does SourceForge.net help you be more productive? Does it
help you create better code? SHARE THE LOVE, and help us help
YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
Does SourceForge.net help you be more productive? Does it
help you create better code? SHARE THE LOVE, and help us help
YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
Does SourceForge.net help you be more productive? Does it
help you create better code? SHARE THE LOVE, and help us help
YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
|
|
From: David K. <dav...@al...> - 2003-11-04 21:55:06
|
Hi Dejan! Dejan Krsmanovic wrote: > David, > how difficult is to extend FtpScanner to use SFTP > protocol? In my opinion it would be better to have one > scanner with one or more new config options that would > enable SFTP support than having new Scanner with > similiar functionality... I contemplated doing that at first, but then I got caught up with the inner workings of the SFTP protocol, sessions, channels and whatnot, and it was about then that I began wandering off from the original FTP scanner structure and more-or-less arrived at something rather (a bit) different. Then again, I guess it would be possible to refactor both in one sweep and make them parts of the same whole. That would indeed be pretty nice, especially as the below... > On Commons VFS, I really like that project and I am > hoping that it will leave sandbox soon. In fact, I had > plans to implement VfsScanner/pipeline stage that > would enable using one scanner/stage for all file > systems supported by Commons VFS. ...sounds *very* interesting indeed. A VfsScanner with (perhaps?) VfsReader and VfsWriter (e.g) PipelineStage implementations would be very useful indeed. I like it! (The Jakarta SFTP Vfs is a bit innefficient in its use of channels, though - only a pool of one active and one idle - one of my changes was to refactor that detail in an effort to make especially the recursive scanning more speedy. Then again, that is probably a consequence from the different underlying filesystem abstraction of the Vfs as compared to the more connection-oriented scanner code. Can probably be re-designed as well. We'll see.) Anyway: - I need the code together with 1.2 for production use, but there is no problem for me doing that with this particular scanner in my own module for some time. - Noone else has expressed any immediate need for an SFTP scanner at this time, so there is no rush... - I like your idea (very much) regarding VFS support more generally in Babeldoc, and I would be interested in contributing code to that end during 1.3 timeframe. > I guess it is time for 1.3 CVS branch. Or this could > be included in 1.2? Anyway, I really think it is time > for you to become commiter! Bruce, we should put David > to developer list regardless his decision! ;-) He can > say yes later! Uhm...heh... *blush* :-) > Dejan /David |
|
From: McDonald, B. <Bru...@ba...> - 2003-11-04 21:24:25
|
Thats right Dejan - I can tie the blindfold - you can hold the pistol!! -----Original Message----- From: Dejan Krsmanovic [mailto:dej...@ya...] Sent: Tuesday, November 04, 2003 4:18 PM To: bab...@li... Subject: Re: [Babeldoc-devel] On a brighter note (SFTP Scanner Impl) ... David, how difficult is to extend FtpScanner to use SFTP protocol? In my opinion it would be better to have one scanner with one or more new config options that would enable SFTP support than having new Scanner with similiar functionality... On Commons VFS, I really like that project and I am hoping that it will leave sandbox soon. In fact, I had plans to implement VfsScanner/pipeline stage that would enable using one scanner/stage for all file systems supported by Commons VFS. I guess it is time for 1.3 CVS branch. Or this could be included in 1.2? Anyway, I really think it is time for you to become commiter! Bruce, we should put David to developer list regardless his decision! ;-) He can say yes later! Dejan --- David Kinnvall <dav...@al...> wrote: > Hi again, everybody! > > I felt like tossing off a slightly more happy note > to the > list, in addition to my threading problems... :-) > Anyway: > > I have put together a new little scanner > implementation, > that I needed for my current project at work, namely > an > SFTP scanner. Initially based on the current FTP > scanner > but subsequently restructured a bit, and using ideas > from > the Jakarta Commons VFS sandbox project and their > SFTP > implementation, with the JSch library underneath > (same as > used by the Jakarta VFS). > > Highlights: > - Overall functionality pretty much the same as for > the > FTP scanner, including remote move from an > "incoming" > to a "processed" folder, as well as a local > backup, > recursive scanning etc. > - No compression support (could not get it to work > in a > reliable way. Yet.) > - Initially only tested password-based > authentication, > but planning on doing public-key-based auth as > well. > - Opens several (currently capped at fixed number of > 5, > but that will be configurable) channels to the > SFTP > server, for *much* better scanning performance. > - Not I18N-enabled yet, but easy enough to do. > > Underlying library and inspirational code all using > a > compatible license for incorporation inside > Babeldoc. > > I have it as part of a company specific module, at > this > time, but I would be happy to submit it for review > and > consideration, and in that case *with* I18N done... > It > could certainly use a bit more widespread testing > too. > > Anybody interested? If so, recommended next step? > > /David > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback > Program. > Does SourceForge.net help you be more productive? > Does it > help you create better code? SHARE THE LOVE, and > help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel __________________________________ Do you Yahoo!? Protect your identity with Yahoo! Mail AddressGuard http://antispam.yahoo.com/whatsnewfree ------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ _______________________________________________ Babeldoc-devel mailing list Bab...@li... https://lists.sourceforge.net/lists/listinfo/babeldoc-devel |
|
From: Dejan K. <dej...@ya...> - 2003-11-04 21:17:39
|
David, how difficult is to extend FtpScanner to use SFTP protocol? In my opinion it would be better to have one scanner with one or more new config options that would enable SFTP support than having new Scanner with similiar functionality... On Commons VFS, I really like that project and I am hoping that it will leave sandbox soon. In fact, I had plans to implement VfsScanner/pipeline stage that would enable using one scanner/stage for all file systems supported by Commons VFS. I guess it is time for 1.3 CVS branch. Or this could be included in 1.2? Anyway, I really think it is time for you to become commiter! Bruce, we should put David to developer list regardless his decision! ;-) He can say yes later! Dejan --- David Kinnvall <dav...@al...> wrote: > Hi again, everybody! > > I felt like tossing off a slightly more happy note > to the > list, in addition to my threading problems... :-) > Anyway: > > I have put together a new little scanner > implementation, > that I needed for my current project at work, namely > an > SFTP scanner. Initially based on the current FTP > scanner > but subsequently restructured a bit, and using ideas > from > the Jakarta Commons VFS sandbox project and their > SFTP > implementation, with the JSch library underneath > (same as > used by the Jakarta VFS). > > Highlights: > - Overall functionality pretty much the same as for > the > FTP scanner, including remote move from an > "incoming" > to a "processed" folder, as well as a local > backup, > recursive scanning etc. > - No compression support (could not get it to work > in a > reliable way. Yet.) > - Initially only tested password-based > authentication, > but planning on doing public-key-based auth as > well. > - Opens several (currently capped at fixed number of > 5, > but that will be configurable) channels to the > SFTP > server, for *much* better scanning performance. > - Not I18N-enabled yet, but easy enough to do. > > Underlying library and inspirational code all using > a > compatible license for incorporation inside > Babeldoc. > > I have it as part of a company specific module, at > this > time, but I would be happy to submit it for review > and > consideration, and in that case *with* I18N done... > It > could certainly use a bit more widespread testing > too. > > Anybody interested? If so, recommended next step? > > /David > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback > Program. > Does SourceForge.net help you be more productive? > Does it > help you create better code? SHARE THE LOVE, and > help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel __________________________________ Do you Yahoo!? Protect your identity with Yahoo! Mail AddressGuard http://antispam.yahoo.com/whatsnewfree |
|
From: David K. <dav...@al...> - 2003-11-04 19:45:08
|
Hi again, everybody! I felt like tossing off a slightly more happy note to the list, in addition to my threading problems... :-) Anyway: I have put together a new little scanner implementation, that I needed for my current project at work, namely an SFTP scanner. Initially based on the current FTP scanner but subsequently restructured a bit, and using ideas from the Jakarta Commons VFS sandbox project and their SFTP implementation, with the JSch library underneath (same as used by the Jakarta VFS). Highlights: - Overall functionality pretty much the same as for the FTP scanner, including remote move from an "incoming" to a "processed" folder, as well as a local backup, recursive scanning etc. - No compression support (could not get it to work in a reliable way. Yet.) - Initially only tested password-based authentication, but planning on doing public-key-based auth as well. - Opens several (currently capped at fixed number of 5, but that will be configurable) channels to the SFTP server, for *much* better scanning performance. - Not I18N-enabled yet, but easy enough to do. Underlying library and inspirational code all using a compatible license for incorporation inside Babeldoc. I have it as part of a company specific module, at this time, but I would be happy to submit it for review and consideration, and in that case *with* I18N done... It could certainly use a bit more widespread testing too. Anybody interested? If so, recommended next step? /David |
|
From: David K. <dav...@al...> - 2003-11-04 19:26:22
|
Hi Bruce! McDonald, Bruce wrote: > Another data point... Great! (Well, I am happy to see that the problem wasn't only occuring in my own setup, since I would then be even *more* stumped than I am currently. :-) Now that the problem has been confirmed elsewhere I feel much better about the whole thing. Then again, I feel sorry for causing a delay in the 1.2 release. Oh, well, the things is pretty serious...) > If I synchronize the getConfig methods on the ConfigService class (as opposed to the internal synchronization) then I can run the scanner > with only the first and second group (each group has 4 files) of data files causing an exception. All subsequent runs are perfect. Hmm. How many threads are doing work in the subsequent runs? > go figure... Indeed... As an added data point (not exactly new, but anyway), I can have the problem occur in a multi-pipeline setup as well, in which I have two pipelines doing initial processing for two separate incoming data-sources, and when done they both call a third pipeline for common processing. I have had the crash occur in the third, common, pipeline. So, the problem travels across Call-stages as well. FWIW. I am available (during Swedish working hours, at least) for running more or less whatever tests anybody can come up with that are worth running, if it helps. I can do and test code changes as well, as long as I can grasp what they intend to do. :-) Right now, I feel that I have put in synchronization both here and there to no avail, so I am *obviously* missing the target. Hints to get me on the right track are welcome! > Bruce. /David |
|
From: McDonald, B. <Bru...@ba...> - 2003-11-04 16:02:54
|
Another data point...
If I synchronize the getConfig methods on the ConfigService class (as opposed to the internal synchronization) then I can run the scanner
with only the first and second group (each group has 4 files) of data files causing an exception. All subsequent runs are perfect.
go figure...
Bruce.
-----Original Message-----
From: McDonald, Bruce
Sent: Tuesday, November 04, 2003 10:19 AM
To: David Kinnvall; bab...@li...
Subject: RE: [Babeldoc-devel] Multithreading problems...
Here is complete example - ready to test.
Start the scanner (babeldoc scanner)
and then copy in the data?.xml files to the input directory and watch it
break.
We need some help here - Lets just say some giving back from however babeldoc might have helped you in the past.
-----Original Message-----
From: McDonald, Bruce
Sent: Tuesday, November 04, 2003 9:38 AM
To: David Kinnvall; bab...@li...
Subject: RE: [Babeldoc-devel] Multithreading problems...
David,
Thank you - I will be working on this. I would appreciate any help that we can get here.
THIS IS THE LAST BIG ISSUE BEFORE BABELDOC 1.2 GOES FINAL.
PLEASE HELP.
regards,
Bruce.
-----Original Message-----
From: David Kinnvall [mailto:dav...@al...]
Sent: Tuesday, November 04, 2003 9:25 AM
To: bab...@li...
Subject: Re: [Babeldoc-devel] Multithreading problems...
Hi everybody!
David Kinnvall (me) wrote at a previous time:
[stuff about my multithreading problems and the lack of datapoints]
I now have a very small setup that triggers the problem, suggesting
that there is nothing really particularly strange about my setup,
but rather that it should be reproducible (debuggable) for others.
Interesting datapoint: The *first* thread seems to always survive.
It is the remaining one(s) that crashes, sooner or later.
Relevant config snippets to (hopefully) be able to reproduce:
Setup a pipeline called whatever and a directory scanner that
feeds it, using the configs outlined below. Adjust as necessary
for your environment(s).
feeder/config.properties:
=========================
# Scanner feeder implementation
scanner.type=asynchronous
#scanner.type=synchronous
scanner.queue=disk
scanner.queueDir=scanner/queue
scanner.queueName=scanner
scanner.poolSize=3
pipeline/whatever/whatever.properties:
======================================
entryStage=entry
entry.stageType=Null
entry.nextStage=enricher
entry.tracked=false
enricher.stageType=XpathExtract
enricher.nextStage=writer
enricher.XPath.foo=/foo/@name
writer.stageType=FileWriter
writer.nextStage=null
writer.outputFile=output/whatever/${ticket.getValue()}_${document.get("foo")}.xml
Input document(s) - create a couple of these, named <anything>.xml:
===================================================================
<?xml version="1.0" encoding="iso-8859-1" standalone="no" ?>
<foo name="bar1"/>
Now, armed with an active Babeldoc instance, using config as
above, give a bunch (say 4) of input xml documents of the above
form to the scanner. For me it (almost) every time bombs for the
second and third threads in the enricher stage. You might have to
retry a few times, but for me it is very consistent, with an NPE
in VariableProcessor.mustExpand()
Important: You need to give multiple input documents to the
scanner at once, to trigger the problem *quickly* ...but...
However: It is certainly possible to trigger it with one single
document at a time as well! Just keep feeding the scanner+pipeline
one of the documents, watch it process it (successfully at least
once), and keep going, until a different thread from the first
gets to work on it. Boom.
The first thread seems to correctly get its configuration for
pipeline stages et al, but the subsequent configurations do
have a tendency to lack values, particularly, as Bruce noticed,
the ones of "complex" nature. A dump of the config data for an
XpathExtract stage, as configured in the snippet above, during
exception handling of one of the NPEs shows this (formatted)
com.babeldoc.core.option.ConfigData@11c2b67[
name=enricher,value=<null>,children={
stageType=com.babeldoc.core.option.ConfigData@659db7[
name=stageType,value=XpathExtract,children=<null>
],
XPath=com.babeldoc.core.option.ConfigData@1556d12[
name=XPath,value=<null>,children={
foo=com.babeldoc.core.option.ConfigData@edf389[
name=foo,value=<null>,children=<null>
]
}
],
nextStage=com.babeldoc.core.option.ConfigData@16be68f[
name=nextStage,value=writer,children=<null>
]
}
]
from which we learn that what should have been
enricher.XPath.foo=/foo/@name
has instead become
enricher.XPath.foo=<null>
which wasn't expected by the pipeline stage. Hence the NPE.
As before, I need help in resolving this. I have so far been
utterly unable to find a solution on my own. I hope that the
additional data in the above text might be of some assistance
for others to have a go at the problem as well, or at least
verify that the problem exists outside of my particular setup.
Best regards,
David
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
Does SourceForge.net help you be more productive? Does it
help you create better code? SHARE THE LOVE, and help us help
YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
Does SourceForge.net help you be more productive? Does it
help you create better code? SHARE THE LOVE, and help us help
YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
|
|
From: McDonald, B. <Bru...@ba...> - 2003-11-04 15:19:59
|
Here is complete example - ready to test.
Start the scanner (babeldoc scanner)
and then copy in the data?.xml files to the input directory and watch it
break.
We need some help here - Lets just say some giving back from however babeldoc might have helped you in the past.
-----Original Message-----
From: McDonald, Bruce
Sent: Tuesday, November 04, 2003 9:38 AM
To: David Kinnvall; bab...@li...
Subject: RE: [Babeldoc-devel] Multithreading problems...
David,
Thank you - I will be working on this. I would appreciate any help that we can get here.
THIS IS THE LAST BIG ISSUE BEFORE BABELDOC 1.2 GOES FINAL.
PLEASE HELP.
regards,
Bruce.
-----Original Message-----
From: David Kinnvall [mailto:dav...@al...]
Sent: Tuesday, November 04, 2003 9:25 AM
To: bab...@li...
Subject: Re: [Babeldoc-devel] Multithreading problems...
Hi everybody!
David Kinnvall (me) wrote at a previous time:
[stuff about my multithreading problems and the lack of datapoints]
I now have a very small setup that triggers the problem, suggesting
that there is nothing really particularly strange about my setup,
but rather that it should be reproducible (debuggable) for others.
Interesting datapoint: The *first* thread seems to always survive.
It is the remaining one(s) that crashes, sooner or later.
Relevant config snippets to (hopefully) be able to reproduce:
Setup a pipeline called whatever and a directory scanner that
feeds it, using the configs outlined below. Adjust as necessary
for your environment(s).
feeder/config.properties:
=========================
# Scanner feeder implementation
scanner.type=asynchronous
#scanner.type=synchronous
scanner.queue=disk
scanner.queueDir=scanner/queue
scanner.queueName=scanner
scanner.poolSize=3
pipeline/whatever/whatever.properties:
======================================
entryStage=entry
entry.stageType=Null
entry.nextStage=enricher
entry.tracked=false
enricher.stageType=XpathExtract
enricher.nextStage=writer
enricher.XPath.foo=/foo/@name
writer.stageType=FileWriter
writer.nextStage=null
writer.outputFile=output/whatever/${ticket.getValue()}_${document.get("foo")}.xml
Input document(s) - create a couple of these, named <anything>.xml:
===================================================================
<?xml version="1.0" encoding="iso-8859-1" standalone="no" ?>
<foo name="bar1"/>
Now, armed with an active Babeldoc instance, using config as
above, give a bunch (say 4) of input xml documents of the above
form to the scanner. For me it (almost) every time bombs for the
second and third threads in the enricher stage. You might have to
retry a few times, but for me it is very consistent, with an NPE
in VariableProcessor.mustExpand()
Important: You need to give multiple input documents to the
scanner at once, to trigger the problem *quickly* ...but...
However: It is certainly possible to trigger it with one single
document at a time as well! Just keep feeding the scanner+pipeline
one of the documents, watch it process it (successfully at least
once), and keep going, until a different thread from the first
gets to work on it. Boom.
The first thread seems to correctly get its configuration for
pipeline stages et al, but the subsequent configurations do
have a tendency to lack values, particularly, as Bruce noticed,
the ones of "complex" nature. A dump of the config data for an
XpathExtract stage, as configured in the snippet above, during
exception handling of one of the NPEs shows this (formatted)
com.babeldoc.core.option.ConfigData@11c2b67[
name=enricher,value=<null>,children={
stageType=com.babeldoc.core.option.ConfigData@659db7[
name=stageType,value=XpathExtract,children=<null>
],
XPath=com.babeldoc.core.option.ConfigData@1556d12[
name=XPath,value=<null>,children={
foo=com.babeldoc.core.option.ConfigData@edf389[
name=foo,value=<null>,children=<null>
]
}
],
nextStage=com.babeldoc.core.option.ConfigData@16be68f[
name=nextStage,value=writer,children=<null>
]
}
]
from which we learn that what should have been
enricher.XPath.foo=/foo/@name
has instead become
enricher.XPath.foo=<null>
which wasn't expected by the pipeline stage. Hence the NPE.
As before, I need help in resolving this. I have so far been
utterly unable to find a solution on my own. I hope that the
additional data in the above text might be of some assistance
for others to have a go at the problem as well, or at least
verify that the problem exists outside of my particular setup.
Best regards,
David
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
Does SourceForge.net help you be more productive? Does it
help you create better code? SHARE THE LOVE, and help us help
YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
Does SourceForge.net help you be more productive? Does it
help you create better code? SHARE THE LOVE, and help us help
YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
|
|
From: McDonald, B. <Bru...@ba...> - 2003-11-04 14:39:52
|
David,
Thank you - I will be working on this. I would appreciate any help that we can get here.
THIS IS THE LAST BIG ISSUE BEFORE BABELDOC 1.2 GOES FINAL.
PLEASE HELP.
regards,
Bruce.
-----Original Message-----
From: David Kinnvall [mailto:dav...@al...]
Sent: Tuesday, November 04, 2003 9:25 AM
To: bab...@li...
Subject: Re: [Babeldoc-devel] Multithreading problems...
Hi everybody!
David Kinnvall (me) wrote at a previous time:
[stuff about my multithreading problems and the lack of datapoints]
I now have a very small setup that triggers the problem, suggesting
that there is nothing really particularly strange about my setup,
but rather that it should be reproducible (debuggable) for others.
Interesting datapoint: The *first* thread seems to always survive.
It is the remaining one(s) that crashes, sooner or later.
Relevant config snippets to (hopefully) be able to reproduce:
Setup a pipeline called whatever and a directory scanner that
feeds it, using the configs outlined below. Adjust as necessary
for your environment(s).
feeder/config.properties:
=========================
# Scanner feeder implementation
scanner.type=asynchronous
#scanner.type=synchronous
scanner.queue=disk
scanner.queueDir=scanner/queue
scanner.queueName=scanner
scanner.poolSize=3
pipeline/whatever/whatever.properties:
======================================
entryStage=entry
entry.stageType=Null
entry.nextStage=enricher
entry.tracked=false
enricher.stageType=XpathExtract
enricher.nextStage=writer
enricher.XPath.foo=/foo/@name
writer.stageType=FileWriter
writer.nextStage=null
writer.outputFile=output/whatever/${ticket.getValue()}_${document.get("foo")}.xml
Input document(s) - create a couple of these, named <anything>.xml:
===================================================================
<?xml version="1.0" encoding="iso-8859-1" standalone="no" ?>
<foo name="bar1"/>
Now, armed with an active Babeldoc instance, using config as
above, give a bunch (say 4) of input xml documents of the above
form to the scanner. For me it (almost) every time bombs for the
second and third threads in the enricher stage. You might have to
retry a few times, but for me it is very consistent, with an NPE
in VariableProcessor.mustExpand()
Important: You need to give multiple input documents to the
scanner at once, to trigger the problem *quickly* ...but...
However: It is certainly possible to trigger it with one single
document at a time as well! Just keep feeding the scanner+pipeline
one of the documents, watch it process it (successfully at least
once), and keep going, until a different thread from the first
gets to work on it. Boom.
The first thread seems to correctly get its configuration for
pipeline stages et al, but the subsequent configurations do
have a tendency to lack values, particularly, as Bruce noticed,
the ones of "complex" nature. A dump of the config data for an
XpathExtract stage, as configured in the snippet above, during
exception handling of one of the NPEs shows this (formatted)
com.babeldoc.core.option.ConfigData@11c2b67[
name=enricher,value=<null>,children={
stageType=com.babeldoc.core.option.ConfigData@659db7[
name=stageType,value=XpathExtract,children=<null>
],
XPath=com.babeldoc.core.option.ConfigData@1556d12[
name=XPath,value=<null>,children={
foo=com.babeldoc.core.option.ConfigData@edf389[
name=foo,value=<null>,children=<null>
]
}
],
nextStage=com.babeldoc.core.option.ConfigData@16be68f[
name=nextStage,value=writer,children=<null>
]
}
]
from which we learn that what should have been
enricher.XPath.foo=/foo/@name
has instead become
enricher.XPath.foo=<null>
which wasn't expected by the pipeline stage. Hence the NPE.
As before, I need help in resolving this. I have so far been
utterly unable to find a solution on my own. I hope that the
additional data in the above text might be of some assistance
for others to have a go at the problem as well, or at least
verify that the problem exists outside of my particular setup.
Best regards,
David
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
Does SourceForge.net help you be more productive? Does it
help you create better code? SHARE THE LOVE, and help us help
YOU! Click Here: http://sourceforge.net/donate/
_______________________________________________
Babeldoc-devel mailing list
Bab...@li...
https://lists.sourceforge.net/lists/listinfo/babeldoc-devel
|
|
From: David K. <dav...@al...> - 2003-11-04 14:24:51
|
Hi everybody!
David Kinnvall (me) wrote at a previous time:
[stuff about my multithreading problems and the lack of datapoints]
I now have a very small setup that triggers the problem, suggesting
that there is nothing really particularly strange about my setup,
but rather that it should be reproducible (debuggable) for others.
Interesting datapoint: The *first* thread seems to always survive.
It is the remaining one(s) that crashes, sooner or later.
Relevant config snippets to (hopefully) be able to reproduce:
Setup a pipeline called whatever and a directory scanner that
feeds it, using the configs outlined below. Adjust as necessary
for your environment(s).
feeder/config.properties:
=========================
# Scanner feeder implementation
scanner.type=asynchronous
#scanner.type=synchronous
scanner.queue=disk
scanner.queueDir=scanner/queue
scanner.queueName=scanner
scanner.poolSize=3
pipeline/whatever/whatever.properties:
======================================
entryStage=entry
entry.stageType=Null
entry.nextStage=enricher
entry.tracked=false
enricher.stageType=XpathExtract
enricher.nextStage=writer
enricher.XPath.foo=/foo/@name
writer.stageType=FileWriter
writer.nextStage=null
writer.outputFile=output/whatever/${ticket.getValue()}_${document.get("foo")}.xml
Input document(s) - create a couple of these, named <anything>.xml:
===================================================================
<?xml version="1.0" encoding="iso-8859-1" standalone="no" ?>
<foo name="bar1"/>
Now, armed with an active Babeldoc instance, using config as
above, give a bunch (say 4) of input xml documents of the above
form to the scanner. For me it (almost) every time bombs for the
second and third threads in the enricher stage. You might have to
retry a few times, but for me it is very consistent, with an NPE
in VariableProcessor.mustExpand()
Important: You need to give multiple input documents to the
scanner at once, to trigger the problem *quickly* ...but...
However: It is certainly possible to trigger it with one single
document at a time as well! Just keep feeding the scanner+pipeline
one of the documents, watch it process it (successfully at least
once), and keep going, until a different thread from the first
gets to work on it. Boom.
The first thread seems to correctly get its configuration for
pipeline stages et al, but the subsequent configurations do
have a tendency to lack values, particularly, as Bruce noticed,
the ones of "complex" nature. A dump of the config data for an
XpathExtract stage, as configured in the snippet above, during
exception handling of one of the NPEs shows this (formatted)
com.babeldoc.core.option.ConfigData@11c2b67[
name=enricher,value=<null>,children={
stageType=com.babeldoc.core.option.ConfigData@659db7[
name=stageType,value=XpathExtract,children=<null>
],
XPath=com.babeldoc.core.option.ConfigData@1556d12[
name=XPath,value=<null>,children={
foo=com.babeldoc.core.option.ConfigData@edf389[
name=foo,value=<null>,children=<null>
]
}
],
nextStage=com.babeldoc.core.option.ConfigData@16be68f[
name=nextStage,value=writer,children=<null>
]
}
]
from which we learn that what should have been
enricher.XPath.foo=/foo/@name
has instead become
enricher.XPath.foo=<null>
which wasn't expected by the pipeline stage. Hence the NPE.
As before, I need help in resolving this. I have so far been
utterly unable to find a solution on my own. I hope that the
additional data in the above text might be of some assistance
for others to have a go at the problem as well, or at least
verify that the problem exists outside of my particular setup.
Best regards,
David
|
|
From: Michael A. <mic...@ze...> - 2003-11-03 01:42:57
|
Hi, all,
I've noticed a small anomoly under a particular condition. When I start
JBoss, the babeldoc.home variable isn't set. Before I start running
docs through the JMX scanner, I normally set the babeldoc.home, restart
the scanner, and all is well. Sometimes, however, I forget to set
babeldoc.home. When I run the first document through, it gets picked
up, queued, deleted from the source directory (I'm using a directory
scanner), and then babeldoc errors, because it presumably can't find the
babeldoc.home, so it doesn't know where the processing libraries are.
This is all fine. So, I stop the scanner using jmx-console, set the
babeldoc.home, and start the scanner again. Now, I get a persistent
error:
23:31:27,958 ERROR [DiskQueue] [DiskQueue.removeNoWait]
java.io.FileNotFoundException: ./scanner22096.tmp (No such file or
directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at
com.babeldoc.core.pipeline.feeder.DiskQueue.deserializeDocument(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.DiskQueue.removeNoWait(Unknown Source)
at com.babeldoc.core.pipeline.feeder.DiskQueue.remove(Unknown
Source)
at
com.babeldoc.core.pipeline.feeder.AsynchronousFeeder$1.run(Unknown
Source)
at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:727)
at java.lang.Thread.run(Thread.java:534)
And it just goes on and on until I stop the scanner. I'm not sure
what's going on here, but it looks like a reference is being left in a
queue somewhere, but the file underneath the reference is being
deleted. I'm not sure what I should do here? Does the feeder need some
debugging? Or is this acceptable behaviour? If so, what escape
mechanism can I use that will allow the user to stop this, without
restarting the whole application server?
Cheers...
MikeA
PS: I have since made it so that the scanner sets the correct
babeldoc.home automatically on startup, but I still need to resolve the
base problem.
|
|
From: Bruce M. <br...@mc...> - 2003-10-31 00:20:35
|
Paul, Babeldoc has a number of documents available: whitepapers, the userguide and the developers guide (woefully inadequate). Please see the documentation section on the www.babeldoc.com website. You may download the current versions of babeldoc at URI: (binary) http://prdownloads.sourceforge.net/babeldoc/babeldoc-bin-1.2.0-RC2.zip?download (source) http://prdownloads.sourceforge.net/babeldoc/babeldoc-src-1.2.0-RC2.zip?download You may also access older versions of the code at: http://sourceforge.net/project/showfiles.php?group_id=56976 You will find the whitepapers (available in both the source and the binary downloads as a very good starting point for documentation. regards, Bruce McDonald. On Thursday 30 October 2003 03:48 pm, you wrote: > Hello Bruce, > > Thank you for your prompt response. > > We are thinking about gathering some brief information describing your > product and compile it together with the other open source EAI technologies > currently available in the market and post it in our EAI User Group for our > users to view it and use it as reference when looking for open source EAI. > (Please provide us with a brief description of your product, logo and > contact information) > > Also, I would like to know if a simple end-to-end documentation is > available for your product. We would like to try each one of the open > source EAI technologies. > > Can we have the latest version of your software available for D/L in our > EAI User Group? We will be creating a special download section just for > open source EAI, this way our users will not have to go all over to find > these technologies. > > If you have any questions or suggestions, please feel free to contact me > via e-mail or phone. Any suggestions will be greatly appreciated. > > Regards, > Paul Kyung > EAI User Group > www.eaiusergroup.com > pk...@ea... > 917.477.0281 (Work) > 213.804.2709 (cell) > > -----Original Message----- > From: Bruce McDonald [mailto:br...@mc...] > Sent: Wednesday, October 29, 2003 7:09 PM > To: pk...@ea...; br...@tr...; dej...@ya...; > ful...@ya... > Subject: Re: EAI User Group - Open Source EAI > > Paul, > > Thank you for contacting us. We would be glad to help however we can. > Please > feel free to email me at this address for further information, etc. > > regards, > Bruce McDonald. > > On Wednesday 29 October 2003 01:09 pm, Paul Kyung wrote: > > Hello, > > > > > > > > I am contacting you from EAI User Group, we are an independent community > > focused specifically in the integration industry. > > > > > > > > Recently we came across some users interested in open source EAI > > technologies available in the current market. So we are thinking about > > writing up a review to our users on each of the open source EAI > > technologies currently available in the market. > > > > > > > > I would like to get in touch with someone in your organization for some > > assistance and approval. Any help would be greatly appreciated. > > > > > > > > My contact information is bellow. > > > > > > > > Regards, > > > > Paul Kyung > > > > EAI User Group > > > > www.eaiusergroup.com <http://www.eaiusergroup.com/> > > > > pk...@ea... > > > > 917.477.0281 |
|
From: Michael A. <mic...@ze...> - 2003-10-30 23:49:13
|
OK, I think I've found my problem. I was previously using JBoss 3.2.2 RC4, when receiving the error below. I've since installed the production release of 3.2.2, and it seems to work. There must have been an anomoly in the JBOss class loader that was causing it. I got the example directory scanner to load a file, and it was processesd through the example pipeline. So, on to other challenges ;-) MikeA On Tue, 2003-10-28 at 22:36, Michael Ansley wrote: > Hi, > > Has anybody deployed the JMX Scanner recently? I've successfully gotten > it deployed, but when the received document is placed into the pipeline, > it causes a CNF Exception, and I can't make out why. Here is the stack > trace: > 21:14:42,606 INFO [AsynchronousFeeder] Processing document... > 21:14:42,933 INFO [SyncPipelineStageProcessor] PipelineStage name: > entry > 21:14:42,994 ERROR [ResourceLoader] [ResourceLoader.getClass] Error when > trying to load: com.babeldoc.core.pipeline.stage.NullPipelineStage > java.lang.ClassNotFoundException: Unexpected error during load of: > com.babeldoc.core.pipeline.stage.NullPipelineStage, > msg=com/babeldoc/core/pipeline/PipelineStage > at > org.jboss.mx.loading.UnifiedClassLoader3.loadClass(UnifiedClassLoader3.java:196) > at java.lang.ClassLoader.loadClass(ClassLoader.java:235) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:302) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:141) > at com.babeldoc.core.ResourceLoader.getClass(Unknown Source) > at > com.babeldoc.core.service.ServiceFactory.getServiceClass(Unknown Source) > at com.babeldoc.core.Type.getTypeClass(Unknown Source) > > > Is there some special config for the pipeline or the scanner that needs > to be included? Any other tips/hints/suspicions that will be useful? > Perhaps it's not JBoss at all, and I'm just configuring incorrectly. > I've got scanner/config.properties, feeder/config.properties and > pipeline/config.properties files in the babeldoc.sar directory, and the > scanner config is definitely making it into JBoss, because the scanner > that is being loaded comes from the scanner/config.properties. > > This is a dir listing of the babledoc.sar directory: > bash-2.05b$ dir > babeldoc_babelfish.jar babeldoc-scanner.jar commons-collections.jar > commons-net.jar META-INF > babeldoc_conversion.jar babeldoc_soap.jar commons-dbcp.jar > commons-pool.jar pipeline > babeldoc_core.jar babeldoc_sql.jar commons-digester.jar > csv19.jar poi.jar > babeldoc_crypto.jar babeldoc_web.jar commons-discovery.jar > feeder scanner > babeldoc_gui.jar babeldoc_xslfo.jar commons-httpclient.jar > journal.log velocity.jar > babeldoc_init.jar commons-beanutils.jar commons-lang.jar > junit.jar > babeldoc_j2ee.jar commons-cli.jar commons-logging.jar > log4j.jar > > It occured to me that it may be because JBoss is using it's own > classloader, but if the classloader can load com.babeldoc.core.Type, > then why can't it load com.babeldoc.core.pipeline.PipelineStage? > > Cheers... > > > MikeA > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel |
Update of /cvsroot/babeldoc/babeldoc/modules/conversion/src/com/babeldoc/conversion/flatfile/digester
In directory sc8-pr-cvs1:/tmp/cvs-serv1337/modules/conversion/src/com/babeldoc/conversion/flatfile/digester
Modified Files:
Conversion.java Field.java Header.java InputDocument.java
LineSegment.java OutputDocument.java
Log Message:
Updates for segmented lines / cvs lines in the Conversion utilities.
Index: Conversion.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/conversion/src/com/babeldoc/conversion/flatfile/digester/Conversion.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** Conversion.java 19 Jul 2003 13:16:47 -0000 1.1
--- Conversion.java 30 Oct 2003 00:15:56 -0000 1.2
***************
*** 70,74 ****
/**
! * Digester class. This will get used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
--- 70,74 ----
/**
! * Digester class. This will getChild used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
***************
*** 113,117 ****
/**
! * get the documentation
*
* @return
--- 113,117 ----
/**
! * getChild the documentation
*
* @return
Index: Field.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/conversion/src/com/babeldoc/conversion/flatfile/digester/Field.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** Field.java 19 Jul 2003 13:16:47 -0000 1.1
--- Field.java 30 Oct 2003 00:15:56 -0000 1.2
***************
*** 67,71 ****
/**
! * Digester class. This will get used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
--- 67,71 ----
/**
! * Digester class. This will getChild used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
Index: Header.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/conversion/src/com/babeldoc/conversion/flatfile/digester/Header.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** Header.java 19 Jul 2003 13:16:47 -0000 1.1
--- Header.java 30 Oct 2003 00:15:56 -0000 1.2
***************
*** 67,71 ****
/**
! * Digester class. This will get used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
--- 67,71 ----
/**
! * Digester class. This will getChild used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
Index: InputDocument.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/conversion/src/com/babeldoc/conversion/flatfile/digester/InputDocument.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** InputDocument.java 19 Jul 2003 13:16:47 -0000 1.1
--- InputDocument.java 30 Oct 2003 00:15:56 -0000 1.2
***************
*** 67,71 ****
/**
! * Digester class. This will get used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
--- 67,71 ----
/**
! * Digester class. This will getChild used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
***************
*** 94,98 ****
/**
! * get the conversion type
*
* @return
--- 94,98 ----
/**
! * getChild the conversion type
*
* @return
Index: LineSegment.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/conversion/src/com/babeldoc/conversion/flatfile/digester/LineSegment.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** LineSegment.java 19 Jul 2003 13:16:47 -0000 1.1
--- LineSegment.java 30 Oct 2003 00:15:56 -0000 1.2
***************
*** 70,74 ****
/**
! * Digester class. This will get used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
--- 70,74 ----
/**
! * Digester class. This will getChild used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
***************
*** 237,241 ****
* @param csvField DOCUMENT ME!
*/
! public void addCsvFields(Field csvField) {
// System.out.println("[LineSegment.addCsvFields]");
this.csvFields.add(csvField);
--- 237,241 ----
* @param csvField DOCUMENT ME!
*/
! public void addCsvField(Field csvField) {
// System.out.println("[LineSegment.addCsvFields]");
this.csvFields.add(csvField);
Index: OutputDocument.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/conversion/src/com/babeldoc/conversion/flatfile/digester/OutputDocument.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** OutputDocument.java 7 Aug 2003 21:09:02 -0000 1.2
--- OutputDocument.java 30 Oct 2003 00:15:56 -0000 1.3
***************
*** 67,71 ****
/**
! * Digester class. This will get used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
--- 67,71 ----
/**
! * Digester class. This will getChild used by the apache Digester to build the
* hierarchy of classes that represent the conversion xml. This class is
* simply a value object representing the xml document.
|
|
From: <tr...@us...> - 2003-10-30 00:15:59
|
Update of /cvsroot/babeldoc/babeldoc/modules/conversion/src/com/babeldoc/conversion/flatfile
In directory sc8-pr-cvs1:/tmp/cvs-serv1337/modules/conversion/src/com/babeldoc/conversion/flatfile
Modified Files:
ConversionUnmarshaller.java FlatFileConverter.java
Log Message:
Updates for segmented lines / cvs lines in the Conversion utilities.
Index: ConversionUnmarshaller.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/conversion/src/com/babeldoc/conversion/flatfile/ConversionUnmarshaller.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -C2 -d -r1.2 -r1.3
*** ConversionUnmarshaller.java 7 Aug 2003 21:09:02 -0000 1.2
--- ConversionUnmarshaller.java 30 Oct 2003 00:15:56 -0000 1.3
***************
*** 106,110 ****
/**
! * get the interparagraph skip
*
* @return
--- 106,110 ----
/**
! * getChild the interparagraph skip
*
* @return
***************
*** 113,117 ****
/**
! * get the characters to skip from the left
*
* @return
--- 113,117 ----
/**
! * getChild the characters to skip from the left
*
* @return
***************
*** 127,131 ****
/**
! * get the line separator
*
* @return
--- 127,131 ----
/**
! * getChild the line separator
*
* @return
***************
*** 148,152 ****
/**
! * get the row element
*
* @return
--- 148,152 ----
/**
! * getChild the row element
*
* @return
Index: FlatFileConverter.java
===================================================================
RCS file: /cvsroot/babeldoc/babeldoc/modules/conversion/src/com/babeldoc/conversion/flatfile/FlatFileConverter.java,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** FlatFileConverter.java 19 Jul 2003 13:16:47 -0000 1.1
--- FlatFileConverter.java 30 Oct 2003 00:15:56 -0000 1.2
***************
*** 29,33 ****
/**
* handle csv conversions. Watch the errors and make sure that xml start and
! * end tags get handled correctly.
*
* @param is the data
--- 29,33 ----
/**
* handle csv conversions. Watch the errors and make sure that xml start and
! * end tags getChild handled correctly.
*
* @param is the data
***************
*** 85,89 ****
/**
* Convert the data. Depending on the setting of the type in the
! * unmarshaller the different converters get called.
*
* @param element the element underwhich to add the new elements (!!!)
--- 85,89 ----
/**
* Convert the data. Depending on the setting of the type in the
! * unmarshaller the different converters getChild called.
*
* @param element the element underwhich to add the new elements (!!!)
***************
*** 128,131 ****
--- 128,132 ----
*/
protected void errorComment(Element element, Exception e) {
+ e.printStackTrace();
element.addComment(e.toString());
}
***************
*** 213,217 ****
// the input stream from the field number that does not exist
// And protect against data duplication.
! if (fieldNum > pairs.length) {
throw new ConversionException(I18n.get("conversion.002",
new Integer(getUnmarshaller().getFields().length),
--- 214,218 ----
// the input stream from the field number that does not exist
// And protect against data duplication.
! if (fieldNum > pairs.length-1) {
throw new ConversionException(I18n.get("conversion.002",
new Integer(getUnmarshaller().getFields().length),
***************
*** 230,235 ****
private static void addElements(Element row, NameValuePair[] pairs) {
for(int j = 0; j < pairs.length; ++j) {
! Element element = row.addElement(pairs[j].getName());
! element.addText(pairs[j].getValue());
}
}
--- 231,238 ----
private static void addElements(Element row, NameValuePair[] pairs) {
for(int j = 0; j < pairs.length; ++j) {
! if(pairs[j]!=null) {
! Element element = row.addElement(pairs[j].getName());
! element.addText(pairs[j].getValue());
! }
}
}
***************
*** 268,278 ****
}
! // Ok get data - first check if we can get all the data
if (end > line.length()) {
! // can we get any of it?
if (line.length() < start) {
value = "";
}
! // Well, we get what we can.
else {
value = line.substring(start);
--- 271,281 ----
}
! // Ok getChild data - first check if we can getChild all the data
if (end > line.length()) {
! // can we getChild any of it?
if (line.length() < start) {
value = "";
}
! // Well, we getChild what we can.
else {
value = line.substring(start);
|
|
From: Michael A. <mic...@ze...> - 2003-10-28 22:40:05
|
Hi,
Has anybody deployed the JMX Scanner recently? I've successfully gotten
it deployed, but when the received document is placed into the pipeline,
it causes a CNF Exception, and I can't make out why. Here is the stack
trace:
21:14:42,606 INFO [AsynchronousFeeder] Processing document...
21:14:42,933 INFO [SyncPipelineStageProcessor] PipelineStage name:
entry
21:14:42,994 ERROR [ResourceLoader] [ResourceLoader.getClass] Error when
trying to load: com.babeldoc.core.pipeline.stage.NullPipelineStage
java.lang.ClassNotFoundException: Unexpected error during load of:
com.babeldoc.core.pipeline.stage.NullPipelineStage,
msg=com/babeldoc/core/pipeline/PipelineStage
at
org.jboss.mx.loading.UnifiedClassLoader3.loadClass(UnifiedClassLoader3.java:196)
at java.lang.ClassLoader.loadClass(ClassLoader.java:235)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:302)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:141)
at com.babeldoc.core.ResourceLoader.getClass(Unknown Source)
at
com.babeldoc.core.service.ServiceFactory.getServiceClass(Unknown Source)
at com.babeldoc.core.Type.getTypeClass(Unknown Source)
Is there some special config for the pipeline or the scanner that needs
to be included? Any other tips/hints/suspicions that will be useful?
Perhaps it's not JBoss at all, and I'm just configuring incorrectly.
I've got scanner/config.properties, feeder/config.properties and
pipeline/config.properties files in the babeldoc.sar directory, and the
scanner config is definitely making it into JBoss, because the scanner
that is being loaded comes from the scanner/config.properties.
This is a dir listing of the babledoc.sar directory:
bash-2.05b$ dir
babeldoc_babelfish.jar babeldoc-scanner.jar commons-collections.jar
commons-net.jar META-INF
babeldoc_conversion.jar babeldoc_soap.jar commons-dbcp.jar
commons-pool.jar pipeline
babeldoc_core.jar babeldoc_sql.jar commons-digester.jar
csv19.jar poi.jar
babeldoc_crypto.jar babeldoc_web.jar commons-discovery.jar
feeder scanner
babeldoc_gui.jar babeldoc_xslfo.jar commons-httpclient.jar
journal.log velocity.jar
babeldoc_init.jar commons-beanutils.jar commons-lang.jar
junit.jar
babeldoc_j2ee.jar commons-cli.jar commons-logging.jar
log4j.jar
It occured to me that it may be because JBoss is using it's own
classloader, but if the classloader can load com.babeldoc.core.Type,
then why can't it load com.babeldoc.core.pipeline.PipelineStage?
Cheers...
MikeA
|
|
From: Michael A. <mic...@ze...> - 2003-10-27 18:04:40
|
PS: Sorry, I should have said which revision number ;-) On Mon, 2003-10-27 at 10:44, Dejan Krsmanovic wrote: > Hi Mike! > It took me half an hour to find exact version where it was removed! But I > found it. > So, the short answer is that we have implemented "Holywood principle" (Don't > call me, I'll call you) or Template method pattern (I think it is called > that way!). > > The long answer is: > In old Babeldoc code there was concrete method execute() in BabeldocCommand > whoch have done few common tasks. All subclasses were expected to call > super.execute() at the begining of their own implementation. > This wasn't good solution since there was no guarantee that subclasses will > actually call execute method. So we have introduced new method called > executeCommand and copied method body from BabeldocCommand.execute() method > which became abstract. Also, at the end of executeCommand there is now call > to execute method. In BabeldocCommand constructor now is called > executeCommand instead of execute. > This was we have achieved following: > 1. Code from BabeldocCommand.executeCommand() will be always executed (we > are not rellying on subclass to call super method) > 2. Subclasses are now required to implement execute() method since it is > abstract > > I hope I have answered your question! Feel free to ask more if something > left unclear! > > Regards, > Dejan > > ----- Original Message ----- > From: "Michael Ansley" <mic...@ze...> > To: <Bru...@ba...> > Cc: <bab...@li...> > Sent: Saturday, October 25, 2003 12:44 AM > Subject: [Babeldoc-devel] Scanner change > > > > Hi, Bruce, > > > > In August, you checked in a change which removed a line from > > Scanner.java, in method execute, which called the super method. Could > > you please let me know why this was removed, if you still remember why? > > > > Thanks... > > > > > > MikeA > > > > > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: The SF.net Donation Program. > > Do you like what SourceForge.net is doing for the Open > > Source Community? Make a contribution, and help us add new > > features and functionality. Click here: http://sourceforge.net/donate/ > > _______________________________________________ > > Babeldoc-devel mailing list > > Bab...@li... > > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel > > |
|
From: Michael A. <mic...@ze...> - 2003-10-27 18:04:03
|
Hi, Dejan, OK, I understand that, thanks. I've used that technique before to ensure that certain code gets called before other code. Cheers... MikeA On Mon, 2003-10-27 at 10:44, Dejan Krsmanovic wrote: > Hi Mike! > It took me half an hour to find exact version where it was removed! But I > found it. > So, the short answer is that we have implemented "Holywood principle" (Don't > call me, I'll call you) or Template method pattern (I think it is called > that way!). > > The long answer is: > In old Babeldoc code there was concrete method execute() in BabeldocCommand > whoch have done few common tasks. All subclasses were expected to call > super.execute() at the begining of their own implementation. > This wasn't good solution since there was no guarantee that subclasses will > actually call execute method. So we have introduced new method called > executeCommand and copied method body from BabeldocCommand.execute() method > which became abstract. Also, at the end of executeCommand there is now call > to execute method. In BabeldocCommand constructor now is called > executeCommand instead of execute. > This was we have achieved following: > 1. Code from BabeldocCommand.executeCommand() will be always executed (we > are not rellying on subclass to call super method) > 2. Subclasses are now required to implement execute() method since it is > abstract > > I hope I have answered your question! Feel free to ask more if something > left unclear! > > Regards, > Dejan > > ----- Original Message ----- > From: "Michael Ansley" <mic...@ze...> > To: <Bru...@ba...> > Cc: <bab...@li...> > Sent: Saturday, October 25, 2003 12:44 AM > Subject: [Babeldoc-devel] Scanner change > > > > Hi, Bruce, > > > > In August, you checked in a change which removed a line from > > Scanner.java, in method execute, which called the super method. Could > > you please let me know why this was removed, if you still remember why? > > > > Thanks... > > > > > > MikeA > > > > > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: The SF.net Donation Program. > > Do you like what SourceForge.net is doing for the Open > > Source Community? Make a contribution, and help us add new > > features and functionality. Click here: http://sourceforge.net/donate/ > > _______________________________________________ > > Babeldoc-devel mailing list > > Bab...@li... > > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: The SF.net Donation Program. > Do you like what SourceForge.net is doing for the Open > Source Community? Make a contribution, and help us add new > features and functionality. Click here: http://sourceforge.net/donate/ > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel |
|
From: Dejan K. <dej...@nb...> - 2003-10-27 10:48:38
|
Hi Mike! It took me half an hour to find exact version where it was removed! But I found it. So, the short answer is that we have implemented "Holywood principle" (Don't call me, I'll call you) or Template method pattern (I think it is called that way!). The long answer is: In old Babeldoc code there was concrete method execute() in BabeldocCommand whoch have done few common tasks. All subclasses were expected to call super.execute() at the begining of their own implementation. This wasn't good solution since there was no guarantee that subclasses will actually call execute method. So we have introduced new method called executeCommand and copied method body from BabeldocCommand.execute() method which became abstract. Also, at the end of executeCommand there is now call to execute method. In BabeldocCommand constructor now is called executeCommand instead of execute. This was we have achieved following: 1. Code from BabeldocCommand.executeCommand() will be always executed (we are not rellying on subclass to call super method) 2. Subclasses are now required to implement execute() method since it is abstract I hope I have answered your question! Feel free to ask more if something left unclear! Regards, Dejan ----- Original Message ----- From: "Michael Ansley" <mic...@ze...> To: <Bru...@ba...> Cc: <bab...@li...> Sent: Saturday, October 25, 2003 12:44 AM Subject: [Babeldoc-devel] Scanner change > Hi, Bruce, > > In August, you checked in a change which removed a line from > Scanner.java, in method execute, which called the super method. Could > you please let me know why this was removed, if you still remember why? > > Thanks... > > > MikeA > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: The SF.net Donation Program. > Do you like what SourceForge.net is doing for the Open > Source Community? Make a contribution, and help us add new > features and functionality. Click here: http://sourceforge.net/donate/ > _______________________________________________ > Babeldoc-devel mailing list > Bab...@li... > https://lists.sourceforge.net/lists/listinfo/babeldoc-devel |
2 messages has been excluded from this view by a project administrator.
