babeldoc-devel

[Babeldoc-devel] static IConfigInfo's in scanners intentional?

[David K. <dav...@al...>]

Hi!

I'm currently using Babeldoc (tracking CVS HEAD) to build a
small newsfeed integration app for my company, where we will
receive realtime news in xml format via ftp to our servers.

First: I reeeeally like Babeldoc! Nice work, people!


I am having a slight problem with the DirectoryScanner as it
is implemented right now. It uses a static IConfigInfo, which
means that I cannot define two instances with separate config
for each in my scanner/config.properties, since one of the
config blocks ends up being applied to and used by both of
the scanner workers. My solution right now is to do this:

   public final IConfigInfo scannerInfo = new ...;

instead of this:

   public static final IConfigInfo scannerInfo = new ...;

in the DirectoryScanner (only scanner I currently use),
which seems to work nicely. Also, when checking the other
scanner implementations, the only one having an IConfigInfo
member that is *not* static is the SqlScanner. It seems to
me that every scanner needing configuration data should have
a non-static IConfigInfo member rather than a static one?


Also, I have put together two very simple pipeline stages:

- SqlPreparedWriter; similar to the SqlWriter but it supports
   enumerating each parameter to go into the query, and then
   uses PreparedStatement.setFoo(index, ...) to set it. I had
   to do this to be able to properly escape large text blocks
   which may or may not contain embedded html into the database,
   which in this case is MySQL. I failed to make the SqlWriter
   do this without MySQL balking at the input.

- FileTransfer; very simple stage to move data from one stream
   to another. In my case I use this to stream associated files
   via http from the news provider's website to the filesystem
   on our servers, based on links embedded in the news xml. It
   is required for us to know if any transfer failed as well as
   recording where the transfered files ended up in the filesystem
   and add this information to the master news xml document before
   final processing and insertion into the target database via the
   stage above. Works somewhat like a FileReader directly piped to
   a FileWriter without affecting the master document and still
   providing feedback (status) to it.

If there is any interest in the above (very simple!) stages, I
would be happy to submit them for inclusion and refinement into
Babeldoc proper.

Oh, if the devel list is not the proper place for posts like
this, I apologize. I haven't seen any traffic at all on the
users list so...

Best regards,

David Kinnvall

Re: [Babeldoc-devel] static IConfigInfo's in scanners intentional?

[Dejan K. <dej...@nb...>]

> Hi!
>
> I'm currently using Babeldoc (tracking CVS HEAD) to build a
> small newsfeed integration app for my company, where we will
> receive realtime news in xml format via ftp to our servers.
>
> First: I reeeeally like Babeldoc! Nice work, people!

Thanks, David. It is very important to us that there are users that are not
afraid to work with most recent files from CVS HEAD. You like living on the
edge, right! ;)

>
>
> I am having a slight problem with the DirectoryScanner as it
> is implemented right now. It uses a static IConfigInfo, which
> means that I cannot define two instances with separate config
> for each in my scanner/config.properties, since one of the
> config blocks ends up being applied to and used by both of
> the scanner workers. My solution right now is to do this:
>
>    public final IConfigInfo scannerInfo = new ...;
>
> instead of this:
>
>    public static final IConfigInfo scannerInfo = new ...;
>
> in the DirectoryScanner (only scanner I currently use),
> which seems to work nicely. Also, when checking the other
> scanner implementations, the only one having an IConfigInfo
> member that is *not* static is the SqlScanner. It seems to
> me that every scanner needing configuration data should have
> a non-static IConfigInfo member rather than a static one?
>

I guess you are right. The problem is that ConfigInfo object were used to
containt meta information about scanner worker. Since all workers of the
same type has same meta information (description, parameter names...) it was
defined as static field. But recently this was changed and now these objects
contains configuration, too. So they should not be static. I guess we should
use the same aproach here as with pipeline stages where Info object is used
as parameter of constructor...

The problem is that I am the one mostly responisble for scanner module. But
these days I really haven't time for Babeldoc, so I haven't tested new code
that Bruce has changed.

I would also ask you to report this as a bug.

> Also, I have put together two very simple pipeline stages:
>
> - SqlPreparedWriter; similar to the SqlWriter but it supports
>    enumerating each parameter to go into the query, and then
>    uses PreparedStatement.setFoo(index, ...) to set it. I had
>    to do this to be able to properly escape large text blocks
>    which may or may not contain embedded html into the database,
>    which in this case is MySQL. I failed to make the SqlWriter
>    do this without MySQL balking at the input.

Is it possible to add this functionality to SqlWriter instead of creting new
pipeline stage?

> - FileTransfer; very simple stage to move data from one stream
>    to another. In my case I use this to stream associated files
>    via http from the news provider's website to the filesystem
>    on our servers, based on links embedded in the news xml. It
>    is required for us to know if any transfer failed as well as
>    recording where the transfered files ended up in the filesystem
>    and add this information to the master news xml document before
>    final processing and insertion into the target database via the
>    stage above. Works somewhat like a FileReader directly piped to
>    a FileWriter without affecting the master document and still
>    providing feedback (status) to it.
>
> If there is any interest in the above (very simple!) stages, I
> would be happy to submit them for inclusion and refinement into
> Babeldoc proper.

I think you should upload your code to SF (feature suggestions or pathces
section) so we can review it.

> Oh, if the devel list is not the proper place for posts like
> this, I apologize. I haven't seen any traffic at all on the
> users list so...

We have created user mailing list for some less tehnical questions but there
are no posts there yet. All Babeldoc users sooner or later become developers
;) So, yes, this is the right place for your questions!

Dejan

Re: [Babeldoc-devel] static IConfigInfo's in scanners intentional?

[Dejan K. <dej...@nb...>]

David, all,
I have removed static fields form worker classes so now every worker now has
each own's ConfigInfo object. However, I have no time right now for testing.
So, if you have a time, please checkout CVS HEAD and let me know the
results...

Dejan