[Babeldoc-devel] Avoiding incomplete reads in DirectoryScanner

[David K. <dav...@al...>]

Hi people!

I am getting bitten by the fact that a slow writer using FTP to
upload files to our server for processing using Babeldoc via the
DirectoryScanner sometimes, due to timing obviously, causes the read
file to be incomplete, thereby causing parsing/transformation errors
and so forth.  (Long sentence, yeah I know... ;-p )

To fix this, I have looked at various alternatives, including potential
ftp daemon settings (nope), kernel module (not very nice), the fam thing
(from SGI, old Perl module, didn't work) to avoid the read from taking
place until the file was written completely.

After failing enough at the above trials...I turned to our beloved
Babeldoc instead, for a dirty(?) little hack:

I extended the DirectoryScanner to take another config parameter
'minLastModified', default 0, specififying the least number of
milliseconds that must have passed since the file's last modification
timestamp, as reported by the File instance.

I am working under the assumption that the modification timestamp is
indeed correctly updated during the ftp upload, and it seems to be the
case, at least on my Red Hat 7.2 installation where I am testing this.

Using <scanner_name>.minLastModified = 5000 I have yet to see any failed
reads so far. But then again, I have only tested it for about an hour or
so... :-)  I'll know better tomorrow what the actual effect will be.

Diff (unified) against DirectoryScanner.java (per current CVS) attached.

Note: I haven't made any changes to the I18N stuff, and I do use a
logInfo() here and there, for debugging, so please treat the patch as
a proof-of-concept needing some (hopefully little) cleanup.

Any hope of getting something like this into Babeldoc proper?

(Oh, yeah, I haven't had the time to properly follow up on the last
  mail I sent about the SqlPreparedWriter and FileTransfer stages.
  Sorry about that. Still pondering if it makes sense to merge the
  Sql thing into the standard SqlWriter, since my prepared version
  has no support for multiple statements nor comments in it...I'll
  get back to the list on that in the (hopefully) near future.)

Regards,

David Kinnvall