Re: [Babeldoc-devel] Avoiding incomplete reads in DirectoryScanner

[David K. <dav...@al...>]

Hi, Bruce,

McDonald, Bruce wrote:
> This issue is really problematic generally!   We did build in a .done file solution into the directory scanner which looks for the existance of the .done flag file.  This file indicates that the transfer has completed.  Why dont you look into this?

Hm...there is no .done file solution in the DirectoryScanner.

There is one in the FileWriterPipelineStage, though, but that
is of no real help in the (our) DirectoryScanner case, since:

- Someone else is doing the writing, and can be slow at
   it, to make things worse...perhaps even via modem. :-)
- I can't make the writer (a different organization)
   change their system to provide an additional done file
   to indicate completion in this case, unfortunately.
- The scanner submits the read document, in whatever state
   it is, to the pipeline queue, for processing, and then
   lets go of it, hence I cannot go back to the scanner if
   I detect in the PipelineStage that something is wrong,
   at least not in any clean way that I can see.

Have I missed anything?

The configurable delay approach in the DirectoryScanner seems
to work so far, at least for me, and I can't really see any
obviously cleaner solution when there is no possibility of
modifying the sender's behavior.

A bit more research indicates that there is really no clean
way (even at the OS level) in Linux to detect when a file has
been completely written and closed. So it seems to me at this
time that the delay might be a viable trick.

I have modified the patch every so slightly so that it checks
the modification timestamp after each delay, to cover the case
with a really slow writer appending a number of bytes each and
every second, for some time. Manually tested using "touch". :-)

Adjusted patch attached, FWIW.

/David