dsfile.setFileContents sometimes finds corruption
Brought to you by:
eries,
malcolmredheron
dsfile.setFileContents reads the file back and checks that it
is the same as what it was trying to write. This is to detect
cases where the file becomes corrupt for some reason.
Sometimes this check finds problems.
Things to note:
- setFileContents and fileContents already use (or try to use)
locking to prevent two threads from accessing a file at the
same time.
- the corruption does always not look like disk-level
corrution. Sometimes it looks like garbage bytes but other
times it looks like a different but valid pickled python file.
I have added logging after we acquire the lock and before
we release it to work out whether threading is the problem.