Re: [Samtools-devel] Samtools cache corruption

[James B. <jk...@sa...>]

On Mon, Jan 18, 2016 at 07:30:15AM -0800, Terence Gliedt wrote:
> samtools needs to be using flock when creating the cache file. In almost
> all cases this will cause no delay, but in the few instances where two
> instances of samtools are writing to the same data, it'll prevent
> corruption.

Which data files are you talking about?  We really need to know which
local cache files you are seeing problems with.

I know samtools caches the reference sequence.  That file is opened
with exclusive mode:

	do {
	    sprintf(path_tmp, "%s.tmp_%d", path, /*getpid(),*/ i);
	    i++;
	    fp = fopen(path_tmp, "wx");
	} while (fp == NULL && errno == EEXIST);

(The file later gets renamed.)  Mode "wx" is exclusive which should do
locking, but there is potential for locking to break if the output
filename is on NFS... I think!

This isn't perhaps ideal as multiple instances canstart trying to
fetch the same reference at the same time, but the rename prevents
instances from using a partial file.  Locally we use a little web
server to do the population to disk and specify REF_PATH to be
/some/path/cram_cache/%2s/%2s/%s:http:://some-server::8000/%s so it
tries NFS first and http secondary, with the some-server downloading
(once) from EBI.

The only other place I can think of caching local files is when
reading a remote BAM/CRAM file and populating the local .bai / .crai
index file.  This is done in htslib test_and_fetch(), but I see it
uses fp = fopen(p, "w") instead of "wx" so it could possibly clobber.
This does look like a potential problem.

This scenario is trivial to avoid though as it writes to local
directory.  If each job makes its own temporary working directory then
the problem is avoided, although it's inefficient.  Are you accessing
the same remote BAMs many times over from different processes?

James

-- 
James Bonfield (jk...@sa...) | Hora aderat briligi. Nunc et Slythia Tova
                                  | Plurima gyrabant gymbolitare vabo;
  A Staden Package developer:     | Et Borogovorum mimzebant undique formae,
https://sf.net/projects/staden/   | Momiferique omnes exgrabure Rathi. 


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.