[fuse-devel] FUSE mmap(2) trouble

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

hi,

i'm having trouble with FUSE and mmap(2). i think the kernel is
caching pages of an mmap(2)ed file, and handing them back to another
client that calls open(2) before the original client calls munmap(2).
this is a problem if the file has changed in the meantime: the later
client sees a mix of current data with stale data from the time of the
mmap(2).

i've boiled my problem down to a small test file system and test
program (attached, along with a Makefile you can use to run the test,
guaranteed to be the lowest-quality Makefile you read all day).

the output is shown below, but basically the test fs has a special
path "/usr/bin/magic_changing_file" whose stat and read calls are
either served by statting and reading /usr/bin/perl on the root fs, or
/usr/bin/make on the root fs after the test fs sees a stat of
"/special-path-that-causes-change".

the accompanying test:

1. runs stat(1) and md5sum(1) on "/usr/bin/magic_changing_file", which
shows the correct md5sum for /usr/bin/perl.
2. opens and mmaps "/usr/bin/magic_changing_file".
3. reads 8KiB of data via the returned pointer.
4. runs md5sum(1) again, getting the wrong result (see below).
5. munmap(2)s the file.
6. runs md5sum(1) one last time, which shows the correct sum for /usr/ 
bin/make.

the problem, then, is the md5sum at step 4. i'd expect to see the
md5sum for /usr/bin/make, but i don't. i don't see the sum for
/usr/bin/perl, either. i see the sum of the first 8KiB of perl and the
rest of make. (the 8KiB comes about because that's how much i read via
the mmap. changing the amount read in the test changes the md5sum(1)
output each time we cause another page to be read.)

if you look at the detailed output below, you can see my userspace fs
simply isn't asked for the first 8KiB. the reads for the second md5sum
start at offset 8192.

i wondered if the problem was that my fs lets FUSE handle the inode
numbering, but you'll see that adding the "use_ino" mount option
doesn't help. stat(1) shows distinct inode numbers, but that first
read is still missing and so the md5sum(1) output is still wrong.

my questions:

  1. is this my fault?
  1a. if so, how can i fix it?
  1b. if not, is there a work-around?

thanks,
  elliott

  $ make test
  sudo fusermount -u /tmp/mbfs || echo
  fusermount: entry for /tmp/mbfs not found in /etc/mtab

  cp mmap-bug-fs /tmp
  mkdir -p /tmp/mbfs
  sudo /tmp/mmap-bug-fs -f -r -o nonempty,allow_other /tmp/mbfs &
  sleep 1; ./mmap-bug-test || echo
  mmap-bug-fs: getattr("/")
  mmap-bug-fs: getattr("/")
  mmap-bug-fs: getattr("/")
  mmap-bug-fs: getattr("/")
  getpagesize() = 4096
  sysconf(_SC_PAGESIZE) = 4096
  mmap-bug-fs: getattr("/usr")
  mmap-bug-fs: getattr("/usr/bin")
  mmap-bug-fs: getattr("/usr/bin/magic_changing_file")
  mmap-bug-fs: translating "/usr/bin/magic_changing_file" to "/usr/bin/ 
perl"...
  File: `/tmp/mbfs/usr/bin/magic_changing_file'
  Size: 1059104         Blocks: 2080       IO Block: 4096   regular file
  Device: 1bh/27d Inode: 4           Links: 2
  Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/     
root)
  Access: 2008-03-20 14:39:50.000000000 -0700
  Modify: 2007-12-04 10:58:04.000000000 -0800
  Change: 2008-02-21 07:42:38.000000000 -0800
  mmap-bug-fs: open("/usr/bin/magic_changing_file", 32768)
  mmap-bug-fs: translating "/usr/bin/magic_changing_file" to "/usr/bin/ 
perl"...
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=0,  
size=16384)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=16384,  
size=32768)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=49152,  
size=65536)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=114688,  
size=131072)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=245760,  
size=131072)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=376832,  
size=131072)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=507904,  
size=131072)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=638976,  
size=131072)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=770048,  
size=131072)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=901120,  
size=131072)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=1032192,  
size=28672)
  mmap-bug-fs: release("/usr/bin/magic_changing_file")
  16eb4198b229fd82e7f58e295ea5ebc5  /tmp/mbfs/usr/bin/ 
magic_changing_file
  mmap-bug-fs: getattr("/usr/bin/magic_changing_file")
  mmap-bug-fs: translating "/usr/bin/magic_changing_file" to "/usr/bin/ 
perl"...
  mmap-bug-fs: open("/usr/bin/magic_changing_file", 32768)
  mmap-bug-fs: translating "/usr/bin/magic_changing_file" to "/usr/bin/ 
perl"...
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=0,  
size=131072)
  mmap-bug-fs: getattr("/special-path-that-causes-change")
  mmap-bug-fs: getattr("/usr/bin/magic_changing_file")
  mmap-bug-fs: translating "/usr/bin/magic_changing_file" to "/usr/bin/ 
make"...
  File: `/tmp/mbfs/usr/bin/magic_changing_file'
  Size: 145516          Blocks: 296        IO Block: 4096   regular file
  Device: 1bh/27d Inode: 4           Links: 1
  Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/     
root)
  Access: 2008-03-20 14:46:53.000000000 -0700
  Modify: 2005-12-16 19:01:11.000000000 -0800
  Change: 2008-02-21 07:40:33.000000000 -0800
  mmap-bug-fs: open("/usr/bin/magic_changing_file", 32768)
  mmap-bug-fs: translating "/usr/bin/magic_changing_file" to "/usr/bin/ 
make"...
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=8192,  
size=8192)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=16384,  
size=32768)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=49152,  
size=65536)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=114688,  
size=32768)
  mmap-bug-fs: release("/usr/bin/magic_changing_file")
  00ca31b14c4af18daaeba498dce357f8  /tmp/mbfs/usr/bin/ 
magic_changing_file
  mmap-bug-fs: release("/usr/bin/magic_changing_file")
  mmap-bug-fs: getattr("/usr/bin/magic_changing_file")
  mmap-bug-fs: translating "/usr/bin/magic_changing_file" to "/usr/bin/ 
make"...
  File: `/tmp/mbfs/usr/bin/magic_changing_file'
  Size: 145516          Blocks: 296        IO Block: 4096   regular file
  Device: 1bh/27d Inode: 4           Links: 1
  Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/     
root)
  Access: 2008-03-20 14:46:55.000000000 -0700
  Modify: 2005-12-16 19:01:11.000000000 -0800
  Change: 2008-02-21 07:40:33.000000000 -0800
  mmap-bug-fs: open("/usr/bin/magic_changing_file", 32768)
  mmap-bug-fs: translating "/usr/bin/magic_changing_file" to "/usr/bin/ 
make"...
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=0,  
size=16384)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=16384,  
size=32768)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=49152,  
size=65536)
  mmap-bug-fs: read("/usr/bin/magic_changing_file", offset=114688,  
size=32768)
  mmap-bug-fs: release("/usr/bin/magic_changing_file")
  7caeb19a544e72b462528488ccd76f44  /tmp/mbfs/usr/bin/ 
magic_changing_file
  sudo fusermount -u /tmp/mbfs
  mmap-bug-fs: getattr("/")
  mmap-bug-fs: getattr("/")
  mmap-bug-fs: getattr("/")
  $