[fuse-devel] inode problem when using git on a sshfs filesystem

[Yann D. <ya...@dr...>]

Hi,

For some days, my usage of git is not as seamless as before.

I'm using git along sshfs/fuse (don't blame me for that), and 
each time I try to rebase one of my branch, I have a conflict when applying
the third commit. Doing the same operation on a local filesystem works without any problem.

===== Part one: git =====

When I try to rebase one specific branch, git rebase failed when applying the third commit,
telling me about uncommited 

I've tried to do it from scratch, using git format-patch / git am
but git am also abort on the third patch with the error message:

error: <path>/<filename>: does not match index

So I've tried to diagnose the problem using :

 - git diff / git status : doesn't return anything.

 - git ls-tree HEAD -l <path>/<filename> : returns the correct mode and file size.

 - git log --raw --all --full-history -- <path>/<filename> : 
   returns valid information matching the one retrieved above.

 - git hash-object <path>/<filename> :
   gives the correct sha1 for the file, as recorded in the patch extracted using git format-patch 
   and reported by git ls-tree or git log (see before)

 - git diff-files : it shows a lot a file, all of them in same directory

   :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename0>
   :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename1>
   :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename2>
   :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename3>
   :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename>

BTW, there's no conflict when applying the patch manually with patch: the patch itself is fine
Using git apply --index also work, but only if it's applied alone:
apply each patches in series and git apply fails in the same third patch.

After diving into git source code and some debugging session, I've found
that the inode number recorded in the active_cache doesn't match the one
on the filesystem for <pach>/<filename> : that's why git apply --index refuse to apply the patch.

Then I tried to monitor stat() information for the file in <path> during
git operations.
1) After applying the first patch, files in <path> were affected different inode number
2) Using strace, I checked that git apply didn't make anything specials to thoses files.
The only thing strange git did, was trying to unlink(<path>), but this failed since the <path>
directory wasn't empty.

Note: the first patch remove, change and add some files in <path> directory, while 
the third patch changes another file in <path> directory

As a workaround: running git diff / git diff --cached / git status between each
git apply --index command seems to update the cache and allows me to apply all the patches
without problem. But it's not an easy path to follow when rebasing branches.

Surprisingly, when looking at strace output, it seems that git apply, once work done, is calling lstat() 
for all the files under <path>, and it sees the new inodes allocated to those files, but I don't know what 
it is doing with those information, if it's not stored in the index.

To conclude, it was a bit hard to diagnose from git point of view.

====== Part two: sshfs / fuse ======

At this time sshfs seems to be guilty of bad behavior, breaking somes POSIX rules.

So I tried the following testcase on another computer to reproduce the
problem outside of git.

Here the results:

$ mkdir dir
$ touch dir/a dir/b
$ stat -t dir/*
dir/a 0 0 81b4 500 500 15 3 1 0 0 1297882724 1297882724 1297882724 4096
dir/b 0 0 81b4 500 500 15 4 1 0 0 1297882726 1297882726 1297882726 4096
$ rmdir dir
rmdir: failed to remove `dir1': Operation not permitted
$ stat -t dir/*
dir/a 0 0 81b4 500 500 15 6 1 0 0 1297882724 1297882724 1297882724 4096
dir/b 0 0 81b4 500 500 15 7 1 0 0 1297882726 1297882726 1297882726 4096

One can see that inode 3 became inode 6 and inode 4 became inode 7 after the failed
unlink operation on dir. Which seems to be a bit uncommon for me.

Note: on a local filesystem, rmdir failed with message rmdir: failed to remove `dir1': Directory not empty

I try to add some debug support to fuse / sshfs in order to locate more precisely the problem:
(lines beginning by -/+ where added by me in libfuse, line beginning with --/++ in sshfs)

$ sshfs localhost:<export> <mount> -o sshfs_debug,debug,cache=no -d -f -s

unique: 22, opcode: FORGET (2), nodeid: 4, insize: 48, pid: 0
- forget 4
FORGET 4/1
DELETE: 4
+ forget 4
unique: 23, opcode: FORGET (2), nodeid: 3, insize: 48, pid: 0
- forget 3
FORGET 3/1
DELETE: 3
+ forget 3
unique: 24, opcode: RMDIR (11), nodeid: 1, insize: 44, pid: 9044
- rmdir 1 dir
rmdir /dir
-- rmdir(/dir)
[00020] RMDIR
  [00020]         STATUS       28bytes (0ms)
++ rmdir(/dir) = -1
   unique: 24, error: -1 (Operation not permitted), outsize: 16
+ rmdir 1 dir
unique: 25, opcode: FORGET (2), nodeid: 2, insize: 48, pid: 0
- forget 2
FORGET 2/1
DELETE: 2
+ forget 2

One can see that the reference to files under the directory are asked by
the kernel to be forgotten, even if the directory is not yet removed.

This seems a bit illogical since a directory with files under it can't
be removed (but FORGET could apply to file deleted but still referenced
by a process).

Note: if the file is opened, the inode associated to the file name
didn't change. Hopefully.

I've tried to reproduce the problem with other virtual filesystem like
shm / tmpfs / devtmpfs / ramfs : no problem.

I've also tried with NFS (local), and there's no problem too (the inode
numbers reported from NFS client side are the same than the server
side).

So it seems this a FUSE only problem, and I haven't found exactly why.

Regards.

-- 
Yann Droneaud