Re: [fuse-devel] vanished files after rename

[Wolfgang B. <w.b...@pr...>]

The same issue seems to also affect link(). Specifically linkat() with
AT_EMPTY_PATH, and none of the available flags, not even that dirty
hidden-file workaround tackle this problem at all.

What basically happens is the kernel starts off with a `lookup` on the
file, getting an inode for the path without creating a full reference
(in the eyes of libfuse). Then the other task gets to execute its
`rename()`, thereby deleting the old inode's *path*. It's *not*
hidden, because this hack only applies when the file is actually
opened, but it's not yet open, it has only been looked up. After
this, the `open()` on the old *inode* fails because it fails to
lookup the *path* for said inode, which has been removed by
`rename()`.

Similar behavior can be observed with `linkat()`: Create a file and
make a hard-link to it, then try to open one, delete it, and use
`linkat()` with AT_EMPTY_PATH. (the hardlink is there because
AT_EMPTY_PATH isn't required to work on files with a zero reference
count). Eg:

# echo stuff > file
# cp -l file another_link

Now do the following sequence:

int f = open("file", O_PATH);
unlink("file");
linkat(f, "", AT_FDCWD, "file", AT_EMPTY_PATH);

This works without fuse, but with fuse `link()` fails to rebuild a
path for the inode of `f` and thus can't execute the userspace's
link() function. (get_path()).

Tell me if I'm on a completely wrong track here, but the problem
seems to be that the kernel works with inodes, but the userspace
doesn't. The current API is fine for functions where the 'nopath'
option applies, but does not work for `open()` and `link()`.

So how should we solve this? This affects *all* fuse filesystems
regardless of mount-options.
The way I see it, this is a design-bug.
So we either need an update to the user-side API to pass through
lookup/forget (since we can't simply open the file at `lookup()` time
since we don't know the open-mode yet - lookup just does
`getattr()`), or the kernel needs to make the combination of
lookup+open atomic, or the "hidden-file" hack needs to be improved (if
you just remove the is_open() checks you end up moving from ENOENT to
EBUSY because it performs multiple actions which each lock and unlock
rather than taking one big lock over the entire process which would
seem more reasonable). Or get rid of `lookup()` on the kernel-side
altogether?

Here's the same bug in glusterfs, reported back in 2010:
https://bugzilla.redhat.com/show_bug.cgi?id=762766
"Severity high, priority low", really?

Thoughts?

> On October 14, 2015 at 9:52 AM Dietmar Maurer <di...@pr...> wrote:
> we recently observed very strange behavior with 'rename'. Our software
> uses temporary files to make sure file creation is atomic. For example,
> we first create 'testfile.tmp.XYZ', write the file, and then simply
> rename the file to 'testfile'. The strange thing is that 'testfile'
> vanish from time to time when read from other processes.
> 
> I assembled a test case using 'fusexmp.c'. Simply download attached 
> fuse-rename-bug.tgz package and do:
(...)
> The real test is inside test_fuse_exmp.pl, which mounts fusexmp at 
> directory './testmnt'. So when you are done with testing you should
> 
> # umount ./testmnt
> 
> AFAIK this bug triggers with any fuse based file system, for example sshfs.
> I guess it is quite dangerous for any software that assume a rename
> is an atomic action.