Filesystem in Userspace

On 17-Aug-09, at 3:43 PM, Nikolaus Rath wrote:

> Is a null-path only used under special circumstances? I had hoped I
> could insert an 'assert path == NULL' into all my functions, so that  
> the
> code makes clear that file system entries are exclusively identified  
> by
> their inode.
>
>
You should use the low-level interface (fuse_lowlevel.h), which  
mirrors the VFS interface much more closely. All operations are based  
on inode number, or parent inode number + file name, and you never get  
a full path.

Regards,

John.

"John Muir" <muirj@...> writes:
> On 17-Aug-09, at 3:43 PM, Nikolaus Rath wrote:
>
>> Is a null-path only used under special circumstances? I had hoped I
>> could insert an 'assert path == NULL' into all my functions, so that  
>> the
>> code makes clear that file system entries are exclusively identified  
>> by
>> their inode.
>>
>>
> You should use the low-level interface (fuse_lowlevel.h), which  
> mirrors the VFS interface much more closely. All operations are based  
> on inode number, or parent inode number + file name, and you never get  
> a full path.

So true. I never looked at the low-level API before because I expected
it to be too low-level for my purposes, but it seems to be exactly what
I need.


Thanks!

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

Hi,


>> You should use the low-level interface (fuse_lowlevel.h), which  
>> mirrors the VFS interface much more closely. All operations are based  
>> on inode number, or parent inode number + file name, and you never get  
>> a full path.
>>     
>
> So true. I never looked at the low-level API before because I expected
> it to be too low-level for my purposes, but it seems to be exactly what
> I need.
>   

Just note that, currently, fuse has no option to check
the Posix ACLs (though a patch was recently posted
here to that aim), and you have to check them in the
file system.... which generally implies to check access
to the parent directory, and which you cannot do with
the low level API because you have no information
about which parent directory was used to access the file
(when a file is linked into several parent directories).

Regards

Jean-Pierre

On 19-Aug-09, at 7:57 AM, Jean-Pierre André wrote:
> Just note that, currently, fuse has no option to check
> the Posix ACLs (though a patch was recently posted
> here to that aim), and you have to check them in the
> file system.... which generally implies to check access
> to the parent directory, and which you cannot do with
> the low level API because you have no information
> about which parent directory was used to access the file
> (when a file is linked into several parent directories).


For which file-system operations do you need to check the permissions  
of the parent directory where fuse does not provide the inode number  
of the parent directory?

John.

On Wed, 19 Aug 2009, Jean-Pierre André wrote:
> Just note that, currently, fuse has no option to check
> the Posix ACLs (though a patch was recently posted
> here to that aim), and you have to check them in the
> file system.... which generally implies to check access
> to the parent directory, and which you cannot do with
> the low level API because you have no information
> about which parent directory was used to access the file
> (when a file is linked into several parent directories).

Oh, you *can* do that with the low level api, just need to check
permissions in the .lookup() method.

BTW, the implementation of the high level API uses the low level API,
so everything that can be done with the high level one can also be
done with the low level one (although there's usually a price in
complexity).

Thanks,
Miklos

Hi Miklos,

Miklos Szeredi wrote:
> On Wed, 19 Aug 2009, Jean-Pierre André wrote:
>   
>> Just note that, currently, fuse has no option to check
>> the Posix ACLs (though a patch was recently posted
>> here to that aim), and you have to check them in the
>> file system.... which generally implies to check access
>> to the parent directory, and which you cannot do with
>> the low level API because you have no information
>> about which parent directory was used to access the file
>> (when a file is linked into several parent directories).
>>     
>
> Oh, you *can* do that with the low level api, just need to check
> permissions in the .lookup() method.
>   

So. I must have overlooked that one.

Thank you for your reply.

Jean-Pierre

Hi John,

John Muir wrote:
>
> On 19-Aug-09, at 7:57 AM, Jean-Pierre André wrote:
>> Just note that, currently, fuse has no option to check
>> the Posix ACLs (though a patch was recently posted
>> here to that aim), and you have to check them in the
>> file system.... which generally implies to check access
>> to the parent directory, and which you cannot do with
>> the low level API because you have no information
>> about which parent directory was used to access the file
>> (when a file is linked into several parent directories).
>
>
> For which file-system operations do you need to check the permissions 
> of the parent directory where fuse does not provide the inode number 
> of the parent directory?

For most of them, for instance in getattr() you are
given the inode of the file, not its parent directory.

... but Miklos has just posted how you can do the
check.

Regards

Jean-Pierre

Hi Miklos,

Miklos Szeredi wrote:
> On Wed, 19 Aug 2009, Jean-Pierre André wrote:
>   
>> Just note that, currently, fuse has no option to check
>> the Posix ACLs (though a patch was recently posted
>> here to that aim), and you have to check them in the
>> file system.... which generally implies to check access
>> to the parent directory, and which you cannot do with
>> the low level API because you have no information
>> about which parent directory was used to access the file
>> (when a file is linked into several parent directories).
>>     
>
> Oh, you *can* do that with the low level api, just need to check
> permissions in the .lookup() method.
>   

There still is something I cannot get right with fuse
cacheing enabled. Consider the following, where
mkdir and chmod both return the current stat with
a timeout of one second :

chmod 755 trydir
rm -rf trydir
mkdir trydir
echo file > trydir/file
ls -li trydir/file    
chmod 000 trydir
chmod 444 trydir/file
ls -li trydir/file

I get the following result :

293 -rw-rw-r-- 1 linux linux 5 2009-08-25 17:50 trydir/file
293 -r--r--r-- 1 linux linux 5 2009-08-25 17:50 trydir/file

Which shows the last chmod has been accepted
though just after one denying access.

Analyzing fuse calls, I see that no lookup is issued
between the two chmod, and I tend to believe the
state returned by the chmod on the dir is not used
to decide over the access for the second chmod, the
state resulting from mkdir being used instead until
its timeout elapses.

Actually it has to be so, because the decision on the
lookup depends on the ACLs, and the chmod does
not return the ACL. I see no ACL request either
between the two chmod.

Am I wrong or do I miss something ?

I see a forget method, but it is going the wrong way.

Note : I am checking the protections in the fs to
benefit from the Posix ACLs.

> BTW, the implementation of the high level API uses the low level API,
> so everything that can be done with the high level one can also be
> done with the low level one (although there's usually a price in
> complexity).
>   

So there must be a solution to the above problem...

Regards

Jean-Pierre

Hi,

Jean-Pierre André wrote:
> Hi Miklos,
>
> Miklos Szeredi wrote:
>> On Wed, 19 Aug 2009, Jean-Pierre André wrote:
>>  
>>> Just note that, currently, fuse has no option to check
>>> the Posix ACLs (though a patch was recently posted
>>> here to that aim), and you have to check them in the
>>> file system.... which generally implies to check access
>>> to the parent directory, and which you cannot do with
>>> the low level API because you have no information
>>> about which parent directory was used to access the file
>>> (when a file is linked into several parent directories).
>>>     
>>
>> Oh, you *can* do that with the low level api, just need to check
>> permissions in the .lookup() method.
>>   
>
> There still is something I cannot get right with fuse
> cacheing enabled. Consider the following, where
> mkdir and chmod both return the current stat with
> a timeout of one second :
>
> chmod 755 trydir
> rm -rf trydir
> mkdir trydir
> echo file > trydir/file
> ls -li trydir/file    chmod 000 trydir
> chmod 444 trydir/file
> ls -li trydir/file
>
> I get the following result :
>
> 293 -rw-rw-r-- 1 linux linux 5 2009-08-25 17:50 trydir/file
> 293 -r--r--r-- 1 linux linux 5 2009-08-25 17:50 trydir/file
>
> Which shows the last chmod has been accepted
> though just after one denying access.
>
> Analyzing fuse calls, I see that no lookup is issued
> between the two chmod, and I tend to believe the
> state returned by the chmod on the dir is not used
> to decide over the access for the second chmod, the
> state resulting from mkdir being used instead until
> its timeout elapses.
>
> Actually it has to be so, because the decision on the
> lookup depends on the ACLs, and the chmod does
> not return the ACL. I see no ACL request either
> between the two chmod.
>
> Am I wrong or do I miss something ?
>
> I see a forget method, but it is going the wrong way.
>
> Note : I am checking the protections in the fs to
> benefit from the Posix ACLs.
>
>> BTW, the implementation of the high level API uses the low level API,
>> so everything that can be done with the high level one can also be
>> done with the low level one (although there's usually a price in
>> complexity).
>>   
>
> So there must be a solution to the above problem...

Investigating on this, I see the cacheing of attributes
is dealt with in the kernel module. Cancelling of
saved attributes is done in fuse_invalidate_attr()
which is called when doing unlink, rmdir, rename, etc.
but not in setattr and setxattr which both impact
access control (chmod, chown, setfacl).

Is this correct ? (it should have occurred to other users,
and I do not remember this to have been mentioned
before).

Is there a reason for that ?

Regards

Jean-Pierre

Hi

[I had a problem at first try, sorry if this is sent twice]

Jean-Pierre André wrote:
> Hi Miklos,
>
> Miklos Szeredi wrote:
>   
>> On Wed, 19 Aug 2009, Jean-Pierre André wrote:
>>   
>>     
>>> Just note that, currently, fuse has no option to check
>>> the Posix ACLs (though a patch was recently posted
>>> here to that aim), and you have to check them in the
>>> file system.... which generally implies to check access
>>> to the parent directory, and which you cannot do with
>>> the low level API because you have no information
>>> about which parent directory was used to access the file
>>> (when a file is linked into several parent directories).
>>>     
>>>       
>> Oh, you *can* do that with the low level api, just need to check
>> permissions in the .lookup() method.
>>   
>>     
>
> There still is something I cannot get right with fuse
> cacheing enabled. Consider the following, where
> mkdir and chmod both return the current stat with
> a timeout of one second :
>
> chmod 755 trydir
> rm -rf trydir
> mkdir trydir
> echo file > trydir/file
> ls -li trydir/file    
> chmod 000 trydir
> chmod 444 trydir/file
> ls -li trydir/file
>   

The above is sometimes bad formatted for unknown
reason, trying again :

chmod 755 trydir
rm -rf trydir
mkdir trydir
echo file > trydir/file
ls -li trydir/file
chmod 000 trydir
chmod 444 trydir/file
ls -li trydir/file

> I get the following result :
>
> 293 -rw-rw-r-- 1 linux linux 5 2009-08-25 17:50 trydir/file
> 293 -r--r--r-- 1 linux linux 5 2009-08-25 17:50 trydir/file
>
> Which shows the last chmod has been accepted
> though just after one denying access.
>
> Analyzing fuse calls, I see that no lookup is issued
> between the two chmod, and I tend to believe the
> state returned by the chmod on the dir is not used
> to decide over the access for the second chmod, the
> state resulting from mkdir being used instead until
> its timeout elapses.
I can now confirm the diagnosis : when checking
whether is directory is searchable (in lookup)
with cacheing enabled, fuse only checks the
directory entry timeout, without checking whether
a chmod/chown/setfacl has changed the permissions
on the directory in the meantime.

As I do not know of any way to cancel the directory
cache in chmod/chown/setfacl, I had to get fuse
to check the attributes time out to decide whether
a new lookup has to be done.

Appended is the patch for that, which relies on
chmod/chown/setfacl to return with a null timeout
(at least when they impact a directory), to force
a new lookup the next time the directory is accessed.
Actually for setfacl, the file system cannot return
the updated stat, so something more has to be done
within fuse, and probably also in the kernel module.

With the patch I get the expected result :

78 -rw-rw-r-- 1 linux linux 5 2009-08-28 09:47 trydir/file
chmod: cannot access `trydir/file': Permission denied
ls: cannot access trydir/file: Permission denied

The patch is a "quick and dirty" one, and I would
really appreciate comments.

Regards

Jean-Pierre

Hi

Jean-Pierre André wrote:
> There still is something I cannot get right with fuse
> cacheing enabled. Consider the following, where
> mkdir and chmod both return the current stat with
> a timeout of one second :
>
> chmod 755 trydir
> rm -rf trydir
> mkdir trydir
> echo file > trydir/file
> ls -li trydir/file    
> chmod 000 trydir
> chmod 444 trydir/file
> ls -li trydir/file
>
> I get the following result :
>
> 293 -rw-rw-r-- 1 linux linux 5 2009-08-25 17:50 trydir/file
> 293 -r--r--r-- 1 linux linux 5 2009-08-25 17:50 trydir/file
>
> Which shows the last chmod has been accepted
> though just after one denying access.
>
> Analyzing fuse calls, I see that no lookup is issued
> between the two chmod, and I tend to believe the
> state returned by the chmod on the dir is not used
> to decide over the access for the second chmod, the
> state resulting from mkdir being used instead until
> its timeout elapses.
>   

I can now confirm the diagnosis : when checking
whether is directory is searchable (in lookup)
with cacheing enabled, fuse only checks the
directory entry timeout, without checking whether
a chmod/chown/setfacl has changed the permissions
on the directory.

As I do not know of any way to cancel the directory
cache in chmod/chown/setfacl, I had to get fuse
to check the attributes time out to decide whether
a new lookup has to be done.

Appended is the patch for that, which relies on
chmod/chown/setfacl to return with a null timeout
(at least when they impact a directory), to force
a new lookup the next time the directory is accessed.

With the patch I get the expected result :

78 -rw-rw-r-- 1 linux linux 5 2009-08-28 09:47 trydir/file
chmod: cannot access `trydir/file': Permission denied
ls: cannot access trydir/file: Permission denied

The patch is a "quick and dirty" one, and I would
really appreciate comments.

Regards

Jean-Pierre

Hi Jean-Pierre,

On Fri, 28 Aug 2009, Jean-Pierre André wrote:
> I can now confirm the diagnosis : when checking
> whether is directory is searchable (in lookup)
> with cacheing enabled, fuse only checks the
> directory entry timeout, without checking whether
> a chmod/chown/setfacl has changed the permissions
> on the directory in the meantime.

Hmm, yes I can see the problem now.  Thanks for investigating!

In fact it's a whole lot more complicated than I first thought...  For
example a directory may have ACL set such that user A has search
permission while user B does not.  In that case checking permissions
in one LOOKUP is not enough, each time the user changes they must be
re-checked again.

One way to solve this would be to go with the "high level lib"
approach of always checking the full path up to the root in the
filesystem.  This has problems with hard links, because the path
leading up to the inode is not unique.  Currently the high level lib
ignores the whole problem of hard links and works as if they didn't
exist (which has its own problems: e.g. hard linked pipes and sockets
don't work the way they should)

Another way would be to move ACL checking into the kernel.  But that
introduces another problem: needing to cache extended attributes for
better performance.

Another approach would be to cache permissions per-user or more
precisely per fsgid/fsuid pair for each inode.  This is perhaps the
most correct solution, but would probably require quite a bit of work
to implement...

Thanks,
Miklos

Hi Miklos,

Thank you for your reply, I really appreciate.

Miklos Szeredi wrote:
> Hi Jean-Pierre,
>
> On Fri, 28 Aug 2009, Jean-Pierre André wrote:
>   
>> I can now confirm the diagnosis : when checking
>> whether is directory is searchable (in lookup)
>> with cacheing enabled, fuse only checks the
>> directory entry timeout, without checking whether
>> a chmod/chown/setfacl has changed the permissions
>> on the directory in the meantime.
>>     
>
> Hmm, yes I can see the problem now.  Thanks for investigating!
>
> In fact it's a whole lot more complicated than I first thought...  For
> example a directory may have ACL set such that user A has search
> permission while user B does not.  In that case checking permissions
> in one LOOKUP is not enough, each time the user changes they must be
> re-checked again.
>   

Please note that my example has nothing to do with ACLs,
and I really do not understand why nobody has reported
about that before. It is a basic situation.

Alas my patch is wrong, though giving correct results
most of the time. It is wrong because it forces a lookup
on a wrong inode, and the result is wrong in other
situations.
> One way to solve this would be to go with the "high level lib"
> approach of always checking the full path up to the root in the
> filesystem.  This has problems with hard links, because the path
> leading up to the inode is not unique.  Currently the high level lib
> ignores the whole problem of hard links and works as if they didn't
> exist (which has its own problems: e.g. hard linked pipes and sockets
> don't work the way they should)
>   

Yes I know that, and that was part of the problem
I wanted to solve.

> Another way would be to move ACL checking into the kernel.  But that
> introduces another problem: needing to cache extended attributes for
> better performance.
>
> Another approach would be to cache permissions per-user or more
> precisely per fsgid/fsuid pair for each inode.  This is perhaps the
> most correct solution, but would probably require quite a bit of work
> to implement...
>   

Well, the correct solution for the ACLs, must be the
patch recently posted here and not to use the timeouts
(which is a pity, because cacheing avoids a lot of calls to
the file system and improves performance significantly).

But why are the lookup cached results not discarded
upon a chmod when there is no ACL in the picture ?
(and why should they be correct at the high level
interface ?)

Thank you for your reply

Jean-Pierre

Miklos Szeredi <miklos@...> writes:
> One way to solve this would be to go with the "high level lib"
> approach of always checking the full path up to the root in the
> filesystem.  This has problems with hard links, because the path
> leading up to the inode is not unique.

The problem is only with *directory* hard links, isn't it? But are they
really used in practice? My impression is that they are a potential
source of trouble in so many cases that whatever advantage they may
offer is just not worth it.  

> Another way would be to move ACL checking into the kernel.  But that
> introduces another problem: needing to cache extended attributes for
> better performance.
>
> Another approach would be to cache permissions per-user or more
> precisely per fsgid/fsuid pair for each inode.  This is perhaps the
> most correct solution, but would probably require quite a bit of work
> to implement...

I agree with that though, no matter if directory hard links are common
or not, caching permissions or xattrs is the better(tm) solution.

Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

Hi Nicolas,

Nikolaus Rath wrote:
> Miklos Szeredi <miklos@...> writes:
>   
>> One way to solve this would be to go with the "high level lib"
>> approach of always checking the full path up to the root in the
>> filesystem.  This has problems with hard links, because the path
>> leading up to the inode is not unique.
>>     
>
> The problem is only with *directory* hard links, isn't it? But are they
> really used in practice? My impression is that they are a potential
> source of trouble in so many cases that whatever advantage they may
> offer is just not worth it. 
>   

This happens when a file is linked into several
directories (hard links on files). Hard links on
directories is not allowed by Posix. Of course you
may prefer not to support hard linked files and
not claim Posix conformance.

>> Another way would be to move ACL checking into the kernel.  But that
>> introduces another problem: needing to cache extended attributes for
>> better performance.
>>
>> Another approach would be to cache permissions per-user or more
>> precisely per fsgid/fsuid pair for each inode.  This is perhaps the
>> most correct solution, but would probably require quite a bit of work
>> to implement...
>>     
>
> I agree with that though, no matter if directory hard links are common
> or not, caching permissions or xattrs is the better(tm) solution.
>   

I will post later measurements in several situations,
(high level, low level, basic permissions, ACLs,
cacheing), which, I hope, will clarify the needs.

Regards

Jean-Pierre

Hi,


Jean-Pierre André wrote:
> Hi Nicolas,
>
> Nikolaus Rath wrote:
>> Miklos Szeredi <miklos@...> writes:
>>  
>
> I will post later measurements in several situations,
> (high level, low level, basic permissions, ACLs,
> cacheing), which, I hope, will clarify the needs.
>

As was promised, I have compared several situations.
They are available with my initial comments at 
http://pagesperso-orange.fr/b.andre/fuse-interfaces.html

However my knowledge of the fuse internal is somewhat
poor and I cannot explain all the results. I may also
have made misinterpretations.

So comments are much welcome.


Regards

Jean-Pierre

On Thu, 10 Sep 2009, Jean-Pierre André wrote:
> Hi,
> As was promised, I have compared several situations.
> They are available with my initial comments at 
> http://pagesperso-orange.fr/b.andre/fuse-interfaces.html

Very interesting.  The numbers seem to indicate that permission
checking in the kernel increases the number of GETATTR calls, which
translates into worse performance...

And I think I know why: if the kernel permission check fails fuse will
unconditionally refresh the attributes (just in case they changed) and
redoes the check.  Can you verify by applying the following patch and
rerunning the cached low-level test?

In any case this should be configurable, so I'll think about a proper
solution.

Thanks,
Miklos


Index: linux-2.6/fs/fuse/dir.c
===================================================================
--- linux-2.6.orig/fs/fuse/dir.c	2009-07-03 13:12:06.000000000 +0200
+++ linux-2.6/fs/fuse/dir.c	2009-09-11 16:23:41.000000000 +0200
@@ -1001,7 +1001,8 @@ static int fuse_permission(struct inode
 		/* If permission is denied, try to refresh file
 		   attributes.  This is also needed, because the root
 		   node will at first have no permissions */
-		if (err == -EACCES && !refreshed) {
+		if (err == -EACCES && !refreshed &&
+		    get_node_id(inode) == FUSE_ROOT_ID) {
 			err = fuse_do_getattr(inode, NULL, NULL);
 			if (!err)
 				err = generic_permission(inode, mask, NULL);

Hi Miklos,

Thank you for analyzing the results.

Your suspected cause does not look like being the
right one. I suspected this because the measures
are related to a quite standard compiling where
permission checks are not likely to fail.

The extra count (about 1750) must be related to
checking whether a directory is writeable. There
were 997 create, 707 unlinks and 42 rename
(same numbers today). However this is assuming
these checks are never based on cached data.

The new counts, with ACLs enabled, and cacheing
enabled were :

a) original kernel module :
4403 getattr for checks by fs
6149 getattr for checks by kernel
9404 getxattr for checks by fs
10816 getxattr for checks by kernel

b) modified kernel module :
4396 getattr for checks by fs
6158 getattr for checks by kernel
9404 getxattr for checks by fs
10816 getxattr for checks by kernel

For getxattr, I have checked what they were
about, and I only found 1412 related to querying
ACLs for checks by kernel and none for checks
by fs. The remaining calls (9404 or 10816-1412)
are queries of security.capability.

So there are no ACL queries when checks are
done by the file system. Good point.

Regards

Jean-Pierre


Miklos Szeredi wrote:
> On Thu, 10 Sep 2009, Jean-Pierre André wrote:
>   
>> Hi,
>> As was promised, I have compared several situations.
>> They are available with my initial comments at 
>> http://pagesperso-orange.fr/b.andre/fuse-interfaces.html
>>     
>
> Very interesting.  The numbers seem to indicate that permission
> checking in the kernel increases the number of GETATTR calls, which
> translates into worse performance...
>
> And I think I know why: if the kernel permission check fails fuse will
> unconditionally refresh the attributes (just in case they changed) and
> redoes the check.  Can you verify by applying the following patch and
> rerunning the cached low-level test?
>
> In any case this should be configurable, so I'll think about a proper
> solution.
>
> Thanks,
> Miklos
>

Hi,

I have updated my figures for two reasons
(see http://pagesperso-orange.fr/b.andre/fuse-interfaces.html)

1) the figures for Posix ACLs in high level were
wrong because the ACLs had not been enabled.
2) I report the number of getacls instead of the
number of getxattr, because most of getxattr
were not related to ACLs.

I also found another broken configuration, and
the conclusion has to be "do not use the cacheing
in any situation" as dirty entries are not discarded,
or show me where I am misusing fuse. I find it
quite strange that nobody has reported such
problems before.

Regards

Jean-Pierre



Miklos Szeredi wrote:
> On Thu, 10 Sep 2009, Jean-Pierre André wrote:
>   
>> Hi,
>> As was promised, I have compared several situations.
>> They are available with my initial comments at 
>> http://pagesperso-orange.fr/b.andre/fuse-interfaces.html
>>     
>
> Very interesting.  The numbers seem to indicate that permission
> checking in the kernel increases the number of GETATTR calls, which
> translates into worse performance...
>
> And I think I know why: if the kernel permission check fails fuse will
> unconditionally refresh the attributes (just in case they changed) and
> redoes the check.  Can you verify by applying the following patch and
> rerunning the cached low-level test?
>
> In any case this should be configurable, so I'll think about a proper
> solution.
>
> Thanks,
> Miklos
>

This is really terrible news as I had already previously discovered an  
posted to the list (my partticular problem was different,  
default_permissions mount option caused caching of stat and access
Calls simply not to get cached at all, and without it there are grave  
security issues And still recently modified files are not cached as  
stated above). Alas I was told "proof or gtfo" on this list even  
though the proof was only a matter to run Zfs fuse daemon with the  
right options.

NOT cool.

Sent from my iPhone

On 12/09/2009, at 5:53, Jean-Pierre André <jean- 
pierre.andre@...> wrote:

> Hi,
>
> I have updated my figures for two reasons
> (see http://pagesperso-orange.fr/b.andre/fuse-interfaces.html)
>
> 1) the figures for Posix ACLs in high level were
> wrong because the ACLs had not been enabled.
> 2) I report the number of getacls instead of the
> number of getxattr, because most of getxattr
> were not related to ACLs.
>
> I also found another broken configuration, and
> the conclusion has to be "do not use the cacheing
> in any situation" as dirty entries are not discarded,
> or show me where I am misusing fuse. I find it
> quite strange that nobody has reported such
> problems before.
>
> Regards
>
> Jean-Pierre
>
>
>
> Miklos Szeredi wrote:
>> On Thu, 10 Sep 2009, Jean-Pierre André wrote:
>>
>>> Hi,
>>> As was promised, I have compared several situations.
>>> They are available with my initial comments at
>>> http://pagesperso-orange.fr/b.andre/fuse-interfaces.html
>>>
>>
>> Very interesting.  The numbers seem to indicate that permission
>> checking in the kernel increases the number of GETATTR calls, which
>> translates into worse performance...
>>
>> And I think I know why: if the kernel permission check fails fuse  
>> will
>> unconditionally refresh the attributes (just in case they changed)  
>> and
>> redoes the check.  Can you verify by applying the following patch and
>> rerunning the cached low-level test?
>>
>> In any case this should be configurable, so I'll think about a proper
>> solution.
>>
>> Thanks,
>> Miklos
>>
>
>
>
>
>
>
> --- 
> --- 
> --- 
> ---------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008  
> 30-Day
> trial. Simplify your report design, integration and deployment - and  
> focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> fuse-devel mailing list
> fuse-devel@...
> https://lists.sourceforge.net/lists/listinfo/fuse-devel

On Fri, 11 Sep 2009, Jean-Pierre André wrote:
> Hi Miklos,
> 
> Thank you for analyzing the results.
> 
> Your suspected cause does not look like being the
> right one. I suspected this because the measures
> are related to a quite standard compiling where
> permission checks are not likely to fail.
> 
> The extra count (about 1750) must be related to
> checking whether a directory is writeable. There
> were 997 create, 707 unlinks and 42 rename
> (same numbers today).

Good observation.  The extra count is not because of checks for
writable, but because these operations invalidate the attribute on the
parent directory (due to size, mtime and nlink changes), and the next
permission check will try to refresh it.  This is not necessary,
however, since the mode bits and the owner/group haven't changed.

So the attribute invalidation could refined to tell the difference
when the attributes which determine permission might also have changed
or only ones which don't have any effect on permission checks.

Can you try this patch?

Thanks,
Miklos

Index: linux-2.6/fs/fuse/dir.c
===================================================================
--- linux-2.6.orig/fs/fuse/dir.c	2009-09-14 11:38:03.000000000 +0200
+++ linux-2.6/fs/fuse/dir.c	2009-09-14 12:11:14.000000000 +0200
@@ -89,6 +89,16 @@ void fuse_invalidate_attr(struct inode *
 	get_fuse_inode(inode)->i_time = 0;
 }
 
+void fuse_invalidate_attr_mask(struct inode *inode, enum fuse_attr_mask attr)
+{
+	struct fuse_conn *fc = get_fuse_conn(inode);
+	struct fuse_inode *fi = get_fuse_inode(inode);
+
+	spin_lock(&fc->lock);
+	fi->attr_invalid |= attr;
+	spin_unlock(&fc->lock);
+}
+
 /*
  * Just mark the entry as stale, so that a next attempt to look it up
  * will result in a new lookup call to userspace
@@ -451,7 +461,7 @@ static int fuse_create_open(struct inode
 	fuse_put_request(fc, forget_req);
 	d_instantiate(entry, inode);
 	fuse_change_entry_timeout(entry, &outentry);
-	fuse_invalidate_attr(dir);
+	fuse_invalidate_attr_mask(dir, FUA_DIRCHANGE);
 	file = lookup_instantiate_filp(nd, entry, generic_file_open);
 	if (IS_ERR(file)) {
 		fuse_sync_release(ff, flags);
@@ -534,7 +544,7 @@ static int create_new_entry(struct fuse_
 		d_instantiate(entry, inode);
 
 	fuse_change_entry_timeout(entry, &outarg);
-	fuse_invalidate_attr(dir);
+	fuse_invalidate_attr_mask(dir, FUA_DIRCHANGE);
 	return 0;
 
  out_put_forget_req:
@@ -646,8 +656,8 @@ static int fuse_unlink(struct inode *dir
 		 * lookup/getattr.
 		 */
 		clear_nlink(inode);
-		fuse_invalidate_attr(inode);
-		fuse_invalidate_attr(dir);
+		fuse_invalidate_attr_mask(inode, FUA_TIME | FUA_NLINK);
+		fuse_invalidate_attr_mask(dir, FUA_DIRCHANGE);
 		fuse_invalidate_entry_cache(entry);
 	} else if (err == -EINTR)
 		fuse_invalidate_entry(entry);
@@ -672,7 +682,7 @@ static int fuse_rmdir(struct inode *dir,
 	fuse_put_request(fc, req);
 	if (!err) {
 		clear_nlink(entry->d_inode);
-		fuse_invalidate_attr(dir);
+		fuse_invalidate_attr_mask(dir, FUA_DIRCHANGE);
 		fuse_invalidate_entry_cache(entry);
 	} else if (err == -EINTR)
 		fuse_invalidate_entry(entry);
@@ -705,11 +715,11 @@ static int fuse_rename(struct inode *old
 	fuse_put_request(fc, req);
 	if (!err) {
 		/* ctime changes */
-		fuse_invalidate_attr(oldent->d_inode);
+		fuse_invalidate_attr_mask(oldent->d_inode, FUA_TIME);
 
-		fuse_invalidate_attr(olddir);
+		fuse_invalidate_attr_mask(olddir, FUA_DIRCHANGE);
 		if (olddir != newdir)
-			fuse_invalidate_attr(newdir);
+			fuse_invalidate_attr_mask(newdir, FUA_DIRCHANGE);
 
 		/* newent will end up negative */
 		if (newent->d_inode)
@@ -755,7 +765,7 @@ static int fuse_link(struct dentry *entr
 	   etc.)
 	*/
 	if (!err || err == -EINTR)
-		fuse_invalidate_attr(inode);
+		fuse_invalidate_attr_mask(inode, FUA_NLINK | FUA_TIME);
 	return err;
 }
 
@@ -835,13 +845,14 @@ static int fuse_do_getattr(struct inode
 }
 
 int fuse_update_attributes(struct inode *inode, struct kstat *stat,
-			   struct file *file, bool *refreshed)
+			   struct file *file, bool *refreshed,
+			   enum fuse_attr_mask mask)
 {
 	struct fuse_inode *fi = get_fuse_inode(inode);
 	int err;
 	bool r;
 
-	if (fi->i_time < get_jiffies_64()) {
+	if (fi->i_time < get_jiffies_64() || (fi->attr_invalid & mask)) {
 		r = true;
 		err = fuse_do_getattr(inode, stat, file);
 	} else {
@@ -990,7 +1001,7 @@ static int fuse_permission(struct inode
 	 */
 	if ((fc->flags & FUSE_DEFAULT_PERMISSIONS) ||
 	    ((mask & MAY_EXEC) && S_ISREG(inode->i_mode))) {
-		err = fuse_update_attributes(inode, NULL, NULL, &refreshed);
+		err = fuse_update_attributes(inode, NULL, NULL, &refreshed, 0);
 		if (err)
 			return err;
 	}
@@ -1085,7 +1096,7 @@ static int fuse_readdir(struct file *fil
 				    filldir);
 
 	__free_page(page);
-	fuse_invalidate_attr(inode); /* atime changed */
+	fuse_invalidate_attr_mask(inode, FUA_TIME);
 	return err;
 }
 
@@ -1118,7 +1129,7 @@ static char *read_link(struct dentry *de
 		link[req->out.args[0].size] = '\0';
  out:
 	fuse_put_request(fc, req);
-	fuse_invalidate_attr(inode); /* atime changed */
+	fuse_invalidate_attr_mask(inode, FUA_TIME);
 	return link;
 }
 
@@ -1381,7 +1392,7 @@ static int fuse_getattr(struct vfsmount
 	if (!fuse_allow_task(fc, current))
 		return -EACCES;
 
-	return fuse_update_attributes(inode, stat, NULL, NULL);
+	return fuse_update_attributes(inode, stat, NULL, NULL, FUA_ALL);
 }
 
 static int fuse_setxattr(struct dentry *entry, const char *name,
Index: linux-2.6/fs/fuse/fuse_i.h
===================================================================
--- linux-2.6.orig/fs/fuse/fuse_i.h	2009-09-14 11:38:03.000000000 +0200
+++ linux-2.6/fs/fuse/fuse_i.h	2009-09-14 12:10:42.000000000 +0200
@@ -55,6 +55,14 @@ extern struct list_head fuse_conn_list;
 /** Global mutex protecting fuse_conn_list and the control filesystem */
 extern struct mutex fuse_mutex;
 
+enum fuse_attr_mask {
+	FUA_SIZE	= (1 << 0),
+	FUA_NLINK	= (1 << 1),
+	FUA_TIME	= (1 << 2),
+	FUA_DIRCHANGE	= FUA_NLINK | FUA_SIZE | FUA_TIME,
+	FUA_ALL		= ~0U,
+};
+
 /** FUSE inode */
 struct fuse_inode {
 	/** Inode data */
@@ -80,6 +88,9 @@ struct fuse_inode {
 	/** Version of last attribute change */
 	u64 attr_version;
 
+	/** Mask indicating which attributes may be invalid */
+	enum fuse_attr_mask attr_invalid;
+
 	/** Files usable in writepage.  Protected by fc->lock */
 	struct list_head write_files;
 
@@ -671,6 +682,8 @@ void fuse_abort_conn(struct fuse_conn *f
  */
 void fuse_invalidate_attr(struct inode *inode);
 
+void fuse_invalidate_attr_mask(struct inode *inode, enum fuse_attr_mask attr);
+
 void fuse_invalidate_entry_cache(struct dentry *entry);
 
 /**
@@ -713,7 +726,8 @@ int fuse_allow_task(struct fuse_conn *fc
 u64 fuse_lock_owner_id(struct fuse_conn *fc, fl_owner_t id);
 
 int fuse_update_attributes(struct inode *inode, struct kstat *stat,
-			   struct file *file, bool *refreshed);
+			   struct file *file, bool *refreshed,
+			   enum fuse_attr_mask mask);
 
 void fuse_flush_writepages(struct inode *inode);
 
Index: linux-2.6/fs/fuse/inode.c
===================================================================
--- linux-2.6.orig/fs/fuse/inode.c	2009-09-14 11:38:03.000000000 +0200
+++ linux-2.6/fs/fuse/inode.c	2009-09-14 11:38:08.000000000 +0200
@@ -131,6 +131,7 @@ void fuse_change_attributes_common(struc
 
 	fi->attr_version = ++fc->attr_version;
 	fi->i_time = attr_valid;
+	fi->attr_invalid = 0;
 
 	inode->i_ino     = attr->ino;
 	inode->i_mode    = (inode->i_mode & S_IFMT) | (attr->mode & 07777);
Index: linux-2.6/fs/fuse/file.c
===================================================================
--- linux-2.6.orig/fs/fuse/file.c	2009-09-14 11:38:03.000000000 +0200
+++ linux-2.6/fs/fuse/file.c	2009-09-14 11:38:08.000000000 +0200
@@ -506,7 +506,7 @@ static int fuse_readpage(struct file *fi
 		SetPageUptodate(page);
 	}
 
-	fuse_invalidate_attr(inode); /* atime changed */
+	fuse_invalidate_attr_mask(inode, FUA_TIME);
  out:
 	unlock_page(page);
 	return err;
@@ -527,7 +527,7 @@ static void fuse_readpages_end(struct fu
 		fuse_read_update_size(inode, pos, req->misc.read.attr_ver);
 	}
 
-	fuse_invalidate_attr(inode); /* atime changed */
+	fuse_invalidate_attr_mask(inode, FUA_TIME);
 
 	for (i = 0; i < req->num_pages; i++) {
 		struct page *page = req->pages[i];
@@ -635,7 +635,8 @@ static ssize_t fuse_file_aio_read(struct
 		 * If trying to read past EOF, make sure the i_size
 		 * attribute is up-to-date.
 		 */
-		err = fuse_update_attributes(inode, NULL, iocb->ki_filp, NULL);
+		err = fuse_update_attributes(inode, NULL, iocb->ki_filp, NULL,
+					     FUA_SIZE);
 		if (err)
 			return err;
 	}
@@ -744,7 +745,7 @@ static int fuse_buffered_write(struct fi
 		if (count == PAGE_CACHE_SIZE)
 			SetPageUptodate(page);
 	}
-	fuse_invalidate_attr(inode);
+	fuse_invalidate_attr_mask(inode, FUA_TIME | FUA_SIZE);
 	return err ? err : nres;
 }
 
@@ -905,7 +906,7 @@ static ssize_t fuse_perform_write(struct
 	if (res > 0)
 		fuse_write_update_size(inode, pos);
 
-	fuse_invalidate_attr(inode);
+	fuse_invalidate_attr_mask(inode, FUA_TIME | FUA_SIZE);
 
 	return res > 0 ? res : err;
 }
@@ -1082,7 +1083,7 @@ static ssize_t fuse_direct_read(struct f
 
 	res = fuse_direct_io(file, buf, count, ppos, 0);
 
-	fuse_invalidate_attr(inode);
+	fuse_invalidate_attr_mask(inode, FUA_TIME);
 
 	return res;
 }
@@ -1106,7 +1107,7 @@ static ssize_t fuse_direct_write(struct
 	}
 	mutex_unlock(&inode->i_mutex);
 
-	fuse_invalidate_attr(inode);
+	fuse_invalidate_attr_mask(inode, FUA_TIME | FUA_SIZE);
 
 	return res;
 }
@@ -1541,7 +1542,8 @@ static loff_t fuse_file_llseek(struct fi
 	mutex_lock(&inode->i_mutex);
 	switch (origin) {
 	case SEEK_END:
-		retval = fuse_update_attributes(inode, NULL, file, NULL);
+		retval = fuse_update_attributes(inode, NULL, file, NULL,
+						FUA_SIZE);
 		if (retval)
 			goto exit;
 		offset += i_size_read(inode);

On Sun, 13 Sep 2009, Manuel Amador (Rudd-O) wrote:
> This is really terrible news as I had already previously discovered an  
> posted to the list (my partticular problem was different,  
> default_permissions mount option caused caching of stat and access
> Calls simply not to get cached at all, and without it there are grave  
> security issues And still recently modified files are not cached as  
> stated above). Alas I was told "proof or gtfo" on this list even  
> though the proof was only a matter to run Zfs fuse daemon with the  
> right options.

Not exactly how I remember: I asked you politely if you could run the
example programs.  The "gtfo" part is purely your invention.

People sometimes think that I have infinite time to set up and debug
their filesystems, unfortunately that is not the case.  I want to
solve any fuse related problems, but I will not (and sometimes cannot)
solve them without help from the person experiencing the problem.

Thanks,
Miklos

Hi Miklos,

Thank you for trying to solve the issue.

Miklos Szeredi wrote:
> On Fri, 11 Sep 2009, Jean-Pierre André wrote:
>   
>> Hi Miklos,
>>
>> Thank you for analyzing the results.
>>
>> Your suspected cause does not look like being the
>> right one. I suspected this because the measures
>> are related to a quite standard compiling where
>> permission checks are not likely to fail.
>>
>> The extra count (about 1750) must be related to
>> checking whether a directory is writeable. There
>> were 997 create, 707 unlinks and 42 rename
>> (same numbers today).
>>     
>
> Good observation.  The extra count is not because of checks for
> writable, but because these operations invalidate the attribute on the
> parent directory (due to size, mtime and nlink changes), and the next
> permission check will try to refresh it.  This is not necessary,
> however, since the mode bits and the owner/group haven't changed.
>
> So the attribute invalidation could refined to tell the difference
> when the attributes which determine permission might also have changed
> or only ones which don't have any effect on permission checks.
>   

I understand you are trying to get the permission flags
refreshed for the file on which they were changed.
But I think the main problem is after changing permissions
on a directory, when you have to update access to files
within the directory, and I suspect this is located
elsewhere.

> Can you try this patch?
>   

I am using kernel 2.6.30 and a couple of modifications
did not get, so I had to patch two functions manually.
This is what I have actually tried :

static ssize_t fuse_direct_read(struct file *file, char __user *buf,
                                     size_t count, loff_t *ppos)
{
//      return fuse_direct_io(file, buf, count, ppos, 0);
        struct inode *inode = file->f_path.dentry->d_inode;
        ssize_t res = fuse_direct_io(file, buf, count, ppos, 0);
       
        fuse_invalidate_attr_mask(inode, FUA_TIME);
 
        return res;
}      

static ssize_t fuse_direct_write(struct file *file, const char __user *buf,
                                 size_t count, loff_t *ppos)
{
        struct inode *inode = file->f_path.dentry->d_inode;
        ssize_t res;
        /* Don't allow parallel writes to the same file */
        mutex_lock(&inode->i_mutex);
        res = generic_write_checks(file, ppos, &count, 0);
        if (!res)
                res = fuse_direct_io(file, buf, count, ppos, 1);
 
        mutex_unlock(&inode->i_mutex);
       
        fuse_invalidate_attr_mask(inode, FUA_TIME | FUA_SIZE);

        return res;
}

And the number of calls for getattr at low level are :

kernel checks and no cache : 79500
kernel checks and cache : 4254 and 4217 (two tries)
fs checks and no cache : 9554
fs checks and cache : 4218

So the number of getattr calls is now the same for
the two checking modes (disregarding small
differences due to variations on execution times)

But I do not see much difference when it comes to
access control. You just have saved unneeded calls.

Regards

Jean-Pierre

> access control. You just have saved unneeded calls.

Right, that was my purpose.  Which means that the no-ACL, low-level,
caching config should have the same performance with and without
'default_permissions' now.  Can you verify that?

Using 'default_permissions' with caching should be the easiest to
implement properly, that is why I'd like to get its performance as
high as without 'default_permissions'.

(And I'd like to deal with the ACL problems separately once the
non-ACL issues have been resolved)

Thanks,
Miklos

Hi again,

Miklos Szeredi wrote:
>> access control. You just have saved unneeded calls.
>>     
>
> Right, that was my purpose.  Which means that the no-ACL, low-level,
> caching config should have the same performance with and without
> 'default_permissions' now.  Can you verify that?
>   

Well, this does not really translate to better performance.

with 'default_permissions' (permissions checked by kernel),
I got 29.42s and 29.92s (two tries) for the alsa compilation,
and without (permissions checked by file system), I got
26.00s and 25.70s.

These figures are higher than the ones I got previously, so I
have restarted the tests with the standard fuse kernel, and
I got 28.79s (kernel) and 25.69 (fs). The old times were
27.88 (kernel) and 24.11. I am not sure the differences between
fuse versions are significant, but the differences between
kernel and fs checks are always greater than 3s and must
be significant.

There must be some transfer of load. The differences in the
number of inode openings are not significant :
kernel and standard fuse : 74225
kernel and patched fuse : 72924
fs and standard fuse : 72131
fs and patched fuse : 71937
note : 74225-72924 is less than the number of getattrs you
saved (which is around 1750). Probably checks on non-existent
files.

> Using 'default_permissions' with caching should be the easiest to
> implement properly, that is why I'd like to get its performance as
> high as without 'default_permissions'.
>   

That is reasonable, but what is the point of measuring the
performances before the cache is fixed ?

> (And I'd like to deal with the ACL problems separately once the
> non-ACL issues have been resolved)
>   

I agree with the steps. I only hope to get a fix before
I get out of business.

Regards

Jean-Pierre

El Lunes 14 Septiembre 2009, Miklos Szeredi escribió:
> On Sun, 13 Sep 2009, Manuel Amador (Rudd-O) wrote:
> > This is really terrible news as I had already previously discovered an
> > posted to the list (my partticular problem was different,
> > default_permissions mount option caused caching of stat and access
> > Calls simply not to get cached at all, and without it there are grave
> > security issues And still recently modified files are not cached as
> > stated above). Alas I was told "proof or gtfo" on this list even
> > though the proof was only a matter to run Zfs fuse daemon with the
> > right options.
>
> Not exactly how I remember: I asked you politely if you could run the
> example programs.  The "gtfo" part is purely your invention.

OK, OK, you did indeed tell me politely but abruptly.

>
> People sometimes think that I have infinite time to set up and debug
> their filesystems, unfortunately that is not the case.  I want to
> solve any fuse related problems, but I will not (and sometimes cannot)
> solve them without help from the person experiencing the problem.

Well, we now have a confirmation of the problem and we can move forward with 
the solution.  What would you like me to do to help?

>
> Thanks,
> Miklos

On Mon, 14 Sep 2009, Jean-Pierre André wrote:
> Hi again,
> 
> Miklos Szeredi wrote:
> >> access control. You just have saved unneeded calls.
> >>     
> >
> > Right, that was my purpose.  Which means that the no-ACL, low-level,
> > caching config should have the same performance with and without
> > 'default_permissions' now.  Can you verify that?
> >   
> 
> Well, this does not really translate to better performance.
> 
> with 'default_permissions' (permissions checked by kernel),
> I got 29.42s and 29.92s (two tries) for the alsa compilation,
> and without (permissions checked by file system), I got
> 26.00s and 25.70s.
> 
> These figures are higher than the ones I got previously, so I
> have restarted the tests with the standard fuse kernel, and
> I got 28.79s (kernel) and 25.69 (fs). The old times were
> 27.88 (kernel) and 24.11. I am not sure the differences between
> fuse versions are significant, but the differences between
> kernel and fs checks are always greater than 3s and must
> be significant.
> 
> There must be some transfer of load. The differences in the
> number of inode openings are not significant :
> kernel and standard fuse : 74225
> kernel and patched fuse : 72924
> fs and standard fuse : 72131
> fs and patched fuse : 71937
> note : 74225-72924 is less than the number of getattrs you
> saved (which is around 1750). Probably checks on non-existent
> files.

Okay, I have no other ideas what could be causing the slowdown.

Could you please try to profile (see 'man gprof') the filesystem to
see where the extra time is going?

> > Using 'default_permissions' with caching should be the easiest to
> > implement properly, that is why I'd like to get its performance as
> > high as without 'default_permissions'.
> >   
> 
> That is reasonable, but what is the point of measuring the
> performances before the cache is fixed ?

Caching doesn't work properly without 'default_permissions'.  That's
something I won't be able to fix quickly.

So first I rather want to fix the performance problems associated with
kernel checking of permissions (default_permissions), which _should_
be easy, we just need to find the cause of it, there's nothing
fundamental that would make it slower.

Thanks,
Miklos

On Mon, 14 Sep 2009, Manuel Amador (Rudd-O) wrote:
> > People sometimes think that I have infinite time to set up and debug
> > their filesystems, unfortunately that is not the case.  I want to
> > solve any fuse related problems, but I will not (and sometimes cannot)
> > solve them without help from the person experiencing the problem.
> 
> Well, we now have a confirmation of the problem and we can move forward with 
> the solution.  What would you like me to do to help?

As I said, you could run the "fusexmp_fh" filesystem to see if you can
reproduce the problem.  I couldn't reproduce it, I get the same high
performance with or without 'default_permissions' and with or without
touching the file.

If you can reproduce it with the example filesystem, that means
there's some difference between our setups.

If you can't reproduce it with the example filesystem, then zfs-fuse
is doing something differently.  That doesn't necessarily mean the
slowdown is zfs-fuse's fault.  At that point it would be interesting
to see debug traces of fuse requests (passing "debug" option to
libfuse) and perhaps profiling zfs-fuse with gprof.

Thanks,
Miklos

> So first I rather want to fix the performance problems associated with
> kernel checking of permissions (default_permissions), which _should_
> be easy, we just need to find the cause of it, there's nothing
> fundamental that would make it slower.

I agree with you.  My observation is that putting -o default_permissions as 
mount option (for filesystems coded using the low-level interface) basically 
causes the kernel NOT to cache any lookups whatsoever (i.e. every stat() or 
access() call results in a corresponding transaction against the userspace 
filesystem).

>
> Thanks,
> Miklos
>
> ---------------------------------------------------------------------------
>--- Come build with us! The BlackBerry® Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9-12, 2009. Register now!
> http://p.sf.net/sfu/devconf
> _______________________________________________
> fuse-devel mailing list
> fuse-devel@...
> https://lists.sourceforge.net/lists/listinfo/fuse-devel

On Wed, 16 Sep 2009, Manuel Amador (Rudd-O) wrote:
> I agree with you.  My observation is that putting -o default_permissions as 
> mount option (for filesystems coded using the low-level interface) basically 
> causes the kernel NOT to cache any lookups whatsoever (i.e. every stat() or 
> access() call results in a corresponding transaction against the userspace 
> filesystem).

There could be several reasons:

a) kernel permission check is failing (unlikely in normal situations)
b) attributes are invalidated because of a read or write
c) filesystem is not setting attr_timeout to a large enough value

If ZFS is setting up the cache timeouts properly then you should check
out the patches I posted for a) and b) here:

  http://article.gmane.org/gmane.comp.file-systems.fuse.devel/8170

and here:

  http://article.gmane.org/gmane.comp.file-systems.fuse.devel/8177

Thanks,
Miklos

Hi Miklos,

Miklos Szeredi wrote:
> On Mon, 14 Sep 2009, Jean-Pierre André wrote:
>   
>> Hi again,
>>
>> Miklos Szeredi wrote:
>>     
>>>> access control. You just have saved unneeded calls.
>>>>     
>>>>         
>>> Right, that was my purpose.  Which means that the no-ACL, low-level,
>>> caching config should have the same performance with and without
>>> 'default_permissions' now.  Can you verify that?
>>>   
>>>       
>> Well, this does not really translate to better performance.
>>
>> with 'default_permissions' (permissions checked by kernel),
>> I got 29.42s and 29.92s (two tries) for the alsa compilation,
>> and without (permissions checked by file system), I got
>> 26.00s and 25.70s.
>>
>> These figures are higher than the ones I got previously, so I
>> have restarted the tests with the standard fuse kernel, and
>> I got 28.79s (kernel) and 25.69 (fs). The old times were
>> 27.88 (kernel) and 24.11. I am not sure the differences between
>> fuse versions are significant, but the differences between
>> kernel and fs checks are always greater than 3s and must
>> be significant.
>>
>> There must be some transfer of load. The differences in the
>> number of inode openings are not significant :
>> kernel and standard fuse : 74225
>> kernel and patched fuse : 72924
>> fs and standard fuse : 72131
>> fs and patched fuse : 71937
>> note : 74225-72924 is less than the number of getattrs you
>> saved (which is around 1750). Probably checks on non-existent
>> files.
>>     
>
> Okay, I have no other ideas what could be causing the slowdown.
>
> Could you please try to profile (see 'man gprof') the filesystem to
> see where the extra time is going?
>   

I have put the profiling results into
http://pagesperso-orange.fr/b.andre/gprof-ntfs.tar.gz
The times have not been recorded, I do not know why,
I do not see an option related to time measurement.
What are the options you recommend ?

There are eight files, all for low-level interface,
with cacheing and no ACL.

The files gprof-ntfs* are about compiling ntfs-3g
the files gprof-alsa* are about compiling alsa

the suffix 3 means kernel permission checking
the suffix 6 means fs permission checking
the suffixes 3b and 6b mean using your recent
patch.

Note the inode openings (ntfs_inode_open) :
they are much more frequent when kernel checks
are used...

>>> Using 'default_permissions' with caching should be the easiest to
>>> implement properly, that is why I'd like to get its performance as
>>> high as without 'default_permissions'.
>>>   
>>>       
>> That is reasonable, but what is the point of measuring the
>> performances before the cache is fixed ?
>>     
>
> Caching doesn't work properly without 'default_permissions'.  That's
> something I won't be able to fix quickly.
>
> So first I rather want to fix the performance problems associated with
> kernel checking of permissions (default_permissions), which _should_
> be easy, we just need to find the cause of it, there's nothing
> fundamental that would make it slower.
>   

Ok to consider that as a priority, I just hope the cache
problem will disappear in the process.

Regards

Jean-Pierre

> Note the inode openings (ntfs_inode_open) :
> they are much more frequent when kernel checks
> are used...

Seems to confirm my observation of increased lookup calls when -o 
default_permissions is specified to the mount program.