Filesystem in Userspace

Miklos Szeredi <miklos@...> wrote:
>
> Hi Andrew!
> 
> Do you plan to send FUSE to Linus for 2.6.14?

Haven't thought about it all much.  Have spent most of my time in the last
month admiring the contents of kernel bugzilla, and the ongoing attempts to
increase them.

> I know you have some doubts about usefulness, etc.  Here are a couple
> of facts, that I hope show that Linux should benefit from having FUSE:
> 
>  - total number of downloads from SF: ~25000
> 
>  - number of downloads of last release (during 3 months): ~7000
> 
>  - number of distros carrying official packages: 2 (debian, gentoo)
> 
>  - number of publicly available filesystems known: 27
> 
>  - of which at least 2 are carried by debian (and maybe others)
> 
>  - number of language bindings: 7 (native: C, java, python, perl, C#, sh, TCL)
> 
>  - biggest known commercial user: ~110TB exported, total bandwidth: 1.5TB/s
> 
>  - mailing list traffic 100-200 messages/month
> 
>  - have been in -mm since 2005 january
> 

I agree that lots of people would like the functionality.  I regret that
although it appears that v9fs could provide it, there seems to be no
interest in working on that.

The main sticking point with FUSE remains the permission tricks around
fuse_allow_task().  AFAIK it remains the case that nobody has come up with
any better idea, so I'm inclined to merge the thing.

On Friday 02 September 2005 15:34, Andrew Morton wrote:
> Miklos Szeredi <miklos@...> wrote:
> > Hi Andrew!
> >
> > Do you plan to send FUSE to Linus for 2.6.14?
=2E..
> I agree that lots of people would like the functionality.  I regret that
> although it appears that v9fs could provide it, there seems to be no
> interest in working on that.

I evaluated both v9fs and FUSE for my project (I don't want to link to it=20
until it does something actually useful ;) ) ... and it seemed that v9fs=20
just wasn't UNIXy enough for my purposes -- the Plan9 way and the UNIX way=
=20
were different enough to make me nervous.  I don't remember the specific=20
details (this was a few months ago), but I do remember that v9fs had no=20
extended attribute support, which was a showstopper for me.  Also, I=20
couldn't find any userspace library for writing 9P servers.

Others may have reached similar conclusions.  Or maybe FUSE is just=20
better-marketed. ;)

Either way, I am a happy FUSE user.  I think it offers things v9fs doesn't,=
=20
and I'd like to see it in mainline. :)

=2D- Josh

=2D-=20
Joshua J. Berry

"I haven't lost my mind -- it's backed up on tape somewhere."
    -- /usr/games/fortune

On Fri, 2005-09-02 at 15:34 -0700, Andrew Morton wrote:
> Miklos Szeredi <miklos@...> wrote:
> >
> > Hi Andrew!
> > 
> > Do you plan to send FUSE to Linus for 2.6.14?
> 
<snip>

i use fuse too, and i like it, it works good, and its quite fast and
easy. it has given me no problems at all, i suggest merging, it harms
nothing, and seems to be well maintained

> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@...
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

> Haven't thought about it all much.  Have spent most of my time in the last
> month admiring the contents of kernel bugzilla, and the ongoing attempts to
> increase them.

A penal system could be created, for example if someone is caught
introducing a bug, he will have to choose three additional reports
from bugzilla and analyze/fix them ;)

> >  - number of language bindings: 7 (native: C, java, python, perl,
> >  - C#, sh, TCL)

8 now, someone just sent a private mail about bindings for the Pliant
(never heard of it) language.

> I agree that lots of people would like the functionality.  I regret that
> although it appears that v9fs could provide it,

I think you are wrong there.  You don't appreciate all the complexity
FUSE _lacks_ by not being network transparent.  Just look at the error
text to errno conversion muck that v9fs has.  And their problems with
trying to do generic uid/gid mappings.

> there seems to be no interest in working on that.

It would mean adding a plethora of extensions to the 9P protocol, that
would take away all it's beauty.  I think you should realize that
these are different interfaces for different purposes. There may be
some overlap, but not enough to warrant trying to massage them into
one big ball.

> The main sticking point with FUSE remains the permission tricks around
> fuse_allow_task().  AFAIK it remains the case that nobody has come up with
> any better idea, so I'm inclined to merge the thing.

Do you promise?  I can do a resplit and submit to Linus, if that takes
some load off you.

Thanks,
Miklos

Miklos Szeredi <miklos@...> wrote:
>
>  > The main sticking point with FUSE remains the permission tricks around
>  > fuse_allow_task().  AFAIK it remains the case that nobody has come up with
>  > any better idea, so I'm inclined to merge the thing.
> 
>  Do you promise?

I troll.  What others think matters.  But at this stage, objections would
need to be substantial, IMO.  We're rather deadlocked on the permission
thing, but if we can't come up with anything better then I'm inclined to
say what-the-hell.

>   I can do a resplit and submit to Linus, if that takes
>  some load off you.

Nah, then I'd just have to check that everything is the same.

> >  > The main sticking point with FUSE remains the permission tricks around
> >  > fuse_allow_task().  AFAIK it remains the case that nobody has come up with
> >  > any better idea, so I'm inclined to merge the thing.
> > 
> >  Do you promise?
> 
> I troll.  What others think matters.  But at this stage, objections would
> need to be substantial, IMO.

Fair enough.

> We're rather deadlocked on the permission thing, but if we can't
> come up with anything better then I'm inclined to say what-the-hell.

There's no disadvantage IMO.  It adds nearly zero complexity.  If
someone doesn't like it, it can be configured out in userspace.  And
it leaves no legacy interfaces to support if later a better method is
found.

> >   I can do a resplit and submit to Linus, if that takes
> >  some load off you.
> 
> Nah, then I'd just have to check that everything is the same.

OK.

Thanks,
Miklos

On Sat, 03 Sep 2005 03:02:18 +0500, Miklos Szeredi <miklos@...>  
wrote:

>
>  - number of distros carrying official packages: 2
>  (debian, gentoo)
>

do not forget about Source Mage GNU/Linux, spell exists and supports 2.3.0  
and CVS versions of FUSE ;-)

>
>
>>>  - number of language bindings: 7 (native: C, java, python, perl,
>>>  - C#, sh, TCL)
>>>
>
> 8 now, someone just sent a private mail about bindings for the Pliant
> (never heard of it) language.
>

9 now (there is an ocaml binding, and if you dont know ocaml, shame  
on you).

I would just like to add
  "please, merge fuse in linux, pleaseeeeeeeeeeee"
I am an happy user of debian so I have no problem, but when I want other
people to install my fuse-advanced-not-yet-public-like-spotlight-and- 
winfs-just-beter filesystem,
then there is a big problem.

On 9/3/05, Miklos Szeredi <miklos@...> wrote:
>=20
> > I agree that lots of people would like the functionality.  I regret tha=
t
> > although it appears that v9fs could provide it,
>=20
> I think you are wrong there.  You don't appreciate all the complexity
> FUSE _lacks_ by not being network transparent.  Just look at the error
> text to errno conversion muck that v9fs has.  And their problems with
> trying to do generic uid/gid mappings.
>

While FUSE doesn't handle it directly, doesn't it have to punt it to
its network file systems, how to the sshfs and what not handle this
sort of mapping?  Not really a criticism, just curious.  This doesn't
so much relate to FUSE, but I've been wrestling with what to do about
this chunk of (mapping) code -- it seems like it might be a good idea
to have some common code shared amongst the networked file systems to
handle this sort of thing.  The NFS idmapd service seems
overcomplicated, but something like that in the common code could
provide the same level of service.  What do folks think? Should
someone (me?) take a whack at a common id mapping service for the
kernel (or just extract idmapd from NFS) -- or is this something
better implemented filesystem-to-filesystem?
=20
> > there seems to be no interest in working on that.
>=20
> It would mean adding a plethora of extensions to the 9P protocol, that
> would take away all it's beauty.  I think you should realize that
> these are different interfaces for different purposes. There may be
> some overlap, but not enough to warrant trying to massage them into
> one big ball.
>=20

A very good point.  I toyed with the idea of looking at creating a
FUSE-API-compatible v9fs file server library - but there are a good
deal of features (like extended attributes) that we don't have
provisions for in the protocol -- and most likely a good deal of
complexity supporting some of these features  that we may not want to
deal with just yet.

Miklos is right, for the moment FUSE and v9fs have some overlap, but
they remain very different things.  FUSE is far more focused on
delivering user-space file servers, and as such has a better solution
for developing user-space file servers.  We are still focusing on
getting the core of v9fs worked out, when we eventually have that
working smoothly, I like to think we'd be able to spend some time
developing a file server SDK as rich as FUSE (perhaps something
API-compatible as I mentioned before) -- but we want to focus on
getting the core protocol implementation right first - since it has
uses beyond user-space file servers.

         -eric

> While FUSE doesn't handle it directly, doesn't it have to punt it to
> its network file systems, how to the sshfs and what not handle this
> sort of mapping?

Sshfs handles it by not handling it.  In this case it is neither
possible, nor needed to be able to correctly map the id space.

Yes, it may confuse the user.  It may even confuse the kernel for
sticky directories(*).  But basically it just works, and is very
simple.

> Not really a criticism, just curious.  This doesn't so much relate
> to FUSE, but I've been wrestling with what to do about this chunk of
> (mapping) code -- it seems like it might be a good idea to have some
> common code shared amongst the networked file systems to handle this
> sort of thing.  The NFS idmapd service seems overcomplicated, but
> something like that in the common code could provide the same level
> of service.  What do folks think? Should someone (me?) take a whack
> at a common id mapping service for the kernel (or just extract
> idmapd from NFS) -- or is this something better implemented
> filesystem-to-filesystem?

If more than one filesystem would use it, it would make sense to
abstract it out.  FUSE doesn't need it since it can happily do the
mapping in userspace.

Miklos

(*) I think the correct behavior would be if checking sticky
permissions could also be delegated to the filesystem, like checking
normal permissions can be.

On 9/3/05, Miklos Szeredi <miklos@...> wrote:
> > While FUSE doesn't handle it directly, doesn't it have to punt it to
> > its network file systems, how to the sshfs and what not handle this
> > sort of mapping?
>=20
> Sshfs handles it by not handling it.  In this case it is neither
> possible, nor needed to be able to correctly map the id space.
>=20
> Yes, it may confuse the user.  It may even confuse the kernel for
> sticky directories(*).  But basically it just works, and is very
> simple.
>=20

In principal, Plan 9 file servers handle permission checking
server-side, so we could likewise punt -- but it seemed a good idea to
have some form of mapping for directory listings (and things like
sticky directories) to make sense.

               -eric

> > Yes, it may confuse the user.  It may even confuse the kernel for
> > sticky directories(*).  But basically it just works, and is very
> > simple.
> > 
> 
> In principal, Plan 9 file servers handle permission checking
> server-side, so we could likewise punt -- but it seemed a good idea to
> have some form of mapping for directory listings (and things like
> sticky directories) to make sense.

Yes if the user/group names are available (as in 9P), then doing the
mapping based on /etc/passwd for example makes sense.

But sshfs only transfers the numeric uid/gid, and hence there's simply
no info to base any transformation on.

It could transfer /etc/passwd from the remote server, and use that to
do mapping, but that is getting more complex than the problem actually
warrants IMO.

Miklos

Since FUSE is being included into the mainline kernel, it would make sense to provide userspace parts of fuse as separate packages, especially for distributions, that will provide the fuse module in their kernel, but not the userspace parts.

Is there any initiative to provide such packages in the FUSE project? Is there a plan to provide such packages, or will that be left entirely to the distributions to provide (or not)?

regards,
Rok

Hi Miklos,

On Fri, 2005-09-16 at 13:25 +0200, Rok Ruzic wrote:
> Since FUSE is being included into the mainline kernel, it would make
> sense to provide userspace parts of fuse as separate packages,
> especially for distributions, that will provide the fuse module in
> their kernel, but not the userspace parts.
> 
> Is there any initiative to provide such packages in the FUSE project?
> Is there a plan to provide such packages, or will that be left
> entirely to the distributions to provide (or not)?

Interesting question, but whilst talking about this I wanted to point
out that you should update linux/Documentation/Changes to state the
minimum fuse-utils (or whatever it is going to be called if you are
going to make such a package) version for that kernel.  And then please
keep it up-to-date as you update fuse...  It would be good to have such
a version statement in there before 2.6.14 comes out.

Best regards,

        Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

> Since FUSE is being included into the mainline kernel, it would make
> sense to provide userspace parts of fuse as separate packages,
> especially for distributions, that will provide the fuse module in
> their kernel, but not the userspace parts.

The source package for FUSE already automatically detects, whether the
kernel module already exists (and is configured in, or is a module) in
the current kernel, and in that case does not compile and install it.
This can be overridden by a configure option.

I don't plan to split the source package.

> Is there any initiative to provide such packages in the FUSE
> project? Is there a plan to provide such packages, or will that be
> left entirely to the distributions to provide (or not)?

It's up to the distributions.  I know that debian and gentoo already
provide official packages, and there are unofficial packages for
fedora.

If you lack FUSE support on your favourite distro, please ask them to
provide one.

Miklos

> On Fri, 2005-09-16 at 13:25 +0200, Rok Ruzic wrote:
> > Since FUSE is being included into the mainline kernel, it would make
> > sense to provide userspace parts of fuse as separate packages,
> > especially for distributions, that will provide the fuse module in
> > their kernel, but not the userspace parts.
> > 
> > Is there any initiative to provide such packages in the FUSE project?
> > Is there a plan to provide such packages, or will that be left
> > entirely to the distributions to provide (or not)?
> 
> Interesting question, but whilst talking about this I wanted to point
> out that you should update linux/Documentation/Changes to state the
> minimum fuse-utils (or whatever it is going to be called if you are
> going to make such a package) version for that kernel.  And then please
> keep it up-to-date as you update fuse...  It would be good to have such
> a version statement in there before 2.6.14 comes out.

OK.  I hope to release FUSE-2.4.0 soonish, so distributions can pick
it up before 2.6.14 comes out.

It mostly works with 2.3.0 too, the library only needs to be updated
if the mount options 'direct_io' or 'kernel_cache' are used.

Miklos

Miklos Szeredi wrote:

> there are unofficial packages for fedora

Where?

V.

-- 
Please note that I do not read the e-mail address used in the from field but
I read vincenzo_ml at yahoo dot it
Attenzione: non leggo l'indirizzo di posta usato nel campo from, ma leggo
vincenzo_ml at yahoo dot it

> > there are unofficial packages for fedora
> 
> Where?

I know of this one by Richard Dawe:

http://homepages.nildram.co.uk/~phekda/richdawe/fedora/FC3/