Filesystem Suggestions

From fuse

Contents 1 Filesystem Suggestions 1.1 Apt-FS 1.2 OCMFS 1.3 MEDIAFS 1.4 REMOTEFS 1.5 TXTFS 1.6 GAIMFS 1.7 IMAPFS 1.8 Filesystems list 1.9 Union or merge filesystem 1.10 Switch filesystem 1.11 MIME type filesystem 1.12 Filter filesystem 1.13 tmpfs 1.14 Confirmation filesystem 1.15 Execution FS 1.16 Patch FS 1.17 Copy-on-write filesystem 1.18 XML filesystem and DOM filesystem 1.19 Split filesystem 1.20 Rename filesystem 1.21 Delayed write filesystem 1.22 Ignore setattr filesystem 1.23 Reverse filesystem 1.24 Sort filesystem 1.25 XWindows windows FS 1.26 HTML rename filesystem 1.27 VMWare filesystem 1.28 make filesystem

Filesystem Suggestions

Apt-FS

Or whatever other package manager you like, a filesystem via which you can browse your package repositories, with all metadata as directories so that you can go to for example /mnt/aptfs/all/abiword/reccommends/abiword-plugins/reccommends/gimp/install, which could simply contain "#!/bin/bash\nsudo apt-get install gimp" and be owner:mod root:755.

OCMFS

The Overlay Metadata Caching File System.

One of the biggest turn-offs of the Overlay filesystems is performance. This filesystem, to be mounted over top of an Overlay or any filesystem with an inefficient metadata structure, or on media with high seek times, would act like COWFS, but also maintain b-tree metadata caches for the underlying filesystem(s). This would also help with consistency in some cases, and allow certain paths from the bottom to rise to the top. If an underlying filesystem is unmounted the metadata can be removed from the cache on access (quick test whether the mount exists), and the next underlying filesystem may be cached instead. This could even allow overlays where the intermediate layers are not in fact mounted at all but merely imported into the metadata cache, since the media can be accessed for block data without being mounted.

This would improve performance spectacularly for optical media, and also improve Union/Overlay performance to the point where it could be acceptable for a distro to consider issuing a static base system image and service packs as (optionally compressed) filesystem images.

Caveats: Write support could be tricky if the underlying layers (more than one) have write support, that would require FAM or some such. Most of this functionality is already available by mixing other FUSE layers.

MEDIAFS

File system that demuxes and exports frames for multimedia streams, for example animated .gif, .avi and .ogg files, etc. Somewhat like tar:/ and audiocd:/ under KDE. So for example mounting demo.avi on /mnt reveals /audio and /video, files describing the container and format, resolution, channels, chapters/indexes, and the title. /audio contains the tracks converted to trackN.wav, trackN.raw (unconverted stream), and files describing the format of the raw streams. /video contains trackN.YUV (stream of uncompressed frames), trackN.raw, files describing the raw format, and /frames. /video/frames contains timestamped .YUV and .PNM files, maybe even .JPG could be thrown in since the source is already FFT encoded it would be a pretty lightweight operation.

Write support would turn standard shell tools into a non-linear multimedia editing suite.  ;-) Obviously write support would be easier to implement on an HUFFYUV/WAV formatted .avi file, which is pretty standard for video editing anyhow.

I admit that all of this can be achieved by using mplayer and co. to extract the media file into a directory, work with the decoded files, and then re-encode the file, but I think this functionality is needed and reusable by a number of programs, from Konqueror (thumbnailing, previews) to biff, and pretty much any media player/editor.

It could easily replace a large number of the programs in use today. It would become an API for multimedia work, hopefully help join the forces working on multimedia APIs in general, make codec inclusion transparent to the apps, etc. Also while this sounds naive compared to mboxfs and other brilliant user-space re-interpretations of metadata and namespace views, media streams are generally simple linear storage containers, much larger in size than typical email, and therefore much more relatively efficient to look at through a mount.

SerhijStasyuk says: IMHO better to implement this via Hybrid objects.

REMOTEFS

File system that mounts a website (in a particular format) as a file system. An example: mount eric.code.web.utk.edu as /mnt/ericcode requires the hosting website to keep some meta-data in specific places, and some extra information for security's sake. mounts as read-only.

Large first step toward remote computing as described in http://www.onlamp.com/pub/a/python/2007/01/18/why-i-stopped-coding.html Obvious security issues to consider.

TXTFS

File system that holds all meta data in a single txt file (or possibly a specified set of text files) and all files in a single directory. Names would be remapped and stored in the text file (so that same name errors won't occur), and the meta data files ought to be formatted to be human-readable. This fs is intended largely as a first step to other file systems, ala gmailfs.

GAIMFS

Essentially a file system that is similar to samba, but operates via gaim messages and file send/receive. The config file would be extensive, to indicate who else is using the file system. A complicated handshake would be necessary, to match versions and grant mutual permission, but remain 'polite' - non-confusing to accidental targets that should not be receiving the messages. This gets somewhat more complicated when we allow a user to be signed on more than once (as gaim does) - some type of union fs and/or target resolution would be required.

Ideally, all the meta data would be in the messages, but putting it in files to be transferred would keep the chat box traffic down a little. Definitely should be human readable, to the extent that putting a person on the other end of the file system instead of another client would not cause any problems (aside from very very high file system latency)

If this type of system were implemented well and simply, I could see it being fairly popular- aim's 'get file' system was/is rather liked.. Might be substantially simpler to implement this as a FUSE fs + a gaim extension.

(obviously this would have to be read-only, except in special cases.. gaim is not built to be secure)

IMAPFS

A filesystem that allows one access to IMAP servers like they are local Maildirs or mboxs or MH(mount-time-option)

OR

How about a filesystem that yields a full tree that's useful without an email client proper? I built something called mboxfs (also on FUSE) a few years ago, which turned an mbox back-end into a tree. I'd like to do the same for an IMAP back-end. Here are some nice things you can do with such a filesystem:

$ cd ~/mailbox/inbox/search/from/my@best.friend/order/recv/desc/0
$ cat body
$ cp replytemplate ~/mailbox/temp/replytomybest_friend
$ vi ~/mailbox/temp/replytomybestfriend
$ mv ~/mailbox/temp/replytomybestfriend ~/mailbox/tosend/

Using a file manager like mc, this alone is quite efficient, compared to many email clients. (I built it as a replacement for Mutt.)

Now, if you don't like file managers, there's nothing stopping you from doing this:

$ cat ~/bin/mail_reply
#!/bin/sh
cp "$1" ~/mailbox/temp/temp.reply
vi ~/mailbox/temp/temp.reply
echo "Send?" >&2; read SENDIT;
if [ "$SENDITx" = "yesx" ]; then
 mv ~/mailbox/temp/temp.reply ~/mailbox/tosend
else
 mv ~/mailbox/temp/temp.reply ~/mailbox/f/drafts
fi
$ exit

...which, in fact, is almost exactly what I did, using slightly more complicated PERL scripts :-) Doing it for mbox was annoying, since I had to coordinate locking and resyncing to the physical backend. Since IMAP already provides those primitives, it should be a lot simpler. (You also save the mbox parse, which makes life a lot more fun.)

OR

A filesystem that allows one to store/retreive/delete files and folders on an IMAP server. Since the IMAP protocol is commonplace and can be encrypted with IMAPS it would be an alternative to VPN. It would mostly likely still be possible to get access to files through the regular IMAP clients too. Although updating would not be possible users might be able to get the attachments from the emails--even over a web mail interface.

SerhijStasyuk says: I like this idea. Especially with inotify support. I have tried to create such FS, but have no time for this :( Also, it can be not just IMAP, but also SMTP/POP/NNTP (I was trying to create it over http://libetpan.sourceforge.net/, but there are some others). This FS would be espesially usable with MIME parts decoding and Hybrid objects libetpan.

See also Plan9's upas/fs or the new upas/nfs (mailfs) in plan9ports.

Filesystems list

There should be an FS which will list all known (preferably not only installed but all existing) FUSE filesystems (as directories). Well, there should be option whether to list all known filesystems or only installed.

Invent some way to auto-mount these filesystems. (Discuss in this Wiki.)

It could be

$ ls /mnt/fuse/
...
Fuseftp/
...
$ ls /mnt/fuse/Fuseftp/
mnt
info
$ cat /mnt/fuse/Fuseftp
$ ls /mnt/fuse/Fuseftp/mnt/ftp.example.org/dir
...

Instead of "info" files, we could alternately use extended file (well, directory) attributes (xattrs) to provide descriptions of filesystems.

Union or merge filesystem

There should be a filesystem which will unite (merge) any number of specified directories. One (specified by a mount option) of these several directories will be written when the merged FS is written.

For example we could merge /mnt/disk1, /mnt/disk2, /mnt/disk3 together. If a file x/y is requested, it could check first /mnt/disk1/x/y or if file doesn't exist then /mnt/disk2/x/y or if file doesn't exist then /mnt/disk3/x/y. Writes could be redirected for example to /mnt/disk2.

It is unclear what to do if it is requested removal of a file which is present not only in the directory specified as writable but also in other merged directories. Probably should return EPERM or EACCESS error? Or should be removed from /mnt/disk2 (for example) but not disappear from directory listing? Choice between these two variants by mount option? Discuss here.

Eric says: -- You missed a few other options here. You could allow the non-writable files to be 'deleted', simulating their deletion in the union without actually removing them from the non-writable systems. You could mandate that all files present in the non-writable fs' have permission read-only. You could even have all files that are different in different file systems be visible in all versions, via renaming ("readme.txt.FILESYSTEM1, readme.txt.FILESYSTEM2..."), though that would be fairly painful to mount initially.

It depends on the application which is best, but for the sake of emulating a single filesystem, I vote for my first option - delete from the writable system and note it's deletion, simulating deletion from the non-writable ones as well. This makes the result indistinguishable from that you would get with a conventional file system.

Switch filesystem

This filesystem would dynamically switch (by user request) between several similar filesystems without unmounting.

That is we could switch between several different views of one directory, e.g. between viewing shell script themselves (localfs) or viewing their output (execfs) or, for example, seeing them as syntax colorized HTML files.

It could be also nice if FUSE kernel itself could be able switch filesystems without unmounting. (Then we will not need switch filesystem as we will have it in any case.)

Eric says: This one seems a bit .. pointless. Not because it's not useful, but because it is essentially the same as having two mounted file systems and switching a softlink from one to the other. Aside from that, switching would have to take a long time, as the switch system needs to make sure NONE of the files being 'removed' are in use.. which is basically what mounting and unmounting does.

MIME type filesystem

This filesystem would add MIME type to files as an extended attribute (xattr). It could use libfile to know what is the MIME type of a file.

MIME type should be also settable (and stored somewhere) by setxattr.

Filter filesystem

This filesystem would execute arbitrary filter shell commands (given by mount option) when reading/writing files.

This would be for example, more general replacement for all compressed filesystems.

There should be also binary (C) interface for mounting this to pass as filters C functions instead of shell commands.

We should have a special *filter* C interface which could be a more special interface than a filesystem. Probably it could not only filter files but also rename them (see Rename Filesystem).

tmpfs

There are tmpfs in Linux kernel but as of Linux kernel 2.6.15.4 tmpfs is limited to half of physical memory (without swap) — It is too bad size limit. Half of physical memory is the default (can be changed with the size= option) and has been like that since at least 2.6.12 -- Dermoth.

We could create user space tmpfs as an alternative to the kernel tmpfs.

Wellons says: Not sure what the above information is talking about. The tmpfs on the Linux kernel lives in virtual memory and uses swap. If your swap is 10G, your tmpfs can be about 10G. Has been since about 2.4.18: http://lxr.linux.no/source/Documentation/filesystems/tmpfs.txt . Perhaps this would still be useful for users who want personal temporary space.

Confirmation filesystem

This filesystem would ask user confirmation (e.g. using X11, the X display can be passed as a mount option) for certain file operations.

For example it could ask confirmation when reading big files (more than a certain size) to reduce network traffic for a networked FS.

Also it can ask delete confirmation (dependently on user options it could be asked only for files, only for directories, etc.) However it is impossible to ask recursive directory deletion confirmation on our API level. We need more general API with recursive directory unlinking (using directory locks?)

We may return EIO for user-aborted operations.

Execution FS

Should (based on an existing directory in an other filesystem) for executable files return their output instead of file content. That is this FS should run programs.

What mount options we could use to specify used program arguments? Anything special about shell scripts? What to do with SUID files? What to do with SUID shell scripts? What to do with non-executable files? (Should these dependently on mount option indeed be executed using as if were executable scripts?)

It could be (among other) a lightweight fast alternative to a Web server to debug CGI scripts if we will somehow support so complex operations as imitating Web server environment.

What should be done if writing into an executable? Probably it should keep per-process state and if the same process reads from an executable where it has written it should return command output run with that input which was written to. What then do if one process continues to write to an executable after it has finished execution? Should SIGPIPE be raised? Should execute the command several times?

Patch FS

It would automatically apply a set of patches (.diff files) to a directory. A set of patches would be specified by shell globbing (e.g. /dir/patches/*.diff,/dir/patches/*.patch).

Copy-on-write filesystem

It would optimize file copy that is sendfile() operation which should used by cp shell command by using copy-on-write technique (do not copy file content while it is not modified, but copy only a reference to file object; actual copy is done only if a file is modified).

(Does sendfile() work with the current FUSE version?)

XML filesystem and DOM filesystem

Browse an XML file as a hierarchy of directories and (text) files.

E.g. tag names would map to directory names.

More generally it should be DOM filesystem (see [www.w3.org] about DOM).

Ideally it should support XML (and HTML) files, XML database protocol and DOM (through at least C/C++ and Java interfaces).

Split filesystem

It would represent a file as a directory with several files. A file should be split into sections (and possibly also subsections) by some user-specified shell command (e.g. split).

It would be great to automatically split HTML files into chapters.

Eric says: the premise of this fs is interesting, but the description doesn't work straightforwardly. The simplest solution I could come up with is to specify the split-string as a mount parameter.. Really, this is better implemented as a command line tool instead of a file system.

Rename filesystem

It would present files under different names. (We could use a shell command or just glob-pattern/replacement as file names filter.)

For example after a compressed filesystem is mounted over a directory with .gz files we would want also to remove .gz extension from file names.

Delayed write filesystem

It would delay all write operations to an other filesystem until unmounting.

Writing should also be cancelable (user to be able to cancel all changes).

Eric says: So basically a unionfs with a 'commit-changes' on the end? It's interesting, and feels close to cvsfs. There are definitely uses for this one.

Ignore setattr filesystem

This filesystem would mirror an other filesystem except that setattr would always do nothing and return success.

It is sometimes convenient for e.g. DOS/Windows filesystems to be dealt from Unix.

Reverse filesystem

Probably a stupid idea: Use file content as filename and filename as file content. :-) There are indeed some cases when this would be convenient.

Eric: O.O Be careful there some files have protected contents, but not protected names! .ssh/identity, for example...

Sort filesystem

It would sort files in a directory accordingly a user given sorting criteria (e.g. sort first by date, then by size). It could also be able to sort using a shell command for specifying a sorting criteria (or a C function pointed passed as a binary option when mounting from a C program). Eric: I notice a lot of the ideas here involve control from the command line. Not only does that seem complicated to implement, any time you do something like that, you are changing the filesystem without checking for use first (otherwise you might as well just remount it).

XWindows windows FS

Browser X11 windows as a hierarchy of directories.

HTML rename filesystem

Moving a file would update all HTML (and XML using XLink) files in the same filesystem to link to updated location.

It could use MIME type filesystem (see above) to decide which files are HTML.

Eric: Definitely useful Windows has a primitive version of this hacked into its shell, but this could be very useful. Could just as easily be done by aliasing the mv command with a sed script + move, and using an environment variable for $CURRENTHTMLDIR though :-\

VMWare filesystem

Something like Ken Kato's VDK driver for Windows - but on Linux. Replacement for vmware-mount.pl. Current implementation of vmware-mount is crap (using NBD on localhost to mount vmware virtual disk files). It`s ancient, unstable, unflexible (e.g. cannot map entire disk) and it`s closed source, too. Since the VMWare virtual disk format is open spec. now, one could write a Fuse module to access any filesystem (linux supports) inside virtual disk files. Not sure, if we could have raw access to partitions with this, too. So , maybe it`s better to write a device-mapper target for this, but i`m not sure, so i`m posting it here.

make filesystem

Probably in combination with chroot, this would be a compilation environment to get around frustrating library and include dependancies; instead of ever falling through with a "File not found", the file system would root through the "real" filesystem looking for the appropriate library or header file. It could even search the internet and install developer packages etc. As a mount option, the user could specify a different target operating system or build environment, easing cross compilation (see also http://distcc.samba.org/). Hopefully this would all be done much faster/smaller/cleaner than running an entire virtual machine just to do compilation.

Probably a horrible, horrible idea in the long run.