fuse-devel

[fuse-devel] multiple fuse filesystems served by the same process

[Boris P. <bpr...@ho...>]

Hi, guys, 

I am considering serving multiple fuse filesystem instances from the same process. 
The reason for this is to allow for multiple filesystem instances served by a single 
storage backend, and the convenience/performance of interfacing to that backend 
directly, rather than over IPC. 

If I create a thread per filesystem instance and call fuse_main() with arguments 
specifying the mount point for that instance, I will get multiple mount points served 
from the same process. I have reviewed libfuse code, and it seems that architecturally, 
there should not be an issue with this approach. I played with fusexmp.c example to 
quickly prototype this approach, and things come up fine, each mount point seems to 
work fine by itself, but multi-threaded tests running against multiple mount points 
concurrently break with various errors, including "bad file descriptor". 

I wonder if anyone had experience with this type of aproach. Alternatively, can 
anyone comment on the feasibility/potential issues with it ? Presumably, it should 
not make a difference for fuse.ko whether connections lead to the same or different 
processes, but it might be the case that I am overlooking some fuse.ko-level issues. 
Finally, can anyone point to potential source of teh "bad file descriptor" errors ? 

Best regards, 
Boris. 


 		 	   		  

[fuse-devel] multiple fuse filesystems served by the same process

[Michael R. <fb0...@ra...>]

>If I create a thread per filesystem instance and call fuse_main() with arguments 
>specifying the mount point for that instance, I will get multiple mount points served 
>from the same process. I have reviewed libfuse code, and it seems that architecturally, 
>there should not be an issue with this approach. I played with fusexmp.c example to 
>quickly prototype this approach, and things come up fine, each mount point seems to 
>work fine by itself, but multi-threaded tests running against multiple mount points 
>concurrently break with various errors, including "bad file descriptor". 

My experience shows that the setup stage of fuse_main does presume that serving one 
filesystem is the only goal of the process. Why not just launch event loops? You can
copy all the useful things of fuse_main in a small function, actually.

Re: [fuse-devel] multiple fuse filesystems served by the same process

[Boris P. <bpr...@ho...>]

So, I think that libfuse should be fine with starting multiple threads calling fuse_main(). 

I am starting to look into the fuse.ko code to see if it might be assumping things about 
per-process structures, etc. 

Clearly, fuse.ko is OK with multiple backends running out of separate process address 
spaces. Would anyone know (perhaps Miklos :)) if fuse.ko is OK with multiple backends 
running out of a single process address space ? 

Boris.

> Date: Thu, 3 Feb 2011 12:35:38 +0300
> From: fb0...@ra...
> To: bpr...@ho...
> CC: fus...@li...
> Subject: [fuse-devel] multiple fuse filesystems served by the same process
> 
> >If I create a thread per filesystem instance and call fuse_main() with arguments 
> >specifying the mount point for that instance, I will get multiple mount points served 
> >from the same process. I have reviewed libfuse code, and it seems that architecturally, 
> >there should not be an issue with this approach. I played with fusexmp.c example to 
> >quickly prototype this approach, and things come up fine, each mount point seems to 
> >work fine by itself, but multi-threaded tests running against multiple mount points 
> >concurrently break with various errors, including "bad file descriptor". 
> 
> My experience shows that the setup stage of fuse_main does presume that serving one 
> filesystem is the only goal of the process. Why not just launch event loops? You can
> copy all the useful things of fuse_main in a small function, actually.
> 
> 
> 
 		 	   		  

Re: [fuse-devel] multiple fuse filesystems served by the same process

[Stef B. <st...@gm...>]

Hi,

I've managed to write a fs which served multiple backends.

I followed a different approach, completly in the fuse fs.

My idea was to make my fs read a configuration when ever a directory is created.
Now when a directory is configured to contain an audio cdrom, the fs
takes the right callbackend
functions for that backend, an audio cdrom.
So I've written

read_audiocdrom
open_audiocdrom
release_audiocdrom

and

getattr_audiocdrom
opendir_audiocdrom
etc.

Whenever my fs enters the just created directory, is uses these callbacks.

For example:

<mountpoint>/Hardware/1-AudioCD

Of course the different options/parameters are set, like device
(/dev/sr0) are set, and the device is containing an audio cdrom.

For nomal io, my fs using normal functions. In general, my fs is an
overlay fs, and using normal native pread, pwrite, stat etc calls.

I have left this idea, cause it was getting complicated and not
necessary, cause I have already an construction to enter different
backend via the automounter.

I hope this helps,

Stef

[fuse-devel] FW: multiple fuse filesystems served by the same process

[Boris P. <bpr...@ho...>]

Hi, guys, 

a quick update for those who might be interested in this topic.

Serving multiple mount points out of single address space by invoking fuse_main() 
(-f argument) in separate threads actually seems to work fine. My original experiments 
must have been compromized by some bugs in my backend. I don't see any reason why 
a similar setup would not work with low-level API, but I did not verify that. 

Couple points on thread safety of the libfuse code: the fuse_signal.c and the 
module-related code might need some (fairly minimal) work; otherwise, things look fine. 

I have hacked the fusexmp.c example and was able to serve about 100 concurrent mounts 
and was able to run iozone with data integrity verification against those 100 mount points 
concurrently :) Seems like fairly rasonable base to start further multi-threaded development. 

Best regards, 
Boris. 

From: bpr...@ho...
To: fus...@li...
Subject: multiple fuse filesystems served by the same process
Date: Wed, 2 Feb 2011 21:49:32 -0500








Hi, guys, 

I am considering serving multiple fuse filesystem instances from the same process. 
The reason for this is to allow for multiple filesystem instances served by a single 
storage backend, and the convenience/performance of interfacing to that backend 
directly, rather than over IPC. 

If I create a thread per filesystem instance and call fuse_main() with arguments 
specifying the mount point for that instance, I will get multiple mount points served 
from the same process. I have reviewed libfuse code, and it seems that architecturally, 
there should not be an issue with this approach. I played with fusexmp.c example to 
quickly prototype this approach, and things come up fine, each mount point seems to 
work fine by itself, but multi-threaded tests running against multiple mount points 
concurrently break with various errors, including "bad file descriptor". 

I wonder if anyone had experience with this type of aproach. Alternatively, can 
anyone comment on the feasibility/potential issues with it ? Presumably, it should 
not make a difference for fuse.ko whether connections lead to the same or different 
processes, but it might be the case that I am overlooking some fuse.ko-level issues. 
Finally, can anyone point to potential source of teh "bad file descriptor" errors ? 

Best regards, 
Boris. 


 		 	   		  

Re: [fuse-devel] FW: multiple fuse filesystems served by the same process

[Stef B. <st...@gm...>]

Well that's interesting!

I'm interested indeed.

A sort time ago I've been able to create another mainloop which is
build around one epoll, listing to various channels (=fd's),
distributing the incoming signals to the worker threads. That may be
something interesting... if you have more mountpoints and more
channels to listen to and several threads to do the work this is
handy.

Please let me know where I can find your code, if you would like to publish.

Stef

2011/2/18 Boris Protopopov <bpr...@ho...>:
>
> Hi, guys,
>
> a quick update for those who might be interested in this topic.
>
> Serving multiple mount points out of single address space by invoking fuse_main()
> (-f argument) in separate threads actually seems to work fine. My original experiments
> must have been compromized by some bugs in my backend. I don't see any reason why
> a similar setup would not work with low-level API, but I did not verify that.
>
> Couple points on thread safety of the libfuse code: the fuse_signal.c and the
> module-related code might need some (fairly minimal) work; otherwise, things look fine.
>
> I have hacked the fusexmp.c example and was able to serve about 100 concurrent mounts
> and was able to run iozone with data integrity verification against those 100 mount points
> concurrently :) Seems like fairly rasonable base to start further multi-threaded development.
>
> Best regards,
> Boris.
>
> From: bpr...@ho...
> To: fus...@li...
> Subject: multiple fuse filesystems served by the same process
> Date: Wed, 2 Feb 2011 21:49:32 -0500
>
>
>
>
>
>
>
>
> Hi, guys,
>
> I am considering serving multiple fuse filesystem instances from the same process.
> The reason for this is to allow for multiple filesystem instances served by a single
> storage backend, and the convenience/performance of interfacing to that backend
> directly, rather than over IPC.
>
> If I create a thread per filesystem instance and call fuse_main() with arguments
> specifying the mount point for that instance, I will get multiple mount points served
> from the same process. I have reviewed libfuse code, and it seems that architecturally,
> there should not be an issue with this approach. I played with fusexmp.c example to
> quickly prototype this approach, and things come up fine, each mount point seems to
> work fine by itself, but multi-threaded tests running against multiple mount points
> concurrently break with various errors, including "bad file descriptor".
>
> I wonder if anyone had experience with this type of aproach. Alternatively, can
> anyone comment on the feasibility/potential issues with it ? Presumably, it should
> not make a difference for fuse.ko whether connections lead to the same or different
> processes, but it might be the case that I am overlooking some fuse.ko-level issues.
> Finally, can anyone point to potential source of teh "bad file descriptor" errors ?
>
> Best regards,
> Boris.
>
>
>
> ------------------------------------------------------------------------------
> The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> Pinpoint memory and threading errors before they happen.
> Find and fix more than 250 security defects in the development cycle.
> Locate bottlenecks in serial and parallel code that limit performance.
> http://p.sf.net/sfu/intel-dev2devfeb
> _______________________________________________
> fuse-devel mailing list
> fus...@li...
> https://lists.sourceforge.net/lists/listinfo/fuse-devel
>

[fuse-devel] FW: multiple fuse filesystems served by the same process

[Boris P. <bpr...@ho...>]

Hi, Stef, 

sure, I don't mind sharing the modified fusexmp.c, with the usual disclamers plus
"not a production quality but an example for illustrative purposes only :)  work 
in progress - not has been polished/cleaned up". 

At present, the hacked example makes the following assumptions: 

- the number of mount points to create is given with -n N argument
- all fuse arguments are OK except for the mountpoint: the mount points 
in the form "/tmp/xmp%i" are hardcoded, dirs are created if not there. 

So, I run it as follows: 
# ./fusexmp -n 100 -f

this runs in the foreground, and 100 mount points are mounted over 
/tmp/xmp0, /tmp/xmp1, ...., /tmp/xmp99 dirs. 

I unmount as follows: 
# for i in `seq 0 99`; do fusermount -u /tmp/xmp$i; done

and then terminate the main thread
# killall fusexmp_mt

If you see any problems/issues with this approach, please let me know.

The code is attached below, 
best regards,
Boris. 

P.S. If there is more interest, I can cleanup/submit as another example. 

/*
  FUSE: Filesystem in Userspace
  Copyright (C) 2001-2007  Miklos Szeredi <mi...@sz...>

  This program can be distributed under the terms of the GNU GPL.
  See the file COPYING.

  gcc -Wall -D_FILE_OFFSET_BITS=64 `pkg-config fuse --cflags --libs` -pthread fusexmp.c -o fusexmp_mt
*/

#define FUSE_USE_VERSION 26

#ifdef HAVE_CONFIG_H
#include <config.h>
#endif

#ifdef linux
/* For pread()/pwrite() */
#define _XOPEN_SOURCE 500
#endif

#include <fuse.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <dirent.h>
#include <errno.h>
#include <sys/time.h>
#include <sys/stat.h>
#include <sys/types.h>
#ifdef HAVE_SETXATTR
#include <sys/xattr.h>
#endif

static int xmp_getattr(const char *path, struct stat *stbuf)
{
    int res;

    res = lstat(path, stbuf);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_access(const char *path, int mask)
{
    int res;

    res = access(path, mask);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_readlink(const char *path, char *buf, size_t size)
{
    int res;

    res = readlink(path, buf, size - 1);
    if (res == -1)
        return -errno;

    buf[res] = '\0';
    return 0;
}


static int xmp_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
                       off_t offset, struct fuse_file_info *fi)
{
    DIR *dp;
    struct dirent *de;

    (void) offset;
    (void) fi;

    dp = opendir(path);
    if (dp == NULL)
        return -errno;

    while ((de = readdir(dp)) != NULL) {
        struct stat st;
        memset(&st, 0, sizeof(st));
        st.st_ino = de->d_ino;
        st.st_mode = de->d_type << 12;
        if (filler(buf, de->d_name, &st, 0))
            break;
    }

    closedir(dp);
    return 0;
}

static int xmp_mknod(const char *path, mode_t mode, dev_t rdev)
{
    int res;

    /* On Linux this could just be 'mknod(path, mode, rdev)' but this
       is more portable */
    if (S_ISREG(mode)) {
        res = open(path, O_CREAT | O_EXCL | O_WRONLY, mode);
        if (res >= 0)
            res = close(res);
    } else if (S_ISFIFO(mode))
        res = mkfifo(path, mode);
    else
        res = mknod(path, mode, rdev);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_mkdir(const char *path, mode_t mode)
{
    int res;

    res = mkdir(path, mode);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_unlink(const char *path)
{
    int res;

    res = unlink(path);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_rmdir(const char *path)
{
    int res;

    res = rmdir(path);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_symlink(const char *from, const char *to)
{
    int res;

    res = symlink(from, to);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_rename(const char *from, const char *to)
{
    int res;

    res = rename(from, to);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_link(const char *from, const char *to)
{
    int res;

    res = link(from, to);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_chmod(const char *path, mode_t mode)
{
    int res;

    res = chmod(path, mode);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_chown(const char *path, uid_t uid, gid_t gid)
{
    int res;

    res = lchown(path, uid, gid);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_truncate(const char *path, off_t size)
{
    int res;

    res = truncate(path, size);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_utimens(const char *path, const struct timespec ts[2])
{
    int res;
    struct timeval tv[2];

    tv[0].tv_sec = ts[0].tv_sec;
    tv[0].tv_usec = ts[0].tv_nsec / 1000;
    tv[1].tv_sec = ts[1].tv_sec;
    tv[1].tv_usec = ts[1].tv_nsec / 1000;

    res = utimes(path, tv);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_open(const char *path, struct fuse_file_info *fi)
{
    int res;

    res = open(path, fi->flags);
    if (res == -1)
        return -errno;

    close(res);
    return 0;
}

static int xmp_read(const char *path, char *buf, size_t size, off_t offset,
                    struct fuse_file_info *fi)
{
    int fd;
    int res;

    (void) fi;
    fd = open(path, O_RDONLY);
    if (fd == -1)
        return -errno;

    res = pread(fd, buf, size, offset);
    if (res == -1)
        res = -errno;

    close(fd);
    return res;
}

static int xmp_write(const char *path, const char *buf, size_t size,
                     off_t offset, struct fuse_file_info *fi)
{
    int fd;
    int res;

    (void) fi;
    fd = open(path, O_WRONLY);
    if (fd == -1)
        return -errno;

    res = pwrite(fd, buf, size, offset);
    if (res == -1)
        res = -errno;

    close(fd);
    return res;
}

static int xmp_statfs(const char *path, struct statvfs *stbuf)
{
    int res;

    res = statvfs(path, stbuf);
    if (res == -1)
        return -errno;

    return 0;
}

static int xmp_release(const char *path, struct fuse_file_info *fi)
{
    /* Just a stub.  This method is optional and can safely be left
       unimplemented */

    (void) path;
    (void) fi;
    return 0;
}

static int xmp_fsync(const char *path, int isdatasync,
                     struct fuse_file_info *fi)
{
    /* Just a stub.  This method is optional and can safely be left
       unimplemented */

    (void) path;
    (void) isdatasync;
    (void) fi;
    return 0;
}

#ifdef HAVE_SETXATTR
/* xattr operations are optional and can safely be left unimplemented */
static int xmp_setxattr(const char *path, const char *name, const char *value,
                        size_t size, int flags)
{
    int res = lsetxattr(path, name, value, size, flags);
    if (res == -1)
        return -errno;
    return 0;
}

static int xmp_getxattr(const char *path, const char *name, char *value,
                        size_t size)
{
    int res = lgetxattr(path, name, value, size);
    if (res == -1)
        return -errno;
    return res;
}

static int xmp_listxattr(const char *path, char *list, size_t size)
{
    int res = llistxattr(path, list, size);
    if (res == -1)
        return -errno;
    return res;
}

static int xmp_removexattr(const char *path, const char *name)
{
    int res = lremovexattr(path, name);
    if (res == -1)
        return -errno;
    return 0;
}
#endif /* HAVE_SETXATTR */

static struct fuse_operations xmp_oper = {
    .getattr    = xmp_getattr,
    .access     = xmp_access,
    .readlink   = xmp_readlink,
    .readdir    = xmp_readdir,
    .mknod      = xmp_mknod,
    .mkdir      = xmp_mkdir,
    .symlink    = xmp_symlink,
    .unlink     = xmp_unlink,
    .rmdir      = xmp_rmdir,
    .rename     = xmp_rename,
    .link       = xmp_link,
    .chmod      = xmp_chmod,
    .chown      = xmp_chown,
    .truncate   = xmp_truncate,
    .utimens    = xmp_utimens,
    .open       = xmp_open,
    .read       = xmp_read,
    .write      = xmp_write,
    .statfs     = xmp_statfs,
    .release    = xmp_release,
    .fsync      = xmp_fsync,
#ifdef HAVE_SETXATTR
    .setxattr   = xmp_setxattr,
    .getxattr   = xmp_getxattr,
    .listxattr  = xmp_listxattr,
    .removexattr    = xmp_removexattr,
#endif
};

// quick test for multi-mount point fuse setup
#include <stdlib.h>
#include <pthread.h>
#include <signal.h>

typedef struct {
    char mntpt[256];
    int argc;
    char **argv;
} xmp_arg;
  
static void *xmp_thread_function(void *data) 
{
    xmp_arg *arg = (xmp_arg *)data;
    int thread_argc = arg->argc+2;
    char **thread_argv = NULL; 
    int i = 0, rc = 0;

    if ((thread_argv = (char **)malloc((thread_argc*sizeof(char*)))) == NULL) {
        printf("Could not malloc %lu bytes, exiting", thread_argc*(sizeof(char*)));
        return NULL;
    } 

    // setup the arguments for the thread
    for (i = 0; i < arg->argc; i++) {
        thread_argv[i] = arg->argv[i];
    } 
    // additional arguments
    thread_argv[i++] = arg->mntpt;
    thread_argv[i] = NULL;

    rc = fuse_main(thread_argc-1, thread_argv, &xmp_oper, NULL);
  
    if (rc) 
        fprintf(stderr, "fuse_main() returned %d\n", rc);
    else 
        printf("fuse_main() exited\n");

    free(thread_argv);

    return NULL; 
}

static int shutdown = 0; 

static void xmp_handler(int sig) 
{
    printf("got signal %d\n", sig);
    shutdown = 1;
} 

static int set_sig_handler(int sig, void (*handler)(int))
{
    struct sigaction sa;
    memset(&sa, 0, sizeof(sa));
    sa.sa_handler = handler;
    sigaddset(&sa.sa_mask, sig);
    return sigaction(sig, &sa, NULL);
} 

int main(int argc, char *argv[])
{
    int i = 0;
    int num_mount_points = 0;
    pthread_t *xmp_thread;
    xmp_arg *xmp_thread_arg; 
    char **newargv = NULL;
    int newargc = 0;
    umask(0);
    
    // get the number of mount points to create
    // we will need to take the -n N arguments out before passing 
    // args to fuse_main()
    if ((newargv = malloc(sizeof(char*)*(argc+1))) == NULL) {
        perror("malloc() failed");
        exit(-1);
    }

    for (i = 0; i< argc; i++) {
        if ((strcmp(argv[i], "-n") == 0)) {
            num_mount_points = atoi(argv[++i]);
        } else {
            newargv[newargc++] = argv[i];
        } 
    }
    newargv[newargc] = NULL;
    
    // install signal handlers to override fuse signal handling
    if (((set_sig_handler(SIGTERM, xmp_handler)) == -1) ||
        ((set_sig_handler(SIGINT, xmp_handler)) == -1) ||
        ((set_sig_handler(SIGHUP, xmp_handler)) == -1) ||
        ((set_sig_handler(SIGPIPE, SIG_IGN)) == -1)) {
        perror("could not set signal handler");
        exit(-1);
    } 
  
    // create a number of threads that mount different mount points
    if (((xmp_thread = malloc(num_mount_points*sizeof(pthread_t))) == NULL) ||
        ((xmp_thread_arg = malloc(num_mount_points*sizeof(xmp_arg)))) == NULL)
    {
        perror("malloc() failed");
        exit(-1);
    }
    memset(xmp_thread, 0, num_mount_points*sizeof(pthread_t));
    memset(xmp_thread_arg, 0, num_mount_points*sizeof(xmp_arg));
    for (i = 0; i < num_mount_points; i++) {
        int rc = 0;
        // all will be mounted at /tmp/xmp%i - for i-th mount point
        sprintf(xmp_thread_arg[i].mntpt, "/tmp/xmp%d", i);
        // make sure mount point exists
        if ((rc = mkdir(xmp_thread_arg[i].mntpt, 0755)) && 
            (errno != EEXIST)) {
            perror("mkdir() failed");
            exit(-1);
        }
        xmp_thread_arg[i].argc = newargc;
        xmp_thread_arg[i].argv = newargv;
        if ((pthread_create(&xmp_thread[i], NULL, xmp_thread_function, 
                            (void *)&xmp_thread_arg[i]))) {
            perror("pthread_create() failed");
            exit(-1);
        }
    }
  
    // wait for shutdown
    // each mount pont can be unmounted separately, once all are unmounted, 
    // the user is expected to send "killall basename(argv[0])" to exit 
    while(shutdown == 0) {
        sleep(1); 
    } 

    // join the threads
    for (i = 0; i < num_mount_points; i++) {
        pthread_join(xmp_thread[i], NULL);
    } 

    return 0;
}




> Date: Fri, 18 Feb 2011 23:24:50 +0100
> Subject: Re: [fuse-devel] FW: multiple fuse filesystems served by the same process
> From: st...@gm...
> To: bpr...@ho...
> CC: fus...@li...
> 
> Well that's interesting!
> 
> I'm interested indeed.
> 
> A sort time ago I've been able to create another mainloop which is
> build around one epoll, listing to various channels (=fd's),
> distributing the incoming signals to the worker threads. That may be
> something interesting... if you have more mountpoints and more
> channels to listen to and several threads to do the work this is
> handy.
> 
> Please let me know where I can find your code, if you would like to publish.
> 
> Stef
> 
> 2011/2/18 Boris Protopopov <bpr...@ho...>:
> >
> > Hi, guys,
> >
> > a quick update for those who might be interested in this topic.
> >
> > Serving multiple mount points out of single address space by invoking fuse_main()
> > (-f argument) in separate threads actually seems to work fine. My original experiments
> > must have been compromized by some bugs in my backend. I don't see any reason why
> > a similar setup would not work with low-level API, but I did not verify that.
> >
> > Couple points on thread safety of the libfuse code: the fuse_signal.c and the
> > module-related code might need some (fairly minimal) work; otherwise, things look fine.
> >
> > I have hacked the fusexmp.c example and was able to serve about 100 concurrent mounts
> > and was able to run iozone with data integrity verification against those 100 mount points
> > concurrently :) Seems like fairly rasonable base to start further multi-threaded development.
> >
> > Best regards,
> > Boris.
> >
> > From: bpr...@ho...
> > To: fus...@li...
> > Subject: multiple fuse filesystems served by the same process
> > Date: Wed, 2 Feb 2011 21:49:32 -0500
> >
> >
> >
> >
> >
> >
> >
> >
> > Hi, guys,
> >
> > I am considering serving multiple fuse filesystem instances from the same process.
> > The reason for this is to allow for multiple filesystem instances served by a single
> > storage backend, and the convenience/performance of interfacing to that backend
> > directly, rather than over IPC.
> >
> > If I create a thread per filesystem instance and call fuse_main() with arguments
> > specifying the mount point for that instance, I will get multiple mount points served
> > from the same process. I have reviewed libfuse code, and it seems that architecturally,
> > there should not be an issue with this approach. I played with fusexmp.c example to
> > quickly prototype this approach, and things come up fine, each mount point seems to
> > work fine by itself, but multi-threaded tests running against multiple mount points
> > concurrently break with various errors, including "bad file descriptor".
> >
> > I wonder if anyone had experience with this type of aproach. Alternatively, can
> > anyone comment on the feasibility/potential issues with it ? Presumably, it should
> > not make a difference for fuse.ko whether connections lead to the same or different
> > processes, but it might be the case that I am overlooking some fuse.ko-level issues.
> > Finally, can anyone point to potential source of teh "bad file descriptor" errors ?
> >
> > Best regards,
> > Boris.
> >
> >
> >
> > ------------------------------------------------------------------------------
> > The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
> > Pinpoint memory and threading errors before they happen.
> > Find and fix more than 250 security defects in the development cycle.
> > Locate bottlenecks in serial and parallel code that limit performance.
> > http://p.sf.net/sfu/intel-dev2devfeb
> > _______________________________________________
> > fuse-devel mailing list
> > fus...@li...
> > https://lists.sourceforge.net/lists/listinfo/fuse-devel
> >
 		 	   		  

Re: [fuse-devel] FW: multiple fuse filesystems served by the same process

[Stef B. <st...@gm...>]

Thanks a lot for sharing. It looks fine!

I'm busy right now, but when I've been testing it with my epoll-thread
event loop, I will send you the result.

With the epoll thread eventloop, there is one central eventloop, even
with your multiple mountpoints
fs.
The epoll will detect activity on the fd's (one fd per mountpoint) and
will start a worker thread.
In my single mounpoint fs, I use 9 worker threads. This number does
not change runtime.

With your fs it's possible to use say 40 threads while there are 100
mountpoints.... It's very unlikely there is activity on all
mountpoints at the same time. The number 40 is just a guess, but
you'll get my point.

There is only one thing important I guess. The mainloop (epoll) has to
link the incoming request (coming from one of the fd's) to the right
fuse_process_session aka session.

Stef

Re: [fuse-devel] FW: multiple fuse filesystems served by the same process

[Boris P. <bpr...@ho...>]

Great, thanks, Stef!
Boris. 

> Date: Wed, 23 Feb 2011 16:10:38 +0100
> Subject: Re: [fuse-devel] FW: multiple fuse filesystems served by the same process
> From: st...@gm...
> To: bpr...@ho...
> CC: fus...@li...
> 
> Thanks a lot for sharing. It looks fine!
> 
> I'm busy right now, but when I've been testing it with my epoll-thread
> event loop, I will send you the result.
> 
> With the epoll thread eventloop, there is one central eventloop, even
> with your multiple mountpoints
> fs.
> The epoll will detect activity on the fd's (one fd per mountpoint) and
> will start a worker thread.
> In my single mounpoint fs, I use 9 worker threads. This number does
> not change runtime.
> 
> With your fs it's possible to use say 40 threads while there are 100
> mountpoints.... It's very unlikely there is activity on all
> mountpoints at the same time. The number 40 is just a guess, but
> you'll get my point.
> 
> There is only one thing important I guess. The mainloop (epoll) has to
> link the incoming request (coming from one of the fd's) to the right
> fuse_process_session aka session.
> 
> Stef
 		 	   		  

Re: [fuse-devel] FW: multiple fuse filesystems served by the same process

[Stef B. <st...@gm...>]

Maybe you've expected a result earlier. I'm looking at your example right now.

The sollution I'm thinking of uses the lowlevel version of fuse. With
this much more is possible, even required to do the thing I'm thinking
of.

The main of my ll fs looks like:

   if (fuse_parse_cmdline(&xmpfs_args, &xmpfs_mountpoint, NULL,
&foreground) != -1 ) {

        if ( (tmp_chan = fuse_mount(xmpfs_mountpoint, &xmpfs_args)) != NULL) {

            xmpfs_session=fuse_lowlevel_new(&xmpfs_args, &xmpfs_oper,
sizeof(xmpfs_oper), NULL);

            if ( xmpfs_session != NULL ) {

                res = fuse_daemonize(foreground);

                if (res==0) {


                    fuse_session_add_chan(xmpfs_session, xmpfs_chan);

                    syslog(LOG_DEBUG, "Channel created, starting
fuse_session_loop_epoll_mt.\n");

                    res=fuse_session_loop_epoll_mt(xmpfs_session);

                    fuse_session_remove_chan(xmpfs_chan);

                }

                fuse_session_destroy(xmpfs_session);

            } else {

                syslog(LOG_DEBUG, "Error starting a new session.\n");

            }

            fuse_unmount(xmpfs_mountpoint, xmpfs_chan);

        } else {

            syslog(LOG_DEBUG, "Error mounting and setting up a channel.\n");

        }

    } else {

        syslog(LOG_DEBUG, "Error parsing options.\n");

    }

    fuse_opt_free_args(&xmpfs_args);


Now as you look like it, the setting of the handlers are not there.
This is done in fuse_session_loop_epoll_mt. The epoll thing there is
able to listen to the signals via signalfd, so that's done there. That
is usefull when launching an external command, the signal SIG_CHLD
(send to the parent when a child is terminated) is caught there and
handled in the right way (by calling a wait function).

So a lowlevel fs is required. Well that's not hard, to make a lowlevel
version of the example fs. I've got this already.

In plaintext it should work as follows:

read the number of desired mountpoints from the commandline
(do not use the function fuse_parse_cmdline like above of course)

create an array (or linked list) of session, for every mountpoint a session.
and init every element with fuse_mount() (giving back a channel[i])
and fuse_lowlevel_new (giving back session[i]).
(



































the function fuse_mount does


Daemonize. I shouldnt know why this should stay in foreground.

Add the channel[i] to session[i].

give this list/array of sessions AND extra the number of sessions as
parameter to
fuse_session_loop_epoll_mt.

This will add the channels (=fd) to the epoll instance. The clever
thing with epoll is that it's possible to link additional data to a
epoll_event. This epoll_event is available when there is data on the
associated fd:

epoll_event leads to the right session, which leads again to the right channel.

So it's possible to feed the data to a worker thread, which calls
fuse_session_process(session, buffer with request, size of request,
channel).

You see here that the mountpoint is not mentioned here.

So there is one main eventloop handling all the mountpoints here...
listning to all the fd's, one for every mountpoint.

Hope you understand the working of this.
I would like to add that this is pure experimental, I mean even the
whole using more than one mountpoint. Can you think of some use??

Stef Bon

Notes: my fuse_loop_epoll_mt.c is attached. It's attached here as
seperate file, but I'm using this directly in my fs. If anyone is
interested, and thinks it's usefull to add to the fuse source, please
let me know. I will make a patch for it or so.

One thing: the multithread loop already in the source
fuse_session_loop_mt is using a lock around the calling of
fuse_session_process: &mt->lock. It looks like this is preventing
*real* multithreading behaviour. I do not know why this is done this
way. Maybe for not thread safe fs's.

This event loop does not have that. So no blocking there. One thing is
thus required: the fs using this loop has to be thread safe!!

In my fs (a layer around the normal system, providing a neat GoboLinux
like environment) I sometimes see "bursts" of 3/4/5 threads busy, not
more. One note: only the things a user/initiated by the user does are
handled by the fuse fs, others (like the startup of the KDE which
causes a lot of IO of course) is going via the fuse fs. For example
browing the /Computer map, opening a file on a USB stick, going to the
/Network map etc...

Re: [fuse-devel] FW: multiple fuse filesystems served by the same process

[Stef B. <st...@gm...>]

Sorry for the white space in the middle. I was playing with the
battery of a new keyboard)

What I wanted to add, that the function fuse_mount does something with
the fd 0, 1, 2. Look at the
code in lib/helper.c, line 204. This is not required when using more
then one mountpoint here.

Stef

Re: [fuse-devel] FW: multiple fuse filesystems served by the same process

[Boris P. <bpr...@ho...>]

Hi, Stef, 
 
I am looking into low-level APis as well, but I did not have a chance to allocate enough time for that just yet, 
 
Boris. 
 
> Date: Thu, 10 Mar 2011 14:32:22 +0100
> Subject: Re: [fuse-devel] FW: multiple fuse filesystems served by the same process
> From: st...@gm...
> To: bpr...@ho...
> CC: fus...@li...
> 
> Sorry for the white space in the middle. I was playing with the
> battery of a new keyboard)
> 
> What I wanted to add, that the function fuse_mount does something with
> the fd 0, 1, 2. Look at the
> code in lib/helper.c, line 204. This is not required when using more
> then one mountpoint here.
> 
> Stef