Re: [fuse-devel] FUSE threads blocked writing response to kernel

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi All---

Oops.  Forgot to include version numbers.  We are using

fuse verion 2.8.4-1
kernel version 2.6.30.10-105.2.23

Regards,
Steve

On Tue, Oct 26, 2010 at 3:58 PM, S F <sfr...@gm...> wrote:
> Hi All---
> We are seeing some strange behavior with our FUSE application and I
> was wondering if this behavior has been noticed by others.
>
> Our FUSE file system sits underneath samba, and under heavy write
> loads we sometimes see our application memory size explode while samba
> starts returning time-outs to the client.  Eventually our FUSE
> applicaiton runs out of memory and dumps core.  Looking into the core
> file we see that the memory is being consumed because FUSE has created
> a very large number of threads, but all of the threads are blocked
> waiting to write their results back to the kernel in
> fuse_kern_chan_send().  It appears that requests keep coming up from
> Samba through the kernel,  and new threads continue to be created by
> FUSE, but existing threads are not cleaned up because they are blocked
> waiting to write results back to the FUSE device.
>
> Currently we have only observed this at customer sites, and have not
> been able to recreate the situation under controlled conditions in our
> lab.  What we know about the customer workloads is that they are
> running a recursive backup operation targeting our product.  Basically
> there is a very high load of file/directory creates and writes.  I
> don't have much experience with the kernel side of things so I'm
> somewhat at a loss to explain why the FUSE device would end up
> blocking writes from the FUSE application.  Any ideas?  Anyone seen a
> similar issue?
>
> Here is some gdb output showing the thread situation along with a
> couple representative stack traces:
>
> Program terminated with signal 6, Aborted.
> #0  0x0088b424 in __kernel_vsyscall ()
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.10.2-1.i686 libattr-2.4.43-3.fc11.i586
> libcap-2.16-4.fc11.1.i586 libgcc-4.4.1-2.fc11.i586
> libstdc++-4.4.1-2.fc11.i586 libxml2-2.7.6-1.fc11.i586
> openssl-0.9.8n-2.fc11.i686 zlib-1.2.3-22.fc11.i586
> (gdb) info thread
>  300 Thread 21884  0x0088b424 in __kernel_vsyscall ()
>  299 Thread 21808  0x0088b424 in __kernel_vsyscall ()
>  298 Thread 21867  0x0088b424 in __kernel_vsyscall ()
>  297 Thread 21766  0x0088b424 in __kernel_vsyscall ()
>  296 Thread 21838  0x0088b424 in __kernel_vsyscall ()
>  295 Thread 21777  0x0088b424 in __kernel_vsyscall ()
>  294 Thread 21819  0x0088b424 in __kernel_vsyscall ()
>  293 Thread 21743  0x0088b424 in __kernel_vsyscall ()
>  292 Thread 21788  0x0088b424 in __kernel_vsyscall ()
>  291 Thread 21848  0x0088b424 in __kernel_vsyscall ()
>  290 Thread 21829  0x0088b424 in __kernel_vsyscall ()
>  289 Thread 21799  0x0088b424 in __kernel_vsyscall ()
>  288 Thread 21872  0x0088b424 in __kernel_vsyscall ()
>  287 Thread 21754  0x0088b424 in __kernel_vsyscall ()
>  286 Thread 21698  0x0088b424 in __kernel_vsyscall ()
>  285 Thread 21809  0x0088b424 in __kernel_vsyscall ()
>  284 Thread 21824  0x0088b424 in __kernel_vsyscall ()
>  283 Thread 21841  0x0088b424 in __kernel_vsyscall ()
>  282 Thread 21856  0x0088b424 in __kernel_vsyscall ()
>  281 Thread 21871  0x0088b424 in __kernel_vsyscall ()
>  280 Thread 21686  0x0088b424 in __kernel_vsyscall ()
>  279 Thread 21792  0x0088b424 in __kernel_vsyscall ()
>  278 Thread 21869  0x0088b424 in __kernel_vsyscall ()
>  277 Thread 21855  0x0088b424 in __kernel_vsyscall ()
>  276 Thread 21837  0x0088b424 in __kernel_vsyscall ()
>  275 Thread 21873  0x0088b424 in __kernel_vsyscall ()
>  274 Thread 21840  0x0088b424 in __kernel_vsyscall ()
> ---Type <return> to continue, or q <return> to quit---q
> Quit
> (gdb) thread 280
> [Switching to thread 280 (Thread 21686)]#0  0x0088b424 in __kernel_vsyscall ()
> (gdb) bt
> #0  0x0088b424 in __kernel_vsyscall ()
> #1  0x0029b836 in writev () from /lib/libc.so.6
> #2  0x001248cf in fuse_kern_chan_send (ch=0x8ae0af8, iov=0x7c7a5200,
> count=2) at fuse_kern_chan.c:61
> #3  0x00129dad in fuse_chan_send (ch=0x8ae0af8, iov=0x7c7a5200,
> count=2) at fuse_session.c:192
> #4  0x001264b7 in fuse_send_reply_iov_nofree (req=0x8c3cc90, error=0,
> iov=0x7c7a5200, count=2)
>    at fuse_lowlevel.c:147
> #5  0x00126596 in send_reply_iov (req=0x8c3cc90, error=2,
> iov=0x7c7a5200, count=2) at fuse_lowlevel.c:155
> #6  0x001265dc in send_reply (argsize=<value optimized out>,
> arg=<value optimized out>,
>    error=<value optimized out>, req=<value optimized out>) at
> fuse_lowlevel.c:170
> #7  send_reply_ok (argsize=<value optimized out>, arg=<value optimized
> out>, error=<value optimized out>,
>    req=<value optimized out>) at fuse_lowlevel.c:243
> #8  0x00127455 in fuse_reply_xattr (req=0x8c3cc90, count=44) at
> fuse_lowlevel.c:397
> #9  0x0012295e in fuse_lib_getxattr (req=0x8c3cc90, ino=221727,
> name=0x9798250 "system.posix_acl_access", size=0)
>    at fuse.c:2924
> #10 0x00126c1f in do_getxattr (req=0x18, nodeid=221727,
> inarg=0x9798248) at fuse_lowlevel.c:916
> #11 0x00126962 in fuse_ll_process (data=0x8ae0c40, buf=0x9798220 "H",
> len=72, ch=0x8ae0af8) at fuse_lowlevel.c:1514
> #12 0x00129bf6 in fuse_session_process (se=0x8ae0490, buf=0x9798220
> "H", len=72, ch=0x8ae0af8) at fuse_session.c:80
> #13 0x00124f63 in fuse_do_work (data=0x8c30cb8) at fuse_loop_mt.c:107
> #14 0x0086d8f5 in start_thread () from /lib/libpthread.so.0
> #15 0x002a3fce in clone () from /lib/libc.so.6
> (gdb) thread 279
> [Switching to thread 279 (Thread 21792)]#0  0x0088b424 in __kernel_vsyscall ()
> (gdb) bt
> #0  0x0088b424 in __kernel_vsyscall ()
> #1  0x0029b836 in writev () from /lib/libc.so.6
> #2  0x001248cf in fuse_kern_chan_send (ch=0x8ae0af8, iov=0x36735230,
> count=2) at fuse_kern_chan.c:61
> #3  0x00129dad in fuse_chan_send (ch=0x8ae0af8, iov=0x36735230,
> count=2) at fuse_session.c:192
> #4  0x001264b7 in fuse_send_reply_iov_nofree (req=0x8c38b30, error=0,
> iov=0x36735230, count=2)
>    at fuse_lowlevel.c:147
> #5  0x00126596 in send_reply_iov (req=0x8c38b30, error=2,
> iov=0x36735230, count=2) at fuse_lowlevel.c:155
> #6  0x001265dc in send_reply (argsize=<value optimized out>,
> arg=<value optimized out>,
>    error=<value optimized out>, req=<value optimized out>) at
> fuse_lowlevel.c:170
> #7  send_reply_ok (argsize=<value optimized out>, arg=<value optimized
> out>, error=<value optimized out>,
>    req=<value optimized out>) at fuse_lowlevel.c:243
> #8  0x00122911 in fuse_lib_getxattr (req=0x8c38b30, ino=307204,
> name=0xa5845b0 "system.posix_acl_access", size=60)
>    at fuse.c:2917
> #9  0x00126c1f in do_getxattr (req=0x4c, nodeid=307204,
> inarg=0xa5845a8) at fuse_lowlevel.c:916
> #10 0x00126962 in fuse_ll_process (data=0x8ae0c40, buf=0xa584580 "H",
> len=72, ch=0x8ae0af8) at fuse_lowlevel.c:1514
> #11 0x00129bf6 in fuse_session_process (se=0x8ae0490, buf=0xa584580
> "H", len=72, ch=0x8ae0af8) at fuse_session.c:80
> #12 0x00124f63 in fuse_do_work (data=0x8c39118) at fuse_loop_mt.c:107
> #13 0x0086d8f5 in start_thread () from /lib/libpthread.so.0
> #14 0x002a3fce in clone () from /lib/libc.so.6
> (gdb)
>
> etc.
>
> Regards,
> Steve Fridella
>