Re: [fuse-devel] Filesystem requests not being sent to FUSE application

[Lucas C. Villa Real <lucasvr@go...>]

On Thu, Feb 2, 2012 at 10:53 AM, James Rhodes
<jrhodes@...> wrote:
> This is the output from GDB when inspecting the backtrace of threads when it
> locks up:

[...]

> Interestingly enough it appears as though this is always occuring on unique
> 3078 (so it's reproducible), but I can't see why Thread 2 would be stopping
> where it is (in the middle of a read operation) specifically on 3078.
>
> I'm going to take a closer look at the variables surrounding the read()
> operation on Thread 2 tomorrow.  In the meantime if anyone needs a code
> reference to look at regarding the above stack traces, the source code is
> at http://code.redpointsoftware.com.au/System/Linux/AppTools/files/af73922c8b6c0aefa80513e55655a8546f00e9e4/appfs.

Yes, that's where I'd suggest you to put more efforts on. Out of
curiosity, which file type are you reading from on that function? I
assume that's not a pipe, right?

> Regards, James Rhodes.
> Redpoint Software
>
> http://about.me/james.rhodes

Best regards,

-- 
Lucas
"If you're looking for a reason I've a reason to give: pleasure,
little treasure"

Re: [fuse-devel] Filesystem requests not being sent to FUSE application

[James Rhodes <jrhodes@re...>]

This is the output from GDB when inspecting the backtrace of threads when
it locks up:

(gdb) thread apply all bt

Thread 4 (Thread 0x7ffff5aa3700 (LWP 6675)):
#0  0x00007ffff7bcddcd in read () from /lib64/libpthread.so.0
#1  0x00000000004229fd in read (__nbytes=135168, __buf=0x7ffff7f71010,
__fd=<optimized out>) at /usr/include/bits/unistd.h:45
#2  fuse_kern_chan_receive (chp=<optimized out>, buf=0x7ffff7f71010 "8",
size=135168) at fuse_kern_chan.c:28
#3  0x0000000000429da6 in fuse_do_work (data=0x7fffe8000aa0) at
fuse_loop_mt.c:76
#4  0x00007ffff7bc6f05 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff6b7f63d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7ffff62a4700 (LWP 6653)):
#0  0x00007ffff7bcddcd in read () from /lib64/libpthread.so.0
#1  0x00000000004229fd in read (__nbytes=135168, __buf=0x7ffff7f93010,
__fd=<optimized out>) at /usr/include/bits/unistd.h:45
#2  fuse_kern_chan_receive (chp=<optimized out>, buf=0x7ffff7f93010 ":",
size=135168) at fuse_kern_chan.c:28
#3  0x0000000000429da6 in fuse_do_work (data=0x7ffff00008c0) at
fuse_loop_mt.c:76
#4  0x00007ffff7bc6f05 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff6b7f63d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7ffff6aa5700 (LWP 6652)):
#0  0x00007ffff7bcddcd in read () from /lib64/libpthread.so.0
#1  0x00007ffff792f0d8 in std::__basic_file<char>::xsgetn(char*, long) ()
from /usr/lib64/libstdc++.so.6
#2  0x00007ffff7930c75 in std::basic_filebuf<char, std::char_traits<char>
>::underflow() () from /usr/lib64/libstdc++.so.6
#3  0x00007ffff7959aba in std::basic_streambuf<char, std::char_traits<char>
>::uflow() () from /usr/lib64/libstdc++.so.6
#4  0x00007ffff7959da1 in std::basic_streambuf<char, std::char_traits<char>
>::xsgetn(char*, long) () from /usr/lib64/libstdc++.so.6
#5  0x00007ffff793047e in std::basic_filebuf<char, std::char_traits<char>
>::xsgetn(char*, long) () from /usr/lib64/libstdc++.so.6
#6  0x00007ffff793a8f3 in std::istream::read(char*, long) () from
/usr/lib64/libstdc++.so.6
#7  0x000000000040bc36 in AppLib::LowLevel::BlockStream::read
(this=0x64a4d0, out=0x7ffff6a9f32c "tr_A\200\066\064", count=4) at
blockstream.cpp:86
#8  0x0000000000417e13 in AppLib::LowLevel::Endian::doR (fd=0x64a4d0,
data=0x7ffff6a9f32c "tr_A\200\066\064", size=4) at endian.cpp:41
#9  0x0000000000416798 in AppLib::LowLevel::FreeList::getIndexInList
(this=0x64a450, pos=0) at freelist.cpp:111
#10 0x000000000041659a in AppLib::LowLevel::FreeList::freeBlock
(this=0x64a450, pos=4472832) at freelist.cpp:72
#11 0x000000000040f2e9 in AppLib::LowLevel::FS::resetBlock (this=0x64a010,
pos=4472832) at fs.cpp:806
#12 0x0000000000406647 in AppLib::FUSE::FuseLink::rmdir
(path=0x7ffff0000b00 "/tr_ABCDEF") at fuselink.cpp:304
#13 0x000000000041f155 in fuse_lib_rmdir (req=0x7ffff0000a30, parent=1,
name=0x7ffff7fb5038 "tr_ABCDEF") at fuse.c:2287
#14 0x0000000000429e16 in fuse_do_work (data=0x64cd90) at fuse_loop_mt.c:107
#15 0x00007ffff7bc6f05 in start_thread () from /lib64/libpthread.so.0
#16 0x00007ffff6b7f63d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7fd8720 (LWP 6648)):
#0  0x00007ffff7bcd010 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000429fd8 in fuse_session_loop_mt (se=0x64cd30) at
fuse_loop_mt.c:222
#2  0x0000000000427cdd in fuse_main_common (argc=<optimized out>,
argv=<optimized out>, op=<optimized out>, op_size=<optimized out>,
user_data=<optimized out>, compat=<optimized out>) at helper.c:328
#3  0x0000000000405a8c in AppLib::FUSE::Mounter::Mounter (this=0x64a3d0,
disk_image=0x7fffffffe2d9
"/home/james/Projects/AppTools/appfs/tests/working/test.afs",
mount_point=0x7fffffffe314
"/home/james/Projects/AppTools/appfs/tests/working/mount", foreground=true,
continuefunc=0x4053f4 <appmount_continue()>) at fuselink.cpp:138
#4  0x00000000004050b3 in appmount_start (argc=3, argv=0x7fffffffde48) at
appmount.cpp:109
#5  0x0000000000404b24 in main (argc=3, argv=0x7fffffffde48) at main.cpp:26

Interestingly enough it appears as though this is always occuring on unique
3078 (so it's reproducible), but I can't see why Thread 2 would be stopping
where it is (in the middle of a read operation) specifically on 3078.

I'm going to take a closer look at the variables surrounding the read()
operation on Thread 2 tomorrow.  In the meantime if anyone needs a code
reference to look at regarding the above stack traces, the source code is
at
http://code.redpointsoftware.com.au/System/Linux/AppTools/files/af73922c8b6c0aefa80513e55655a8546f00e9e4/appfs
.

Regards, James Rhodes.
Redpoint Software

http://about.me/james.rhodes




On Thu, Feb 2, 2012 at 11:16 PM, Lucas C. Villa Real
<lucasvr@...>wrote:

> On Thu, Feb 2, 2012 at 2:16 AM, James Rhodes
> <jrhodes@...> wrote:
> > Hi everyone,
> >
> > Just confirmed that applying the 'sync' option does not fix this issue.
>  In
> > addition, for the last 3 tests it has always finished around request
> unique
> > ID 3078.
> >
> > Regards, James Rhodes.
> > Redpoint Software
> >
> > http://about.me/james.rhodes
>
> Hi James,
>
> From the stacktrace it looks like FUSE/kernel is waiting for your
> filesystem daemon to respond to the rmdir request.
>
> [ 3985.016197] rmdir           S 0000000000000000     0  7943   6389
> 0x00000000
> [ 3985.016197]  ffff88000020bd78 0000000000000082 ffff88000020bd48
> ffffffff8103f626
> [ 3985.016197]  ffff88000020bfd8 ffff88000020bfd8 ffff88000020bfd8
> 00000000000123c0
> [ 3985.016197]  ffff88000aa06240 ffff8800021a26c0 0000000000000246
> ffff8800089daca8
> [ 3985.016197] Call Trace:
> [ 3985.016197]  [<ffffffffa029a4d2>] wait_answer_interruptible+0x72/0xb0
> [fuse]
> [ 3985.016197]  [<ffffffffa029a7f5>] request_wait_answer+0x145/0x220 [fuse]
> [ 3985.016197]  [<ffffffffa029a93b>] fuse_request_send+0x6b/0xa0 [fuse]
> [ 3985.016197]  [<ffffffffa029df6b>] fuse_rmdir+0x7b/0x100 [fuse]
> [ 3985.016197]  [<ffffffff811574d7>] vfs_rmdir.part.29+0xa7/0xf0
> [ 3985.016197]  [<ffffffff811598d3>] do_rmdir+0x113/0x120
> [ 3985.016197]  [<ffffffff81545692>] system_call_fastpath+0x16/0x1b
> [ 3985.016197]  [<00007fa3755fac17>] 0x7fa3755fac16
>
> Perhaps you have some deadlock situation in your code (or an ancient
> version of libfuse which contains some bug)? Have you tried to attach
> gdb to your process?
>
> --
> Lucas
> "If you're looking for a reason I've a reason to give: pleasure,
> little treasure"
>

Re: [fuse-devel] Filesystem requests not being sent to FUSE application

[Lucas C. Villa Real <lucasvr@go...>]

On Thu, Feb 2, 2012 at 2:16 AM, James Rhodes
<jrhodes@...> wrote:
> Hi everyone,
>
> Just confirmed that applying the 'sync' option does not fix this issue.  In
> addition, for the last 3 tests it has always finished around request unique
> ID 3078.
>
> Regards, James Rhodes.
> Redpoint Software
>
> http://about.me/james.rhodes

Hi James,

>From the stacktrace it looks like FUSE/kernel is waiting for your
filesystem daemon to respond to the rmdir request.

[ 3985.016197] rmdir           S 0000000000000000     0  7943   6389 0x00000000
[ 3985.016197]  ffff88000020bd78 0000000000000082 ffff88000020bd48
ffffffff8103f626
[ 3985.016197]  ffff88000020bfd8 ffff88000020bfd8 ffff88000020bfd8
00000000000123c0
[ 3985.016197]  ffff88000aa06240 ffff8800021a26c0 0000000000000246
ffff8800089daca8
[ 3985.016197] Call Trace:
[ 3985.016197]  [<ffffffffa029a4d2>] wait_answer_interruptible+0x72/0xb0 [fuse]
[ 3985.016197]  [<ffffffffa029a7f5>] request_wait_answer+0x145/0x220 [fuse]
[ 3985.016197]  [<ffffffffa029a93b>] fuse_request_send+0x6b/0xa0 [fuse]
[ 3985.016197]  [<ffffffffa029df6b>] fuse_rmdir+0x7b/0x100 [fuse]
[ 3985.016197]  [<ffffffff811574d7>] vfs_rmdir.part.29+0xa7/0xf0
[ 3985.016197]  [<ffffffff811598d3>] do_rmdir+0x113/0x120
[ 3985.016197]  [<ffffffff81545692>] system_call_fastpath+0x16/0x1b
[ 3985.016197]  [<00007fa3755fac17>] 0x7fa3755fac16

Perhaps you have some deadlock situation in your code (or an ancient
version of libfuse which contains some bug)? Have you tried to attach
gdb to your process?

-- 
Lucas
"If you're looking for a reason I've a reason to give: pleasure,
little treasure"

Re: [fuse-devel] Filesystem requests not being sent to FUSE application

[James Rhodes <jrhodes@re...>]

Hi everyone,

Just confirmed that applying the 'sync' option does not fix this issue.  In
addition, for the last 3 tests it has always finished around request unique
ID 3078.

Regards, James Rhodes.
Redpoint Software

http://about.me/james.rhodes




On Thu, Feb 2, 2012 at 3:08 PM, James Rhodes <
jrhodes@...> wrote:

> Just a thought, does FUSE invoke requests asynchronously (i.e. is it
> possible for the mkdir and rmdir handles to be executing at the exact same
> time)?  If this is the case, that might be the cause of this issue given
> that the filesystem is not designed for asynchronous write operations (read
> is okay though).
>
> Regards, James Rhodes.
> Redpoint Software
>
> http://about.me/james.rhodes
>
>
>
>
> On Thu, Feb 2, 2012 at 2:05 PM, James Rhodes <
> jrhodes@...> wrote:
>
>> Sorry guys,
>>
>> Looks like codepad.org truncates pasted text after a certain limit.
>>  Here's the full paste: http://pastebin.com/raw.php?i=hxTUYr4q.
>>
>> Regards, James Rhodes.
>> Redpoint Software
>>
>> http://about.me/james.rhodes
>>
>>
>>
>>
>> On Thu, Feb 2, 2012 at 1:58 PM, James Rhodes <
>> jrhodes@...> wrote:
>>
>>> Hi everyone,
>>>
>>> Here's the output from dmesg generated when invoking 'echo t >
>>> /proc/sysrq-trigger' (while the filesystem is locked up in sem_wait).
>>>
>>> http://codepad.org/eGM7vVzb (linked since it's a very large amount of
>>> text)
>>>
>>> Regards, James Rhodes.
>>> Redpoint Software
>>>
>>> http://about.me/james.rhodes
>>>
>>>
>>>
>>>
>>> On Thu, Feb 2, 2012 at 1:39 PM, Lucas C. Villa Real <
>>> lucasvr@...> wrote:
>>>
>>>> On Thu, Feb 2, 2012 at 12:23 AM, James Rhodes
>>>> <jrhodes@...> wrote:
>>>> > Hi everyone,
>>>> >
>>>> > I'm writing a small filesystem in FUSE and I'm noticing some very
>>>> strange
>>>> > behaviour when doing simple reliability tests.  The reliability test
>>>> > consists of:
>>>> >
>>>> > while (true); do
>>>> >        mkdir $DIR_MOUNT/tr_ABCDEF
>>>> >        rmdir $DIR_MOUNT/tr_ABCDEF
>>>> > done
>>>> >
>>>> > This works fine for the X (it varies significantly) or so requests to
>>>> the
>>>> > FUSE filesystem and then... FUSE just stops sending requests to it.
>>>>  The
>>>> > application doesn't crash or lock up in my code, but sits waiting in
>>>> > sem_wait inside fuse_session_loop.  The full backtrace when
>>>> interrupting
>>>> > GDB is:
>>>> >
>>>> > #0  0x00007ffff7bcd010 in sem_wait () from /lib64/libpthread.so.0
>>>> > #1  0x0000000000429fd8 in fuse_session_loop_mt (se=0x64cd30) at
>>>> > fuse_loop_mt.c:222
>>>> > #2  0x0000000000427cdd in fuse_main_common (argc=<optimized out>,
>>>> > argv=<optimized out>, op=<optimized out>, op_size=<optimized out>,
>>>> > user_data=<optimized out>, compat=<optimized out>) at helper.c:328
>>>> > #3  0x0000000000405a8c in AppLib::FUSE::Mounter::Mounter
>>>> (this=0x64a3d0,
>>>> > disk_image=0x7fffffffe345
>>>> > "/home/james/Projects/AppTools/appfs/tests/working/test.afs",
>>>> > mount_point=0x7fffffffe380
>>>> > "/home/james/Projects/AppTools/appfs/tests/working/mount",
>>>> foreground=true,
>>>> > continuefunc=0x4053f4 <appmount_continue()>) at fuselink.cpp:138
>>>> > #4  0x00000000004050b3 in appmount_start (argc=3,
>>>> argv=0x7fffffffdee8) at
>>>> > appmount.cpp:109
>>>> > #5  0x0000000000404b24 in main (argc=3, argv=0x7fffffffdee8) at
>>>> main.cpp:26
>>>> >
>>>> > This to me at least doesn't make sense.  I can't see any reason why
>>>> the
>>>> > mkdir / rmdir requests being passed to the kernel and then to FUSE
>>>> would
>>>> > not then make their way to my actual application code, given that it
>>>> > already worked for X requests previously.  When I proceed to then do:
>>>> >
>>>> > ls $DIR_MOUNT/
>>>> >
>>>> > the FUSE filesystem again locks up in sem_wait and never begins
>>>> executing
>>>> > my code, meanwhile ls is locked up on the terminal waiting for the
>>>> getattr
>>>> > request to finish.
>>>> >
>>>> > Is there anything here that could be causing such a strange issue?
>>>>
>>>> Hi James,
>>>>
>>>> A kernel stack trace is going to be very helpful to have someone help
>>>> you to debug that. You can invoke 'echo t > /proc/sysrq-trigger' and
>>>> then send its output (which goes to the kernel log -- see dmesg) to
>>>> the list.
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>> Lucas
>>>> "If you're looking for a reason I've a reason to give: pleasure,
>>>> little treasure"
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Keep Your Developer Skills Current with LearnDevNow!
>>>> The most comprehensive online learning library for Microsoft developers
>>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>>>> Metro Style Apps, more. Free future releases when you subscribe now!
>>>> http://p.sf.net/sfu/learndevnow-d2d
>>>> _______________________________________________
>>>> fuse-devel mailing list
>>>> fuse-devel@...
>>>> https://lists.sourceforge.net/lists/listinfo/fuse-devel
>>>>
>>>
>>>
>>
>

Re: [fuse-devel] Filesystem requests not being sent to FUSE application

[James Rhodes <jrhodes@re...>]

Just a thought, does FUSE invoke requests asynchronously (i.e. is it
possible for the mkdir and rmdir handles to be executing at the exact same
time)?  If this is the case, that might be the cause of this issue given
that the filesystem is not designed for asynchronous write operations (read
is okay though).

Regards, James Rhodes.
Redpoint Software

http://about.me/james.rhodes




On Thu, Feb 2, 2012 at 2:05 PM, James Rhodes <
jrhodes@...> wrote:

> Sorry guys,
>
> Looks like codepad.org truncates pasted text after a certain limit.
>  Here's the full paste: http://pastebin.com/raw.php?i=hxTUYr4q.
>
> Regards, James Rhodes.
> Redpoint Software
>
> http://about.me/james.rhodes
>
>
>
>
> On Thu, Feb 2, 2012 at 1:58 PM, James Rhodes <
> jrhodes@...> wrote:
>
>> Hi everyone,
>>
>> Here's the output from dmesg generated when invoking 'echo t >
>> /proc/sysrq-trigger' (while the filesystem is locked up in sem_wait).
>>
>> http://codepad.org/eGM7vVzb (linked since it's a very large amount of
>> text)
>>
>> Regards, James Rhodes.
>> Redpoint Software
>>
>> http://about.me/james.rhodes
>>
>>
>>
>>
>> On Thu, Feb 2, 2012 at 1:39 PM, Lucas C. Villa Real <
>> lucasvr@...> wrote:
>>
>>> On Thu, Feb 2, 2012 at 12:23 AM, James Rhodes
>>> <jrhodes@...> wrote:
>>> > Hi everyone,
>>> >
>>> > I'm writing a small filesystem in FUSE and I'm noticing some very
>>> strange
>>> > behaviour when doing simple reliability tests.  The reliability test
>>> > consists of:
>>> >
>>> > while (true); do
>>> >        mkdir $DIR_MOUNT/tr_ABCDEF
>>> >        rmdir $DIR_MOUNT/tr_ABCDEF
>>> > done
>>> >
>>> > This works fine for the X (it varies significantly) or so requests to
>>> the
>>> > FUSE filesystem and then... FUSE just stops sending requests to it.
>>>  The
>>> > application doesn't crash or lock up in my code, but sits waiting in
>>> > sem_wait inside fuse_session_loop.  The full backtrace when
>>> interrupting
>>> > GDB is:
>>> >
>>> > #0  0x00007ffff7bcd010 in sem_wait () from /lib64/libpthread.so.0
>>> > #1  0x0000000000429fd8 in fuse_session_loop_mt (se=0x64cd30) at
>>> > fuse_loop_mt.c:222
>>> > #2  0x0000000000427cdd in fuse_main_common (argc=<optimized out>,
>>> > argv=<optimized out>, op=<optimized out>, op_size=<optimized out>,
>>> > user_data=<optimized out>, compat=<optimized out>) at helper.c:328
>>> > #3  0x0000000000405a8c in AppLib::FUSE::Mounter::Mounter
>>> (this=0x64a3d0,
>>> > disk_image=0x7fffffffe345
>>> > "/home/james/Projects/AppTools/appfs/tests/working/test.afs",
>>> > mount_point=0x7fffffffe380
>>> > "/home/james/Projects/AppTools/appfs/tests/working/mount",
>>> foreground=true,
>>> > continuefunc=0x4053f4 <appmount_continue()>) at fuselink.cpp:138
>>> > #4  0x00000000004050b3 in appmount_start (argc=3, argv=0x7fffffffdee8)
>>> at
>>> > appmount.cpp:109
>>> > #5  0x0000000000404b24 in main (argc=3, argv=0x7fffffffdee8) at
>>> main.cpp:26
>>> >
>>> > This to me at least doesn't make sense.  I can't see any reason why the
>>> > mkdir / rmdir requests being passed to the kernel and then to FUSE
>>> would
>>> > not then make their way to my actual application code, given that it
>>> > already worked for X requests previously.  When I proceed to then do:
>>> >
>>> > ls $DIR_MOUNT/
>>> >
>>> > the FUSE filesystem again locks up in sem_wait and never begins
>>> executing
>>> > my code, meanwhile ls is locked up on the terminal waiting for the
>>> getattr
>>> > request to finish.
>>> >
>>> > Is there anything here that could be causing such a strange issue?
>>>
>>> Hi James,
>>>
>>> A kernel stack trace is going to be very helpful to have someone help
>>> you to debug that. You can invoke 'echo t > /proc/sysrq-trigger' and
>>> then send its output (which goes to the kernel log -- see dmesg) to
>>> the list.
>>>
>>> Thanks,
>>>
>>> --
>>> Lucas
>>> "If you're looking for a reason I've a reason to give: pleasure,
>>> little treasure"
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Keep Your Developer Skills Current with LearnDevNow!
>>> The most comprehensive online learning library for Microsoft developers
>>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>>> Metro Style Apps, more. Free future releases when you subscribe now!
>>> http://p.sf.net/sfu/learndevnow-d2d
>>> _______________________________________________
>>> fuse-devel mailing list
>>> fuse-devel@...
>>> https://lists.sourceforge.net/lists/listinfo/fuse-devel
>>>
>>
>>
>

Re: [fuse-devel] Filesystem requests not being sent to FUSE application

[James Rhodes <jrhodes@re...>]

Sorry guys,

Looks like codepad.org truncates pasted text after a certain limit.  Here's
the full paste: http://pastebin.com/raw.php?i=hxTUYr4q.

Regards, James Rhodes.
Redpoint Software

http://about.me/james.rhodes




On Thu, Feb 2, 2012 at 1:58 PM, James Rhodes <
jrhodes@...> wrote:

> Hi everyone,
>
> Here's the output from dmesg generated when invoking 'echo t >
> /proc/sysrq-trigger' (while the filesystem is locked up in sem_wait).
>
> http://codepad.org/eGM7vVzb (linked since it's a very large amount of
> text)
>
> Regards, James Rhodes.
> Redpoint Software
>
> http://about.me/james.rhodes
>
>
>
>
> On Thu, Feb 2, 2012 at 1:39 PM, Lucas C. Villa Real <lucasvr@...
> > wrote:
>
>> On Thu, Feb 2, 2012 at 12:23 AM, James Rhodes
>> <jrhodes@...> wrote:
>> > Hi everyone,
>> >
>> > I'm writing a small filesystem in FUSE and I'm noticing some very
>> strange
>> > behaviour when doing simple reliability tests.  The reliability test
>> > consists of:
>> >
>> > while (true); do
>> >        mkdir $DIR_MOUNT/tr_ABCDEF
>> >        rmdir $DIR_MOUNT/tr_ABCDEF
>> > done
>> >
>> > This works fine for the X (it varies significantly) or so requests to
>> the
>> > FUSE filesystem and then... FUSE just stops sending requests to it.  The
>> > application doesn't crash or lock up in my code, but sits waiting in
>> > sem_wait inside fuse_session_loop.  The full backtrace when interrupting
>> > GDB is:
>> >
>> > #0  0x00007ffff7bcd010 in sem_wait () from /lib64/libpthread.so.0
>> > #1  0x0000000000429fd8 in fuse_session_loop_mt (se=0x64cd30) at
>> > fuse_loop_mt.c:222
>> > #2  0x0000000000427cdd in fuse_main_common (argc=<optimized out>,
>> > argv=<optimized out>, op=<optimized out>, op_size=<optimized out>,
>> > user_data=<optimized out>, compat=<optimized out>) at helper.c:328
>> > #3  0x0000000000405a8c in AppLib::FUSE::Mounter::Mounter (this=0x64a3d0,
>> > disk_image=0x7fffffffe345
>> > "/home/james/Projects/AppTools/appfs/tests/working/test.afs",
>> > mount_point=0x7fffffffe380
>> > "/home/james/Projects/AppTools/appfs/tests/working/mount",
>> foreground=true,
>> > continuefunc=0x4053f4 <appmount_continue()>) at fuselink.cpp:138
>> > #4  0x00000000004050b3 in appmount_start (argc=3, argv=0x7fffffffdee8)
>> at
>> > appmount.cpp:109
>> > #5  0x0000000000404b24 in main (argc=3, argv=0x7fffffffdee8) at
>> main.cpp:26
>> >
>> > This to me at least doesn't make sense.  I can't see any reason why the
>> > mkdir / rmdir requests being passed to the kernel and then to FUSE would
>> > not then make their way to my actual application code, given that it
>> > already worked for X requests previously.  When I proceed to then do:
>> >
>> > ls $DIR_MOUNT/
>> >
>> > the FUSE filesystem again locks up in sem_wait and never begins
>> executing
>> > my code, meanwhile ls is locked up on the terminal waiting for the
>> getattr
>> > request to finish.
>> >
>> > Is there anything here that could be causing such a strange issue?
>>
>> Hi James,
>>
>> A kernel stack trace is going to be very helpful to have someone help
>> you to debug that. You can invoke 'echo t > /proc/sysrq-trigger' and
>> then send its output (which goes to the kernel log -- see dmesg) to
>> the list.
>>
>> Thanks,
>>
>> --
>> Lucas
>> "If you're looking for a reason I've a reason to give: pleasure,
>> little treasure"
>>
>>
>> ------------------------------------------------------------------------------
>> Keep Your Developer Skills Current with LearnDevNow!
>> The most comprehensive online learning library for Microsoft developers
>> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
>> Metro Style Apps, more. Free future releases when you subscribe now!
>> http://p.sf.net/sfu/learndevnow-d2d
>> _______________________________________________
>> fuse-devel mailing list
>> fuse-devel@...
>> https://lists.sourceforge.net/lists/listinfo/fuse-devel
>>
>
>

Re: [fuse-devel] Filesystem requests not being sent to FUSE application

[James Rhodes <jrhodes@re...>]

Hi everyone,

Here's the output from dmesg generated when invoking 'echo t >
/proc/sysrq-trigger' (while the filesystem is locked up in sem_wait).

http://codepad.org/eGM7vVzb (linked since it's a very large amount of text)

Regards, James Rhodes.
Redpoint Software

http://about.me/james.rhodes




On Thu, Feb 2, 2012 at 1:39 PM, Lucas C. Villa Real
<lucasvr@...>wrote:

> On Thu, Feb 2, 2012 at 12:23 AM, James Rhodes
> <jrhodes@...> wrote:
> > Hi everyone,
> >
> > I'm writing a small filesystem in FUSE and I'm noticing some very strange
> > behaviour when doing simple reliability tests.  The reliability test
> > consists of:
> >
> > while (true); do
> >        mkdir $DIR_MOUNT/tr_ABCDEF
> >        rmdir $DIR_MOUNT/tr_ABCDEF
> > done
> >
> > This works fine for the X (it varies significantly) or so requests to the
> > FUSE filesystem and then... FUSE just stops sending requests to it.  The
> > application doesn't crash or lock up in my code, but sits waiting in
> > sem_wait inside fuse_session_loop.  The full backtrace when interrupting
> > GDB is:
> >
> > #0  0x00007ffff7bcd010 in sem_wait () from /lib64/libpthread.so.0
> > #1  0x0000000000429fd8 in fuse_session_loop_mt (se=0x64cd30) at
> > fuse_loop_mt.c:222
> > #2  0x0000000000427cdd in fuse_main_common (argc=<optimized out>,
> > argv=<optimized out>, op=<optimized out>, op_size=<optimized out>,
> > user_data=<optimized out>, compat=<optimized out>) at helper.c:328
> > #3  0x0000000000405a8c in AppLib::FUSE::Mounter::Mounter (this=0x64a3d0,
> > disk_image=0x7fffffffe345
> > "/home/james/Projects/AppTools/appfs/tests/working/test.afs",
> > mount_point=0x7fffffffe380
> > "/home/james/Projects/AppTools/appfs/tests/working/mount",
> foreground=true,
> > continuefunc=0x4053f4 <appmount_continue()>) at fuselink.cpp:138
> > #4  0x00000000004050b3 in appmount_start (argc=3, argv=0x7fffffffdee8) at
> > appmount.cpp:109
> > #5  0x0000000000404b24 in main (argc=3, argv=0x7fffffffdee8) at
> main.cpp:26
> >
> > This to me at least doesn't make sense.  I can't see any reason why the
> > mkdir / rmdir requests being passed to the kernel and then to FUSE would
> > not then make their way to my actual application code, given that it
> > already worked for X requests previously.  When I proceed to then do:
> >
> > ls $DIR_MOUNT/
> >
> > the FUSE filesystem again locks up in sem_wait and never begins executing
> > my code, meanwhile ls is locked up on the terminal waiting for the
> getattr
> > request to finish.
> >
> > Is there anything here that could be causing such a strange issue?
>
> Hi James,
>
> A kernel stack trace is going to be very helpful to have someone help
> you to debug that. You can invoke 'echo t > /proc/sysrq-trigger' and
> then send its output (which goes to the kernel log -- see dmesg) to
> the list.
>
> Thanks,
>
> --
> Lucas
> "If you're looking for a reason I've a reason to give: pleasure,
> little treasure"
>
>
> ------------------------------------------------------------------------------
> Keep Your Developer Skills Current with LearnDevNow!
> The most comprehensive online learning library for Microsoft developers
> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
> Metro Style Apps, more. Free future releases when you subscribe now!
> http://p.sf.net/sfu/learndevnow-d2d
> _______________________________________________
> fuse-devel mailing list
> fuse-devel@...
> https://lists.sourceforge.net/lists/listinfo/fuse-devel
>

Re: [fuse-devel] Filesystem requests not being sent to FUSE application

[Lucas C. Villa Real <lucasvr@go...>]

On Thu, Feb 2, 2012 at 12:23 AM, James Rhodes
<jrhodes@...> wrote:
> Hi everyone,
>
> I'm writing a small filesystem in FUSE and I'm noticing some very strange
> behaviour when doing simple reliability tests.  The reliability test
> consists of:
>
> while (true); do
>        mkdir $DIR_MOUNT/tr_ABCDEF
>        rmdir $DIR_MOUNT/tr_ABCDEF
> done
>
> This works fine for the X (it varies significantly) or so requests to the
> FUSE filesystem and then... FUSE just stops sending requests to it.  The
> application doesn't crash or lock up in my code, but sits waiting in
> sem_wait inside fuse_session_loop.  The full backtrace when interrupting
> GDB is:
>
> #0  0x00007ffff7bcd010 in sem_wait () from /lib64/libpthread.so.0
> #1  0x0000000000429fd8 in fuse_session_loop_mt (se=0x64cd30) at
> fuse_loop_mt.c:222
> #2  0x0000000000427cdd in fuse_main_common (argc=<optimized out>,
> argv=<optimized out>, op=<optimized out>, op_size=<optimized out>,
> user_data=<optimized out>, compat=<optimized out>) at helper.c:328
> #3  0x0000000000405a8c in AppLib::FUSE::Mounter::Mounter (this=0x64a3d0,
> disk_image=0x7fffffffe345
> "/home/james/Projects/AppTools/appfs/tests/working/test.afs",
> mount_point=0x7fffffffe380
> "/home/james/Projects/AppTools/appfs/tests/working/mount", foreground=true,
> continuefunc=0x4053f4 <appmount_continue()>) at fuselink.cpp:138
> #4  0x00000000004050b3 in appmount_start (argc=3, argv=0x7fffffffdee8) at
> appmount.cpp:109
> #5  0x0000000000404b24 in main (argc=3, argv=0x7fffffffdee8) at main.cpp:26
>
> This to me at least doesn't make sense.  I can't see any reason why the
> mkdir / rmdir requests being passed to the kernel and then to FUSE would
> not then make their way to my actual application code, given that it
> already worked for X requests previously.  When I proceed to then do:
>
> ls $DIR_MOUNT/
>
> the FUSE filesystem again locks up in sem_wait and never begins executing
> my code, meanwhile ls is locked up on the terminal waiting for the getattr
> request to finish.
>
> Is there anything here that could be causing such a strange issue?

Hi James,

A kernel stack trace is going to be very helpful to have someone help
you to debug that. You can invoke 'echo t > /proc/sysrq-trigger' and
then send its output (which goes to the kernel log -- see dmesg) to
the list.

Thanks,

-- 
Lucas
"If you're looking for a reason I've a reason to give: pleasure,
little treasure"

[fuse-devel] Filesystem requests not being sent to FUSE application

[James Rhodes <jrhodes@re...>]

Hi everyone,

I'm writing a small filesystem in FUSE and I'm noticing some very strange
behaviour when doing simple reliability tests.  The reliability test
consists of:

while (true); do
        mkdir $DIR_MOUNT/tr_ABCDEF
        rmdir $DIR_MOUNT/tr_ABCDEF
done

This works fine for the X (it varies significantly) or so requests to the
FUSE filesystem and then... FUSE just stops sending requests to it.  The
application doesn't crash or lock up in my code, but sits waiting in
sem_wait inside fuse_session_loop.  The full backtrace when interrupting
GDB is:

#0  0x00007ffff7bcd010 in sem_wait () from /lib64/libpthread.so.0
#1  0x0000000000429fd8 in fuse_session_loop_mt (se=0x64cd30) at
fuse_loop_mt.c:222
#2  0x0000000000427cdd in fuse_main_common (argc=<optimized out>,
argv=<optimized out>, op=<optimized out>, op_size=<optimized out>,
user_data=<optimized out>, compat=<optimized out>) at helper.c:328
#3  0x0000000000405a8c in AppLib::FUSE::Mounter::Mounter (this=0x64a3d0,
disk_image=0x7fffffffe345
"/home/james/Projects/AppTools/appfs/tests/working/test.afs",
mount_point=0x7fffffffe380
"/home/james/Projects/AppTools/appfs/tests/working/mount", foreground=true,
continuefunc=0x4053f4 <appmount_continue()>) at fuselink.cpp:138
#4  0x00000000004050b3 in appmount_start (argc=3, argv=0x7fffffffdee8) at
appmount.cpp:109
#5  0x0000000000404b24 in main (argc=3, argv=0x7fffffffdee8) at main.cpp:26

This to me at least doesn't make sense.  I can't see any reason why the
mkdir / rmdir requests being passed to the kernel and then to FUSE would
not then make their way to my actual application code, given that it
already worked for X requests previously.  When I proceed to then do:

ls $DIR_MOUNT/

the FUSE filesystem again locks up in sem_wait and never begins executing
my code, meanwhile ls is locked up on the terminal waiting for the getattr
request to finish.

Is there anything here that could be causing such a strange issue?

Regards, James Rhodes.
Redpoint Software

http://about.me/james.rhodes