|
From: Albert S. <fu...@gm...> - 2011-07-28 19:02:38
|
Hello all I have been testing Valgrind 3.6.1 and the latest from SVN with some Go binaries. A bug I reported against 3.6.0 still seems to exit: https://bugs.kde.org/show_bug.cgi?id=262916 Running Go binaries under Valgrind lead to a segfault in vgPlain_get_StackTrace_wrk. To reproduce: Follow instructions at http://golang.org/doc/install.html cd $GOROOT/src/pkg/ebnf make test valgrind ./6.out Are there any developers that could take a look? It would be really nice to have Valgrind working with Go binaries that call C libraries. Thanks! Regards Albert |
|
From: Albert S. <fu...@gm...> - 2011-07-28 20:21:53
|
Hello again On Thu, Jul 28, 2011 at 9:02 PM, Albert Strasheim <fu...@gm...> wrote: > I have been testing Valgrind 3.6.1 and the latest from SVN with some > Go binaries. > A bug I reported against 3.6.0 still seems to exit: > https://bugs.kde.org/show_bug.cgi?id=262916 > Running Go binaries under Valgrind lead to a segfault in > vgPlain_get_StackTrace_wrk. The line that crashes is this: m_stacktrace.c:334: uregs.xip = (((UWord*)uregs.xbp)[1]); (gdb) p uregs $1 = {xip = 1066225967935, xsp = 75771824, xbp = 4294966222} Above the code there is this comment: /* Note: re "- 1 * sizeof(UWord)", need to take account of the fact that we are prodding at & ((UWord*)fp)[1] and so need to adjust the limit check accordingly. Omitting this has been observed to cause segfaults on rare occasions. */ so it seems Go binaries are triggering this "rare occasion". Any thoughts on how to proceed would be appreciated. Thanks! Regards Albert |
|
From: Alexander P. <gl...@go...> - 2011-07-28 20:46:47
|
Running the binaries under Nullgrind (valgrind --tool=none) is better to start with, because Go programs are likely to trigger some incompatibilities in the Valgrind runtime itself, not necessarily Memcheck. For example, goroutines may be tricky to mirror in a straightforward way (using threads), because there can be thousands of them, while Valgrind has a strict limit for the number of threads. Second, they use segmented stacks in Go, which are extended if needed. Valgrind does not know about this and may be confused (I suppose the segfaults in your report could denote the demand for increasing the stack size) A side question: is there any particular reason you need Valgrind for Go? Alex |
|
From: Albert S. <fu...@gm...> - 2011-07-28 20:57:54
|
Hello On Thu, Jul 28, 2011 at 10:46 PM, Alexander Potapenko <gl...@go...> wrote: > Running the binaries under Nullgrind (valgrind --tool=none) is better > to start with, because Go programs are likely to trigger some > incompatibilities in the Valgrind runtime itself, not necessarily > Memcheck. Thanks. I can successfully run all of the Go test programs I tried with Nullgrind. > For example, goroutines may be tricky to mirror in a straightforward > way (using threads), because there can be thousands of them, while > Valgrind has a strict limit for the number of threads. Go also spawns threads. Valgrind probably shouldn't have care about goroutines. > Second, they use segmented stacks in Go, which are extended if needed. > Valgrind does not know about this and may be confused (I suppose the > segfaults in your report could denote the demand for increasing the > stack size) Valgrind does say: ==20554== Warning: client switching stacks? SP change: 0x7ff000440 --> 0x4841fb8 ==20554== to suppress, use: --max-stackframe=34267194504 or greater ==20554== Warning: client switching stacks? SP change: 0x4841f68 --> 0x7ff000470 ==20554== to suppress, use: --max-stackframe=34267194632 or greater ==20554== Warning: client switching stacks? SP change: 0x7ff0003f0 --> 0xf84000f050 ==20554== to suppress, use: --max-stackframe=1031882730592 or greater ==20554== further instances of this message will not be shown. but I still have to read through the documentation of --max-stackframe and friends more carefully. > A side question: is there any particular reason you need Valgrind for Go? Go programs can use C libraries through something called cgo. I'd like to be able to check this C code inside my Go binaries. Also, Valgrind might be able to find bugs in the C part of the Go runtime (goroutine scheduler, garbage collector, etc.). Regards Albert |
|
From: Rich C. <rc...@wi...> - 2011-07-28 20:52:44
|
Hi,
Using go tip:9218 on linux x86_64, I couldn't reproduce this.
What about your environment makes this issue unique ?
Rich
On Thu, 28 Jul 2011 21:02:11 +0200
Albert Strasheim <fu...@gm...> wrote:
> Hello all
>
> I have been testing Valgrind 3.6.1 and the latest from SVN with some
> Go binaries.
>
> A bug I reported against 3.6.0 still seems to exit:
>
> https://bugs.kde.org/show_bug.cgi?id=262916
>
> Running Go binaries under Valgrind lead to a segfault in
> vgPlain_get_StackTrace_wrk.
>
> To reproduce:
>
> Follow instructions at http://golang.org/doc/install.html
> cd $GOROOT/src/pkg/ebnf
> make test
> valgrind ./6.out
>
> Are there any developers that could take a look? It would be really
> nice to have Valgrind working with Go binaries that call C libraries.
>
> Thanks!
>
> Regards
>
> Albert
>
> ------------------------------------------------------------------------------
> Got Input? Slashdot Needs You.
> Take our quick survey online. Come on, we don't ask for help often.
> Plus, you'll get a chance to win $100 to spend on ThinkGeek.
> http://p.sf.net/sfu/slashdot-survey
> _______________________________________________
> Valgrind-developers mailing list
> Val...@li...
> https://lists.sourceforge.net/lists/listinfo/valgrind-developers
--
Rich Coe rc...@wi...
|
|
From: Philippe W. <phi...@sk...> - 2011-07-28 20:53:43
|
> Above the code there is this comment: > > /* Note: re "- 1 * sizeof(UWord)", need to take account of the > fact that we are prodding at & ((UWord*)fp)[1] and so need to > adjust the limit check accordingly. Omitting this has been > observed to cause segfaults on rare occasions. */ > > so it seems Go binaries are triggering this "rare occasion". > > Any thoughts on how to proceed would be appreciated. I do not know much about how Valgrind computes the stack trace of the simulated cpu, so the below suggestion might be useless but it does not cost much to suggest :). So, in 3.7.0 svn, Valgrind has an embedded gdbserver. To see what goes wrong in the stack trace of the simulated cpu, you might try to compare how the Valgrind core computes a backtrace ( the one that fails) to the way gdb computes a backtrace (through the Valgrind gdbserver) to the way gdb computes a backtrace when debugging natively 6.out. (you will have for the last two to put a break just before the 6.out instruction which causes the problematic stacktrace). Philippe |
|
From: Albert S. <fu...@gm...> - 2011-07-28 21:02:06
|
Hello On Thu, Jul 28, 2011 at 10:52 PM, Rich Coe <rc...@wi...> wrote: > Hi, > > Using go tip:9218 on linux x86_64, I couldn't reproduce this. > What about your environment makes this issue unique ? > > Rich You're right! I guess I was accidentally still running 3.6.1 instead of the build from SVN. I guess bug 262916 can be closed now. The next thing to do is to take a closer look at the warnings produced. Thanks. Regards Albert |
|
From: Albert S. <fu...@gm...> - 2011-07-28 21:20:03
|
Hello again On Thu, Jul 28, 2011 at 11:01 PM, Albert Strasheim <fu...@gm...> wrote: > Hello > On Thu, Jul 28, 2011 at 10:52 PM, Rich Coe <rc...@wi...> wrote: >> Hi, >> Using go tip:9218 on linux x86_64, I couldn't reproduce this. >> What about your environment makes this issue unique ? >> Rich > You're right! I guess I was accidentally still running 3.6.1 instead > of the build from SVN. > I guess bug 262916 can be closed now. > The next thing to do is to take a closer look at the warnings produced. I've done some more tests, and most packages work, but the 6.out from the reflect package still produces a crash at m_stacktrace.c:334. I'm definitely running the code from SVN now. Regards Albert |
|
From: Albert S. <fu...@gm...> - 2011-07-28 21:33:40
|
Hello On Thu, Jul 28, 2011 at 10:46 PM, Alexander Potapenko <gl...@go...> wrote: > For example, goroutines may be tricky to mirror in a straightforward > way (using threads), because there can be thousands of them, while > Valgrind has a strict limit for the number of threads. > Second, they use segmented stacks in Go, which are extended if needed. > Valgrind does not know about this and may be confused (I suppose the > segfaults in your report could denote the demand for increasing the > stack size) On this note, I see Valgrind has three client requests that might be useful: VALGRIND_STACK_REGISTER VALGRIND_STACK_DEREGISTER VALGRIND_STACK_CHANGE Adding these to the Go runtime should be relatively straight-forward. However, the documentation at http://valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.clientreq says: Warning: Unfortunately, this client request is unreliable and best avoided. Is this still the case with Valgrind in SVN? If so, is this something that could be fixed relatively easily or are there dragons here? Regards Albert |
|
From: Philippe W. <phi...@sk...> - 2011-07-28 21:46:51
|
On Thu, 2011-07-28 at 23:33 +0200, Albert Strasheim wrote: > Warning: Unfortunately, this client request is unreliable and best avoided. > > Is this still the case with Valgrind in SVN? > > If so, is this something that could be fixed relatively easily or are > there dragons here? The svn doc still contains the same warning. But the doc also tells: "Use this if you're using a user-level thread package and are noticing spurious errors from Valgrind about uninitialized memory reads." I am not sure an valgrind SEGV matches this. Do you see the same crash with other tools ? (e.g. callgrind/cachegrind/massif/...) ? |
|
From: Albert S. <fu...@gm...> - 2011-07-28 21:51:16
|
Hello On Thu, Jul 28, 2011 at 11:46 PM, Philippe Waroquiers <phi...@sk...> wrote: > On Thu, 2011-07-28 at 23:33 +0200, Albert Strasheim wrote: >> Warning: Unfortunately, this client request is unreliable and best avoided. >> Is this still the case with Valgrind in SVN? >> If so, is this something that could be fixed relatively easily or are >> there dragons here? > The svn doc still contains the same warning. > But the doc also tells: > "Use this if you're using a user-level thread package and are noticing > spurious errors from Valgrind about uninitialized memory reads." I'm seeing what I think are spurious errors for the Go binaries that do run to completion under Valgrind. I don't think this will fix the SIGSEGV, > I am not sure an valgrind SEGV matches this. > Do you see the same crash with other tools ? > (e.g. callgrind/cachegrind/massif/...) ? Agreed. The segfault is a separate problem that has to be addressed. Regards Albert |
|
From: Philippe W. <phi...@sk...> - 2011-07-28 22:33:21
|
On Thu, 2011-07-28 at 23:50 +0200, Albert Strasheim wrote:
> Agreed. The segfault is a separate problem that has to be addressed.
You might try to compare how stacktraces are computed between
Valgrind core, Valgrind gdbserver, native gdb (cfr previous mail).
Alternatively, you might experiment by disabling various Valgrind JIT
optimisations e.g. using the options:
--vex-iropt-level=<0..2> [2]
--vex-iropt-precise-memory-exns=no|yes [no]
--vex-iropt-unroll-thresh=<0..400> [120]
--vex-guest-chase-thresh=<0..99> [10]
I think to disable as much as possible; you have to use respectively
0, yes, 0, 0.
Philippe
|
|
From: Rich C. <rc...@wi...> - 2011-07-28 22:37:32
|
Hi Albert,
I reproduced the crash with src/pgk/reflect.
Here's what is happening to cause this:
- V detects an error (might) occur.
- V starts to capture the stack of the occurance.
- if we assume the stack pointer and stack frames are correct,
as V walks the stack, one of the frames does not conform to ABI
and the previous frame pointer in the current frame points to
non-existing memory.
I have not proved that the frame does not conform to the ABI, there
are many other reasons for the invalid pointer.
Since this happened on frame 7, a work-around is to run V with
--num-callers=6, and the program runs to completion.
Rich
On Thu, 28 Jul 2011 23:19:36 +0200
Albert Strasheim <fu...@gm...> wrote:
> Hello again
>
> On Thu, Jul 28, 2011 at 11:01 PM, Albert Strasheim <fu...@gm...> wrote:
> > Hello
> > On Thu, Jul 28, 2011 at 10:52 PM, Rich Coe <rc...@wi...> wrote:
> >> Hi,
> >> Using go tip:9218 on linux x86_64, I couldn't reproduce this.
> >> What about your environment makes this issue unique ?
> >> Rich
> > You're right! I guess I was accidentally still running 3.6.1 instead
> > of the build from SVN.
> > I guess bug 262916 can be closed now.
> > The next thing to do is to take a closer look at the warnings produced.
>
> I've done some more tests, and most packages work, but the 6.out from
> the reflect package still produces a crash at m_stacktrace.c:334.
>
> I'm definitely running the code from SVN now.
>
> Regards
>
> Albert
--
Rich Coe rc...@wi...
|
|
From: Philippe W. <phi...@sk...> - 2011-07-28 23:12:04
|
On Thu, 2011-07-28 at 17:37 -0500, Rich Coe wrote: > Hi Albert, > > I reproduced the crash with src/pgk/reflect. > Here's what is happening to cause this: > - V detects an error (might) occur. > - V starts to capture the stack of the occurance. > - if we assume the stack pointer and stack frames are correct, > as V walks the stack, one of the frames does not conform to ABI > and the previous frame pointer in the current frame points to > non-existing memory. > > I have not proved that the frame does not conform to the ABI, there > are many other reasons for the invalid pointer. You might use the Valgrind gdbserver to use gdb to do the stacktrace just before the error is reported by Valgrind. If gdb+Valgrind gdbserver can produce a stacktrace of the simulated cpu and Valgrind "core" cannot make a stacktrace with the same simulated cpu state, then it looks like the Valgrind stacktrace logic is to be enhanced/corrected. If both can't make a stacktrace, but the native gdb can do a stacktrace, then that looks more like a Valgrind simulation bug. Philippe |