Thread: [Rpy] More seg faults (Rpy 2.2.0beta3)

Hello,

I experienced seg faults and core dumps related to rpy with several
recent versions of rpy, and reported some of them on this list. Here's
another one, this time with rpy 2.2.0beta3 (and python 2.7.1)
occurring during one of the tests coming with the source code. After
unpacking,

   cd rpy2-2.2.0beta3/rpy/rinterface/tests
   python test_SexpVector.py

gives

   testAssignItemComplex (__main__.SexpVectorTestCase) ... ok
   testAssignItemDifferentType (__main__.SexpVectorTestCase) ... ok

      [...several more tests that pass ok...]

   testNACharacterInVector (__main__.NAValuesTestCase) ... ok
   testNACharacterRepr (__main__.NAValuesTestCase) ... ok
   testNACharactertoR (__main__.NAValuesTestCase) ... ok
   testNAIntegerBinaryfunc (__main__.NAValuesTestCase) ... Segmentation fault

in my setup which is admittedly somewhat special (Intel 11.1 compilers
used for compiling python and all modules, for example, on an OpenSuse
11.1 Linux).

I have played with that particular test script; when replacing the
test suite setup at the end of the test script with

   def suite():
    suite = unittest.TestLoader().loadTestsFromTestCase(ByteSexpVectorTestCase)
    return suite

I get another odd behaviour:

   >python test_SexpVector.py

   testInitFromBytes (__main__.ByteSexpVectorTestCase) ... ok
   testInitFromSeqInvalidByte (__main__.ByteSexpVectorTestCase) ... ok
   testInitFromSeqOfBytes (__main__.ByteSexpVectorTestCase) ... ok

   ----------------------------------------------------------------------
   Ran 3 tests in 0.002s

   OK
   Segmentation fault

i.e. a seg fault upon exit (something I had seen before in a different
context; see http://sourceforge.net/mailarchive/forum.php?thread_name=4D9CDE71.1070705%40gmail.com&forum_name=rpy-list).
Before switching to 2.2.0beta3, we had been using 2.2.0alpha2 without
producing core dumps;but when updating to a newer version of netcdf
and Jeff Whitaker's the netCDF4 module, a rather large application of
ours suddenly started to produce segmentation faults in the middle of
nowhere (meaning that the crash occurred reproducibly in the middle of
some calculation entirely unrelated to rpy; the program just had the
rpy module loaded). Removing the import of rpy modules made it working
as before. When importing rpy2.2.0beta3 it core dumps at the same
place, but runs absolutely fine if rpy2.robjects is not imported.

Going through the mailing list archive and the bug tracker, I see that
there are different reports of similar (= segmentation fault) behavior
on different platforms which are all very difficult to reproduce, but
still bother people. Is there a way to debug rpy2 to get an idea what
might go wrong during import and shortly thereafter to track this
down? From my experience I would not be surprised if the above
behavior cannot be easily reproduced by others - and yet rpy2 seems to
produce crashes for quite some people.

Sorry - my intention is not to complain. I just feel terribly
frustrated that I cannot even provide the slightest idea where that
problem comes from, and even less how to drill it down further.

   Christian.

On Fri, May 13, 2011 at 03:49:49PM +0000, Christian Marquardt wrote:
> Hello,
> 
> I experienced seg faults and core dumps related to rpy with several
> recent versions of rpy, and reported some of them on this list. Here's
> another one, this time with rpy 2.2.0beta3 (and python 2.7.1)
> occurring during one of the tests coming with the source code. After
> unpacking,

[...]
> in my setup which is admittedly somewhat special (Intel 11.1 compilers
> used for compiling python and all modules, for example, on an OpenSuse
> 11.1 Linux).

[...]

> Sorry - my intention is not to complain. I just feel terribly
> frustrated that I cannot even provide the slightest idea where that
> problem comes from, and even less how to drill it down further.

>From my experience segv with different compilers or architectures
expose problems with bad memory management (which can be not the case
in our situation). Actually, I find it very reassuring that there
are people who use rpy on different architectures or using different
compilers. Hopefully, it will be made solid like a rock soon.

Anyway, how does the backtrace look like in your case?

On mine (on Fedora 14 using Python 3.2) while using attached script (t.py)

    (gdb) set args t.py
    (gdb) r
    Starting program: /home/wrobell/opt/bin/python3 t.py
    [Thread debugging using libthread_db enabled]
    Detaching after fork from child process 8221.
    Detaching after fork from child process 8222.
    end

    Program received signal SIGSEGV, Segmentation fault.
    0x00000070006d0063 in ?? ()
    (gdb) bt
    #0  0x00000070006d0063 in ?? ()
    #1  0x000000000051e447 in insertdict (mp=0xadc580, key=0xa58570, 
        hash=953885456351248617, value=0x7959c0) at Objects/dictobject.c:538
    #2  0x00000000005200ce in PyDict_SetItem (op=0xadc580, 
        key=<value optimized out>, value=0x7959c0) at Objects/dictobject.c:810
    #3  0x0000000000524a72 in _PyModule_Clear (m=<value optimized out>)
        at Objects/moduleobject.c:297
    #4  0x0000000000477e5a in PyImport_Cleanup () at Python/import.c:520
    #5  0x000000000048553e in Py_Finalize () at Python/pythonrun.c:430
    #6  0x000000000049cc53 in Py_Main (argc=<value optimized out>, 
        argv=<value optimized out>) at Modules/main.c:711
    #7  0x0000000000415551 in main (argc=2, argv=0x7fffffffe6a8)
        at ./Modules/python.c:59
    (gdb) x/i 0x00000070006d0063
    => 0x70006d0063:	Cannot access memory at address 0x70006d0063

Best regards,

Artur

I tried your popen script before, but it's running fine - no core dump. I'll might try to compile a debuggable python over the weekend...

   Chris. 




On 13 May 2011, at 18:09, Artur Wroblewski <wr...@pl...> wrote:

> On Fri, May 13, 2011 at 03:49:49PM +0000, Christian Marquardt wrote:
>> Hello,
>> 
>> I experienced seg faults and core dumps related to rpy with several
>> recent versions of rpy, and reported some of them on this list. Here's
>> another one, this time with rpy 2.2.0beta3 (and python 2.7.1)
>> occurring during one of the tests coming with the source code. After
>> unpacking,
> 
> [...]
>> in my setup which is admittedly somewhat special (Intel 11.1 compilers
>> used for compiling python and all modules, for example, on an OpenSuse
>> 11.1 Linux).
> 
> [...]
> 
>> Sorry - my intention is not to complain. I just feel terribly
>> frustrated that I cannot even provide the slightest idea where that
>> problem comes from, and even less how to drill it down further.
> 
>> From my experience segv with different compilers or architectures
> expose problems with bad memory management (which can be not the case
> in our situation). Actually, I find it very reassuring that there
> are people who use rpy on different architectures or using different
> compilers. Hopefully, it will be made solid like a rock soon.
> 
> Anyway, how does the backtrace look like in your case?
> 
> On mine (on Fedora 14 using Python 3.2) while using attached script (t.py)
> 
>    (gdb) set args t.py
>    (gdb) r
>    Starting program: /home/wrobell/opt/bin/python3 t.py
>    [Thread debugging using libthread_db enabled]
>    Detaching after fork from child process 8221.
>    Detaching after fork from child process 8222.
>    end
> 
>    Program received signal SIGSEGV, Segmentation fault.
>    0x00000070006d0063 in ?? ()
>    (gdb) bt
>    #0  0x00000070006d0063 in ?? ()
>    #1  0x000000000051e447 in insertdict (mp=0xadc580, key=0xa58570, 
>        hash=953885456351248617, value=0x7959c0) at Objects/dictobject.c:538
>    #2  0x00000000005200ce in PyDict_SetItem (op=0xadc580, 
>        key=<value optimized out>, value=0x7959c0) at Objects/dictobject.c:810
>    #3  0x0000000000524a72 in _PyModule_Clear (m=<value optimized out>)
>        at Objects/moduleobject.c:297
>    #4  0x0000000000477e5a in PyImport_Cleanup () at Python/import.c:520
>    #5  0x000000000048553e in Py_Finalize () at Python/pythonrun.c:430
>    #6  0x000000000049cc53 in Py_Main (argc=<value optimized out>, 
>        argv=<value optimized out>) at Modules/main.c:711
>    #7  0x0000000000415551 in main (argc=2, argv=0x7fffffffe6a8)
>        at ./Modules/python.c:59
>    (gdb) x/i 0x00000070006d0063
>    => 0x70006d0063:    Cannot access memory at address 0x70006d0063
> 
> Best regards,
> 
> Artur
> 
> <t.py>
> ------------------------------------------------------------------------------
> Achieve unprecedented app performance and reliability
> What every C/C++ and Fortran developer should know.
> Learn how Intel has extended the reach of its next-generation tools
> to help boost performance applications - inlcuding clusters.
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list

On Fri, May 13, 2011 at 7:01 PM, Christian Marquardt
<mar...@gm...> wrote:
> I tried your popen script before, but it's running fine - no core dump. I'll might try to compile a debuggable python over the weekend...

I have just discovered that compiling Python
with "--with-wide-unicode" helps in my os.popen
test case.

What options are used for "configure" script
in your case?

[...]

Regards,

Artur

Ah, Unicode. I believe the builds from python.org are narrow unicode (i.e.
UTF-16), whereas those shipped in many Linux distros (including the one I
use) are wide unicode (UTF-32). This might explain why I can't replicate
your segfaults.

Thomas

On 18 May 2011 23:56, Artur Wroblewski <wr...@pl...> wrote:

> On Fri, May 13, 2011 at 7:01 PM, Christian Marquardt
> <mar...@gm...> wrote:
> > I tried your popen script before, but it's running fine - no core dump.
> I'll might try to compile a debuggable python over the weekend...
>
> I have just discovered that compiling Python
> with "--with-wide-unicode" helps in my os.popen
> test case.
>
> What options are used for "configure" script
> in your case?
>
> [...]
>
> Regards,
>
> Artur
>
>
> ------------------------------------------------------------------------------
> What Every C/C++ and Fortran developer Should Know!
> Read this article and learn how Intel has extended the reach of its
> next-generation tools to help Windows* and Linux* C/C++ and Fortran
> developers boost performance applications - including clusters.
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list
>

On Thu, May 19, 2011 at 12:19 AM, Thomas Kluyver <ta...@gm...> wrote:
> Ah, Unicode. I believe the builds from python.org are narrow unicode (i.e.
> UTF-16), whereas those shipped in many Linux distros (including the one I
> use) are wide unicode (UTF-32). This might explain why I can't replicate
> your segfaults.

The question is what is the real reason of segv. :)

Best regards,

Artur

On 19 May 2011 08:19, Artur Wroblewski <wr...@pl...> wrote:

> On Thu, May 19, 2011 at 12:19 AM, Thomas Kluyver <ta...@gm...> wrote:
> > Ah, Unicode. I believe the builds from python.org are narrow unicode
> (i.e.
> > UTF-16), whereas those shipped in many Linux distros (including the one I
> > use) are wide unicode (UTF-32). This might explain why I can't replicate
> > your segfaults.
>
> The question is what is the real reason of segv. :)
>

At a guess: if C code in rpy2 is somehow assuming that PyUnicode objects are
stored as 4-bytes per character, then when it tries to access one that's
stored as only 2 bytes per character, it would try to use a block of memory
twice the size of the object, which may take it outside the memory it owns
(i.e. a segfault).

Why that would be happening on import, or interacting with os.popen, I don't
know. But if we can only trigger the segfaults on narrow unicode Python
builds, we've got a place to start looking, at least.

Anyone else who's tried to replicate this: can you get a segfault, and is
your Python wide- or narrow-unicode? If you're not sure about the latter,
check sys.maxunicode - it should be 65535 for narrow, or 1114111 for wide.

Thomas

Hmm,

I've been building our python (2.7.1) with --enable-unicode=ucs4 which I believe stands for 4-byte Unicode. In any case, Rpy handles (should handle?) the Unicode conversion through python C-API calls which should behave consistently with the way the original python has been build. 

As I said before, our python is not build with FCC, but with Intel's C/C++ compilers, and I can't reproduce the popen() segmentation fault, but get one at a different place. So I agree with Artur: what is the real reason of the seg fault(s) that come up for various people on various platforms with various compilers?

   Chris. 

On 19 May 2011, at 11:22, Thomas Kluyver <ta...@gm...> wrote:

> On 19 May 2011 08:19, Artur Wroblewski <wr...@pl...> wrote:
> On Thu, May 19, 2011 at 12:19 AM, Thomas Kluyver <ta...@gm...> wrote:
> > Ah, Unicode. I believe the builds from python.org are narrow unicode (i.e.
> > UTF-16), whereas those shipped in many Linux distros (including the one I
> > use) are wide unicode (UTF-32). This might explain why I can't replicate
> > your segfaults.
> 
> The question is what is the real reason of segv. :)
> 
> At a guess: if C code in rpy2 is somehow assuming that PyUnicode objects are stored as 4-bytes per character, then when it tries to access one that's stored as only 2 bytes per character, it would try to use a block of memory twice the size of the object, which may take it outside the memory it owns (i.e. a segfault).
> 
> Why that would be happening on import, or interacting with os.popen, I don't know. But if we can only trigger the segfaults on narrow unicode Python builds, we've got a place to start looking, at least.
> 
> Anyone else who's tried to replicate this: can you get a segfault, and is your Python wide- or narrow-unicode? If you're not sure about the latter, check sys.maxunicode - it should be 65535 for narrow, or 1114111 for wide.
> 
> Thomas
> ------------------------------------------------------------------------------
> What Every C/C++ and Fortran developer Should Know!
> Read this article and learn how Intel has extended the reach of its 
> next-generation tools to help Windows* and Linux* C/C++ and Fortran 
> developers boost performance applications - including clusters. 
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list

On Thu, May 19, 2011 at 11:16 AM, Christian Marquardt
<mar...@gm...> wrote:
> Hmm,
> I've been building our python (2.7.1) with --enable-unicode=ucs4 which I
> believe stands for 4-byte Unicode. In any case, Rpy handles (should handle?)
> the Unicode conversion through python C-API calls which should behave
> consistently with the way the original python has been build.
> As I said before, our python is not build with FCC, but with Intel's C/C++
> compilers, and I can't reproduce the popen() segmentation fault, but get one
> at a different place. So I agree with Artur: what is the real reason of the
> seg fault(s) that come up for various people on various platforms with
> various compilers?

Exactly! :)

My os.popen test case was created because my application fails
- the app uses ctypes
- ctypes uses os.popen to determine system and libraries available

While the os.popen does not cause my app to crash with UCS4, still
my application segfaults... just in different place. I will try to narrow it
and provide some stack trace later this evening.

Best regards,

Artur

> On 19 May 2011, at 11:22, Thomas Kluyver <ta...@gm...> wrote:
>
> On 19 May 2011 08:19, Artur Wroblewski <wr...@pl...> wrote:
>>
>> On Thu, May 19, 2011 at 12:19 AM, Thomas Kluyver <ta...@gm...> wrote:
>> > Ah, Unicode. I believe the builds from python.org are narrow unicode
>> > (i.e.
>> > UTF-16), whereas those shipped in many Linux distros (including the one
>> > I
>> > use) are wide unicode (UTF-32). This might explain why I can't replicate
>> > your segfaults.
>>
>> The question is what is the real reason of segv. :)
>
> At a guess: if C code in rpy2 is somehow assuming that PyUnicode objects are
> stored as 4-bytes per character, then when it tries to access one that's
> stored as only 2 bytes per character, it would try to use a block of memory
> twice the size of the object, which may take it outside the memory it owns
> (i.e. a segfault).
>
> Why that would be happening on import, or interacting with os.popen, I don't
> know. But if we can only trigger the segfaults on narrow unicode Python
> builds, we've got a place to start looking, at least.
>
> Anyone else who's tried to replicate this: can you get a segfault, and is
> your Python wide- or narrow-unicode? If you're not sure about the latter,
> check sys.maxunicode - it should be 65535 for narrow, or 1114111 for wide.
>
> Thomas
>
> ------------------------------------------------------------------------------
> What Every C/C++ and Fortran developer Should Know!
> Read this article and learn how Intel has extended the reach of its
> next-generation tools to help Windows* and Linux* C/C++ and Fortran
> developers boost performance applications - including clusters.
> http://p.sf.net/sfu/intel-dev2devmay
>
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list
>
> ------------------------------------------------------------------------------
> What Every C/C++ and Fortran developer Should Know!
> Read this article and learn how Intel has extended the reach of its
> next-generation tools to help Windows* and Linux* C/C++ and Fortran
> developers boost performance applications - including clusters.
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list
>
>

On Thu, May 19, 2011 at 12:46 PM, Artur Wroblewski
<wr...@pl...> wrote:
[...]
> While the os.popen does not cause my app to crash with UCS4, still
> my application segfaults... just in different place. I will try to narrow it
> and provide some stack trace later this evening.

At the moment the segfault can be reproduced as follows (using UCS4
based build)

1. Start Python interactive shell.
2. Run

    >>> import rpy2.robjects as ro
    >>> ro.NA_Real

Sometimes it will segv on import. If not then always when accessing NA_Real.
I wrote "accessing" as it fails on the following code too

    >>> x = ro.FloatVector([ro.NA_Real, 1, 0, ro.NA_Real])

It will not segfault if you put above code into a script and run it.


gdb session:

$ gdb python3
GNU gdb (GDB) PLD Linux (7.2-1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pld-linux".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/users/wrobell/opt/bin/python3...done.
(gdb) r
Starting program: /home/users/wrobell/opt/bin/python3
[Thread debugging using libthread_db enabled]
Python 3.2 (r32:88445, May 18 2011, 23:54:16)
[GCC 4.5.3 20110515 (release)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpy2.robjects as ro
Detaching after fork from child process 25710.
>>> ro.NA_Real

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6d62908 in ?? ()
(gdb) bt
#0  0x00007ffff6d62908 in ?? ()
#1  0x00007ffff7a47ac8 in PyObject_Repr (v=0x870a10) at Objects/object.c:384
#2  0x00007ffff7a2a989 in PyFile_WriteObject (v=0x870a10, f=<value
optimized out>, flags=<value optimized out>) at
Objects/fileobject.c:145
#3  0x00007ffff7ad3bd7 in sys_displayhook (self=<value optimized out>,
o=0x870a10) at ./Python/sysmodule.c:158
#4  0x00007ffff7a09979 in PyObject_Call (func=0x7ffff6d7a200,
arg=<value optimized out>, kw=<value optimized out>) at
Objects/abstract.c:2149
#5  0x00007ffff7aa0327 in PyEval_CallObjectWithKeywords
(func=0x7ffff6d7a200, arg=0x7791d0, kw=<value optimized out>) at
Python/ceval.c:3755
#6  0x00007ffff7aa4d03 in PyEval_EvalFrameEx (f=<value optimized out>,
throwflag=<value optimized out>) at Python/ceval.c:1832
#7  0x00007ffff7aa7fe6 in PyEval_EvalCodeEx (_co=<value optimized
out>, globals=<value optimized out>, locals=<value optimized out>,
    args=<value optimized out>, argcount=0, kws=0x0, kwcount=0,
defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3311
#8  0x00007ffff7aa823b in PyEval_EvalCode (co=<value optimized out>,
globals=<value optimized out>, locals=<value optimized out>)
    at Python/ceval.c:761
#9  0x00007ffff7acd466 in run_mod (fp=<value optimized out>,
filename=0x7ffff7b3d9f7 "<stdin>", flags=0x7fffffffe660) at
Python/pythonrun.c:1760
#10 PyRun_InteractiveOneFlags (fp=<value optimized out>,
filename=0x7ffff7b3d9f7 "<stdin>", flags=0x7fffffffe660) at
Python/pythonrun.c:1148
#11 0x00007ffff7acd70e in PyRun_InteractiveLoopFlags
(fp=0x7ffff71086c0, filename=0x7ffff7b3d9f7 "<stdin>",
flags=0x7fffffffe660)
    at Python/pythonrun.c:1038
#12 0x00007ffff7acdc1c in PyRun_AnyFileExFlags (fp=0x7ffff71086c0,
filename=0x7ffff7b3d9f7 "<stdin>", closeit=0, flags=0x7fffffffe660)
    at Python/pythonrun.c:1007
#13 0x00007ffff7ae30a1 in run_file (argc=<value optimized out>,
argv=<value optimized out>) at Modules/main.c:297
#14 Py_Main (argc=<value optimized out>, argv=<value optimized out>)
at Modules/main.c:692
#15 0x0000000000400aa9 in main (argc=1, argv=0x7fffffffe7d8) at
./Modules/python.c:59
(gdb)


Best regards,

Artur

On 19 May 2011 19:36, Artur Wroblewski <wr...@pl...> wrote:

> At the moment the segfault can be reproduced as follows (using UCS4
> based build)
>

OK, interesting. I still can't replicate it, in either Python 2.7 or 3.2
wide-unicode builds, with 2.2.0 beta 3.

Thomas

On Thu, May 19, 2011 at 8:32 PM, Thomas Kluyver <ta...@gm...> wrote:
> On 19 May 2011 19:36, Artur Wroblewski <wr...@pl...> wrote:
>>
>> At the moment the segfault can be reproduced as follows (using UCS4
>> based build)
>
> OK, interesting. I still can't replicate it, in either Python 2.7 or 3.2
> wide-unicode builds, with 2.2.0 beta 3.

What version of gcc, binutils and glibc are you using?

BTW. Interesting is, that every time (be it os.popen or now) it is
about NA_Real. For example, there has been no problem with
NA_Integer.

Best regards,

Artur

On 19 May 2011 20:38, Artur Wroblewski <wr...@pl...> wrote:

> What version of gcc, binutils and glibc are you using?
>

gcc 4.5.2
binutils 2.21.0
glibc 2.13

I don't compile Python myself, though - I let Ubuntu's build servers take
care of that.


> BTW. Interesting is, that every time (be it os.popen or now) it is
> about NA_Real. For example, there has been no problem with
> NA_Integer.
>

Interesting. In 2.7, NA_Real appears as "0.0", and in 3.2 as "NA_Real_".

Thomas

On 2011-05-19 22:00, Thomas Kluyver wrote:
> On 19 May 2011 20:38, Artur Wroblewski <wr...@pl... 
> <mailto:wr...@pl...>> wrote:
>
>     What version of gcc, binutils and glibc are you using?
>
>
> gcc 4.5.2
> binutils 2.21.0
> glibc 2.13
>
> I don't compile Python myself, though - I let Ubuntu's build servers 
> take care of that.
>
>     BTW. Interesting is, that every time (be it os.popen or now) it is
>     about NA_Real. For example, there has been no problem with
>     NA_Integer.
>
>
> Interesting. In 2.7, NA_Real appears as "0.0", and in 3.2 as "NA_Real_".

Not here:

 >>> import rpy2.rinterface as ri
 >>> ri.initr()
0
 >>> ri.NA_Real
NA_real_
 >>> import sys
 >>> sys.version
'2.7.1+ (r271:86832, Mar 24 2011, 00:37:39) \n[GCC 4.5.2]'

>
> Thomas
>
>
> ------------------------------------------------------------------------------
> What Every C/C++ and Fortran developer Should Know!
> Read this article and learn how Intel has extended the reach of its
> next-generation tools to help Windows* and Linux* C/C++ and Fortran
> developers boost performance applications - including clusters.
> http://p.sf.net/sfu/intel-dev2devmay
>
>
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list

On 19 May 2011 21:08, Laurent Gautier <lga...@gm...> wrote:

> >>> ri.NA_Real
> NA_real_
> >>> import sys
> >>> sys.version
> '2.7.1+ (r271:86832, Mar 24 2011, 00:37:39) \n[GCC 4.5.2]'
>

OK, that's interesting:

>>> ri.NA_Real
0.0
>>> sys.version
'2.7.1+ (r271:86832, Apr 11 2011, 18:05:24) \n[GCC 4.5.2]'

The value of NA_Real can be displayed as 0.0, or as a very small float (e-41
or smaller), which can change on different runs.

It also gives me another problem: after rpy2.rinterface (or rpy2.robjects)
has been imported, doing any more imports fails with a SyntaxError. It's not
100% consistent, but I can reproduce it quite a few times in a row (closing
and relaunching the interactive interpreter). This only seems to be the case
in 2.7 - I can't reproduce it in 3.2.

>>> import rpy2.robjects
>>> import sys
  File "<stdin>", line 1
    import sys
             ^
SyntaxError: invalid syntax

Thomas

On 2011-05-19 22:27, Thomas Kluyver wrote:
> On 19 May 2011 21:08, Laurent Gautier <lga...@gm... 
> <mailto:lga...@gm...>> wrote:
>
>     >>> ri.NA_Real
>     NA_real_
>     >>> import sys
>     >>> sys.version
>     '2.7.1+ (r271:86832, Mar 24 2011, 00:37:39) \n[GCC 4.5.2]'
>
>
> OK, that's interesting:
>
> >>> ri.NA_Real
> 0.0
> >>> sys.version
> '2.7.1+ (r271:86832, Apr 11 2011, 18:05:24) \n[GCC 4.5.2]'
>
> The value of NA_Real can be displayed as 0.0, or as a very small float 
> (e-41 or smaller), which can change on different runs.

Well, technically there is a numerical value behind NA_Real; it is an 
"NA" by convention.
The annoying part is that this is changing from run to run: this is 
suggesting uninitialized memory.

Does
 >>> import ri
 >>> ri.R_VERSION_BUILD
correspond to the R version you are running rpy2 against ?

Do you have older versions of rpy2 installed ? If yes, does removing 
them solve the problem ?

>
> It also gives me another problem: after rpy2.rinterface (or 
> rpy2.robjects) has been imported, doing any more imports fails with a 
> SyntaxError. It's not 100% consistent, but I can reproduce it quite a 
> few times in a row (closing and relaunching the interactive 
> interpreter). This only seems to be the case in 2.7 - I can't 
> reproduce it in 3.2.
>
> >>> import rpy2.robjects
> >>> import sys
>   File "<stdin>", line 1
>     import sys
>              ^
> SyntaxError: invalid syntax
>
> Thomas

On 19 May 2011 22:00, Laurent Gautier <lga...@gm...> wrote:

>
> Does
> >>> import ri
> >>> ri.R_VERSION_BUILD
> correspond to the R version you are running rpy2 against ?
>

Yes: 2.12.1, svn rev 53855


> Do you have older versions of rpy2 installed ? If yes, does removing them
> solve the problem ?


I did have rpy2 2.1.9 installed from the repositories. Removing it doesn't
seem to change anything.

Extra info: before I do ri.initr(), ri.NA_Real is 0.0. Once I've done
initr(), ri.NA_Real appears as a small float. Additionally, once it's not
0.0, it changes between successive evaluations. There appears to be a
repeating pattern: [a, a, b, b, nan], where a and b are different small
floats and nan is the special 'not a number' float value. a changes between
successive runs of the interactive shell, whereas b is consistent, at least
so far. But I guess there's not much to be gleaned from that.

Thomas

I'm in no way knowledgeable about python (or R) internals, but for me this smells like Rpy putting some parts of it's initialisation into memory areas which are also accessed by other parts of the python interpreter. And that probably shouldn't happen...



On 19 May 2011, at 23:18, Thomas Kluyver <ta...@gm...> wrote:

> On 19 May 2011 22:00, Laurent Gautier <lga...@gm...> wrote:
> 
> Does
> >>> import ri
> >>> ri.R_VERSION_BUILD
> correspond to the R version you are running rpy2 against ?
> 
> Yes: 2.12.1, svn rev 53855
>  
> Do you have older versions of rpy2 installed ? If yes, does removing them solve the problem ?
> 
> I did have rpy2 2.1.9 installed from the repositories. Removing it doesn't seem to change anything.
> 
> Extra info: before I do ri.initr(), ri.NA_Real is 0.0. Once I've done initr(), ri.NA_Real appears as a small float. Additionally, once it's not 0.0, it changes between successive evaluations. There appears to be a repeating pattern: [a, a, b, b, nan], where a and b are different small floats and nan is the special 'not a number' float value. a changes between successive runs of the interactive shell, whereas b is consistent, at least so far. But I guess there's not much to be gleaned from that.
> 
> Thomas
>  
> 
> ------------------------------------------------------------------------------
> What Every C/C++ and Fortran developer Should Know!
> Read this article and learn how Intel has extended the reach of its 
> next-generation tools to help Windows* and Linux* C/C++ and Fortran 
> developers boost performance applications - including clusters. 
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list

On 2011-05-19 23:18, Thomas Kluyver wrote:
> On 19 May 2011 22:00, Laurent Gautier <lga...@gm... 
> <mailto:lga...@gm...>> wrote:
>
>
>     Does
>     >>> import ri
>     >>> ri.R_VERSION_BUILD
>     correspond to the R version you are running rpy2 against ?
>
>
> Yes: 2.12.1, svn rev 53855
>
>     Do you have older versions of rpy2 installed ? If yes, does
>     removing them solve the problem ?
>
>
> I did have rpy2 2.1.9 installed from the repositories. Removing it 
> doesn't seem to change anything.
>
> Extra info: before I do ri.initr(), ri.NA_Real is 0.0. Once I've done 
> initr(), ri.NA_Real appears as a small float. Additionally, once it's 
> not 0.0, it changes between successive evaluations. There appears to 
> be a repeating pattern: [a, a, b, b, nan], where a and b are different 
> small floats and nan is the special 'not a number' float value. a 
> changes between successive runs of the interactive shell, whereas b is 
> consistent, at least so far. But I guess there's not much to be 
> gleaned from that.

Mmmmh... the problem seems to be rooted in what I would believe to be an 
oddity with R.
After a quick look, it appears that while the symbol NA_REAL is defined 
in headers (as an alias of R_NaReal) it is only initialized when the R 
engine is initialized.

void attribute_hidden InitArithmetic()
{
     R_NaInt = INT_MIN;
     R_NaN = 0.0/R_Zero_Hack;
     R_NaReal = R_ValueOfNA();
     R_PosInf = 1.0/R_Zero_Hack;
     R_NegInf = -1.0/R_Zero_Hack;
}

This is a problem since rpy2's rinterface is obviously initialized 
before the embedded R is (rpy2.initr() must be called to initialize R).
I am not certain of why this is done dynamically in R at initialization 
time; it seems that this could be a static declaration.

static double R_ValueOfNA(void)
{
     /* The gcc shipping with RedHat 9 gets this wrong without
      * the volatile declaration. Thanks to Marc Schwartz. */
     volatile ieee_double x;
     x.word[hw] = 0x7ff00000;
     x.word[lw] = 1954;
     return x.value;
}

Something is already puzzling: why was this sometimes working ?
Moving the specification of the numerical value stored in rpy2's NA_Real 
to something resolved at compile-time, now I have issue with Python 
mixing up methods from the class "NAReal_Type" and the class "float". 
There is something tricky going on, that might also be in Python itself 
(as it seems that I am venturing in lesser explored corners of Python - 
I have not found an other project defining a subtype for PyFloat_Type at 
the C level).

This is quite annoying, and certainly a release blocker until resolved.


L.

>
> Thomas
>

On Fri, May 20, 2011 at 2:16 PM, Laurent Gautier <lga...@gm...> wrote:
[...]
> This is a problem since rpy2's rinterface is obviously initialized before
> the embedded R is (rpy2.initr() must be called to initialize R).
[...]
> Something is already puzzling: why was this sometimes working ?

Maybe because, under some conditions, sequence.c is initializing
NAReal with NAReal_New as new one? Just guessing as I still do
not understand the semantics of NA*_New functions and associated
RPY_NA_NEW macro - why NAReal_New(1) is called once again
in sequence.c... shouldn't be there just one NA_Real or do
I misunderstand the meaing of "new" parameter?

[...]

Best regards,

Artur

On 2011-05-20 15:55, Artur Wroblewski wrote:
> On Fri, May 20, 2011 at 2:16 PM, Laurent Gautier<lga...@gm...>  wrote:
> [...]
>> This is a problem since rpy2's rinterface is obviously initialized before
>> the embedded R is (rpy2.initr() must be called to initialize R).
> [...]
>> Something is already puzzling: why was this sometimes working ?
> Maybe because, under some conditions, sequence.c is initializing
> NAReal with NAReal_New as new one?

I mean working with some people, not with others, and in my case it 
stops working when I hack in explicitly the numerical value R_NaReal 
takes after initialization.
Otherwise the NA* classes are singletons; PyObject *self is initialized 
only once.

> Just guessing as I still do
> not understand the semantics of NA*_New functions and associated
> RPY_NA_NEW macro - why NAReal_New(1) is called once again
> in sequence.c... shouldn't be there just one NA_Real or do
> I misunderstand the meaing of "new" parameter?

The parameter "new" indicates whether this is a new reference (reference 
count incremented by one) or a borrowed reference.
The macro ensure that the same strategy is applied to all NA* (yet only 
NAReal is causing trouble). Reading the Python source (floatobject.c) 
suggests that there might be dark corner when creating subtypes of float.


Best,

Laurent
> [...]
>
> Best regards,
>
> Artur
>
> ------------------------------------------------------------------------------
> What Every C/C++ and Fortran developer Should Know!
> Read this article and learn how Intel has extended the reach of its
> next-generation tools to help Windows* and Linux* C/C++ and Fortran
> developers boost performance applications - including clusters.
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list

On Sat, May 21, 2011 at 12:35 PM, Laurent Gautier <lga...@gm...> wrote:
[...]
> Out of curiosity, do people experiencing problems have 32-bit or 64-bit OS ?

i have two x86 64-bit systems (PLD Linux and Fedora 14) on which
I get segfaults.

Out of curiosity I have just checked it with quite old ppc 32-bit system.
No segfaults there (python 3.2, gcc 4.5.1, glibc 2.10).

Regards,

w

On 2011-05-21 14:40, Artur Wroblewski wrote:
> On Sat, May 21, 2011 at 12:35 PM, Laurent Gautier<lga...@gm...>  wrote:
> [...]
>> Out of curiosity, do people experiencing problems have 32-bit or 64-bit OS ?
> i have two x86 64-bit systems (PLD Linux and Fedora 14) on which
> I get segfaults.
>
> Out of curiosity I have just checked it with quite old ppc 32-bit system.
> No segfaults there (python 3.2, gcc 4.5.1, glibc 2.10).

I forget to mention that I only manage to reproduce the segfault on 
Python 2.7 (2.7.1, I did not try on 2.6). Python 3.2 seems to be working 
(well, no crash although the unitests reports errors for things that 
appear to be working when run interactively).

> Regards,
>
> w
>
> ------------------------------------------------------------------------------
> What Every C/C++ and Fortran developer Should Know!
> Read this article and learn how Intel has extended the reach of its
> next-generation tools to help Windows* and Linux* C/C++ and Fortran
> developers boost performance applications - including clusters.
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list

Please use the bug tracker on bitbucket.
Otherwise there is a great chance this falls between the cracks.

On 2011-05-13 17:49, Christian Marquardt wrote:
> Hello,
>
> I experienced seg faults and core dumps related to rpy with several
> recent versions of rpy, and reported some of them on this list. Here's
> another one, this time with rpy 2.2.0beta3 (and python 2.7.1)
> occurring during one of the tests coming with the source code. After
> unpacking,
>
>     cd rpy2-2.2.0beta3/rpy/rinterface/tests
>     python test_SexpVector.py
>
> gives
>
>     testAssignItemComplex (__main__.SexpVectorTestCase) ... ok
>     testAssignItemDifferentType (__main__.SexpVectorTestCase) ... ok
>
>        [...several more tests that pass ok...]
>
>     testNACharacterInVector (__main__.NAValuesTestCase) ... ok
>     testNACharacterRepr (__main__.NAValuesTestCase) ... ok
>     testNACharactertoR (__main__.NAValuesTestCase) ... ok
>     testNAIntegerBinaryfunc (__main__.NAValuesTestCase) ... Segmentation fault
>
> in my setup which is admittedly somewhat special (Intel 11.1 compilers
> used for compiling python and all modules, for example, on an OpenSuse
> 11.1 Linux).
>
> I have played with that particular test script; when replacing the
> test suite setup at the end of the test script with
>
>     def suite():
>      suite = unittest.TestLoader().loadTestsFromTestCase(ByteSexpVectorTestCase)
>      return suite
>
> I get another odd behaviour:
>
>     >python test_SexpVector.py
>
>     testInitFromBytes (__main__.ByteSexpVectorTestCase) ... ok
>     testInitFromSeqInvalidByte (__main__.ByteSexpVectorTestCase) ... ok
>     testInitFromSeqOfBytes (__main__.ByteSexpVectorTestCase) ... ok
>
>     ----------------------------------------------------------------------
>     Ran 3 tests in 0.002s
>
>     OK
>     Segmentation fault
>
> i.e. a seg fault upon exit (something I had seen before in a different
> context; see http://sourceforge.net/mailarchive/forum.php?thread_name=4D9CDE71.1070705%40gmail.com&forum_name=rpy-list).
> Before switching to 2.2.0beta3, we had been using 2.2.0alpha2 without
> producing core dumps;but when updating to a newer version of netcdf
> and Jeff Whitaker's the netCDF4 module, a rather large application of
> ours suddenly started to produce segmentation faults in the middle of
> nowhere (meaning that the crash occurred reproducibly in the middle of
> some calculation entirely unrelated to rpy; the program just had the
> rpy module loaded). Removing the import of rpy modules made it working
> as before. When importing rpy2.2.0beta3 it core dumps at the same
> place, but runs absolutely fine if rpy2.robjects is not imported.
>
> Going through the mailing list archive and the bug tracker, I see that
> there are different reports of similar (= segmentation fault) behavior
> on different platforms which are all very difficult to reproduce, but
> still bother people. Is there a way to debug rpy2 to get an idea what
> might go wrong during import and shortly thereafter to track this
> down? From my experience I would not be surprised if the above
> behavior cannot be easily reproduced by others - and yet rpy2 seems to
> produce crashes for quite some people.
>
> Sorry - my intention is not to complain. I just feel terribly
> frustrated that I cannot even provide the slightest idea where that
> problem comes from, and even less how to drill it down further.
>
>     Christian.
>
> ------------------------------------------------------------------------------
> Achieve unprecedented app performance and reliability
> What every C/C++ and Fortran developer should know.
> Learn how Intel has extended the reach of its next-generation tools
> to help boost performance applications - inlcuding clusters.
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list

Done - issue #74

   Christian. 

On 14 May 2011, at 08:37, Laurent Gautier <lga...@gm...> wrote:

> Please use the bug tracker on bitbucket.
> Otherwise there is a great chance this falls between the cracks.
> 
> On 2011-05-13 17:49, Christian Marquardt wrote:
>> Hello,
>> 
>> I experienced seg faults and core dumps related to rpy with several
>> recent versions of rpy, and reported some of them on this list. Here's
>> another one, this time with rpy 2.2.0beta3 (and python 2.7.1)
>> occurring during one of the tests coming with the source code. After
>> unpacking,
>> 
>>    cd rpy2-2.2.0beta3/rpy/rinterface/tests
>>    python test_SexpVector.py
>> 
>> gives
>> 
>>    testAssignItemComplex (__main__.SexpVectorTestCase) ... ok
>>    testAssignItemDifferentType (__main__.SexpVectorTestCase) ... ok
>> 
>>       [...several more tests that pass ok...]
>> 
>>    testNACharacterInVector (__main__.NAValuesTestCase) ... ok
>>    testNACharacterRepr (__main__.NAValuesTestCase) ... ok
>>    testNACharactertoR (__main__.NAValuesTestCase) ... ok
>>    testNAIntegerBinaryfunc (__main__.NAValuesTestCase) ... Segmentation fault
>> 
>> in my setup which is admittedly somewhat special (Intel 11.1 compilers
>> used for compiling python and all modules, for example, on an OpenSuse
>> 11.1 Linux).
>> 
>> I have played with that particular test script; when replacing the
>> test suite setup at the end of the test script with
>> 
>>    def suite():
>>     suite = unittest.TestLoader().loadTestsFromTestCase(ByteSexpVectorTestCase)
>>     return suite
>> 
>> I get another odd behaviour:
>> 
>>> python test_SexpVector.py
>> 
>>    testInitFromBytes (__main__.ByteSexpVectorTestCase) ... ok
>>    testInitFromSeqInvalidByte (__main__.ByteSexpVectorTestCase) ... ok
>>    testInitFromSeqOfBytes (__main__.ByteSexpVectorTestCase) ... ok
>> 
>>    ----------------------------------------------------------------------
>>    Ran 3 tests in 0.002s
>> 
>>    OK
>>    Segmentation fault
>> 
>> i.e. a seg fault upon exit (something I had seen before in a different
>> context; see http://sourceforge.net/mailarchive/forum.php?thread_name=4D9CDE71.1070705%40gmail.com&forum_name=rpy-list).
>> Before switching to 2.2.0beta3, we had been using 2.2.0alpha2 without
>> producing core dumps;but when updating to a newer version of netcdf
>> and Jeff Whitaker's the netCDF4 module, a rather large application of
>> ours suddenly started to produce segmentation faults in the middle of
>> nowhere (meaning that the crash occurred reproducibly in the middle of
>> some calculation entirely unrelated to rpy; the program just had the
>> rpy module loaded). Removing the import of rpy modules made it working
>> as before. When importing rpy2.2.0beta3 it core dumps at the same
>> place, but runs absolutely fine if rpy2.robjects is not imported.
>> 
>> Going through the mailing list archive and the bug tracker, I see that
>> there are different reports of similar (= segmentation fault) behavior
>> on different platforms which are all very difficult to reproduce, but
>> still bother people. Is there a way to debug rpy2 to get an idea what
>> might go wrong during import and shortly thereafter to track this
>> down? From my experience I would not be surprised if the above
>> behavior cannot be easily reproduced by others - and yet rpy2 seems to
>> produce crashes for quite some people.
>> 
>> Sorry - my intention is not to complain. I just feel terribly
>> frustrated that I cannot even provide the slightest idea where that
>> problem comes from, and even less how to drill it down further.
>> 
>>    Christian.
>> 
>> ------------------------------------------------------------------------------
>> Achieve unprecedented app performance and reliability
>> What every C/C++ and Fortran developer should know.
>> Learn how Intel has extended the reach of its next-generation tools
>> to help boost performance applications - inlcuding clusters.
>> http://p.sf.net/sfu/intel-dev2devmay
>> _______________________________________________
>> rpy-list mailing list
>> rpy...@li...
>> https://lists.sourceforge.net/lists/listinfo/rpy-list
> 
> 
> ------------------------------------------------------------------------------
> Achieve unprecedented app performance and reliability
> What every C/C++ and Fortran developer should know.
> Learn how Intel has extended the reach of its next-generation tools
> to help boost performance applications - inlcuding clusters.
> http://p.sf.net/sfu/intel-dev2devmay
> _______________________________________________
> rpy-list mailing list
> rpy...@li...
> https://lists.sourceforge.net/lists/listinfo/rpy-list