plt-spy-discuss Mailing List for PLT Spy
Brought to you by:
aphexairlines
You can subscribe to this list here.
2004 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(2) |
Aug
(16) |
Sep
(1) |
Oct
|
Nov
|
Dec
(4) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2005 |
Jan
|
Feb
(1) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
From: Daniel P. de M. e S. <ds...@cc...> - 2005-02-15 19:26:30
|
We don't need to change MzScheme's call to dlopen after all. Putting the following line in Spy's scheme_initialize function still copies the libpython symbols to the global namespace: dlopen("/usr/lib/libpython2.3.so", RTLD_LAZY | RTLD_GLOBAL); Daniel |
From: Christopher A. <ra...@gm...> - 2004-12-19 08:53:29
|
On Sun, 19 Dec 2004 17:06:30 +1100, Christopher Armstrong <ra...@gm...> wrote: > Is Spy embedding python with a fundamentally different mechanism than this? Agh, of course, I forgot that there is yet another indirection. It is not Spy dynlink Python dlopen Pygtk dynlink Python it is Scheme dlopen Spy dynlink Python dlopen Pygtk dynlink Python. :-( I will write a minimal test (both without and with Python) of this immediately to verify that this is what causes the bug. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash |
From: Christopher A. <ra...@gm...> - 2004-12-19 06:06:37
|
On Sat, 18 Dec 2004 16:18:47 +0900, Daniel Pinto de Mello e Silva <ds...@cc...> wrote: > Others have had this problem, so I'm not sure if it's a Spy bug. See these threads first: > > http://mail.python.org/pipermail/python-list/2000-January/019956.html > http://mail.python.org/pipermail/python-list/1999-July/007387.html > http://mail.python.org/pipermail/c++-sig/2003-May/004023.html > http://mail.iptel.org/pipermail/sems/2004-May/000371.html > http://dbforums.com/archive/97/2002/03/4/331168 Still, the difference between the dead-simple python embedded program that imports numarray and spy hasn't been discovered. For example, this works:: --- main.c --- #include <Python.h> int main(int argc, char *argv[]) { Py_Initialize(); PyRun_SimpleString("import numarray\nprint dir(numarray)\n"); Py_Finalize(); return 0; } --- snip --- when compiled with "gcc -lpython2.3 -I/usr/include/python2.3 -omain main.c". Is Spy embedding python with a fundamentally different mechanism than this? > Also look for the section under "Installing With Tomcat" here: > http://jepp.sourceforge.net/ > > His advice seems to work: Good to know, but I'm still going to research the possibility of sane linking :-) -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash |
From: Daniel P. de M. e S. <ds...@cc...> - 2004-12-18 07:18:57
|
On Sat, 2004-12-18 at 10:30 +1100, Christopher Armstrong wrote: > The current embedded version of PLT-Spy uses a weird hack during > building to dynamically link all the installed extension modules, > because if it doesn't, importing (for example) 'gtk' will blow up with > an error about pygtk.so not being able to find a Python API symbol. > > That is, Spy dynlinks python dlopens gtk dynlinks python. For some > reason, when Spy tells Python to import (dlopen) gtk, gtk can't find > python's symbols. Similar things happen with all C extension modules, > not just gtk. > > I tried to write some minimal reproductions of this bug the last few days. > > One, minimal-dl-bug, has program A dynlink to B.so which dlopens C.so > which dynlinks and calls B.so. A.foo calls B.bar calls C.baz calls > B.quux. This is, to my understanding, the situation we have with > PLT-Spy, Python, and GTK, respectively. The program runs without > problem. > > Two, python-dl-bug, has program A dynlink to python2.3.so and import > gtk. This also works without problem. > > So, the only conclusion I can draw from this, assuming my minimal > examples match up with the current situation correctly, is that it's > actually a bug in PLT-Spy. I'm beginning to dive into the embedding > code to find out what the problem is. > > Thoughts? Are my minimal examples misled? Any direction on where to look in Spy? > > Thanks. > Others have had this problem, so I'm not sure if it's a Spy bug. See these threads first: http://mail.python.org/pipermail/python-list/2000-January/019956.html http://mail.python.org/pipermail/python-list/1999-July/007387.html http://mail.python.org/pipermail/c++-sig/2003-May/004023.html http://mail.iptel.org/pipermail/sems/2004-May/000371.html http://dbforums.com/archive/97/2002/03/4/331168 Also look for the section under "Installing With Tomcat" here: http://jepp.sourceforge.net/ His advice seems to work: $ mzscheme --require python.ss Welcome to MzScheme version 299.22, Copyright (c) 2004 PLT Scheme, Inc. [scheme_initialize] Initializing libpython... [scheme_initialize] Initialized libpython. > (py-eval "import numarray") [eval-python] (begin (begin (define rhs1306 (python-load-module/dotted "numarray")) (define numarray rhs1306))) Traceback (most recent call last): File "/usr/lib/python2.3/site-packages/numarray/__init__.py", line 42, in ? from numarrayall import * File "/usr/lib/python2.3/site-packages/numarray/numarrayall.py", line 2, in ? from generic import * File "/usr/lib/python2.3/site-packages/numarray/generic.py", line 11, in ? import math as _math ImportError: /usr/lib/python2.3/lib-dynload/math.so: undefined symbol: PyExc_FloatingPointError $ LD_PRELOAD=/usr/lib/libpython2.3.so.1.0 mzscheme --require python.ss Welcome to MzScheme version 299.22, Copyright (c) 2004 PLT Scheme, Inc. [scheme_initialize] Initializing libpython... [scheme_initialize] Initialized libpython. > (py-eval "import numarray") [eval-python] (begin (begin (define rhs1306 (python-load-module/dotted "numarray")) (define numarray rhs1306))) [spy-import] imported module: <module 'numarray' from '/usr/lib/python2.3/site-packages/numarray/__init__.pyc'> () > Daniel |
From: Christopher A. <ra...@gm...> - 2004-12-17 23:30:10
|
The current embedded version of PLT-Spy uses a weird hack during building to dynamically link all the installed extension modules, because if it doesn't, importing (for example) 'gtk' will blow up with an error about pygtk.so not being able to find a Python API symbol. That is, Spy dynlinks python dlopens gtk dynlinks python. For some reason, when Spy tells Python to import (dlopen) gtk, gtk can't find python's symbols. Similar things happen with all C extension modules, not just gtk. I tried to write some minimal reproductions of this bug the last few days. One, minimal-dl-bug, has program A dynlink to B.so which dlopens C.so which dynlinks and calls B.so. A.foo calls B.bar calls C.baz calls B.quux. This is, to my understanding, the situation we have with PLT-Spy, Python, and GTK, respectively. The program runs without problem. Two, python-dl-bug, has program A dynlink to python2.3.so and import gtk. This also works without problem. So, the only conclusion I can draw from this, assuming my minimal examples match up with the current situation correctly, is that it's actually a bug in PLT-Spy. I'm beginning to dive into the embedding code to find out what the problem is. Thoughts? Are my minimal examples misled? Any direction on where to look in Spy? Thanks. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash |
From: Daniel P. de M. e S. <ds...@cc...> - 2004-09-10 14:31:37
|
Ok, so this Python function: def gen(lst): for x in lst: yield x is this in Scheme: (define (gen lst) (mk-it (lambda (set-k!) (let/ec return (for-each (lambda (x) (let/cc k (set-k! k) (return x))) lst) (raise "stop iteration"))))) where mk-it is defined here: (require (lib "class.ss")) (define iterator% (class object% (init-field initial-k) (define k initial-k) (define set-k! (lambda (c) (set! k c))) (define/public (next) (k set-k!)) (define/public (reset) (set! k initial-k)) (super-new))) (define (mk-it f) (new iterator% (initial-k f))) so: > (define g (gen '(1 2 3 4))) > g #<struct:object:iterator%> > (send g next) 1 > (send g next) 2 > (send g next) 3 > (send g reset) > (send g next) 1 > (send g next) 2 > (send g next) 3 > (send g next) 4 > (send g next) uncaught exception: "stop iteration" |
From: Daniel P. de M. e S. <ds...@cc...> - 2004-08-09 17:44:40
|
On Mon, 2004-08-09 at 23:48, Christopher Armstrong wrote: > On Mon, 09 Aug 2004 19:25:10 +0900, Daniel Pinto de Mello e Silva > <ds...@cc...> wrote: > > Do you know how to declare/define a libpython type as a subtype of > > another one in C? > > No, I don't remember :-). There's a "xxsubtype.c" or something like > that somewhere in Python's source, as an example. > > > If you remember the problem we had with module objects, they have a > > PyModule_GetDict function that should return a dictionary. We need to > > implement our own kind of module dictionary (wrapping a scheme > > namespace), so we have to make a subtype of PyDict_Type. Do you > > remember how to do that? > > Making a subtype of PyDict isn't going to help - the problem is that a > lot of code calls PyDict_* functions on this thing returned from > PyModule_GetDict, meaning any overriding we do in __setitem__ or > whatever will *not* be honored -- it'll pass through straight to the > low-level dict calls. What we had decided to do before (a while ago on > AIM, I think) was to modify the PyDict_* implementations to do a > PyDict_CheckExact(self), and if it fails, fall back on PyObject_* (and > avoid infinite loops somehow), which does the check for overridden > magic-methods. > > And when we do that, there won't be any point in subtyping PyDict -- > we can just make our own type that implements (the C-equivalent of) > __setitem__, __getitem__, and so on. Subtyping would only be necessary > if we do actually want to share implementation. Ok, I modified dictobject.c's PyDict_SetItem to call PySchemeNS_SetItem when the obj is a PySchemeNS obj, and the same for GetItem and DelItem. Now I'm trying to splice together the scheme-based module importing code and the libpython module importing code. Ughhh. This is probably my least favorite part. How much do you think should be done in C versus Scheme? libpython's find_module (from libpython's import.c, renamed spymod-import.c in Spy) fails to "find" a module sitting in the current directory. Maybe the fact that initsite() failed during init (and then I disabled it) has something to do with it... Daniel |
From: Christopher A. <ra...@gm...> - 2004-08-09 14:48:48
|
On Mon, 09 Aug 2004 19:25:10 +0900, Daniel Pinto de Mello e Silva <ds...@cc...> wrote: > Do you know how to declare/define a libpython type as a subtype of > another one in C? No, I don't remember :-). There's a "xxsubtype.c" or something like that somewhere in Python's source, as an example. > If you remember the problem we had with module objects, they have a > PyModule_GetDict function that should return a dictionary. We need to > implement our own kind of module dictionary (wrapping a scheme > namespace), so we have to make a subtype of PyDict_Type. Do you > remember how to do that? Making a subtype of PyDict isn't going to help - the problem is that a lot of code calls PyDict_* functions on this thing returned from PyModule_GetDict, meaning any overriding we do in __setitem__ or whatever will *not* be honored -- it'll pass through straight to the low-level dict calls. What we had decided to do before (a while ago on AIM, I think) was to modify the PyDict_* implementations to do a PyDict_CheckExact(self), and if it fails, fall back on PyObject_* (and avoid infinite loops somehow), which does the check for overridden magic-methods. And when we do that, there won't be any point in subtyping PyDict -- we can just make our own type that implements (the C-equivalent of) __setitem__, __getitem__, and so on. Subtyping would only be necessary if we do actually want to share implementation. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com |
From: Daniel P. de M. e S. <ds...@cc...> - 2004-08-09 10:25:06
|
Do you know how to declare/define a libpython type as a subtype of another one in C? If you remember the problem we had with module objects, they have a PyModule_GetDict function that should return a dictionary. We need to implement our own kind of module dictionary (wrapping a scheme namespace), so we have to make a subtype of PyDict_Type. Do you remember how to do that? Daniel |
From: Matthew F. <mf...@cs...> - 2004-08-09 00:38:12
|
At Sun, 8 Aug 2004 14:58:22 -0400, Eli Barzilay wrote: > Anyway, the GC thing is subtle. The thing is that usually you don't > collect objects that have a pointer to them -- but if you keep a > pointer to an offset within them then the GC wouldn't see it. (This > is the 3m precise GC, I don't know what Bohm does.) The Boehm GC supports "interior pointers" by default, but there's a compile-time flag to disable it, and it's disabled in MzScheme. My impression is that disabling interior pointers makes a significant performance difference. > The only real solution I can think of is to make pointers offset-able. > Either add a new kind of pointer that has an (integer) offset value, > or extend the current pointer object with an offset value. I'm CCing > Matthew so he can say if this sounds like a reasonable solution. I think I've missed too much context. It certainly makes sense to have a record that combines a pointer to the start of a GCable object and an offset into the object. (One of the GCs uses this strategy to track "weak reference", where the reference that's weak can be in the middle of a GCable object.) Matthew |
From: Eli B. <el...@ba...> - 2004-08-08 22:30:53
|
On Aug 9, Daniel Pinto de Mello e Silva wrote: > I like the idea of a field pointer. So in the case of: > > struct PyString { > PyTypeObject* type; > int size; > char* buf; > }; > > would you write in scheme: > > (get-ffi-obj "PyObject_ToPyString" pylib > (_fun _py-obj-ptr -> (_pointer-field (_py-obj-type _int) _string))) > > where (_py-obj-type _int) is a list of the fields before the one you > care about? (so that the ffi lib can calculate the offset) I'm thinking of something more direct, something like a simple ptr-offset function (or maybe a ptr-offset! which will be convenient for loops where you don't want to allocate a new pointer object in each iteration). For the offset amount there is foreign-sizeof. In any case, I don't think that this is relevant in this case -- the Scheme pointer will have a pointer into the python object which is a bad idea. In other words, a pointer-with-offset is something you can have, but slapping that into a function like make-sized-byte-string will create a Scheme wrapper that has the actual pointer value. I think that there are offset versions of the make-sized-... functions that can be used for this, but the main problem is still Scheme code and Pythone code that might have different assumptions on what should be there (especially if one of them mutates something). > What if you want both the size and the char buf? > > (get-ffi-obj "PyObject_ToPyString" pylib > (_fun _py-obj-ptr -> (ret: _pointer) > -> (pystr->string ret)) > > (define (pystr->string pystr) > (libmzscheme:scheme-make-sized-string > (ffi:c->scheme ret > (_pointer-field (_pointer) _int)) > (ffi:c->scheme ret > (_pointer-field (_pointer _int) _pointer)))) > > Or is that too roundabout? Same as above. I think it's best to keep the struct as is, and only create a string object if you need one. Won't it be better to use a Scheme cstruct? -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://www.barzilay.org/ Maze is Life! |
From: Daniel P. de M. e S. <ds...@cc...> - 2004-08-08 19:24:26
|
On Mon, 2004-08-09 at 03:58, Eli Barzilay wrote: > On Aug 8, Christopher Armstrong wrote: > > On Mon, 09 Aug 2004 03:03:04 +0900, Daniel Pinto de Mello e Silva > > <ds...@cc...> wrote: > > > > > > Eli's ffi is powerful stuff. Can python's ctypes library do that? > > > > (Ccing Eli since he might be interested) > > > > Yeah. Actually, from what Eli says, it seems that ctypes has it > > easier than his library, since Python's GC doesn't work on such a > > low level (or something). You can get the offset of a struct member > > and create a pointer to the member without worrying about the GC > > molesting that pointer. > > Aha, so this is why the simultaneous ffi discussions on two seemingly > different fronts... > > Anyway, the GC thing is subtle. The thing is that usually you don't > collect objects that have a pointer to them -- but if you keep a > pointer to an offset within them then the GC wouldn't see it. (This > is the 3m precise GC, I don't know what Bohm does.) More than that -- > it'll make it think that there is an object where there isn't one. > With 3m, it is possible to make the GC use these things, but it needs > to be allocated in a different page, so the whole things becomes > heavier. > > I think that a better solution is to have new kind of pointer -- a > GC-invisible one: different from the current one only in its mark > procedure which will go over the tag object but not over the pointer > itself. So Scheme programmers can use it as long as they're holding a > reference to the real object so it's not collected. BUT... this is > not a solution -- the moving collect beats all dirty hack -- the > problem is that if the GC doesn't see it, then it won't modify it if > there was a collection, which means that your offset pointer is no > pointing at the remains of the old object and using it will have no > effect at best, and burn your computer at worst. > > The only real solution I can think of is to make pointers offset-able. > Either add a new kind of pointer that has an (integer) offset value, > or extend the current pointer object with an offset value. I'm CCing > Matthew so he can say if this sounds like a reasonable solution. I like the idea of a field pointer. So in the case of: struct PyString { PyTypeObject* type; int size; char* buf; }; would you write in scheme: (get-ffi-obj "PyObject_ToPyString" pylib (_fun _py-obj-ptr -> (_pointer-field (_py-obj-type _int) _string))) where (_py-obj-type _int) is a list of the fields before the one you care about? (so that the ffi lib can calculate the offset) What if you want both the size and the char buf? (get-ffi-obj "PyObject_ToPyString" pylib (_fun _py-obj-ptr -> (ret: _pointer) -> (pystr->string ret)) (define (pystr->string pystr) (libmzscheme:scheme-make-sized-string (ffi:c->scheme ret (_pointer-field (_pointer) _int)) (ffi:c->scheme ret (_pointer-field (_pointer _int) _pointer)))) Or is that too roundabout? Daniel |
From: Eli B. <el...@ba...> - 2004-08-08 19:18:39
|
On Aug 9, Daniel Pinto de Mello e Silva wrote: > I don't understand. If a struct has two ints: > > struct S { int x; int y; }; > > and you have a pointer to the y of an S instance: > > int* ptr = some_s->y; > > where the address of the instance object is 32, the address of the y > is 36 (32 + offset 4), and the object is later moved to address 512 > by a GC cycle, how does the GC know to fix the 36 pointer to 516? I > thought it would only look at pointers to object addresses (the 32 > in this case). If you store the 36 address in a GC-able pointer, then it will assume that there is an object there and Bad Stuff might happen. If you don't tell it that your ptr is GC-able, it wouldn't consider it as a pointer, and as a result it won't update it so Bad Stuff might happen if you use ptr later. My suggestion is something like: void *ptr = some_s; int ofs = (int)(((void*)some_s) - ((void*)(&(some_s->y)))); which solves the pointer problem. -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://www.barzilay.org/ Maze is Life! |
From: Daniel P. de M. e S. <ds...@cc...> - 2004-08-08 19:01:59
|
On Mon, 2004-08-09 at 03:43, Christopher Armstrong wrote: > On Mon, 09 Aug 2004 03:03:04 +0900, Daniel Pinto de Mello e Silva > <ds...@cc...> wrote: > > On Mon, 2004-08-09 at 02:24, Christopher Armstrong wrote: > > > Oh! Never mind: > > > > > > ... [py-string->stringp "PyString_AsString" > > > (_fun _py-string-pointer -> _pointer)] ... > > > > > > (define (py-string->string py-str) > > > (make-sized-byte-string (py-string->stringp py-str) > > > (py-var-object-size (ptr-ref py-str _py-var-object)))) > > > > > > *then* it prints #"f\0oo"! :-D > > > > Eli's ffi is powerful stuff. Can python's ctypes library do that? > > (Ccing Eli since he might be interested) > > Yeah. Actually, from what Eli says, it seems that ctypes has it easier > than his library, since Python's GC doesn't work on such a low level > (or something). You can get the offset of a struct member and create a > pointer to the member without worrying about the GC molesting that > pointer. I don't understand. If a struct has two ints: struct S { int x; int y; }; and you have a pointer to the y of an S instance: int* ptr = some_s->y; where the address of the instance object is 32, the address of the y is 36 (32 + offset 4), and the object is later moved to address 512 by a GC cycle, how does the GC know to fix the 36 pointer to 516? I thought it would only look at pointers to object addresses (the 32 in this case). Daniel |
From: Eli B. <el...@ba...> - 2004-08-08 18:58:39
|
On Aug 8, Christopher Armstrong wrote: > On Mon, 09 Aug 2004 03:03:04 +0900, Daniel Pinto de Mello e Silva > <ds...@cc...> wrote: > > > > Eli's ffi is powerful stuff. Can python's ctypes library do that? > > (Ccing Eli since he might be interested) > > Yeah. Actually, from what Eli says, it seems that ctypes has it > easier than his library, since Python's GC doesn't work on such a > low level (or something). You can get the offset of a struct member > and create a pointer to the member without worrying about the GC > molesting that pointer. Aha, so this is why the simultaneous ffi discussions on two seemingly different fronts... Anyway, the GC thing is subtle. The thing is that usually you don't collect objects that have a pointer to them -- but if you keep a pointer to an offset within them then the GC wouldn't see it. (This is the 3m precise GC, I don't know what Bohm does.) More than that -- it'll make it think that there is an object where there isn't one. With 3m, it is possible to make the GC use these things, but it needs to be allocated in a different page, so the whole things becomes heavier. I think that a better solution is to have new kind of pointer -- a GC-invisible one: different from the current one only in its mark procedure which will go over the tag object but not over the pointer itself. So Scheme programmers can use it as long as they're holding a reference to the real object so it's not collected. BUT... this is not a solution -- the moving collect beats all dirty hack -- the problem is that if the GC doesn't see it, then it won't modify it if there was a collection, which means that your offset pointer is no pointing at the remains of the old object and using it will have no effect at best, and burn your computer at worst. The only real solution I can think of is to make pointers offset-able. Either add a new kind of pointer that has an (integer) offset value, or extend the current pointer object with an offset value. I'm CCing Matthew so he can say if this sounds like a reasonable solution. -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://www.barzilay.org/ Maze is Life! |
From: Christopher A. <ra...@gm...> - 2004-08-08 18:43:30
|
On Mon, 09 Aug 2004 03:03:04 +0900, Daniel Pinto de Mello e Silva <ds...@cc...> wrote: > On Mon, 2004-08-09 at 02:24, Christopher Armstrong wrote: > > Oh! Never mind: > > > > ... [py-string->stringp "PyString_AsString" > > (_fun _py-string-pointer -> _pointer)] ... > > > > (define (py-string->string py-str) > > (make-sized-byte-string (py-string->stringp py-str) > > (py-var-object-size (ptr-ref py-str _py-var-object)))) > > > > *then* it prints #"f\0oo"! :-D > > Eli's ffi is powerful stuff. Can python's ctypes library do that? (Ccing Eli since he might be interested) Yeah. Actually, from what Eli says, it seems that ctypes has it easier than his library, since Python's GC doesn't work on such a low level (or something). You can get the offset of a struct member and create a pointer to the member without worrying about the GC molesting that pointer. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com |
From: Daniel P. de M. e S. <ds...@cc...> - 2004-08-08 18:28:36
|
On Mon, 2004-08-09 at 02:24, Christopher Armstrong wrote: > On Mon, 09 Aug 2004 02:00:05 +0900, Daniel Pinto de Mello e Silva > <ds...@cc...> wrote: > > On Mon, 2004-08-09 at 00:45, Christopher Armstrong wrote: > > > Anyway, I'm trying to get Eli to fix some of the bigger problems in > > > his FFI now. Currently there's no sensible way to extract the string > > > from PyString (or any struct {int size; char val[1];}) since there's > > > no way to get a pointer to a member of a struct, citing some strange > > > GC issues that are unfamiliar to me. Once I get him to iron out all > > > the kinks I'll look at wrapping more of libpython and start patching > > > Spy to use it. > > > > Why not use PyString_AsString? > > > > (define pystr->string > > (get-ffi-obj "PyString_AsString" spylib > > '(_fun _py-string-pointer -> _string))) > > Oh, duh, I should have thought of that. But, hmm, that's problematic: > _string is definitely inappropriate. > > (define a-py-string > (string-and-size->pystring "f\0oo" 4)) > > (printf "string: ~s\n" > (py-string->string a-py-string)) > > expectedly, prints out string: "f". And I can't use (_bytes o 4), > either, since '4' needs to be calculated per-struct-instance.... I'll > bug Eli some more (I hope he doesn't get annoyed with all the messages > I've been sending him ;-) > > Oh! Never mind: > > ... [py-string->stringp "PyString_AsString" > (_fun _py-string-pointer -> _pointer)] ... > > (define (py-string->string py-str) > (make-sized-byte-string (py-string->stringp py-str) > (py-var-object-size (ptr-ref py-str _py-var-object)))) > > *then* it prints #"f\0oo"! :-D Eli's ffi is powerful stuff. Can python's ctypes library do that? Daniel |
From: Christopher A. <ra...@gm...> - 2004-08-08 17:24:11
|
On Mon, 09 Aug 2004 02:00:05 +0900, Daniel Pinto de Mello e Silva <ds...@cc...> wrote: > On Mon, 2004-08-09 at 00:45, Christopher Armstrong wrote: > > Anyway, I'm trying to get Eli to fix some of the bigger problems in > > his FFI now. Currently there's no sensible way to extract the string > > from PyString (or any struct {int size; char val[1];}) since there's > > no way to get a pointer to a member of a struct, citing some strange > > GC issues that are unfamiliar to me. Once I get him to iron out all > > the kinks I'll look at wrapping more of libpython and start patching > > Spy to use it. > > Why not use PyString_AsString? > > (define pystr->string > (get-ffi-obj "PyString_AsString" spylib > '(_fun _py-string-pointer -> _string))) Oh, duh, I should have thought of that. But, hmm, that's problematic: _string is definitely inappropriate. (define a-py-string (string-and-size->pystring "f\0oo" 4)) (printf "string: ~s\n" (py-string->string a-py-string)) expectedly, prints out string: "f". And I can't use (_bytes o 4), either, since '4' needs to be calculated per-struct-instance.... I'll bug Eli some more (I hope he doesn't get annoyed with all the messages I've been sending him ;-) Oh! Never mind: ... [py-string->stringp "PyString_AsString" (_fun _py-string-pointer -> _pointer)] ... (define (py-string->string py-str) (make-sized-byte-string (py-string->stringp py-str) (py-var-object-size (ptr-ref py-str _py-var-object)))) *then* it prints #"f\0oo"! :-D > > BTW, what's a better way to do that "Forgive me, O Lord" bit? :-) > > With a macro! Thanks :-) -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com |
From: Daniel P. de M. e S. <ds...@cc...> - 2004-08-08 17:00:04
|
On Mon, 2004-08-09 at 00:45, Christopher Armstrong wrote: > On Sun, 08 Aug 2004 22:34:48 +0900, Daniel Pinto de Mello e Silva > <ds...@cc...> wrote: > > Nice! That raises a question though: should Spy script libpython or > > should it be a modified libpython? > > I terribly wish it didn't need to modify libpython, but remember those > problems with PyDict_*? AFAICT, if you want to use the fancy PLT > namespaces instead of regular Python dicts for python namespaces, > we're going to have to modify *some* of the library; but this FFI > should totally obsolete all that explicit wrapping for scheme that's > necessary in Spy now. Hmm, and I think there are other things > necessary to reimplement, like the GC functions and the > evaluation/import functions and .... > Yep, Spy still needs its own module objects, eval, and memory allocation. I'm thinking about letting what's left of libpython handle importing though. Our approach is the same as theirs: keep a table of modules already loaded and refuse to load ones already there. Might as well use the same code to do that. > Well, unless you can think of another way to go about it that I'm missing. Yeah, probably not a good idea to just embed cpython. > Anyway, I'm trying to get Eli to fix some of the bigger problems in > his FFI now. Currently there's no sensible way to extract the string > from PyString (or any struct {int size; char val[1];}) since there's > no way to get a pointer to a member of a struct, citing some strange > GC issues that are unfamiliar to me. Once I get him to iron out all > the kinks I'll look at wrapping more of libpython and start patching > Spy to use it. Why not use PyString_AsString? (define pystr->string (get-ffi-obj "PyString_AsString" spylib '(_fun _py-string-pointer -> _string))) > BTW, what's a better way to do that "Forgive me, O Lord" bit? :-) With a macro! (define-syntax (define-ffi-objs stx) (syntax-case stx () [(define-ffi-objs lib [id c-name type] ...) (syntax (define-values (id ...) (values (get-ffi-obj c-name lib type) ...)))])) (define-ffi-objs pylib [py-initialize "Py_Initialize" (_fun -> _void)] [py-finalize "Py_Finalize" (_fun -> _void)] [string-and-size->pystring "PyString_FromStringAndSize" (_fun _string _int -> _py-string-pointer)] [py-run-string "PyRun_SimpleString" (_fun _string -> _void)]) or: (define-syntax (define-ffi-objs stx) (syntax-case stx () [(define-ffi-objs lib [id c-name type] ...) (syntax (begin (define id (get-ffi-obj c-name lib type)) ...))])) if you prefer "define" instead of "define-values". Daniel |
From: Christopher A. <ra...@gm...> - 2004-08-08 15:45:54
|
On Sun, 08 Aug 2004 22:34:48 +0900, Daniel Pinto de Mello e Silva <ds...@cc...> wrote: > Nice! That raises a question though: should Spy script libpython or > should it be a modified libpython? I terribly wish it didn't need to modify libpython, but remember those problems with PyDict_*? AFAICT, if you want to use the fancy PLT namespaces instead of regular Python dicts for python namespaces, we're going to have to modify *some* of the library; but this FFI should totally obsolete all that explicit wrapping for scheme that's necessary in Spy now. Hmm, and I think there are other things necessary to reimplement, like the GC functions and the evaluation/import functions and .... Well, unless you can think of another way to go about it that I'm missing. Anyway, I'm trying to get Eli to fix some of the bigger problems in his FFI now. Currently there's no sensible way to extract the string from PyString (or any struct {int size; char val[1];}) since there's no way to get a pointer to a member of a struct, citing some strange GC issues that are unfamiliar to me. Once I get him to iron out all the kinks I'll look at wrapping more of libpython and start patching Spy to use it. BTW, what's a better way to do that "Forgive me, O Lord" bit? :-) -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com |
From: Daniel P. de M. e S. <ds...@cc...> - 2004-08-08 13:34:50
|
Nice! That raises a question though: should Spy script libpython or should it be a modified libpython? Daniel On Sun, 2004-08-08 at 03:38, Christopher Armstrong wrote: > Check this out. I've been bugging Eli to explain/improve his FFI > interface -- it's really awesome so far. > > > #!/bin/sh > #| > exec mzscheme -r "$0" "$@" > |# > > (require (lib "foreign.ss")) > > ;; Data structure definitions > (define-cstruct _py-object > ([refcount _int] > [type _pointer])) > > (define-cstruct _py-var-object > ([py-object _py-object] > [size _int])) > > (define-cstruct _py-string > ([py-var-object _py-var-object] > [hash _long] > [state _int] > [*val _int8])) > > ;; API function imports > > (define pylib (ffi-lib "libpython2.3")) > > (define imported-functions > '([py-initialize "Py_Initialize" > (_fun -> _void)] > [py-finalize "Py_Finalize" > (_fun -> _void)] > [string-and-size->pystring "PyString_FromStringAndSize" > (_fun _string _int -> _py-string-pointer)] > [py-run-string "PyRun_SimpleString" > (_fun _string -> _void)] > )) > > ;; Forgive me, O Lord, for I know not what I do > (map (lambda (triplet) > (eval `(define ,(car triplet) > (get-ffi-obj ,(cadr triplet) pylib ,(caddr triplet))))) > imported-functions) > > > ;; Higher-level wrappers > > ;; This doesn't work yet, as I have no way to get a pointer to a > ;; struct member. > ;(define (py-string-val py-str) > ; (make-sized-byte-string > ; (py-string-*val py-str) > ; (size-offset (ptr-ref py-str _py-var-object)))) > > > > ;; Main > > (py-initialize) > > (py-run-string "print 'hello, world!'") > > (define a-py-string > (ptr-ref (string-and-size->pystring "foo" 3) _py-string)) > > (printf "ref: ~s size: ~s string [first char as int :-(]: ~s\n" > (py-object-refcount (ptr-ref a-py-string _py-object)) > (py-var-object-size (ptr-ref a-py-string _py-var-object)) > (py-string-*val (ptr-ref a-py-string _py-string)) > ) > > (py-finalize) > |
From: Christopher A. <ra...@gm...> - 2004-08-07 18:38:25
|
Check this out. I've been bugging Eli to explain/improve his FFI interface -- it's really awesome so far. #!/bin/sh #| exec mzscheme -r "$0" "$@" |# (require (lib "foreign.ss")) ;; Data structure definitions (define-cstruct _py-object ([refcount _int] [type _pointer])) (define-cstruct _py-var-object ([py-object _py-object] [size _int])) (define-cstruct _py-string ([py-var-object _py-var-object] [hash _long] [state _int] [*val _int8])) ;; API function imports (define pylib (ffi-lib "libpython2.3")) (define imported-functions '([py-initialize "Py_Initialize" (_fun -> _void)] [py-finalize "Py_Finalize" (_fun -> _void)] [string-and-size->pystring "PyString_FromStringAndSize" (_fun _string _int -> _py-string-pointer)] [py-run-string "PyRun_SimpleString" (_fun _string -> _void)] )) ;; Forgive me, O Lord, for I know not what I do (map (lambda (triplet) (eval `(define ,(car triplet) (get-ffi-obj ,(cadr triplet) pylib ,(caddr triplet))))) imported-functions) ;; Higher-level wrappers ;; This doesn't work yet, as I have no way to get a pointer to a ;; struct member. ;(define (py-string-val py-str) ; (make-sized-byte-string ; (py-string-*val py-str) ; (size-offset (ptr-ref py-str _py-var-object)))) ;; Main (py-initialize) (py-run-string "print 'hello, world!'") (define a-py-string (ptr-ref (string-and-size->pystring "foo" 3) _py-string)) (printf "ref: ~s size: ~s string [first char as int :-(]: ~s\n" (py-object-refcount (ptr-ref a-py-string _py-object)) (py-var-object-size (ptr-ref a-py-string _py-var-object)) (py-string-*val (ptr-ref a-py-string _py-string)) ) (py-finalize) -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com |
From: Daniel P. de M. e S. <ds...@cc...> - 2004-07-15 12:39:23
|
On Thu, 2004-07-15 at 03:43, Christopher Armstrong wrote: > Hey Daniel, I saw you made some commits to Spy :-). Has anything > happened with that project someone was working on to get a better FFI > into PLT, I wonder? Yep, Eli Barzilay provided a nice interface to libffi that will be bundled with the v299 release. > Unfortunately I probably won't be able to touch Spy for another little > while, as I'm moving to Australia for a new job shortly. Once I get > settled in there, though, I have a few projects that I'd like to work on > that would have Spy as a prerequisite, so I may get some more time in on it. No problem, I've been slow at it too. You're welcome to work on it whenever you're ready. I'm trying to get all the cpython C files linked (Objects/, Modules/, and parts of Python/). I'll let you know how it goes. Daniel |
From: Christopher A. <ra...@tw...> - 2004-07-14 18:43:49
|
Hey Daniel, I saw you made some commits to Spy :-). Has anything happened with that project someone was working on to get a better FFI into PLT, I wonder? Unfortunately I probably won't be able to touch Spy for another little while, as I'm moving to Australia for a new job shortly. Once I get settled in there, though, I have a few projects that I'd like to work on that would have Spy as a prerequisite, so I may get some more time in on it. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/ |