## psyco-devel — Technical or open discussion

You can subscribe to this list here.

 2001 2002 2003 2004 2005 2006 2007 2009 2010 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec (39) Jan (37) Feb (35) Mar (23) Apr (14) May (2) Jun (7) Jul Aug (5) Sep (10) Oct Nov (4) Dec Jan (5) Feb (5) Mar (2) Apr May (7) Jun (8) Jul (16) Aug (8) Sep (15) Oct (4) Nov Dec (3) Jan Feb Mar (1) Apr (3) May (2) Jun (12) Jul (8) Aug (13) Sep Oct (9) Nov Dec (1) Jan (2) Feb Mar (6) Apr (1) May Jun Jul (1) Aug Sep Oct Nov Dec Jan (1) Feb (2) Mar (2) Apr (1) May (3) Jun (2) Jul (3) Aug (1) Sep Oct (4) Nov (11) Dec (20) Jan (11) Feb Mar Apr May (1) Jun Jul Aug Sep Oct Nov Dec Jan (1) Feb (1) Mar (5) Apr (4) May (6) Jun Jul (25) Aug Sep (2) Oct Nov Dec Jan Feb Mar Apr (1) May Jun Jul Aug Sep Oct Nov Dec
S M T W T F S

1

2

3
(1)
4
(5)
5
(5)
6

7
(1)
8
(6)
9
(3)
10
(3)
11
(6)
12
(1)
13
(2)
14

15
(2)
16

17
(2)
18

19

20

21

22

23

24

25

26

27

28

29

30

31

Showing results of 37

<< < 1 2 (Page 2 of 2)
 [Psyco-devel] Re: Attempting the 'quick' float solution From: Tim Hochberg - 2002-01-07 18:31:35 ```Following up myself: you can ignore this last message. By trying several different things and mindless copying of pintobject.h I now have rudimentary float support working in psyco. Only add and subtract are working right now, and conversion from ints to floats is not yet supported, but the following silly function runs 4.5 times faster than base python in my version of Psyco, versus 2.5 times faster in CVS psyco. def f10(): z = 0.0 for i in range(100000): z = z - 5.0 y = z + 10. x = y + 10. y = z - 10. x = y + 10. y = z + 10. x = y - 10. y = z + 10. x = y + 10. x = y + 10. y = z - 10. x = y + 10. y = z + 10. x = y + 10. z = z - x return z The next step it to get the other operations working, which should be easy, and get coercion working which I _hope_ will be easy, but I haven't looked into it yet... -tim ----- Original Message ----- From: "Tim Hochberg" To: "Armin Rigo" Cc: Sent: Saturday, January 05, 2002 2:42 PM Subject: Attempting the 'quick' float solution > > Some time ago Armin wrote: > > >[SNIP] let me first > >describe a quick solution that would probably still give a serious > >speed-up to all FP operations. > > I'm attempting to do this on the theory that as I work on the little pieces, > I may absorb enough by osmosis to understand how Psyco actually works. This > may be false, but perhaps I can accomplish something useful anyway. If > someone else is already working on this, let me know and I'll try something > else. I'm sure I'll be asking for guidance repeatedly, but I'll try not to > be too much of a pest... > > These parts are 'done' (they will certainly require some debugging): > > * cimpl_fp_XXX > * PsycoFloat_FROM_DOUBLE() > * PsycoFloat_AS_DOUBLE_1() and PsycoFloat_AS_DOUBLE_2() > > Hmmm.... that list looks awfully short so far ... sigh. Anyway, my question > for today involves ploat_add and friends. Armin suggested: > > >2) meta-implementation: just like we have pint_add()&co, we need > >pfloat_add()&co. A priori, pfloat_add() is something like: > > > >static vinfo_t* pfloat_add(PsycoObject* po, vinfo_t* v, vinfo_t* w) > >{ > > vinfo_t *a1, *a2, *b1, *b2, *x; > > vinfo_array_t* result; > > a1 = PsycoFloat_AS_DOUBLE_1(po, v); > > a2 = PsycoFloat_AS_DOUBLE_2(po, v); > > b1 = PsycoFloat_AS_DOUBLE_1(po, w); > > b2 = PsycoFloat_AS_DOUBLE_2(po, w); > > /* ... */ > >} > > Is there any reason that this can't be done in the same way as pint_add. For > example: > > static vinfo_t* pfloat_add(PsycoObject* po, vinfo_t* v, vinfo_t* w) > { > vinfo_t *a1, *a2, *b1, *b2, *x; > vinfo_array_t* result; > CONVERT_TO_DOUBLE(v, a1, a2); > CONVERT_TO_DOUBLE(w, b1, b2); > /* ... */ > } > > Where CONVERT_TO_DOUBLE would start out as: > > define CONVERT_TO_DOUBLE(vobj, vlng1, vlng2) \ > if (Psyco_TypeSwitch(po, vobj, &psyfs_float) == 0) { \ > PsycoFloat_AS_DOUBLE(po, vobj1, vobj2); \ > if (vlng1 == NULL || vlng2 == NULL) \ > return NULL; \ > } \ > else { \ > if (PycException_Occurred(po)) \ > return NULL; \ > vinfo_incref(psyco_viNotImplemented); \ > return psyco_viNotImplemented; \ > } > > > Eventually it would get extended as we allowed integers and other types to > be converted to floats. psyfs_float would have to be defined in pobject.c > and presumably entered into some array of fixed_switch values somewhere, but > I haven't got that far yet. > > Is this a reasonable way to approach this if my goal is to first get > float-float operations working and then to branch out from there, or am I > way off target here.... > > Thanks, > > -tim > > ```
 [Psyco-devel] Attempting the 'quick' float solution From: Tim Hochberg - 2002-01-05 21:43:02 ```Some time ago Armin wrote: >[SNIP] let me first >describe a quick solution that would probably still give a serious >speed-up to all FP operations. I'm attempting to do this on the theory that as I work on the little pieces, I may absorb enough by osmosis to understand how Psyco actually works. This may be false, but perhaps I can accomplish something useful anyway. If someone else is already working on this, let me know and I'll try something else. I'm sure I'll be asking for guidance repeatedly, but I'll try not to be too much of a pest... These parts are 'done' (they will certainly require some debugging): * cimpl_fp_XXX * PsycoFloat_FROM_DOUBLE() * PsycoFloat_AS_DOUBLE_1() and PsycoFloat_AS_DOUBLE_2() Hmmm.... that list looks awfully short so far ... sigh. Anyway, my question for today involves ploat_add and friends. Armin suggested: >2) meta-implementation: just like we have pint_add()&co, we need >pfloat_add()&co. A priori, pfloat_add() is something like: > >static vinfo_t* pfloat_add(PsycoObject* po, vinfo_t* v, vinfo_t* w) >{ > vinfo_t *a1, *a2, *b1, *b2, *x; > vinfo_array_t* result; > a1 = PsycoFloat_AS_DOUBLE_1(po, v); > a2 = PsycoFloat_AS_DOUBLE_2(po, v); > b1 = PsycoFloat_AS_DOUBLE_1(po, w); > b2 = PsycoFloat_AS_DOUBLE_2(po, w); > /* ... */ >} Is there any reason that this can't be done in the same way as pint_add. For example: static vinfo_t* pfloat_add(PsycoObject* po, vinfo_t* v, vinfo_t* w) { vinfo_t *a1, *a2, *b1, *b2, *x; vinfo_array_t* result; CONVERT_TO_DOUBLE(v, a1, a2); CONVERT_TO_DOUBLE(w, b1, b2); /* ... */ } Where CONVERT_TO_DOUBLE would start out as: define CONVERT_TO_DOUBLE(vobj, vlng1, vlng2) \ if (Psyco_TypeSwitch(po, vobj, &psyfs_float) == 0) { \ PsycoFloat_AS_DOUBLE(po, vobj1, vobj2); \ if (vlng1 == NULL || vlng2 == NULL) \ return NULL; \ } \ else { \ if (PycException_Occurred(po)) \ return NULL; \ vinfo_incref(psyco_viNotImplemented); \ return psyco_viNotImplemented; \ } Eventually it would get extended as we allowed integers and other types to be converted to floats. psyfs_float would have to be defined in pobject.c and presumably entered into some array of fixed_switch values somewhere, but I haven't got that far yet. Is this a reasonable way to approach this if my goal is to first get float-float operations working and then to branch out from there, or am I way off target here.... Thanks, -tim ```
 Re: [Psyco-devel] Bug report From: Petru Paler - 2002-01-05 17:49:47 ```Hello, On Sat, 2002-01-05 at 19:26, Armin Rigo wrote: > Where can we find pybench? http://www.lemburg.com/files/python/ -- Petru Paler, http://www.ppetru.net ```
 Re: [Psyco-devel] Bug report From: Armin Rigo - 2002-01-05 17:44:22 ```Hello Petru, Petru Paler wrote: > I tried to run pybench using Kjetil's new selective compilation function > written in C. Where can we find pybench? > Looks like bad things happen when it tries to compile builtin stuff... I do not think it is related to its being built-in: I guess the 'split' mentionned is not the built-in one, but the one from the standard module 'string'. It might be a coincidence, as 'string.split' seems to compile fine on other examples. Armin. ```
 Re: [Psyco-devel] Re: selective compilation From: Armin Rigo - 2002-01-05 17:44:16 ```Hello Samuele, Samuele Pedroni wrote: > A question: it is because it is ineherently difficult to start compilin= g > from the middle of an execution given psyco approach? Well, it was not meant to be done so. I am not sure it is inherently difficult, but at least it is not trivial: there is quite a lot of info stored in Python-specific structures like frames which would need to be translated into their Psyco equivalent. Maybe this should be discussed in a more general setting in which Psyco could produce incrementally optimized code. Doing so would require us to stop the exeuction at any point and restart it elsewhere, in the newly re-optimized code. Or we could believe that two levels will always be enough: not compiled at all (run by Python) and compiled with all optimizations (only for the innermost loops). There is another thing that I did not discuss up to now, but which could become very meaningful: optimizing data structures. There is little point in optimizing code that loops over all items of a list if all these items are Python objects whose type must be checked at each iteration anyway. So some structures like lists or instance of user-defined classes need to be optimized. For lists this is not quite easy: we cannot just arbitrarily replace lists created by user code by Python objects of another type, e.g. "Psyco-aware list". This would break code: C code (clearly) but also Python code, as in 'if type(x)=3D=3DListType'. Instances are more flexible: we could use Python 2.2's new type system and define a Psyco metaclass. With some tricks, most user-level classes would then use Psyco's metaclass instead of Python's. The interest is that metaclasses control the way instance attributes are stored. This would in theory let us detect that a given attribute always contains an integer, and store directly the integer value instead of a Python integer object in the instance. In both cases, however, we would have much more knowledge about an object if it were built by Psyco-compiled code. Why? Because if, say, a list is populated by Python code, and only later we see it, then we must guess what could be the common points of all objects in the list (e.g. being a integer objects, being tuples of length 2, and so on). On the other hand, if the list is built by Psyco-compiled code, the Psyco compiler has seen the calls to list.append(), and generally knows something about the objects that were passed as arguments to list.append() -- for example, they were all virtual-time integer objects. The same applies to instances: Psyco usually knows something about 'x' when it compiles 'obj.attr =3D x'. With this point of view, Psyco would work much better if *all* the code had been compiled by it. Well, maybe it would be better to see what occurs on real-world scripts if we just compile everything. The problem is rather about memory than speed, I guess, as compiling infrequently-used code might only be a minor time loss. So in summary, I believe that we should postpone the discussion about when Psyco should compile what until serious advances in the core of Psyco or in its Python support files. The neural network example 'bpnn.py' shows that Psyco is not efficient on lists of lists nor instance objects. I propose that we give ourselves as the next goal: make 'bpnn.py' work significantly faster in Psyco than in Python (it is currently significantly slower). Just compiling everything is fine for 'bpnn.py'. Virtualizing float objects as I discussed in a previous e-mail would be a first step. Virtualizing lists and instances is the final goal. Please ask if you would like to help about floats but don't understand what I told in (the first part of) the e-mail about floats. A bient=F4t, Armin. ```
 Re: [Psyco-devel] Re: selective compilation From: Samuele Pedroni - 2002-01-05 12:36:47 ```[Armin Rigo] > > Kjetil Jacobsen wrote: > > the problem with the current approach to selective compilation is that a > > function may be called only once and still account for most of the > > computation time. > > Yes, sure. Well, even with deep messing into Python's internals, I > believe it would be difficult and wrong to try to start compiling a > function in the middle of its execution; so it means that we cannot > handle the case of long functions only executed once. A question: it is because it is ineherently difficult to start compiling from the middle of an execution given psyco approach? From my readings a typical approach to detect the need of compilation for long running methods is to count "looping" (e.g. decrementing a counter associated with the function on every looping (back) branch and triggering compilation when it reaches zero. ) So at least compilation would be naturally triggered at a merge-point, could this help? regards, Samuele Pedroni. ```
 [Psyco-devel] Bug report From: Petru Paler - 2002-01-04 23:41:56 ```I tried to run pybench using Kjetil's new selective compilation function written in C. Using the latest CVS source, I added an "import psyco" line at the top of pybench.py and tried to run it. I get a segfault with debugging disabled, and this: psyco: compiling function split [40][36][60][32][36][36][120]nonnull_refcount: item 2 nonnull_refcount: in array item 5 python: c/vcompiler.c:323: psyco_assert_coherent: Assertion `!err' failed. Aborted with debugging enabled. Looks like bad things happen when it tries to compile builtin stuff... -- Petru Paler, http://www.ppetru.net ```
 Re: [Psyco-devel] Compilation on Windows? From: Petru Paler - 2002-01-04 23:30:13 ```On Fri, 2002-01-04 at 19:01, Tim Hochberg wrote: > I suppose the underscores are meant to signal that the macros aren't really > meant to be used by the unwary. I'll submit a patch to Psyco at sourceforge. I applied the patch to CVS. Armin: I don't have permission to modify the bug status, could you please grant it to me? -- Petru Paler, http://www.ppetru.net ```
 [Psyco-devel] Re: selective compilation From: Armin Rigo - 2002-01-04 17:58:25 ```Hello Kjetil, A few quick notes, more to come... Kjetil Jacobsen wrote: > the problem with the current approach to selective compilation is that a > function may be called only once and still account for most of the > computation time. Yes, sure. Well, even with deep messing into Python's internals, I believe it would be difficult and wrong to try to start compiling a function in the middle of its execution; so it means that we cannot handle the case of long functions only executed once. We still have to work on Psyco before we know if we could let it compile a large part of the code or if it should target only a few crucial functions. > (solution) be to use the execution time instead of the number of invocations as a > metric for determining whether a function should be compiled, since proc0 > in this example is the function having the longest execution time. Yes, that would be a good solution if we have a very fast way of measuring time. More tests needed there. Also note that if we measure the time, this is done at the end, when the function returns; if dynamically rebinding occurs at this moment, it solves the problem you mentionned about functions being regularily executed once even after rebinding, if that rebinding occurs at the beginning. Finally, collecting statistics between runs would be a good idea too, but should be planned only with a Psyco-specific kind of '.pyc' file format that would also store already-compiled code buffers. The latter has some drawbacks: on a Linux box, all .py files from the standard library are stored in directories where users cannot write; where should Psyco write its own optimization files? Unlike regular '.pyc' files, Psyco's will change all the time. Armin ```
 Re: [Psyco-devel] Compilation on Windows? From: Tim Hochberg - 2002-01-04 17:01:49 ```[Tim Hochberg] > > Has anyone successfully compiled Psyco on Windows? I'm plugging along trying > > to get it to compile on Windows 2000 with Python 2.2 final. I've made some > > progress, everything now compiles, albeit with a boatload of warnings about > > truncating ints to chars, but can't link because of an unresolved reference > > to __PyGC_generation0. [Michael Hudson] > IIRC, you want to change an occurence of something like > > PyGC_Object_TRACK > > to something like > > PyGC_Object_Track Thanks! That did the trick. If anyone else is trying this, the specific substitutions occur in pscyo.c and are: _PyObject_GC_TRACK -> PyObject_GC_Track _PyObject_GC_UNTRACK -> PyObject_GC_UnTrack I suppose the underscores are meant to signal that the macros aren't really meant to be used by the unwary. I'll submit a patch to Psyco at sourceforge. > > I'm wondering if the declaration in Python's objimpl.h requires a DL_IMPORT > > declaration, FWIW, this didn't appear to help and I wasn't inclined to pursue it much after I noticed that the macros were marked with a forbidding underscore. -tim ```
 Re: [Psyco-devel] Compilation on Windows? From: Michael Hudson - 2002-01-04 10:54:47 ```On Thu, 3 Jan 2002, Tim Hochberg wrote: > > Has anyone successfully compiled Psyco on Windows? I'm plugging along trying > to get it to compile on Windows 2000 with Python 2.2 final. I've made some > progress, everything now compiles, albeit with a boatload of warnings about > truncating ints to chars, but can't link because of an unresolved reference > to __PyGC_generation0. IIRC, you want to change an occurence of something like PyGC_Object_TRACK to something like PyGC_Object_Track > I'm wondering if the decleration in Python's objimpl.h requires a DL_IMPORT > declaration, maybe. > but that's pure speculation as most of that dynamic library is > pure black magic to me. me too. > I'll probably try changing that -- requiring a recompile of python, Would be interesting to see if it helped. Submit a patch to sf (for Python) if it does? > but I'm wondering if anyone has a better idea of how to do this. > Also, anyone know of a good way to search the archives of this list? Um, emailing me regexp patterns? I've got a complete mbox archive if anyone wants a copy. Cheers, M. ```
 [Psyco-devel] Compilation on Windows? From: Tim Hochberg - 2002-01-03 23:09:30 ```Has anyone successfully compiled Psyco on Windows? I'm plugging along trying to get it to compile on Windows 2000 with Python 2.2 final. I've made some progress, everything now compiles, albeit with a boatload of warnings about truncating ints to chars, but can't link because of an unresolved reference to __PyGC_generation0. I'm wondering if the decleration in Python's objimpl.h requires a DL_IMPORT declaration, but that's pure speculation as most of that dynamic library is pure black magic to me. I'll probably try changing that -- requiring a recompile of python, but I'm wondering if anyone has a better idea of how to do this. Also, anyone know of a good way to search the archives of this list? -tim ```

Showing results of 37

<< < 1 2 (Page 2 of 2)