sqlobject-discuss

Re: [SQLObject] SQLObject Speed ?

[Stef T. <st...@um...>]

Simon Cross wrote:
> On Thu, Jul 30, 2009 at 3:09 PM, Stef Telford<st...@um...> wrote:
>   
>>    Let me re-ask that question; is there any point in submitting
>> patches to use gmpy to you if to get things working requires a patched
>> psycopg2 (which will probably never go 'mainstream') ?
>>     
>
> I think your best bet is going to be to attempt to get a fast C
> implementation of Decimal accepted into Python itself.  There has
> already been some work on this which is outlined in the Python bug
> tracker [1] and given that there was talk of accepting the patches for
> 3.1 there is some hope that you could get them into 2.7 / 3.2.
>
> [1] http://bugs.python.org/issue2486
>
> Schiavo
> Simon
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
> trial. Simplify your report design, integration and deployment - and focus on 
> what you do best, core application coding. Discover what's new with 
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> sqlobject-discuss mailing list
> sql...@li...
> https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
>   
Hello Simon,
    Well.. sadly.. how realistic is this ? No offense meant to anyone, 
but, gmpy has been around since (at least) python 2.4 (perhaps earlier ? 
not sure 100%) and it still hasn't been accepted in. I do agree that 
fixing Decimal at a fundamental level would be MUCH nicer, but, the BDFL 
(Guido) seems to not be moving on it. That bug was opened on 2008-03 .. 
over a year ago.

    I think that the quickest (no pun intended) is to fix it in the 
adaptor if possible. Jst my 2c :)

    Regards
    Stef

Re: [SQLObject] SQLObject Speed ?

[Simon C. <hod...@gm...>]

On Thu, Jul 30, 2009 at 5:39 PM, Stef Telford<st...@um...> wrote:
>   Well.. sadly.. how realistic is this ? No offense meant to anyone, but,
> gmpy has been around since (at least) python 2.4 (perhaps earlier ? not sure
> 100%) and it still hasn't been accepted in. I do agree that fixing Decimal
> at a fundamental level would be MUCH nicer, but, the BDFL (Guido) seems to
> not be moving on it. That bug was opened on 2008-03 .. over a year ago.

Agreed that getting any substantial change into Python is going to be
a bit of pain for anyone with commit access, but I'm don't think
getting a C implementation of an existing module in is going to be
anywhere near as difficult as getting a whole new library accepted
(and one that introduces a new somewhat dodgy C library dependency at
that).

> I think that the quickest (no pun intended) is to fix it in the adaptor if
> possible. Jst my 2c :)

Sure. But it would be cool to have a C implementation of Decimal in
standard Python. :)

Schiavo
Simon

Re: [SQLObject] SQLObject Speed ?

[Oleg B. <ph...@ph...>]

On Tue, Jul 28, 2009 at 04:04:28PM -0400, Stef Telford wrote:
> Oleg Broytmann wrote:
>>    Can I ask you do an experiment with a different program? What if you use
>> sqlbuilder.Select() - a lower-level interface? How long it'd take to draw
>> all these rows?
>
>    Sorry, I have almost -0- experience with that low a level .. do you  
> have any nice canned example I could tailor to suit ?

   Very easy: sqlbuilder.Select(list_of_columns), convert the expression to
a string (SQL query), execute the query and get back the results:

connection = connectionForURI('...')

class Test(SQLObject):
    _connection = connection
    a = IntCol()
    b = IntCol()

Test.createTable()

Test(a=1, b=2)
Test(a=2, b=1)

for row in connection.queryAll(connection.sqlrepr(
      sqlbuilder.Select([Test.q.a, Test.q.b]))):
    print row

   In case you want to pass the list of columns as a list of strings - pass
the list of tables for the FROM clause:

      Select(['a', 'b'], staticTables=['test'])

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            ph...@ph...
           Programmers don't die, they just GOSUB without RETURN.

Re: [SQLObject] SQLObject Speed ?

[Stef T. <st...@um...>]

Oleg Broytmann wrote:
> On Tue, Jul 28, 2009 at 04:04:28PM -0400, Stef Telford wrote:
>   
>> Oleg Broytmann wrote:
>>     
>>>    Can I ask you do an experiment with a different program? What if you use
>>> sqlbuilder.Select() - a lower-level interface? How long it'd take to draw
>>> all these rows?
>>>       
>>    Sorry, I have almost -0- experience with that low a level .. do you  
>> have any nice canned example I could tailor to suit ?
>>     
>
>    Very easy: sqlbuilder.Select(list_of_columns), convert the expression to
> a string (SQL query), execute the query and get back the results:
>
> connection = connectionForURI('...')
>
> class Test(SQLObject):
>     _connection = connection
>     a = IntCol()
>     b = IntCol()
>
> Test.createTable()
>
> Test(a=1, b=2)
> Test(a=2, b=1)
>
> for row in connection.queryAll(connection.sqlrepr(
>       sqlbuilder.Select([Test.q.a, Test.q.b]))):
>     print row
>
>    In case you want to pass the list of columns as a list of strings - pass
> the list of tables for the FROM clause:
>
>       Select(['a', 'b'], staticTables=['test'])
>
> Oleg.
>   
Hello Oleg,
    Here is the output from the lowlevel connection select as you 
suggested above (mostly) ordered by internal time. Note that it selects 
all 220k bookings, as opposed to last time when it 'only' selected 40k. 
It seems that decimal.__new__ is killer .. I could be reading this wrong 
(of course) but.. the tottime would seem to back that up I think.

         133167599 function calls (133165268 primitive calls) in 392.615 
CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  9883936   93.137    0.000   93.137    0.000 {built-in method __new__ 
of type object at 0x814fa00}
  9883851   90.187    0.000  242.693    0.000 decimal.py:515(__new__)
        1   53.209   53.209  295.902  295.902 {method 'fetchall' of 
'psycopg2._psycopg.cursor' objects}
  9883845   35.014    0.000   42.012    0.000 decimal.py:830(__str__)
  9884218   25.102    0.000   25.102    0.000 {built-in method match}
  9883845   21.096    0.000   63.108    0.000 decimal.py:825(__repr__)
 39536056   20.103    0.000   20.103    0.000 {built-in method group}
        1   16.715   16.715  392.589  392.589 t.lowlevel:3(<module>)
        1   16.522   16.522   16.522   16.522 {method 'execute' of 
'psycopg2._psycopg.cursor' objects}
24350308/24349893    9.742    0.000    9.742    0.000 {len}
  9883846    6.461    0.000    6.461    0.000 {method 'lstrip' of 'str' 
objects}
  9885976    4.971    0.000    4.971    0.000 {isinstance}
        1    0.031    0.031    0.031    0.031 
pgconnection.py:108(makeConnection)
        1    0.026    0.026  392.615  392.615 {execfile}
   371/62    0.015    0.000    0.048    0.001 sre_parse.py:385(_parse)
        1    0.014    0.014  312.470  312.470 
dbconnection.py:257(_runWithConnection)
     4824    0.012    0.000    0.015    0.000 sre_parse.py:188(__next)
   740/61    0.011    0.000    0.033    0.001 sre_compile.py:38(_compile)
      261    0.010    0.000    0.011    0.000 {eval}
      339    0.010    0.000    0.015    0.000 
sre_compile.py:213(_optimize_charset)
 1054/380    0.008    0.000    0.009    0.000 sre_parse.py:146(getwidth)
      664    0.006    0.000    0.009    0.000 {built-in method sub}
    12091    0.005    0.000    0.005    0.000 {method 'append' of 'list' 
objects}
     4074    0.005    0.000    0.018    0.000 sre_parse.py:207(get)
       43    0.004    0.000    0.004    0.000 sre_compile.py:264(_mk_bitmap)
      128    0.004    0.000    0.017    0.000 
pkg_resources.py:1645(find_on_path)
        1    0.004    0.004    0.005    0.005 socket.py:43(<module>)
        1    0.004    0.004    0.031    0.031 converters.py:1(<module>)
        1    0.004    0.004    0.007    0.007 mimetypes.py:199(readfp)
     6273    0.003    0.000    0.003    0.000 {method 'startswith' of 
'str' objects}


    Regards
    Stef