Re: [opendbx] slow insert with sqlite

[Mariano M. P. <mar...@gm...>]

On Tue, Jul 5, 2011 at 11:46 PM, Alain Rastoul <al...@fr...> wrote:

> **
> Hi  Mariano,
> I don't want to do multi threading with sqlite because I know it doesn't
> work.
> I was curious about the squeakdbx (or opendbx architecture) because of the
> not so good performance and the time spent in waiting , I do not
> understand the squeakdbx package vs opendbx package: the doc is mentioning a
> squeakdbx plugin dll but I have no squeakdbx dll ?
>

Sorry. THat's outdated. Once (2 years ago) Esteban Lorenzano tried to write
a plugin to avoid FFI. The idea was that such plugin could avoid locking the
VM. But I don't remember why we didn't succeded.


>
> You are saying that in that case the external call is counted on the
> InputEventPollingFetcher>> wait and not in primitives (?).
>


maybe :)  but I don't know


> I will investigate with FFI/SQlite and it should be the same (I've seen
> some messages about incorrect profiling reports in primitives),
>
>

Yes, primitives are not really well measured in profilers. Check the new
profiler announced by Eliot Miranda, it fixes this problem.


> I expected much better performance with sqlite , and glorp is  very good
> (5% of the time), I would have expected the contrary.
>

Sorry I didn't understand.


>
> Thanks
>
> Cheers
> Alain
>
> "Mariano Martinez Peck" <mar...@gm...> a écrit dans le message de
> news:CAA+-=mVV...@ma......
>
>
> On Tue, Jul 5, 2011 at 10:50 PM, Alain Rastoul <al...@fr...> wrote:
>
>> Hi,
>> (sorry for sending this mail again, my pc was off for a long time and the
>> message was dated from 2007, people who sort their messages would not see
>> it)
>>
>> I've done a small program in Pharo 1.3 with glorp+opendbx that insert 1000
>> rows in a customer table in a sqlite db.
>> The 1000 insert takes 140 sec (very slow), but the Pharo profiler says
>> that
>> it spend 95%
>> of the time waiting for input.
>> (in InputEventPollingFetcher>> waitForInput)
>> I was wondering if the queries are executed in another thread than the vm
>> thread ?
>>
>
> Hi Alain. No. Squeak/Pharo's thread architecture is the so called green
> thread, that is, only ONE OS thread is used. Internally, the language
> reifies Process, Scheduler, #fork: , etc etc etc. But from the OS point of
> view there is only one thread for the VM. So.....the regular FFI blocks the
> VM. What does it mean? that while the C function called by FFI is being
> executed, the WHOLE VM is block. Notihgn can happen at the same time.
> Imagine the function that retrieves the results and needs to wait for
> them.....TERRIBLE. So...if the backend does not support async quieries, then
> you are screw and dbx may be slow in Pharo. Nothing to do.
>
> However, some backends support async queries, and opendbx let us configure
> this. This is explained in:
>
> http://www.squeakdbx.org/Architecture%20and%20desing?_s=FlIhkPQOOFSlqf8C&_k=j-3_7Kw_&_n&18
> where it says "External call implementation"
>
> You can see the list of backends that support async queries in here:
>
> http://www.squeakdbx.org/documentation/Asynchronous%20queries?_s=FlIhkPQOOFSlqf8C&_k=j-3_7Kw_&_n&17
>
> Notice that there is some room for improvements, but we didn't have time so
> far. Hernik told us some good ideas. But since we didn't need more power so
> far we couldn't find time to integrate his ideas. I am forwarding now the
> emails to the mailing list. If you can take a look and provide code, it
> would be awesome. Basically, it improves how and how much we wait in each
> side: image and opendbx.
>
> Finally, notice that Eliot is working in a multithreared FFI for Cog, but
> it is not yet available as far as I know.
>
> Cheers
>
> Mariano
>
> (I thought I've seen a document about opendbx architecture but could'nt
>> find
>> it on the site).
>>
>> TIA
>> Alain
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> All of the data generated in your IT infrastructure is seriously valuable.
>> Why? It contains a definitive record of application performance, security
>> threats, fraudulent activity, and more. Splunk takes this data and makes
>> sense of it. IT sense. And common sense.
>> http://p.sf.net/sfu/splunk-d2d-c2
>> _______________________________________________
>> libopendbx-devel mailing list
>> lib...@pu...
>>
>> https://lists.sourceforge.net/lists/listinfo/libopendbx-devel
>> http://www.linuxnetworks.de/doc/index.php/OpenDBX
>>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>
>  ------------------------------
>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
>
> ------------------------------
>
>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> libopendbx-devel mailing list
> lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libopendbx-devel
> http://www.linuxnetworks.de/doc/index.php/OpenDBX
>
>


-- 
Mariano
http://marianopeck.wordpress.com