Re: [opendbx] slow insert with sqlite
Brought to you by:
nose
Menu
▾
▴
Re: [opendbx] slow insert with sqlite
|
From: Alain R. <al...@fr...> - 2011-07-06 17:54:46
|
Thank you Diogenes, I will try your advice about synchronous pragma and send you my code, but I have to work for a customer this evening :( I will subscribe to the squeak dbx mailing list too. thanks again Regards, Alain "Diogenes Moreira" <dio...@gm...> a écrit dans le message de news: CAL...@ma...... hi alan. i dont know if sqlite implemetation has a problem.. but in glorp some times do unnecesary read.. by example when you making "N times" CommitAndContinue.. and because it, you see a lot time spending in InputEventPollingFetcher>>wait. if you send to me or send to SqueakDbx list you glorp code(ST), may be, I can send to you some advices. or may be, we can find the delay source. Best. pd: please check in you sqlite the pragmas.. by example synchronous (http://www.sqlite.org/pragma.html#pragma_synchronous) that increment the performance to much. On Tue, Jul 5, 2011 at 6:46 PM, Alain Rastoul <al...@fr...> wrote: Hi Mariano, I don't want to do multi threading with sqlite because I know it doesn't work. I was curious about the squeakdbx (or opendbx architecture) because of the not so good performance and the time spent in waiting , I do not understand the squeakdbx package vs opendbx package: the doc is mentioning a squeakdbx plugin dll but I have no squeakdbx dll ? You are saying that in that case the external call is counted on the InputEventPollingFetcher>> wait and not in primitives (?). I will investigate with FFI/SQlite and it should be the same (I've seen some messages about incorrect profiling reports in primitives), I expected much better performance with sqlite , and glorp is very good (5% of the time), I would have expected the contrary. Thanks Cheers Alain "Mariano Martinez Peck" <mar...@gm...> a écrit dans le message de news:CAA+-=mVV3zvP...@pu...... On Tue, Jul 5, 2011 at 10:50 PM, Alain Rastoul <al...@fr...> wrote: Hi, (sorry for sending this mail again, my pc was off for a long time and the message was dated from 2007, people who sort their messages would not see it) I've done a small program in Pharo 1.3 with glorp+opendbx that insert 1000 rows in a customer table in a sqlite db. The 1000 insert takes 140 sec (very slow), but the Pharo profiler says that it spend 95% of the time waiting for input. (in InputEventPollingFetcher>> waitForInput) I was wondering if the queries are executed in another thread than the vm thread ? Hi Alain. No. Squeak/Pharo's thread architecture is the so called green thread, that is, only ONE OS thread is used. Internally, the language reifies Process, Scheduler, #fork: , etc etc etc. But from the OS point of view there is only one thread for the VM. So.....the regular FFI blocks the VM. What does it mean? that while the C function called by FFI is being executed, the WHOLE VM is block. Notihgn can happen at the same time. Imagine the function that retrieves the results and needs to wait for them.....TERRIBLE. So...if the backend does not support async quieries, then you are screw and dbx may be slow in Pharo. Nothing to do. However, some backends support async queries, and opendbx let us configure this. This is explained in: http://www.squeakdbx.org/Architecture%20and%20desing?_s=FlIhkPQOOFSlqf8C&_k=j-3_7Kw_&_n&18 where it says "External call implementation" You can see the list of backends that support async queries in here: http://www.squeakdbx.org/documentation/Asynchronous%20queries?_s=FlIhkPQOOFSlqf8C&_k=j-3_7Kw_&_n&17 Notice that there is some room for improvements, but we didn't have time so far. Hernik told us some good ideas. But since we didn't need more power so far we couldn't find time to integrate his ideas. I am forwarding now the emails to the mailing list. If you can take a look and provide code, it would be awesome. Basically, it improves how and how much we wait in each side: image and opendbx. Finally, notice that Eliot is working in a multithreared FFI for Cog, but it is not yet available as far as I know. Cheers Mariano (I thought I've seen a document about opendbx architecture but could'nt find it on the site). TIA Alain ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ libopendbx-devel mailing list libopendbx-...@pu... https://lists.sourceforge.net/lists/listinfo/libopendbx-devel http://www.linuxnetworks.de/doc/index.php/OpenDBX -- Mariano http://marianopeck.wordpress.com -------------------------------------------------------------------------- ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 -------------------------------------------------------------------------- ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ libopendbx-devel mailing list lib...@li... https://lists.sourceforge.net/lists/listinfo/libopendbx-devel http://www.linuxnetworks.de/doc/index.php/OpenDBX ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ------------------------------------------------------------------------------ |
