From: Michael G. H. <mic...@gm...> - 2009-01-30 16:14:56
|
I has been about an hour now since I started the select s from Equity s; query. I just terminated it after doing a top saw that 11.8 gb and 43.3% memory utilized. 4874 root 19 0 11.8g 3.4g 701m S 2 43.3 1:45.58 eyedbd Anyway, in general unconstrained queries are not particularly important to my needs. ? bench("count(select s from Equity s where s.symbol == \"AMD\" or s.symbol == \"INTC\")"); Elapsed time: 6783.575000 ms = 581189 ? Which is not too bad given? ls -lh total 36G -rw------- 1 root root 34G 2009-01-28 02:50 trades.dat -rw------- 1 root root 245K 2009-01-27 18:56 trades.dbs -rw------- 1 root root 270M 2009-01-28 02:50 trades.dmp -rw------- 1 root root 0 2009-01-27 18:56 trades.lck -rw------- 1 root root 1.5G 2009-01-28 02:50 trades.omp -rw------- 1 root root 500M 2009-01-27 18:56 trades.shm Michael. On Friday 30 January 2009 10:37:25 am Eric Viara wrote: > > I am fairly sure I have solved the Issue. I changed the my ulimit to > > match yours. > > > > ulimit -m unlimited > > ulimit -v unlimited > > > > That solved the select s from Equity s where s.symbol == "AMD"; > > Good news ! > > > Now I am waiting for the > > > > select s from Equity s; > > Generaly speaking, any query that returns "many" elements is slow > because of a non scaleable implementation of the select. > An OQL select statement returns a bag (or a list in case of an order by > clause). This bag is built by the OQL engine, and the construct of such > a bag (1) can be very long (2) uses a lot of memory. > In case of the memory used is more that the size of the memory available > on your computer, such a query can take ages... > > On my 16 GB computer, such a query on a 20M objects takes 26 seconds > (hot access), a couple of minutes (cold access). > On my 4 GB computer, such a query takes... 1800 seconds (hot or cold > access). > > Eric > > > It is still running after 15 minutes. Before it would exit within 30 > > seconds. Som I think that this is a good sign that all the issues were > > ulimit related > > > > By default my ulimit -a is > > ulimit -a > > core file size (blocks, -c) 0 > > data seg size (kbytes, -d) unlimited > > scheduling priority (-e) 0 > > file size (blocks, -f) unlimited > > pending signals (-i) 81920 > > max locked memory (kbytes, -l) 32 > > max memory size (kbytes, -m) 6947220 > > open files (-n) 1024 > > pipe size (512 bytes, -p) 8 > > POSIX message queues (bytes, -q) 819200 > > real-time priority (-r) 0 > > stack size (kbytes, -s) 8192 > > cpu time (seconds, -t) unlimited > > max user processes (-u) 81920 > > virtual memory (kbytes, -v) 8222160 > > file locks (-x) unlimited > > > >>>>>>>> ? select Record;Catchpoint 2 (exception thrown)0x00002b15bcdd1da0 > >>>>>>>> in __cxa_throw () from /usr/lib64/libstdc++.so.6(gdb) bt > >>>>>>>> #0 0x00002b15bcdd1da0 in __cxa_throw () from > >>>>>>>> /usr/lib64/libstdc++.so.6 #1 0x00002b15bcdd2299 in operator new > >>>>>>>> () from /usr/lib64/libstdc++.so.6 #2 0x00002b15bb9d8bf3 in > >>>>>>>> eyedb::oqmlAtom::make_atom () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #3 0x00002b15bba1d9b8 in eyedb::oqml_scan () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #4 0x00002b15bba0d66b in eyedb::oqmlIdent::evalQuery () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #5 0x00002b15bba0d8c3 in eyedb::oqmlIdent::eval () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #6 0x00002b15bba4cdd6 in eyedb::oqmlSelect::eval () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #7 0x00002b15bb9d8991 in eyedb::oqmlNode::realize () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #8 0x00002b15bbb06ecb in oqlparse () from > >>>>>>>> /usr/local/lib/libeyedb-2.8.8.so #9 0x00002b15bbb08c2d in > >>>>>>>> eyedb::oqml_realize () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #10 0x00002b15bbb08e80 in eyedb::oqml_realize () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #11 0x00002b15bb9bfac6 in > >>>>>>>> eyedb::OQLBEIteratorOQL::OQLBEIteratorOQL () from > >>>>>>>> /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #12 0x00002b15bb9be83c in eyedb::OQLBE::OQLBE () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #13 0x00002b15bb9818a6 in eyedb::IDB_oqlCreate () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #14 0x00002b15bb96af0d in eyedb::oqlCreate () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #15 0x00002b15bb9bedb1 in eyedb::OQL::execute () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #16 0x00002b15bb9beef5 in eyedb::OQL::execute () > >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so > >>>>>>>> #17 0x000000000040e789 in eyedb::OQLParser::oql_send () > >>>>>>>> #18 0x000000000040f1d1 in eyedb::OQLParser::parse () > >>>>>>>> #19 0x000000000040f307 in eyedb::OQLParser::parse () > >>>>>>>> #20 0x0000000000407509 in main () > >>>>>>>> (gdb) > >>>>>>>> > >>>>>>>>> Sorry to bother you with this, but this is the only way to have a > >>>>>>>>> (little) chance to fix this bug. > >>>>>>>> > >>>>>>>> No bother at all :-). As as much as you want to. > >>>>>>>> > >>>>>>>>> Another question: does the failure appears immediately or after a > >>>>>>>>> few seconds (or minutes) ? > >>>>>>>> > >>>>>>>> After a few seconds. > >>>>>>>> It does seem to have to do with how many objects is coming back > >>>>>>>> from a query; > >>>>>>>> > >>>>>>>> Michael -- I, the unwilling, was led by the unqualified, to do the unbelievable for so long with so little, that I attempted the impossible with nothing......" |