Re: [eyedb-general] [eyedb-devel] BUG: eyedbadmin datafile create has a bug in the --size parameter

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

I has been about an hour now since I started the 

select s from Equity s; 

query. 

I just terminated it after doing a  top saw that 11.8 gb and 43.3% memory 
utilized.
 4874 root      19   0 11.8g 3.4g 701m S    2 43.3   1:45.58 eyedbd

Anyway, in general unconstrained queries are not particularly important to my 
needs. 

? bench("count(select s from Equity s where s.symbol == \"AMD\" or s.symbol == 
\"INTC\")");
Elapsed time: 6783.575000 ms
= 581189
?    

Which is not too bad given?
ls -lh
total 36G
-rw------- 1 root root  34G 2009-01-28 02:50 trades.dat
-rw------- 1 root root 245K 2009-01-27 18:56 trades.dbs
-rw------- 1 root root 270M 2009-01-28 02:50 trades.dmp
-rw------- 1 root root    0 2009-01-27 18:56 trades.lck
-rw------- 1 root root 1.5G 2009-01-28 02:50 trades.omp
-rw------- 1 root root 500M 2009-01-27 18:56 trades.shm

Michael.

On Friday 30 January 2009 10:37:25 am Eric Viara wrote:
> > I am fairly sure I have solved the Issue. I changed the my ulimit to
> > match yours.
> >
> > ulimit -m unlimited
> > ulimit -v unlimited
> >
> > That solved the select s from Equity s where s.symbol == "AMD";
>
> Good news !
>
> > Now I am waiting for the
> >
> > select s from Equity s;
>
> Generaly speaking, any query that returns "many" elements is slow
> because of a non scaleable implementation of the select.
> An OQL select statement returns a bag (or a list in case of an order by
> clause). This bag is built by the OQL engine, and the construct of such
> a bag (1) can be very long (2) uses a lot of memory.
> In case of the memory used is more that the size of the memory available
> on your computer, such a query can take ages...
>
> On my 16 GB computer, such a query on a 20M objects takes 26 seconds
> (hot access), a couple of minutes (cold access).
> On my 4 GB computer, such a query takes... 1800 seconds (hot or cold
> access).
>
> Eric
>
> > It is still running after 15 minutes. Before it would exit within 30
> > seconds. Som I think that this is a good sign that all the issues were
> > ulimit related
> >
> > By default my ulimit -a is
> > ulimit -a
> > core file size          (blocks, -c) 0
> > data seg size           (kbytes, -d) unlimited
> > scheduling priority             (-e) 0
> > file size               (blocks, -f) unlimited
> > pending signals                 (-i) 81920
> > max locked memory       (kbytes, -l) 32
> > max memory size         (kbytes, -m) 6947220
> > open files                      (-n) 1024
> > pipe size            (512 bytes, -p) 8
> > POSIX message queues     (bytes, -q) 819200
> > real-time priority              (-r) 0
> > stack size              (kbytes, -s) 8192
> > cpu time               (seconds, -t) unlimited
> > max user processes              (-u) 81920
> > virtual memory          (kbytes, -v) 8222160
> > file locks                      (-x) unlimited
> >
> >>>>>>>> ? select Record;Catchpoint 2 (exception thrown)0x00002b15bcdd1da0
> >>>>>>>> in __cxa_throw () from /usr/lib64/libstdc++.so.6(gdb) bt
> >>>>>>>> #0  0x00002b15bcdd1da0 in __cxa_throw () from
> >>>>>>>> /usr/lib64/libstdc++.so.6 #1  0x00002b15bcdd2299 in operator new
> >>>>>>>> () from /usr/lib64/libstdc++.so.6 #2  0x00002b15bb9d8bf3 in
> >>>>>>>> eyedb::oqmlAtom::make_atom ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #3  0x00002b15bba1d9b8 in eyedb::oqml_scan ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #4  0x00002b15bba0d66b in eyedb::oqmlIdent::evalQuery ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #5  0x00002b15bba0d8c3 in eyedb::oqmlIdent::eval ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #6  0x00002b15bba4cdd6 in eyedb::oqmlSelect::eval ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #7  0x00002b15bb9d8991 in eyedb::oqmlNode::realize ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #8  0x00002b15bbb06ecb in oqlparse () from
> >>>>>>>> /usr/local/lib/libeyedb-2.8.8.so #9  0x00002b15bbb08c2d in
> >>>>>>>> eyedb::oqml_realize ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #10 0x00002b15bbb08e80 in eyedb::oqml_realize ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #11 0x00002b15bb9bfac6 in
> >>>>>>>> eyedb::OQLBEIteratorOQL::OQLBEIteratorOQL () from
> >>>>>>>> /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #12 0x00002b15bb9be83c in eyedb::OQLBE::OQLBE ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #13 0x00002b15bb9818a6 in eyedb::IDB_oqlCreate ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #14 0x00002b15bb96af0d in eyedb::oqlCreate ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #15 0x00002b15bb9bedb1 in eyedb::OQL::execute ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #16 0x00002b15bb9beef5 in eyedb::OQL::execute ()
> >>>>>>>> from /usr/local/lib/libeyedb-2.8.8.so
> >>>>>>>> #17 0x000000000040e789 in eyedb::OQLParser::oql_send ()
> >>>>>>>> #18 0x000000000040f1d1 in eyedb::OQLParser::parse ()
> >>>>>>>> #19 0x000000000040f307 in eyedb::OQLParser::parse ()
> >>>>>>>> #20 0x0000000000407509 in main ()
> >>>>>>>> (gdb)
> >>>>>>>>
> >>>>>>>>> Sorry to bother you with this, but this is the only way to have a
> >>>>>>>>> (little) chance to fix this bug.
> >>>>>>>>
> >>>>>>>> No bother at all :-). As as much as you want to.
> >>>>>>>>
> >>>>>>>>> Another question: does the failure appears immediately or after a
> >>>>>>>>> few seconds (or minutes) ?
> >>>>>>>>
> >>>>>>>> After a few seconds.
> >>>>>>>> It does seem to have to do with how many objects is coming back
> >>>>>>>> from a query;
> >>>>>>>>
> >>>>>>>> Michael

-- 
I, the unwilling, was led by the unqualified, to do the unbelievable for so 
long with so little, that I attempted the impossible with nothing......"