Re: [eyedb-general] BIG SHOW STOPPER .. eyedbadmin datafile resize .. I think that this is a BUG

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Hi Michael,

> Hi 
>   I do believe that there is a bug in the size parameter of the "eyedbadmin 
> datafiel resize...."
>
> The Harddisk space I have avaliable on my developer for EyeDB is 230GB. I 
> wanted to have 220GB use for the stocks database. 
>
> Shown below is the maximum eyedb "stocks.dat" achived for the 
> given "eyedbadmin datafiel resize"
>
> To me it clearly shows that in all cases the number after 0 is not the maximum 
> size of "stocks.dat" file. Am I interpreting the number after 0 correctly. 
> The --help says the number in in MB
>
> MAXIMUM HARD DISK SPACE THAT WILL BE USED IS 38GB
> eyedbadmin datafile resize stocks 0 220000
> storage manager: invalid datafile maxsize: datafile 'stocks.dat' is partially 
> used: size can be reduced to a minimum of '393216' Kbytes or size maybe 
> extended.
>
> MAXIMUM HARD DISK SPACE THAT WILL BE USED IS45GB
> eyedbadmin datafile resize stocks 0 22000000
>
> MAXIMUM HARD DISK SPACE THAT WILL BE USED IS 61GB
> eyedbadmin datafile resize stocks 0 2200 000 000
> storage manager: invalid datafile maxsize: datafile 'stocks.dat' is partially 
> used: size can be reduced to a minimum of '393216' Kbytes or size maybe 
> extended.
>   

All these problems come from a non-detected size overflow. The maximum 
size (in bytes) of a datafile is: MAXSIZE = 4G * SLOTSIZE.
So for a datafile with slotsize=16 (the default value), the maximum size 
is 4G * 16 = 64 GB

If you try with the current version in the subversion repositery:

eyedbadmin datafile resize stocks 0 220000
you will get a clear error message and the datafile will not be created.

eyedbadmin datafile resize stocks 0 65535
will be ok, but any value greater than 65355 will be refused.

With the version you use, some values greated than 65535 seem to be OK 
(no error messages were displayed), but in this case, the datafile could 
be corrupted (containing conflicting information about its size which 
can lead to strange behaviour).

To use hundreds of GB of disk space you need to create several datafiles 
tied to one dataspace. But note that you can create datafiles with a 
greater slotsize, for instance 64 (always power of 2):
eyedbadmin datafile create --slotsize=64 DBNAME DATFILE
in this case, the maximum size will be 4G * 64 = 256 GB.

Last point: if you do not create datafiles whose size is greater than 4G 
* SLOTSIZE,  you do not need to upgrade your eyedb version.

Hope that is clear !

Eric

> Currently
> -rw------- 1 root root  61G 2009-01-23 22:55 stocks.dat
> -rw------- 1 root root 245K 2009-01-23 15:29 stocks.dbs
> -rw------- 1 root root 481M 2009-01-23 22:55 stocks.dmp
> -rw------- 1 root root    0 2009-01-23 15:29 stocks.lck
> -rw------- 1 root root 2.4G 2009-01-23 22:55 stocks.omp
> -rw------- 1 root root 500M 2009-01-23 15:29 stocks.shm
>
> NOTICE the size parameters I had to use in order not to get a failure on the 
> resizing 
> eyedbadmin datafile resize stocks 0 22000000000
> storage manager: invalid datafile maxsize: datafile 'stocks.dat' is partially 
> used: size can be reduced to a minimum of '4194304' Kbytes or size maybe 
> extended.
>
> XMan-3:/home/XMan # eyedbadmin datafile resize stocks 0 220000000000
> storage manager: invalid datafile maxsize: datafile 'stocks.dat' is partially 
> used: size can be reduced to a minimum of '4194304' Kbytes or size maybe 
> extended.
>
> XMan-3:/home/XMan # eyedbadmin datafile resize stocks 0 2200000000000
>
> NOW my database insert application completly craps out. BIG SHOW STOPPER
> *** glibc detected 
> *** /home/XMan/Projects/marketdata/dtn-2/optimized/src/eyedb/mdl-eyedb: 
> double free or corruption (!prev): 0x00000000008207a0 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x2b8bcfdb221d]
> /lib64/libc.so.6(cfree+0x76)[0x2b8bcfdb3f76]
> /usr/local/lib/libeyedbsm-2.8.7.so(_ZN7eyedbsm4HIdx10makeObjectERNS0_11CListHeaderEjRNS_3OidERiRNS0_14CListObjHeaderERNS0_10CellHeaderEj+0x1f8)
> [0x2b8bceb32598]
> /usr/local/lib/libeyedbsm-2.8.7.so(_ZN7eyedbsm4HIdx7getCellEjRNS0_11CListHeaderEjRNS_3OidERNS0_14CListObjHeaderERiRNS0_10CellHeaderE+0x287)
> [0x2b8bceb32937]
> /usr/local/lib/libeyedbsm-2.8.7.so(_ZN7eyedbsm4HIdx14insert_performEPKvRSt6vectorIS2_SaIS2_EEj+0x386)
> [0x2b8bceb35166]
> /usr/local/lib/libeyedbsm-2.8.7.so(_ZN7eyedbsm4HIdx14insert_performEPKvS2_j+0x64)
> [0x2b8bceb35444]
> /usr/local/lib/libeyedb-2.8.7.so(_ZN5eyedb21IDB_collClassRegisterEPNS_8DbHandleEPKN7eyedbsm3OidES5_NS_4BoolE+0x115)
> [0x2b8bce4a05b5]
> /usr/local/lib/libeyedb-2.8.7.so[0x2b8bce4a079f]
> /usr/local/lib/libeyedb-2.8.7.so[0x2b8bce4a0bcf]
> /usr/local/lib/libeyedb-2.8.7.so(_ZN5eyedb16IDB_objectCreateEPNS_8DbHandleEsPhPN7eyedbsm3OidEPvPS2_S6_+0x511)
> [0x2b8bce4ac611]
> /usr/local/lib/libeyedb-2.8.7.so(_ZN5eyedb12objectCreateEPNS_8DbHandleEsPhPN7eyedbsm3OidE+0x56)
> [0x2b8bce48a496]
> /usr/local/lib/libeyedb-2.8.7.so(_ZN5eyedb10Collection14create_realizeEPKNS_7RecModeE+0x38a)
> [0x2b8bce4099fa]
> /usr/local/lib/libeyedb-2.8.7.so(_ZN5eyedb10Collection7realizeEPKNS_7RecModeE+0x21b)
> [0x2b8bce40ae0b]
> /usr/local/lib/libeyedb-2.8.7.so(_ZN5eyedb10Collection14realizePerformERKNS_3OidES3_RNS_14AttrIdxContextEPKNS_7RecModeE+0xbb)
> [0x2b8bce4084fb]
> /usr/local/lib/libeyedb-2.8.7.so(_ZNK5eyedb10AttrDirect7realizeEPNS_8DatabaseEPNS_6ObjectERKNS_3OidES7_RNS_14AttrIdxContextEPKNS_7RecModeE+0x10d)
> [0x2b8bce3d87fd]
> /usr/local/lib/libeyedb-2.8.7.so(_ZN5eyedb7Agregat14realizePerformERKNS_3OidES3_RNS_14AttrIdxContextEPKNS_7RecModeE+0x133)
> [0x2b8bce3b37b3]
> /usr/local/lib/libeyedb-2.8.7.so(_ZN5eyedb7Agregat7realizeEPKNS_7RecModeE+0x397)
> [0x2b8bce3b3d77]
> /home/XMan/Projects/marketdata/dtn-2/optimized/src/eyedb/mdl-eyedb[0x421d2c]
> /home/XMan/Projects/marketdata/dtn-2/optimized/src/eyedb/mdl-eyedb[0x44b389]
> /home/XMan/Projects/marketdata/dtn-2/optimized/src/eyedb/mdl-eyedb[0x41cea5]
> /home/XMan/Projects/marketdata/dtn-2/optimized/src/eyedb/mdl-eyedb[0x41d4ec]
> /home/XMan/Projects/marketdata/dtn-2/optimized/src/eyedb/mdl-eyedb[0x412564]
> /lib64/libc.so.6(__libc_start_main+0xf4)[0x2b8bcfd61b54]
> /home/XMan/Projects/marketdata/dtn-2/optimized/src/eyedb/mdl-eyedb(__gxx_personality_v0+0x2e1)
> [0x40f419]
> ======= Memory map: ========
> 00400000-0046f000 r-xp 00000000 08:41 
> 30310441                           /mnt/sde/home/XMan/Projects/marketdata/dtn-2/optimized/src/eyedb/mdl-eyedb
> 0066e000-0066f000 r--p 0006e000 08:41 
> 30310441                           /mnt/sde/home/XMan/Projects/marketdata/dtn-2/optimized/src/eyedb/mdl-eyedb
> 0066f000-00670000 rw-p 0006f000 08:41 
> 30310441                           /mnt/sde/home/XMan/Projects/marketdata/dtn-2/optimized/src/eyedb/mdl-eyedb
> 00670000-00953000 rw-p 00670000 00:00 0                                  
> [heap]
> 40000000-40001000 ---p 40000000 00:00 0
> 40001000-40801000 rw-p 40001000 00:00 0
> 2aaaaaaab000-2aaaab27c000 r--s 007d1000 08:01 
> 872232                     /usr/local/var/lib/eyedb/db/dbmdb.dat
> 2aaaab27c000-2aaaabc88000 rw-p 2aaaab27c000 00:00 0
> 2aaaabc88000-2aaacb088000 rw-s 00000000 08:21 
> 10747908                   /mnt/sdc/eyedb/stocks.shm
> 2aaacb088000-2aaacb0c6000 rw-s 00000000 08:21 
> 10747906                   /mnt/sdc/eyedb/stocks.dbs
> 2aaacb0c6000-2aaae90c6000 rw-s 00000000 08:21 
> 10747910                   /mnt/sdc/eyedb/stocks.dmp
> 2aaae90c6000-2aac4ead2000 rw-s 00000000 08:21 
> 10747907                   /mnt/sdc/eyedb/stocks.omp
> 2aac4ead2000-2aac4fa74000 rw-s 00000000 08:21 
> 10747909                   /mnt/sdc/eyedb/stocks.dat
> 2aac4fa74000-2aac4fa95000 r-xp 00000000 08:01 
> 426841                     /usr/local/lib/eyedb/utilsmthfe-2.8.7.so
> 2aac4fa95000-2aac4fc95000 ---p 00021000 08:01 
> 426841                     /usr/local/lib/eyedb/utilsmthfe-2.8.7.so
> 2aac4fc95000-2aac4fc96000 r--p 00021000 08:01 
> 426841                     /usr/local/lib/eyedb/utilsmthfe-2.8.7.so
> 2aac4fc96000-2aac4fc97000 rw-p 00022000 08:01 
> 426841                     /usr/local/lib/eyedb/utilsmthfe-2.8.7.so
> 2aac4fc97000-2aac50468000 rw-s 967c1d000 08:21 
> 10747909                  /mnt/sdc/eyedb/stocks.dat
> 2aac50468000-2aac50c39000 rw-s b17c19000 08:21 
> 10747909                  /mnt/sdc/eyedb/stocks.dat
> 2aac50c39000-2aac5140a000 rw-s ba9ff3000 08:21 
> 10747909                  /mnt/sdc/eyedb/stocks.dat
> 2aac5140a000-2aac51bdb000 rw-s effc17000 08:21 
> 10747909                  /mnt/sdc/eyedb/stocks.dat
> 2aac54000000-2aac54021000 rw-p 2aac54000000 00:00 0
> 2aac54021000-2aac58000000 ---p 2aac54021000 00:00 0
> 2b8bcd9cf000-2b8bcd9eb000 r-xp 00000000 08:01 
> 481710                     /lib64/ld-2.6.1.so
> 2b8bcd9eb000-2b8bcd9ed000 rw-p 2b8bcd9eb000 00:00 0
> 2b8bcda20000-2b8bcda21000 rw-p 2b8bcda20000 00:00 0
> 2b8bcdbea000-2b8bcdbec000 rw-p 0001b000 08:01 
> 481710                     /lib64/ld-2.6.1.so
> 2b8bcdbec000-2b8bcdbff000 r-xp 00000000 08:01 
> 230101                     /usr/local/lib/libboost_filesystem-gcc42-mt-1_37.so.1.37.0
> 2b8bcdbff000-2b8bcddff000 ---p 00013000 08:01 
> 230101                     /usr/local/lib/libboost_filesystem-gcc42-mt-1_37.so.1.37.0
> 2b8bcddff000-2b8bcde00000 r--p 00013000 08:01 
> 230101                     /usr/local/lib/libboost_filesystem-gcc42-mt-1_37.so.1.37.0
> 2b8bcde00000-2b8bcde01000 rw-p 00014000 08:01 
> 230101                     /usr/local/lib/libboost_filesystem-gcc42-mt-1_37.so.1.37.0
> 2b8bcde01000-2b8bcde11000 r-xp 00000000 08:01 
> 46775                      /usr/local/lib/libboost_date_time-gcc42-mt-1_37.so.1.37.0
> 2b8bcde11000-2b8bce011000 ---p 00010000 08:01 
> 46775                      /usr/local/lib/libboost_date_time-gcc42-mt-1_37.so.1.37.0
> 2b8bce011000-2b8bce012000 r--p 00010000 08:01 
> 46775                      /usr/local/lib/libboost_date_time-gcc42-mt-1_37.so.1.37.0
> 2b8bce012000-2b8bce013000 rw-p 00011000 08:01 
> 46775                      /usr/local/lib/libboost_date_time-gcc42-mt-1_37.so.1.37.0
> 2b8bce013000-2b8bce05a000 r-xp 00000000 08:01 
> 46758                      /usr/local/lib/libboost_program_options-gcc42-mt-1_37.so.1.37.0
> 2b8bce05a000-2b8bce25a000 ---p 00047000 08:01 
> 46758                      /usr/local/lib/libboost_program_options-gcc42-mt-1_37.so.1.37.0
> 2b8bce25a000-2b8bce25d000 r--p 00047000 08:01 
> 46758                      /usr/local/lib/libboost_program_options-gcc42-mt-1_37.so.1.37.0
> 2b8bce25d000-2b8bce25e000 rw-p 0004a000 08:01 
> 46758                      /usr/local/lib/libboost_program_options-gcc42-mt-1_37.so.1.37.0
> 2b8bce25e000-2b8bce25f000 rw-p 2b8bce25e000 00:00 0
> 2b8bce25f000-2b8bce6cd000 r-xp 00000000 08:01 
> 70232                      /usr/local/lib/libeyedb-2.8.7.so
> 2b8bce6cd000-2b8bce8cd000 ---p 0046e000 08:01 
> 70232                      /usr/local/lib/libeyedb-2.8.7.so
> 2b8bce8cd000-2b8bce8f2000 r--p 0046e000 08:01 
> 70232                      /usr/local/lib/libeyedb-2.8.7.so
> 2b8bce8f2000-2b8bce8f9000 rw-p 00493000 08:01 
> 70232                      /usr/local/lib/libeyedb-2.8.7.so
> 2b8bce8f9000-2b8bce902000 rw-p 2b8bce8f9000 00:00 0
> 2b8bce902000-2b8bce909000 r-xp 00000000 08:01 
> 69935                      /usr/local/lib/libeyedbrpcfe-2.8.7.so
> 2b8bce909000-2b8bceb08000 ---p 00007000 08:01 
> 69935                      /usr/local/lib/libeyedbrpcfe-2.8.7.so
> 2b8bceb08000-2b8bceb09000 r--p 00006000 08:01 
> 69935                      /usr/local/lib/libeyedbrpcfe-2.8.7.so
> Aborted=14 Bytes=7917, pbytes=877007000 08:01 
> 69935                      /usr/local/lib/libeyedPackets=1 Bytes=936, 
> pbytes=936
>   