Menu

#7 execption when inserting record in DISKLESS mode

v1.0_(example)
open
nobody
None
5
2013-09-25
2013-09-24
yymailb
No

fastdb version: 3.73
env: windows 8, visual c++ 2010 sp1

desc:
I wrote a short program which using fastdb in DISKLESS mode (#define DISKLESS_CONFIGURATION 1), which created a db instance with initialsize = 1000000000(1GB), and repeat insert a small struct to db. fastdb throw a FileLimitExeeded execption when the No.198181 record is inserting.
I've checked the call stack, the execption throwed in function inline void dbDatabase::extend(offs_t size), where the parameter size = 0xfff82000, maybe something wrong?

the program is attached.

1 Attachments

Related

Bugs: #7

Discussion

  • Konstantin Knizhnik

    In diskless mode FastDB is not able to reallocate database , so initial
    database size is actually maximal database size.
    Please notice that specified size of database determines size of virtual
    memory region and doesn't cause allocation of correspondent amount of
    physical memory. So you can specify very larger initial database size in
    diskless mode - much more than for example physical memory at your
    system. It will not cause inefficient usage of system resources.

    On 09/24/2013 01:35 PM, yymailb wrote:


    [bugs:#7] http://sourceforge.net/p/fastdb/bugs/7/ execption when
    inserting record in DISKLESS mode

    Status: open
    Created: Tue Sep 24, 2013 09:35 AM UTC by yymailb
    Last Updated: Tue Sep 24, 2013 09:35 AM UTC
    Owner: nobody

    fastdb version: 3.73
    env: windows 8, visual c++ 2010 sp1

    desc:
    I wrote a short program which using fastdb in DISKLESS mode (#define
    DISKLESS_CONFIGURATION 1), which created a db instance with
    initialsize = 1000000000(1GB), and repeat insert a small struct to db.
    fastdb throw a FileLimitExeeded execption when the No.198181 record is
    inserting.
    I've checked the call stack, the execption throwed in function inline
    void dbDatabase::extend(offs_t size), where the parameter size =
    0xfff82000, maybe something wrong?

    the program is attached.


    Sent from sourceforge.net because you indicated interest in
    https://sourceforge.net/p/fastdb/bugs/7/

    To unsubscribe from further messages, please visit
    https://sourceforge.net/auth/subscriptions/

     

    Related

    Bugs: #7

  • yymailb

    yymailb - 2013-09-24

    I've tried some different initial size, such as 100MB, 1GB, and 2GB. in each condition, the program always throw exception at No. 198181 record. the computer has 16GB phisical memory. I noticed that the fastdb call function dbDatabase::extend(offs_t size) with parameter size = 0xfff82000(4.29GB), I'm not sure if this value exceed some limit for a 32bit process.

     
  • Konstantin Knizhnik

    Are you using 32-bit or 64-bit version of Windows?
    Did you build FastDB and you applications as 64-bit application?
    Fro 32-bit application size of database is naturally limited by 4Gb.

    On 09/24/2013 05:03 PM, yymailb wrote:

    I've tried some different initial size, such as 100MB, 1GB, and 2GB.
    in each condition, the program always throw exception at No. 198181
    record. the computer has 16GB phisical memory. I noticed that the
    fastdb call function dbDatabase::extend(offs_t size) with parameter
    size = 0xfff82000(4.29GB), I'm not sure if this value exceed some
    limit for a 32bit process.


    [bugs:#7] http://sourceforge.net/p/fastdb/bugs/7/ execption when
    inserting record in DISKLESS mode

    Status: open
    Created: Tue Sep 24, 2013 09:35 AM UTC by yymailb
    Last Updated: Tue Sep 24, 2013 09:35 AM UTC
    Owner: nobody

    fastdb version: 3.73
    env: windows 8, visual c++ 2010 sp1

    desc:
    I wrote a short program which using fastdb in DISKLESS mode (#define
    DISKLESS_CONFIGURATION 1), which created a db instance with
    initialsize = 1000000000(1GB), and repeat insert a small struct to db.
    fastdb throw a FileLimitExeeded execption when the No.198181 record is
    inserting.
    I've checked the call stack, the execption throwed in function inline
    void dbDatabase::extend(offs_t size), where the parameter size =
    0xfff82000, maybe something wrong?

    the program is attached.


    Sent from sourceforge.net because you indicated interest in
    https://sourceforge.net/p/fastdb/bugs/7/

    To unsubscribe from further messages, please visit
    https://sourceforge.net/auth/subscriptions/

     

    Related

    Bugs: #7

  • yymailb

    yymailb - 2013-09-24

    I'm using a 64bit windows 8. fastdb and my program are compiled as 32bit. I know that the 4gb limit for 32bit process. In this case, the struct size is 48 byte, 198181 records means about 9.5MB, much less than initial db size, why dbDatabase::extend try resize to 4.2GB?

     
  • Konstantin Knizhnik

    I find out a problem - it is caused by my recent attempt to avoid too
    deep recursion in memory allocator.
    For diskless mode which is not able to extend database this approach
    doesn't work.
    I have committed the fix in SVN repository at SourceForge.

    On 09/24/2013 05:43 PM, yymailb wrote:

    I'm using a 64bit windows 8. fastdb and my program are compiled as
    32bit. I know that the 4gb limit for 32bit process. In this case, the
    struct size is 48 byte, 198181 records means about 9.5MB, much less
    than initial db size, why dbDatabase::extend try resize to 4.2GB?


    [bugs:#7] http://sourceforge.net/p/fastdb/bugs/7/ execption when
    inserting record in DISKLESS mode

    Status: open
    Created: Tue Sep 24, 2013 09:35 AM UTC by yymailb
    Last Updated: Tue Sep 24, 2013 01:03 PM UTC
    Owner: nobody

    fastdb version: 3.73
    env: windows 8, visual c++ 2010 sp1

    desc:
    I wrote a short program which using fastdb in DISKLESS mode (#define
    DISKLESS_CONFIGURATION 1), which created a db instance with
    initialsize = 1000000000(1GB), and repeat insert a small struct to db.
    fastdb throw a FileLimitExeeded execption when the No.198181 record is
    inserting.
    I've checked the call stack, the execption throwed in function inline
    void dbDatabase::extend(offs_t size), where the parameter size =
    0xfff82000, maybe something wrong?

    the program is attached.


    Sent from sourceforge.net because you indicated interest in
    https://sourceforge.net/p/fastdb/bugs/7/

    To unsubscribe from further messages, please visit
    https://sourceforge.net/auth/subscriptions/

     

    Related

    Bugs: #7

  • yymailb

    yymailb - 2013-09-25

    I've checked out the newest version, it's works well. Now I can insert 255826 records in a 100MB db before exception, each record cost 390 bytes while the record struct size is 48 bytes. Is this normal?

     

    Last edit: yymailb 2013-09-25
  • Konstantin Knizhnik

    On 09/25/2013 05:42 AM, yymailb wrote:

    I've checked out the newest svn version and tried it. it's very much
    slower than version 3.73, 1000 inserts cost about 20 seconds. init db
    size is 1GB.

    Sorry, I can not reproduce the problem. In my case performance is about
    10 seconds for inserting all objects.
    Please check that you are still using diskless mode.
    In any case - transaction commit after each insert is not so efficient
    even in diskless mode (and in normal mode it will be extremely sow
    because each transaction commit will require synchronous write to the
    disk). You can reduce insert time by doing all inserts within one
    transaction or using delayed commit policy or at least perform commit
    after each N inserts.


    [bugs:#7] http://sourceforge.net/p/fastdb/bugs/7/ execption when
    inserting record in DISKLESS mode

    Status: open
    Created: Tue Sep 24, 2013 09:35 AM UTC by yymailb
    Last Updated: Tue Sep 24, 2013 01:43 PM UTC
    Owner: nobody

    fastdb version: 3.73
    env: windows 8, visual c++ 2010 sp1

    desc:
    I wrote a short program which using fastdb in DISKLESS mode (#define
    DISKLESS_CONFIGURATION 1), which created a db instance with
    initialsize = 1000000000(1GB), and repeat insert a small struct to db.
    fastdb throw a FileLimitExeeded execption when the No.198181 record is
    inserting.
    I've checked the call stack, the execption throwed in function inline
    void dbDatabase::extend(offs_t size), where the parameter size =
    0xfff82000, maybe something wrong?

    the program is attached.


    Sent from sourceforge.net because you indicated interest in
    https://sourceforge.net/p/fastdb/bugs/7/

    To unsubscribe from further messages, please visit
    https://sourceforge.net/auth/subscriptions/

     

    Related

    Bugs: #7

  • yymailb

    yymailb - 2013-09-25

    I've checked out the newest version, it's works well. Now I can insert 255826 records in a 100MB db before exception, each record cost 390 bytes while the record struct size is 48 bytes. Is this normal?

     
  • Konstantin Knizhnik

    On 09/25/2013 10:38 AM, yymailb wrote:

    I've checked out the newest version, it's works well. Now I can insert
    255826 records in a 100MB db before exception, each record cost 390
    bytes while the record struct size is 48 bytes. Is this normal?

    Estimated average size of DbMarginRate object indatabase is 64 bytes (8
    + 4 + 8*4 + 4 + 10) align on 16 byte quantum.
    Also there 3 indexes - about 24 extra bytes per object. Plus two
    OID->offset maps - up tp 16 bytes per object.
    Large number of short transactions (frequent commits) requires
    allocation of huge number of shadow page copies. Them are deallocated on
    commit, but it leads to fragmentation and also can cause growth of
    database size.

    If you are building FastDB with LARGE_DATABASE_SUPPORT ( I am not sure
    that it happen i you case because you are using 32-bit mode),
    then FastDB supports up tp 1Tb database and reserve about 100 space for
    bitmap pages.
    What is the size of empty database in your case (when you create
    database and just insert 1 object)?


    [bugs:#7] http://sourceforge.net/p/fastdb/bugs/7/ execption when
    inserting record in DISKLESS mode

    Status: open
    Created: Tue Sep 24, 2013 09:35 AM UTC by yymailb
    Last Updated: Wed Sep 25, 2013 01:42 AM UTC
    Owner: nobody

    fastdb version: 3.73
    env: windows 8, visual c++ 2010 sp1

    desc:
    I wrote a short program which using fastdb in DISKLESS mode (#define
    DISKLESS_CONFIGURATION 1), which created a db instance with
    initialsize = 1000000000(1GB), and repeat insert a small struct to db.
    fastdb throw a FileLimitExeeded execption when the No.198181 record is
    inserting.
    I've checked the call stack, the execption throwed in function inline
    void dbDatabase::extend(offs_t size), where the parameter size =
    0xfff82000, maybe something wrong?

    the program is attached.


    Sent from sourceforge.net because you indicated interest in
    https://sourceforge.net/p/fastdb/bugs/7/

    To unsubscribe from further messages, please visit
    https://sourceforge.net/auth/subscriptions/

     

    Related

    Bugs: #7

    • yymailb

      yymailb - 2013-09-25

      I created database and just insert 1 object, db.getDatabaseSize() return 1000341504, db.getUsedDatabaseSize() return 4337664.

       
  • yymailb

    yymailb - 2013-09-25
     

    Last edit: yymailb 2013-09-25
  • yymailb

    yymailb - 2013-09-25

    I created database and just insert 1 object, db.getDatabaseSize() return 1000341504, db.getUsedDatabaseSize() return 4337664.

     

Log in to post a comment.