Concurrent access problems

Help
gunjanv
2007-12-14
2013-04-09
  • gunjanv
    gunjanv
    2007-12-14

    Hi All - I'm trying to resolve an occasional but debilitating DB_RUNRECOVERY error. We have two threads accessing the bdb concurrently. the threads runs fine up to around 80k entries, but then at a random entry poop out with a DB_RUNRECOVERY error.

    much obliged for any assistance.

    thread A: for(i = 0; i < 100000; i++)
            {
            creates a new key/value hash each time
            writes new log entries to the db.
            sleep(random ms upto 20ms)
            }
             updated synch variable WritingIsDone = true
            
    thread B. while Not WritingIsDone
            {
            open a ReadUncommitted cursor,
                gets all log keys
            closes cursor
            For each log key found
                {
                check value hash
                delete the key
                }
                    sleep(random upto 200ms)
                }
    =================================================================
    Here's how the db is created:

          // create and open the BDB
          _DbEnv = new BerkeleyDb.Env(EnvCreateFlags.None);
          const Env.OpenFlags envOpenFlags =
            //Env.OpenFlags.InitCDB | Env.OpenFlags.InitMPool | Env.OpenFlags.Create;
            Env.OpenFlags.InitMPool | Env.OpenFlags.Create | Env.OpenFlags.SystemMem | Env.OpenFlags.InitLock;

          _DbEnv.Open(BDBDir, envOpenFlags, 0);    

          _Database = _DbEnv.CreateDatabase(DbCreateFlags.None);

          _DbTree =
            (DbBTree)_Database.Open(null, BDBDir + "\\&quot; + BDB_FILE, null,
                                       BerkeleyDb.DbType.BTree, Db.OpenFlags.ReadUncommitted| Db.OpenFlags.Create, 0);

    =================================================================
    Here's how the cursor is setup:

            using (BerkeleyDb.DbBTreeCursor cursor = _DbTree.OpenCursor(null, DbBTreeCursor.CreateFlags.ReadUncommitted))
            {
              DbEntry key = DbEntry.Out(new byte[256]);
              DbEntry data = DbEntry.Out(new byte[8192]);

              while (ReadStatus.Success == cursor.Get(ref key, ref data, DbBTreeCursor.GetMode.Next, DbBTreeCursor.ReadFlags.ReadUncommitted))
              {
                string keystr = new string(System.Text.Encoding.ASCII.GetChars(key.Buffer));
                keystr = keystr.Trim('\0');

                if (keystr.StartsWith(keyPrefix))
                {
                  string datastr = new string(System.Text.Encoding.ASCII.GetChars(data.Buffer));
                  entries[keystr] = datastr.Trim('\0');
                }
                key = DbEntry.Out(new byte[256]);
                data = DbEntry.Out(new byte[8192]);
              }
            } //using closes the cursor

     
    • gunjanv
      gunjanv
      2007-12-14

      Also wanted to add that the same database DbBTree handle is passed to both threads.

       
      • Karl Waclawek
        Karl Waclawek
        2007-12-14

        Have you tried opening the database with the addition flag Db.OpenFlags.ThreadSafe?

        I am not sure this will help, since I am already synchronizing on the db handle,
        but it is worth a try.

         
    • Karl Waclawek
      Karl Waclawek
      2007-12-15

      Just an unrelated comment: you don't need to allocate a new buffer every time your loop iterates.
      Just allocate the key and data buffers once, like

        byte[] keyBuf = new byte[256];
        byte[] dataBuf = new byte[8192];

      and then, in the loop, initialize the DbEntry instance on each iteration:

        key = DbEntry.Out(keyBuf);
        data = DbEntry.Out(dataBuf);

       

       
    • gunjanv
      gunjanv
      2007-12-18

      So I tried the Db.OpenFlags.ThreadSafe flag to no avail. It doesn't make sense why it fails after 70-80k entries. What general strategies do you recommend for preventing the database from getting corrupt? I tried syncing every 10000 entries, but it didn't change anything.

      Thanks for the optimization, btw.

       
      • Karl Waclawek
        Karl Waclawek
        2007-12-18

        Hmm,

        These issues are probably more related to how BDB is used than my .NET adapter.
        If you like, you could e-mail me (karl at waclawek dot net) a small self-contained
        application which generates the error, and I will have a look.

        But you should also try the Berkeley DB support forums at Oracle:
        http://forums.oracle.com/forums/forum.jspa?forumID=271
        They know better than anyone how BDB works.

        Karl