Concurrent access problems

[gunjanv]

Hi All - I'm trying to resolve an occasional but debilitating DB_RUNRECOVERY error. We have two threads accessing the bdb concurrently. the threads runs fine up to around 80k entries, but then at a random entry poop out with a DB_RUNRECOVERY error.

much obliged for any assistance.

thread A: for(i = 0; i < 100000; i++)
        {
        creates a new key/value hash each time
        writes new log entries to the db.
        sleep(random ms upto 20ms)
        }
         updated synch variable WritingIsDone = true
        
thread B. while Not WritingIsDone
        {
        open a ReadUncommitted cursor,
            gets all log keys
        closes cursor
        For each log key found
            {
            check value hash
            delete the key
            }
                sleep(random upto 200ms)
            }
=================================================================
Here's how the db is created:

      // create and open the BDB
      _DbEnv = new BerkeleyDb.Env(EnvCreateFlags.None);
      const Env.OpenFlags envOpenFlags =
        //Env.OpenFlags.InitCDB | Env.OpenFlags.InitMPool | Env.OpenFlags.Create;
        Env.OpenFlags.InitMPool | Env.OpenFlags.Create | Env.OpenFlags.SystemMem | Env.OpenFlags.InitLock;

      _DbEnv.Open(BDBDir, envOpenFlags, 0);    

      _Database = _DbEnv.CreateDatabase(DbCreateFlags.None);

      _DbTree =
        (DbBTree)_Database.Open(null, BDBDir + "\\&quot; + BDB_FILE, null,
                                   BerkeleyDb.DbType.BTree, Db.OpenFlags.ReadUncommitted| Db.OpenFlags.Create, 0);

=================================================================
Here's how the cursor is setup:

        using (BerkeleyDb.DbBTreeCursor cursor = _DbTree.OpenCursor(null, DbBTreeCursor.CreateFlags.ReadUncommitted))
        {
          DbEntry key = DbEntry.Out(new byte[256]);
          DbEntry data = DbEntry.Out(new byte[8192]);

          while (ReadStatus.Success == cursor.Get(ref key, ref data, DbBTreeCursor.GetMode.Next, DbBTreeCursor.ReadFlags.ReadUncommitted))
          {
            string keystr = new string(System.Text.Encoding.ASCII.GetChars(key.Buffer));
            keystr = keystr.Trim('\0');

            if (keystr.StartsWith(keyPrefix))
            {
              string datastr = new string(System.Text.Encoding.ASCII.GetChars(data.Buffer));
              entries[keystr] = datastr.Trim('\0');
            }
            key = DbEntry.Out(new byte[256]);
            data = DbEntry.Out(new byte[8192]);
          }
        } //using closes the cursor

[gunjanv]

Also wanted to add that the same database DbBTree handle is passed to both threads.

[Karl Waclawek]

Have you tried opening the database with the addition flag Db.OpenFlags.ThreadSafe?

I am not sure this will help, since I am already synchronizing on the db handle,
but it is worth a try.

[Karl Waclawek]

Just an unrelated comment: you don't need to allocate a new buffer every time your loop iterates.
Just allocate the key and data buffers once, like

  byte[] keyBuf = new byte[256];
  byte[] dataBuf = new byte[8192];

and then, in the loop, initialize the DbEntry instance on each iteration:

  key = DbEntry.Out(keyBuf);
  data = DbEntry.Out(dataBuf);

 

[gunjanv]

So I tried the Db.OpenFlags.ThreadSafe flag to no avail. It doesn't make sense why it fails after 70-80k entries. What general strategies do you recommend for preventing the database from getting corrupt? I tried syncing every 10000 entries, but it didn't change anything.

Thanks for the optimization, btw.

[Karl Waclawek]

Hmm,

These issues are probably more related to how BDB is used than my .NET adapter.
If you like, you could e-mail me (karl at waclawek dot net) a small self-contained
application which generates the error, and I will have a look.

But you should also try the Berkeley DB support forums at Oracle:
http://forums.oracle.com/forums/forum.jspa?forumID=271
They know better than anyone how BDB works.

Karl