It looks like Kevin intended to send this to the list, so I'm forwarding it.  Thanks for the reply Kevin, I'll respond separately.

-------- Original Message --------
Subject: re: [Jdbm-general] jdbm2 mailing list; jdbm corruption
Date: Thu, 12 Aug 2010 09:56:48 -0700
From: Kevin Day <kevin@trumpetinc.com>
Organization: Trumpet, Inc.
To: Jim Newsham <jnewsham@referentia.com>


The log file is actually cleared if things are shut down properly (only needed in the event of a crash) - just something to consider - theoretically, you could put the log file in a temp folder or something, but you have to make sure that if a crash occurs, the user immediately re-opens the app.
 
Feasibility for putting the log into the db file is not good at all - it would be a nightmare.
 
I haven't started playing with jdbm2 yet - if it's creating multiple files, I'd be interested if the jdbm2 lead has any comments on that.
 
 
For jdbm1, file corruption is something that we've seen in one very, very rare (and well defined) case.  The latest code in SVN has a fix for the issue (it was related to recoverable write failures - namely if the disk ran out of space at one point, then suddenly had enough space).  I'm pretty sure I commited a patch that fixes this, you can look at the SVN history notes to make sure.
 
In your code, you are comitting every single change (this shouldn't cause problems, I just wanted to point that out)...
 
But outside of that, your code does look solid. 
 
 
Other folks may need to weigh in here (Alex??), but here are my thoughts about how to maybe go about troubleshooting this:
 
Is there any way for you to make backup copies of the db and lg files before you run each iteration, and halt the iterations as soon as you get the exception?  I'm not positive that this will help, but this would at least give the db and log files that are required to make the failure happen. 
 
You should be able to rename the lg file and actually bring up the db file by itself and not get errors.  If that is indeed the case, then there is something about the log file that is resulting in the corruption, and we may be able to track that down.
 
Would you be willing to run the same test on the jdbm2 codebase?  It would interesting to know if the problem exists in both places (if it does, it would give at least a hint of where the problem is coming from).
 
- K
 
 
 
 
----------------------- Original Message -----------------------
  
From: Jim Newsham <jnewsham@referentia.com>
To: jdbm-general@lists.sourceforge.net
Cc: 
Date: Wed, 11 Aug 2010 18:00:39 -1000
Subject: [Jdbm-general] jdbm2 mailing list; jdbm corruption
  

I just came across jdbm and am considering using it (or jdbm2) for a
project.

I see that jdbm2 was recently forked from jdbm, and is hosted elsewhere
(http://code.google.com/p/jdbm2/).  However, I couldn't find any mailing
list at that location.  Is this list the place for jdbm2 questions, or
did I miss something?

On the subject of jdbm/jdbm2...  I need a structured file format which
is robust (will not corrupt), and ideally just a single file (since from
the user perspective it will be an opaque configuration file that he may
want to manipulate -- i.e., backup, copy, move).  I see that jdbm can do
what I need with 2 files rather than 1, which is actually acceptable
(not 100% ideal, but not too bad; on a side note, what is the technical
feasibility of stuffing the data and log into a single file, without
foregoing corruption-safety).

I considered using jdbm2 since it's more active/recent, but when I tried
out a simple test I discovered that it creates 8 database files
immediately -- ouch, not good.  So I'm giving up on jdbm2 for now.  
Unless jdbm2 can be made to use just 1 or 2 files.

Back to jdbm, I wrote a very simple corruption test, and it seemed to
run fine for a few hours but finally started throwing Errors.  The test
consists of (1) a class called CrashTest2 which opens a db, iterates all
keys/values, then repeatedly puts random key/value pairs forever (the
"domain" of possible keys is limited to 10,000, so puts will eventually
overwrite); and (2) a class called Driver which starts the db access
class in another process (Runtime.exec()), waits for some random amount
of time, then kills the process -- over and over, forever.

Stack trace follows:

Exception in thread "main" java.lang.Error: CRITICAL: page header magic
for block 11603 not OK 0
        at jdbm.recman.PageHeader.<init>(PageHeader.java:74)
        at jdbm.recman.DataPage.<init>(DataPage.java:63)
        at jdbm.recman.DataPage.getDataPageView(DataPage.java:75)
        at
jdbm.recman.PhysicalRowIdManager.allocNew(PhysicalRowIdManager.java:201)
        at
jdbm.recman.PhysicalRowIdManager.alloc(PhysicalRowIdManager.java:175)
        at
jdbm.recman.PhysicalRowIdManager.insert(PhysicalRowIdManager.java:81)
        at jdbm.recman.BaseRecordManager.insert(BaseRecordManager.java:225)
        at
jdbm.recman.CacheRecordManager.insert(CacheRecordManager.java:158)
        at
jdbm.recman.CacheRecordManager.insert(CacheRecordManager.java:141)
        at jdbm.htree.HTree.createInstance(HTree.java:94)
        at scratch.jdbm.CrashTest2.getTree(CrashTest2.java:52)
        at scratch.jdbm.CrashTest2.check(CrashTest2.java:66)
        at scratch.jdbm.CrashTest2.main(CrashTest2.java:42)


And the code follows:

public class CrashTest2 {

  private static final String TREE_NAME = "tree";

  private static int commitCount;

  public static void main(String... args) throws Exception {
    Runtime.getRuntime().addShutdownHook(new Thread() {
      public void run() {
        System.out.println(String.format("terminating after approx. %s
commits", commitCount));
      }
    });

    RecordManager recordManager =
RecordManagerFactory.createRecordManager("crashtest2");
    long startNanos = System.nanoTime();
    check(recordManager);
    long endNanos = System.nanoTime();
    System.out.println(String.format("checked file in %.3fms",
(endNanos - startNanos) / 1000000.0));
    runTest(recordManager);
  }

  private static HTree getTree(RecordManager recordManager, String
treeName) throws Exception {
    long recordId = recordManager.getNamedObject(treeName);
    if (recordId == 0) {
      HTree tree = HTree.createInstance(recordManager);
      recordManager.setNamedObject("htree", tree.getRecid());
      return tree;
    }
    else {
      return HTree.load(recordManager, recordId);
    }
  }

  private static void check(RecordManager recordManager) throws Exception {
    // we're not really checking that everything we expect is here...
    // we just traverse the entire tree to make sure everything is readable
    int keyCount = 0;
    int valueCount = 0;
    HTree tree = getTree(recordManager, TREE_NAME);
    FastIterator iter = tree.keys();
    String key;
    while ((key = (String) iter.next()) != null) {
      keyCount++;
      String value = (String) tree.get(key);
      if (value == null) {
        throw new RuntimeException("null value");
      }
    }
    iter = tree.keys();
    while (iter.next() != null) {
      valueCount++;
    }
    if (keyCount != valueCount) {
      throw new RuntimeException("expected the same number of keys and
values");
    }
  }

  private static void runTest(RecordManager recordManager) throws
Exception {
    HTree tree = getTree(recordManager, TREE_NAME);

    Random random = new Random();
    while (true) {
      int key = random.nextInt(10000);
      tree.put((Integer) key, (Integer) key);
      recordManager.commit();
      commitCount++;
    }
  }

}

public class Driver {

  private static final Executor executor = Executors.newCachedThreadPool();

  public static void main(String... args) throws Exception {
    String command = "java -classpath build\\classes;lib\\jdbm-1.0.jar
scratch.jdbm.CrashTest2";
    while (true) {
      Process process = Runtime.getRuntime().exec(command);
      drain(process.getInputStream());
      drain(process.getErrorStream());
      sleepRandomly(200L, 500L);
      process.destroy();
      process.waitFor();
    }
  }

  private static void drain(final InputStream in) {
    executor.execute(new Runnable() {
      public void run() {
        int c;
        try {
          while ((c = in.read()) != -1) {
            System.out.print((char) c);
          }
        }
        catch(IOException ioe) {
          ioe.printStackTrace();
        }
      }
    });
  }

  private static void sleepRandomly(long wait, long range) {
    final long sleepTime = wait + (long) (Math.random() * range);
    try {
      Thread.sleep(sleepTime);
    }
    catch(InterruptedException ie) {
      // do nothing
    }
  }

}

Thanks and regards,
Jim


------------------------------------------------------------------------------
This SF.net email is sponsored by

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev
_______________________________________________
Jdbm-general mailing list
Jdbm-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jdbm-general