[Jdbm-general] jdbm2 mailing list; jdbm corruption

[Jim N. <jne...@re...>]

I just came across jdbm and am considering using it (or jdbm2) for a 
project.

I see that jdbm2 was recently forked from jdbm, and is hosted elsewhere 
(http://code.google.com/p/jdbm2/).  However, I couldn't find any mailing 
list at that location.  Is this list the place for jdbm2 questions, or 
did I miss something?

On the subject of jdbm/jdbm2...  I need a structured file format which 
is robust (will not corrupt), and ideally just a single file (since from 
the user perspective it will be an opaque configuration file that he may 
want to manipulate -- i.e., backup, copy, move).  I see that jdbm can do 
what I need with 2 files rather than 1, which is actually acceptable 
(not 100% ideal, but not too bad; on a side note, what is the technical 
feasibility of stuffing the data and log into a single file, without 
foregoing corruption-safety).

I considered using jdbm2 since it's more active/recent, but when I tried 
out a simple test I discovered that it creates 8 database files 
immediately -- ouch, not good.  So I'm giving up on jdbm2 for now.  
Unless jdbm2 can be made to use just 1 or 2 files.

Back to jdbm, I wrote a very simple corruption test, and it seemed to 
run fine for a few hours but finally started throwing Errors.  The test 
consists of (1) a class called CrashTest2 which opens a db, iterates all 
keys/values, then repeatedly puts random key/value pairs forever (the 
"domain" of possible keys is limited to 10,000, so puts will eventually 
overwrite); and (2) a class called Driver which starts the db access 
class in another process (Runtime.exec()), waits for some random amount 
of time, then kills the process -- over and over, forever.

Stack trace follows:

Exception in thread "main" java.lang.Error: CRITICAL: page header magic 
for block 11603 not OK 0
         at jdbm.recman.PageHeader.<init>(PageHeader.java:74)
         at jdbm.recman.DataPage.<init>(DataPage.java:63)
         at jdbm.recman.DataPage.getDataPageView(DataPage.java:75)
         at 
jdbm.recman.PhysicalRowIdManager.allocNew(PhysicalRowIdManager.java:201)
         at 
jdbm.recman.PhysicalRowIdManager.alloc(PhysicalRowIdManager.java:175)
         at 
jdbm.recman.PhysicalRowIdManager.insert(PhysicalRowIdManager.java:81)
         at jdbm.recman.BaseRecordManager.insert(BaseRecordManager.java:225)
         at 
jdbm.recman.CacheRecordManager.insert(CacheRecordManager.java:158)
         at 
jdbm.recman.CacheRecordManager.insert(CacheRecordManager.java:141)
         at jdbm.htree.HTree.createInstance(HTree.java:94)
         at scratch.jdbm.CrashTest2.getTree(CrashTest2.java:52)
         at scratch.jdbm.CrashTest2.check(CrashTest2.java:66)
         at scratch.jdbm.CrashTest2.main(CrashTest2.java:42)


And the code follows:

public class CrashTest2 {

   private static final String TREE_NAME = "tree";

   private static int commitCount;

   public static void main(String... args) throws Exception {
     Runtime.getRuntime().addShutdownHook(new Thread() {
       public void run() {
         System.out.println(String.format("terminating after approx. %s 
commits", commitCount));
       }
     });

     RecordManager recordManager = 
RecordManagerFactory.createRecordManager("crashtest2");
     long startNanos = System.nanoTime();
     check(recordManager);
     long endNanos = System.nanoTime();
     System.out.println(String.format("checked file in %.3fms", 
(endNanos - startNanos) / 1000000.0));
     runTest(recordManager);
   }

   private static HTree getTree(RecordManager recordManager, String 
treeName) throws Exception {
     long recordId = recordManager.getNamedObject(treeName);
     if (recordId == 0) {
       HTree tree = HTree.createInstance(recordManager);
       recordManager.setNamedObject("htree", tree.getRecid());
       return tree;
     }
     else {
       return HTree.load(recordManager, recordId);
     }
   }

   private static void check(RecordManager recordManager) throws Exception {
     // we're not really checking that everything we expect is here...
     // we just traverse the entire tree to make sure everything is readable
     int keyCount = 0;
     int valueCount = 0;
     HTree tree = getTree(recordManager, TREE_NAME);
     FastIterator iter = tree.keys();
     String key;
     while ((key = (String) iter.next()) != null) {
       keyCount++;
       String value = (String) tree.get(key);
       if (value == null) {
         throw new RuntimeException("null value");
       }
     }
     iter = tree.keys();
     while (iter.next() != null) {
       valueCount++;
     }
     if (keyCount != valueCount) {
       throw new RuntimeException("expected the same number of keys and 
values");
     }
   }

   private static void runTest(RecordManager recordManager) throws 
Exception {
     HTree tree = getTree(recordManager, TREE_NAME);

     Random random = new Random();
     while (true) {
       int key = random.nextInt(10000);
       tree.put((Integer) key, (Integer) key);
       recordManager.commit();
       commitCount++;
     }
   }

}

public class Driver {

   private static final Executor executor = Executors.newCachedThreadPool();

   public static void main(String... args) throws Exception {
     String command = "java -classpath build\\classes;lib\\jdbm-1.0.jar 
scratch.jdbm.CrashTest2";
     while (true) {
       Process process = Runtime.getRuntime().exec(command);
       drain(process.getInputStream());
       drain(process.getErrorStream());
       sleepRandomly(200L, 500L);
       process.destroy();
       process.waitFor();
     }
   }

   private static void drain(final InputStream in) {
     executor.execute(new Runnable() {
       public void run() {
         int c;
         try {
           while ((c = in.read()) != -1) {
             System.out.print((char) c);
           }
         }
         catch(IOException ioe) {
           ioe.printStackTrace();
         }
       }
     });
   }

   private static void sleepRandomly(long wait, long range) {
     final long sleepTime = wait + (long) (Math.random() * range);
     try {
       Thread.sleep(sleepTime);
     }
     catch(InterruptedException ie) {
       // do nothing
     }
   }

}

Thanks and regards,
Jim