Error releasing deferred frees using 1.0.6 against a 1.0.4 journal

[thompsonbry]

The following stack trace can appear when attempting to commit data using 1.0.6 against a journal which had been written on by 1.0.4.

Note: This exception is only observed when using the recycler (minReleaseAge GT 0) (versus session protection).

Caused by: java.lang.RuntimeException: Problem with entry at -176050709462618
        at com.bigdata.rwstore.RWStore.freeDeferrals(RWStore.java:3692) ~[bigdata-1.0.6.jar:na]
        at com.bigdata.rwstore.RWStore.checkDeferredFrees(RWStore.java:2517) ~[bigdata-1.0.6.jar:na]
        at com.bigdata.journal.RWStrategy.checkDeferredFrees(RWStrategy.java:673) ~[bigdata-1.0.6.jar:na]
        at com.bigdata.journal.AbstractJournal.commitNow(AbstractJournal.java:2423) ~[bigdata-1.0.6.jar:na]
        at com.bigdata.journal.AbstractJournal.commit(AbstractJournal.java:2285) ~[bigdata-1.0.6.jar:na]

[thompsonbry]

In 1.0.4, the journal was not pruning the commit record index when processing deferred frees in an attempt to work around [1] and only the commit records from "lastDeferredReleaseTime" forward were processed.

When the journal is then opened under 1.0.6, all historical commit records are considered (the start time is 0L). This causes the RWStore to consider commit records whose deferred frees have already been release, which is the root cause of the thrown exception.

Martyn is working on a migration utility which will simply prune the commit record index, removing all entries up to the "lastDeferredReleaseTime". This utility could then be applied to a 1.0.4 journal to make it compatible with 1.0.6.

[1] http://sourceforge.net/apps/trac/bigdata/ticket/440 (BTree can not be cast to Name2Addr)

[thompsonbry]

Commit of journal migration utility by Martyn. The utility prunes entries in the commit record index whose deferred deletes have already been released. This utility is necessary when upgrading from bigdata 1.0.4 to 1.0.6 if deferred deletes were enabled for the journal (minReleaseAge > 0). When minReleaseAge is ZERO (0), the database uses session protection mode and the older commit record entries will not cause a problem.

This utility might also be necessary when moving from 1.1.0 to 1.1.1.

Committed revision r6008.

[thompsonbry]

This issue and the existence of the migration utility have also been documented on the wiki.

See https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=DataMigration

[thompsonbry]

Modified RWStore to NOT recycle commit records starting at commitTime := 0L. Instead, it starts recycling from lastDeferredReleaseTime+1. This is correct and also handles the case where the commit record index was not pruned when deferred deletes were recycled (bigdata 1.0.4).

Commented AbstractJournal#getCommitRecord?(long) to indicate that we must continue to check against the lastReleaseTime and refuse to return a commit record from the commit record index which is older than the lastReleaseTime.

Committed Revision r6009 (1.0.x and 1.1.x maintenance branches)

[thompsonbry]

The data migration utility is:

java ... com.bigdata.journal.VerifyCommitRecordIndex <journal-file>

The only argument is the name of the journal file. The utility will prune the commit record index up to the lastDeferredReleaseTime and then commit the journal. After running this utility, the exception associated with this ticket should no longer be thrown.