[AgileWiki] Backup utilities, TKS properties

[Bill La F. <Wil...@Su...>]

In AgileWiki2 there were utilities for dumping and restoring, for the 
whole database or only a single Cabinet. There was also a snap utility 
which would dump only present state. This latter was used to help manage 
the size of the database, as well as being very helpful when we wanted 
to change the low-level datastructures. AgileWiki2 was built on 
BerklyDB, which handled the transaction processing, log files and recovery.

In AgileWiki3 we are not using BerklyDB and all the files are at least 
an order of magnitude smaller. And between conversion to Java and 
writing our own database, we've seen a speed increase which sometimes 
reaches 3 orders of magnitude. But then, this is a custom database 
designed just for AgileWiki. However, both TKS (the AgileWiki3 database) 
and BerklyDB are btrees and support transactions.

So we are creating our own log file in AgileWiki3. And the first utility 
will be to rebuild the database from scratch using only a set of log files.

The log files consist of a series of transactions, but the content is 
comprehensive. In most cases, the entire request event (XML document) 
from the client is captured. As well, every database property change is 
recorded. However, updates to the inverted indexes are not captured 
separately. The trick here is that TKS properties with a name beginning 
with "_" are inverted properties.

Hmm. Another thing to keep in mind is that only the doc property can 
contain line feeds. This property is then given special handling in the 
log file.

Oh and then there are property tables. Properties with names ending with 
"-" have as their value a series of key/value pairs. And the order is 
significant (think list, not set). Of course, inverted table properties 
are supported as well, though in this case the values are not used in 
the inverted table.

Bill