[CZT-Devel] new section manager - comments?

[Leo F. <leo...@ne...>]

Hi guys,

Andrius and I have been working this week on the manager and we just created a branch. Soon we will merge changes 
to the main trunk. Here is an update on what's been happening :-)

---- 1) in general 

We are nearly there in the implementation of the new section manager: its main difference is that is calculates 
dependencies correctly keeping track of both dependants (downward) and dependencies (upward). 
Thanks for the comments - and Petra, thanks for the link to the cztsecman paper: it kept us in check on the facilities envisaged.

We are not finished yet, as some tests still need updating. And there is a small problem with LatexMarkupFunction: the most
difficult one of all to keep track of dependencies within the transaction. Parser tests pass, except for ZEves ones, as we have 
some unconventional cases (e.g., files with multiple sections or sections not in synch with the file), that need updating.

We branched the main trunk into 

http://czt.svn.sourceforge.net/viewvc/czt/branches/tsm/

Please have a look at SectionManager.java in session and in the commands giving your comments…
This is changing and not ready yet.

---- 1.1) using it

For the user,  we tried to keep the protocol just as before with get/puts. So, most of the code around is the same.
Command implementations must change, however. Most places of use do not need to change, unless they had
misused the section manager (e.g., taken advantage of duplicate keys; or used command exceptions to get keys; etc).
When keys are removed, all dependants/dependencies are removed as well.

For IDEs, we envisage having a section manager per project, where all files involved within a project are handled
by a single manager. This improves performance by reducing rework, and with the dependencies tracking keeps 
all in check for soundness. We are also going to use time stamps for consistency.

---- 1.3) the future - concurrency

The overall plan is to have concurrent section management (!!!) The plan is once we understand the dependencies graph well,
we would be able to predict what could be split into different threads so that we could speed up some processes, in particular
parsing/lexing, which takes most time. VC generation can be quite expensive as well, and would benefit from this.

This is just an idea for now, but the hope is to factor / extend the manager to allow concurrency as an option.

Best,
Leo & Andrius 

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
---- 2) in detail

We plan to document this in a small document, as well as in various comments along the implementation.
We will also have a bunch of examples of both usage and implementation.

---- 2.1) the dependencies

This is the tricky bit, and we are documenting each command/key and its dependencies soon. It will be part of
the documentation of each command - or perhaps even within the section manager itself.

Overall the protocol/implementation works well, is simple (enough), and conservative/strict. For instance, we do not allow:
	a) key duplication (e.g., two transactions trying to calculate the same key?); 
	b) key removal of ongoing transaction (e.g., interference? - removing a key removes dependants/dependencies);
	c) new transaction on cached keys (e.g., would lead to key duplication); 
	d) and there are other cases...

The only tricky case is LatexMarkupFunction (LMF), and somewhat the OpTable. 

Like in kernels, the main trouble lies within bootstrapping, and that is parsing. When parsing ZSect, its OpTable gets
created and triggers parsing of parent tables, which is fine. Trouble with LMF is that it is lexing, so during lexing of a 
section, we might start parsing of its parent. And when that involves Spec or ZSect with cycles, things get messy. 
Another messy case is the one with multiple sections in a file or even repeated/redeclared sections….. we are getting there.

The main trouble with redeclared sections is that it is caught as a type error (!!!) despite the fact it manifest itself as a lexing problem!
And since we are no longer allowing for an inconsistent section manager (e.g., key duplication), sorting this one out is getting hard. 

---- 2.2) the protocol

get(Key)
	startTransaction(key)
	try
	{
		compute(key)
	}
	catch (CmdExpt e)
	{
		cancel(key)
	}

put(Key, Value)
	startTransaction(key)
	endTransaction(key, value)

Command.compute
	if okay
		endTransaction(key, value)
	else
		cancelTransaction(key) or
		throw new CommandException().

Just like in some of the parser visitors, we compute implicit dependencies within get(Key).
We also allow explicit dependencies (e.g., those not existing dependencies that do not appear
in the get protocol; or top-level dependencies needed like Spec/ZSect; etc). 

The idea here is that more dependencies will make the manager more sensitive to change, which is okay.
Later we will fine tune those to see the best balance. We are keeping it very conservative in this calc.

There are various consistency checks to ensure the soundness of transactions and dependencies. 

--- 2.3) the implementation

 We added transactions to the manager in the follow way:

a) we keep a stack of transactions by (key, index) pair
b) we keep a list of dependency keys

The transaction stack contains the user given key plus a pointer to the dependency keys list. 
That key will depend on all elements in b) from the index to the size. The dependencies list
grows at get(Key) and shrinks at endTransaction when the last transaction finishes (e.g., it
contains the transitive dependencies of a key).

Upon endTransaction we store the mapping from key->value within the manager and calculate
the dependency maps in both directions. There are two methods getDependants(key) and getDependencies(key).

When removing a key (e.g., on reset()), we remove all dependants and dependencies. We ensure you
cannot remove dependants for a key of an ongoing transaction. On reset/cloning  we require that there are no ongoing transactions. 

A small addition is the method on keys to consider permanent (e.g., those that don't get removed on reset).
Originally, it was on all those involving prelude or toolkit in names. It remains this way, but we give the user the chance to
register if there are any more.

For debugging, there is a detailed tracing mechanism that dumps how the transactions and dependencies progress. 

-------- 

Phew…. please give us comments / suggestions / opinions…. 

as soon as we clear all the test cases, we will merge these changes to the main trunk.

Best,
Leo & Andrius