[Bigdata-developers] Group commit status

[Bryan T. <br...@sy...>]

Kirk,

As you know, I've picked up the support for group commit (aka concurrent
unisolated writers). This is ticket http://trac.bigdata.com/ticket/566.
The support is based on the existing job/task-oriented group commit
mechanisms used by the scale-out architecture.  The work is happening in
the NSS_GROUP_COMMIT branch of the SourceForge git repository.

I am finding a some issues related to the servlet api - specifically when
os.flush() or writer.flush() is invoked.  The problem is that flush()
commits the response and we need to be much more careful about when this is
invoked since it can now be called before the atomic commit point. This is
because flush() was being called from within the body of the task once it
calls conn.commit(), but the actual group commit does not occur until the
task exits its call() method. The conn.commit() method flushes the sail's
assertion buffers, but the task has to actually finish executing (return
from its call() method) before the group commit logic takes over,
checkpoints its indices, and allows the task to participate in the next
group commit meld. If you want to understand better what is going on, look
at AbstractTask.doUnisolatedReadWriteTask(), its call to
checkpointIndices(), and the before/after task hooks that are used to
coordinate the job-oriented concurrency control with the group commit logic
in the WriteExecutorService.  Unisolated tasks are required to pre-declare
their locks (which are then ordered to prevent deadlocks) and will run when
those locks are held by (or on behalf of) the task.

The other issue that I need to resolve is how we create/destroy the bigdata
namespaces. This will need to be done using an AbstractApiTask, similar to
the current RegisterIndexTask or DropIndexTask.  That needs to get hooked
into the test suite, HAJournalServer, the servlet life cycle listener, and
possibly BigdataSail.createLTS().  Once that is done, I will be able to
conditionally disable the journal.commit() call inside of
LocalTripleStore.commit() based on whether or not group commit is enabled
in BigdataStatics.  A big caution is that invoking Journal.commit()
directly will go around the isolation mechanisms of group commit.  This can
force partially checkpointed index updates to become visible immediately
rather than when the next group commit rolls around.  Applications that
code to AbstractApiTask.submit() will not have this problem.

@kirk: Since I am out most of tomorrow, I probably won't get a chance to
clear all of these issues up until early next week.  In the meantime, could
you look into developing a test suite for the AbstractApiTask.submit()
layer. This could be based on the existing TestConcurrentRestApiRequests if
you like, and you could also extend the coverage there if you have time.
Those test suites should pass if BigdataStatics.NSS_GROUP_COMMIT is false.
Until I address the issues mentioned above they will fail if that field is
set to true.

Thanks,
Bryan