Re: [Bigdata-developers] Results of SPARQL compliance tests on bigdata scale-out

[<dav...@no...>]

On the subject of the dataset tests…

I had a quick look at those a few weeks ago because they are also run as part of the ‘rdf.sail’ unit test suite.  It seemed that the test manifest did not include the graphs to be loaded. So, unless we derive this information from elsewhere, I would expect all the dataset tests that should return a non-empty result set to fail.

David

From: ext Mike Personick [mailto:mi...@sy...]
Sent: Monday, August 30, 2010 9:05 PM
To: Levine Brian (Nokia-MS/Boston); 'big...@li...'
Subject: Re: [Bigdata-developers] Results of SPARQL compliance tests on bigdata scale-out

Brian,

Also, since you're in the scale-out quads query mindset, have you given any more thought to a scale-out quads query performance benchmark test that we can use in the refactor? Like a quads version of BSBM or something?

Thanks,
Mike

________________________________
From: Bri...@no... <Bri...@no...>
To: Mike Personick; big...@li... <big...@li...>
Sent: Mon Aug 30 14:34:39 2010
Subject: RE: [Bigdata-developers] Results of SPARQL compliance tests on bigdata scale-out
Thanks for the detailed response Mike!

=b


From: ext Mike Personick [mailto:mi...@sy...]
Sent: Monday, August 30, 2010 2:54 PM
To: Levine Brian (Nokia-MS/Boston); big...@li...
Subject: RE: [Bigdata-developers] Results of SPARQL compliance tests on bigdata scale-out

Brian,

The dataset test cases use the class “org.openrdf.repository.dataset.DatasetRepository” as a wrapper around the BigdataSailRepository.  The DatasetRepository class allows you to load data into the repository from a query itself rather than invoking explicit load operations and then running a query afterwards.  There is some incompatibility between the Sesame DatasetRepository class and the BigdataSailRepository that causes the query-time data load to never occur.  We’ve never bothered to track down the cause of this because it’s never been important to anyone and we don’t recommend using the Sesame DatasetRepository wrapper class for concurrency reasons.  If you look closely at those dataset test cases you will see that the ones that “succeed” are the ones where success happens to mean no results.  Also if you load the data in manually via a load operation and then perform the same query everything works fine.  We can take a closer look at this during the quads query refactor if this feature is important to you.

The iterator is not progressing warning is an obscure concurrency issue deep in the bidata core that is on Bryan’s plate to eventually track down and eliminate.  This has been around for a while and has not caused any actual errors that I am aware of.

The ISOLATABLE_INDICES issue has to do with full read/write transactions, which as you know are not supported in scale-out.  You are doing the right thing running the database in unisolated mode for the TCK, that is the only way you would be able to get it to pass.  However this is not what you’d want to do in production – there you’d want to segregate your reads and writes, do your writes against the unisolated indices and do your reads against read-only views.

Thanks,
Mike

________________________________
From: Bri...@no... [mailto:Bri...@no...]
Sent: Monday, August 30, 2010 11:44 AM
To: big...@li...
Subject: [Bigdata-developers] Results of SPARQL compliance tests on bigdata scale-out

Hi all,

Last week I ran the Sesame SPARQL compliance tests against bigdata scale-out. This email includes the results of those tests.  The attached TAR file contains maven surefire output (TXT, XML and HTML) for each configuration—one directory per config.  Also included in each directory is a file called failed.txt which lists the tests that failed for that configuration.  I haven’t looked at the failure cases in detail yet.

Systap folks, if you could comment on these results in general and especially the issues I’ve raised prefixed by “Systap:”, I’d appreciate it.

-brian

Summary:

The quads-ascii-noinline configuration most accurately duplicates the configuration of the scale-up compliance test. A total of 13 tests reported errors in this configuration.  Of those, 8 tests were dataset tests which bigdata apparently can’t handle. These tests are purposely filtered out in the scale-up compliance test. Note that these same 8 tests do indeed fail in the scale-up tests when that filter is disabled. The remaining scale-out errors are in 5 graph-based compliance tests.

Systap: Could you provide some background on the dataset test case problem?  What is the limitation in bigdata that causes this functionality to be unsupported?

Description of the test:

The BigdataSparqlTest is a JUnit test included with bigdata.  It extends Sesame’s SPARQLQueryTest which is an implementation of the W3C RDF compliance tests. This test is meant to run against a scale-up instance of bigdata.

To test scale-out, we cloned BigdataSparqlTest and created ScaleoutSparqlTest. This is essentially identical to BigdataSparqlTest except for the following:

•         SPARQLQueryTest tears down and creates a new repository for each test in the suite.  This was too expensive for the scale-out test (and I’m not sure that it even works programmatically) and so a single repository was created and used for an entire test run. However, a new repository (different namespace) was used for each of the 4 configurations.

•         SPARQLQueryTest also closes and reopens the RepositoryConnection for each test.  This started flooding the log with Zookeeper errors (I didn’t spend a lot of time researching why this happens). So ScaleoutSparqlTest maintains a single connection and does a connection.clear() between each test to remove all statements.

•         BigdataSparqlTest identifies a number of test cases that fail due to the recent changes for inlining of literals. For the scale-up case, inlining programmatically disabled when one of these test cases is encountered.  This is done by setting an override property when the repository is created. Since ScaleoutSparqlTest does not create a new repository for each run, this was not possible and those test cases were allowed to run (and fail). Note that one of the test configurations (see below) disables inlining for all test cases to ensure that these test cases pass when inlining is disabled (which they do).

These failures are a result of doing lexical comparisons of literals (.e.g. “01” vs “1”). Since inlining stores the canonical form of the literal, these comparisons fail. It’s debatable as to whether we’ll care about this in production.

•         Four configurations (property settings) were tested. For each config, I’ve included the number of tests that failed. In each case, 8 of the failures are dataset-related test cases:

o   Quads enabled, ASCII collator, inline test excluded: 13 errors

o   Quads enabled, ICU collator, inline tests excluded: 14 errors. Note: using the ICU collator causes the normalization-01 test to fail.

o   Quads enabled, ICU collator, inline tests included: 29 errors

o   Quads disabled, ASCII collator, inline tests included: 32 errors

The first two configurations are probably the most relevant for our purposes.


Additional notes/questions:



•         These tests were run against a scale-out instance running on a single machine (a/k/a single-node cluster) and against an 8-node cluster (2 CS, 5DS, 1 infra). Test results were identical except for execution times.

•         During the course of these tests, there were many of the following warnings written to the log:

WARN : com.bigdata.relation.accesspath.BlockingBuffer$BlockingIterator._hasNext(BlockingBuffer.java:1920): Iterator is not progressing: ntries=298, elapsed=2003ms : BlockingIterator{ open=true, futureIsDone=true, bufferIsOpen=false, nextE=false}



This occurred on both the 1-node and 8-node configurations.

Systap:  Does this indicate a performance issue?  A configuration issue? Is this at all related to issuing un-isolated scale-out queries (see next item).



•         The repository connection was created using a SAIL that wrapped an AbstractTripleStore created with ITx.UNISOLATED. Consequently, a warning regarding issuing an “un-isolated scale-out query” was seen for each query which appeared to be benign. Comments in the code indicate that this is indeed benign, but not recommended.

Systap: Not sure what the proper workaround is for this.  If we instantiated an AbstractTripleStore with a known commit timestamp, we’d have a read-only store and therefore a read-only connection which would not allow the test to write out the test triples. Setting the ISOLATABLE_INDICES override property is not supported for scale-out due a cast of an IIndexManager to Journal in BigdataSail.




Brian Levine
Principal Software Engineer
Services/Ovi Cloud
Nokia