bigdata-developers — List for bigdata developers

[Bigdata-developers] trac 409, 729 GRAPH X {} construct

[Jeremy J C. <jj...@sy...>]

I have committed test cases, and have commented on the trac items

Jeremy J Carroll
Principal Architect
Syapse, Inc.

Re: [Bigdata-developers] What inferences are expected with OwlAxioms?

[Mike P. <mi...@sy...>]

Jim,

Can you provide a sample data set that demonstrates the problem with the
first query?  What is the correct answer for that query on your data set?

The second query uses an unsupported OWL feature.  Bigdata supports the
RDFS Plus inference profile.  This includes rdfs:subClassOf,
rdfs:subPropertyOf, rdfs:domain, rdfs:range, owl:equivalentClass,
owl:equivalentProperty, owl:sameAs, owl:inverseOf, owl:TransitiveProperty,
owl:SymmetricProperty, owl:InverseFunctionalProperty, and
owl:FunctionalProperty.  Bigdata does not support higher level reasoning,
including owl:someValuesFrom.  In general bigdata is aimed at large scale
knowledge bases that do not require the computational complexity required
by higher level reasoning.

Thanks,
Mike


On 8/21/13 12:34 PM, "Bryan Thompson" <br...@sy...> wrote:

>Copying the developers list. The SPAM filter on source forge is not
>adjustable until we convert the project over to the new project management
>system, which is currently blocking on the correct migration of the trac
>tickets (and pending while I figure out if the new project management
>ticket tracker is going to work out for us).
>
>Please see below.
>Bryan
>
>On 8/21/13 2:23 PM, "Jim Balhoff" <ba...@ne...> wrote:
>
>>Hi,
>>
>>I tried to post this on the Bigdata forum but it is blocked by the
>>Sourceforge spam check.
>>
>>I am using Bigdata 1.2.3 and trying out some of the inference support. Is
>>there any documentation about what inferences can be expected for the
>>OwlAxioms mode? I haven't found much on the wiki covering inference. In
>>my RWStore.properties, I set
>>com.bigdata.rdf.store.AbstractTripleStore.quads=false and
>>com.bigdata.rdf.store.AbstractTripleStore.axiomsClass=com.bigdata.rdf.axi
>>o
>>ms.OwlAxioms.
>>
>>I tried a basic subclass query and don't seem to get any inferences:
>>
>>PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
>>PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
>>PREFIX bone_element: <http://purl.obolibrary.org/obo/UBERON_0001474>
>>SELECT (COUNT(DISTINCT ?term) AS ?count)
>>WHERE
>>{
>>?term rdfs:subClassOf bone_element: .
>>?term rdfs:label ?label .
>>}
>>
>>The result is 22, but when I use a property path rdfs:subClassOf* the
>>result is 1121. I expected them to be the same with OWL inferences.
>>
>>I was hoping that some types of class expressions would be supported,
>>like this:
>>
>>PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
>>PREFIX owl: <http://www.w3.org/2002/07/owl#>
>>PREFIX part_of: <http://purl.obolibrary.org/obo/BFO_0000050>
>>PREFIX head: <http://purl.obolibrary.org/obo/UBERON_0000033>
>>SELECT (COUNT(DISTINCT ?term) AS ?count)
>>WHERE
>>{
>>?term rdfs:subClassOf ?restriction .
>>?restriction owl:onProperty part_of: .
>>?restriction owl:someValuesFrom head: .
>>?term rdfs:label ?label .
>>}
>>
>>The result of this query to Bigdata is 56, but using the ELK reasoner for
>>OWL EL in Protege I find 1697 descendant classes of (part_of some head).
>>
>>Regarding the first (bone_element) query, it seems simple so might I have
>>something configured incorrectly? The explanation does show
>>"includeInferred=true". Are expression queries like the second supported?
>>
>>Thank you,
>>Jim
>>
>>____________________________________________
>>James P. Balhoff, Ph.D.
>>National Evolutionary Synthesis Center
>>2024 West Main St., Suite A200
>>Durham, NC 27705
>>USA
>>
>>
>>
>

Re: [Bigdata-developers] What inferences are expected with OwlAxioms?

[Bryan T. <br...@sy...>]

Copying the developers list. The SPAM filter on source forge is not
adjustable until we convert the project over to the new project management
system, which is currently blocking on the correct migration of the trac
tickets (and pending while I figure out if the new project management
ticket tracker is going to work out for us).

Please see below.
Bryan

On 8/21/13 2:23 PM, "Jim Balhoff" <ba...@ne...> wrote:

>Hi,
>
>I tried to post this on the Bigdata forum but it is blocked by the
>Sourceforge spam check.
>
>I am using Bigdata 1.2.3 and trying out some of the inference support. Is
>there any documentation about what inferences can be expected for the
>OwlAxioms mode? I haven't found much on the wiki covering inference. In
>my RWStore.properties, I set
>com.bigdata.rdf.store.AbstractTripleStore.quads=false and
>com.bigdata.rdf.store.AbstractTripleStore.axiomsClass=com.bigdata.rdf.axio
>ms.OwlAxioms.
>
>I tried a basic subclass query and don't seem to get any inferences:
>
>PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
>PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
>PREFIX bone_element: <http://purl.obolibrary.org/obo/UBERON_0001474>
>SELECT (COUNT(DISTINCT ?term) AS ?count)
>WHERE
>{
>?term rdfs:subClassOf bone_element: .
>?term rdfs:label ?label .
>}
>
>The result is 22, but when I use a property path rdfs:subClassOf* the
>result is 1121. I expected them to be the same with OWL inferences.
>
>I was hoping that some types of class expressions would be supported,
>like this:
>
>PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
>PREFIX owl: <http://www.w3.org/2002/07/owl#>
>PREFIX part_of: <http://purl.obolibrary.org/obo/BFO_0000050>
>PREFIX head: <http://purl.obolibrary.org/obo/UBERON_0000033>
>SELECT (COUNT(DISTINCT ?term) AS ?count)
>WHERE
>{
>?term rdfs:subClassOf ?restriction .
>?restriction owl:onProperty part_of: .
>?restriction owl:someValuesFrom head: .
>?term rdfs:label ?label .
>}
>
>The result of this query to Bigdata is 56, but using the ELK reasoner for
>OWL EL in Protege I find 1697 descendant classes of (part_of some head).
>
>Regarding the first (bone_element) query, it seems simple so might I have
>something configured incorrectly? The explanation does show
>"includeInferred=true". Are expression queries like the second supported?
>
>Thank you,
>Jim
>
>____________________________________________
>James P. Balhoff, Ph.D.
>National Evolutionary Synthesis Center
>2024 West Main St., Suite A200
>Durham, NC 27705
>USA
>
>
>

[Bigdata-developers] FW: [bigdata®] #530: Journal HA

[Bryan T. <br...@sy...>]

All, I have brought back the changes that we have made in
branches/READ_CACHE2 into branches/READ_CACHE in order improve HA
stability in the READ_CACHE branch.  Work on stability and failover
characteristics will continue in branches/READ_CACHE.

Thanks,
Bryan

On 8/14/13 2:12 PM, "bigdata®" <no...@so...> wrote:

>#530: Journal HA
>-------------------------+------------------------------------------------
>--
> Reporter:  thompsonbry  |       Owner:  thompsonbry
>     Type:  enhancement  |      Status:  accepted
> Priority:  major        |   Milestone:  High Availability
>Component:  Journal      |     Version:  BIGDATA_RELEASE_1_1_0
> Keywords:               |
>-------------------------+------------------------------------------------
>--
>
>Comment(by thompsonbry):
>
> Merging changes from branches/READ_CACHE2 back into branches/READ_CACHE.
> These changes are designed to improve the stability and failover
> characteristics of the HA replication cluster.  Work continues on this
> under branches/READ_CACHE2.
>
>Š.
>
> See #530 (Journal HA)
> See #720 (HA3 simultaneous start failures)
>
> Committed Revision r7292.
>
>-- 
>Ticket URL: 
><http://sourceforge.net/apps/trac/bigdata/ticket/530#comment:158>
>bigdata® <http://www.bigdata.com/blog>
>bigdata® is a scale-out storage and computing fabric supporting optional
>transactions, very high concurrency, and very high aggregate IO rates.

Re: [Bigdata-developers] 711 and 704 fix (Protocol issues)

[Bryan T. <br...@sy...>]

Jeremy,

I was able to specify to the eclipse patch tool that it should ignore the 1st path component for the patch. That appears to have produced a reasonable outcome.  (The diff has leading a/ and b/ path prefixes.)

- I tend to be a fanatic about using 'final' whenever a variable is immutable.  Likewise, I like to leave vertical whitespace between methods and common their intention.

- I tend to link the code back to the ticket when fixing a bug or creating a new test/test suite.

- Please update the 2.7 migration ticket to link back to the code change that will be required in ConnegUtil.

Pedantics aside, this looks fine.  I am not sure whether or not we already have some of those methods.

Looks like several excellent finds in there!  Please go ahead and commit.

Thanks,
Bryan


From: Jeremy Carroll <jj...@sy...<mailto:jj...@sy...>>
Date: Friday, August 9, 2013 1:27 PM
To: "Big...@li...<mailto:Big...@li...>" <Big...@li...<mailto:Big...@li...>>
Subject: [Bigdata-developers] 711 and 704 fix (Protocol issues)

I have attached a patch to trac 711 which:
- fixes 711 (POST of application/sparql-query)
- fixes 704 (ASK and json)
- provides tests for both
- provides test framework for 'protocol' tests
- makes it easier to run on-the-wire tests

This message is to ask for review, and any further instructions.

I have two question for the reviewer(s)
 - about a new utility I/O method
 - about a not quite relate change to MiniMime

OVERVIEW

I provide a summary of my changes by reference to the diff on github
https://github.com/jeremycarroll/bigdata/compare/fd7d9e0...trac-711-post
- Actual fixes to:
   704
    ConnegUtil.java
  711
    BigdataRDFContext.java
    QueryServlet.java (with question I/O utility)
    RESTServlet.java
  other
    MiniMime.java (with question not this bug)

- Test enhancement
   TestConneg.java
   AbstractProtocolTest.java
         ExampleProtocolTest.java
         TestAskJsonTrac704.java
         TestPostNotURLEncoded.java
         TestRelease123Protocol.java

- Test launching enhancements
   ProxySuiteHelper.java
   TestProtocolAll.java


- Actual fixes to:
   704
    ConnegUtil.java
https://github.com/jeremycarroll/bigdata/compare/fd7d9e0...d53f287#diff-2
    The new code is lines 66 - 99 with line 92 being the heart of it
    I include line 70 that disables the fix and issues a warning once sesame upgrade is in place


  711
    BigdataRDFContext.java

Missing mime type for update responses

   QueryServlet.java (with question I/O utility)
    RESTServlet.java

Use mime type as well as params for control flow POST requests; and use body for query not parameters, where appropriate.
This does not check for multiply defined queries (e.g. both in body and on parameter) or other bad cases

In QueryServlet there is a method readFully which takes a Reader and returns a String of its content.
https://github.com/jeremycarroll/bigdata/compare/fd7d9e0...d53f287#diff-3 line 67
Do we already have such a method? Should this method be moved to a utility class?


  other
    MiniMime.java (with question not this bug)

I noticed that this class makes assumptions about the absence of whitespace that is permitted by the relevant RFC.
I fixed that in one detail (the content-type) but not in the other detail (the q processing).
This was not part of defect 711 or 704; and there is no motivating test.

- Test enhancement
   TestConneg.java

   Additional test related to 704

   AbstractProtocolTest.java

    Provides fairly low level control of the wire protocol for protocol related tests rather than query related tests.
    The intent is to subclass as follows, and for this class to be largely a black box ….

         ExampleProtocolTest.java
               A very simple example of how to use the superclass.

         TestRelease123Protocol.java
               A set of tests that capture some aspects of the  behavior for queries and sparql update of the current version.
               Note: these caught the missing mime type in BigdataRDFContent

         TestAskJsonTrac704.java
               Simple test for 704


         TestPostNotURLEncoded.java
                 These are the tests for 711

- Test launching enhancements
   ProxySuiteHelper.java
       This provides two methods encapsulating the 'how to launch tests' advice from this week's meeting, in a way that tries to avoid temporary copies and kludges during development.
       Each of the above test classes can be launched stand-alone without change.

   TestProtocolAll.java
         A test suite of all the above tests, that using the glue from the ProxySuiteHelper can also run successfully in TestAll, as well as stand-alone




Jeremy J Carroll
Principal Architect
Syapse, Inc.

[Bigdata-developers] 711 and 704 fix (Protocol issues)

[Jeremy J C. <jj...@sy...>]

I have attached a patch to trac 711 which:
- fixes 711 (POST of application/sparql-query)
- fixes 704 (ASK and json)
- provides tests for both
- provides test framework for 'protocol' tests
- makes it easier to run on-the-wire tests

This message is to ask for review, and any further instructions.

I have two question for the reviewer(s)
 - about a new utility I/O method
 - about a not quite relate change to MiniMime

OVERVIEW

I provide a summary of my changes by reference to the diff on github
https://github.com/jeremycarroll/bigdata/compare/fd7d9e0...trac-711-post
- Actual fixes to:
   704
    ConnegUtil.java
  711
    BigdataRDFContext.java
    QueryServlet.java (with question I/O utility)
    RESTServlet.java
  other
    MiniMime.java (with question not this bug)

- Test enhancement
   TestConneg.java
   AbstractProtocolTest.java
         ExampleProtocolTest.java
         TestAskJsonTrac704.java
         TestPostNotURLEncoded.java
         TestRelease123Protocol.java

- Test launching enhancements
   ProxySuiteHelper.java
   TestProtocolAll.java
          
  
- Actual fixes to:
   704
    ConnegUtil.java
https://github.com/jeremycarroll/bigdata/compare/fd7d9e0...d53f287#diff-2
    The new code is lines 66 - 99 with line 92 being the heart of it
    I include line 70 that disables the fix and issues a warning once sesame upgrade is in place


  711
    BigdataRDFContext.java
   
Missing mime type for update responses 

   QueryServlet.java (with question I/O utility)
    RESTServlet.java

Use mime type as well as params for control flow POST requests; and use body for query not parameters, where appropriate.
This does not check for multiply defined queries (e.g. both in body and on parameter) or other bad cases

In QueryServlet there is a method readFully which takes a Reader and returns a String of its content.
https://github.com/jeremycarroll/bigdata/compare/fd7d9e0...d53f287#diff-3 line 67
Do we already have such a method? Should this method be moved to a utility class? 


  other
    MiniMime.java (with question not this bug)

I noticed that this class makes assumptions about the absence of whitespace that is permitted by the relevant RFC.
I fixed that in one detail (the content-type) but not in the other detail (the q processing).
This was not part of defect 711 or 704; and there is no motivating test.

- Test enhancement
   TestConneg.java

   Additional test related to 704

   AbstractProtocolTest.java

    Provides fairly low level control of the wire protocol for protocol related tests rather than query related tests.
    The intent is to subclass as follows, and for this class to be largely a black box ….

         ExampleProtocolTest.java
               A very simple example of how to use the superclass.

         TestRelease123Protocol.java
               A set of tests that capture some aspects of the  behavior for queries and sparql update of the current version.
               Note: these caught the missing mime type in BigdataRDFContent

         TestAskJsonTrac704.java
               Simple test for 704


         TestPostNotURLEncoded.java
                 These are the tests for 711

- Test launching enhancements
   ProxySuiteHelper.java
       This provides two methods encapsulating the 'how to launch tests' advice from this week's meeting, in a way that tries to avoid temporary copies and kludges during development.
       Each of the above test classes can be launched stand-alone without change.

   TestProtocolAll.java
         A test suite of all the above tests, that using the glue from the ProxySuiteHelper can also run successfully in TestAll, as well as stand-alone




Jeremy J Carroll
Principal Architect
Syapse, Inc.

Re: [Bigdata-developers] Merge of 1.2.x to HA branch

[Bryan T. <br...@sy...>]

A correction to the above message. The target branch is branches/READ_CACHE, not branches/BIGDATA_READ_CACHE.  There branches/BIGDATA_READ_CACHE branch does not exist.

From: Bryan Thompson <br...@sy...<mailto:br...@sy...>>
Date: Friday, August 9, 2013 8:37 AM
To: "Big...@li...<mailto:Big...@li...>" <Big...@li...<mailto:Big...@li...>>
Subject: [Bigdata-developers] Merge of 1.2.x to HA branch

I have just completed this merge. This catches up the HA development branch with the 1.2.x maintenance branch.

I am going to merge the changes from the HA development branch (branches/BIGDATA_READ_CACHE) into the branch where we have been working on some HA failover issues (branches/READ_CACHE2)  prior to bringing the changes in the latter back into the former.

Bryan

[Bigdata-developers] Merge of 1.2.x to HA branch

[Bryan T. <br...@sy...>]

I have just completed this merge. This catches up the HA development branch with the 1.2.x maintenance branch.

I am going to merge the changes from the HA development branch (branches/BIGDATA_READ_CACHE) into the branch where we have been working on some HA failover issues (branches/READ_CACHE2)  prior to bringing the changes in the latter back into the former.

Bryan

Re: [Bigdata-developers] FYGI moved from easy mock to httpclient (trac 704, 711)

[Bryan T. <br...@sy...>]

Sounds good.  Excellent on the TestAll2 stuff.

I did notice that the JSON and CSV tests for SPARQL result sets were (and
are) not enabled because the RemoteRepository is unable to parse them (the
parsers are not present in openrdf 2.6.x).  I am tracking that on the
ticket for a 2.7 migration and will update those tests then to ensure
coverage at that time.

Bryan

On 8/8/13 2:15 PM, "Jeremy J Carroll" <jj...@sy...> wrote:

>I have migrated my tests to use HttpClient as we discussed on the call
>yesterday, and, I think as Bryan suggested, it revealed issues which
>easymock did not reveal Š.
>
>so what felt a bit like busy-work was actually very worthwhile
>
>Oh also, I took Mike's TestAll2 approach and triggered it by a static
>boolean in AbstractIndexManagerTestCase, flipped in the static
>initializer of TestNanoSparqlServerWithProxyIndexManager, so that a
>suite() method can tell whether we are in the stand-alone case or the
>running-the-test-suite case, and do the right thing.
>
>I need to fix the two minor issues revealed by the tests and I will have
>a patch for 704 and 711
>
>More later
>
>Jeremy J Carroll
>Principal Architect
>Syapse, Inc.
>
>
>
>
>--------------------------------------------------------------------------
>----
>Get 100% visibility into Java/.NET code with AppDynamics Lite!
>It's a free troubleshooting tool designed for production.
>Get down to code-level detail for bottlenecks, with <2% overhead.
>Download for free and get started troubleshooting in minutes.
>http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktr
>k
>_______________________________________________
>Bigdata-developers mailing list
>Big...@li...
>https://lists.sourceforge.net/lists/listinfo/bigdata-developers

[Bigdata-developers] FYGI moved from easy mock to httpclient (trac 704, 711)

[Jeremy J C. <jj...@sy...>]

I have migrated my tests to use HttpClient as we discussed on the call yesterday, and, I think as Bryan suggested, it revealed issues which easymock did not reveal ….

so what felt a bit like busy-work was actually very worthwhile

Oh also, I took Mike's TestAll2 approach and triggered it by a static boolean in AbstractIndexManagerTestCase, flipped in the static initializer of TestNanoSparqlServerWithProxyIndexManager, so that a suite() method can tell whether we are in the stand-alone case or the running-the-test-suite case, and do the right thing.

I need to fix the two minor issues revealed by the tests and I will have a patch for 704 and 711

More later

Jeremy J Carroll
Principal Architect
Syapse, Inc.

Re: [Bigdata-developers] trac 704 fixed: tests using easy mock

[Jeremy J C. <jj...@sy...>]

In light of this, I have added the following code to my earlier fix:


	static {
	// Work-around for sesame not handling ask and json (see trac 704)
		
		if (BooleanQueryResultFormat.forMIMEType(BigdataRDFServlet.MIME_SPARQL_RESULTS_JSON)==null) {
			log.warn("Workaround for sesame 2.6 BooleanQueryResultFormat defect no longer needed", new RuntimeException("location of issue"));
		} else {
			final BooleanQueryResultFormat askJsonFormat = BooleanQueryResultFormat.register("SPARQL/JSON",BigdataRDFServlet.MIME_SPARQL_RESULTS_JSON,"srj");

….


		}
	}
	

Jeremy J Carroll
Principal Architect
Syapse, Inc.



On Aug 6, 2013, at 4:40 PM, Jeremy J Carroll <jj...@sy...> wrote:

> From Bryan's note about sesame upgrade I note that the following commit is the fix at the sesame end:
> 
> https://bitbucket.org/openrdf/sesame/commits/12ad5ab457cd34d2ac26d5871cfa7e8769747a12
> 
> 
> Jeremy J Carroll
> Principal Architect
> Syapse, Inc.
> 
> 
> 
> On Aug 6, 2013, at 4:33 PM, Jeremy J Carroll <jj...@sy...> wrote:
> 
>> 
>> Hi I have fixed trac 704 concerning ASK and JSON
>> 
>> Underlying problem was sesame support for this was missing. 
>> I added it to ConnegUtil with:
>> 
>> 
>> 
>> 	static {
>> 	// Work-around for sesame not handling ask and json (see trac 704)
>> 		final BooleanQueryResultFormat askJsonFormat = BooleanQueryResultFormat.register("SPARQL/JSON","application/sparql-results+json","srj");
>> 		BooleanQueryResultWriterRegistry.getInstance().add(new BooleanQueryResultWriterFactory(){
>> 
>> 			@Override
>> 			public BooleanQueryResultFormat getBooleanQueryResultFormat() {
>> 				return askJsonFormat;
>> 			}
>> 
>> 			@Override
>> 			public BooleanQueryResultWriter getWriter(final OutputStream out) {
>> 				return new BooleanQueryResultWriter(){
>> 
>> 					@Override
>> 					public BooleanQueryResultFormat getBooleanQueryResultFormat() {
>> 						return askJsonFormat;
>> 					}
>> 
>> 					@Override
>> 					public void write(boolean arg0) throws IOException {
>> 						String answer = "{ \"head\":{ } , \"boolean\": " +  Boolean.toString(arg0) + " }";
>> 						out.write(answer.getBytes("utf-8"));
>> 						
>> 					}};
>> 			}});
>> 	}
>> 
>> 
>> 
>> The test framework additions allow mocking a HttpRequest and Responses to a servlet, to test for such protocol issues. I used easy mock 3.2, apache 2.0 license, to help do this.
>> 
>> I am running
>> com.bigdata.rdf.sail.TestAll
>> to verify
>> 
>> The changes are in a patch on the trac ticket, and can also be seen in Git at:
>> 
>> https://github.com/jeremycarroll/bigdata/compare/base-for-servlet-testing...trac-704-ask-json
>> 
>> Jeremy
>> 
>> 
>> 
>> Jeremy J Carroll
>> Principal Architect
>> Syapse, Inc.
>> 
>> 
>> 
> 

Re: [Bigdata-developers] trac 704 fixed: tests using easy mock

[Jeremy J C. <jj...@sy...>]

From Bryan's note about sesame upgrade I note that the following commit is the fix at the sesame end:

https://bitbucket.org/openrdf/sesame/commits/12ad5ab457cd34d2ac26d5871cfa7e8769747a12


Jeremy J Carroll
Principal Architect
Syapse, Inc.



On Aug 6, 2013, at 4:33 PM, Jeremy J Carroll <jj...@sy...> wrote:

> 
> Hi I have fixed trac 704 concerning ASK and JSON
> 
> Underlying problem was sesame support for this was missing. 
> I added it to ConnegUtil with:
> 
> 
> 
> 	static {
> 	// Work-around for sesame not handling ask and json (see trac 704)
> 		final BooleanQueryResultFormat askJsonFormat = BooleanQueryResultFormat.register("SPARQL/JSON","application/sparql-results+json","srj");
> 		BooleanQueryResultWriterRegistry.getInstance().add(new BooleanQueryResultWriterFactory(){
> 
> 			@Override
> 			public BooleanQueryResultFormat getBooleanQueryResultFormat() {
> 				return askJsonFormat;
> 			}
> 
> 			@Override
> 			public BooleanQueryResultWriter getWriter(final OutputStream out) {
> 				return new BooleanQueryResultWriter(){
> 
> 					@Override
> 					public BooleanQueryResultFormat getBooleanQueryResultFormat() {
> 						return askJsonFormat;
> 					}
> 
> 					@Override
> 					public void write(boolean arg0) throws IOException {
> 						String answer = "{ \"head\":{ } , \"boolean\": " +  Boolean.toString(arg0) + " }";
> 						out.write(answer.getBytes("utf-8"));
> 						
> 					}};
> 			}});
> 	}
> 
> 
> 
> The test framework additions allow mocking a HttpRequest and Responses to a servlet, to test for such protocol issues. I used easy mock 3.2, apache 2.0 license, to help do this.
> 
> I am running
> com.bigdata.rdf.sail.TestAll
> to verify
> 
> The changes are in a patch on the trac ticket, and can also be seen in Git at:
> 
> https://github.com/jeremycarroll/bigdata/compare/base-for-servlet-testing...trac-704-ask-json
> 
> Jeremy
> 
> 
> 
> Jeremy J Carroll
> Principal Architect
> Syapse, Inc.
> 
> 
> 

[Bigdata-developers] heads up - java issue on Mac java 7

[Jeremy J C. <jj...@sy...>]

Running the bigdata sails test suite resulted in a core dump on Mac OS X 10.8.4 with

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x00007fff888dc351, pid=67214, tid=0
#
# JRE version: 7.0_21-b12
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [libxpc.dylib+0xd351]  _xpc_mach_port_get_context+0x70
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/jeremycarroll/BIGDATA_RELEASE_1_2_0/hs_err_pid67214.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp

as far as I can tell this is Mac specific. I have downgraded to Java 6 and seem to be having better luck

Jeremy


Jeremy J Carroll
Principal Architect
Syapse, Inc.

[Bigdata-developers] trac 704 fixed: tests using easy mock

[Jeremy J C. <jj...@sy...>]

Hi I have fixed trac 704 concerning ASK and JSON

Underlying problem was sesame support for this was missing. 
I added it to ConnegUtil with:



	static {
	// Work-around for sesame not handling ask and json (see trac 704)
		final BooleanQueryResultFormat askJsonFormat = BooleanQueryResultFormat.register("SPARQL/JSON","application/sparql-results+json","srj");
		BooleanQueryResultWriterRegistry.getInstance().add(new BooleanQueryResultWriterFactory(){

			@Override
			public BooleanQueryResultFormat getBooleanQueryResultFormat() {
				return askJsonFormat;
			}

			@Override
			public BooleanQueryResultWriter getWriter(final OutputStream out) {
				return new BooleanQueryResultWriter(){

					@Override
					public BooleanQueryResultFormat getBooleanQueryResultFormat() {
						return askJsonFormat;
					}

					@Override
					public void write(boolean arg0) throws IOException {
						String answer = "{ \"head\":{ } , \"boolean\": " +  Boolean.toString(arg0) + " }";
						out.write(answer.getBytes("utf-8"));
						
					}};
			}});
	}



The test framework additions allow mocking a HttpRequest and Responses to a servlet, to test for such protocol issues. I used easy mock 3.2, apache 2.0 license, to help do this.

I am running
com.bigdata.rdf.sail.TestAll
to verify

The changes are in a patch on the trac ticket, and can also be seen in Git at:

https://github.com/jeremycarroll/bigdata/compare/base-for-servlet-testing...trac-704-ask-json

Jeremy



Jeremy J Carroll
Principal Architect
Syapse, Inc.

[Bigdata-developers] heads up - java issue on Mac java 7

[Jeremy C. <jj...@gm...>]

Running the bigdata sails test suite resulted in a core dump on Mac OS X 10.8.4 with

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x00007fff888dc351, pid=67214, tid=0
#
# JRE version: 7.0_21-b12
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [libxpc.dylib+0xd351]  _xpc_mach_port_get_context+0x70
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/jeremycarroll/BIGDATA_RELEASE_1_2_0/hs_err_pid67214.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp

as far as I can tell this is Mac specific. I have downgraded to Java 6 and seem to be having better luck

Jeremy

[Bigdata-developers] trac 704 fixed: tests using easy mock

[Jeremy C. <jj...@gm...>]

Hi I have fixed trac 704 concerning ASK and JSON

Underlying problem was sesame support for this was missing. 
I added it to ConnegUtil with:



	static {
	// Work-around for sesame not handling ask and json (see trac 704)
		final BooleanQueryResultFormat askJsonFormat = BooleanQueryResultFormat.register("SPARQL/JSON","application/sparql-results+json","srj");
		BooleanQueryResultWriterRegistry.getInstance().add(new BooleanQueryResultWriterFactory(){

			@Override
			public BooleanQueryResultFormat getBooleanQueryResultFormat() {
				return askJsonFormat;
			}

			@Override
			public BooleanQueryResultWriter getWriter(final OutputStream out) {
				return new BooleanQueryResultWriter(){

					@Override
					public BooleanQueryResultFormat getBooleanQueryResultFormat() {
						return askJsonFormat;
					}

					@Override
					public void write(boolean arg0) throws IOException {
						String answer = "{ \"head\":{ } , \"boolean\": " +  Boolean.toString(arg0) + " }";
						out.write(answer.getBytes("utf-8"));
						
					}};
			}});
	}



The test framework additions allow mocking a HttpRequest and Responses to a servlet, to test for such protocol issues. I used easy mock 3.2, apache 2.0 license, to help do this.

I am running
com.bigdata.rdf.sail.TestAll
to verify

The changes are in a patch on the trac ticket, and can also be seen in Git at:

https://github.com/jeremycarroll/bigdata/compare/base-for-servlet-testing...trac-704-ask-json

Jeremy

Re: [Bigdata-developers] FW: [bigdata(R)] #714: Migrate to openrdf 2.7

[Peter A. <ans...@gm...>]

On 7 August 2013 00:48, Bryan Thompson <br...@sy...> wrote:

> FYI, I have created a ticket to act as a placeholder for the openrdf 2.7
> migration. This migration is not yet scheduled.
> Bryan
>
> On 8/6/13 10:43 AM, "bigdata®" <no...@so...> wrote:
>
> >#714: Migrate to openrdf 2.7
> >--------------------------+-----------------------------------------------
> >--
> > Reporter:  thompsonbry   |       Owner:  mrpersonick
> >     Type:  enhancement   |      Status:  new
> > Priority:  major         |   Milestone:  Project management
> >Component:  Bigdata SAIL  |     Version:  BIGDATA_RELEASE_1_2_3
> > Keywords:                |
> >--------------------------+-----------------------------------------------
> >--
> > This is a feature request to migrate to openrdf 2.7. This request has
> >been
> > discussed on the bigdata-developers email list. There have also been some
> > questions about the timing of this migration on the forum.  The migration
> > has not yet been scheduled.
> >
> > 2.7 introduces some changes in tx support/scope that we need to look at.
> > If someone wants to attempt the 2.7 migration I can point at what needs
> >to
> > be done. The main issues are any api semantics changes (and 2.7 may have
> > some, capturing the delta in any classes where we maintain a fork
> > (including the test suites), picking up any new test suites for
> > compliance,  and capturing any missing functionality as openrdf continues
> > to capture the final sparql 1.1 recommendation.
> >
> > One of the other reasons to go to 2.7 is if it supports SPARQL VALUES
> > (rather than BINDINGS).  However, I do not yet have confirmation that it
> > does.
>
>
>
I am using SPARQL VALUES in a 2.7 application, so it is there from 2.7.0
[1].

As far as I know there is only one remaining failing test for SPARQL-1.1 to
get to full compliance with the query and update specs [2]

A list of changes for developers is on the documentation site at [3]

Cheers,

Peter

[1] https://openrdf.atlassian.net/browse/SES-1020
[2] https://openrdf.atlassian.net/browse/SES-937
[3]
http://openrdf.callimachus.net/sesame/2.7/docs/articles/upgrade-notes.docbook?view

[Bigdata-developers] FW: [bigdata®] #714: Migrate to openrdf 2.7

[Bryan T. <br...@sy...>]

FYI, I have created a ticket to act as a placeholder for the openrdf 2.7
migration. This migration is not yet scheduled.
Bryan

On 8/6/13 10:43 AM, "bigdata®" <no...@so...> wrote:

>#714: Migrate to openrdf 2.7
>--------------------------+-----------------------------------------------
>--
> Reporter:  thompsonbry   |       Owner:  mrpersonick
>     Type:  enhancement   |      Status:  new
> Priority:  major         |   Milestone:  Project management
>Component:  Bigdata SAIL  |     Version:  BIGDATA_RELEASE_1_2_3
> Keywords:                |
>--------------------------+-----------------------------------------------
>--
> This is a feature request to migrate to openrdf 2.7. This request has
>been
> discussed on the bigdata-developers email list. There have also been some
> questions about the timing of this migration on the forum.  The migration
> has not yet been scheduled.
>
> 2.7 introduces some changes in tx support/scope that we need to look at.
> If someone wants to attempt the 2.7 migration I can point at what needs
>to
> be done. The main issues are any api semantics changes (and 2.7 may have
> some, capturing the delta in any classes where we maintain a fork
> (including the test suites), picking up any new test suites for
> compliance,  and capturing any missing functionality as openrdf continues
> to capture the final sparql 1.1 recommendation.
>
> One of the other reasons to go to 2.7 is if it supports SPARQL VALUES
> (rather than BINDINGS).  However, I do not yet have confirmation that it
> does.
>
> Also, see #593 (Upgrade to Sesame 2.6.10)
>
>-- 
>Ticket URL: <http://sourceforge.net/apps/trac/bigdata/ticket/714>
>bigdata® <http://www.bigdata.com/blog>
>bigdata® is a scale-out storage and computing fabric supporting optional
>transactions, very high concurrency, and very high aggregate IO rates.

Re: [Bigdata-developers] Two new test failures

[Mike P. <mi...@sy...>]

The first one is a test I created yesterday for ticket 693.  Looks like I forgot to check in its data file.

I'll look at the second one now.


From: Bryan Thompson <br...@sy...<mailto:br...@sy...>>
Date: Tuesday, August 6, 2013 5:59 AM
To: "Big...@li...<mailto:Big...@li...>" <Big...@li...<mailto:Big...@li...>>
Cc: Jeremy Carroll <jj...@sy...<mailto:jj...@sy...>>, Mike Personick <mi...@sy...<mailto:mi...@sy...>>
Subject: Two new test failures

Mike/Jeremy,

There are two new test failures in CI.  I suspect that these are related to the recent checkin of the changes to the join optimizer for property paths and possibly to the #709

 1.  com.bigdata.rdf.sail.TestTicket693.testBug1<http://192.168.1.10:8080/job/bigdata-release-1.2.0/3671/testReport/junit/com.bigdata.rdf.sail/TestTicket693/testBug1/>
 2.  com.bigdata.rdf.sparql.ast.optimizers.TestASTStaticJoinOptimizer.test_nestedOptionals03<http://192.168.1.10:8080/job/bigdata-release-1.2.0/3671/testReport/junit/com.bigdata.rdf.sparql.ast.optimizers/TestASTStaticJoinOptimizer/test_nestedOptionals03_2/>

#693 is OneOrMorePath SPARQL property path expression ignored<https://sourceforge.net/apps/trac/bigdata/ticket/693>. The stack trace for the error is:

java.lang.NullPointerException

        at org.openrdf.repository.base.RepositoryConnectionBase.add(RepositoryConnectionBase.java:284)
        at com.bigdata.rdf.sail.TestTicket693.testBug1(TestTicket693.java:154)

The stack trace for TestASTStaticJoinOptimizer.test_nestedOptionals03 is:

junit.framework.AssertionFailedError: Expecting: x, actual=y
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:680)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:542)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:542)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:542)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:542)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:542)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:586)
        at com.bigdata.rdf.sparql.ast.AbstractASTEvaluationTestCase.assertSameAST(AbstractASTEvaluationTestCase.java:172)
        at com.bigdata.rdf.sparql.ast.optimizers.TestASTStaticJoinOptimizer.test_nestedOptionals03(TestASTStaticJoinOptimizer.java:1513)


Is this related to the following ticket?

#709<https://sourceforge.net/apps/trac/bigdata/ticket/709>      select
 ?g { Graph ?g {} } incorrect<https://sourceforge.net/apps/trac/bigdata/ticket/709>

Thanks,

Bryan

[Bigdata-developers] Two new test failures

[Bryan T. <br...@sy...>]

Mike/Jeremy,

There are two new test failures in CI.  I suspect that these are related to the recent checkin of the changes to the join optimizer for property paths and possibly to the #709

 1.  com.bigdata.rdf.sail.TestTicket693.testBug1<http://192.168.1.10:8080/job/bigdata-release-1.2.0/3671/testReport/junit/com.bigdata.rdf.sail/TestTicket693/testBug1/>
 2.  com.bigdata.rdf.sparql.ast.optimizers.TestASTStaticJoinOptimizer.test_nestedOptionals03<http://192.168.1.10:8080/job/bigdata-release-1.2.0/3671/testReport/junit/com.bigdata.rdf.sparql.ast.optimizers/TestASTStaticJoinOptimizer/test_nestedOptionals03_2/>

#693 is OneOrMorePath SPARQL property path expression ignored<https://sourceforge.net/apps/trac/bigdata/ticket/693>. The stack trace for the error is:

java.lang.NullPointerException

        at org.openrdf.repository.base.RepositoryConnectionBase.add(RepositoryConnectionBase.java:284)
        at com.bigdata.rdf.sail.TestTicket693.testBug1(TestTicket693.java:154)

The stack trace for TestASTStaticJoinOptimizer.test_nestedOptionals03 is:

junit.framework.AssertionFailedError: Expecting: x, actual=y
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:680)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:542)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:542)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:542)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:542)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:542)
        at com.bigdata.bop.engine.AbstractQueryEngineTestCase.diff(AbstractQueryEngineTestCase.java:586)
        at com.bigdata.rdf.sparql.ast.AbstractASTEvaluationTestCase.assertSameAST(AbstractASTEvaluationTestCase.java:172)
        at com.bigdata.rdf.sparql.ast.optimizers.TestASTStaticJoinOptimizer.test_nestedOptionals03(TestASTStaticJoinOptimizer.java:1513)


Is this related to the following ticket?

#709<https://sourceforge.net/apps/trac/bigdata/ticket/709>      select ?g { Graph ?g {} } incorrect<https://sourceforge.net/apps/trac/bigdata/ticket/709>

Thanks,

Bryan

Re: [Bigdata-developers] 712 fixed

[Jeremy J C. <jj...@sy...>]

On Aug 2, 2013, at 4:38 PM, Bryan Thompson <br...@sy...> wrote:

> Have you run through the following test suites?
> TestBigdataSailWithQuads
> com.bigdata.rdf.sparql.TestAll

Both get the same failures as at the CI page

> If so, please commit to the 1.2.x branch and then monitor CI here [1], which is linked from [2].


The commit is done.

Note I 'squashed' my (local) commits down to a single commit on SVN: to try and behave compatibly with the project

FWIW my three commits were
https://github.com/jeremycarroll/bigdata/compare/base_for_trac_709_712...trac_712_fixed
added 2 test cases exercising the issue 712
simplified test
fixed 712 - SELECT ?x { OPTIONAL { FILTER (false) }}

It makes for a cleaner history, but slightly less informative, to have it squashed

Re: [Bigdata-developers] 712 fixed

[Bryan T. <br...@sy...>]

Sorry, Jeremy.  Been a long day.  Apologies for the misspelling of your name.
Bryan

From: Bryan Thompson <br...@sy...<mailto:br...@sy...>>
Date: Friday, August 2, 2013 7:38 PM
To: Jeremy Carroll <jj...@sy...<mailto:jj...@sy...>>, "Big...@li...<mailto:Big...@li...>" <Big...@li...<mailto:Big...@li...>>
Subject: Re: [Bigdata-developers] 712 fixed

Jerry,

Have you run through the following test suites?

 *   TestBigdataSailWithQuads
 *   com.bigdata.rdf.sparql.TestAll

If so, please commit to the 1.2.x branch and then monitor CI here [1], which is linked from [2].

CI does not run completely green. There are several tests that are known failures and a few that tare stochastic, so you should look at what is failing before you commit and familiarize yourself with that list.   It should be relatively obvious if new failures emerge from the commit.

I certainly agree that case 4 does not look correct.  You can not merge an optional group into a non-optional group, which is what is suggested by that comment.

4. JoinGroup1 [optional=false] { JoinGroup2 [optional=true]  { ... } } -> JoinGroup2 [optional=true]  { ... }

I have Cc'd Mike as he was the person who did most of the work on the empty group optimizer. He might have a different insight into the code vs the comment block.

Thanks,
Bryan

[1] http://www.bigdata.com/hudson-release-1.2.0/lastSuccessful/archive/BIGDATA_RELEASE_1_2_0/ant-build/classes/test/test-results/report/index.html
[2] http://www.bigdata.com/bigdata/blog/

From: Jeremy Carroll <jj...@sy...<mailto:jj...@sy...>>
Date: Friday, August 2, 2013 7:22 PM
To: "Big...@li...<mailto:Big...@li...>" <Big...@li...<mailto:Big...@li...>>
Subject: [Bigdata-developers] 712 fixed


I have added a patch to the ticket, including a test case, if someone else could review the patch and apply to SVN (or simply the review the patch and give me the go ahead)

The problem was in ASTEmptyGroupOptimizer

There is this comment, with a new line added by me ...

            /*
1. JoinGroup1 [optional=false] { JoinGroup2 [optional=false] { ... } } -> JoinGroup2 [optional=false] { ... }
2. JoinGroup1 [optional=true]  { JoinGroup2 [optional=true]  { ... } } -> JoinGroup2 [optional=true]  { ... }
3. JoinGroup1 [optional=true]  { JoinGroup2 [optional=false] { ... } } -> JoinGroup2 [optional=true]  { ... }

This case 4 appears to be misconceived: Jeremy Carroll.
4. JoinGroup1 [optional=false] { JoinGroup2 [optional=true]  { ... } } -> JoinGroup2 [optional=true]  { ... }
             */

and avoiding case 4 fixed the issue

Jeremy J Carroll
Principal Architect
Syapse, Inc.

Re: [Bigdata-developers] 712 fixed

[Bryan T. <br...@sy...>]

Jerry,

Have you run through the following test suites?

 *   TestBigdataSailWithQuads
 *   com.bigdata.rdf.sparql.TestAll

If so, please commit to the 1.2.x branch and then monitor CI here [1], which is linked from [2].

CI does not run completely green. There are several tests that are known failures and a few that tare stochastic, so you should look at what is failing before you commit and familiarize yourself with that list.   It should be relatively obvious if new failures emerge from the commit.

I certainly agree that case 4 does not look correct.  You can not merge an optional group into a non-optional group, which is what is suggested by that comment.

4. JoinGroup1 [optional=false] { JoinGroup2 [optional=true]  { ... } } -> JoinGroup2 [optional=true]  { ... }

I have Cc'd Mike as he was the person who did most of the work on the empty group optimizer. He might have a different insight into the code vs the comment block.

Thanks,
Bryan

[1] http://www.bigdata.com/hudson-release-1.2.0/lastSuccessful/archive/BIGDATA_RELEASE_1_2_0/ant-build/classes/test/test-results/report/index.html
[2] http://www.bigdata.com/bigdata/blog/

From: Jeremy Carroll <jj...@sy...<mailto:jj...@sy...>>
Date: Friday, August 2, 2013 7:22 PM
To: "Big...@li...<mailto:Big...@li...>" <Big...@li...<mailto:Big...@li...>>
Subject: [Bigdata-developers] 712 fixed


I have added a patch to the ticket, including a test case, if someone else could review the patch and apply to SVN (or simply the review the patch and give me the go ahead)

The problem was in ASTEmptyGroupOptimizer

There is this comment, with a new line added by me ...

            /*
1. JoinGroup1 [optional=false] { JoinGroup2 [optional=false] { ... } } -> JoinGroup2 [optional=false] { ... }
2. JoinGroup1 [optional=true]  { JoinGroup2 [optional=true]  { ... } } -> JoinGroup2 [optional=true]  { ... }
3. JoinGroup1 [optional=true]  { JoinGroup2 [optional=false] { ... } } -> JoinGroup2 [optional=true]  { ... }

This case 4 appears to be misconceived: Jeremy Carroll.
4. JoinGroup1 [optional=false] { JoinGroup2 [optional=true]  { ... } } -> JoinGroup2 [optional=true]  { ... }
             */

and avoiding case 4 fixed the issue

Jeremy J Carroll
Principal Architect
Syapse, Inc.

[Bigdata-developers] 712 fixed

[Jeremy J C. <jj...@sy...>]

I have added a patch to the ticket, including a test case, if someone else could review the patch and apply to SVN (or simply the review the patch and give me the go ahead)

The problem was in ASTEmptyGroupOptimizer

There is this comment, with a new line added by me ...

            	/*
1. JoinGroup1 [optional=false] { JoinGroup2 [optional=false] { ... } } -> JoinGroup2 [optional=false] { ... }
2. JoinGroup1 [optional=true]  { JoinGroup2 [optional=true]  { ... } } -> JoinGroup2 [optional=true]  { ... }
3. JoinGroup1 [optional=true]  { JoinGroup2 [optional=false] { ... } } -> JoinGroup2 [optional=true]  { ... }

This case 4 appears to be misconceived: Jeremy Carroll.
4. JoinGroup1 [optional=false] { JoinGroup2 [optional=true]  { ... } } -> JoinGroup2 [optional=true]  { ... }
            	 */

and avoiding case 4 fixed the issue

Jeremy J Carroll
Principal Architect
Syapse, Inc.

[Bigdata-developers] race condition ...

[Jeremy J C. <jj...@sy...>]

I added a new ticket

https://sourceforge.net/apps/trac/bigdata/ticket/708

which is worse I think than earlier issues I have raised.

I hope to find time to do bigdata work late this week early next, and I think my pipeline is probably

708 (race) 704 (json and ask)  697 (sparql 1.1 graph store protocol)

any advice on this last one would be welcome.

I will change my query to workaround for now


Jeremy J Carroll
Principal Architect
Syapse, Inc.