bigdata-developers — List for bigdata developers

[Bigdata-developers] FW: Build failed in Jenkins: bigdata-release-1.2.0 #3841

[Bryan T. <br...@sy...>]

Ah. Another JDK 7 issue.  I need to change the jenkins version for it to
succeed.


We have also had a problem with jenkins double-counting test failures.

Bryan


On 8/28/13 5:03 PM, "Bryan Thompson" <br...@sy...> wrote:

>See <http://192.168.1.10:8080/job/bigdata-release-1.2.0/3841/changes>
>
>Changes:
>
>[thompsonbry] Deprecated the old StaticFrontier class.
>
>Refactored the new StaticFrontier2 class so that the time required to
>copy in the new frontier and the time to sort it show up clearly in the
>profiler and can be clearly distinguished.
>
>[thompsonbry] javadoc
>
>[thompsonbry] Modified the thread-local scheduler to reuse the same
>backing array to sort the thread-local frontier in each round.
>
>[thompsonbry] Added a managed array abstraction and replaced the static
>frontier implementation with that abstraction.
>
>The code paths that compact and sort the frontier have been modified to
>reuse an array that grows as necessary.  This should reduce the overhead
>associated with memory allocation for the frontier.
>
>I still need to optimize this for the thread-local frontier.
>
>I still need to use a parallel sort for the frontiers (other than the
>thread-local one, which already breaks the sort into N threads since each
>per-thread frontier is sorted independently).
>
>------------------------------------------
>[...truncated 58749 lines...]
>    [junit]
>    [junit] ID: http://www.dajobe.org/foaf.rdf#i
>    [junit]
>    [junit] ID: http://dig.csail.mit.edu/People/yosi#YES
>    [junit]
>    [junit] ID: http://www.cambridgesemantics.com/people/lee
>    [junit]
>    [junit] ID: http://handtwerk.de/foaf.rdf#arne
>    [junit]
>    [junit] ID: http://dbpedia.org/resource/Yochai_Benkler
>    [junit]
>    [junit] ID: http://xmlns.com/foaf/0.1/Person
>    [junit]
>    [junit] ID: http://my.opera.com/chaals/xml/foaf#me
>    [junit]
>    [junit] ID: http://people.csail.mit.edu/ryanlee/about#ryanlee
>    [junit]
>    [junit] ID: http://www.w3.org/People/Berners-Lee/card#i
>    [junit] http://www.w3.org/1999/02/22-rdf-syntax-ns#type:
>http://xmlns.com/foaf/0.1/Person
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://users.ecs.soton.ac.uk/mc/mcfoaf.rdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://teole.jfouffa.org/People/Teole/card.rdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://swordfish.rdfweb.org/people/libby/rdfweb/webwho.xrdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://rit.mellon.org/Members/ihf/foaf.rdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://research.microsoft.com/~henrikn/foaf.xml#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://qdos.com/people/tom.xrdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://presbrey.mit.edu/foaf.rdf#presbrey
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://people.w3.org/simon/foaf#i
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://people.csail.mit.edu/ryanlee/about#ryanlee
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://people.csail.mit.edu/psz/foaf.rdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://people.csail.mit.edu/lkagal/foaf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://people.csail.mit.edu/lkagal/foaf.rdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://people.csail.mit.edu/crowell/foaf.rdf#crowell
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://people.apache.org/~oshani/foaf.rdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://norman.walsh.name/knows/who#norman-walsh
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://myopenlink.net/dataspace/person/kidehen#this
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://my.opera.com/howcome/xml/foaf#howcome
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://my.opera.com/chaals/xml/foaf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://inamidst.com/sbp/foaf#Sean
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://id.ecs.soton.ac.uk/person/60
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://id.ecs.soton.ac.uk/person/2686
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://id.ecs.soton.ac.uk/person/1650
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://id.ecs.soton.ac.uk/person/1269
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://hometown.aol.com/chbussler/foaf/chbussler.foaf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://heddley.com/edd/foaf.rdf#edd
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://eikeon.com/foaf.rdf#eikeon
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://dig.csail.mit.edu/People/yosi#YES
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://dig.csail.mit.edu/People/RRS
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://dig.csail.mit.edu/2007/wiki/people/RobertHoffmann#RMH
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://dig.csail.mit.edu/2007/wiki/people/JoeLambda#JL
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://dbpedia.org/resource/Tim_Bray
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://dbpedia.org/resource/John_Seely_Brown
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://dbpedia.org/resource/John_Markoff
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://dbpedia.org/resource/John_Klensin
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://dbpedia.org/resource/John_Gage
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://danbri.org/foaf.rdf#danbri
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://bblfish.net/people/henry/card#me
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x2
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x1
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.w3.org/People/karl/karl-foaf.xrdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.w3.org/People/Jacobs/contact.rdf#IanJacobs
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.w3.org/People/EM/contact#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.w3.org/People/djweitzner/foaf#DJW
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.w3.org/People/Connolly/#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.w3.org/People/Berners-Lee/card#edd
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.w3.org/People/Berners-Lee/card#dj
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.w3.org/People/Berners-Lee/card#cm
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.w3.org/People/Berners-Lee/card#amy
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.mindswap.org/2004/owl/mindswappers#Jennifer.Golbeck
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.mindswap.org/2004/owl/mindswappers#Bijan.Parsia
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.lassila.org/ora.rdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.kjetil.kjernsmo.net/foaf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.ivan-herman.net/foaf.rdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.isi.edu/~gil/foaf.rdf#me
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.dajobe.org/foaf.rdf#i
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.cs.umd.edu/~hendler/2003/foaf.rdf#jhendler
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.cambridgesemantics.com/people/lee
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://www.aaronsw.com/about.xrdf#aaronsw
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://web.mit.edu/shinnyih/foaf.rdf#
>    [junit] http://xmlns.com/foaf/0.1/knows:
>http://web.mit.edu/ruthdhan/www/foaf.rdf#ruthdhan
>    [junit] http://www.w3.org/2000/01/rdf-schema#label: "Tim Berners-Lee"
>    [junit]
>    [junit] ID: http://dbpedia.org/resource/John_Gage
>    [junit]
>    [junit] ID: http://myopenlink.net/dataspace/person/kidehen#this
>    [junit]
>    [junit] ID: http://dig.csail.mit.edu/People/RRS
>    [junit]
>    [junit] ID: http://people.w3.org/simon/foaf#i
>    [junit] http://www.w3.org/1999/02/22-rdf-syntax-ns#type:
>http://xmlns.com/foaf/0.1/Person
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x13
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x12
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x11
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x10
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x9
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x8
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x7
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x6
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x5
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x4
>    [junit] http://xmlns.com/foaf/0.1/knows: _:node1833bq6t5x3
>    [junit] http://www.w3.org/2000/01/rdf-schema#label: "Simon J.
>Hernandez"
>    [junit]
>    [junit] ID: http://www.w3.org/People/djweitzner/foaf#DJW
>    [junit]
>    [junit] ID: http://axel.deri.ie/~axepol/foaf.rdf#me
>    [junit]
>    [junit] ID: http://torrez.us/who#elias
>    [junit]
>    [junit] ID: node1833bq6t5x4
>    [junit]
>    [junit] ID: node1833bq6t5x3
>    [junit]
>    [junit] ID: node1833bq6t5x2
>    [junit]
>    [junit] ID: node1833bq6t5x1
>    [junit]
>    [junit] ID: node1833bq6t5x8
>    [junit]
>    [junit] ID: http://www.w3.org/People/EM/contact#me
>    [junit]
>    [junit] ID: http://id.ecs.soton.ac.uk/person/60
>    [junit]
>    [junit] ID: node1833bq6t5x7
>    [junit]
>    [junit] ID: node1833bq6t5x6
>    [junit]
>    [junit] ID: http://people.csail.mit.edu/psz/foaf.rdf#me
>    [junit]
>    [junit] ID: node1833bq6t5x5
>    [junit]
>    [junit] ID: http://web.mit.edu/shinnyih/foaf.rdf#
>    [junit]
>    [junit] ID: node1833bq6t5x11
>    [junit]
>    [junit] ID: node1833bq6t5x10
>    [junit]
>    [junit] ID: node1833bq6t5x13
>    [junit]
>    [junit] ID: node1833bq6t5x12
>    [junit]
>    [junit] ID: http://norman.walsh.name/knows/who#norman-walsh
>    [junit]
>    [junit] ID: http://dbpedia.org/resource/Tim_Bray
>    [junit]
>    [junit] ID: node1833bq6t5x9
>    [junit]
>    [junit] ID: http://dblp.l3s.de/d2r/page/authors/Christian_Bizer
>    [junit]
>    [junit] ID: http://dbpedia.org/resource/James_Gosling
>    [junit]
>    [junit] ID: http://semedia.deit.univpm.it/people/christian/foaf.rdf#me
>    [junit]
>    [junit] ID: http://data.boab.info/david/foaf.rdf#me
>    [junit]
>    [junit] ID: http://teole.jfouffa.org/People/Teole/card.rdf#me
>    [junit]
>    [junit] ID: http://www.w3.org/People/Jacobs/contact.rdf#IanJacobs
>    [junit]
>    [junit] ID: http://www.isi.edu/~gil/foaf.rdf#me
>    [junit]
>    [junit] ID: http://www.w3.org/People/Berners-Lee/card#edd
>    [junit]
>    [junit] ID: http://hometown.aol.com/chbussler/foaf/chbussler.foaf#me
>    [junit]
>    [junit] ID: http://www.w3.org/People/Berners-Lee/card#cm
>    [junit]
>    [junit] ID: http://people.apache.org/~oshani/foaf.rdf#me
>    [junit]
>    [junit] ID: http://dig.csail.mit.edu/2007/wiki/people/JoeLambda#JL
>    [junit]
>    [junit] ID: http://people.csail.mit.edu/lkagal/foaf#me
>    [junit]
>    [junit] ID: http://id.ecs.soton.ac.uk/person/1269
>    [junit]
>    [junit] ID: http://www.anjeve.de/foaf.rdf#AnjaJentzsch
>    [junit]
>    [junit] ID: http://research.microsoft.com/~henrikn/foaf.xml#me
>    [junit]
>    [junit] ID: http://b4mad.net/FOAF/goern.rdf#goern
>    [junit]
>    [junit] ID: http://eikeon.com/foaf.rdf#eikeon
>    [junit]
>    [junit] ID: http://inamidst.com/sbp/foaf#Sean
>    [junit]
>    [junit] ID: http://www.ibiblio.org/hhalpin/#me
>    [junit]
>    [junit] ID: http://my.opera.com/howcome/xml/foaf#howcome
>    [junit]
>    [junit] ID: http://id.ecs.soton.ac.uk/person/2686
>    [junit]
>    [junit] ID: http://davelevy.info/foaf.rdf#me
>    [junit]
>    [junit] ID: http://www.w3.org/People/Berners-Lee/card#dj
>    [junit]
>    [junit] ID: http://rit.mellon.org/Members/ihf/foaf.rdf#me
>    [junit]
>    [junit] ID: http://www.lassila.org/ora.rdf#me
>    [junit]
>    [junit] ID: http://www.w3.org/People/Connolly/#me
>    [junit]
>    [junit] ID: http://dbpedia.org/resource/John_Klensin
>    [junit]
>    [junit] ID: http://crschmidt.net/foaf.rdf#crschmidt
>    [junit]
>    [junit] ID: http://qdos.com/people/tom.xrdf#me
>    [junit]
>    [junit] ID:
>http://purl.org/captsolo/semweb/foaf-captsolo.rdf#Uldis_Bojars
>    [junit]
>    [junit] Tests run: 50, Failures: 0, Errors: 0, Time elapsed: 20.215
>sec
>
>clean-sparql-test-suite:
>     [echo] "clearing: /tmp/sparql-*"
>[junitreport] Processing
><http://192.168.1.10:8080/job/bigdata-release-1.2.0/ws/BIGDATA_RELEASE_1_2
>_0/ant-build/classes/test/test-results/TESTS-TestSuites.xml> to
>/tmp/null753061698
>[junitreport] Loading stylesheet
>jar:file:/usr/java/apache-ant-1.8.2/lib/ant-junit.jar!/org/apache/tools/an
>t/taskdefs/optional/junit/xsl/junit-frames.xsl
>[junitreport] : Error! The first argument to the non-static Java function
>'replace' is not a valid object reference.
>[junitreport] : Error! Could not compile stylesheet
>[junitreport] : Fatal Error! Could not compile stylesheet Cause: Cannot
>convert data-type 'void' to 'reference'.
>[junitreport] Failed to process
><http://192.168.1.10:8080/job/bigdata-release-1.2.0/ws/BIGDATA_RELEASE_1_2
>_0/ant-build/classes/test/test-results/TESTS-TestSuites.xml>
>
>BUILD FAILED
><http://192.168.1.10:8080/job/bigdata-release-1.2.0/ws/BIGDATA_RELEASE_1_2
>_0/build.xml>:1764: The following error occurred while executing this
>line:
><http://192.168.1.10:8080/job/bigdata-release-1.2.0/ws/BIGDATA_RELEASE_1_2
>_0/build.xml>:2154: Errors while applying transformations: Fatal error
>during transformation
>
>Total time: 62 minutes 58 seconds
>FATAL: 3566911
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566911
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566912
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566912
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566912
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566914
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566915
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566915
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566915
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566915
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566916
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566915
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566916
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566915
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>FATAL: 3566915
>com.bigdata.service.jini.JiniFederation.executorService1
>com.bigdata.service.DataService$DataServiceFederationDelegate.isServiceRea
>dy(DataService.java:562): Store manager not open - will shutdown.
>Build step 'Invoke Ant' marked build as failure
>Archiving artifacts
>Performing Post build task...
>Could not match :JUNIT RUN COMPLETE  : False
>Logical operation result is FALSE
>Skipping script  : ~/rsync.sh release-1.2.0
>
>rm -rf /var/tmp/zookeeper
>END OF POST BUILD TASK         : 0
>Recording test results

Re: [Bigdata-developers] Java 7?

[Bryan T. <br...@sy...>]

Another big advantage of Java 7 is that the G1 garbage collector finally
appears to work.  This means that you can start to use larger heaps.  I
have not yet felt out the rough spots here, but that is an exciting
development.
Bryan

On 8/28/13 4:45 PM, "Peter Ansell" <ans...@gm...> wrote:

>It would only be an issue in the case you say. From a users point of view
>not much changes, comparatively, which is the main reason for not doing
>it from my impressions.
>
>It will be nice to see the performance boosts from the new methods.
>
>--
>Peter
>
>On 29/08/2013, at 6:34 AM, Jeremy J Carroll <jj...@sy...> wrote:
>
>> presumably it isn't an issue for us if the sesame code remains java 6
>>compatible?
>> A minor detail is that we may mistakenly suggest non-compatible bug
>>fixes to themŠ
>> 
>> Jeremy J Carroll
>> Principal Architect
>> Syapse, Inc.
>> 
>> 
>> 
>> On Aug 28, 2013, at 1:31 PM, Bryan Thompson <br...@sy...> wrote:
>> 
>>> Thing thing which is forcing this issue is work that we are doing on
>>>graph
>>> mining on RDF graphs.  Threading is a big issue there.  Access to AIO,
>>> parallelized sorts, and fork/join pools could really help out as well.
>>> This would in turn be a good opportunity to move the platform to take
>>> advantage of those features in the QueryEngine.  As originally
>>>designed,
>>> we need to use unbounded thread pools because all IOs can block.  A
>>> fork/join pattern might produce significant performance gains.  This is
>>> easier to test in the context of the RDF graph mining since that code
>>>is
>>> new, but I think that this would apply to query as well.
>>> 
>>> Thanks,
>>> Bryan
>>> 
>>> On 8/28/13 4:27 PM, "Peter Ansell" <ans...@gm...> wrote:
>>> 
>>>> 
>>>> On 29/08/2013, at 5:03 AM, Bryan Thompson <br...@sy...> wrote:
>>>> 
>>>>> Any thoughts on switching to Java 7 as a requirement?  So far we have
>>>>> maintained Java 6 compatibility.  However, there are new threading
>>>>> patterns in Java 7 that could be very useful.  There is also the
>>>>> asynchronous file IO support.  Java 7 had some very bad initial
>>>>> releases, but seems to be Ok now and Oracle has sunset Java 6.
>>>>> 
>>>>> Thanks,
>>>>> Bryan
>>>> 
>>>> The Sesame API and core is indefinitely going to stay at Java6
>>>> compatibility from what I can tell. My Java7 patches to add support
>>>>for
>>>> autocloseable were vetoed by the main companies backing the other
>>>> developers as not worthwhile. That, and native support for BCP47 were
>>>>the
>>>> main drivers for updating the API in my view. The threading and file
>>>>io
>>>> changes don't directly impact the core enough to convince them of it
>>>>as
>>>> far as I can tell.
>>>> 
>>>> Peter
>>> 
>>> 
>>> 
>>>------------------------------------------------------------------------
>>>------
>>> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
>>> Discover the easy way to master current and previous Microsoft
>>>technologies
>>> and advance your career. Get an incredible 1,500+ hours of step-by-step
>>> tutorial videos with LearnDevNow. Subscribe today and save!
>>> 
>>>http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clk
>>>trk
>>> _______________________________________________
>>> Bigdata-developers mailing list
>>> Big...@li...
>>> https://lists.sourceforge.net/lists/listinfo/bigdata-developers
>> 

Re: [Bigdata-developers] Java 7?

[Peter A. <ans...@gm...>]

It would only be an issue in the case you say. From a users point of view not much changes, comparatively, which is the main reason for not doing it from my impressions.

It will be nice to see the performance boosts from the new methods.

--
Peter

On 29/08/2013, at 6:34 AM, Jeremy J Carroll <jj...@sy...> wrote:

> presumably it isn't an issue for us if the sesame code remains java 6 compatible?
> A minor detail is that we may mistakenly suggest non-compatible bug fixes to them…
> 
> Jeremy J Carroll
> Principal Architect
> Syapse, Inc.
> 
> 
> 
> On Aug 28, 2013, at 1:31 PM, Bryan Thompson <br...@sy...> wrote:
> 
>> Thing thing which is forcing this issue is work that we are doing on graph
>> mining on RDF graphs.  Threading is a big issue there.  Access to AIO,
>> parallelized sorts, and fork/join pools could really help out as well.
>> This would in turn be a good opportunity to move the platform to take
>> advantage of those features in the QueryEngine.  As originally designed,
>> we need to use unbounded thread pools because all IOs can block.  A
>> fork/join pattern might produce significant performance gains.  This is
>> easier to test in the context of the RDF graph mining since that code is
>> new, but I think that this would apply to query as well.
>> 
>> Thanks,
>> Bryan
>> 
>> On 8/28/13 4:27 PM, "Peter Ansell" <ans...@gm...> wrote:
>> 
>>> 
>>> On 29/08/2013, at 5:03 AM, Bryan Thompson <br...@sy...> wrote:
>>> 
>>>> Any thoughts on switching to Java 7 as a requirement?  So far we have
>>>> maintained Java 6 compatibility.  However, there are new threading
>>>> patterns in Java 7 that could be very useful.  There is also the
>>>> asynchronous file IO support.  Java 7 had some very bad initial
>>>> releases, but seems to be Ok now and Oracle has sunset Java 6.
>>>> 
>>>> Thanks,
>>>> Bryan
>>> 
>>> The Sesame API and core is indefinitely going to stay at Java6
>>> compatibility from what I can tell. My Java7 patches to add support for
>>> autocloseable were vetoed by the main companies backing the other
>>> developers as not worthwhile. That, and native support for BCP47 were the
>>> main drivers for updating the API in my view. The threading and file io
>>> changes don't directly impact the core enough to convince them of it as
>>> far as I can tell.
>>> 
>>> Peter
>> 
>> 
>> ------------------------------------------------------------------------------
>> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
>> Discover the easy way to master current and previous Microsoft technologies
>> and advance your career. Get an incredible 1,500+ hours of step-by-step
>> tutorial videos with LearnDevNow. Subscribe today and save!
>> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Bigdata-developers mailing list
>> Big...@li...
>> https://lists.sourceforge.net/lists/listinfo/bigdata-developers
> 

Re: [Bigdata-developers] Java 7?

[Jeremy J C. <jj...@sy...>]

presumably it isn't an issue for us if the sesame code remains java 6 compatible?
A minor detail is that we may mistakenly suggest non-compatible bug fixes to them…

Jeremy J Carroll
Principal Architect
Syapse, Inc.



On Aug 28, 2013, at 1:31 PM, Bryan Thompson <br...@sy...> wrote:

> Thing thing which is forcing this issue is work that we are doing on graph
> mining on RDF graphs.  Threading is a big issue there.  Access to AIO,
> parallelized sorts, and fork/join pools could really help out as well.
> This would in turn be a good opportunity to move the platform to take
> advantage of those features in the QueryEngine.  As originally designed,
> we need to use unbounded thread pools because all IOs can block.  A
> fork/join pattern might produce significant performance gains.  This is
> easier to test in the context of the RDF graph mining since that code is
> new, but I think that this would apply to query as well.
> 
> Thanks,
> Bryan
> 
> On 8/28/13 4:27 PM, "Peter Ansell" <ans...@gm...> wrote:
> 
>> 
>> On 29/08/2013, at 5:03 AM, Bryan Thompson <br...@sy...> wrote:
>> 
>>> Any thoughts on switching to Java 7 as a requirement?  So far we have
>>> maintained Java 6 compatibility.  However, there are new threading
>>> patterns in Java 7 that could be very useful.  There is also the
>>> asynchronous file IO support.  Java 7 had some very bad initial
>>> releases, but seems to be Ok now and Oracle has sunset Java 6.
>>> 
>>> Thanks,
>>> Bryan
>> 
>> The Sesame API and core is indefinitely going to stay at Java6
>> compatibility from what I can tell. My Java7 patches to add support for
>> autocloseable were vetoed by the main companies backing the other
>> developers as not worthwhile. That, and native support for BCP47 were the
>> main drivers for updating the API in my view. The threading and file io
>> changes don't directly impact the core enough to convince them of it as
>> far as I can tell.
>> 
>> Peter
> 
> 
> ------------------------------------------------------------------------------
> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> Discover the easy way to master current and previous Microsoft technologies
> and advance your career. Get an incredible 1,500+ hours of step-by-step
> tutorial videos with LearnDevNow. Subscribe today and save!
> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
> _______________________________________________
> Bigdata-developers mailing list
> Big...@li...
> https://lists.sourceforge.net/lists/listinfo/bigdata-developers

Re: [Bigdata-developers] Java 7?

[Bryan T. <br...@sy...>]

Thing thing which is forcing this issue is work that we are doing on graph
mining on RDF graphs.  Threading is a big issue there.  Access to AIO,
parallelized sorts, and fork/join pools could really help out as well.
This would in turn be a good opportunity to move the platform to take
advantage of those features in the QueryEngine.  As originally designed,
we need to use unbounded thread pools because all IOs can block.  A
fork/join pattern might produce significant performance gains.  This is
easier to test in the context of the RDF graph mining since that code is
new, but I think that this would apply to query as well.

Thanks,
Bryan

On 8/28/13 4:27 PM, "Peter Ansell" <ans...@gm...> wrote:

>
>On 29/08/2013, at 5:03 AM, Bryan Thompson <br...@sy...> wrote:
>
>> Any thoughts on switching to Java 7 as a requirement?  So far we have
>>maintained Java 6 compatibility.  However, there are new threading
>>patterns in Java 7 that could be very useful.  There is also the
>>asynchronous file IO support.  Java 7 had some very bad initial
>>releases, but seems to be Ok now and Oracle has sunset Java 6.
>> 
>> Thanks,
>> Bryan
>
>The Sesame API and core is indefinitely going to stay at Java6
>compatibility from what I can tell. My Java7 patches to add support for
>autocloseable were vetoed by the main companies backing the other
>developers as not worthwhile. That, and native support for BCP47 were the
>main drivers for updating the API in my view. The threading and file io
>changes don't directly impact the core enough to convince them of it as
>far as I can tell.
>
>Peter

Re: [Bigdata-developers] Java 7?

[Peter A. <ans...@gm...>]

On 29/08/2013, at 5:03 AM, Bryan Thompson <br...@sy...> wrote:

> Any thoughts on switching to Java 7 as a requirement?  So far we have maintained Java 6 compatibility.  However, there are new threading patterns in Java 7 that could be very useful.  There is also the asynchronous file IO support.  Java 7 had some very bad initial releases, but seems to be Ok now and Oracle has sunset Java 6.  
> 
> Thanks,
> Bryan

The Sesame API and core is indefinitely going to stay at Java6 compatibility from what I can tell. My Java7 patches to add support for autocloseable were vetoed by the main companies backing the other developers as not worthwhile. That, and native support for BCP47 were the main drivers for updating the API in my view. The threading and file io changes don't directly impact the core enough to convince them of it as far as I can tell.

Peter

Re: [Bigdata-developers] Java 7?

[Jeremy J C. <jj...@sy...>]

well then supporting 6 and 7 costs money (in principle) with no benefit … seems like an easy decision!

Jeremy J Carroll
Principal Architect
Syapse, Inc.



On Aug 28, 2013, at 12:17 PM, Bryan Thompson <br...@sy...> wrote:

> All paying customers have upgraded long ago….
> 
> Bryan
> 
> From: Jeremy Carroll <jj...@sy...>
> Date: Wednesday, August 28, 2013 3:15 PM
> To: Bryan Thompson <br...@sy...>
> Cc: "Big...@li..." <Big...@li...>
> Subject: Re: [Bigdata-developers] Java 7?
> 
> I am generally using java 7 and I am pretty happy. 
> Isn't this more about whether customers are happy to upgrade or not? (i.e. a business rather than a technical decision)
> 
> Jeremy J Carroll
> Principal Architect
> Syapse, Inc.
> 
> 
> 
> On Aug 28, 2013, at 12:03 PM, Bryan Thompson <br...@sy...> wrote:
> 
>> Any thoughts on switching to Java 7 as a requirement?  So far we have maintained Java 6 compatibility.  However, there are new threading patterns in Java 7 that could be very useful.  There is also the asynchronous file IO support.  Java 7 had some very bad initial releases, but seems to be Ok now and Oracle has sunset Java 6.  
>> 
>> Thanks,
>> Bryan
>> ------------------------------------------------------------------------------
>> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
>> Discover the easy way to master current and previous Microsoft technologies
>> and advance your career. Get an incredible 1,500+ hours of step-by-step
>> tutorial videos with LearnDevNow. Subscribe today and save!
>> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk_______________________________________________
>> Bigdata-developers mailing list
>> Big...@li...
>> https://lists.sourceforge.net/lists/listinfo/bigdata-developers
> 

Re: [Bigdata-developers] Java 7?

[Bryan T. <br...@sy...>]

All paying customers have upgraded long ago….

Bryan

From: Jeremy Carroll <jj...@sy...<mailto:jj...@sy...>>
Date: Wednesday, August 28, 2013 3:15 PM
To: Bryan Thompson <br...@sy...<mailto:br...@sy...>>
Cc: "Big...@li...<mailto:Big...@li...>" <Big...@li...<mailto:Big...@li...>>
Subject: Re: [Bigdata-developers] Java 7?

I am generally using java 7 and I am pretty happy.
Isn't this more about whether customers are happy to upgrade or not? (i.e. a business rather than a technical decision)

Jeremy J Carroll
Principal Architect
Syapse, Inc.



On Aug 28, 2013, at 12:03 PM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote:

Any thoughts on switching to Java 7 as a requirement?  So far we have maintained Java 6 compatibility.  However, there are new threading patterns in Java 7 that could be very useful.  There is also the asynchronous file IO support.  Java 7 had some very bad initial releases, but seems to be Ok now and Oracle has sunset Java 6.

Thanks,
Bryan
------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk_______________________________________________
Bigdata-developers mailing list
Big...@li...<mailto:Big...@li...>
https://lists.sourceforge.net/lists/listinfo/bigdata-developers

Re: [Bigdata-developers] Java 7?

[Jeremy J C. <jj...@sy...>]

I am generally using java 7 and I am pretty happy. 
Isn't this more about whether customers are happy to upgrade or not? (i.e. a business rather than a technical decision)

Jeremy J Carroll
Principal Architect
Syapse, Inc.



On Aug 28, 2013, at 12:03 PM, Bryan Thompson <br...@sy...> wrote:

> Any thoughts on switching to Java 7 as a requirement?  So far we have maintained Java 6 compatibility.  However, there are new threading patterns in Java 7 that could be very useful.  There is also the asynchronous file IO support.  Java 7 had some very bad initial releases, but seems to be Ok now and Oracle has sunset Java 6.  
> 
> Thanks,
> Bryan
> ------------------------------------------------------------------------------
> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> Discover the easy way to master current and previous Microsoft technologies
> and advance your career. Get an incredible 1,500+ hours of step-by-step
> tutorial videos with LearnDevNow. Subscribe today and save!
> http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk_______________________________________________
> Bigdata-developers mailing list
> Big...@li...
> https://lists.sourceforge.net/lists/listinfo/bigdata-developers

[Bigdata-developers] Java 7?

[Bryan T. <br...@sy...>]

Any thoughts on switching to Java 7 as a requirement?  So far we have maintained Java 6 compatibility.  However, there are new threading patterns in Java 7 that could be very useful.  There is also the asynchronous file IO support.  Java 7 had some very bad initial releases, but seems to be Ok now and Oracle has sunset Java 6.

Thanks,
Bryan

Re: [Bigdata-developers] CBD limits

[Bryan T. <br...@sy...>]

Peter, thanks for mentioning this. The implementations of describe do not overlap, but I am curious to see how they provide for limits on describe iterations. Bryan

On Aug 26, 2013, at 11:56 PM, "Peter Ansell" <ans...@gm...> wrote:

> Hi Jeremy and Bryan,
> 
> Sesame-2.7.4+ contains an implementation of SCBD (except for
> reification) for DESCRIBE queries. Not sure how that relates to
> BigData yet, but it may need to be tracked in the Sesame-2.7 upgrade.
> The relevant issue is https://openrdf.atlassian.net/browse/SES-1876
> 
> Cheers,
> 
> Peter
> 
> On 27/08/2013, Jeremy J Carroll <jj...@sy...> wrote:
>> 
>> This musing is somewhat premature … the other developers are still working
>> on the earlier version of the system, and it will be clearer what
>> performance issues that we have with DESCRIBE in this usage, once we have
>> more real experience in our use case
>> 
>> 
>> Jeremy J Carroll
>> Principal Architect
>> Syapse, Inc.
>> 
>> 
>> 
>> On Aug 26, 2013, at 3:37 PM, Bryan Thompson <br...@sy...> wrote:
>> 
>>> I am not in principle opposed to your suggestion.  However, what is the
>>> problem with setting the iteration limit low and the statement limit high?
>>> Does that achieve what you want?
>>> 
>>> It might be nice to have a UDF that accepts some parameters that could be
>>> used to limit this.  Perhaps by specifying the name of class that
>>> implements that function in a configuration file?  Time might another
>>> criterion of interest which could be passed in (elapsed time since the
>>> start f the CBD expansion).
>>> 
>>> There was a discussion on the forum (or possibly the email list) when this
>>> was implemented.  The current design emerged from that discussion, but I
>>> recognize that it could be enhanced.
>>> 
>>> Bryan
>>> 
>>> 
>>> On Aug 26, 2013, at 5:33 PM, "Jeremy J Carroll" <jj...@sy...> wrote:
>>> 
>>>> I added a minor defect to trac
>>>> https://sourceforge.net/apps/trac/bigdata/ticket/732
>>>> concerning the CBD describe statement limit being mis-implemented (by one
>>>> word)
>>>> 
>>>> However, I am concerned that this limit may also be mis-designed.
>>>> Here is my story, and this suggests a stronger limit (an OR rather than
>>>> an AND ?)
>>>> 
>>>> The current design is expressed as:
>>>> /**
>>>>     * For iterative {@link DescribeModeEnum}s, this property places a
>>>> limit on
>>>>     * the number of statements that will be accumulated before the
>>>> DESCRIBE
>>>>     * query is cut off, providing that the limit on the maximum #of
>>>> iterations
>>>>     * in the description is also satisfied (the cut off requires that
>>>> both
>>>>     * limits are reached). May be ZERO (0) for NO limit.
>>>>     *
>>>>     * @see #DESCRIBE_MODE
>>>>     * @see #DESCRIBE_ITERATION_LIMIT
>>>>     */
>>>> com.bigdata.rdf.sparql.ast.QueryHints.DESCRIBE_STATEMENT_LIMIT
>>>> 
>>>> 
>>>> 
>>>> We are using pubby [1] to give my developers an easy to use front end
>>>> into the data in bigdata.
>>>> Pubby is a SPARQL to HTML gateway, each page displays a single resource,
>>>> and its DESCIBE-tion from SPARQL, so that you can click through on
>>>> related objects.
>>>> This then implements TBL's "follow-your-nose" by combining pubby and
>>>> bigdata.
>>>> 
>>>> Rationale: my developers are coming from a MySQL legacy system which has
>>>> relatively easy to use tools such as Sequel Pro. It is unrealistic to
>>>> expect many of them to learn SPARQL in order to simply get a feel for the
>>>> data and the ontologies.
>>>> pubby provides a simple HTML view of the linked data inside bigdata, with
>>>> triples represented by web links.
>>>> (Note I did several fixes to get pubby to behave how I expected [2])
>>>> 
>>>> Pubby drives itself by sending SPARQL DESCRIBE requests to the configured
>>>> endpoint. Stickler's SCBD is the algorithm of choice for DESCRIBE
>>>> (usually).
>>>> 
>>>> It is a pretty normal usage pattern to then click on some classes and
>>>> arrive at a class with several instances … 'several' may be several
>>>> million, and such a describe query is then not practicably executable in
>>>> the cheap and cheerful browsing around the data fashion. This then needs
>>>> to be constrained by a describe limit query hint (for example), but the
>>>> current definition - where BOTH  the iteration limit AND the statement
>>>> limit need to be reached is probably not what is required, since they are
>>>> no iterations in the case described. (In fact, the mentioning of SCBD is
>>>> somewhat of a red herring here). I wonder whether simply changing the AND
>>>> into an OR may be a good enough improvement.
>>>> i.e. the iterations bottom out at the iteration limit, and when we reach
>>>> the statement limit the whole computation is terminated.
>>>> 
>>>> 
>>>> Jeremy J Carroll
>>>> Principal Architect
>>>> Syapse, Inc.
>>>> 
>>>> 
>>>> [1] http://wifo5-03.informatik.uni-mannheim.de/pubby/
>>>> [2] https://github.com/cygri/pubby/pull/18
>>>> 
>>>> 
>>>> 
>>>> 
>>>> <ATT00001.c>
>>>> <ATT00002.c>
>> 
>> 

Re: [Bigdata-developers] CBD limits

[Peter A. <ans...@gm...>]

Hi Jeremy and Bryan,

Sesame-2.7.4+ contains an implementation of SCBD (except for
reification) for DESCRIBE queries. Not sure how that relates to
BigData yet, but it may need to be tracked in the Sesame-2.7 upgrade.
The relevant issue is https://openrdf.atlassian.net/browse/SES-1876

Cheers,

Peter

On 27/08/2013, Jeremy J Carroll <jj...@sy...> wrote:
>
> This musing is somewhat premature … the other developers are still working
> on the earlier version of the system, and it will be clearer what
> performance issues that we have with DESCRIBE in this usage, once we have
> more real experience in our use case
>
>
> Jeremy J Carroll
> Principal Architect
> Syapse, Inc.
>
>
>
> On Aug 26, 2013, at 3:37 PM, Bryan Thompson <br...@sy...> wrote:
>
>> I am not in principle opposed to your suggestion.  However, what is the
>> problem with setting the iteration limit low and the statement limit high?
>> Does that achieve what you want?
>>
>> It might be nice to have a UDF that accepts some parameters that could be
>> used to limit this.  Perhaps by specifying the name of class that
>> implements that function in a configuration file?  Time might another
>> criterion of interest which could be passed in (elapsed time since the
>> start f the CBD expansion).
>>
>> There was a discussion on the forum (or possibly the email list) when this
>> was implemented.  The current design emerged from that discussion, but I
>> recognize that it could be enhanced.
>>
>> Bryan
>>
>>
>> On Aug 26, 2013, at 5:33 PM, "Jeremy J Carroll" <jj...@sy...> wrote:
>>
>>> I added a minor defect to trac
>>> https://sourceforge.net/apps/trac/bigdata/ticket/732
>>> concerning the CBD describe statement limit being mis-implemented (by one
>>> word)
>>>
>>> However, I am concerned that this limit may also be mis-designed.
>>> Here is my story, and this suggests a stronger limit (an OR rather than
>>> an AND ?)
>>>
>>> The current design is expressed as:
>>>  /**
>>>      * For iterative {@link DescribeModeEnum}s, this property places a
>>> limit on
>>>      * the number of statements that will be accumulated before the
>>> DESCRIBE
>>>      * query is cut off, providing that the limit on the maximum #of
>>> iterations
>>>      * in the description is also satisfied (the cut off requires that
>>> both
>>>      * limits are reached). May be ZERO (0) for NO limit.
>>>      *
>>>      * @see #DESCRIBE_MODE
>>>      * @see #DESCRIBE_ITERATION_LIMIT
>>>      */
>>> com.bigdata.rdf.sparql.ast.QueryHints.DESCRIBE_STATEMENT_LIMIT
>>>
>>>
>>>
>>> We are using pubby [1] to give my developers an easy to use front end
>>> into the data in bigdata.
>>> Pubby is a SPARQL to HTML gateway, each page displays a single resource,
>>> and its DESCIBE-tion from SPARQL, so that you can click through on
>>> related objects.
>>> This then implements TBL's "follow-your-nose" by combining pubby and
>>> bigdata.
>>>
>>> Rationale: my developers are coming from a MySQL legacy system which has
>>> relatively easy to use tools such as Sequel Pro. It is unrealistic to
>>> expect many of them to learn SPARQL in order to simply get a feel for the
>>> data and the ontologies.
>>> pubby provides a simple HTML view of the linked data inside bigdata, with
>>> triples represented by web links.
>>> (Note I did several fixes to get pubby to behave how I expected [2])
>>>
>>> Pubby drives itself by sending SPARQL DESCRIBE requests to the configured
>>> endpoint. Stickler's SCBD is the algorithm of choice for DESCRIBE
>>> (usually).
>>>
>>> It is a pretty normal usage pattern to then click on some classes and
>>> arrive at a class with several instances … 'several' may be several
>>> million, and such a describe query is then not practicably executable in
>>> the cheap and cheerful browsing around the data fashion. This then needs
>>> to be constrained by a describe limit query hint (for example), but the
>>> current definition - where BOTH  the iteration limit AND the statement
>>> limit need to be reached is probably not what is required, since they are
>>> no iterations in the case described. (In fact, the mentioning of SCBD is
>>> somewhat of a red herring here). I wonder whether simply changing the AND
>>> into an OR may be a good enough improvement.
>>> i.e. the iterations bottom out at the iteration limit, and when we reach
>>> the statement limit the whole computation is terminated.
>>>
>>>
>>> Jeremy J Carroll
>>> Principal Architect
>>> Syapse, Inc.
>>>
>>>
>>> [1] http://wifo5-03.informatik.uni-mannheim.de/pubby/
>>> [2] https://github.com/cygri/pubby/pull/18
>>>
>>>
>>>
>>>
>>> <ATT00001.c>
>>> <ATT00002.c>
>
>

Re: [Bigdata-developers] CBD limits

[Jeremy J C. <jj...@sy...>]

This musing is somewhat premature … the other developers are still working on the earlier version of the system, and it will be clearer what performance issues that we have with DESCRIBE in this usage, once we have more real experience in our use case


Jeremy J Carroll
Principal Architect
Syapse, Inc.



On Aug 26, 2013, at 3:37 PM, Bryan Thompson <br...@sy...> wrote:

> I am not in principle opposed to your suggestion.  However, what is the problem with setting the iteration limit low and the statement limit high? Does that achieve what you want?  
> 
> It might be nice to have a UDF that accepts some parameters that could be used to limit this.  Perhaps by specifying the name of class that implements that function in a configuration file?  Time might another criterion of interest which could be passed in (elapsed time since the start f the CBD expansion).
> 
> There was a discussion on the forum (or possibly the email list) when this was implemented.  The current design emerged from that discussion, but I recognize that it could be enhanced.
> 
> Bryan
> 
> 
> On Aug 26, 2013, at 5:33 PM, "Jeremy J Carroll" <jj...@sy...> wrote:
> 
>> I added a minor defect to trac
>> https://sourceforge.net/apps/trac/bigdata/ticket/732
>> concerning the CBD describe statement limit being mis-implemented (by one word)
>> 
>> However, I am concerned that this limit may also be mis-designed.
>> Here is my story, and this suggests a stronger limit (an OR rather than an AND ?)
>> 
>> The current design is expressed as:
>>  /**
>>      * For iterative {@link DescribeModeEnum}s, this property places a limit on
>>      * the number of statements that will be accumulated before the DESCRIBE
>>      * query is cut off, providing that the limit on the maximum #of iterations
>>      * in the description is also satisfied (the cut off requires that both
>>      * limits are reached). May be ZERO (0) for NO limit.
>>      * 
>>      * @see #DESCRIBE_MODE
>>      * @see #DESCRIBE_ITERATION_LIMIT
>>      */
>> com.bigdata.rdf.sparql.ast.QueryHints.DESCRIBE_STATEMENT_LIMIT
>> 
>> 
>> 
>> We are using pubby [1] to give my developers an easy to use front end into the data in bigdata.
>> Pubby is a SPARQL to HTML gateway, each page displays a single resource, and its DESCIBE-tion from SPARQL, so that you can click through on related objects.
>> This then implements TBL's "follow-your-nose" by combining pubby and bigdata.
>> 
>> Rationale: my developers are coming from a MySQL legacy system which has relatively easy to use tools such as Sequel Pro. It is unrealistic to expect many of them to learn SPARQL in order to simply get a feel for the data and the ontologies. 
>> pubby provides a simple HTML view of the linked data inside bigdata, with triples represented by web links.
>> (Note I did several fixes to get pubby to behave how I expected [2])
>> 
>> Pubby drives itself by sending SPARQL DESCRIBE requests to the configured endpoint. Stickler's SCBD is the algorithm of choice for DESCRIBE (usually).
>> 
>> It is a pretty normal usage pattern to then click on some classes and arrive at a class with several instances … 'several' may be several million, and such a describe query is then not practicably executable in the cheap and cheerful browsing around the data fashion. This then needs to be constrained by a describe limit query hint (for example), but the current definition - where BOTH  the iteration limit AND the statement limit need to be reached is probably not what is required, since they are no iterations in the case described. (In fact, the mentioning of SCBD is somewhat of a red herring here). I wonder whether simply changing the AND into an OR may be a good enough improvement.
>> i.e. the iterations bottom out at the iteration limit, and when we reach the statement limit the whole computation is terminated.
>> 
>> 
>> Jeremy J Carroll
>> Principal Architect
>> Syapse, Inc.
>> 
>> 
>> [1] http://wifo5-03.informatik.uni-mannheim.de/pubby/
>> [2] https://github.com/cygri/pubby/pull/18
>> 
>> 
>> 
>> 
>> <ATT00001.c>
>> <ATT00002.c>

Re: [Bigdata-developers] CBD limits

[Bryan T. <br...@sy...>]

I am not in principle opposed to your suggestion.  However, what is the problem with setting the iteration limit low and the statement limit high? Does that achieve what you want?

It might be nice to have a UDF that accepts some parameters that could be used to limit this.  Perhaps by specifying the name of class that implements that function in a configuration file?  Time might another criterion of interest which could be passed in (elapsed time since the start f the CBD expansion).

There was a discussion on the forum (or possibly the email list) when this was implemented.  The current design emerged from that discussion, but I recognize that it could be enhanced.

Bryan


On Aug 26, 2013, at 5:33 PM, "Jeremy J Carroll" <jj...@sy...<mailto:jj...@sy...>> wrote:

I added a minor defect to trac
https://sourceforge.net/apps/trac/bigdata/ticket/732
concerning the CBD describe statement limit being mis-implemented (by one word)

However, I am concerned that this limit may also be mis-designed.
Here is my story, and this suggests a stronger limit (an OR rather than an AND ?)

The current design is expressed as:
 /**
     * For iterative {@link DescribeModeEnum}s, this property places a limit on
     * the number of statements that will be accumulated before the DESCRIBE
     * query is cut off, providing that the limit on the maximum #of iterations
     * in the description is also satisfied (the cut off requires that both
     * limits are reached). May be ZERO (0) for NO limit.
     *
     * @see #DESCRIBE_MODE
     * @see #DESCRIBE_ITERATION_LIMIT
     */
com.bigdata.rdf.sparql.ast.QueryHints.DESCRIBE_STATEMENT_LIMIT



We are using pubby [1] to give my developers an easy to use front end into the data in bigdata.
Pubby is a SPARQL to HTML gateway, each page displays a single resource, and its DESCIBE-tion from SPARQL, so that you can click through on related objects.
This then implements TBL's "follow-your-nose" by combining pubby and bigdata.

Rationale: my developers are coming from a MySQL legacy system which has relatively easy to use tools such as Sequel Pro. It is unrealistic to expect many of them to learn SPARQL in order to simply get a feel for the data and the ontologies.
pubby provides a simple HTML view of the linked data inside bigdata, with triples represented by web links.
(Note I did several fixes to get pubby to behave how I expected [2])

Pubby drives itself by sending SPARQL DESCRIBE requests to the configured endpoint. Stickler's SCBD is the algorithm of choice for DESCRIBE (usually).

It is a pretty normal usage pattern to then click on some classes and arrive at a class with several instances … 'several' may be several million, and such a describe query is then not practicably executable in the cheap and cheerful browsing around the data fashion. This then needs to be constrained by a describe limit query hint (for example), but the current definition - where BOTH  the iteration limit AND the statement limit need to be reached is probably not what is required, since they are no iterations in the case described. (In fact, the mentioning of SCBD is somewhat of a red herring here). I wonder whether simply changing the AND into an OR may be a good enough improvement.
i.e. the iterations bottom out at the iteration limit, and when we reach the statement limit the whole computation is terminated.


Jeremy J Carroll
Principal Architect
Syapse, Inc.


[1] http://wifo5-03.informatik.uni-mannheim.de/pubby/
[2] https://github.com/cygri/pubby/pull/18




<ATT00001.c>
<ATT00002.c>

[Bigdata-developers] CBD limits

[Jeremy J C. <jj...@sy...>]

I added a minor defect to trac
https://sourceforge.net/apps/trac/bigdata/ticket/732
concerning the CBD describe statement limit being mis-implemented (by one word)

However, I am concerned that this limit may also be mis-designed.
Here is my story, and this suggests a stronger limit (an OR rather than an AND ?)

The current design is expressed as:
 /**
     * For iterative {@link DescribeModeEnum}s, this property places a limit on
     * the number of statements that will be accumulated before the DESCRIBE
     * query is cut off, providing that the limit on the maximum #of iterations
     * in the description is also satisfied (the cut off requires that both
     * limits are reached). May be ZERO (0) for NO limit.
     * 
     * @see #DESCRIBE_MODE
     * @see #DESCRIBE_ITERATION_LIMIT
     */
com.bigdata.rdf.sparql.ast.QueryHints.DESCRIBE_STATEMENT_LIMIT



We are using pubby [1] to give my developers an easy to use front end into the data in bigdata.
Pubby is a SPARQL to HTML gateway, each page displays a single resource, and its DESCIBE-tion from SPARQL, so that you can click through on related objects.
This then implements TBL's "follow-your-nose" by combining pubby and bigdata.

Rationale: my developers are coming from a MySQL legacy system which has relatively easy to use tools such as Sequel Pro. It is unrealistic to expect many of them to learn SPARQL in order to simply get a feel for the data and the ontologies. 
pubby provides a simple HTML view of the linked data inside bigdata, with triples represented by web links.
(Note I did several fixes to get pubby to behave how I expected [2])

Pubby drives itself by sending SPARQL DESCRIBE requests to the configured endpoint. Stickler's SCBD is the algorithm of choice for DESCRIBE (usually).

It is a pretty normal usage pattern to then click on some classes and arrive at a class with several instances … 'several' may be several million, and such a describe query is then not practicably executable in the cheap and cheerful browsing around the data fashion. This then needs to be constrained by a describe limit query hint (for example), but the current definition - where BOTH  the iteration limit AND the statement limit need to be reached is probably not what is required, since they are no iterations in the case described. (In fact, the mentioning of SCBD is somewhat of a red herring here). I wonder whether simply changing the AND into an OR may be a good enough improvement.
i.e. the iterations bottom out at the iteration limit, and when we reach the statement limit the whole computation is terminated.


Jeremy J Carroll
Principal Architect
Syapse, Inc.


[1] http://wifo5-03.informatik.uni-mannheim.de/pubby/
[2] https://github.com/cygri/pubby/pull/18

Re: [Bigdata-developers] [bigdata®] #731: CBD and Update leads to 500

[Jeremy C. <jj...@gm...>]

Yep - that exercises the defect.
See commit r7341

com.bigdata.rdf.sail.webapp.client.HttpException: Status Code=500, Status Line=HTTP/1.1 500 Server Error, Response=PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> 
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> 
PREFIX dc: <http://purl.org/dc/elements/1.1/> 
PREFIX foaf: <http://xmlns.com/foaf/0.1/> 
PREFIX ex: <http://example.org/> 
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> 

INSERT { ex:bob rdfs:label "Bob" . } WHERE { hint:Query hint:describeMode "SCBD" }
java.util.concurrent.ExecutionException: java.lang.Exception: org.openrdf.query.UpdateExecutionException: java.lang.IllegalArgumentException
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
	at java.util.concurrent.FutureTask.get(FutureTask.java:111)
	at com.bigdata.rdf.sail.webapp.QueryServlet.doUpdate(QueryServlet.java:401)
	at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:153)
	at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:206)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:534)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:475)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:929)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:864)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114)
	at org.eclipse.jetty.server.Server.handle(Server.java:352)
	at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
	at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1051)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:590)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:212)
	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:508)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.Exception: org.openrdf.query.UpdateExecutionException: java.lang.IllegalArgumentException
	at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:1089)
	at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:1)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	... 1 more
Caused by: org.openrdf.query.UpdateExecutionException: java.lang.IllegalArgumentException
	at com.bigdata.rdf.sparql.ast.eval.ASTEvalHelper.executeUpdate(ASTEvalHelper.java:1093)
	at com.bigdata.rdf.sail.BigdataSailUpdate.execute2(BigdataSailUpdate.java:152)
	at com.bigdata.rdf.sail.webapp.BigdataRDFContext$UpdateTask.doQuery(BigdataRDFContext.java:1392)
	at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:1062)
	... 6 more
Caused by: java.lang.IllegalArgumentException
	at com.bigdata.rdf.sparql.ast.eval.CBD.<init>(CBD.java:144)
	at com.bigdata.rdf.sparql.ast.eval.ASTEvalHelper.evaluateGraphQuery(ASTEvalHelper.java:589)
	at com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.convertDeleteInsert(AST2BOpUpdate.java:943)
	at com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.convertUpdateSwitch(AST2BOpUpdate.java:417)
	at com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.convertUpdate(AST2BOpUpdate.java:279)
	at com.bigdata.rdf.sparql.ast.eval.ASTEvalHelper.executeUpdate(ASTEvalHelper.java:1087)
	... 9 more

	at com.bigdata.rdf.sail.webapp.client.RemoteRepository.checkResponseCode(RemoteRepository.java:1500)
	at com.bigdata.rdf.sail.webapp.client.RemoteRepository$SparqlUpdate.evaluate(RemoteRepository.java:1139)
	at com.bigdata.rdf.sail.webapp.TestCBD731.executeInsert(TestCBD731.java:173)
	at com.bigdata.rdf.sail.webapp.TestCBD731.testSCBD(TestCBD731.java:159)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at junit.framework.TestCase.runTest(TestCase.java:154)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at junit.framework.TestSuite.runTest(TestSuite.java:208)
	at junit.framework.TestSuite.run(TestSuite.java:203)
	at junit.framework.TestSuite.runTest(TestSuite.java:208)
	at junit.framework.TestSuite.run(TestSuite.java:203)
	at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)


On Aug 26, 2013, at 8:34 AM, Jeremy Carroll <jj...@gm...> wrote:

> 
> 
> BTW I am going to spend 1hr trying to get a clean test case for this …
> 
> I had an idea over the weekend, do a simple INSERT or CONSTRUCT with a describeMode hint, if that tickles the defect which I think it might, then it is cost-effective to add the test
> 
> Jeremy
> 
> 
> On Aug 26, 2013, at 4:03 AM, "bigdata®" <no...@so...> wrote:
> 
>> #731: CBD and Update leads to 500
>> -----------------------------+----------------------------------------------
>> Reporter:  jeremy_carroll  |       Owner:  jeremy_carroll       
>>     Type:  defect          |      Status:  closed               
>> Priority:  critical        |   Milestone:                       
>> Component:  Bigdata SAIL    |     Version:  BIGDATA_RELEASE_1_2_3
>> Resolution:  fixed           |    Keywords:                       
>> -----------------------------+----------------------------------------------
>> Changes (by thompsonbry):
>> 
>> * status:  new => closed
>> * resolution:  => fixed
>> 
>> 
>> Comment:
>> 
>> Proposed change set looks good.
>> 
>> CI is clean.
>> 
>> Ticket is closed.
>> 
>> -- 
>> Ticket URL: <http://sourceforge.net/apps/trac/bigdata/ticket/731#comment:4>
>> bigdata® <http://www.bigdata.com/blog>
>> bigdata® is a scale-out storage and computing fabric supporting optional transactions, very high concurrency, and very high aggregate IO rates.
> 

Re: [Bigdata-developers] [bigdata®] #731: CBD and Update leads to 500

[Jeremy C. <jj...@gm...>]

BTW I am going to spend 1hr trying to get a clean test case for this …

I had an idea over the weekend, do a simple INSERT or CONSTRUCT with a describeMode hint, if that tickles the defect which I think it might, then it is cost-effective to add the test

Jeremy


On Aug 26, 2013, at 4:03 AM, "bigdata®" <no...@so...> wrote:

> #731: CBD and Update leads to 500
> -----------------------------+----------------------------------------------
>  Reporter:  jeremy_carroll  |       Owner:  jeremy_carroll       
>      Type:  defect          |      Status:  closed               
>  Priority:  critical        |   Milestone:                       
> Component:  Bigdata SAIL    |     Version:  BIGDATA_RELEASE_1_2_3
> Resolution:  fixed           |    Keywords:                       
> -----------------------------+----------------------------------------------
> Changes (by thompsonbry):
> 
>  * status:  new => closed
>  * resolution:  => fixed
> 
> 
> Comment:
> 
> Proposed change set looks good.
> 
> CI is clean.
> 
> Ticket is closed.
> 
> -- 
> Ticket URL: <http://sourceforge.net/apps/trac/bigdata/ticket/731#comment:4>
> bigdata® <http://www.bigdata.com/blog>
> bigdata® is a scale-out storage and computing fabric supporting optional transactions, very high concurrency, and very high aggregate IO rates.

[Bigdata-developers] FW: [Bigdata-commit] SF.net SVN: bigdata:[7319] branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/ java/com/bigdata/rdf/sparql/ast/eval

[Bryan T. <br...@sy...>]

I rolled back r7319 as it was causing problems with UNION.  You can see
the errors if you reapply the change set by running

com.bigdata.rdf.sparql.ast.eval.TestAll


My change is r7329.

I marked the places in the code where the change was reversed with the
following text to make it easier to find:

>>> FIXME Rolling back r7319 which broke UNION processing.

Thanks,
Bryan

On 8/22/13 3:21 PM, "mrp...@us..."
<mrp...@us...> wrote:

>Revision: 7319
>          http://bigdata.svn.sourceforge.net/bigdata/?rev=7319&view=rev
>Author:   mrpersonick
>Date:     2013-08-22 19:21:47 +0000 (Thu, 22 Aug 2013)
>Log Message:
>-----------
>got rid of the extra copy ops we were using to make Tee work (for Unions)
>
>Modified Paths:
>--------------
>    
>branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql
>/ast/eval/AST2BOpContext.java
>    
>branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql
>/ast/eval/AST2BOpUtility.java
>
>Modified: 
>branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql
>/ast/eval/AST2BOpContext.java
>===================================================================
>--- 
>branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql
>/ast/eval/AST2BOpContext.java	2013-08-22 18:42:59 UTC (rev 7318)
>+++ 
>branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql
>/ast/eval/AST2BOpContext.java	2013-08-22 19:21:47 UTC (rev 7319)
>@@ -66,6 +66,13 @@
> 	private final AtomicInteger idFactory;
> 	
> 	/**
>+	 * Temporary "next id" bypasses the idFactory when we want to be
>explicit
>+	 * about the next bop id.  Used for Tees (Unions).  nextId = -1 means
>use
>+	 * the idFactory.
>+	 */
>+	private transient int nextId = -1;
>+	
>+	/**
> 	 * The KB instance.
> 	 */
> 	protected final AbstractTripleStore db;
>@@ -456,9 +463,34 @@
> 
>     }
>     
>+    /**
>+     * Temporarily set the next bop Id to come out of the context.
>+     */
>+    public void setNextId(final int nextId) {
>+    	
>+    	this.nextId = nextId;
>+    	
>+    }
>+    
>+    /**
>+     * Return the next id from the idFactory, unless there is a temporary
>+     * bop id set, in which case return it and clear it.
>+     */
>     public int nextId() {
> 
>-        return idFactory.incrementAndGet();
>+    	if (nextId == -1) {
>+    		
>+    		return idFactory.incrementAndGet();
>+    		
>+    	} else {
>+    		
>+    		final int tmp = nextId;
>+    		
>+    		nextId = -1;
>+    		
>+    		return tmp;
>+    		
>+    	}
> 
>     }
> 
>
>Modified: 
>branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql
>/ast/eval/AST2BOpUtility.java
>===================================================================
>--- 
>branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql
>/ast/eval/AST2BOpUtility.java	2013-08-22 18:42:59 UTC (rev 7318)
>+++ 
>branches/BIGDATA_RELEASE_1_2_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql
>/ast/eval/AST2BOpUtility.java	2013-08-22 19:21:47 UTC (rev 7319)
>@@ -2243,10 +2243,11 @@
>              * Need to make sure the first operator in the group has the
>right
>              * Id.
>              */
>-            left = new CopyOp(leftOrEmpty(left), NV.asMap(new NV[] {//
>-                    new NV(Predicate.Annotations.BOP_ID,
>subqueryIds[i++]),//
>-                    }));
>-
>+//            left = new CopyOp(leftOrEmpty(left), NV.asMap(new NV[] {//
>+//                    new NV(Predicate.Annotations.BOP_ID,
>subqueryIds[i++]),//
>+//                    }));
>+            ctx.setNextId(subqueryIds[i++]);
>+            
>             // Start with everything already known to be materialized.
>             final Set<IVariable<?>> tmp = new
>LinkedHashSet<IVariable<?>>(
>                     doneSet);
>@@ -2278,11 +2279,12 @@
>         /*
>          * All the subqueries get routed here when they are done.
>          */
>-        left = applyQueryHints(new CopyOp(leftOrEmpty(left),//
>-                new NV(Predicate.Annotations.BOP_ID, downstreamId),//
>-                new NV(BOp.Annotations.EVALUATION_CONTEXT,
>-                        BOpEvaluationContext.CONTROLLER)//
>-                ), ctx.queryHints);
>+//        left = applyQueryHints(new CopyOp(leftOrEmpty(left),//
>+//                new NV(Predicate.Annotations.BOP_ID, downstreamId),//
>+//                new NV(BOp.Annotations.EVALUATION_CONTEXT,
>+//                        BOpEvaluationContext.CONTROLLER)//
>+//                ), ctx.queryHints);
>+        ctx.setNextId(downstreamId);
> 
>         // Add in anything which was known materialized for all child
>groups.
>         doneSet.addAll(doneSetsIntersection);
>
>This was sent by the SourceForge.net collaborative development platform,
>the world's largest Open Source development site.
>
>
>--------------------------------------------------------------------------
>----
>Introducing Performance Central, a new site from SourceForge and
>AppDynamics. Performance Central is your source for news, insights,
>analysis and resources for efficient Application Performance Management.
>Visit us today!
>http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktr
>k
>_______________________________________________
>Bigdata-commit mailing list
>Big...@li...
>https://lists.sourceforge.net/lists/listinfo/bigdata-commit

[Bigdata-developers] tiny fix for CBD, trac 731

[Jeremy J C. <jj...@sy...>]

I just switched to using Concise Bounded Description for describe mode, and hit a trivial defect where describe mode was being used during an update - and then there was an IllegalArgumentException (null pointer)
I have committed a fix in revision 7328; I hope this was not presumptious.

Jeremy J Carroll
Principal Architect
Syapse, Inc.

Re: [Bigdata-developers] What inferences are expected with OwlAxioms?

[Jim B. <ba...@ne...>]

Mike,

On Aug 21, 2013, at 3:46 PM, Mike Personick <mi...@sy...> wrote:

> Jim,
> 
> Can you provide a sample data set that demonstrates the problem with the
> first query?  What is the correct answer for that query on your data set?

I will create a sample data set with expected answers and get back to you soon.


> The second query uses an unsupported OWL feature.  Bigdata supports the
> RDFS Plus inference profile.  This includes rdfs:subClassOf,
> rdfs:subPropertyOf, rdfs:domain, rdfs:range, owl:equivalentClass,
> owl:equivalentProperty, owl:sameAs, owl:inverseOf, owl:TransitiveProperty,
> owl:SymmetricProperty, owl:InverseFunctionalProperty, and
> owl:FunctionalProperty.  Bigdata does not support higher level reasoning,
> including owl:someValuesFrom.  In general bigdata is aimed at large scale
> knowledge bases that do not require the computational complexity required
> by higher level reasoning.

Thanks, this is good to know. We use several complicated ontologies with deep hierarchies and I have been working on ways to make use of the knowledge in them on a "big" (to me) dataset without doing much in the way of reasoning at query time. But I wanted to make sure I had an accurate understanding of the reasoning actually provided with Bigdata before going too far with my other approaches.

- Jim

> 
> Thanks,
> Mike
> 
> 
> On 8/21/13 12:34 PM, "Bryan Thompson" <br...@sy...> wrote:
> 
>> Copying the developers list. The SPAM filter on source forge is not
>> adjustable until we convert the project over to the new project management
>> system, which is currently blocking on the correct migration of the trac
>> tickets (and pending while I figure out if the new project management
>> ticket tracker is going to work out for us).
>> 
>> Please see below.
>> Bryan
>> 
>> On 8/21/13 2:23 PM, "Jim Balhoff" <ba...@ne...> wrote:
>> 
>>> Hi,
>>> 
>>> I tried to post this on the Bigdata forum but it is blocked by the
>>> Sourceforge spam check.
>>> 
>>> I am using Bigdata 1.2.3 and trying out some of the inference support. Is
>>> there any documentation about what inferences can be expected for the
>>> OwlAxioms mode? I haven't found much on the wiki covering inference. In
>>> my RWStore.properties, I set
>>> com.bigdata.rdf.store.AbstractTripleStore.quads=false and
>>> com.bigdata.rdf.store.AbstractTripleStore.axiomsClass=com.bigdata.rdf.axi
>>> o
>>> ms.OwlAxioms.
>>> 
>>> I tried a basic subclass query and don't seem to get any inferences:
>>> 
>>> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
>>> PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
>>> PREFIX bone_element: <http://purl.obolibrary.org/obo/UBERON_0001474>
>>> SELECT (COUNT(DISTINCT ?term) AS ?count)
>>> WHERE
>>> {
>>> ?term rdfs:subClassOf bone_element: .
>>> ?term rdfs:label ?label .
>>> }
>>> 
>>> The result is 22, but when I use a property path rdfs:subClassOf* the
>>> result is 1121. I expected them to be the same with OWL inferences.
>>> 
>>> I was hoping that some types of class expressions would be supported,
>>> like this:
>>> 
>>> PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
>>> PREFIX owl: <http://www.w3.org/2002/07/owl#>
>>> PREFIX part_of: <http://purl.obolibrary.org/obo/BFO_0000050>
>>> PREFIX head: <http://purl.obolibrary.org/obo/UBERON_0000033>
>>> SELECT (COUNT(DISTINCT ?term) AS ?count)
>>> WHERE
>>> {
>>> ?term rdfs:subClassOf ?restriction .
>>> ?restriction owl:onProperty part_of: .
>>> ?restriction owl:someValuesFrom head: .
>>> ?term rdfs:label ?label .
>>> }
>>> 
>>> The result of this query to Bigdata is 56, but using the ELK reasoner for
>>> OWL EL in Protege I find 1697 descendant classes of (part_of some head).
>>> 
>>> Regarding the first (bone_element) query, it seems simple so might I have
>>> something configured incorrectly? The explanation does show
>>> "includeInferred=true". Are expression queries like the second supported?
>>> 
>>> Thank you,
>>> Jim
>>> 
>>> ____________________________________________
>>> James P. Balhoff, Ph.D.
>>> National Evolutionary Synthesis Center
>>> 2024 West Main St., Suite A200
>>> Durham, NC 27705
>>> USA
>>> 
>>> 
>>> 
>> 
> 

____________________________________________
James P. Balhoff, Ph.D.
National Evolutionary Synthesis Center
2024 West Main St., Suite A200
Durham, NC 27705
USA

[Bigdata-developers] next couple of weeks

[Jeremy C. <jj...@gm...>]

Just to set expectations, after my flurry of activity today, I am not expecting to do further bigdata related work for the next couple of weeks.

When I do next have time, I will pick up either one of the correctness issues I have reported or the SPARQL 1.1 graph update protocol issue

I have previously looked at the heisenbug 708 and decided it is too hard for me.

Jeremy

Re: [Bigdata-developers] FW: [bigdata®] #429: Optimization for GRAPH uri {} and GRAPH ?foo {}

[Jeremy J C. <jj...@sy...>]

On Aug 21, 2013, at 2:27 PM, Bryan Thompson <br...@sy...> wrote:

> Bigdata does not distinguish between an empty named graph and a named
> graph that does not exist.  

I have modified the tests in light of this, of the 6 cases I thought of, only 2 give correct results.

Jeremy

[Bigdata-developers] heisenbug test committed: bug 708

[Jeremy J C. <jj...@sy...>]

I managed to get the heisen behavior to reliably repeat as a JUnit test on my machine. This is checked in at
bigdata-rdf/src/test/com/bigdata/rdf/sparql/ast/eval/TestBindHeisenbug708.java

I would be interested as to whether others can observe the same behavior as me: if not we may wish to increase the max number in the test code.

I get failure messages such as:
junit.framework.AssertionFailedError: Test failed 6/10 times

Non-heisen behavior would include: the test passing, and 

junit.framework.AssertionFailedError: Test failed 10/10 times

Obviously the latter is not desirable, but the heisen behavior is much worse in my view.

Jeremy J Carroll
Principal Architect
Syapse, Inc.

Re: [Bigdata-developers] FW: [bigdata®] #429: Optimization for GRAPH uri {} and GRAPH ?foo {}

[Jeremy J C. <jj...@sy...>]

OK - I  will modify the relevant tests to reflect this intent. However, there are still test failures for some cases.

Jeremy J Carroll
Principal Architect
Syapse, Inc.



On Aug 21, 2013, at 2:27 PM, Bryan Thompson <br...@sy...> wrote:

> Bigdata does not distinguish between an empty named graph and a named
> graph that does not exist.  You do not need to explicitly create a named
> graph and a destroy of an empty named graph has no effect.  Could this be
> related to what you are observing?
> 
> Internally, we just carry covering indices (SPOC, etc).
> 
> public static final transient SPOKeyOrder SPOC = new SPOKeyOrder(_SPOC);
>    public static final transient SPOKeyOrder POCS = new
> SPOKeyOrder(_POCS);
>    public static final transient SPOKeyOrder OCSP = new
> SPOKeyOrder(_OCSP);
>    public static final transient SPOKeyOrder CSPO = new
> SPOKeyOrder(_CSPO);
>    public static final transient SPOKeyOrder PCSO = new
> SPOKeyOrder(_PCSO);
>    public static final transient SPOKeyOrder SOPC = new
> SPOKeyOrder(_SOPC);
> 
> If the CSPO graph has nothing for some given C, then the named graph is
> empty.  But we never say that it does or does not exist.
> 
> 
> Bryan
> 
> On 8/21/13 5:21 PM, "bigdata®" <no...@so...> wrote:
> 
>> #429: Optimization for GRAPH uri {} and GRAPH ?foo {}
>> ----------------------------------+---------------------------------------
>> --
>> Reporter:  thompsonbry           |       Owner:  mrpersonick
>>    Type:  enhancement           |      Status:  new
>> Priority:  major                 |   Milestone:  Query
>> Component:  Query Plan Generator  |     Version:
>> Keywords:                        |
>> ----------------------------------+---------------------------------------
>> --
>> 
>> Comment(by jeremy_carroll):
>> 
>> GRAPH uri {} does not work! It always matches, when it shouldn't, if
>> there
>> is no graph in the dataset. Added a test to exercise this point.
>> 
>> I believe it is an implementation variant as to whether or not the empty
>> graph matches. I have tests for this case, and the tests presuppose that
>> bigdata supports empty graphs in the dataset. If this is incorrect
>> assumption, then the two tests will need tweaking
>> 
>> -- 
>> Ticket URL: 
>> <http://sourceforge.net/apps/trac/bigdata/ticket/429#comment:2>
>> bigdata® <http://www.bigdata.com/blog>
>> bigdata® is a scale-out storage and computing fabric supporting optional
>> transactions, very high concurrency, and very high aggregate IO rates.
> 
> 
> ------------------------------------------------------------------------------
> Introducing Performance Central, a new site from SourceForge and 
> AppDynamics. Performance Central is your source for news, insights, 
> analysis and resources for efficient Application Performance Management. 
> Visit us today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
> _______________________________________________
> Bigdata-developers mailing list
> Big...@li...
> https://lists.sourceforge.net/lists/listinfo/bigdata-developers

[Bigdata-developers] FW: [bigdata®] #429: Optimization for GRAPH uri {} and GRAPH ?foo {}

[Bryan T. <br...@sy...>]

Bigdata does not distinguish between an empty named graph and a named
graph that does not exist.  You do not need to explicitly create a named
graph and a destroy of an empty named graph has no effect.  Could this be
related to what you are observing?

Internally, we just carry covering indices (SPOC, etc).

public static final transient SPOKeyOrder SPOC = new SPOKeyOrder(_SPOC);
    public static final transient SPOKeyOrder POCS = new
SPOKeyOrder(_POCS);
    public static final transient SPOKeyOrder OCSP = new
SPOKeyOrder(_OCSP);
    public static final transient SPOKeyOrder CSPO = new
SPOKeyOrder(_CSPO);
    public static final transient SPOKeyOrder PCSO = new
SPOKeyOrder(_PCSO);
    public static final transient SPOKeyOrder SOPC = new
SPOKeyOrder(_SOPC);

If the CSPO graph has nothing for some given C, then the named graph is
empty.  But we never say that it does or does not exist.


Bryan

On 8/21/13 5:21 PM, "bigdata®" <no...@so...> wrote:

>#429: Optimization for GRAPH uri {} and GRAPH ?foo {}
>----------------------------------+---------------------------------------
>--
> Reporter:  thompsonbry           |       Owner:  mrpersonick
>     Type:  enhancement           |      Status:  new
> Priority:  major                 |   Milestone:  Query
>Component:  Query Plan Generator  |     Version:
> Keywords:                        |
>----------------------------------+---------------------------------------
>--
>
>Comment(by jeremy_carroll):
>
> GRAPH uri {} does not work! It always matches, when it shouldn't, if
>there
> is no graph in the dataset. Added a test to exercise this point.
>
> I believe it is an implementation variant as to whether or not the empty
> graph matches. I have tests for this case, and the tests presuppose that
> bigdata supports empty graphs in the dataset. If this is incorrect
> assumption, then the two tests will need tweaking
>
>-- 
>Ticket URL: 
><http://sourceforge.net/apps/trac/bigdata/ticket/429#comment:2>
>bigdata® <http://www.bigdata.com/blog>
>bigdata® is a scale-out storage and computing fabric supporting optional
>transactions, very high concurrency, and very high aggregate IO rates.