Menu
▾
▴
Re: [Bigdata-developers] Blazegraph 1.5.1 on Travis
|
From: James HK <jam...@gm...> - 2015-05-09 18:38:35
|
Hi, > Let me suggest that 256MB is a pretty small heap for the JVM running > the database. Why don't you give it a 4GB heap? Changing the option to "-Xmx4g" did not amount in any significant reduction [0] of the test execution time (compared to the previous cited runs). > PerformanceOptimization (General background on performance tuning for > bigdata) > http://wiki.blazegraph.com/wiki/index.php/PerformanceOptimization I read through the pages but I couldn't find any option that would make the test suite run faster in the Travis-CI environment. [0] https://travis-ci.org/mwjames/SemanticMediaWiki/builds/61901411 Cheers On 5/9/15, Bryan Thompson <br...@sy...> wrote: > James, > > Let me suggest that 256MB is a pretty small heap for the JVM running > the database. Why don't you give it a 4GB heap? > > We provide guidance for performance tuning at the following links: > > PerformanceOptimization (General background on performance tuning for > bigdata) > http://wiki.blazegraph.com/wiki/index.php/PerformanceOptimization > > IOOptimization (How to tune IO performance, SSD, branching factors, etc.) > http://wiki.blazegraph.com/wiki/index.php/IOOptimization > > QueryOptimization (SPARQL specific query performance tricks and tips) > http://wiki.blazegraph.com/wiki/index.php/QueryOptimization > > Thanks, > Bryan > ---- > Bryan Thompson > Chief Scientist & Founder > SYSTAP, LLC > 4501 Tower Road > Greensboro, NC 27410 > br...@sy... > http://blazegraph.com > http://blog.bigdata.com > http://mapgraph.io > > Blazegraph™ is our ultra high-performance graph database that supports > both RDF/SPARQL and Tinkerpop/Blueprints APIs. MapGraph™ is our > disruptive new technology to use GPUs to accelerate data-parallel > graph analytics. > > CONFIDENTIALITY NOTICE: This email and its contents and attachments > are for the sole use of the intended recipient(s) and are confidential > or proprietary to SYSTAP. Any unauthorized review, use, disclosure, > dissemination or copying of this email or its contents or attachments > is prohibited. If you have received this communication in error, > please notify the sender by reply email and permanently delete all > copies of the email and its contents and attachments. > > > On Sat, May 9, 2015 at 12:12 AM, James HK <jam...@gm...> > wrote: >> Hi, >> >> I just wanted to give you some feedback on the Blazegraph >> 1.5.1/Semantic MediaWiki 2.3/Travis integration. >> >> After having tried different ways to use tomcat + bigdata on Travis >> (which wasn't really fruitful, our Travis/sesame runs on tomcat), I >> settled for the more pragmatic [0] approach. Using [0] now allows us >> to run our test suite on Travis [1] against Blazegraph 1.5.1. >> >> I'm not sure why but Blazegraph requires ~6 min to pass all our tests >> while the same tests finish on Fuseki [2], Sesame [3], or Virtuoso [4] >> in about half the time (~3 min). >> >> Maybe there are some tweaks available in order to make our tests run >> faster before adding the Blazegraph service to our normal Travis >> setup. >> >> [0] java -server -Xmx256m >> -Dbigdata.propertyFile=.../build/travis/blazegraph-store.properties >> -jar bigdata-bundled.jar >> >> [1] https://travis-ci.org/mwjames/SemanticMediaWiki/builds/61850318, >> https://travis-ci.org/mwjames/SemanticMediaWiki/builds/61849465 >> >> [2] >> https://travis-ci.org/SemanticMediaWiki/SemanticMediaWiki/jobs/61848314 >> [3] >> https://travis-ci.org/SemanticMediaWiki/SemanticMediaWiki/jobs/61848316 >> [4] >> https://travis-ci.org/SemanticMediaWiki/SemanticMediaWiki/jobs/61848315 >> >> Cheers >> >> On 5/1/15, Bryan Thompson <br...@sy...> wrote: >>> Here is the relevant documentation from the code: >>> >>> to turn this off include the following in your property file: >>> >>> com.bigdata.rdf.store.AbstractTripleStore.inlineDateTimes=false >>> >>> /** >>> * Set up database to inline date/times directly into the statement >>> * indices rather than using the lexicon to map them to term identifiers >>> * and back (default {@value #DEFAULT_INLINE_DATE_TIMES}). Date times >>> * will be converted to UTC, then stored as milliseconds since the >>> * epoch. Thus if you inline date/times you will lose the canonical >>> * representation of the date/time. This has two consequences: (1) you >>> * will not be able to recover the original time zone of the date/time; >>> * and (2) greater than millisecond precision will be lost. >>> * >>> * @see #INLINE_DATE_TIMES_TIMEZONE >>> */ >>> String INLINE_DATE_TIMES = AbstractTripleStore.class.getName() >>> + ".inlineDateTimes"; >>> >>> String DEFAULT_INLINE_DATE_TIMES = "true"; >>> ---- >>> Bryan Thompson >>> Chief Scientist & Founder >>> SYSTAP, LLC >>> 4501 Tower Road >>> Greensboro, NC 27410 >>> br...@sy... >>> http://blazegraph.com >>> http://blog.bigdata.com >>> http://mapgraph.io >>> >>> Blazegraph™ is our ultra high-performance graph database that supports >>> both RDF/SPARQL and Tinkerpop/Blueprints APIs. MapGraph™ is our >>> disruptive new technology to use GPUs to accelerate data-parallel >>> graph analytics. >>> >>> CONFIDENTIALITY NOTICE: This email and its contents and attachments >>> are for the sole use of the intended recipient(s) and are confidential >>> or proprietary to SYSTAP. Any unauthorized review, use, disclosure, >>> dissemination or copying of this email or its contents or attachments >>> is prohibited. If you have received this communication in error, >>> please notify the sender by reply email and permanently delete all >>> copies of the email and its contents and attachments. >>> >>> >>> On Thu, Apr 30, 2015 at 12:34 PM, Bryan Thompson <br...@sy...> >>> wrote: >>>> I am not at a computer now.. see AbstractTripleStore INLINE_DATE_TIMES. >>>> You >>>> need to turn off that property when creating the namespace or in the >>>> RWStore >>>> file in WEB-INF. >>>> >>>> Thanks, >>>> Bryan >>>> >>>> On Apr 30, 2015 7:30 AM, "James HK" <jam...@gm...> >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> When trying to run our unit test suite [0] against a preliminary/local >>>>> Blazegraph 1.5.1 instance the >>>>> following error appeared during the test run (using a vanilla >>>>> Blazegraph with the standard kb namespace) . >>>>> >>>>> Our test suite (and the test mentioned) is run/tested against Virtuoso >>>>> 6.1/Fuseki 1.1.1/Sesame 2.7.14 [0] on Travis-CI therefore it is >>>>> unlikely an issue on our side. >>>>> >>>>> 1) >>>>> SMW\Tests\Integration\MediaWiki\Import\TimeDataTypeTest::testImportOfDifferentDateWithAssortmentOfOutputConversion >>>>> SMW\SPARQLStore\Exception\BadHttpDatabaseResponseException: A SPARQL >>>>> query error has occurred >>>>> Query: >>>>> PREFIX wiki: <http://example.org/id/> >>>>> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> >>>>> PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> >>>>> PREFIX owl: <http://www.w3.org/2002/07/owl#> >>>>> PREFIX swivt: <http://semantic-mediawiki.org/swivt/1.0#> >>>>> PREFIX property: <http://example.org/id/Property-3A> >>>>> PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> >>>>> DELETE { wiki:TimeDataTypeRegressionTest ?p ?o } WHERE { >>>>> wiki:TimeDataTypeRegressionTest ?p ?o } >>>>> Error: Query refused >>>>> Endpoint: http://192.168.1.104:9999/bigdata/namespace/kb/sparql >>>>> HTTP response code: 500 >>>>> >>>>> This translates into an error on the Blazegraph side with (output from >>>>> http://localhost:9999/bigdata/#update): >>>>> >>>>> ERROR: SPARQL-UPDATE: updateStr=PREFIX wiki: >>>>> PREFIX rdf: >>>>> PREFIX rdfs: >>>>> PREFIX owl: >>>>> PREFIX swivt: >>>>> PREFIX property: >>>>> PREFIX xsd: >>>>> DELETE { wiki:TimeDataTypeRegressionTest ?p ?o } WHERE { >>>>> wiki:TimeDataTypeRegressionTest ?p ?o } >>>>> java.util.concurrent.ExecutionException: >>>>> java.util.concurrent.ExecutionException: >>>>> org.openrdf.query.UpdateExecutionException: >>>>> java.lang.IllegalStateException: Already assigned: >>>>> old=LiteralExtensionIV [delegate=XSDLong(6017484188943806464), >>>>> datatype=Vocab(-42)], new=LiteralExtensionIV >>>>> [delegate=XSDLong(204552172800000), datatype=Vocab(-42)], this: >>>>> "8452"^^ >>>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:188) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:261) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlUpdate(QueryServlet.java:359) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:165) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:237) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doPost(MultiTenancyServlet.java:137) >>>>> at >>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:707) >>>>> at >>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >>>>> at >>>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769) >>>>> at >>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) >>>>> at >>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) >>>>> at >>>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) >>>>> at >>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) >>>>> at >>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) >>>>> at >>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) >>>>> at >>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) >>>>> at >>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) >>>>> at >>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >>>>> at >>>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) >>>>> at >>>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) >>>>> at >>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) >>>>> at org.eclipse.jetty.server.Server.handle(Server.java:497) >>>>> at >>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) >>>>> at >>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) >>>>> at >>>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) >>>>> at >>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610) >>>>> at >>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539) >>>>> at java.lang.Thread.run(Thread.java:744) >>>>> Caused by: java.util.concurrent.ExecutionException: >>>>> org.openrdf.query.UpdateExecutionException: >>>>> java.lang.IllegalStateException: Already assigned: >>>>> old=LiteralExtensionIV [delegate=XSDLong(6017484188943806464), >>>>> datatype=Vocab(-42)], new=LiteralExtensionIV >>>>> [delegate=XSDLong(204552172800000), datatype=Vocab(-42)], this: >>>>> "8452"^^ >>>>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >>>>> at java.util.concurrent.FutureTask.get(FutureTask.java:188) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.QueryServlet$SparqlUpdateTask.call(QueryServlet.java:460) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.QueryServlet$SparqlUpdateTask.call(QueryServlet.java:371) >>>>> at >>>>> com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> at >>>>> com.bigdata.rdf.task.AbstractApiTask.submitApiTask(AbstractApiTask.java:365) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:258) >>>>> ... 26 more >>>>> Caused by: org.openrdf.query.UpdateExecutionException: >>>>> java.lang.IllegalStateException: Already assigned: >>>>> old=LiteralExtensionIV [delegate=XSDLong(6017484188943806464), >>>>> datatype=Vocab(-42)], new=LiteralExtensionIV >>>>> [delegate=XSDLong(204552172800000), datatype=Vocab(-42)], this: >>>>> "8452"^^ >>>>> at >>>>> com.bigdata.rdf.sparql.ast.eval.ASTEvalHelper.executeUpdate(ASTEvalHelper.java:1303) >>>>> at >>>>> com.bigdata.rdf.sail.BigdataSailUpdate.execute2(BigdataSailUpdate.java:152) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.BigdataRDFContext$UpdateTask.doQuery(BigdataRDFContext.java:1683) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.innerCall(BigdataRDFContext.java:1310) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:1275) >>>>> at >>>>> com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:517) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> ... 1 more >>>>> Caused by: java.lang.IllegalStateException: Already assigned: >>>>> old=LiteralExtensionIV [delegate=XSDLong(6017484188943806464), >>>>> datatype=Vocab(-42)], new=LiteralExtensionIV >>>>> [delegate=XSDLong(204552172800000), datatype=Vocab(-42)], this: >>>>> "8452"^^ >>>>> at >>>>> com.bigdata.rdf.model.BigdataValueImpl.setIV(BigdataValueImpl.java:139) >>>>> at >>>>> com.bigdata.rdf.internal.LexiconConfiguration.createInlineIV(LexiconConfiguration.java:430) >>>>> at >>>>> com.bigdata.rdf.lexicon.LexiconRelation.getInlineIV(LexiconRelation.java:3150) >>>>> at >>>>> com.bigdata.rdf.lexicon.LexiconRelation.addTerms(LexiconRelation.java:1719) >>>>> at >>>>> com.bigdata.rdf.store.AbstractTripleStore.getAccessPath(AbstractTripleStore.java:2928) >>>>> at >>>>> com.bigdata.rdf.store.AbstractTripleStore.getAccessPath(AbstractTripleStore.java:2874) >>>>> at >>>>> com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.removeStatements(BigdataSail.java:2962) >>>>> at >>>>> com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.removeStatements(BigdataSail.java:2865) >>>>> at >>>>> com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.addOrRemoveStatement(AST2BOpUpdate.java:2054) >>>>> at >>>>> com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.convertDeleteInsert(AST2BOpUpdate.java:989) >>>>> at >>>>> com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.convertUpdateSwitch(AST2BOpUpdate.java:417) >>>>> at >>>>> com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.convertUpdate(AST2BOpUpdate.java:279) >>>>> at >>>>> com.bigdata.rdf.sparql.ast.eval.ASTEvalHelper.executeUpdate(ASTEvalHelper.java:1295) >>>>> ... 9 more >>>>> >>>>> [0] https://travis-ci.org/SemanticMediaWiki/SemanticMediaWiki >>>>> >>>>> Cheers >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> One dashboard for servers and applications across >>>>> Physical-Virtual-Cloud >>>>> Widest out-of-the-box monitoring support with 50+ applications >>>>> Performance metrics, stats and reports that give you Actionable >>>>> Insights >>>>> Deep dive visibility with transaction tracing using APM Insight. >>>>> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y >>>>> _______________________________________________ >>>>> Bigdata-developers mailing list >>>>> Big...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/bigdata-developers >>> > |
