Re: [Bigdata-developers] Write performance degrades during import

[Bryan T. <br...@sy...>]

Lee,

There are quite a few.  Please see the performance optimization section of
the wiki.  The main things to look at are IO Wait.  This can be compensated
in part by the write cache buffers.  Statement buffer size and branching
factors also effect throughput over time.

In other news, we have a fix for the pre-order traversal with annotations
issue (#1210) and it fixes the wildcard rewrite problem.  I've attached the
modified classes so you can test locally.  The fix will be in the next
release.

Let me suggest that we schedule a call for next week to discuss some of the
questions you have had on the list and your goals with the platform and how
we can work with you to achieve them.  Brad Bebee (Cc) is the best point of
contact to set this up.

Thanks,
Bryan

----
Bryan Thompson
Chief Scientist & Founder
SYSTAP, LLC
4501 Tower Road
Greensboro, NC 27410
br...@sy...
http://blazegraph.com
http://blog.bigdata.com <http://bigdata.com>
http://mapgraph.io

Blazegraph™ <http://www.blazegraph.com/> is our ultra high-performance
graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints
APIs.  MapGraph™ <http://www.systap.com/mapgraph> is our disruptive new
technology to use GPUs to accelerate data-parallel graph analytics.

CONFIDENTIALITY NOTICE:  This email and its contents and attachments are
for the sole use of the intended recipient(s) and are confidential or
proprietary to SYSTAP. Any unauthorized review, use, disclosure,
dissemination or copying of this email or its contents or attachments is
prohibited. If you have received this communication in error, please notify
the sender by reply email and permanently delete all copies of the email
and its contents and attachments.

On Mon, Apr 27, 2015 at 11:19 AM, Lee Kitching <le...@sw...> wrote:

> Hi,
>
> We are trying to perform a bulk import into a new blazegraph journal. The
> import process writes quads to an in-process BigdataSailRepository with the
> following configuration based on the 'fastload' settings in the
> bigdata-sails samples directory:
>
> com.bigdata.rdf.store.AbstractTripleStore.quadsMode=true
>
> com.bigdata.rdf.store.AbstractTripleStore.statementIdentifiers=false
>
>
> com.bigdata.rdf.store.AbstractTripleStore.axiomsClass=com.bigdata.rdf.axioms.NoAxioms
>
> com.bigdata.rdf.sail.truthMaintenance=false
>
> com.bigdata.rdf.store.AbstractTripleStore.justify=false
>
> com.bigdata.journal.AbstractJournal.initialExtent=209715200
>
> com.bigdata.journal.AbstractJournal.maximumExtent=209715200
>
> com.bigdata.rdf.store.AbstractTripleStore.textIndex=false
>
> com.bigdata.journal.AbstractJournal.bufferMode=DiskRW
>
> com.bigdata.sail.isolatableIndices=true
>
>
> com.bigdata.rdf.store.AbstractTripleStore.vocabularyClass=com.bigdata.rdf.vocab.NoVocabulary
>
> com.bigdata.journal.AbstractJournal.file=bigdata_conf.jnl
>
> com.bigdata.journal.AbstractJournal.writeCacheBufferCount=2000
>
> com.bigdata.btree.writeRetentionQueue.capacity=8000
>
>
> When run against a native sesame repository, the import takes around 50
> hours. When run against the blazegraph repository the import slows down
> significantly after 2-3 hours and begins logging warnings of the form:
>
> [2015-04-27 07:34:30,238][WARN][com.bigdata.btree.AbstractBTree] wrote:
> name=kb.spo.OCSP, 8 records (#nodes=3, #leaves=5) in 5493ms :
> addrRoot=-244779124025982418
> [2015-04-27 07:47:48,342][WARN][com.bigdata.btree.AbstractBTree] wrote:
> name=kb.spo.SOPC, 1 records (#nodes=1, #leaves=0) in 40841ms :
> addrRoot=-246059333517835846
> [2015-04-27 07:47:48,858][WARN][com.bigdata.btree.AbstractBTree] wrote:
> name=kb.spo.SPOC, 7 records (#nodes=4, #leaves=3) in 42109ms :
> addrRoot=-246099989678259484
> [2015-04-27 07:54:47,743][WARN][com.bigdata.btree.AbstractBTree] wrote:
> name=kb.spo.SOPC, 1 records (#nodes=1, #leaves=0) in 43231ms :
> addrRoot=-245678000551493109
> [2015-04-27 07:54:52,251][WARN][com.bigdata.btree.AbstractBTree] wrote:
> name=kb.spo.SPOC, 1 records (#nodes=1, #leaves=0) in 44875ms :
> addrRoot=-245097441232158259
> [2015-04-27 07:54:52,251][WARN][com.bigdata.btree.AbstractBTree] wrote:
> name=kb.spo.CSPO, 1 records (#nodes=1, #leaves=0) in 34808ms :
> addrRoot=-245097501361700476
> [2015-04-27 07:54:52,251][WARN][com.bigdata.btree.AbstractBTree] wrote:
> name=kb.spo.POCS, 1 records (#nodes=1, #leaves=0) in 44875ms :
> addrRoot=-245097342447910551
>
> Are there any settings we should change or add to the journal
> configuration to prevent this slowdown?
>
> Thanks
>
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Bigdata-developers mailing list
> Big...@li...
> https://lists.sourceforge.net/lists/listinfo/bigdata-developers
>
>