[Bigdata-developers] Write performance degrades during import

[Lee K. <le...@sw...>]

Hi,

We are trying to perform a bulk import into a new blazegraph journal. The
import process writes quads to an in-process BigdataSailRepository with the
following configuration based on the 'fastload' settings in the
bigdata-sails samples directory:

com.bigdata.rdf.store.AbstractTripleStore.quadsMode=true

com.bigdata.rdf.store.AbstractTripleStore.statementIdentifiers=false

com.bigdata.rdf.store.AbstractTripleStore.axiomsClass=com.bigdata.rdf.axioms.NoAxioms

com.bigdata.rdf.sail.truthMaintenance=false

com.bigdata.rdf.store.AbstractTripleStore.justify=false

com.bigdata.journal.AbstractJournal.initialExtent=209715200

com.bigdata.journal.AbstractJournal.maximumExtent=209715200

com.bigdata.rdf.store.AbstractTripleStore.textIndex=false

com.bigdata.journal.AbstractJournal.bufferMode=DiskRW

com.bigdata.sail.isolatableIndices=true

com.bigdata.rdf.store.AbstractTripleStore.vocabularyClass=com.bigdata.rdf.vocab.NoVocabulary

com.bigdata.journal.AbstractJournal.file=bigdata_conf.jnl

com.bigdata.journal.AbstractJournal.writeCacheBufferCount=2000

com.bigdata.btree.writeRetentionQueue.capacity=8000


When run against a native sesame repository, the import takes around 50
hours. When run against the blazegraph repository the import slows down
significantly after 2-3 hours and begins logging warnings of the form:

[2015-04-27 07:34:30,238][WARN][com.bigdata.btree.AbstractBTree] wrote:
name=kb.spo.OCSP, 8 records (#nodes=3, #leaves=5) in 5493ms :
addrRoot=-244779124025982418
[2015-04-27 07:47:48,342][WARN][com.bigdata.btree.AbstractBTree] wrote:
name=kb.spo.SOPC, 1 records (#nodes=1, #leaves=0) in 40841ms :
addrRoot=-246059333517835846
[2015-04-27 07:47:48,858][WARN][com.bigdata.btree.AbstractBTree] wrote:
name=kb.spo.SPOC, 7 records (#nodes=4, #leaves=3) in 42109ms :
addrRoot=-246099989678259484
[2015-04-27 07:54:47,743][WARN][com.bigdata.btree.AbstractBTree] wrote:
name=kb.spo.SOPC, 1 records (#nodes=1, #leaves=0) in 43231ms :
addrRoot=-245678000551493109
[2015-04-27 07:54:52,251][WARN][com.bigdata.btree.AbstractBTree] wrote:
name=kb.spo.SPOC, 1 records (#nodes=1, #leaves=0) in 44875ms :
addrRoot=-245097441232158259
[2015-04-27 07:54:52,251][WARN][com.bigdata.btree.AbstractBTree] wrote:
name=kb.spo.CSPO, 1 records (#nodes=1, #leaves=0) in 34808ms :
addrRoot=-245097501361700476
[2015-04-27 07:54:52,251][WARN][com.bigdata.btree.AbstractBTree] wrote:
name=kb.spo.POCS, 1 records (#nodes=1, #leaves=0) in 44875ms :
addrRoot=-245097342447910551

Are there any settings we should change or add to the journal configuration
to prevent this slowdown?

Thanks