[Virtuoso-users] When is a bulk load done?

[Paul H. <ont...@gm...>]

The image below is a trace of the CPU consumption of a bulk load I did
in Virtuoso.

https://raw.githubusercontent.com/paulhoule/images/master/CpuForVirtuosoLoad.PNG

Here I am loading half a billion or so triples in Amazon EC2 using an
r3.xlarge,  with Virtuoso running on the instance storage SSD.  In
terms of time and cost,  this is a great chart,  but there is one
detail that I'm concerned about.

At first I queued up the 'a' dataset and started one rdf_loader_run.
That used around 50% of system resources on a four core machine,  so
when I started a second loader,  the rate sped up until it reached
100%.

It falls to zero because it finished the 'a' data,  then I queued the
rest of the data and the system ran at 100%.  Then there is a glitch
at one point,  then the CPU consumption falls in two plateaus,  with
another glitch at the end.

Definitely in the second plateau was looking at the machine and
noticed that the  db.dba.load_list said the loading was done but that
the CPU was still running hard and,  more to the point,  I/O was still
being done to the SSD.  I don't see anything in the log that
correlates with the end of this phase.

What is Virtuoso doing in this phase after the loading is said to be
complete?  Would it be safe to shut Virtuoso down at this time?  I can
watch the load_list and the CPU meter to get a pretty good guess of
when it is done,  but is there some other way I know Virtuoso is done
with this activity?



-- 
Paul Houle
Expert on Freebase, DBpedia, Hadoop and RDF
(607) 539 6254    paul.houle on Skype   ont...@gm...