Re: [Virtuoso-users] When is a bulk load done?

[Hugh W. <hwi...@op...>]

Hi Paul,

If there are still rdf_loader_run() processes running then the "status();" would show this to be the case or whatever other process might be running, but if all files are in ll_state=2 in the load_list table then no rdf_loader_run() processes should be in use.

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.      //              http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

On 22 Apr 2014, at 03:56, Paul Houle <ont...@gm...> wrote:

> Well I am watching my Virtuoso instance right now and even though all
> the rows have been in state 2 for at least an hour I am still
> observing a load average of 1.14 thanks to Virtuoso CPU and disk
> activity.  So something is still going on although the load is "done".
> I've seen this behavior on and off in the past and I think that one
> contribution factor could be having more than one rdf_loader_run()
> running.
> 
> On Mon, Apr 21, 2014 at 10:52 PM, Hugh Williams
> <hwi...@op...> wrote:
>> Hi Paul,
>> 
>> Using the Virtuoso RDF Bulk Loader (rdf_loader_run) as you appear to be, the
>> load is complete when all files in the load_list table have an "ll_state" =
>> 2  ,  0 means still to be loaded, 1 means load is in progress.
>> 
>> What does the "status()" command report when the load rate slows as I
>> suspect all the "buffers" (NumberOfBuffers in INI) are being consumed at
>> that point and will thus slow  load rates.
>> 
>> Note you can also monitor the load rates using the Virtuoso LDMeter
>> functions as detailed at:
>> 
>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksGuideLDMeterUtility
>> 
>> Best Regards
>> Hugh Williams
>> Professional Services
>> OpenLink Software, Inc.      //              http://www.openlinksw.com/
>> Weblog   -- http://www.openlinksw.com/blogs/
>> LinkedIn -- http://www.linkedin.com/company/openlink-software/
>> Twitter  -- http://twitter.com/OpenLink
>> Google+  -- http://plus.google.com/100570109519069333827/
>> Facebook -- http://www.facebook.com/OpenLinkSoftware
>> Universal Data Access, Integration, and Management Technology Providers
>> 
>> On 21 Apr 2014, at 23:37, Paul Houle <ont...@gm...> wrote:
>> 
>> The image below is a trace of the CPU consumption of a bulk load I did
>> in Virtuoso.
>> 
>> https://raw.githubusercontent.com/paulhoule/images/master/CpuForVirtuosoLoad.PNG
>> 
>> Here I am loading half a billion or so triples in Amazon EC2 using an
>> r3.xlarge,  with Virtuoso running on the instance storage SSD.  In
>> terms of time and cost,  this is a great chart,  but there is one
>> detail that I'm concerned about.
>> 
>> At first I queued up the 'a' dataset and started one rdf_loader_run.
>> That used around 50% of system resources on a four core machine,  so
>> when I started a second loader,  the rate sped up until it reached
>> 100%.
>> 
>> It falls to zero because it finished the 'a' data,  then I queued the
>> rest of the data and the system ran at 100%.  Then there is a glitch
>> at one point,  then the CPU consumption falls in two plateaus,  with
>> another glitch at the end.
>> 
>> Definitely in the second plateau was looking at the machine and
>> noticed that the  db.dba.load_list said the loading was done but that
>> the CPU was still running hard and,  more to the point,  I/O was still
>> being done to the SSD.  I don't see anything in the log that
>> correlates with the end of this phase.
>> 
>> What is Virtuoso doing in this phase after the loading is said to be
>> complete?  Would it be safe to shut Virtuoso down at this time?  I can
>> watch the load_list and the CPU meter to get a pretty good guess of
>> when it is done,  but is there some other way I know Virtuoso is done
>> with this activity?
>> 
>> 
>> 
>> --
>> Paul Houle
>> Expert on Freebase, DBpedia, Hadoop and RDF
>> (607) 539 6254    paul.houle on Skype   ont...@gm...
>> 
>> ------------------------------------------------------------------------------
>> Start Your Social Network Today - Download eXo Platform
>> Build your Enterprise Intranet with eXo Platform Software
>> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
>> Get Started Now And Turn Your Intranet Into A Collaboration Platform
>> http://p.sf.net/sfu/ExoPlatform
>> _______________________________________________
>> Virtuoso-users mailing list
>> Vir...@li...
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>> 
>> 
> 
> 
> 
> -- 
> Paul Houle
> Expert on Freebase, DBpedia, Hadoop and RDF
> (607) 539 6254    paul.houle on Skype   ont...@gm...
> 
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform
> _______________________________________________
> Virtuoso-users mailing list
> Vir...@li...
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users