From: Rose B. <ros...@gm...> - 2013-12-18 11:53:26
|
Dear Developers, I installed 07.00.3207 OpenLink Virtuoso. After this I tried to load dbpedia to virtuoso. When loading 40+ GB files using DB.DBA.TTLP_MTWhen loading large files (2GB) using DB.DBA.TTLP_MT(file_to_string_output('filename'),'','graphname',512); virtuoso hangs (CPU usage 0%). Within my virtuoso.ini file I have allocated 64 GB RAM to virtuoso. The output after running virtuoso is mentioned below: /usr/local/virtuoso-opensource/bin# ./virtuoso-t -f -c /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.ini 04:48:30 SQL Optimizer enabled (max 1000 layouts) 04:48:31 Compiler unit is timed at 0.000286 msec 04:48:36 built-in procedure "repl_undot_name" overruled by the RDBMS 04:48:38 Roll forward started 04:48:38 Roll forward complete 04:48:39 Checkpoint started 04:48:40 Checkpoint finished, log reused 04:48:40 HTTP/WebDAV server online at 8890 04:48:40 Server online at 1111 (pid 4988) 04:52:54 * Monitor: Locks are held for a long time 04:54:55 * Monitor: Locks are held for a long time 04:56:55 * Monitor: Locks are held for a long time 05:01:07 * Monitor: Locks are held for a long time 05:03:07 * Monitor: Locks are held for a long time 05:05:08 * Monitor: Locks are held for a long time 05:07:09 * Monitor: Locks are held for a long time 05:09:10 * Monitor: Locks are held for a long time 05:11:10 * Monitor: Locks are held for a long time 05:13:11 * Monitor: Locks are held for a long time 05:15:22 * Monitor: Locks are held for a long time 05:17:30 * Monitor: Locks are held for a long time 05:19:43 * Monitor: Locks are held for a long time 05:29:50 * Monitor: Locks are held for a long time 05:31:51 * Monitor: Locks are held for a long time 05:33:54 * Monitor: Locks are held for a long time 05:35:56 * Monitor: Locks are held for a long time 05:37:59 * Monitor: Locks are held for a long time 05:39:59 * Monitor: Locks are held for a long time 05:41:59 * Monitor: Locks are held for a long time 05:44:02 * Monitor: Locks are held for a long time 05:46:02 * Monitor: Locks are held for a long time 05:48:02 * Monitor: Locks are held for a long time 05:49:35 * Monitor: Should read for update because lock escalation from shared to exclusive fails frequently (1) 05:50:03 * Monitor: Locks are held for a long time 05:50:52 Checkpoint started 05:52:06 Checkpoint finished, log reused 05:52:13 * Monitor: Locks are held for a long time 06:02:52 * Monitor: Many lock waits 06:04:52 * Monitor: Many lock waits 06:06:52 * Monitor: Many lock waits I tried to set DefaultIsolation = 2 in virtuoso.ini as mentioned at: https://github.com/openlink/virtuoso-opensource/issues/40. However, still Virtuoso continues to hang with 0% CPU usage. Can someone please help |
From: Hugh W. <hwi...@op...> - 2013-12-18 12:41:46
|
Hi Rose, Are you sure the buffer allocations you set in the INI file have taken effect ? Check with the "status('');" command to confirm they have as I suspect they have not, note you need to restart the server for INI file changes to take effect. Also, note you should use the Virtuoso RDF Bulk loader, which calls the relevant TTLP or RDFXML_LOAD() functions under the covers, for managing such large dataset loads as detailed at: http://www.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoader This being how we load DBpedia datasets into http://dbpedia.org/sparql (which we host) and other large datasets, with an example Dbpedia load detailed at: http://www.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoaderExampleDbpedia Best Regards Hugh Williams Professional Services OpenLink Software, Inc. // http://www.openlinksw.com/ Weblog -- http://www.openlinksw.com/blogs/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Google+ -- http://plus.google.com/100570109519069333827/ Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers On 18 Dec 2013, at 11:53, Rose Beck <ros...@gm...> wrote: > Dear Developers, > > I installed 07.00.3207 OpenLink Virtuoso. After this I tried to load dbpedia to virtuoso. When loading 40+ GB files using DB.DBA.TTLP_MTWhen loading large files (2GB) using DB.DBA.TTLP_MT(file_to_string_output('filename'),'','graphname',512); virtuoso hangs (CPU usage 0%). Within my virtuoso.ini file I have allocated 64 GB RAM to virtuoso. > > The output after running virtuoso is mentioned below: > /usr/local/virtuoso-opensource/bin# ./virtuoso-t -f -c /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.ini > 04:48:30 SQL Optimizer enabled (max 1000 layouts) > 04:48:31 Compiler unit is timed at 0.000286 msec > 04:48:36 built-in procedure "repl_undot_name" overruled by the RDBMS > 04:48:38 Roll forward started > 04:48:38 Roll forward complete > 04:48:39 Checkpoint started > 04:48:40 Checkpoint finished, log reused > 04:48:40 HTTP/WebDAV server online at 8890 > 04:48:40 Server online at 1111 (pid 4988) > 04:52:54 * Monitor: Locks are held for a long time > 04:54:55 * Monitor: Locks are held for a long time > 04:56:55 * Monitor: Locks are held for a long time > 05:01:07 * Monitor: Locks are held for a long time > 05:03:07 * Monitor: Locks are held for a long time > 05:05:08 * Monitor: Locks are held for a long time > 05:07:09 * Monitor: Locks are held for a long time > 05:09:10 * Monitor: Locks are held for a long time > 05:11:10 * Monitor: Locks are held for a long time > 05:13:11 * Monitor: Locks are held for a long time > 05:15:22 * Monitor: Locks are held for a long time > 05:17:30 * Monitor: Locks are held for a long time > 05:19:43 * Monitor: Locks are held for a long time > 05:29:50 * Monitor: Locks are held for a long time > 05:31:51 * Monitor: Locks are held for a long time > 05:33:54 * Monitor: Locks are held for a long time > 05:35:56 * Monitor: Locks are held for a long time > 05:37:59 * Monitor: Locks are held for a long time > 05:39:59 * Monitor: Locks are held for a long time > 05:41:59 * Monitor: Locks are held for a long time > 05:44:02 * Monitor: Locks are held for a long time > 05:46:02 * Monitor: Locks are held for a long time > 05:48:02 * Monitor: Locks are held for a long time > 05:49:35 * Monitor: Should read for update because lock escalation from shared to exclusive fails frequently (1) > 05:50:03 * Monitor: Locks are held for a long time > 05:50:52 Checkpoint started > 05:52:06 Checkpoint finished, log reused > 05:52:13 * Monitor: Locks are held for a long time > 06:02:52 * Monitor: Many lock waits > 06:04:52 * Monitor: Many lock waits > 06:06:52 * Monitor: Many lock waits > > I tried to set DefaultIsolation = 2 in virtuoso.ini as mentioned at: > https://github.com/openlink/virtuoso-opensource/issues/40. However, still Virtuoso continues to hang with 0% CPU usage. > > Can someone please help > > ------------------------------------------------------------------------------ > Rapidly troubleshoot problems before they affect your business. Most IT > organizations don't have a clear picture of how application performance > affects their revenue. With AppDynamics, you get 100% visibility into your > Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro! > http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk_______________________________________________ > Virtuoso-users mailing list > Vir...@li... > https://lists.sourceforge.net/lists/listinfo/virtuoso-users |
From: Dimitris K. <ji...@gm...> - 2013-12-19 05:10:49
|
I had a similar problem with v7.0.0 and big (ttl) files. I had to restart and try to reload many times until vos finally succeeded. I don't remember the status() output but looked like there was a deadlock. I noticed that *most times* the halting was right after I did a select count() to check the numbers of triples inserted Best, Dimitris On Wed, Dec 18, 2013 at 2:55 PM, Hugh Williams <hwi...@op...>wrote: > Hi Rose, > > Are you sure the buffer allocations you set in the INI file have taken > effect ? Check with the "status('');" command to confirm they have as I > suspect they have not, note you need to restart the server for INI file > changes to take effect. > > Also, note you should use the Virtuoso RDF Bulk loader, which calls the > relevant TTLP or RDFXML_LOAD() functions under the covers, for managing > such large dataset loads as detailed at: > > http://www.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoader > > This being how we load DBpedia datasets into http://dbpedia.org/sparql(which we host) and other large datasets, with an example Dbpedia load > detailed at: > > > http://www.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoaderExampleDbpedia > > Best Regards > Hugh Williams > Professional Services > OpenLink Software, Inc. // http://www.openlinksw.com/ > Weblog -- http://www.openlinksw.com/blogs/ > LinkedIn -- http://www.linkedin.com/company/openlink-software/ > Twitter -- http://twitter.com/OpenLink > Google+ -- http://plus.google.com/100570109519069333827/ > Facebook -- http://www.facebook.com/OpenLinkSoftware > Universal Data Access, Integration, and Management Technology Providers > > On 18 Dec 2013, at 11:53, Rose Beck <ros...@gm...> wrote: > > Dear Developers, > > I installed 07.00.3207 OpenLink Virtuoso. After this I tried to load > dbpedia to virtuoso. When loading 40+ GB files using DB.DBA.TTLP_MTWhen > loading large files (2GB) using > DB.DBA.TTLP_MT(file_to_string_output('filename'),'','graphname',512); > virtuoso hangs (CPU usage 0%). Within my virtuoso.ini file I have allocated > 64 GB RAM to virtuoso. > > The output after running virtuoso is mentioned below: > > > /usr/local/virtuoso-opensource/bin# ./virtuoso-t -f -c /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.ini > > 04:48:30 SQL Optimizer enabled (max 1000 layouts) > 04:48:31 Compiler unit is timed at 0.000286 msec > 04:48:36 built-in procedure "repl_undot_name" overruled by the RDBMS > 04:48:38 Roll forward started > 04:48:38 Roll forward complete > 04:48:39 Checkpoint started > 04:48:40 Checkpoint finished, log reused > 04:48:40 HTTP/WebDAV server online at 8890 > 04:48:40 Server online at 1111 (pid 4988) > 04:52:54 * Monitor: Locks are held for a long time > 04:54:55 * Monitor: Locks are held for a long time > 04:56:55 * Monitor: Locks are held for a long time > 05:01:07 * Monitor: Locks are held for a long time > 05:03:07 * Monitor: Locks are held for a long time > 05:05:08 * Monitor: Locks are held for a long time > 05:07:09 * Monitor: Locks are held for a long time > 05:09:10 * Monitor: Locks are held for a long time > 05:11:10 * Monitor: Locks are held for a long time > 05:13:11 * Monitor: Locks are held for a long time > 05:15:22 * Monitor: Locks are held for a long time > 05:17:30 * Monitor: Locks are held for a long time > 05:19:43 * Monitor: Locks are held for a long time > 05:29:50 * Monitor: Locks are held for a long time > 05:31:51 * Monitor: Locks are held for a long time > 05:33:54 * Monitor: Locks are held for a long time > 05:35:56 * Monitor: Locks are held for a long time > 05:37:59 * Monitor: Locks are held for a long time > 05:39:59 * Monitor: Locks are held for a long time > 05:41:59 * Monitor: Locks are held for a long time > 05:44:02 * Monitor: Locks are held for a long time > 05:46:02 * Monitor: Locks are held for a long time > 05:48:02 * Monitor: Locks are held for a long time > 05:49:35 * Monitor: Should read for update because lock escalation from > shared to exclusive fails frequently (1) > 05:50:03 * Monitor: Locks are held for a long time > 05:50:52 Checkpoint started > 05:52:06 Checkpoint finished, log reused > 05:52:13 * Monitor: Locks are held for a long time > 06:02:52 * Monitor: Many lock waits > 06:04:52 * Monitor: Many lock waits > 06:06:52 * Monitor: Many lock waits > > I tried to set DefaultIsolation = 2 in virtuoso.ini as mentioned at: > https://github.com/openlink/virtuoso-opensource/issues/40. However, still > Virtuoso continues to hang with 0% CPU usage. > > Can someone please help > > > ------------------------------------------------------------------------------ > Rapidly troubleshoot problems before they affect your business. Most IT > organizations don't have a clear picture of how application performance > affects their revenue. With AppDynamics, you get 100% visibility into your > Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics > Pro! > > http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk_______________________________________________ > Virtuoso-users mailing list > Vir...@li... > https://lists.sourceforge.net/lists/listinfo/virtuoso-users > > > > > ------------------------------------------------------------------------------ > Rapidly troubleshoot problems before they affect your business. Most IT > organizations don't have a clear picture of how application performance > affects their revenue. With AppDynamics, you get 100% visibility into your > Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics > Pro! > http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk > _______________________________________________ > Virtuoso-users mailing list > Vir...@li... > https://lists.sourceforge.net/lists/listinfo/virtuoso-users > > -- Kontokostas Dimitris |