Menu
▾
▴
bigdata-developers — List for bigdata developers
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
(19) |
Mar
(8) |
Apr
(25) |
May
(16) |
Jun
(77) |
Jul
(131) |
Aug
(76) |
Sep
(30) |
Oct
(7) |
Nov
(3) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(2) |
Jun
(2) |
Jul
(16) |
Aug
(3) |
Sep
(1) |
Oct
|
Nov
(7) |
Dec
(7) |
| 2012 |
Jan
(10) |
Feb
(1) |
Mar
(8) |
Apr
(6) |
May
(1) |
Jun
(3) |
Jul
(1) |
Aug
|
Sep
(1) |
Oct
|
Nov
(8) |
Dec
(2) |
| 2013 |
Jan
(5) |
Feb
(12) |
Mar
(2) |
Apr
(1) |
May
(1) |
Jun
(1) |
Jul
(22) |
Aug
(50) |
Sep
(31) |
Oct
(64) |
Nov
(83) |
Dec
(28) |
| 2014 |
Jan
(31) |
Feb
(18) |
Mar
(27) |
Apr
(39) |
May
(45) |
Jun
(15) |
Jul
(6) |
Aug
(27) |
Sep
(6) |
Oct
(67) |
Nov
(70) |
Dec
(1) |
| 2015 |
Jan
(3) |
Feb
(18) |
Mar
(22) |
Apr
(121) |
May
(42) |
Jun
(17) |
Jul
(8) |
Aug
(11) |
Sep
(26) |
Oct
(15) |
Nov
(66) |
Dec
(38) |
| 2016 |
Jan
(14) |
Feb
(59) |
Mar
(28) |
Apr
(44) |
May
(21) |
Jun
(12) |
Jul
(9) |
Aug
(11) |
Sep
(4) |
Oct
(2) |
Nov
(1) |
Dec
|
| 2017 |
Jan
(20) |
Feb
(7) |
Mar
(4) |
Apr
(18) |
May
(7) |
Jun
(3) |
Jul
(13) |
Aug
(2) |
Sep
(4) |
Oct
(9) |
Nov
(2) |
Dec
(5) |
| 2018 |
Jan
|
Feb
|
Mar
|
Apr
(2) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2019 |
Jan
|
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Bryan T. <br...@sy...> - 2013-03-01 19:25:31
|
Laurent,
This exception would be from using the wrong Sesame JARs. I think we probably failed to update the POM when we updated the JARs.
java.lang.NoSuchFieldError: NQUADS
at com.bigdata.rdf.rio.nquads.NQuadsParserFactory.getRDFFormat(NQuadsParserFactory.java:47)
I have just committed a fix for this. It is building in CI now. A new snapshot JAR should be posted shortly.
Truth maintenance is only supported for triples, not quads. There are several discussions on the forum concerning what would be required to support truth maintenance and inference over quads. Briefly, the problem is specifying which named graph(s) are combined and where the entailments are written. There are also several suggestions of alternative mechanisms involving triples that may be better suited to some applications, especially the triples plus statement level metadata database mode.
Thanks,
Bryan
From: Laurent Pellegrino <lau...@gm...<mailto:lau...@gm...>>
Date: Friday, March 1, 2013 6:41 AM
To: Bryan Thompson <br...@sy...<mailto:br...@sy...>>
Subject: Re: [Bigdata-developers] Jena Adapter?
Hello,
Thank you for the answers. I tried to create a Bigdata repository with support for quads with the following piece of code:
Properties properties =
new Properties();
File journal = File.createTempFile("bigdata", ".jnl");
properties.setProperty(
BigdataSail.Options.FILE, journal.getAbsolutePath());
properties.setProperty(
BigdataSail.Options.QUADS_MODE, "true");
properties.setProperty(
BigdataSail.Options.TRUTH_MAINTENANCE, "false");
BigdataSail sail = new BigdataSail(properties);
Repository repo = new BigdataSailRepository(sail);
repo.initialize();
repo.shutDown();
but the repository initialization seems not to work. I get the following error :
[main] ERROR rio.RDFParserRegistry - Failed to instantiate service
java.lang.NoSuchFieldError: NQUADS
at com.bigdata.rdf.rio.nquads.NQuadsParserFactory.getRDFFormat(NQuadsParserFactory.java:47)
at org.openrdf.rio.RDFParserRegistry.getKey(RDFParserRegistry.java:38)
at org.openrdf.rio.RDFParserRegistry.getKey(RDFParserRegistry.java:15)
at info.aduna.lang.service.ServiceRegistry.add(ServiceRegistry.java:74)
at info.aduna.lang.service.ServiceRegistry.<init>(ServiceRegistry.java:44)
at info.aduna.lang.service.FileFormatServiceRegistry.<init>(FileFormatServiceRegistry.java:20)
at org.openrdf.rio.RDFParserRegistry.<init>(RDFParserRegistry.java:33)
at org.openrdf.rio.RDFParserRegistry.getInstance(RDFParserRegistry.java:26)
at com.bigdata.rdf.ServiceProviderHook.forceLoad(ServiceProviderHook.java:109)
at com.bigdata.rdf.ServiceProviderHook.<clinit>(ServiceProviderHook.java:84)
at com.bigdata.rdf.store.AbstractTripleStore.<init>(AbstractTripleStore.java:1266)
at com.bigdata.rdf.store.AbstractLocalTripleStore.<init>(AbstractLocalTripleStore.java:57)
at com.bigdata.rdf.store.LocalTripleStore.<init>(LocalTripleStore.java:161)
at com.bigdata.rdf.sail.BigdataSail.createLTS(BigdataSail.java:726)
at com.bigdata.rdf.sail.BigdataSail.createLTS(BigdataSail.java:653)
at com.bigdata.rdf.sail.BigdataSail.<init>(BigdataSail.java:630)
at fr.inria.eventcloud.stash.rdfstores.benchmark.BigdataRepository.createBigDataRepository(BigdataRepository.java:53)
at fr.inria.eventcloud.stash.rdfstores.benchmark.RDFStoreBenchmark.testBigdataSequential(RDFStoreBenchmark.java:45)
at fr.inria.eventcloud.stash.rdfstores.benchmark.RDFStoreBenchmark.main(RDFStoreBenchmark.java:77)
I have set truth maintenance to false given that I get the following message if I let it to the default value when quads mode is set to true:
java.lang.UnsupportedOperationException: com.bigdata.rdf.sail.truthMaintenance is not supported with quads (com.bigdata.rdf.store.AbstractTripleStore.quads)
My test was performed by using bigdata version 1.2.2-SNAPSHOT from your maven repository.
I cannot figure out what is wrong in the previous piece of code. Could you help me?
Kind Regards,
Laurent
On Thu, Feb 28, 2013 at 1:12 PM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote:
Laurent,
Unfortunately not. This has been discussed several times, but never implemented. However, the advent of SPARQL UPDATE makes it much easier to write portable applications. If you like, feel free to post questions about how to achieve certain things under bigdata that you currently accomplish with Jena.
Thanks,
Bryan
From: Laurent Pellegrino <lau...@gm...<mailto:lau...@gm...>>
Date: Thursday, February 28, 2013 4:27 AM
To: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>>
Subject: [Bigdata-developers] Jena Adapter?
Hello all,
I am investigating the use of Bigdata in one application that currently uses Jena TDB. I would like to test Bigdata without rewriting all, that's why I wonder if there exists a jena adapter on top of Bigdata?
Kind Regards,
Laurent
|
|
From: Bryan T. <br...@sy...> - 2013-02-28 21:40:12
|
Laurent, Unfortunately not. This has been discussed several times, but never implemented. However, the advent of SPARQL UPDATE makes it much easier to write portable applications. If you like, feel free to post questions about how to achieve certain things under bigdata that you currently accomplish with Jena. Thanks, Bryan From: Laurent Pellegrino <lau...@gm...<mailto:lau...@gm...>> Date: Thursday, February 28, 2013 4:27 AM To: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: [Bigdata-developers] Jena Adapter? Hello all, I am investigating the use of Bigdata in one application that currently uses Jena TDB. I would like to test Bigdata without rewriting all, that's why I wonder if there exists a jena adapter on top of Bigdata? Kind Regards, Laurent |
|
From: Laurent P. <lau...@gm...> - 2013-02-28 09:29:31
|
Hello all, I am investigating the use of Bigdata in one application that currently uses Jena TDB. I would like to test Bigdata without rewriting all, that's why I wonder if there exists a jena adapter on top of Bigdata? Kind Regards, Laurent |
|
From: Bryan T. <br...@sy...> - 2013-02-27 12:23:27
|
To summarize, some recent Java releases for both 1.6.0 and 1.7.0 have broken the serialization contract for custom log Levels used by Apache River (aka Jini). See https://issues.apache.org/jira/browse/RIVER-416. Also see below in this thread. I ran into this problem under OSX where it is extraordinarily difficult to manage the installed java versions. I had accepted an upgrade to Java 1.6.0_41, which hit this bug. I managed to get things working again by moving to 1.7.0_11. There are no OSX installers for the 1.6.x JDK available for versions that are compatible with the current River release. The only available 1.6.0 installer for OSX is for 1.6.0_41, which hits the custom Level serialization problem. I was able to find an installer for an earlier version of 1.7.0 (before the version that introduces the level serialization problem). I am now able to start the LUS again Š. Some notes below for the interested. JAVA OSX VERSION CONTROL Note: Java is not installed consistently for all versions of 1.6 and 1.7. Also, note that only the most recent versions of 1.6 have installers available from Oracle. Finally, note that there appear to be two different paths (/System/Library and /Library) in which different java versions may be visible and that the java version actually installed under that path MIGHT NOT agree with the path name (e.g., 1.6.0_41 appears under the name 1.6.0_35-b10-428.jdk for me.) Finally, note that 1.6.0_41 is NOT compatible with River (see https://issues.apache.org/jira/browse/RIVER-416). # Show the available versions. ls -l /Library/Java/JavaVirtualMachines ls -l /System/Library/Java/JavaVirtualMachines See http://www.oracle.com/technetwork/java/javase/downloads/index.html (downloads) See http://docs.oracle.com/javase/7/docs/webnotes/install/mac/mac-jdk.html#unin stall To run a different version of Java, either specify the full path, or use the java_home tool: /usr/libexec/java_home -v 1.7.0_06 --exec javac -version Thanks, Bryan On 2/27/13 2:54 AM, "Peter Firmstone" <ji...@ze...> wrote: >Oh, I think I know what this is. > >Oracle broke the Serialization contract of Level class, did you >recently upgrade your Java runtime? > >See River-416 on Jira: > >The com.sun.jini.logging.Levels class produces a RuntimeException with >the latest version of Java (both 1.6 and 1.7). The issue surrounds >creation of custom java.util.logging.Level. > >I hope your production systems haven't been upgraded to the latest Java? > >Peter. > > >On 27/02/2013 8:23 AM, Bryan Thompson wrote: >> I did a reboot to try and clear this up. The problem persists. Any >> other thoughts? >> >> It is possible that the problem is unrelated to the QA runs, but if so >> I am not sure what triggered it or why it is failing. >> >> Thanks, >> Bryan >> >> From: Peter <ji...@ze... <mailto:ji...@ze...>> >> Reply-To: Peter <ji...@ze... <mailto:ji...@ze...>> >> Date: Tuesday, February 26, 2013 3:31 PM >> To: Bryan Thompson <br...@sy... <mailto:br...@sy...>> >> Subject: Re: QA Test debugging >> >> They can interract over the network, it's likely there's a port that >> hasn't been freed by a stale process. Probably 4160. >> >> Seems to be causing blocking input streams. >> >> >> ----- Original message ----- >> > Peter, >> > >> > Is there anything about the recent qa runs I've been doing that could >> have >> > messed up my ability to start river? I am now seeing this when I try >> to start >> > the LUS on my laptop. The trace is a bit odd since it has to do with >> an error >> > in the setup of the loggers. I would not have expected any >> interaction between >> > the River QA runs and this code as they are in separate directory >>trees. >> > >> > Thanks, >> > Bryan >> > >> > >> > java.lang.ExceptionInInitializerError >> > >> > at >> > >>net.jini.config.AbstractConfiguration.getEntryInternal(AbstractConfigurat >>ion.java:371) >> > >> > at >> >>net.jini.config.AbstractConfiguration.getEntry(AbstractConfiguration.java >>:193) >> > >> > at net.jini.discovery.LookupDiscovery.init(LookupDiscovery.java:3141) >> > >> > at >> >>net.jini.discovery.LookupDiscovery.beginDiscovery(LookupDiscovery.java:30 >>86) >> > >> > at >>net.jini.discovery.LookupDiscovery.<init>(LookupDiscovery.java:1946) >> > >> > at >> > >>net.jini.discovery.LookupDiscoveryManager.beginDiscovery(LookupDiscoveryM >>anager.java:1549) >> > >> > at >> > >>net.jini.discovery.LookupDiscoveryManager.<init>(LookupDiscoveryManager.j >>ava:902) >> > >> > at >> > >>com.bigdata.service.jini.util.LookupStarter.setupDiscovery(LookupStarter. >>java:192) >> > >> > at >> >>com.bigdata.service.jini.util.LookupStarter.access$0(LookupStarter.java:1 >>75) >> > >> > at >> > >>com.bigdata.service.jini.util.LookupStarter$LookupStartThread.run(LookupS >>tarter.java:148) >> > >> > Caused by: java.lang.RuntimeException: Unexpected exception >> > >> > at com.sun.jini.logging.Levels.createLevel(Levels.java:142) >> > >> > at com.sun.jini.logging.Levels.<clinit>(Levels.java:52) >> > >> > ... 10 more >> > >> > Caused by: java.io.EOFException >> > >> > at >> > >>java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream >>.java:2557) >> > >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1294) >> > >> > at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1950) >> > >> > at >>java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1874) >> > >> > at >> >>java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756) >> > >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326) >> > >> > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) >> > >> > at com.sun.jini.logging.Levels.createLevel(Levels.java:138) >> > >> > ... 11 more |
|
From: Bryan T. <br...@sy...> - 2013-02-07 21:38:32
|
Excellent. I am glad that is working out for you. Bryan From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Thursday, February 7, 2013 4:36 PM To: Bryan Thompson <br...@sy...<mailto:br...@sy...>> Cc: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: Re: [Bigdata-developers] Lock Down a BigData server Bryan, I've checked out the developer branch and tested both readOnly and queryTimeout. POST when in readOnly does not allow me insert and returns a string Not writable. And if queryTimeout is non-zero and exceeded, then it throws an exception java.util.concurrent.ExecutionException: java.lang.Exception: org.openrdf.query.QueryInterruptedException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.bigdata.bop.engine.QueryTimeoutException: Query deadline is expired. Super turnaround. Thanks a bunch, Doug On Thu, Feb 7, 2013 at 1:09 PM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote: Doug, I've just committed a fix for this [1]. Please try it out. I have verified the readOnly feature on a local WAR deployment. I have not tested the queryTimeout limit feature. Thanks, Bryan [1] https://sourceforge.net/apps/trac/bigdata/ticket/643#comment:1 From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Thursday, February 7, 2013 10:39 AM To: Bryan Thompson <br...@sy...<mailto:br...@sy...>> Cc: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: Re: [Bigdata-developers] Lock Down a BigData server How about a simple init-param for each? If they are not there, then let it go unfettered (as it is now). Something like: <servlet> <servlet-name>REST API</servlet-name> <display-name>REST API</display-name> <description>The REST API, including a SPARQL end point, as described at https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=NanoSparqlServer </description> <init-param> <param-name>readOnly</param-name> <param-value>true</param-value> </init-param> <init-param> <param-name>maxQueryMs</param-name> <param-value>20000</param-value> </init-param> <servlet-class>com.bigdata.rdf.sail.webapp.RESTServlet</servlet-class> <load-on-startup>0</load-on-startup> </servlet> At least for my use case, I load the data as an admin. When I'm happy, I lock it down so that neither I nor anyone else can change it. Further, if I wanted to have a public and a private (admin) interface, I could have a separate servlet-mapping and separate servlet up with different init-param tags On Thu, Feb 7, 2013 at 10:20 AM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote: Using POST for Query is a hugely bad idea unless your goal is to defeat the HTTP caching layer. It is easy enough to add a configuration option that disables mutation. However, this would have to be selectively disabled|enabled for specific interfaces in order to let you use the REST API to update the database while denying that privilege to public facing interfaces. Do you have a concrete suggestion on how to specify the set of interfaces for which update should be allowed or denied? See the SparqlEndpointConfig class. This is populated from web.xml in BigdataRDFServletContextListener. If you can suggest a mechanism for allowing update on only certain interfaces, then I am happy to integrate it. We could use a similar mechanism to impose a timeout on queries according to the interface on which they arrive. Thanks, Bryan From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Thursday, February 7, 2013 10:13 AM To: Bryan Thompson <br...@sy...<mailto:br...@sy...>> Cc: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: Re: [Bigdata-developers] Lock Down a BigData server Yes the intent is to expose an idempotent SPARQL 1.1 Query interface. However, I don't think denying POST is an option, as: Many clients use forms to POST *to avoid query strings from showing up on the URL *to avoid URL length potential issues And many frameworks such as ELDA only use post So there is no way to make BigData idempotent without parsing the query string for myself? On Fri, Dec 14, 2012 at 7:58 AM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote: Doug, Is your intention to expose the SPARQL 1.1 Query interface? If so, you can just deny any HTTP method other than GET. You can also restrict the URIs that are visible to only include the …//sparql path for the external interface of your host. That will make things like the performance counters and the status pages unavailable. Many times people narrow the interface further through an application layer rather than exposing the full SPARQL query language. There is an open thread on the discussion forum [1] for imposing a timeout on all external interface requests. That thread suggests how this can be done by modifying the code and some possible approaches to automating this policy. Thanks, Bryan [1] https://sourceforge.net/projects/bigdata/forums/forum/676946/topic/6133565 From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Wednesday, December 12, 2012 10:20 PM To: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: [Bigdata-developers] Lock Down a BigData server I like and use the SPARQL 1.1 interface to load and delete data. However, when I take this puppy to production, I don't want others to be able to modify my endpoint. Sorry for the n00b question, but... After I have loaded data, how do I lock it down? Thanks, Doug |
|
From: Doug W. <dou...@gm...> - 2013-02-07 21:36:43
|
Bryan, I've checked out the developer branch and tested both readOnly and queryTimeout. POST when in readOnly does not allow me insert and returns a string Not writable. And if queryTimeout is non-zero and exceeded, then it throws an exception java.util.concurrent.ExecutionException: java.lang.Exception: org.openrdf.query.QueryInterruptedException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.bigdata.bop.engine.QueryTimeoutException: Query deadline is expired. Super turnaround. Thanks a bunch, Doug On Thu, Feb 7, 2013 at 1:09 PM, Bryan Thompson <br...@sy...> wrote: > Doug, > > I've just committed a fix for this [1]. Please try it out. I have > verified the readOnly feature on a local WAR deployment. I have not tested > the queryTimeout limit feature. > > Thanks, > Bryan > > [1] https://sourceforge.net/apps/trac/bigdata/ticket/643#comment:1 > > From: Doug Whitehead <dou...@gm...> > Date: Thursday, February 7, 2013 10:39 AM > > To: Bryan Thompson <br...@sy...> > Cc: "big...@li..." < > big...@li...> > Subject: Re: [Bigdata-developers] Lock Down a BigData server > > How about a simple init-param for each? If they are not there, then let > it go unfettered (as it is now). Something like: > > <servlet> > <servlet-name>REST API</servlet-name> > <display-name>REST API</display-name> > <description>The REST API, including a SPARQL end point, as > described at > > https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=NanoSparqlServer > </description> > <init-param> > <param-name>readOnly</param-name> > <param-value>true</param-value> > </init-param> > <init-param> > <param-name>maxQueryMs</param-name> > <param-value>20000</param-value> > </init-param> > > <servlet-class>com.bigdata.rdf.sail.webapp.RESTServlet</servlet-class> > <load-on-startup>0</load-on-startup> > </servlet> > > At least for my use case, I load the data as an admin. When I'm happy, I > lock it down so that neither I nor anyone else can change it. > > Further, if I wanted to have a public and a private (admin) interface, I > could have a separate servlet-mapping and separate servlet up with > different init-param tags > > On Thu, Feb 7, 2013 at 10:20 AM, Bryan Thompson <br...@sy...> wrote: > >> Using POST for Query is a hugely bad idea unless your goal is to defeat >> the HTTP caching layer. >> >> It is easy enough to add a configuration option that disables mutation. >> However, this would have to be selectively disabled|enabled for specific >> interfaces in order to let you use the REST API to update the database >> while denying that privilege to public facing interfaces. >> >> Do you have a concrete suggestion on how to specify the set of interfaces >> for which update should be allowed or denied? See the SparqlEndpointConfig >> class. This is populated from web.xml in BigdataRDFServletContextListener. >> If you can suggest a mechanism for allowing update on only certain >> interfaces, then I am happy to integrate it. >> >> We could use a similar mechanism to impose a timeout on queries according >> to the interface on which they arrive. >> >> Thanks, >> Bryan >> * >> * >> >> From: Doug Whitehead <dou...@gm...> >> Date: Thursday, February 7, 2013 10:13 AM >> To: Bryan Thompson <br...@sy...> >> Cc: "big...@li..." < >> big...@li...> >> Subject: Re: [Bigdata-developers] Lock Down a BigData server >> >> Yes the intent is to expose an idempotent SPARQL 1.1 Query interface. >> >> However, I don't think denying POST is an option, as: >> >> Many clients use forms to POST >> *to avoid query strings from showing up on the URL >> *to avoid URL length potential issues >> And many frameworks such as ELDA only use post >> >> So there is no way to make BigData idempotent without parsing the query >> string for myself? >> >> >> On Fri, Dec 14, 2012 at 7:58 AM, Bryan Thompson <br...@sy...> wrote: >> >>> Doug, >>> >>> Is your intention to expose the SPARQL 1.1 Query interface? If so, you >>> can just deny any HTTP method other than GET. You can also restrict the >>> URIs that are visible to only include the …//sparql path for the external >>> interface of your host. That will make things like the performance >>> counters and the status pages unavailable. >>> >>> Many times people narrow the interface further through an application >>> layer rather than exposing the full SPARQL query language. There is an >>> open thread on the discussion forum [1] for imposing a timeout on all >>> external interface requests. That thread suggests how this can be done by >>> modifying the code and some possible approaches to automating this policy. >>> >>> Thanks, >>> Bryan >>> >>> [1] >>> https://sourceforge.net/projects/bigdata/forums/forum/676946/topic/6133565 >>> >>> From: Doug Whitehead <dou...@gm...> >>> Date: Wednesday, December 12, 2012 10:20 PM >>> To: "big...@li..." < >>> big...@li...> >>> Subject: [Bigdata-developers] Lock Down a BigData server >>> >>> I like and use the SPARQL 1.1 interface to load and delete data. >>> However, when I take this puppy to production, I don't want others to be >>> able to modify my endpoint. >>> >>> Sorry for the n00b question, but... >>> After I have loaded data, how do I lock it down? >>> >>> Thanks, >>> Doug >>> >> >> > |
|
From: Bryan T. <br...@sy...> - 2013-02-07 18:30:17
|
Doug, I've just committed a fix for this [1]. Please try it out. I have verified the readOnly feature on a local WAR deployment. I have not tested the queryTimeout limit feature. Thanks, Bryan [1] https://sourceforge.net/apps/trac/bigdata/ticket/643#comment:1 From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Thursday, February 7, 2013 10:39 AM To: Bryan Thompson <br...@sy...<mailto:br...@sy...>> Cc: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: Re: [Bigdata-developers] Lock Down a BigData server How about a simple init-param for each? If they are not there, then let it go unfettered (as it is now). Something like: <servlet> <servlet-name>REST API</servlet-name> <display-name>REST API</display-name> <description>The REST API, including a SPARQL end point, as described at https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=NanoSparqlServer </description> <init-param> <param-name>readOnly</param-name> <param-value>true</param-value> </init-param> <init-param> <param-name>maxQueryMs</param-name> <param-value>20000</param-value> </init-param> <servlet-class>com.bigdata.rdf.sail.webapp.RESTServlet</servlet-class> <load-on-startup>0</load-on-startup> </servlet> At least for my use case, I load the data as an admin. When I'm happy, I lock it down so that neither I nor anyone else can change it. Further, if I wanted to have a public and a private (admin) interface, I could have a separate servlet-mapping and separate servlet up with different init-param tags On Thu, Feb 7, 2013 at 10:20 AM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote: Using POST for Query is a hugely bad idea unless your goal is to defeat the HTTP caching layer. It is easy enough to add a configuration option that disables mutation. However, this would have to be selectively disabled|enabled for specific interfaces in order to let you use the REST API to update the database while denying that privilege to public facing interfaces. Do you have a concrete suggestion on how to specify the set of interfaces for which update should be allowed or denied? See the SparqlEndpointConfig class. This is populated from web.xml in BigdataRDFServletContextListener. If you can suggest a mechanism for allowing update on only certain interfaces, then I am happy to integrate it. We could use a similar mechanism to impose a timeout on queries according to the interface on which they arrive. Thanks, Bryan From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Thursday, February 7, 2013 10:13 AM To: Bryan Thompson <br...@sy...<mailto:br...@sy...>> Cc: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: Re: [Bigdata-developers] Lock Down a BigData server Yes the intent is to expose an idempotent SPARQL 1.1 Query interface. However, I don't think denying POST is an option, as: Many clients use forms to POST *to avoid query strings from showing up on the URL *to avoid URL length potential issues And many frameworks such as ELDA only use post So there is no way to make BigData idempotent without parsing the query string for myself? On Fri, Dec 14, 2012 at 7:58 AM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote: Doug, Is your intention to expose the SPARQL 1.1 Query interface? If so, you can just deny any HTTP method other than GET. You can also restrict the URIs that are visible to only include the …//sparql path for the external interface of your host. That will make things like the performance counters and the status pages unavailable. Many times people narrow the interface further through an application layer rather than exposing the full SPARQL query language. There is an open thread on the discussion forum [1] for imposing a timeout on all external interface requests. That thread suggests how this can be done by modifying the code and some possible approaches to automating this policy. Thanks, Bryan [1] https://sourceforge.net/projects/bigdata/forums/forum/676946/topic/6133565 From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Wednesday, December 12, 2012 10:20 PM To: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: [Bigdata-developers] Lock Down a BigData server I like and use the SPARQL 1.1 interface to load and delete data. However, when I take this puppy to production, I don't want others to be able to modify my endpoint. Sorry for the n00b question, but... After I have loaded data, how do I lock it down? Thanks, Doug |
|
From: Bryan T. <br...@sy...> - 2013-02-07 15:50:00
|
Sounds good. It is much simpler if you admin first, then lock down to make expose to the public. Here is a ticket for this [1]. Bryan [1] https://sourceforge.net/apps/trac/bigdata/ticket/643 (Add web.xml parameters to lock down public NSS end points) From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Thursday, February 7, 2013 10:39 AM To: Bryan Thompson <br...@sy...<mailto:br...@sy...>> Cc: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: Re: [Bigdata-developers] Lock Down a BigData server How about a simple init-param for each? If they are not there, then let it go unfettered (as it is now). Something like: <servlet> <servlet-name>REST API</servlet-name> <display-name>REST API</display-name> <description>The REST API, including a SPARQL end point, as described at https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=NanoSparqlServer </description> <init-param> <param-name>readOnly</param-name> <param-value>true</param-value> </init-param> <init-param> <param-name>maxQueryMs</param-name> <param-value>20000</param-value> </init-param> <servlet-class>com.bigdata.rdf.sail.webapp.RESTServlet</servlet-class> <load-on-startup>0</load-on-startup> </servlet> At least for my use case, I load the data as an admin. When I'm happy, I lock it down so that neither I nor anyone else can change it. Further, if I wanted to have a public and a private (admin) interface, I could have a separate servlet-mapping and separate servlet up with different init-param tags On Thu, Feb 7, 2013 at 10:20 AM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote: Using POST for Query is a hugely bad idea unless your goal is to defeat the HTTP caching layer. It is easy enough to add a configuration option that disables mutation. However, this would have to be selectively disabled|enabled for specific interfaces in order to let you use the REST API to update the database while denying that privilege to public facing interfaces. Do you have a concrete suggestion on how to specify the set of interfaces for which update should be allowed or denied? See the SparqlEndpointConfig class. This is populated from web.xml in BigdataRDFServletContextListener. If you can suggest a mechanism for allowing update on only certain interfaces, then I am happy to integrate it. We could use a similar mechanism to impose a timeout on queries according to the interface on which they arrive. Thanks, Bryan From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Thursday, February 7, 2013 10:13 AM To: Bryan Thompson <br...@sy...<mailto:br...@sy...>> Cc: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: Re: [Bigdata-developers] Lock Down a BigData server Yes the intent is to expose an idempotent SPARQL 1.1 Query interface. However, I don't think denying POST is an option, as: Many clients use forms to POST *to avoid query strings from showing up on the URL *to avoid URL length potential issues And many frameworks such as ELDA only use post So there is no way to make BigData idempotent without parsing the query string for myself? On Fri, Dec 14, 2012 at 7:58 AM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote: Doug, Is your intention to expose the SPARQL 1.1 Query interface? If so, you can just deny any HTTP method other than GET. You can also restrict the URIs that are visible to only include the …//sparql path for the external interface of your host. That will make things like the performance counters and the status pages unavailable. Many times people narrow the interface further through an application layer rather than exposing the full SPARQL query language. There is an open thread on the discussion forum [1] for imposing a timeout on all external interface requests. That thread suggests how this can be done by modifying the code and some possible approaches to automating this policy. Thanks, Bryan [1] https://sourceforge.net/projects/bigdata/forums/forum/676946/topic/6133565 From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Wednesday, December 12, 2012 10:20 PM To: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: [Bigdata-developers] Lock Down a BigData server I like and use the SPARQL 1.1 interface to load and delete data. However, when I take this puppy to production, I don't want others to be able to modify my endpoint. Sorry for the n00b question, but... After I have loaded data, how do I lock it down? Thanks, Doug |
|
From: Doug W. <dou...@gm...> - 2013-02-07 15:40:01
|
How about a simple init-param for each? If they are not there, then let it
go unfettered (as it is now). Something like:
<servlet>
<servlet-name>REST API</servlet-name>
<display-name>REST API</display-name>
<description>The REST API, including a SPARQL end point, as
described at
https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=NanoSparqlServer
</description>
<init-param>
<param-name>readOnly</param-name>
<param-value>true</param-value>
</init-param>
<init-param>
<param-name>maxQueryMs</param-name>
<param-value>20000</param-value>
</init-param>
<servlet-class>com.bigdata.rdf.sail.webapp.RESTServlet</servlet-class>
<load-on-startup>0</load-on-startup>
</servlet>
At least for my use case, I load the data as an admin. When I'm happy, I
lock it down so that neither I nor anyone else can change it.
Further, if I wanted to have a public and a private (admin) interface, I
could have a separate servlet-mapping and separate servlet up with
different init-param tags
On Thu, Feb 7, 2013 at 10:20 AM, Bryan Thompson <br...@sy...> wrote:
> Using POST for Query is a hugely bad idea unless your goal is to defeat
> the HTTP caching layer.
>
> It is easy enough to add a configuration option that disables mutation.
> However, this would have to be selectively disabled|enabled for specific
> interfaces in order to let you use the REST API to update the database
> while denying that privilege to public facing interfaces.
>
> Do you have a concrete suggestion on how to specify the set of interfaces
> for which update should be allowed or denied? See the SparqlEndpointConfig
> class. This is populated from web.xml in BigdataRDFServletContextListener.
> If you can suggest a mechanism for allowing update on only certain
> interfaces, then I am happy to integrate it.
>
> We could use a similar mechanism to impose a timeout on queries according
> to the interface on which they arrive.
>
> Thanks,
> Bryan
> *
> *
>
> From: Doug Whitehead <dou...@gm...>
> Date: Thursday, February 7, 2013 10:13 AM
> To: Bryan Thompson <br...@sy...>
> Cc: "big...@li..." <
> big...@li...>
> Subject: Re: [Bigdata-developers] Lock Down a BigData server
>
> Yes the intent is to expose an idempotent SPARQL 1.1 Query interface.
>
> However, I don't think denying POST is an option, as:
>
> Many clients use forms to POST
> *to avoid query strings from showing up on the URL
> *to avoid URL length potential issues
> And many frameworks such as ELDA only use post
>
> So there is no way to make BigData idempotent without parsing the query
> string for myself?
>
>
> On Fri, Dec 14, 2012 at 7:58 AM, Bryan Thompson <br...@sy...> wrote:
>
>> Doug,
>>
>> Is your intention to expose the SPARQL 1.1 Query interface? If so, you
>> can just deny any HTTP method other than GET. You can also restrict the
>> URIs that are visible to only include the …//sparql path for the external
>> interface of your host. That will make things like the performance
>> counters and the status pages unavailable.
>>
>> Many times people narrow the interface further through an application
>> layer rather than exposing the full SPARQL query language. There is an
>> open thread on the discussion forum [1] for imposing a timeout on all
>> external interface requests. That thread suggests how this can be done by
>> modifying the code and some possible approaches to automating this policy.
>>
>> Thanks,
>> Bryan
>>
>> [1]
>> https://sourceforge.net/projects/bigdata/forums/forum/676946/topic/6133565
>>
>> From: Doug Whitehead <dou...@gm...>
>> Date: Wednesday, December 12, 2012 10:20 PM
>> To: "big...@li..." <
>> big...@li...>
>> Subject: [Bigdata-developers] Lock Down a BigData server
>>
>> I like and use the SPARQL 1.1 interface to load and delete data.
>> However, when I take this puppy to production, I don't want others to be
>> able to modify my endpoint.
>>
>> Sorry for the n00b question, but...
>> After I have loaded data, how do I lock it down?
>>
>> Thanks,
>> Doug
>>
>
>
|
|
From: Bryan T. <br...@sy...> - 2013-02-07 15:21:13
|
Using POST for Query is a hugely bad idea unless your goal is to defeat the HTTP caching layer. It is easy enough to add a configuration option that disables mutation. However, this would have to be selectively disabled|enabled for specific interfaces in order to let you use the REST API to update the database while denying that privilege to public facing interfaces. Do you have a concrete suggestion on how to specify the set of interfaces for which update should be allowed or denied? See the SparqlEndpointConfig class. This is populated from web.xml in BigdataRDFServletContextListener. If you can suggest a mechanism for allowing update on only certain interfaces, then I am happy to integrate it. We could use a similar mechanism to impose a timeout on queries according to the interface on which they arrive. Thanks, Bryan From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Thursday, February 7, 2013 10:13 AM To: Bryan Thompson <br...@sy...<mailto:br...@sy...>> Cc: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: Re: [Bigdata-developers] Lock Down a BigData server Yes the intent is to expose an idempotent SPARQL 1.1 Query interface. However, I don't think denying POST is an option, as: Many clients use forms to POST *to avoid query strings from showing up on the URL *to avoid URL length potential issues And many frameworks such as ELDA only use post So there is no way to make BigData idempotent without parsing the query string for myself? On Fri, Dec 14, 2012 at 7:58 AM, Bryan Thompson <br...@sy...<mailto:br...@sy...>> wrote: Doug, Is your intention to expose the SPARQL 1.1 Query interface? If so, you can just deny any HTTP method other than GET. You can also restrict the URIs that are visible to only include the …//sparql path for the external interface of your host. That will make things like the performance counters and the status pages unavailable. Many times people narrow the interface further through an application layer rather than exposing the full SPARQL query language. There is an open thread on the discussion forum [1] for imposing a timeout on all external interface requests. That thread suggests how this can be done by modifying the code and some possible approaches to automating this policy. Thanks, Bryan [1] https://sourceforge.net/projects/bigdata/forums/forum/676946/topic/6133565 From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Wednesday, December 12, 2012 10:20 PM To: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: [Bigdata-developers] Lock Down a BigData server I like and use the SPARQL 1.1 interface to load and delete data. However, when I take this puppy to production, I don't want others to be able to modify my endpoint. Sorry for the n00b question, but... After I have loaded data, how do I lock it down? Thanks, Doug |
|
From: Doug W. <dou...@gm...> - 2013-02-07 15:13:56
|
Yes the intent is to expose an idempotent SPARQL 1.1 Query interface. However, I don't think denying POST is an option, as: Many clients use forms to POST *to avoid query strings from showing up on the URL *to avoid URL length potential issues And many frameworks such as ELDA only use post So there is no way to make BigData idempotent without parsing the query string for myself? On Fri, Dec 14, 2012 at 7:58 AM, Bryan Thompson <br...@sy...> wrote: > Doug, > > Is your intention to expose the SPARQL 1.1 Query interface? If so, you > can just deny any HTTP method other than GET. You can also restrict the > URIs that are visible to only include the …//sparql path for the external > interface of your host. That will make things like the performance > counters and the status pages unavailable. > > Many times people narrow the interface further through an application > layer rather than exposing the full SPARQL query language. There is an > open thread on the discussion forum [1] for imposing a timeout on all > external interface requests. That thread suggests how this can be done by > modifying the code and some possible approaches to automating this policy. > > Thanks, > Bryan > > [1] > https://sourceforge.net/projects/bigdata/forums/forum/676946/topic/6133565 > > From: Doug Whitehead <dou...@gm...> > Date: Wednesday, December 12, 2012 10:20 PM > To: "big...@li..." < > big...@li...> > Subject: [Bigdata-developers] Lock Down a BigData server > > I like and use the SPARQL 1.1 interface to load and delete data. > However, when I take this puppy to production, I don't want others to be > able to modify my endpoint. > > Sorry for the n00b question, but... > After I have loaded data, how do I lock it down? > > Thanks, > Doug > |
|
From: Bryan T. <br...@sy...> - 2013-02-07 13:19:34
|
Antoni, Incremental inference is not supported in scale-out (in the JiniFederation). It requires a stable (unchanging) view of the committed triples in order to do truth maintenance. This is at odds with the concept that updates could originate from multiple concurrent clients. In order to do truth maintenance, we would have to impose a constraint that all updates flow through a single point. The JiniFederation does support database at once closure. So you can load large data sets and (re-) compute their closure. You can also arrange to coordinate the read-behind view used by the clients for query such that they only see those points when the closure is complete. The HAJournalServer (replication cluster) does support incremental truth maintenance. This is basically the RWStore mode journal using a low-level write replication protocol to mirror the state of the database onto a highly available quorum of machines. The replication cluster is designed to provide high availability and to scale-out the capability for concurrent query. However, it does not shard the data itself so its data scale is limited by the capacity of a single machine. In terms of the NSS REST API, operations should be done by the time the HTTP response is received. All writes should be stable on the disk. However, per above, the database is not computing inferences incrementally during updates on the JiniFederation. If your inferences can be completed entirely within the scope of the triples that you are asserting, then you can partition the inference problem. I.e., sometimes you can do the inference before you load the data. However, if you need inferences that runs across all the data in the database and you need to do low latency updates with incremental truth maintenance on those inferences, then you can not do that on the JiniFederation. The HAJournalServer provides a solution that scales query horizontally and data up to 50,000,000,000 triples/quads. If you need more data scale, then you could use property paths (recently committed to the development branch) or possibly integrate a higher level reasoner into the query engine. Please tell us a little bit more about what you are trying to achieve here. Thanks, Bryan On 2/7/13 4:41 AM, "Antoni Mylka" <ant...@ba...> wrote: >Hi, > >Is inference done asynchronously in Bigdata? > >In my integration tests I see spurious failures: > >1. I run code that does SPARQL UPDATE queries. It uses SPARQLRepository >or direct, handcrafted invocations of the REST API. The production code >uses HTTP Client connection pooling and it's difficult to say how many >physical connections it uses and when they stop. >2. When that code finishes, I create a new SPARQLRepository and use it to >see if content of the repository meets my expectations. >3. The test code sometimes succeeds, sometimes fails. > >The NanoSparqlServer wikipage states that "Mutation operations are ACID >against a standalone database and shard-wise ACID against a bigdata >federation." Is inference done within the write transaction, or does the >ACID only refer to explicit triples? > >Obviously there may be 1001 other reasons for the test failures I'm >seeing. If you clearly confirm that when Bigdata says it's done - it >means it's done I can look further. > >Another idea for investigation would be to check how HTTPClient >connection pooling influences Bigdata. Does it use Keep-Alive and does >this mean that the write transaction isn't actually committed until >HTTPClient chooses to evict a connection from the pool. > >BTW, we're using a .war from rev 6717 deployed in Tomcat. > >-- >Antoni Myłka >Software Engineer > >basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 >http://www.basis06.ch - source of smart business > >-------------------------------------------------------------------------- >---- >Free Next-Gen Firewall Hardware Offer >Buy your Sophos next-gen firewall before the end March 2013 >and get the hardware for free! Learn more. >http://p.sf.net/sfu/sophos-d2d-feb >_______________________________________________ >Bigdata-developers mailing list >Big...@li... >https://lists.sourceforge.net/lists/listinfo/bigdata-developers |
|
From: Antoni M. <ant...@ba...> - 2013-02-07 09:42:02
|
Hi, Is inference done asynchronously in Bigdata? In my integration tests I see spurious failures: 1. I run code that does SPARQL UPDATE queries. It uses SPARQLRepository or direct, handcrafted invocations of the REST API. The production code uses HTTP Client connection pooling and it's difficult to say how many physical connections it uses and when they stop. 2. When that code finishes, I create a new SPARQLRepository and use it to see if content of the repository meets my expectations. 3. The test code sometimes succeeds, sometimes fails. The NanoSparqlServer wikipage states that "Mutation operations are ACID against a standalone database and shard-wise ACID against a bigdata federation." Is inference done within the write transaction, or does the ACID only refer to explicit triples? Obviously there may be 1001 other reasons for the test failures I'm seeing. If you clearly confirm that when Bigdata says it's done - it means it's done I can look further. Another idea for investigation would be to check how HTTPClient connection pooling influences Bigdata. Does it use Keep-Alive and does this mean that the write transaction isn't actually committed until HTTPClient chooses to evict a connection from the pool. BTW, we're using a .war from rev 6717 deployed in Tomcat. -- Antoni Myłka Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business |
|
From: Bryan T. <br...@sy...> - 2013-01-30 12:01:33
|
I've committed a fix for this. r6861. Thanks, Bryan On 1/28/13 1:10 PM, "Antoni Mylka" <am...@ba...> wrote: >Hi Bryan, > >The Exception was thrown out of a debug statement. We've packaged bigdata >in an OSGI bundle, together with its own log4j.jar. It didn't pick up >changes in the logging configuration of our system. It took me a while to >figure that out :). I've decreased the logging level to WARN and the >exceptions disappeared. Thanks for the tips. > >Regards, > >-- >Antoni Myłka >Software Engineer > >basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 >http://www.basis06.ch - source of smart business > >----- Ursprüngliche Mail ----- >Von: "Bryan Thompson" <br...@sy...> >An: "Antoni Mylka" <am...@ba...>, >big...@li... >Gesendet: Montag, 28. Januar 2013 18:39:20 >Betreff: Re: [Bigdata-developers] Debug log output > > > >It looks like it. The temp triple store generally uses the lexicon of the >primary relation. Code can not assume that the lexicon is available in >this context. > >If I understand it, the exception is thrown out of a debug statement, >correct? > >Thanks, >Bryan |
|
From: Antoni M. <am...@ba...> - 2013-01-28 18:10:46
|
Hi Bryan, The Exception was thrown out of a debug statement. We've packaged bigdata in an OSGI bundle, together with its own log4j.jar. It didn't pick up changes in the logging configuration of our system. It took me a while to figure that out :). I've decreased the logging level to WARN and the exceptions disappeared. Thanks for the tips. Regards, -- Antoni Myłka Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business ----- Ursprüngliche Mail ----- Von: "Bryan Thompson" <br...@sy...> An: "Antoni Mylka" <am...@ba...>, big...@li... Gesendet: Montag, 28. Januar 2013 18:39:20 Betreff: Re: [Bigdata-developers] Debug log output It looks like it. The temp triple store generally uses the lexicon of the primary relation. Code can not assume that the lexicon is available in this context. If I understand it, the exception is thrown out of a debug statement, correct? Thanks, Bryan |
|
From: Bryan T. <br...@sy...> - 2013-01-28 17:58:59
|
It looks like it. The temp triple store generally uses the lexicon of the
primary relation. Code can not assume that the lexicon is available in
this context.
If I understand it, the exception is thrown out of a debug statement,
correct?
Thanks,
Bryan
On 1/28/13 12:23 PM, "Antoni Mylka" <am...@ba...> wrote:
>Hi,
>
>I've been debugging an issue we're seeing. After a recent refactoring one
>of our tests started to fail consistently. Our app makes a SPARQL UPDATE
>query. The query is perfectly normal, it just adds 9 triples.
>Unfortunately it fails because bigdata throws an exception. It's a chain
>of 5 exceptions but the root cause is this:
>
>java.lang.NullPointerException
> at
>com.bigdata.rdf.store.AbstractTripleStore.getTerm(AbstractTripleStore.java
>:2320)
> at
>com.bigdata.rdf.store.AbstractTripleStore.toString(AbstractTripleStore.jav
>a:3227)
> at
>com.bigdata.rdf.store.AbstractTripleStore.toString(AbstractTripleStore.jav
>a:3204)
> at com.bigdata.rdf.spo.SPO.toString(SPO.java:587)
> at com.bigdata.rdf.inf.AbstractSPOBuffer.add(AbstractSPOBuffer.java:222)
> at
>com.bigdata.rdf.inf.TruthMaintenance.applyExistingStatements(TruthMaintena
>nce.java:352)
> at
>com.bigdata.rdf.inf.TruthMaintenance.assertAll(TruthMaintenance.java:477)
> at
>com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.flushStatementBuffe
>rs(BigdataSail.java:3125)
> at
>com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.commit2(BigdataSail
>.java:2935)
> at
>com.bigdata.rdf.sail.BigdataSailRepositoryConnection.commit2(BigdataSailRe
>positoryConnection.java:272)
> at
>com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.convertCommit(AST2BOpUpdate.
>java:353)
> at
>com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.convertUpdate(AST2BOpUpdate.
>java:307)
> at
>com.bigdata.rdf.sparql.ast.eval.ASTEvalHelper.executeUpdate(ASTEvalHelper.
>java:1037)
> at
>com.bigdata.rdf.sail.BigdataSailUpdate.execute2(BigdataSailUpdate.java:152
>)
> at
>com.bigdata.rdf.sail.webapp.BigdataRDFContext$UpdateTask.doQuery(BigdataRD
>FContext.java:1256)
> at
>com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(Bigda
>taRDFContext.java:928)
> at
>com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(Bigda
>taRDFContext.java:405)
>
>I tried to debug this and AFAICS it shows an issue. AbstractSPOBuffer has
>a block of code with quite an interesting code:
>
>if (DEBUG) {
> /*
> * Note: If [store] is a TempTripleStore then this will NOT be able
> * to resolve the terms from the ids (since the lexicon is only in
> * the database).
> */
> log.debug(stmt.toString(_db));
>}
>
>Isn't this an explanation of the NPE I'm seeing? The toString method
>fails because the AbstractTripleStore.getTerm tries to call return
>getLexiconRelation().getTerm(iv), but the lexicon relation is null. If
>this is the case, then perhaps there should be some 'ifs' around those
>lines.
>
>--
>Antoni Myłka
>Software Engineer
>
>basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22
>http://www.basis06.ch - source of smart business
>
>--------------------------------------------------------------------------
>----
>Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
>MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
>with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
>MVPs and experts. ON SALE this month only -- learn more at:
>http://p.sf.net/sfu/learnnow-d2d
>_______________________________________________
>Bigdata-developers mailing list
>Big...@li...
>https://lists.sourceforge.net/lists/listinfo/bigdata-developers
|
|
From: Antoni M. <am...@ba...> - 2013-01-28 17:23:16
|
Hi,
I've been debugging an issue we're seeing. After a recent refactoring one of our tests started to fail consistently. Our app makes a SPARQL UPDATE query. The query is perfectly normal, it just adds 9 triples. Unfortunately it fails because bigdata throws an exception. It's a chain of 5 exceptions but the root cause is this:
java.lang.NullPointerException
at com.bigdata.rdf.store.AbstractTripleStore.getTerm(AbstractTripleStore.java:2320)
at com.bigdata.rdf.store.AbstractTripleStore.toString(AbstractTripleStore.java:3227)
at com.bigdata.rdf.store.AbstractTripleStore.toString(AbstractTripleStore.java:3204)
at com.bigdata.rdf.spo.SPO.toString(SPO.java:587)
at com.bigdata.rdf.inf.AbstractSPOBuffer.add(AbstractSPOBuffer.java:222)
at com.bigdata.rdf.inf.TruthMaintenance.applyExistingStatements(TruthMaintenance.java:352)
at com.bigdata.rdf.inf.TruthMaintenance.assertAll(TruthMaintenance.java:477)
at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.flushStatementBuffers(BigdataSail.java:3125)
at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.commit2(BigdataSail.java:2935)
at com.bigdata.rdf.sail.BigdataSailRepositoryConnection.commit2(BigdataSailRepositoryConnection.java:272)
at com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.convertCommit(AST2BOpUpdate.java:353)
at com.bigdata.rdf.sparql.ast.eval.AST2BOpUpdate.convertUpdate(AST2BOpUpdate.java:307)
at com.bigdata.rdf.sparql.ast.eval.ASTEvalHelper.executeUpdate(ASTEvalHelper.java:1037)
at com.bigdata.rdf.sail.BigdataSailUpdate.execute2(BigdataSailUpdate.java:152)
at com.bigdata.rdf.sail.webapp.BigdataRDFContext$UpdateTask.doQuery(BigdataRDFContext.java:1256)
at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:928)
at com.bigdata.rdf.sail.webapp.BigdataRDFContext$AbstractQueryTask.call(BigdataRDFContext.java:405)
I tried to debug this and AFAICS it shows an issue. AbstractSPOBuffer has a block of code with quite an interesting code:
if (DEBUG) {
/*
* Note: If [store] is a TempTripleStore then this will NOT be able
* to resolve the terms from the ids (since the lexicon is only in
* the database).
*/
log.debug(stmt.toString(_db));
}
Isn't this an explanation of the NPE I'm seeing? The toString method fails because the AbstractTripleStore.getTerm tries to call return getLexiconRelation().getTerm(iv), but the lexicon relation is null. If this is the case, then perhaps there should be some 'ifs' around those lines.
--
Antoni Myłka
Software Engineer
basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22
http://www.basis06.ch - source of smart business
|
|
From: Antoni M. <am...@ba...> - 2013-01-28 16:41:39
|
Hi, I've been debugging an issue we're seeing. After a recent refactoring one of our tests started to fail consistently. Our app makes a SPARQL UPDATE query. The query is perfectly normal, it just adds 9 triples. Unfortunately it fails because bigdata throws an exception. It's a chain of 5 exceptions but the root cause is this: -- Antoni Myłka Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business |
|
From: Bryan T. <br...@sy...> - 2012-12-14 12:59:11
|
Doug, Is your intention to expose the SPARQL 1.1 Query interface? If so, you can just deny any HTTP method other than GET. You can also restrict the URIs that are visible to only include the …//sparql path for the external interface of your host. That will make things like the performance counters and the status pages unavailable. Many times people narrow the interface further through an application layer rather than exposing the full SPARQL query language. There is an open thread on the discussion forum [1] for imposing a timeout on all external interface requests. That thread suggests how this can be done by modifying the code and some possible approaches to automating this policy. Thanks, Bryan [1] https://sourceforge.net/projects/bigdata/forums/forum/676946/topic/6133565 From: Doug Whitehead <dou...@gm...<mailto:dou...@gm...>> Date: Wednesday, December 12, 2012 10:20 PM To: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: [Bigdata-developers] Lock Down a BigData server I like and use the SPARQL 1.1 interface to load and delete data. However, when I take this puppy to production, I don't want others to be able to modify my endpoint. Sorry for the n00b question, but... After I have loaded data, how do I lock it down? Thanks, Doug |
|
From: Doug W. <dou...@gm...> - 2012-12-13 03:20:40
|
I like and use the SPARQL 1.1 interface to load and delete data. However, when I take this puppy to production, I don't want others to be able to modify my endpoint. Sorry for the n00b question, but... After I have loaded data, how do I lock it down? Thanks, Doug |
|
From: Bryan T. <br...@sy...> - 2012-11-28 12:33:49
|
Looks like one of the services is down:
Caused by: java.lang.RuntimeException: DataService not found: { partitionId=0, dataServiceUUID=ff5878a8-9d83-418c-8ca8-4926807a8415, leftSeparator=[], rightSeparator=null}
If you have restarted the cluster, you need to make sure that the old service registrations are gone. This happens automatically with a time against jini/river.
Otherwise, check your servers and look for a dead DataServer process or a dead server.
Bryan
From: WanCai Wen <wen...@gm...<mailto:wen...@gm...>>
Date: Tuesday, November 27, 2012 8:50 PM
To: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>>
Cc: Bryan Thompson <br...@sy...<mailto:br...@sy...>>
Subject: Error of NanoSparqlServer in Bigdata Cluster
Caused by: java.lang.RuntimeException: DataService not found: { partitionId=0, dataServiceUUID=ff5878a8-9d83-418c-8ca8-4926807a8415, leftSeparator=[], rightSeparator=null}
|
|
From: WanCai W. <wen...@gm...> - 2012-11-28 01:50:13
|
how to slove the following error?
SELECT * { ?s ?p ?o } LIMIT 1
java.lang.RuntimeException: com.bigdata.service.ndx.ClientException:
Execution failed: index=rdf_test.lex.TERM2ID, ts=1354065608409,
procedure com.bigdata.rdf.lexicon.Term2IdWriteProc : Split{ ntuples=1,
fromIndex=0, toIndex=1, pmd={ partitionId=0,
dataServiceUUID=ff5878a8-9d83-418c-8ca8-4926807a8415,
leftSeparator=[], rightSeparator=null}}
at com.bigdata.rdf.lexicon.LexiconRelation.addTerms(LexiconRelation.java:1844)
at com.bigdata.rdf.lexicon.LexiconRelation.addTerms(LexiconRelation.java:1705)
at com.bigdata.rdf.sail.sparql.BatchRDFValueResolver.process(BatchRDFValueResolver.java:197)
at com.bigdata.rdf.sail.sparql.Bigdata2ASTSPARQLParser.parseQuery2(Bigdata2ASTSPARQLParser.java:323)
at com.bigdata.rdf.sail.webapp.BigdataRDFContext.getQueryTask(BigdataRDFContext.java:1150)
at com.bigdata.rdf.sail.webapp.QueryServlet.doQuery(QueryServlet.java:479)
at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:120)
at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:133)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:534)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:475)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:929)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:864)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114)
at org.eclipse.jetty.server.Server.handle(Server.java:348)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1068)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218)
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:508)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.bigdata.service.ndx.ClientException: Execution failed:
index=rdf_test.lex.TERM2ID, ts=1354065608409, procedure
com.bigdata.rdf.lexicon.Term2IdWriteProc : Split{ ntuples=1,
fromIndex=0, toIndex=1, pmd={ partitionId=0,
dataServiceUUID=ff5878a8-9d83-418c-8ca8-4926807a8415,
leftSeparator=[], rightSeparator=null}}
at com.bigdata.service.ndx.ClientIndexView.runOne(ClientIndexView.java:1699)
at com.bigdata.service.ndx.ClientIndexView.runTasks(ClientIndexView.java:1648)
at com.bigdata.service.ndx.ClientIndexView.submit(ClientIndexView.java:1602)
at com.bigdata.service.ndx.ClientIndexView.submit(ClientIndexView.java:1514)
at com.bigdata.rdf.lexicon.Term2IdWriteTask.call(Term2IdWriteTask.java:216)
at com.bigdata.rdf.lexicon.LexiconRelation.addTerms(LexiconRelation.java:1840)
... 28 more
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: DataService not found: { partitionId=0,
dataServiceUUID=ff5878a8-9d83-418c-8ca8-4926807a8415,
leftSeparator=[], rightSeparator=null}
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262)
at java.util.concurrent.FutureTask.get(FutureTask.java:119)
at com.bigdata.service.ndx.ClientIndexView.runOne(ClientIndexView.java:1692)
... 33 more
Caused by: java.lang.RuntimeException: DataService not found: {
partitionId=0, dataServiceUUID=ff5878a8-9d83-418c-8ca8-4926807a8415,
leftSeparator=[], rightSeparator=null}
at com.bigdata.service.ndx.AbstractDataServiceProcedureTask.submit(AbstractDataServiceProcedureTask.java:279)
at com.bigdata.service.ndx.AbstractDataServiceProcedureTask.call(AbstractDataServiceProcedureTask.java:215)
at com.bigdata.service.ndx.AbstractDataServiceProcedureTask.call(AbstractDataServiceProcedureTask.java:45)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
... 1 more
|
|
From: Bryan T. <br...@sy...> - 2012-11-26 18:25:32
|
This is documented on the wiki: https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=SPARQL_Benchmarks Bryan From: WanCai Wen <wen...@gm...<mailto:wen...@gm...>> Date: Sunday, November 25, 2012 11:55 PM To: "big...@li...<mailto:big...@li...>" <big...@li...<mailto:big...@li...>> Subject: [Bigdata-developers] LUBM in cluster hi, I hava installed the cluster.I want to do LUBM on the cluster,Could you tell me how to do LUBM in the cluster ? |
|
From: WanCai W. <wen...@gm...> - 2012-11-26 04:56:04
|
hi, I hava installed the cluster.I want to do LUBM on the cluster,Could you tell me how to do LUBM in the cluster ? |
|
From: WanCai W. <wen...@gm...> - 2012-11-21 08:36:44
|
hi,
i downloaded zookeeper-3.3.3.tar.gz file to disk and untar it.
in build.properties file ,set the following properties .
when i change status to start in /nas/bigdata/benchmark/state file
i get the following output:
*Waiting 5000ms for service discovery.*
*Zookeeper is not running.*
*Discovered 0 jini service registrars.*
*Discovered 0 services*
*Discovered 0 stale bigdata services.*
*Discovered 0 live bigdata services.*
*Discovered 0 other services.*
*Bigdata services by serviceIface:*
*Bigdata services by hostname:*
how to set zookeeper properties and how to install it ,please?
|
27 messages has been excluded from this view by a project administrator.
