ng4j-namedgraphs — Discussion of Named Graphs in RDF

Re: [namedgraphs] Memory leak in database persistence implementation

[Jennifer C. <jen...@at...>]

What I've tried locally is to use a com.sun.rowset.CachedRowSet to capture
the contents of the ResultSet so that the ResultSet can be cleaned up.
This appears to be slightly reducing (i.e. hurting) performance in the
tests that I have run.  In theory I think it eliminates that particular
memory leak, but I don't have a good set of test data to confirm that.
(Anyone?)

 

I could provide the file to anyone who wants to try it (or attach it to a
bug report, or even commit it if it sounds reasonable).  A short
description of the change is as follows:

 

In QuadDB methods find and listGraphNames, 

.        final ResultSet results = executeQuery(sql); // change this to
not be final (not necessary but helps to find any errors later on)

.        Add this code following that line:
      // Use a CachedRowSet so we can clean-up the ResultSet

            // http://onjava.com/pub/a/onjava/2004/06/23/cachedrowset.html

            final CachedRowSetImpl crs;

            try {

                  crs = new CachedRowSetImpl();

                  crs.populate(results);

            } catch (SQLException e) {

                  throw new JenaException(e);

            } finally {

                  cleanUp(results);

         }

.        In the newly constructed Iterator for return, comment out all
calls to cleanup(results).  (The ResultSet is cleaned up in the finally
above, and there shouldn't be a need to clean-up the CachedRowSet.)  Also
can comment out the "if (!this.hasNext)" statement that only contains a
call to cleanUp.

.        In the newly constructed Iterator for return, replace all
occurrences of "results" with "crs".  (Each one should be showing an error
(if your IDE does that automatically like Eclipse) since results is no
longer final.)

 

One of my concerns is that this modification relies on a com.sun package.
I'd like to find a better solution .

 

Comments and corrections welcome!

 

Jennifer

 

 

From: Graeme Stevenson [mailto:ac...@gm...] 
Sent: Thursday, July 15, 2010 8:20 AM
To: ng4j-namedgraphs
Subject: [namedgraphs] Memory leak in database persistence implementation

 

Hi guys,

Firstly, thanks for all your hard work on the NG4J library. I've found it
extremely useful over the past couple of years. 

I've found a memory leak in the database persistence implementation in
version 0.9.3 caused by ResultSet objects not being closed when a
(positive) check is made for the existence of a Quad. The following
conditions reproduce the bug:

- A call is made to QuadDB.find(..) to check the existence of a Quad
- The hasNext() call on the returned iterator returns true
- No further methods are invoked on the iterator

The implementation of the QuadDB.find(..) currently only closes the
ResultSet object when hasNext() returns false, or an exception occurs.

There are two methods that call QuadDB.find(..) where this will always
occur when a match is found:
QuadDB.insert(...)
NamedGraphSetDB.containsQuads(...)

And this could potentially occur in the following methods if the iterator
is not completely traversed:
NamedGraphSetDB.findQuads(...)
NamedGraphDB.graphBaseFind(...)

Hope this is of use to you,

Cheers,
Graeme.

Re: [namedgraphs] Memory leak in database persistence implementation

[Jennifer C. <jen...@at...>]

I clearly spoke too quickly, that's not a good solution.

 

I'll look into using com.hp.hpl.jena.query.ResultSetFactory.copyResults so
we can clean-up the original ResultSet and be done with it.

 

Jennifer

 

 

From: Jennifer Cormier [mailto:jen...@at...] 
Sent: Thursday, August 26, 2010 9:30 AM
To: 'Graeme Stevenson'; 'ng4j-namedgraphs'
Subject: Re: [namedgraphs] Memory leak in database persistence
implementation

 

Good catch, Graeme!

 

Has anyone looked into this?

 

I think we'll want to have the find method return some sort of closeable
iterator, such as com.hp.hpl.jena.util.iterator.ClosableIterator, instead
of java.util.Iterator, and then the caller should call close if the
iterator is not fully traversed.  Similarly for method listGraphNames in
the same class (QuadDB).

 

I may try that and see .

 

Jennifer

 

 

From: Graeme Stevenson [mailto:ac...@gm...] 
Sent: Thursday, July 15, 2010 8:20 AM
To: ng4j-namedgraphs
Subject: [namedgraphs] Memory leak in database persistence implementation

 

Hi guys,

Firstly, thanks for all your hard work on the NG4J library. I've found it
extremely useful over the past couple of years. 

I've found a memory leak in the database persistence implementation in
version 0.9.3 caused by ResultSet objects not being closed when a
(positive) check is made for the existence of a Quad. The following
conditions reproduce the bug:

- A call is made to QuadDB.find(..) to check the existence of a Quad
- The hasNext() call on the returned iterator returns true
- No further methods are invoked on the iterator

The implementation of the QuadDB.find(..) currently only closes the
ResultSet object when hasNext() returns false, or an exception occurs.

There are two methods that call QuadDB.find(..) where this will always
occur when a match is found:
QuadDB.insert(...)
NamedGraphSetDB.containsQuads(...)

And this could potentially occur in the following methods if the iterator
is not completely traversed:
NamedGraphSetDB.findQuads(...)
NamedGraphDB.graphBaseFind(...)

Hope this is of use to you,

Cheers,
Graeme.

Re: [namedgraphs] Memory leak in database persistence implementation

[Jennifer C. <jen...@at...>]

Good catch, Graeme!

 

Has anyone looked into this?

 

I think we'll want to have the find method return some sort of closeable
iterator, such as com.hp.hpl.jena.util.iterator.ClosableIterator, instead
of java.util.Iterator, and then the caller should call close if the
iterator is not fully traversed.  Similarly for method listGraphNames in
the same class (QuadDB).

 

I may try that and see .

 

Jennifer

 

 

From: Graeme Stevenson [mailto:ac...@gm...] 
Sent: Thursday, July 15, 2010 8:20 AM
To: ng4j-namedgraphs
Subject: [namedgraphs] Memory leak in database persistence implementation

 

Hi guys,

Firstly, thanks for all your hard work on the NG4J library. I've found it
extremely useful over the past couple of years. 

I've found a memory leak in the database persistence implementation in
version 0.9.3 caused by ResultSet objects not being closed when a
(positive) check is made for the existence of a Quad. The following
conditions reproduce the bug:

- A call is made to QuadDB.find(..) to check the existence of a Quad
- The hasNext() call on the returned iterator returns true
- No further methods are invoked on the iterator

The implementation of the QuadDB.find(..) currently only closes the
ResultSet object when hasNext() returns false, or an exception occurs.

There are two methods that call QuadDB.find(..) where this will always
occur when a match is found:
QuadDB.insert(...)
NamedGraphSetDB.containsQuads(...)

And this could potentially occur in the following methods if the iterator
is not completely traversed:
NamedGraphSetDB.findQuads(...)
NamedGraphDB.graphBaseFind(...)

Hope this is of use to you,

Cheers,
Graeme.

[namedgraphs] Memory leak in database persistence implementation

[Graeme S. <ac...@gm...>]

Hi guys,

Firstly, thanks for all your hard work on the NG4J library. I've found it
extremely useful over the past couple of years.

I've found a memory leak in the database persistence implementation in
version 0.9.3 caused by ResultSet objects not being closed when a (positive)
check is made for the existence of a Quad. The following conditions
reproduce the bug:

- A call is made to QuadDB.find(..) to check the existence of a Quad
- The hasNext() call on the returned iterator returns true
- No further methods are invoked on the iterator

The implementation of the QuadDB.find(..) currently only closes the
ResultSet object when hasNext() returns false, or an exception occurs.

There are two methods that call QuadDB.find(..) where this will always occur
when a match is found:
QuadDB.insert(...)
NamedGraphSetDB.containsQuads(...)

And this could potentially occur in the following methods if the iterator is
not completely traversed:
NamedGraphSetDB.findQuads(...)
NamedGraphDB.graphBaseFind(...)

Hope this is of use to you,

Cheers,
Graeme.

Re: [namedgraphs] SPARQL over named graphs with blank nodes

[Olaf H. <ha...@in...>]

Hey,

On Friday 25 June 2010 16:07:59 Blom, J.J.C. (Jaap) wrote:
> [...]
> > I don't see a problem here. If you import such data into a Jena Model (or
> > Graph) then Jena automatically creates unique blank node identifiers.
> 
> </snip>
> 
> Well actually I import it in TriX for transporting and I checked, that
>  means the blank node labels aren't renamed. So that means I would need to
>  loop through all the named graphs and do it. Not something I would want to
>  implement for such a thing tbh.

You mean you import a TriX document in NG4J, right?
Unlike the other parsers (RDF/XML, Turtle, etc.) that are implemented in Jena, 
the TriX parser is part of NG4J, I think. That might be the reason why data 
imported from TriX documents has the same blank node labels internally than
in the document.

Greetings,
Olaf

Re: [namedgraphs] SPARQL over named graphs with blank nodes

[Jennifer C. <jen...@at...>]

This probably is not related to the issues you are seeing, but it might be
worth noting.

Previously I had some problems in a particular use case including NG4J.
Eventually tracked them down to the NG4J-provided parser for TriX.  It
appears that TriX is not fully supported by the parser.  The problems were
completely resolved by a single change - using TriG instead of TriX.  

This was not a matter of incorrectly written TriX on our system's part.
Rather, we were beginning with a named graph model, writing as TriX and
later reading in the TriX - all done using NG4J.  

Jennifer
 

-----Original Message-----
From: Blom, J.J.C. (Jaap) [mailto:jb...@cr...] 
Sent: Friday, June 25, 2010 10:08 AM
To: ng4...@li...
Subject: Re: [namedgraphs] SPARQL over named graphs with blank nodes

Hi again,

Sorry forgot about this for a bit.

<snip>
> But only for debugging purposes. Right? I mean, this data is meant to be
processed automatically and an application doesn't care about labels.
</snip>

Well if that means that running in debugging mode actually enables bugs,
that may not be the best approach. Since I had the choice of keeping the
TRIX a bit readable or putting in labels that seemed to cause errors (not
sure why and the details have since escaped me). So now I declare logical
named blank nodes everywhere (assuming ARQ would hold itself to the
principles of RDF). So I may want to turn back on that decision.

<snip>
> I don't see a problem here. If you import such data into a Jena Model
(or
> Graph) then Jena automatically creates unique blank node identifiers.
</snip>

Well actually I import it in TriX for transporting and I checked, that
means the blank node labels aren't renamed. So that means I would need to
loop through all the named graphs and do it. Not something I would want to
implement for such a thing tbh.

<snip>
> Yes, I think it is a problem in Jena. NG4J uses the Jena implementation
of triple pattern matching over a union of RDF graphs which is implemented
in > > the 'MultiUnion' class (package 'com.hp.hpl.jena.graph.compose');
more precisely, the method 'multiGraphFind'
</snip>

Kay, thanks.

Greetings,
  Jaap Blom

http://www.croon.nl/disclaimer
http://www.croon.nl/inkoopvoorwaarden
Kamer van Koophandel Rotterdam nr. / Chamber of Commerce Rotterdam no.:
24002312

--------------------------------------------------------------------------
----
ThinkGeek and WIRED's GeekDad team up for the Ultimate 
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the 
lucky parental unit.  See the prize list and enter to win: 
http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________
ng4j-namedgraphs mailing list
ng4...@li...
https://lists.sourceforge.net/lists/listinfo/ng4j-namedgraphs

Re: [namedgraphs] SPARQL over named graphs with blank nodes

[Blom, J.J.C. (Jaap) <jb...@cr...>]

Hi again,

Sorry forgot about this for a bit.

<snip>
> But only for debugging purposes. Right? I mean, this data is meant to be processed automatically and an application doesn't care about labels.
</snip>

Well if that means that running in debugging mode actually enables bugs, that may not be the best approach. Since I had the choice of keeping the TRIX a bit readable or putting in labels that seemed to cause errors (not sure why and the details have since escaped me). So now I declare logical named blank nodes everywhere (assuming ARQ would hold itself to the principles of RDF). So I may want to turn back on that decision.

<snip>
> I don't see a problem here. If you import such data into a Jena Model (or
> Graph) then Jena automatically creates unique blank node identifiers.
</snip>

Well actually I import it in TriX for transporting and I checked, that means the blank node labels aren't renamed. So that means I would need to loop through all the named graphs and do it. Not something I would want to implement for such a thing tbh.

<snip>
> Yes, I think it is a problem in Jena. NG4J uses the Jena implementation of triple pattern matching over a union of RDF graphs which is implemented in > > the 'MultiUnion' class (package 'com.hp.hpl.jena.graph.compose'); more precisely, the method 'multiGraphFind'
</snip>

Kay, thanks.

Greetings,
  Jaap Blom

http://www.croon.nl/disclaimer
http://www.croon.nl/inkoopvoorwaarden
Kamer van Koophandel Rotterdam nr. / Chamber of Commerce Rotterdam no.: 24002312

Re: [namedgraphs] SPARQL over named graphs with blank nodes

[Olaf H. <ha...@in...>]

Hey,

On Sunday 20 June 2010 08:56:42 Blom, J.J.C. (Jaap) wrote:
> Hi,
> 
> Wel for a number of reasons:
> A) It reads a little easier to track a blank node with a readable label in
>  respect to a UID.

But only for debugging purposes. Right? I mean, this data is meant to be 
processed automatically and an application doesn't care about labels.

>  B) Data received from sources not using NG4J/Jena (and
>  thus not supporting this "rule" of uniquely naming blank nodes)

I don't see a problem here. If you import such data into a Jena Model (or 
Graph) then Jena automatically creates unique blank node identifiers.

>  C) Because
>  I should be able to do it. The RDF defenitions specify that two blank
>  nodes with the same label in different graphs should not be considered the
>  same node on that fact alone. (Sorry if I seem to be ranting here)

Okay.

> Like I said I doubt this is a bug from NG4J (I think it is Jena in this
>  case), but since there are some elements from NG4J involved I need to be
>  sure of this.

Yes, I think it is a problem in Jena. NG4J uses the Jena implementation of 
triple pattern matching over a union of RDF graphs which is implemented
in the 'MultiUnion' class (package 'com.hp.hpl.jena.graph.compose'); more 
precisely, the method 'multiGraphFind'

Greetings,
Olaf

Re: [namedgraphs] SPARQL over named graphs with blank nodes

[Blom, J.J.C. (Jaap) <jb...@cr...>]

Hi,

Wel for a number of reasons:
A) It reads a little easier to track a blank node with a readable label in respect to a UID.
B) Data received from sources not using NG4J/Jena (and thus not supporting this "rule" of uniquely naming blank nodes)
C) Because I should be able to do it. The RDF defenitions specify that two blank nodes with the same label in different graphs should not be considered the same node on that fact alone. (Sorry if I seem to be ranting here)

Like I said I doubt this is a bug from NG4J (I think it is Jena in this case), but since there are some elements from NG4J involved I need to be sure of this.

Thanks,
  Jaap Blom

-----Original Message-----
From: Olaf Hartig [mailto:ha...@in...]
Sent: zaterdag 19 juni 2010 9:24
To: Blom, J.J.C. (Jaap)
Cc: ng4...@li...
Subject: Re: [namedgraphs] SPARQL over named graphs with blank nodes

Hey,

On Friday 18 June 2010 17:03:54 Blom, J.J.C. (Jaap) wrote:
> Hi,
>
> I'm working on NG4J 0.9.3 and have the following code:
>                 Node graph1 = Node.createURI("http://graph1");
>                 Node graph2 = Node.createURI("http://graph2");
>                 Node start = Node.createURI("http://start");
>                 Node predicate1 = Node.createURI("http://predicate1");
>                 Node predicate2 = Node.createURI("http://predicate2");
>                 Node result1 = Node.createLiteral("expected result");
>                 Node result2 = Node.createLiteral("not expected result");
>                 NamedGraphSet ngs = new NamedGraphSetImpl();
>                 ngs.addQuad(new Quad(graph1, start, predicate1,
> Node.createAnon(new AnonId("blank")))); ngs.addQuad(new Quad(graph1,
> Node.createAnon(new AnonId("blank")), predicate2, result1));
> ngs.addQuad(new Quad(graph2, Node.createAnon(new AnonId("blank")),
> predicate2, result2));//*/ String query =
>                         "Select ?target Where\n" +
>                         "{\n" +
>                         "       <" + start.getURI() + "> <" +
>  predicate1.getURI() + "> ?b .\n" + "       ?b <" + predicate2.getURI() +
>  "> ?target\n" + "}";
>                 System.out.println(query);
>                 Query sparqlQuery = QueryFactory.create(query);
>                 QueryExecution qe =
>  QueryExecutionFactory.create(sparqlQuery, new
> NamedGraphDataset(ngs));  ResultSet results = qe.execSelect();
>                 while(results.hasNext())
>                 {
>                         System.out.println(results.next());
>                 }
>                 System.out.println("done");
>
> This gives the following result:
> Select $target Where
> {
>         <http://start> <http://predicate1> ?b .
>         ?b <http://predicate2> $target } ( ?target = "expected result"
> ) ( ?target = "not expected result" ) done
>
> I didn't expect the "not expected result", since it is linked to a
> blank  node in another graph. Granted the blank nodes have the same
> label, but  RDF states that blank nodes in different graphs are not
> linked by their  label. Am I correct in this one and if so is this a
> NG4J error or a  Jena/ARQ error?

I agree that you would expect only the "expected result" and that would be the correct behavior. The issue is that none ever expected to have two blank nodes with the same label in different graphs. Usually, this would never happen because Jena creates new blank node with unique labels whenever it reads an RDF document. Programmatically, you would typically create a Node object that represents your blank node and use this Node object instead of creating new Node objects with the same label. Hence, to me it seems that your example code uses the Jena API in a way it was not intended to be used.
Why do you want to do it the way your example code outlines?

Greetings,
Olaf

PS: I cannot answer the other questions you sent to the list because I'm not familiar with that part of NG4J.



http://www.croon.nl/disclaimer
http://www.croon.nl/inkoopvoorwaarden
Kamer van Koophandel Rotterdam nr. / Chamber of Commerce Rotterdam no.: 24002312

Re: [namedgraphs] SPARQL over named graphs with blank nodes

[Olaf H. <ha...@in...>]

Hey,

On Friday 18 June 2010 17:03:54 Blom, J.J.C. (Jaap) wrote:
> Hi,
> 
> I'm working on NG4J 0.9.3 and have the following code:
>                 Node graph1 = Node.createURI("http://graph1");
>                 Node graph2 = Node.createURI("http://graph2");
>                 Node start = Node.createURI("http://start");
>                 Node predicate1 = Node.createURI("http://predicate1");
>                 Node predicate2 = Node.createURI("http://predicate2");
>                 Node result1 = Node.createLiteral("expected result");
>                 Node result2 = Node.createLiteral("not expected result");
>                 NamedGraphSet ngs = new NamedGraphSetImpl();
>                 ngs.addQuad(new Quad(graph1, start, predicate1,
>  Node.createAnon(new AnonId("blank")))); ngs.addQuad(new Quad(graph1,
>  Node.createAnon(new AnonId("blank")), predicate2, result1));
>  ngs.addQuad(new Quad(graph2, Node.createAnon(new AnonId("blank")),
>  predicate2, result2));//*/ String query =
>                         "Select ?target Where\n" +
>                         "{\n" +
>                         "       <" + start.getURI() + "> <" +
>  predicate1.getURI() + "> ?b .\n" + "       ?b <" + predicate2.getURI() +
>  "> ?target\n" + "}";
>                 System.out.println(query);
>                 Query sparqlQuery = QueryFactory.create(query);
>                 QueryExecution qe =
>  QueryExecutionFactory.create(sparqlQuery, new NamedGraphDataset(ngs));
>  ResultSet results = qe.execSelect();
>                 while(results.hasNext())
>                 {
>                         System.out.println(results.next());
>                 }
>                 System.out.println("done");
> 
> This gives the following result:
> Select $target Where
> {
>         <http://start> <http://predicate1> ?b .
>         ?b <http://predicate2> $target
> }
> ( ?target = "expected result" )
> ( ?target = "not expected result" )
> done
> 
> I didn't expect the "not expected result", since it is linked to a blank
>  node in another graph. Granted the blank nodes have the same label, but
>  RDF states that blank nodes in different graphs are not linked by their
>  label. Am I correct in this one and if so is this a NG4J error or a
>  Jena/ARQ error?

I agree that you would expect only the "expected result" and that would be the 
correct behavior. The issue is that none ever expected to have two blank nodes 
with the same label in different graphs. Usually, this would never happen 
because Jena creates new blank node with unique labels whenever it reads an 
RDF document. Programmatically, you would typically create a Node object that 
represents your blank node and use this Node object instead of creating new 
Node objects with the same label. Hence, to me it seems that your example code 
uses the Jena API in a way it was not intended to be used.
Why do you want to do it the way your example code outlines?

Greetings,
Olaf

PS: I cannot answer the other questions you sent to the list because I'm not 
familiar with that part of NG4J.

[namedgraphs] Canonical graphs

[Blom, J.J.C. (Jaap) <jb...@cr...>]

Hi,

During my work to figure out how to create canonicalized graphs I came across the following. I have a named graph (http://127.0.0.1/Test1) with the following triples:
http://example.com/graph4 http://example.com/graph5 http://example.com/graph6
http://example.com/graph1 http://example.com/graph2 http://example.com/graph3
_:uuerwl http://example.com/graph2 http://example.com/graph3

After I got out the canonicalized graph it looked like this:
[http://127.0.0.1/Test1, http://example.com/graph1 http://example.com/graph2 _:g000001, http://example.com/graph1 http://example.com/graph2 http://example.com/graph3, http://example.com/graph4 http://example.com/graph5 http://example.com/graph6]

I verified that this result was correct with the digest, but if I take a look at the document explaining the CN14 method (http://www.hpl.hp.com/techreports/2003/HPL-2003-142.html) I can't figure out how to come
to the above result. If my understanding of that method is correct it should either be N-Triples or RDF/XML. But N-Triples should be seperated by a '.' and a line end. The above result however does look a lot like the standard a standard toString() implementation of Java. However I was under the impression that canonical graphs are meant to be in a form independent to implementation or even programming language. Next to that the used blank node is _:g000001 . According to my understanding of the CN14 method, the number of used 0 in the blank node label should be the smallest amount possible so that every blank node label has the same length. This would mean the blank node should be _:g1 .
Am I:
A) Incorrect in my assumption about independability
B) Incorrect in my assumption about N-Triples
C) Correct in my assumption that something is wrong here
D) Anything else I forgot to mention
and if I am wrong where can I find the documentation where this is shown?

Greetings,
  Jaap Blom



  ________________________________
http://www.croon.nl/disclaimer
http://www.croon.nl/inkoopvoorwaarden
Kamer van Koophandel Rotterdam nr. / Chamber of Commerce Rotterdam no.: 24002312

[namedgraphs] SPARQL over named graphs with blank nodes

[Blom, J.J.C. (Jaap) <jb...@cr...>]

Hi,

I'm working on NG4J 0.9.3 and have the following code:
                Node graph1 = Node.createURI("http://graph1");
                Node graph2 = Node.createURI("http://graph2");
                Node start = Node.createURI("http://start");
                Node predicate1 = Node.createURI("http://predicate1");
                Node predicate2 = Node.createURI("http://predicate2");
                Node result1 = Node.createLiteral("expected result");
                Node result2 = Node.createLiteral("not expected result");
                NamedGraphSet ngs = new NamedGraphSetImpl();
                ngs.addQuad(new Quad(graph1, start, predicate1, Node.createAnon(new AnonId("blank"))));
                ngs.addQuad(new Quad(graph1, Node.createAnon(new AnonId("blank")), predicate2, result1));
                ngs.addQuad(new Quad(graph2, Node.createAnon(new AnonId("blank")), predicate2, result2));//*/
                String query =
                        "Select ?target Where\n" +
                        "{\n" +
                        "       <" + start.getURI() + "> <" + predicate1.getURI() + "> ?b .\n" +
                        "       ?b <" + predicate2.getURI() + "> ?target\n" +
                        "}";
                System.out.println(query);
                Query sparqlQuery = QueryFactory.create(query);
                QueryExecution qe = QueryExecutionFactory.create(sparqlQuery, new NamedGraphDataset(ngs));
                ResultSet results = qe.execSelect();
                while(results.hasNext())
                {
                        System.out.println(results.next());
                }
                System.out.println("done");

This gives the following result:
Select $target Where
{
        <http://start> <http://predicate1> ?b .
        ?b <http://predicate2> $target
}
( ?target = "expected result" )
( ?target = "not expected result" )
done

I didn't expect the "not expected result", since it is linked to a blank node in another graph. Granted the blank nodes have the same label, but RDF states that blank nodes in different graphs are not linked by their label. Am I correct in this one and if so is this a NG4J error or a Jena/ARQ error?

Greetings,
  Jaap Blom



  ________________________________
http://www.croon.nl/disclaimer
http://www.croon.nl/inkoopvoorwaarden
Kamer van Koophandel Rotterdam nr. / Chamber of Commerce Rotterdam no.: 24002312

[namedgraphs] Digest encoding

[Blom, J.J.C. (Jaap) <jb...@cr...>]

Hi,

I was wondering why the digest for a data graph is first hexadecimal encoded and then base 64 encoded (instead of only base 64 encoded like the signature for a warrant graph). I conclude this on basis of the code from NG4J 0.9.3. In de.fuberlin.wiwiss.ng4j.swp.util.SWPSignatureUtilities in the function protected static String calculateDigest( String dataInCanonicalForm, Node digestMethod ) on line 225 the following lines of code are present:
        res = Hex.encode( resBuf );
        return new String(Base64.encodeBase64( res )) ;



I suspect that this is a bug, am I correct?

Greetings,
  Jaap Blom



  ________________________________
http://www.croon.nl/disclaimer
http://www.croon.nl/inkoopvoorwaarden
Kamer van Koophandel Rotterdam nr. / Chamber of Commerce Rotterdam no.: 24002312

[namedgraphs] Best Way To Insert Large Amounts of Data Into NG4J DB

[Chris S. <chr...@is...>]

Hey everyone,

    I was able to load approx. 200k triples into the NG4J in memory implementation.  I was able to do this very quickly (under a minute) so I'm very impressed.  However, I'm struggling when it comes to taking those NamedGraphs from memory and insert them into the NG4J Database implementation.  Has anyone had any experience with inserting large amounts of data into the Database implementation?  Any help would be greatly appreciated.

Thanks,
Chris Shellenbarger
Intelligent Software Solutions, Inc.

[namedgraphs] Latest changes, release procedure, copyright notices, site hosting

[Tim P. <ti...@pa...>]

Hi,

I was surprised to find that a lot of changes had been made to a lot
of files when I did a CVS update.
This came as a surprise because the project does not have a commit
notifications list: I cannot fix this
as I am not a project admin however the process is given here:
https://sourceforge.net/apps/trac/sourceforge/wiki/CVS%20syncmail
(first create a mailing list called ng4j-cvs).

Trying to get a handle on what had been changed I did a CVS diff on
two files: the only change was
to the copyright notice date. My understanding is that this is not
good practice.
>From what I can gather copyright is neither granted from creation date
nor from assertion date, but
from date of author's death. In practice files which were created by one person
(according to the @author tag) were copyright another in the copyright
statement and another in the CVS ID.

I believe that the best practice is to leave copyright assertion
statements as year of creation, rather than range.

Olaf, of the incoming changes were there any that were not about copyright?

I have recently discovered how to restructure a project using the
sourceforge utility adminrepo:
http://tim-pizey.blogspot.com/2010/02/bringing-webmacro-up-for-air.html
and also how to automate the upload of project artefacts.

I have seen projects get separated from their documentation and
original authors before and
strongly recommend that we get the project site
http://www4.wiwiss.fu-berlin.de/bizer/ng4j/
to point to ng4j.sourceforge.net, not the other way around as currently.
This would also enable us to fix the other outstanding problem:
that we are using http://paneris.net/~maven2/ as our deployment repository.
I think we should be deploying to http://ng4j.sourceforge.net/maven2/
but I cannot set this up
whilst I am not an admin and we are redirecting all site traffic.

cheers
Tim

Re: [namedgraphs] Updated jenatest

[Tim P. <ti...@pa...>]

Hi Olaf,

On 25 February 2010 15:05, Olaf Hartig  wrote:
> Hey,
>
> I created a package of the NG4J 0.9.3 release and uploaded it to SourceForge.

Cool.

I have just updated webmacro, by restructuring its CVS repository see
http://tim-pizey.blogspot.com/2010/02/bringing-webmacro-up-for-air.html

We might want to do a similar restructure to ng4j.

I have also found out about how to create and deploy the release
bundle in Maven.
I would also recommend deploying our artefact to
http://ng4j.sourceforge.net/maven2/
however this currently relocates to
http://www4.wiwiss.fu-berlin.de/bizer/ng4j/maven2/

About ant - just winging.

cheers
Tim

Re: [namedgraphs] Updated jenatest

[Olaf H. <ha...@in...>]

Hey,

I created a package of the NG4J 0.9.3 release and uploaded it to SourceForge.

On Friday 19 February 2010 17:33:36 Tim Pizey wrote:
> Hi Olaf,
> 
> I looked lib/LIBRARIES.txt but it appears to be outdated now, as it is
>  about our previous upgrade. However I have now upgraded it.

Thanks.

> [...]
> I think that this additional work, of maintaining the lib directory
> and its documentation, is a reason to drop the ant build.

No, I still use ant to build NG4J and I guess I'm not the only one.
I don't think it is that much work replacing the relevant JARs in the lib 
directory and adjusting the LIBRARIES.txt file if one upgrades the libs
used by NG4J.

Olaf

Re: [namedgraphs] Updated jenatest

[Tim P. <ti...@pa...>]

Hi Olaf,

I looked lib/LIBRARIES.txt but it appears to be outdated now, as it is about
our previous upgrade. However I have now upgraded it.

I did upgrade some jars, it is a shame we do not have commit
notification emails enabled,
but from memory, and by inspection, I see jena-2.6.2.jar and
jena-2.6.2-tests.jar

Comparing http://paneris.net/ng4j/dependencies.html with contents of lib
I see that I overlooked, and have now updated, iri-0.6 to iri-0.7 and
arq-2.8.0 to arq-2.8.2

In addition we have an unused, transitive, dependency upon
lucene-core-2.3.1 which I have added.

These changes have kicked off a CI build and all passes.

I think that this additional work, of maintaining the lib directory
and its documentation, is a reason to drop the ant build.

cheers
Tim



On 19 February 2010 15:16, Olaf Hartig  wrote:
> Hey Tim,
>
> Did you miss some of the JARs updated in the Jena 2.6.2 package when you
> upgraded the Jena in NG4J to 2.6.2?
> I don't find information about your upgrade in lib/LIBRARIES.txt
>
> Greetings,
> Olaf

Re: [namedgraphs] Updated jenatest

[Olaf H. <ha...@in...>]

Hey Tim,

Did you miss some of the JARs updated in the Jena 2.6.2 package when you 
upgraded the Jena in NG4J to 2.6.2?
I don't find information about your upgrade in lib/LIBRARIES.txt

Greetings,
Olaf

Re: [namedgraphs] Maven Coordinates?

[Tim P. <ti...@pa...>]

Hi Olaf, 


On Monday 15 February 2010 18:11:44 Olaf Hartig wrote:
> Hey all,
> 
> I'm back from vacation.
> 
> On Wednesday 10 February 2010 19:29:08 Tim Pizey wrote:
> > [...]
> > I have released 0.9.3 and we are now working on 0.9.4-SNAPSHOT
> > [...]
> > I have not updated the wiki nor the sourceforge downloads
> 
> Does it mean in Maven we have a 0.9.3 but "officially" (i.e. on the Web sites) 
> it is still the old 0.9.2?

yes
 
> If so, I better prepare the official 0.9.3 package - i.e. doing the things 
> listed in http://sf.net/apps/mediawiki/ng4j/index.php?title=Release_Process
>  - Right?
> Did you tag the release in CVS?

yes


Of the steps in the wiki:

Steps to Create the Source Package 
cvs co 
 Update and commit the file doc/index.html (adjust the sections News and Download). 

no

 If it is the first release of a new year update the copyright notices in the source files and commit these changes. 

No
(Surely not, the copyright is the year the source is written, no?)

ant clean 
rm lib/ng4j* 

No

Change the version property in build.xml. 

Yes

 Change the version in pom.xml. 

Yes

 Change the version number of the SWClLib in src/de/fuberlin/wiwiss/ng4j/semwebclient/CommandLineQuery.java. 

yes

 Reflect the new release in CHANGES. 

yes

ant
 
 Tag the CVS with the new version number (e.g. cvs -q tag v0_9_1). 

yes

 Change the version in build.xml and in pom.xml to reflect the post-release state. 
 Create a new ng4j-${version} directory somewhere. 
 Unzip the new ZIP file in this new directory. 
 Remove all IDE-specific config and project files from the new directory. 

I think that we should store the IDE specific configs that we use: 
there are not many IDEs. 



 Zip the new directory. 
 Steps to Publish the Source Package 
 Create a new release on SourceForge 
 Kindly ask Chris to upload the updated doc/index.html. 
 Announce the new release. 
 Open a beer ;-) 


I used the maven release tool: 
mvn release:prepare 
mvn release:perform

This does the editting of the pom and the tagging and committing. 

I put both the ant and maven builds into http://hudson.paneris.net/ 


I think that the only thing to do is update doc/index.html, 
run ant and upload artifacts to sourceforge. 


cheers
tim





-- 
We are in dialogue.

Re: [namedgraphs] Maven Coordinates?

[Olaf H. <ha...@in...>]

Hey all,

I'm back from vacation.

On Wednesday 10 February 2010 19:29:08 Tim Pizey wrote:
> [...]
> I have released 0.9.3 and we are now working on 0.9.4-SNAPSHOT
> [...]
> I have not updated the wiki nor the sourceforge downloads

Does it mean in Maven we have a 0.9.3 but "officially" (i.e. on the Web sites) 
it is still the old 0.9.2?

If so, I better prepare the official 0.9.3 package - i.e. doing the things 
listed in http://sf.net/apps/mediawiki/ng4j/index.php?title=Release_Process
 - Right?
Did you tag the release in CVS?

Greetings,
Olaf

[namedgraphs] Updated jenatest

[Tim P. <ti...@pa...>]

Hi,

Yesterday I did not figure out where the jenattest jar had got to.
It is now known as jena-2.6.2-tests.jar

That our main code is dependent upon a test jar should be fixed.

The CI kicked in http://hudson.paneris.net/job/ng4j/43/console
so http://paneris.net/ng4j/dependencies.html is now up to date.

cheers
Tim

Re: [namedgraphs] Maven Coordinates?

[Tim P. <ti...@pa...>]

Hi all,

I have, finally, set up my Hudson server.

http://hudson.paneris.net/

I did not get a reply as to whether to commit, so I went ahead as the
Jena Repository url has changed.

We and arc are still dependent upon jenatest-0.6.0.  The class
NodeUtils should be moved into the main jar.
I was given an assurance on teh Jena list that this would happen a while back.

I then had real problems with the maven release plugin which boiled down to:

export CVS_RSH=ssh

You can now perform a release with

mvn release:prepare release:perform

I have released 0.9.3 and we are now working on 0.9.4-SNAPSHOT

I have setup two CI jobs: ng4j which does a complete maven build:

mvn clean deploy, site site:deploy

which deploys to the maven repo at

http://paneris.net/maven2/

and deploys a Maven generated web site to http://paneris.net/ng4j/

The ant build checks that it still completes successfully (test
failures now are a failure).

I have not updated the wiki nor the sourceforge downloads

cheers
Tim

Re: [namedgraphs] Using NG4J With SDB

[Jennifer C. <jen...@at...>]

Chris,
 
Sounds good.  Please let us know how you make out.  It would be great to
better understand SDB's named graph implementation.  The ability to store
NG4J named graphs in SDB could be a valuable addition to the project,
regardless of its performance.
 
NG4J's database storage is not as efficient as it could be.  One
performance improvement that I'd like to see is using
java.sql.PreparedStatement instead of java.sql.Statement.  Additional
optimization could be done as well.
 
Jennifer
 


  _____  

From: Chris Shellenbarger [mailto:chr...@is...] 
Sent: Tuesday, February 09, 2010 1:24 PM
To: Jennifer Cormier
Subject: RE: [namedgraphs] Using NG4J With SDB



Thank you very much for your prompt reply - I'll look into the concerns
you have mentioned and see how the performance differs with each approach.

 

Thanks again,

 

Chris

 

From: Jennifer Cormier [mailto:jen...@at...] 
Sent: Tuesday, February 09, 2010 11:12 AM
To: Chris Shellenbarger; ng4...@li...
Subject: RE: [namedgraphs] Using NG4J With SDB

 

Hi Chris,

 

I agree, it would be wonderful the SDB and NG4J named graphs were
compatible.

 

Unfortunately, although NG4J does use Jena, the way that it implements
"named graphs" is quite different than that of SDB.

 

My understanding is that SDB supports naming a graph that contains
triples.  So each graph is a separate triple store.

 

NG4J uses a quad store.  Its "statements" are quads.  The first item in
each statement is the name of the graph.

 

When I looked at SDB's support of named graphs the other year, I had the
impression that each graph had to be "manually" named and loaded in from a
config file.  If that's the case, then in order for NG4J to work with SDB
(if that's possible) it seems to me that we need an additional "pipe" or
processor/transform tool that would handle the missing steps and automate
everything.  It would need to create a separate (SDB) graph for each NG4J
named graph and give the SDB graph the name corresponding to the NG4J one.
Doing so might remove much of SDB's performance gain.

 

Another thing to consider is that with NG4J, it is not necessary to "know"
the name of each graph.  All the graphs are stored together (e.g. in the
same database) as quads.  You can easily query (using SPARQL) across all
the graphs.  This is possible without loading the graphs into memory.  The
SPARQL query is converted to a SQL database query.

 

I don't know much about SDB, but I suspect that in order to query across
all graphs it would be necessary to first load them all into memory, or
put them together in a single, larger graph.  Once that is done, how then
does one determine to which named graph a statement belongs?  I don't have
the answers.

 

That's my understanding of the situation.  Clarifications welcome.

 

Jennifer

 

 

  _____  

From: Chris Shellenbarger [mailto:chr...@is...] 
Sent: Tuesday, February 09, 2010 12:13 PM
To: ng4...@li...
Subject: [namedgraphs] Using NG4J With SDB

Hello,

    I am really interested in using NG4J in a project I am working on.
However, the only thing holding me back is that I'd like to back the NG4J
usage with the Jena SDB datastore.  I noticed that NG4J has its own
Database storage implementation but I was wondering if anyone has been
able to successfully plugin SDB in use with NG4J?  Since they both work
with the Jena configuration and support named graphs it seems that there
should be a way to work with both - I'm having trouble getting it to work
however.  If anyone could help, I'd greatly appreciate it.

 

Thanks,

Chris

Re: [namedgraphs] Using NG4J With SDB

[Jennifer C. <jen...@at...>]

Hi Chris,
 
I agree, it would be wonderful the SDB and NG4J named graphs were
compatible.
 
Unfortunately, although NG4J does use Jena, the way that it implements
"named graphs" is quite different than that of SDB.
 
My understanding is that SDB supports naming a graph that contains
triples.  So each graph is a separate triple store.
 
NG4J uses a quad store.  Its "statements" are quads.  The first item in
each statement is the name of the graph.
 
When I looked at SDB's support of named graphs the other year, I had the
impression that each graph had to be "manually" named and loaded in from a
config file.  If that's the case, then in order for NG4J to work with SDB
(if that's possible) it seems to me that we need an additional "pipe" or
processor/transform tool that would handle the missing steps and automate
everything.  It would need to create a separate (SDB) graph for each NG4J
named graph and give the SDB graph the name corresponding to the NG4J one.
Doing so might remove much of SDB's performance gain.
 
Another thing to consider is that with NG4J, it is not necessary to "know"
the name of each graph.  All the graphs are stored together (e.g. in the
same database) as quads.  You can easily query (using SPARQL) across all
the graphs.  This is possible without loading the graphs into memory.  The
SPARQL query is converted to a SQL database query.
 
I don't know much about SDB, but I suspect that in order to query across
all graphs it would be necessary to first load them all into memory, or
put them together in a single, larger graph.  Once that is done, how then
does one determine to which named graph a statement belongs?  I don't have
the answers.
 
That's my understanding of the situation.  Clarifications welcome.
 
Jennifer
 


  _____  

From: Chris Shellenbarger [mailto:chr...@is...] 
Sent: Tuesday, February 09, 2010 12:13 PM
To: ng4...@li...
Subject: [namedgraphs] Using NG4J With SDB



Hello,

    I am really interested in using NG4J in a project I am working on.
However, the only thing holding me back is that I'd like to back the NG4J
usage with the Jena SDB datastore.  I noticed that NG4J has its own
Database storage implementation but I was wondering if anyone has been
able to successfully plugin SDB in use with NG4J?  Since they both work
with the Jena configuration and support named graphs it seems that there
should be a way to work with both - I'm having trouble getting it to work
however.  If anyone could help, I'd greatly appreciate it.

 

Thanks,

Chris