Menu
▾
▴
Re: [Bigdata-developers] SPARQL DELETE, INSERT performance question
|
From: Bryan T. <br...@bl...> - 2016-08-24 19:50:45
|
Great. Glad to hear it.
Thanks,
Bryan
On Wed, Aug 24, 2016 at 3:41 PM, Kevin Ford <kf...@ar...> wrote:
> Thanks, Bryan.
>
> Reducing the amount of max memory, thereby avoiding swapping, had a
> positive impact.
>
> Yours,
> Kevin
>
>
> On 8/19/16 16:06, Bryan Thompson wrote:
> > You might also give it less memory for a small data set. What is
> > important is making sure that there is NO swapping. Java handles
> > swapping very poorly because it needs to do memory scans during GC.
> >
> > Thanks,
> > Bryan
> >
> > On Fri, Aug 19, 2016 at 4:36 PM, Kevin Ford <kf...@ar...
> > <mailto:kf...@ar...>> wrote:
> >
> > Thanks, Bryan.
> >
> > We're looking at whether we can increase the memory on the machine
> > (currently 8gb, but has other applications running on it) and see if
> > that improves matters.
> >
> > Yours,
> > Kevin
> >
> >
> > On 8/18/16 12:32, Bryan Thompson wrote:
> > > Kevin,
> > >
> > > I would not anticipate the kinds of times that you are reporting
> below
> > > for these update requests. Assuming that URI and URI/other are
> > not all
> > > of the triples in your graph, these should be very fast updates.
> > >
> > > Can you confirm that the machine has at least those 4GB of RAM
> > available
> > > beyond the other active processes?
> > >
> > > Note that Linux distributions will begin to swap out processes
> > once 50%
> > > of the physical memory has been used unless you set swappiness to
> zero
> > > (a kernel parameter - see the Blazegraph wiki for more information
> on
> > > this topic).
> > >
> > > Bryan
> > >
> > > On Wednesday, August 17, 2016, Kevin Ford <kf...@ar...
> > <mailto:kf...@ar...>
> > > <mailto:kf...@ar... <mailto:kf...@ar...>>> wrote:
> > >
> > > Dear All,
> > >
> > > I have a question about SPARQL DELETE/INSERT performance.
> > >
> > > We have a job that issues two DELETE queries followed by an
> INSERT
> > > query, per request, to keep our triplestore up to date. The
> > pattern is
> > > as follows:
> > >
> > > DELETE { <URI> ?p ?o };
> > > DELETE { <URI/other> ?p ?o };
> > > INSERT {
> > > <URI> <new> '1' .
> > > <URI> <new> '2' .
> > > <URI> <new> '3' .
> > > <URI/other> <new> '1' .
> > > }
> > >
> > > We've noted two things:
> > >
> > > 1) If a store is empty, this is relatively fast.
> > > 2) IF the store is configured for automatic inferencing, it is
> > slower
> > > than a store without that feature activated.
> > >
> > > Neither of those observations seems particularly surprising,
> > but each of
> > > those two DELETE statements and the one INSERT takes more than
> > 3 seconds
> > > against a store with only 4.6 million triples. See below for
> > a sample
> > > of the output.
> > >
> > > Blazegraph is allocated 4g of memory.
> > >
> > > We're working within a framework that produces these series of
> > queries
> > > so, before making any custom modifications, I was wondering if
> > this
> > > performance is to be expected? If not, would it point to a
> > configuation
> > > issue of some kind, or something else?
> > >
> > > Also, I've noted that issuing a DELETE HTTP request appears to
> > handle
> > > the deletes faster, but is there a more optimal way to
> > construct those
> > > three queries (the DELETE/INSERT WHERE pattern did not appear
> > to improve
> > > matters based on a few isolated tests).
> > >
> > > Cordially,
> > > Kevin
> > >
> > >
> > > $ curl -X POST
> > https://laketsidx/blazegraph/namespace/lakeidx/sparql
> > <https://laketsidx/blazegraph/namespace/lakeidx/sparql>
> > > <https://laketsidx/blazegraph/namespace/lakeidx/sparql
> > <https://laketsidx/blazegraph/namespace/lakeidx/sparql>>
> > > --data @bg-test-02.sparql -H "Content-type:
> > > application/x-www-form-urlencoded"
> > > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
> > > "http://www.w3.org/TR/html4/loose.dtd
> > <http://www.w3.org/TR/html4/loose.dtd>
> > > <http://www.w3.org/TR/html4/loose.dtd
> > <http://www.w3.org/TR/html4/loose.dtd>>"><html><head><meta
> > > http-equiv="Content-Type"
> > > content="text/html;charset=UTF-8"><title>blazegraph™
> by
> > > SYSTAP</title
> > > ></head
> > > ><body<p>totalElapsed=33ms, elapsed=33ms, connFlush=0ms,
> > > batchResolve=0, whereClause=23ms, deleteClause=9ms,
> > insertClause=23ms</p
> > > ><hr><p>totalElapsed=2579ms, elapsed=21ms, connFlush=2524ms,
> > > batchResolve=0, whereClause=21ms, deleteClause=0ms,
> > insertClause=21ms</p
> > > ><hr><p>totalElapsed=2582ms, elapsed=2ms, connFlush=0ms,
> > > batchResolve=0, whereClause=0ms, deleteClause=0ms,
> > insertClause=0ms</p
> > > ><hr><p>COMMIT: totalElapsed=3118ms,
> commitTime=1471453475719,
> > > mutationCount=62</p
> > > ></html
> > >
> > > $ curl -X POST
> > https://laketsidx/blazegraph/namespace/lakeidx/sparql
> > <https://laketsidx/blazegraph/namespace/lakeidx/sparql>
> > > <https://laketsidx/blazegraph/namespace/lakeidx/sparql
> > <https://laketsidx/blazegraph/namespace/lakeidx/sparql>>
> > > --data @bg-test-02.sparql -H "Content-type:
> > > application/x-www-form-urlencoded"
> > > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
> > > "http://www.w3.org/TR/html4/loose.dtd
> > <http://www.w3.org/TR/html4/loose.dtd>
> > > <http://www.w3.org/TR/html4/loose.dtd
> > <http://www.w3.org/TR/html4/loose.dtd>>"><html><head><meta
> > > http-equiv="Content-Type"
> > > content="text/html;charset=UTF-8"><title>blazegraph™
> by
> > > SYSTAP</title
> > > ></head
> > > ><body<p>totalElapsed=24ms, elapsed=24ms, connFlush=0ms,
> > > batchResolve=0, whereClause=12ms, deleteClause=11ms,
> > > insertClause=12ms</p
> > > ><hr><p>totalElapsed=2966ms, elapsed=12ms, connFlush=2929ms,
> > > batchResolve=0, whereClause=12ms, deleteClause=0ms,
> > insertClause=12ms</p
> > > ><hr><p>totalElapsed=2969ms, elapsed=2ms, connFlush=0ms,
> > > batchResolve=0, whereClause=0ms, deleteClause=0ms,
> > insertClause=0ms</p
> > > ><hr><p>COMMIT: totalElapsed=3669ms,
> commitTime=1471453483251,
> > > mutationCount=62</p
> > > ></html
> > >
> > >
> > >
> > ------------------------------------------------------------
> ------------------
> > > _______________________________________________
> > > Bigdata-developers mailing list
> > > Big...@li...
> > <mailto:Big...@li...> <javascript:;>
> > >
> > https://lists.sourceforge.net/lists/listinfo/bigdata-developers
> > <https://lists.sourceforge.net/lists/listinfo/bigdata-developers>
> > >
> > <https://lists.sourceforge.net/lists/listinfo/bigdata-developers
> > <https://lists.sourceforge.net/lists/listinfo/bigdata-developers>>
> > >
> >
> > ------------------------------------------------------------
> ------------------
> > _______________________________________________
> > Bigdata-developers mailing list
> > Big...@li...
> > <mailto:Big...@li...>
> > https://lists.sourceforge.net/lists/listinfo/bigdata-developers
> > <https://lists.sourceforge.net/lists/listinfo/bigdata-developers>
> >
> >
>
> ------------------------------------------------------------
> ------------------
> _______________________________________________
> Bigdata-developers mailing list
> Big...@li...
> https://lists.sourceforge.net/lists/listinfo/bigdata-developers
>
|
