Re: [Bigdata-developers] trivial ASK takes over 5s

[Bryan T. <br...@sy...>]

It is extended in RWStrategy.truncate(), which calls through to
RWStore.establishExtent().  I would recommend turning on the txLog logger.
This provides better information about the different phases of the commit
and their latency.

    public void truncate(final long extent) {



        m_store.establishExtent(extent);



}

Bryan

----
Bryan Thompson
Chief Scientist & Founder
Blazegraph
e: br...@bl...
w: http://blazegraph.com

Blazegraph products help to solve the Graph Cache Thrash to achieve large
scale processing for graph and predictive analytics.  Blazegraph is the
creator of the industry’s first GPU-accelerated high-performance database
for large graphs, has been named as one of the “10 Companies and
Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>.

Blazegraph Database <https://www.blazegraph.com/> is our ultra-high
performance graph database that supports both RDF/SPARQL and
Tinkerpop/Blueprints APIs.   Blazegraph GPU
<https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS
<https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new
technologies that use GPUs to enable extreme scaling that is thousands of
times faster and 40 times more affordable than CPU-based solutions.

CONFIDENTIALITY NOTICE:  This email and its contents and attachments are
for the sole use of the intended recipient(s) and are confidential or
proprietary to SYSTAP, LLC DBA Blazegraph.  Any unauthorized review, use,
disclosure, dissemination or copying of this email or its contents or
attachments is prohibited. If you have received this communication in
error, please notify the sender by reply email and permanently delete all
copies of the email and its contents and attachments.

On Tue, Mar 22, 2016 at 1:26 PM, Jeremy J Carroll <jj...@sy...> wrote:

> With more testing I found that by repeatedly inserting a few million
> triples to a journal with two or three billion triples, normally the
> journal size does not change, but then when it did grow, it grew by about
> 40G (the journal is about 500G).
> I am wondering where in the code that file growth happens, I suspect that
> that could easily take 5 seconds, and lock out other activity while it is
> happening.
>
> The latency reported in the commitNow function was pretty acceptable at 13051
> nanoseconds
>
> Jeremy
>
>
>
>
> On Mar 21, 2016, at 3:26 PM, Bryan Thompson <br...@sy...> wrote:
>
> Certainly. Also, as i indicated, it may be possible to reduce the lockout
> period. In principle, it seems that flushing the write cache outside of the
> lock would be safe. But this would need a careful code review.
>
> Thanks,
> Bryan
> On Mar 21, 2016 6:24 PM, "Jeremy J Carroll" <jj...@sy...> wrote:
>
>> OK,
>> looking at the implementation of commitNow, I take it to be a fairly
>> low-level piece of code where holding the write lock seems reasonable.
>> The easiest way to reduce the length of time that the write lock is being
>> held is to either reduced the amount of data being written or increased the
>> disk speed.
>>
>> For us to understand the length of time the write lock is written it
>> looks like we should enable Info level on the logger and then we will see
>> commit times and this will give us an idea of how bad the problem is.
>>
>> Then if we cannot reduce the commit time sufficiently we should get back
>> to you.
>>
>> Does that sound like a reasonable plan on this issue?
>>
>> Jeremy
>>
>>
>>
>> On Mar 21, 2016, at 1:27 PM, Jeremy J Carroll <jj...@sy...> wrote:
>>
>> Yes - or blocking some other part of the process ...
>>
>>
>> On Mar 21, 2016, at 12:12 PM, Bryan Thompson <br...@sy...> wrote:
>>
>> Is the thought then that there might be some barrier that is blocking the
>> start of a new query during some part of that large create?
>>
>> ----
>> Bryan Thompson
>> Chief Scientist & Founder
>> Blazegraph
>> e: br...@bl...
>> w: http://blazegraph.com
>>
>> Blazegraph products help to solve the Graph Cache Thrash to achieve large
>> scale processing for graph and predictive analytics.  Blazegraph is the
>> creator of the industry’s first GPU-accelerated high-performance database
>> for large graphs, has been named as one of the “10 Companies and
>> Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>.
>>
>>
>> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high
>> performance graph database that supports both RDF/SPARQL and
>> Tinkerpop/Blueprints APIs.   Blazegraph GPU
>> <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS
>> <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive
>> new technologies that use GPUs to enable extreme scaling that is thousands
>> of times faster and 40 times more affordable than CPU-based solutions.
>>
>> CONFIDENTIALITY NOTICE:  This email and its contents and attachments are
>> for the sole use of the intended recipient(s) and are confidential or
>> proprietary to SYSTAP, LLC DBA Blazegraph.  Any unauthorized review, use,
>> disclosure, dissemination or copying of this email or its contents or
>> attachments is prohibited. If you have received this communication in
>> error, please notify the sender by reply email and permanently delete all
>> copies of the email and its contents and attachments.
>>
>> On Mon, Mar 21, 2016 at 12:40 PM, Jeremy J Carroll <jj...@sy...>
>> wrote:
>>
>>> The logs I have are:
>>> - full GC logs
>>> - python level logs of http requests and errors/warnings (of our
>>> application server: not the blazegraph http requests)
>>>
>>> I can map those logs into SPARQL requests approximately, and I see that
>>> at the point where I got the long wait, there were a single large create
>>> operations that use the
>>> INSERT-by-POST operation taking approximately 30s (including python
>>> processing) which normally is much more than 50% of blazegraph time.
>>>
>>> Very minor other activity going on
>>>
>>> No interesting GC activity
>>>
>>> Jeremy
>>>
>>>
>>>
>>>
>>> On Mar 19, 2016, at 11:57 AM, Bryan Thompson <br...@sy...> wrote:
>>>
>>> Do you have full logs for the server during this interval?
>>> Bryan
>>>
>>> ----
>>> Bryan Thompson
>>> Chief Scientist & Founder
>>> Blazegraph
>>> e: br...@bl...
>>> w: http://blazegraph.com
>>>
>>> Blazegraph products help to solve the Graph Cache Thrash to achieve
>>> large scale processing for graph and predictive analytics.  Blazegraph is
>>> the creator of the industry’s first GPU-accelerated high-performance
>>> database for large graphs, has been named as one of the “10 Companies
>>> and Technologies to Watch in 2016”
>>> <http://insideanalysis.com/2016/01/20535/>.
>>>
>>> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high
>>> performance graph database that supports both RDF/SPARQL and
>>> Tinkerpop/Blueprints APIs.   Blazegraph GPU
>>> <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS
>>> <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive
>>> new technologies that use GPUs to enable extreme scaling that is thousands
>>> of times faster and 40 times more affordable than CPU-based solutions.
>>>
>>> CONFIDENTIALITY NOTICE:  This email and its contents and attachments
>>> are for the sole use of the intended recipient(s) and are confidential or
>>> proprietary to SYSTAP, LLC DBA Blazegraph.  Any unauthorized review, use,
>>> disclosure, dissemination or copying of this email or its contents or
>>> attachments is prohibited. If you have received this communication in
>>> error, please notify the sender by reply email and permanently delete all
>>> copies of the email and its contents and attachments.
>>>
>>> On Fri, Mar 18, 2016 at 6:53 PM, Jeremy J Carroll <jj...@sy...>
>>> wrote:
>>>
>>>> I checked the GC logs. We have no GC stoppage of more than a second for
>>>> days either side of the slow query.
>>>>
>>>> The slow query was on March 6th, there was a 3s stoppage on Feb 29th
>>>> and a 1.8s stoppage on March 9th
>>>>
>>>> Any other thoughts?
>>>>
>>>> Jeremy
>>>>
>>>>
>>>> On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy...> wrote:
>>>>
>>>> 5s is probably a major GC event.
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>