Menu
▾
▴
bigdata-developers — List for bigdata developers
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
(19) |
Mar
(8) |
Apr
(25) |
May
(16) |
Jun
(77) |
Jul
(131) |
Aug
(76) |
Sep
(30) |
Oct
(7) |
Nov
(3) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(2) |
Jun
(2) |
Jul
(16) |
Aug
(3) |
Sep
(1) |
Oct
|
Nov
(7) |
Dec
(7) |
| 2012 |
Jan
(10) |
Feb
(1) |
Mar
(8) |
Apr
(6) |
May
(1) |
Jun
(3) |
Jul
(1) |
Aug
|
Sep
(1) |
Oct
|
Nov
(8) |
Dec
(2) |
| 2013 |
Jan
(5) |
Feb
(12) |
Mar
(2) |
Apr
(1) |
May
(1) |
Jun
(1) |
Jul
(22) |
Aug
(50) |
Sep
(31) |
Oct
(64) |
Nov
(83) |
Dec
(28) |
| 2014 |
Jan
(31) |
Feb
(18) |
Mar
(27) |
Apr
(39) |
May
(45) |
Jun
(15) |
Jul
(6) |
Aug
(27) |
Sep
(6) |
Oct
(67) |
Nov
(70) |
Dec
(1) |
| 2015 |
Jan
(3) |
Feb
(18) |
Mar
(22) |
Apr
(121) |
May
(42) |
Jun
(17) |
Jul
(8) |
Aug
(11) |
Sep
(26) |
Oct
(15) |
Nov
(66) |
Dec
(38) |
| 2016 |
Jan
(14) |
Feb
(59) |
Mar
(28) |
Apr
(44) |
May
(21) |
Jun
(12) |
Jul
(9) |
Aug
(11) |
Sep
(4) |
Oct
(2) |
Nov
(1) |
Dec
|
| 2017 |
Jan
(20) |
Feb
(7) |
Mar
(4) |
Apr
(18) |
May
(7) |
Jun
(3) |
Jul
(13) |
Aug
(2) |
Sep
(4) |
Oct
(9) |
Nov
(2) |
Dec
(5) |
| 2018 |
Jan
|
Feb
|
Mar
|
Apr
(2) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2019 |
Jan
|
Feb
|
Mar
(1) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Bryan T. <br...@sy...> - 2016-03-31 12:24:50
|
The 2.1.0-SNAPSHOT release => You can download the WAR from https://oss.sonatype.org/content/repositories/snapshots/com/blazegraph/bigdata-war/2.1.0-SNAPSHOT/ . Please let us know if this corrects the issue. Thanks, Bryan ---- Bryan Thompson Chief Scientist & Founder Blazegraph e: br...@bl... w: http://blazegraph.com Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. On Thu, Mar 31, 2016 at 8:23 AM, Bryan Thompson <br...@sy...> wrote: > This should be fixed in 2.1.0, which is now in QA. Brad can provide you > with a snapshot leading up to the release for testing. > > Our hypothesis is that the problem arose from failing to account for the > accumulation of blank nodes in an internal buffer. The fix takes account of > this and then evicts batches before the buffer would overflow. The overflow > is the index out of bounds exception. > > Thanks, > Bryan > > ---- > Bryan Thompson > Chief Scientist & Founder > Blazegraph > e: br...@bl... > w: http://blazegraph.com > > Blazegraph products help to solve the Graph Cache Thrash to achieve large > scale processing for graph and predictive analytics. Blazegraph is the > creator of the industry’s first GPU-accelerated high-performance database > for large graphs, has been named as one of the “10 Companies and > Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. > > > Blazegraph Database <https://www.blazegraph.com/> is our ultra-high > performance graph database that supports both RDF/SPARQL and > Tinkerpop/Blueprints APIs. Blazegraph GPU > <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS > <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new > technologies that use GPUs to enable extreme scaling that is thousands of > times faster and 40 times more affordable than CPU-based solutions. > > CONFIDENTIALITY NOTICE: This email and its contents and attachments are > for the sole use of the intended recipient(s) and are confidential or > proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, > disclosure, dissemination or copying of this email or its contents or > attachments is prohibited. If you have received this communication in > error, please notify the sender by reply email and permanently delete all > copies of the email and its contents and attachments. > > On Thu, Mar 31, 2016 at 8:00 AM, Andreas Kahl <ka...@bs...> > wrote: > >> Hello everyone, >> >> I did some extensive testing on this error, but I cannot load this file >> into Blazegraph 2.0.1 (I tried the .jar from Sourceforge and compiled the >> latest master revision with Oracle JDK 1.8.0_74 myself). >> Blazegraph runs into this: >> java.lang.RuntimeException: Could not load: >> url=file:///srv/feed-dateien/DNBLOD/GND.ttl.gz, >> cause=java.lang.ArrayIndexOutOfBoundsException: 40005 >> >> Please find the complete Stacktrace in the Logs attached. >> >> The SPARQL LOAD Command: >> curl -d"update=LOAD <file:///srv/feed-dateien/DNBLOD/GND.ttl.gz>" >> -d"monitor=true" http://localhost:8080/blazegraph/sparql 2>&1 >> >>/var/log/bigdata/loadGnd.log >> >> The data tested can be downloaded from >> http://datendienst.dnb.de/cgi-bin/mabit.pl?cmd=fetch&userID=opendata&pass=opendata&mabheft=GND.ttl.gz >> >> >> Blazegraph was running with this call: >> nohup java -server -Xmx6g -XX:+UseG1GC >> -Djava.io.tmpdir=/mnt/triplestore/tomcat8/temp/ -cp >> /mnt/triplestore/bigdata/blazegraph-jar-2.0.1.jar >> com.bigdata.rdf.sail.webapp.NanoSparqlServer 8080 kb >> /etc/bigdata/RWStore.properties >/var/log/bigdata/blazegraph.log & >> >> Java-Version on the server: >> java -version >> java version "1.8.0_65" >> Java(TM) SE Runtime Environment (build 1.8.0_65-b17) >> Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode) >> >> RWStore.properties was practically the original one (I just switched to >> Triples Mode and altered the Journal file's location. No custom >> Vocabularies or namespaces were used. >> >> Is there any known Issue? Should I try some specific revision from Github? >> >> Thanks for any hints. >> >> Andreas >> >> >> ------------------------------------------------------------------------------ >> Transform Data into Opportunity. >> Accelerate data analysis in your applications with >> Intel Data Analytics Acceleration Library. >> Click to learn more. >> http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140 >> _______________________________________________ >> Bigdata-developers mailing list >> Big...@li... >> https://lists.sourceforge.net/lists/listinfo/bigdata-developers >> >> > |
|
From: Bryan T. <br...@sy...> - 2016-03-31 12:23:44
|
This should be fixed in 2.1.0, which is now in QA. Brad can provide you with a snapshot leading up to the release for testing. Our hypothesis is that the problem arose from failing to account for the accumulation of blank nodes in an internal buffer. The fix takes account of this and then evicts batches before the buffer would overflow. The overflow is the index out of bounds exception. Thanks, Bryan ---- Bryan Thompson Chief Scientist & Founder Blazegraph e: br...@bl... w: http://blazegraph.com Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. On Thu, Mar 31, 2016 at 8:00 AM, Andreas Kahl <ka...@bs...> wrote: > Hello everyone, > > I did some extensive testing on this error, but I cannot load this file > into Blazegraph 2.0.1 (I tried the .jar from Sourceforge and compiled the > latest master revision with Oracle JDK 1.8.0_74 myself). > Blazegraph runs into this: > java.lang.RuntimeException: Could not load: > url=file:///srv/feed-dateien/DNBLOD/GND.ttl.gz, > cause=java.lang.ArrayIndexOutOfBoundsException: 40005 > > Please find the complete Stacktrace in the Logs attached. > > The SPARQL LOAD Command: > curl -d"update=LOAD <file:///srv/feed-dateien/DNBLOD/GND.ttl.gz>" > -d"monitor=true" http://localhost:8080/blazegraph/sparql 2>&1 > >>/var/log/bigdata/loadGnd.log > > The data tested can be downloaded from > http://datendienst.dnb.de/cgi-bin/mabit.pl?cmd=fetch&userID=opendata&pass=opendata&mabheft=GND.ttl.gz > > > Blazegraph was running with this call: > nohup java -server -Xmx6g -XX:+UseG1GC > -Djava.io.tmpdir=/mnt/triplestore/tomcat8/temp/ -cp > /mnt/triplestore/bigdata/blazegraph-jar-2.0.1.jar > com.bigdata.rdf.sail.webapp.NanoSparqlServer 8080 kb > /etc/bigdata/RWStore.properties >/var/log/bigdata/blazegraph.log & > > Java-Version on the server: > java -version > java version "1.8.0_65" > Java(TM) SE Runtime Environment (build 1.8.0_65-b17) > Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode) > > RWStore.properties was practically the original one (I just switched to > Triples Mode and altered the Journal file's location. No custom > Vocabularies or namespaces were used. > > Is there any known Issue? Should I try some specific revision from Github? > > Thanks for any hints. > > Andreas > > > ------------------------------------------------------------------------------ > Transform Data into Opportunity. > Accelerate data analysis in your applications with > Intel Data Analytics Acceleration Library. > Click to learn more. > http://pubads.g.doubleclick.net/gampad/clk?id=278785471&iu=/4140 > _______________________________________________ > Bigdata-developers mailing list > Big...@li... > https://lists.sourceforge.net/lists/listinfo/bigdata-developers > > |
|
From: Andreas K. <ka...@bs...> - 2016-03-31 12:20:39
|
Hello everyone, I did some extensive testing on this error, but I cannot load this file into Blazegraph 2.0.1 (I tried the .jar from Sourceforge and compiled the latest master revision with Oracle JDK 1.8.0_74 myself). Blazegraph runs into this: java.lang.RuntimeException: Could not load: url=file:///srv/feed-dateien/DNBLOD/GND.ttl.gz, cause=java.lang.ArrayIndexOutOfBoundsException: 40005 Please find the complete Stacktrace in the Logs attached. The SPARQL LOAD Command: curl -d"update=LOAD <file:///srv/feed-dateien/DNBLOD/GND.ttl.gz>" -d"monitor=true" http://localhost:8080/blazegraph/sparql 2>&1 >>/var/log/bigdata/loadGnd.log The data tested can be downloaded from http://datendienst.dnb.de/cgi-bin/mabit.pl?cmd=fetch&userID=opendata&pass=opendata&mabheft=GND.ttl.gz Blazegraph was running with this call: nohup java -server -Xmx6g -XX:+UseG1GC -Djava.io.tmpdir=/mnt/triplestore/tomcat8/temp/ -cp /mnt/triplestore/bigdata/blazegraph-jar-2.0.1.jar com.bigdata.rdf.sail.webapp.NanoSparqlServer 8080 kb /etc/bigdata/RWStore.properties >/var/log/bigdata/blazegraph.log & Java-Version on the server: java -version java version "1.8.0_65" Java(TM) SE Runtime Environment (build 1.8.0_65-b17) Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode) RWStore.properties was practically the original one (I just switched to Triples Mode and altered the Journal file's location. No custom Vocabularies or namespaces were used. Is there any known Issue? Should I try some specific revision from Github? Thanks for any hints. Andreas |
|
From: Stas M. <sma...@wi...> - 2016-03-28 23:06:52
|
Hi! > Do you plan to add support for JSONP in your SPARLQ endpoint to overcome cross-domain restrictions? I think cross-domain restrictions can be also worked around by having a proxy in front of Blazegraph that adds header 'Access-Control-Allow-Origin: *'. -- Stas Malyshev sma...@wi... |
|
From: Joakim S. <joa...@bl...> - 2016-03-28 21:17:49
|
Hi Do you plan to add support for JSONP in your SPARLQ endpoint to overcome cross-domain restrictions? |
|
From: Bryan T. <br...@sy...> - 2016-03-22 17:31:32
|
It is extended in RWStrategy.truncate(), which calls through to
RWStore.establishExtent(). I would recommend turning on the txLog logger.
This provides better information about the different phases of the commit
and their latency.
public void truncate(final long extent) {
m_store.establishExtent(extent);
}
Bryan
----
Bryan Thompson
Chief Scientist & Founder
Blazegraph
e: br...@bl...
w: http://blazegraph.com
Blazegraph products help to solve the Graph Cache Thrash to achieve large
scale processing for graph and predictive analytics. Blazegraph is the
creator of the industry’s first GPU-accelerated high-performance database
for large graphs, has been named as one of the “10 Companies and
Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>.
Blazegraph Database <https://www.blazegraph.com/> is our ultra-high
performance graph database that supports both RDF/SPARQL and
Tinkerpop/Blueprints APIs. Blazegraph GPU
<https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS
<https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new
technologies that use GPUs to enable extreme scaling that is thousands of
times faster and 40 times more affordable than CPU-based solutions.
CONFIDENTIALITY NOTICE: This email and its contents and attachments are
for the sole use of the intended recipient(s) and are confidential or
proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use,
disclosure, dissemination or copying of this email or its contents or
attachments is prohibited. If you have received this communication in
error, please notify the sender by reply email and permanently delete all
copies of the email and its contents and attachments.
On Tue, Mar 22, 2016 at 1:26 PM, Jeremy J Carroll <jj...@sy...> wrote:
> With more testing I found that by repeatedly inserting a few million
> triples to a journal with two or three billion triples, normally the
> journal size does not change, but then when it did grow, it grew by about
> 40G (the journal is about 500G).
> I am wondering where in the code that file growth happens, I suspect that
> that could easily take 5 seconds, and lock out other activity while it is
> happening.
>
> The latency reported in the commitNow function was pretty acceptable at 13051
> nanoseconds
>
> Jeremy
>
>
>
>
> On Mar 21, 2016, at 3:26 PM, Bryan Thompson <br...@sy...> wrote:
>
> Certainly. Also, as i indicated, it may be possible to reduce the lockout
> period. In principle, it seems that flushing the write cache outside of the
> lock would be safe. But this would need a careful code review.
>
> Thanks,
> Bryan
> On Mar 21, 2016 6:24 PM, "Jeremy J Carroll" <jj...@sy...> wrote:
>
>> OK,
>> looking at the implementation of commitNow, I take it to be a fairly
>> low-level piece of code where holding the write lock seems reasonable.
>> The easiest way to reduce the length of time that the write lock is being
>> held is to either reduced the amount of data being written or increased the
>> disk speed.
>>
>> For us to understand the length of time the write lock is written it
>> looks like we should enable Info level on the logger and then we will see
>> commit times and this will give us an idea of how bad the problem is.
>>
>> Then if we cannot reduce the commit time sufficiently we should get back
>> to you.
>>
>> Does that sound like a reasonable plan on this issue?
>>
>> Jeremy
>>
>>
>>
>> On Mar 21, 2016, at 1:27 PM, Jeremy J Carroll <jj...@sy...> wrote:
>>
>> Yes - or blocking some other part of the process ...
>>
>>
>> On Mar 21, 2016, at 12:12 PM, Bryan Thompson <br...@sy...> wrote:
>>
>> Is the thought then that there might be some barrier that is blocking the
>> start of a new query during some part of that large create?
>>
>> ----
>> Bryan Thompson
>> Chief Scientist & Founder
>> Blazegraph
>> e: br...@bl...
>> w: http://blazegraph.com
>>
>> Blazegraph products help to solve the Graph Cache Thrash to achieve large
>> scale processing for graph and predictive analytics. Blazegraph is the
>> creator of the industry’s first GPU-accelerated high-performance database
>> for large graphs, has been named as one of the “10 Companies and
>> Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>.
>>
>>
>> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high
>> performance graph database that supports both RDF/SPARQL and
>> Tinkerpop/Blueprints APIs. Blazegraph GPU
>> <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS
>> <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive
>> new technologies that use GPUs to enable extreme scaling that is thousands
>> of times faster and 40 times more affordable than CPU-based solutions.
>>
>> CONFIDENTIALITY NOTICE: This email and its contents and attachments are
>> for the sole use of the intended recipient(s) and are confidential or
>> proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use,
>> disclosure, dissemination or copying of this email or its contents or
>> attachments is prohibited. If you have received this communication in
>> error, please notify the sender by reply email and permanently delete all
>> copies of the email and its contents and attachments.
>>
>> On Mon, Mar 21, 2016 at 12:40 PM, Jeremy J Carroll <jj...@sy...>
>> wrote:
>>
>>> The logs I have are:
>>> - full GC logs
>>> - python level logs of http requests and errors/warnings (of our
>>> application server: not the blazegraph http requests)
>>>
>>> I can map those logs into SPARQL requests approximately, and I see that
>>> at the point where I got the long wait, there were a single large create
>>> operations that use the
>>> INSERT-by-POST operation taking approximately 30s (including python
>>> processing) which normally is much more than 50% of blazegraph time.
>>>
>>> Very minor other activity going on
>>>
>>> No interesting GC activity
>>>
>>> Jeremy
>>>
>>>
>>>
>>>
>>> On Mar 19, 2016, at 11:57 AM, Bryan Thompson <br...@sy...> wrote:
>>>
>>> Do you have full logs for the server during this interval?
>>> Bryan
>>>
>>> ----
>>> Bryan Thompson
>>> Chief Scientist & Founder
>>> Blazegraph
>>> e: br...@bl...
>>> w: http://blazegraph.com
>>>
>>> Blazegraph products help to solve the Graph Cache Thrash to achieve
>>> large scale processing for graph and predictive analytics. Blazegraph is
>>> the creator of the industry’s first GPU-accelerated high-performance
>>> database for large graphs, has been named as one of the “10 Companies
>>> and Technologies to Watch in 2016”
>>> <http://insideanalysis.com/2016/01/20535/>.
>>>
>>> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high
>>> performance graph database that supports both RDF/SPARQL and
>>> Tinkerpop/Blueprints APIs. Blazegraph GPU
>>> <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS
>>> <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive
>>> new technologies that use GPUs to enable extreme scaling that is thousands
>>> of times faster and 40 times more affordable than CPU-based solutions.
>>>
>>> CONFIDENTIALITY NOTICE: This email and its contents and attachments
>>> are for the sole use of the intended recipient(s) and are confidential or
>>> proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use,
>>> disclosure, dissemination or copying of this email or its contents or
>>> attachments is prohibited. If you have received this communication in
>>> error, please notify the sender by reply email and permanently delete all
>>> copies of the email and its contents and attachments.
>>>
>>> On Fri, Mar 18, 2016 at 6:53 PM, Jeremy J Carroll <jj...@sy...>
>>> wrote:
>>>
>>>> I checked the GC logs. We have no GC stoppage of more than a second for
>>>> days either side of the slow query.
>>>>
>>>> The slow query was on March 6th, there was a 3s stoppage on Feb 29th
>>>> and a 1.8s stoppage on March 9th
>>>>
>>>> Any other thoughts?
>>>>
>>>> Jeremy
>>>>
>>>>
>>>> On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy...> wrote:
>>>>
>>>> 5s is probably a major GC event.
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
|
|
From: Jeremy J C. <jj...@sy...> - 2016-03-22 17:26:17
|
With more testing I found that by repeatedly inserting a few million triples to a journal with two or three billion triples, normally the journal size does not change, but then when it did grow, it grew by about 40G (the journal is about 500G). I am wondering where in the code that file growth happens, I suspect that that could easily take 5 seconds, and lock out other activity while it is happening. The latency reported in the commitNow function was pretty acceptable at 13051 nanoseconds Jeremy > On Mar 21, 2016, at 3:26 PM, Bryan Thompson <br...@sy...> wrote: > > Certainly. Also, as i indicated, it may be possible to reduce the lockout period. In principle, it seems that flushing the write cache outside of the lock would be safe. But this would need a careful code review. > > Thanks, > Bryan > > On Mar 21, 2016 6:24 PM, "Jeremy J Carroll" <jj...@sy... <mailto:jj...@sy...>> wrote: > OK, > looking at the implementation of commitNow, I take it to be a fairly low-level piece of code where holding the write lock seems reasonable. > The easiest way to reduce the length of time that the write lock is being held is to either reduced the amount of data being written or increased the disk speed. > > For us to understand the length of time the write lock is written it looks like we should enable Info level on the logger and then we will see commit times and this will give us an idea of how bad the problem is. > > Then if we cannot reduce the commit time sufficiently we should get back to you. > > Does that sound like a reasonable plan on this issue? > > Jeremy > > > >> On Mar 21, 2016, at 1:27 PM, Jeremy J Carroll <jj...@sy... <mailto:jj...@sy...>> wrote: >> >> Yes - or blocking some other part of the process ... >> >> >>> On Mar 21, 2016, at 12:12 PM, Bryan Thompson <br...@sy... <mailto:br...@sy...>> wrote: >>> >>> Is the thought then that there might be some barrier that is blocking the start of a new query during some part of that large create? >>> >>> ---- >>> Bryan Thompson >>> Chief Scientist & Founder >>> Blazegraph >>> e: br...@bl... <mailto:br...@bl...> >>> w: http://blazegraph.com <http://blazegraph.com/> >>> >>> Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. >>> >>> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. >>> >>> CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. >>> >>> >>> On Mon, Mar 21, 2016 at 12:40 PM, Jeremy J Carroll <jj...@sy... <mailto:jj...@sy...>> wrote: >>> The logs I have are: >>> - full GC logs >>> - python level logs of http requests and errors/warnings (of our application server: not the blazegraph http requests) >>> >>> I can map those logs into SPARQL requests approximately, and I see that at the point where I got the long wait, there were a single large create operations that use the >>> INSERT-by-POST operation taking approximately 30s (including python processing) which normally is much more than 50% of blazegraph time. >>> >>> Very minor other activity going on >>> >>> No interesting GC activity >>> >>> Jeremy >>> >>> >>> >>> >>>> On Mar 19, 2016, at 11:57 AM, Bryan Thompson <br...@sy... <mailto:br...@sy...>> wrote: >>>> >>>> Do you have full logs for the server during this interval? >>>> Bryan >>>> >>>> ---- >>>> Bryan Thompson >>>> Chief Scientist & Founder >>>> Blazegraph >>>> e: br...@bl... <mailto:br...@bl...> >>>> w: http://blazegraph.com <http://blazegraph.com/> >>>> >>>> Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. >>>> >>>> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. >>>> >>>> CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. >>>> >>>> >>>> On Fri, Mar 18, 2016 at 6:53 PM, Jeremy J Carroll <jj...@sy... <mailto:jj...@sy...>> wrote: >>>> I checked the GC logs. We have no GC stoppage of more than a second for days either side of the slow query. >>>> >>>> The slow query was on March 6th, there was a 3s stoppage on Feb 29th and a 1.8s stoppage on March 9th >>>> >>>> Any other thoughts? >>>> >>>> Jeremy >>>> >>>> >>>>> On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy... <mailto:br...@sy...>> wrote: >>>>> >>>>> 5s is probably a major GC event. >>>> >>>> >>> >>> >> > |
|
From: Bryan T. <br...@sy...> - 2016-03-21 22:26:53
|
Certainly. Also, as i indicated, it may be possible to reduce the lockout period. In principle, it seems that flushing the write cache outside of the lock would be safe. But this would need a careful code review. Thanks, Bryan On Mar 21, 2016 6:24 PM, "Jeremy J Carroll" <jj...@sy...> wrote: > OK, > looking at the implementation of commitNow, I take it to be a fairly > low-level piece of code where holding the write lock seems reasonable. > The easiest way to reduce the length of time that the write lock is being > held is to either reduced the amount of data being written or increased the > disk speed. > > For us to understand the length of time the write lock is written it looks > like we should enable Info level on the logger and then we will see commit > times and this will give us an idea of how bad the problem is. > > Then if we cannot reduce the commit time sufficiently we should get back > to you. > > Does that sound like a reasonable plan on this issue? > > Jeremy > > > > On Mar 21, 2016, at 1:27 PM, Jeremy J Carroll <jj...@sy...> wrote: > > Yes - or blocking some other part of the process ... > > > On Mar 21, 2016, at 12:12 PM, Bryan Thompson <br...@sy...> wrote: > > Is the thought then that there might be some barrier that is blocking the > start of a new query during some part of that large create? > > ---- > Bryan Thompson > Chief Scientist & Founder > Blazegraph > e: br...@bl... > w: http://blazegraph.com > > Blazegraph products help to solve the Graph Cache Thrash to achieve large > scale processing for graph and predictive analytics. Blazegraph is the > creator of the industry’s first GPU-accelerated high-performance database > for large graphs, has been named as one of the “10 Companies and > Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. > > > Blazegraph Database <https://www.blazegraph.com/> is our ultra-high > performance graph database that supports both RDF/SPARQL and > Tinkerpop/Blueprints APIs. Blazegraph GPU > <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS > <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new > technologies that use GPUs to enable extreme scaling that is thousands of > times faster and 40 times more affordable than CPU-based solutions. > > CONFIDENTIALITY NOTICE: This email and its contents and attachments are > for the sole use of the intended recipient(s) and are confidential or > proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, > disclosure, dissemination or copying of this email or its contents or > attachments is prohibited. If you have received this communication in > error, please notify the sender by reply email and permanently delete all > copies of the email and its contents and attachments. > > On Mon, Mar 21, 2016 at 12:40 PM, Jeremy J Carroll <jj...@sy...> wrote: > >> The logs I have are: >> - full GC logs >> - python level logs of http requests and errors/warnings (of our >> application server: not the blazegraph http requests) >> >> I can map those logs into SPARQL requests approximately, and I see that >> at the point where I got the long wait, there were a single large create >> operations that use the >> INSERT-by-POST operation taking approximately 30s (including python >> processing) which normally is much more than 50% of blazegraph time. >> >> Very minor other activity going on >> >> No interesting GC activity >> >> Jeremy >> >> >> >> >> On Mar 19, 2016, at 11:57 AM, Bryan Thompson <br...@sy...> wrote: >> >> Do you have full logs for the server during this interval? >> Bryan >> >> ---- >> Bryan Thompson >> Chief Scientist & Founder >> Blazegraph >> e: br...@bl... >> w: http://blazegraph.com >> >> Blazegraph products help to solve the Graph Cache Thrash to achieve large >> scale processing for graph and predictive analytics. Blazegraph is the >> creator of the industry’s first GPU-accelerated high-performance database >> for large graphs, has been named as one of the “10 Companies and >> Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. >> >> >> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high >> performance graph database that supports both RDF/SPARQL and >> Tinkerpop/Blueprints APIs. Blazegraph GPU >> <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS >> <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive >> new technologies that use GPUs to enable extreme scaling that is thousands >> of times faster and 40 times more affordable than CPU-based solutions. >> >> CONFIDENTIALITY NOTICE: This email and its contents and attachments are >> for the sole use of the intended recipient(s) and are confidential or >> proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, >> disclosure, dissemination or copying of this email or its contents or >> attachments is prohibited. If you have received this communication in >> error, please notify the sender by reply email and permanently delete all >> copies of the email and its contents and attachments. >> >> On Fri, Mar 18, 2016 at 6:53 PM, Jeremy J Carroll <jj...@sy...> wrote: >> >>> I checked the GC logs. We have no GC stoppage of more than a second for >>> days either side of the slow query. >>> >>> The slow query was on March 6th, there was a 3s stoppage on Feb 29th and >>> a 1.8s stoppage on March 9th >>> >>> Any other thoughts? >>> >>> Jeremy >>> >>> >>> On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy...> wrote: >>> >>> 5s is probably a major GC event. >>> >>> >>> >> >> > > > |
|
From: Jeremy J C. <jj...@sy...> - 2016-03-21 22:24:23
|
OK, looking at the implementation of commitNow, I take it to be a fairly low-level piece of code where holding the write lock seems reasonable. The easiest way to reduce the length of time that the write lock is being held is to either reduced the amount of data being written or increased the disk speed. For us to understand the length of time the write lock is written it looks like we should enable Info level on the logger and then we will see commit times and this will give us an idea of how bad the problem is. Then if we cannot reduce the commit time sufficiently we should get back to you. Does that sound like a reasonable plan on this issue? Jeremy > On Mar 21, 2016, at 1:27 PM, Jeremy J Carroll <jj...@sy...> wrote: > > Yes - or blocking some other part of the process ... > > >> On Mar 21, 2016, at 12:12 PM, Bryan Thompson <br...@sy... <mailto:br...@sy...>> wrote: >> >> Is the thought then that there might be some barrier that is blocking the start of a new query during some part of that large create? >> >> ---- >> Bryan Thompson >> Chief Scientist & Founder >> Blazegraph >> e: br...@bl... <mailto:br...@bl...> >> w: http://blazegraph.com <http://blazegraph.com/> >> >> Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. >> >> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. >> >> CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. >> >> >> On Mon, Mar 21, 2016 at 12:40 PM, Jeremy J Carroll <jj...@sy... <mailto:jj...@sy...>> wrote: >> The logs I have are: >> - full GC logs >> - python level logs of http requests and errors/warnings (of our application server: not the blazegraph http requests) >> >> I can map those logs into SPARQL requests approximately, and I see that at the point where I got the long wait, there were a single large create operations that use the >> INSERT-by-POST operation taking approximately 30s (including python processing) which normally is much more than 50% of blazegraph time. >> >> Very minor other activity going on >> >> No interesting GC activity >> >> Jeremy >> >> >> >> >>> On Mar 19, 2016, at 11:57 AM, Bryan Thompson <br...@sy... <mailto:br...@sy...>> wrote: >>> >>> Do you have full logs for the server during this interval? >>> Bryan >>> >>> ---- >>> Bryan Thompson >>> Chief Scientist & Founder >>> Blazegraph >>> e: br...@bl... <mailto:br...@bl...> >>> w: http://blazegraph.com <http://blazegraph.com/> >>> >>> Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. >>> >>> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. >>> >>> CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. >>> >>> >>> On Fri, Mar 18, 2016 at 6:53 PM, Jeremy J Carroll <jj...@sy... <mailto:jj...@sy...>> wrote: >>> I checked the GC logs. We have no GC stoppage of more than a second for days either side of the slow query. >>> >>> The slow query was on March 6th, there was a 3s stoppage on Feb 29th and a 1.8s stoppage on March 9th >>> >>> Any other thoughts? >>> >>> Jeremy >>> >>> >>>> On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy... <mailto:br...@sy...>> wrote: >>>> >>>> 5s is probably a major GC event. >>> >>> >> >> > |
|
From: Bryan T. <br...@sy...> - 2016-03-21 20:38:33
|
That is possible. There are some barriers around commit processing that
might prevent new tx starts. So you could have a commit that was taking a
few seconds and during that time new queries might block. In particular,
AbstractJournal.commitNow() takes the following lock.
final WriteLock lock = _fieldReadWriteLock.writeLock();
lock.lock();
That lock is contended by some other code paths. For example,
getIndexLocal() in the same class needs that lock if there is a cache miss.
We do use a read/write lock there, but once commitNow() gets the write lock
no other thread will be able to proceed.
It is possible that we could defer acquiring that lock. The slowest part of
the commit is flushing the indices to the disk. If we take a lock that is
specific to the commit, and then wait until we have already flushed the
indices to the disk, we might be able to reduce this latency. Of course,
such changes raise the possibility of lock ordering problems which could
lead to a deadlock. So we would have to take a pretty close look at this
before making the change.
Thanks,
Bryan
----
Bryan Thompson
Chief Scientist & Founder
Blazegraph
e: br...@bl...
w: http://blazegraph.com
Blazegraph products help to solve the Graph Cache Thrash to achieve large
scale processing for graph and predictive analytics. Blazegraph is the
creator of the industry’s first GPU-accelerated high-performance database
for large graphs, has been named as one of the “10 Companies and
Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>.
Blazegraph Database <https://www.blazegraph.com/> is our ultra-high
performance graph database that supports both RDF/SPARQL and
Tinkerpop/Blueprints APIs. Blazegraph GPU
<https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS
<https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new
technologies that use GPUs to enable extreme scaling that is thousands of
times faster and 40 times more affordable than CPU-based solutions.
CONFIDENTIALITY NOTICE: This email and its contents and attachments are
for the sole use of the intended recipient(s) and are confidential or
proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use,
disclosure, dissemination or copying of this email or its contents or
attachments is prohibited. If you have received this communication in
error, please notify the sender by reply email and permanently delete all
copies of the email and its contents and attachments.
On Mon, Mar 21, 2016 at 4:27 PM, Jeremy J Carroll <jj...@sy...> wrote:
> Yes - or blocking some other part of the process ...
>
>
> On Mar 21, 2016, at 12:12 PM, Bryan Thompson <br...@sy...> wrote:
>
> Is the thought then that there might be some barrier that is blocking the
> start of a new query during some part of that large create?
>
> ----
> Bryan Thompson
> Chief Scientist & Founder
> Blazegraph
> e: br...@bl...
> w: http://blazegraph.com
>
> Blazegraph products help to solve the Graph Cache Thrash to achieve large
> scale processing for graph and predictive analytics. Blazegraph is the
> creator of the industry’s first GPU-accelerated high-performance database
> for large graphs, has been named as one of the “10 Companies and
> Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>.
>
>
> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high
> performance graph database that supports both RDF/SPARQL and
> Tinkerpop/Blueprints APIs. Blazegraph GPU
> <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS
> <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new
> technologies that use GPUs to enable extreme scaling that is thousands of
> times faster and 40 times more affordable than CPU-based solutions.
>
> CONFIDENTIALITY NOTICE: This email and its contents and attachments are
> for the sole use of the intended recipient(s) and are confidential or
> proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use,
> disclosure, dissemination or copying of this email or its contents or
> attachments is prohibited. If you have received this communication in
> error, please notify the sender by reply email and permanently delete all
> copies of the email and its contents and attachments.
>
> On Mon, Mar 21, 2016 at 12:40 PM, Jeremy J Carroll <jj...@sy...> wrote:
>
>> The logs I have are:
>> - full GC logs
>> - python level logs of http requests and errors/warnings (of our
>> application server: not the blazegraph http requests)
>>
>> I can map those logs into SPARQL requests approximately, and I see that
>> at the point where I got the long wait, there were a single large create
>> operations that use the
>> INSERT-by-POST operation taking approximately 30s (including python
>> processing) which normally is much more than 50% of blazegraph time.
>>
>> Very minor other activity going on
>>
>> No interesting GC activity
>>
>> Jeremy
>>
>>
>>
>>
>> On Mar 19, 2016, at 11:57 AM, Bryan Thompson <br...@sy...> wrote:
>>
>> Do you have full logs for the server during this interval?
>> Bryan
>>
>> ----
>> Bryan Thompson
>> Chief Scientist & Founder
>> Blazegraph
>> e: br...@bl...
>> w: http://blazegraph.com
>>
>> Blazegraph products help to solve the Graph Cache Thrash to achieve large
>> scale processing for graph and predictive analytics. Blazegraph is the
>> creator of the industry’s first GPU-accelerated high-performance database
>> for large graphs, has been named as one of the “10 Companies and
>> Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>.
>>
>>
>> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high
>> performance graph database that supports both RDF/SPARQL and
>> Tinkerpop/Blueprints APIs. Blazegraph GPU
>> <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS
>> <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive
>> new technologies that use GPUs to enable extreme scaling that is thousands
>> of times faster and 40 times more affordable than CPU-based solutions.
>>
>> CONFIDENTIALITY NOTICE: This email and its contents and attachments are
>> for the sole use of the intended recipient(s) and are confidential or
>> proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use,
>> disclosure, dissemination or copying of this email or its contents or
>> attachments is prohibited. If you have received this communication in
>> error, please notify the sender by reply email and permanently delete all
>> copies of the email and its contents and attachments.
>>
>> On Fri, Mar 18, 2016 at 6:53 PM, Jeremy J Carroll <jj...@sy...> wrote:
>>
>>> I checked the GC logs. We have no GC stoppage of more than a second for
>>> days either side of the slow query.
>>>
>>> The slow query was on March 6th, there was a 3s stoppage on Feb 29th and
>>> a 1.8s stoppage on March 9th
>>>
>>> Any other thoughts?
>>>
>>> Jeremy
>>>
>>>
>>> On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy...> wrote:
>>>
>>> 5s is probably a major GC event.
>>>
>>>
>>>
>>
>>
>
>
|
|
From: Jeremy J C. <jj...@sy...> - 2016-03-21 20:27:40
|
Yes - or blocking some other part of the process ... > On Mar 21, 2016, at 12:12 PM, Bryan Thompson <br...@sy...> wrote: > > Is the thought then that there might be some barrier that is blocking the start of a new query during some part of that large create? > > ---- > Bryan Thompson > Chief Scientist & Founder > Blazegraph > e: br...@bl... <mailto:br...@bl...> > w: http://blazegraph.com <http://blazegraph.com/> > > Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. > > Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. > > CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. > > > On Mon, Mar 21, 2016 at 12:40 PM, Jeremy J Carroll <jj...@sy... <mailto:jj...@sy...>> wrote: > The logs I have are: > - full GC logs > - python level logs of http requests and errors/warnings (of our application server: not the blazegraph http requests) > > I can map those logs into SPARQL requests approximately, and I see that at the point where I got the long wait, there were a single large create operations that use the > INSERT-by-POST operation taking approximately 30s (including python processing) which normally is much more than 50% of blazegraph time. > > Very minor other activity going on > > No interesting GC activity > > Jeremy > > > > >> On Mar 19, 2016, at 11:57 AM, Bryan Thompson <br...@sy... <mailto:br...@sy...>> wrote: >> >> Do you have full logs for the server during this interval? >> Bryan >> >> ---- >> Bryan Thompson >> Chief Scientist & Founder >> Blazegraph >> e: br...@bl... <mailto:br...@bl...> >> w: http://blazegraph.com <http://blazegraph.com/> >> >> Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. >> >> Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. >> >> CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. >> >> >> On Fri, Mar 18, 2016 at 6:53 PM, Jeremy J Carroll <jj...@sy... <mailto:jj...@sy...>> wrote: >> I checked the GC logs. We have no GC stoppage of more than a second for days either side of the slow query. >> >> The slow query was on March 6th, there was a 3s stoppage on Feb 29th and a 1.8s stoppage on March 9th >> >> Any other thoughts? >> >> Jeremy >> >> >>> On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy... <mailto:br...@sy...>> wrote: >>> >>> 5s is probably a major GC event. >> >> > > |
|
From: Bryan T. <br...@sy...> - 2016-03-21 19:12:13
|
Is the thought then that there might be some barrier that is blocking the start of a new query during some part of that large create? ---- Bryan Thompson Chief Scientist & Founder Blazegraph e: br...@bl... w: http://blazegraph.com Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. On Mon, Mar 21, 2016 at 12:40 PM, Jeremy J Carroll <jj...@sy...> wrote: > The logs I have are: > - full GC logs > - python level logs of http requests and errors/warnings (of our > application server: not the blazegraph http requests) > > I can map those logs into SPARQL requests approximately, and I see that at > the point where I got the long wait, there were a single large create > operations that use the > INSERT-by-POST operation taking approximately 30s (including python > processing) which normally is much more than 50% of blazegraph time. > > Very minor other activity going on > > No interesting GC activity > > Jeremy > > > > > On Mar 19, 2016, at 11:57 AM, Bryan Thompson <br...@sy...> wrote: > > Do you have full logs for the server during this interval? > Bryan > > ---- > Bryan Thompson > Chief Scientist & Founder > Blazegraph > e: br...@bl... > w: http://blazegraph.com > > Blazegraph products help to solve the Graph Cache Thrash to achieve large > scale processing for graph and predictive analytics. Blazegraph is the > creator of the industry’s first GPU-accelerated high-performance database > for large graphs, has been named as one of the “10 Companies and > Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. > > > Blazegraph Database <https://www.blazegraph.com/> is our ultra-high > performance graph database that supports both RDF/SPARQL and > Tinkerpop/Blueprints APIs. Blazegraph GPU > <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS > <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new > technologies that use GPUs to enable extreme scaling that is thousands of > times faster and 40 times more affordable than CPU-based solutions. > > CONFIDENTIALITY NOTICE: This email and its contents and attachments are > for the sole use of the intended recipient(s) and are confidential or > proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, > disclosure, dissemination or copying of this email or its contents or > attachments is prohibited. If you have received this communication in > error, please notify the sender by reply email and permanently delete all > copies of the email and its contents and attachments. > > On Fri, Mar 18, 2016 at 6:53 PM, Jeremy J Carroll <jj...@sy...> wrote: > >> I checked the GC logs. We have no GC stoppage of more than a second for >> days either side of the slow query. >> >> The slow query was on March 6th, there was a 3s stoppage on Feb 29th and >> a 1.8s stoppage on March 9th >> >> Any other thoughts? >> >> Jeremy >> >> >> On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy...> wrote: >> >> 5s is probably a major GC event. >> >> >> > > |
|
From: Jeremy J C. <jj...@sy...> - 2016-03-21 16:41:01
|
The logs I have are: - full GC logs - python level logs of http requests and errors/warnings (of our application server: not the blazegraph http requests) I can map those logs into SPARQL requests approximately, and I see that at the point where I got the long wait, there were a single large create operations that use the INSERT-by-POST operation taking approximately 30s (including python processing) which normally is much more than 50% of blazegraph time. Very minor other activity going on No interesting GC activity Jeremy > On Mar 19, 2016, at 11:57 AM, Bryan Thompson <br...@sy...> wrote: > > Do you have full logs for the server during this interval? > Bryan > > ---- > Bryan Thompson > Chief Scientist & Founder > Blazegraph > e: br...@bl... <mailto:br...@bl...> > w: http://blazegraph.com <http://blazegraph.com/> > > Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. > > Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. > > CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. > > > On Fri, Mar 18, 2016 at 6:53 PM, Jeremy J Carroll <jj...@sy... <mailto:jj...@sy...>> wrote: > I checked the GC logs. We have no GC stoppage of more than a second for days either side of the slow query. > > The slow query was on March 6th, there was a 3s stoppage on Feb 29th and a 1.8s stoppage on March 9th > > Any other thoughts? > > Jeremy > > >> On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy... <mailto:br...@sy...>> wrote: >> >> 5s is probably a major GC event. > > |
|
From: Bryan T. <br...@sy...> - 2016-03-19 18:58:08
|
Do you have full logs for the server during this interval? Bryan ---- Bryan Thompson Chief Scientist & Founder Blazegraph e: br...@bl... w: http://blazegraph.com Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. On Fri, Mar 18, 2016 at 6:53 PM, Jeremy J Carroll <jj...@sy...> wrote: > I checked the GC logs. We have no GC stoppage of more than a second for > days either side of the slow query. > > The slow query was on March 6th, there was a 3s stoppage on Feb 29th and a > 1.8s stoppage on March 9th > > Any other thoughts? > > Jeremy > > > On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy...> wrote: > > 5s is probably a major GC event. > > > |
|
From: Jeremy J C. <jj...@sy...> - 2016-03-18 22:53:52
|
I checked the GC logs. We have no GC stoppage of more than a second for days either side of the slow query. The slow query was on March 6th, there was a 3s stoppage on Feb 29th and a 1.8s stoppage on March 9th Any other thoughts? Jeremy > On Mar 17, 2016, at 2:54 PM, Bryan Thompson <br...@sy...> wrote: > > 5s is probably a major GC event. |
|
From: Bryan T. <br...@sy...> - 2016-03-17 21:55:06
|
You might try the EST_CARD interface for a lightweight request. It will return a range count without the overhead of parsing a SPARQL request. ---- Bryan Thompson Chief Scientist & Founder Blazegraph e: br...@bl... w: http://blazegraph.com Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. On Thu, Mar 17, 2016 at 5:51 PM, Jeremy J Carroll <jj...@sy...> wrote: > > As part of our health check that ensures we know when our system go down, > we send the following query to all our blazegraph instances every minute: > > ASK > FROM <http://does-not-exist.example.org/an/empty/named/graph> > FROM NAMED <http://does-not-exist.example.org/an/empty/named/graph> > > WHERE { > > } > > > Normally this is very quick! > > However, during a period of heavy write activity using INSERT-by-POST > commands that seem to be taking around 30s (this is from logs one level > out, so there is room for doubt), we were seeing these queries occasionally > take over 5s (the trivial ASK). > > Is this expected? > > My general belief was that writes and concurrent reads did not strongly > conflict in blazegraph. > > > Jeremy > > > > > ------------------------------------------------------------------------------ > Transform Data into Opportunity. > Accelerate data analysis in your applications with > Intel Data Analytics Acceleration Library. > Click to learn more. > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > _______________________________________________ > Bigdata-developers mailing list > Big...@li... > https://lists.sourceforge.net/lists/listinfo/bigdata-developers > |
|
From: Bryan T. <br...@sy...> - 2016-03-17 21:54:12
|
5s is probably a major GC event. ---- Bryan Thompson Chief Scientist & Founder Blazegraph e: br...@bl... w: http://blazegraph.com Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. On Thu, Mar 17, 2016 at 5:51 PM, Jeremy J Carroll <jj...@sy...> wrote: > > As part of our health check that ensures we know when our system go down, > we send the following query to all our blazegraph instances every minute: > > ASK > FROM <http://does-not-exist.example.org/an/empty/named/graph> > FROM NAMED <http://does-not-exist.example.org/an/empty/named/graph> > > WHERE { > > } > > > Normally this is very quick! > > However, during a period of heavy write activity using INSERT-by-POST > commands that seem to be taking around 30s (this is from logs one level > out, so there is room for doubt), we were seeing these queries occasionally > take over 5s (the trivial ASK). > > Is this expected? > > My general belief was that writes and concurrent reads did not strongly > conflict in blazegraph. > > > Jeremy > > > > > ------------------------------------------------------------------------------ > Transform Data into Opportunity. > Accelerate data analysis in your applications with > Intel Data Analytics Acceleration Library. > Click to learn more. > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > _______________________________________________ > Bigdata-developers mailing list > Big...@li... > https://lists.sourceforge.net/lists/listinfo/bigdata-developers > |
|
From: Jeremy J C. <jj...@sy...> - 2016-03-17 21:51:31
|
As part of our health check that ensures we know when our system go down, we send the following query to all our blazegraph instances every minute: ASK FROM <http://does-not-exist.example.org/an/empty/named/graph> FROM NAMED <http://does-not-exist.example.org/an/empty/named/graph> WHERE { } Normally this is very quick! However, during a period of heavy write activity using INSERT-by-POST commands that seem to be taking around 30s (this is from logs one level out, so there is room for doubt), we were seeing these queries occasionally take over 5s (the trivial ASK). Is this expected? My general belief was that writes and concurrent reads did not strongly conflict in blazegraph. Jeremy |
|
From: Brad B. <be...@bl...> - 2016-03-16 21:17:26
|
Edgar, Thank you. Is there any chance that the journal was open for writing while it was being copied? Thanks, --Brad On Wed, Mar 16, 2016 at 4:36 PM, Edgar Rodriguez-Diaz <ed...@sy...> wrote: > Hi Brad, > > > On Mar 16, 2016, at 11:05 AM, Brad Bebee <be...@bl...> wrote: > > > > Edgar, > > > > Can you provide a few more details on the version and steps used that > produced the error? > > This happened using Blazegraph 1.5.3 on a journal file with ~3B triples. > We usually store and manage journal files in AWS switching volume > snapshots as we require. > > Cheers, > Edgar -- _______________ Brad Bebee CEO Blazegraph e: be...@bl... m: 202.642.7961 w: www.blazegraph.com Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. |
|
From: Edgar Rodriguez-D. <ed...@sy...> - 2016-03-16 20:36:35
|
Hi Brad, > On Mar 16, 2016, at 11:05 AM, Brad Bebee <be...@bl...> wrote: > > Edgar, > > Can you provide a few more details on the version and steps used that produced the error? This happened using Blazegraph 1.5.3 on a journal file with ~3B triples. We usually store and manage journal files in AWS switching volume snapshots as we require. Cheers, Edgar |
|
From: Brad B. <be...@bl...> - 2016-03-16 18:06:04
|
Edgar, Can you provide a few more details on the version and steps used that produced the error? Thanks, --Brad On Wed, Mar 16, 2016 at 1:55 PM, Edgar Rodriguez-Diaz <ed...@sy...> wrote: > Hello, > > We are hitting the following error when starting blazegraph: > > Mar 15,2016 13:21:38 PDT - FATAL: 1046 main > com.bigdata.rdf.sail.webapp.NanoSparqlServer.awaitServerStart(NanoSparqlServer.java:505): > Server did not start. > Mar 15,2016 13:21:38 PDT - ERROR: 1063 main > com.bigdata.Banner$1.uncaughtException(Banner.java:112): Uncaught exception > in thread > java.lang.Error: Two allocators at same address > at > com.bigdata.rwstore.FixedAllocator.compareTo(FixedAllocator.java:106) > at java.util.ComparableTimSort.mergeLo(ComparableTimSort.java:684) > at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:481) > at > java.util.ComparableTimSort.mergeCollapse(ComparableTimSort.java:406) > at java.util.ComparableTimSort.sort(ComparableTimSort.java:213) > at java.util.Arrays.sort(Arrays.java:1312) > at java.util.Arrays.sort(Arrays.java:1506) > at java.util.ArrayList.sort(ArrayList.java:1454) > at java.util.Collections.sort(Collections.java:141) > at > com.bigdata.rwstore.RWStore.readAllocationBlocks(RWStore.java:1682) > at com.bigdata.rwstore.RWStore.initfromRootBlock(RWStore.java:1557) > at com.bigdata.rwstore.RWStore.<init>(RWStore.java:969) > at com.bigdata.journal.RWStrategy.<init>(RWStrategy.java:137) > at > com.bigdata.journal.AbstractJournal.<init>(AbstractJournal.java:1265) > at com.bigdata.journal.Journal.<init>(Journal.java:275) > at com.bigdata.journal.Journal.<init>(Journal.java:268) > at > com.bigdata.rdf.sail.webapp.BigdataRDFServletContextListener.openIndexManager(BigdataRDFServletContextListener.java:798) > at > com.bigdata.rdf.sail.webapp.BigdataRDFServletContextListener.contextInitialized(BigdataRDFServletContextListener.java:276) > at > org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:798) > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) > at > org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:789) > at > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) > at > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1341) > at > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1334) > at > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) > at > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:497) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) > at org.eclipse.jetty.server.Server.start(Server.java:387) > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) > at > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) > at org.eclipse.jetty.server.Server.doStart(Server.java:354) > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > com.bigdata.rdf.sail.webapp.NanoSparqlServer.awaitServerStart(NanoSparqlServer.java:485) > at > com.bigdata.rdf.sail.webapp.NanoSparqlServer.main(NanoSparqlServer.java:449) > > I was wondering if this could be related to > https://jira.blazegraph.com/browse/BLZG-1128 or if there may be any other > issue producing this error. > Any thoughts? > > Cheers, > Edgar > > > ------------------------------------------------------------------------------ > Transform Data into Opportunity. > Accelerate data analysis in your applications with > Intel Data Analytics Acceleration Library. > Click to learn more. > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > _______________________________________________ > Bigdata-developers mailing list > Big...@li... > https://lists.sourceforge.net/lists/listinfo/bigdata-developers > > -- _______________ Brad Bebee CEO Blazegraph e: be...@bl... m: 202.642.7961 w: www.blazegraph.com Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. |
|
From: Edgar Rodriguez-D. <ed...@sy...> - 2016-03-16 17:56:00
|
Hello,
We are hitting the following error when starting blazegraph:
Mar 15,2016 13:21:38 PDT - FATAL: 1046 main com.bigdata.rdf.sail.webapp.NanoSparqlServer.awaitServerStart(NanoSparqlServer.java:505): Server did not start.
Mar 15,2016 13:21:38 PDT - ERROR: 1063 main com.bigdata.Banner$1.uncaughtException(Banner.java:112): Uncaught exception in thread
java.lang.Error: Two allocators at same address
at com.bigdata.rwstore.FixedAllocator.compareTo(FixedAllocator.java:106)
at java.util.ComparableTimSort.mergeLo(ComparableTimSort.java:684)
at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:481)
at java.util.ComparableTimSort.mergeCollapse(ComparableTimSort.java:406)
at java.util.ComparableTimSort.sort(ComparableTimSort.java:213)
at java.util.Arrays.sort(Arrays.java:1312)
at java.util.Arrays.sort(Arrays.java:1506)
at java.util.ArrayList.sort(ArrayList.java:1454)
at java.util.Collections.sort(Collections.java:141)
at com.bigdata.rwstore.RWStore.readAllocationBlocks(RWStore.java:1682)
at com.bigdata.rwstore.RWStore.initfromRootBlock(RWStore.java:1557)
at com.bigdata.rwstore.RWStore.<init>(RWStore.java:969)
at com.bigdata.journal.RWStrategy.<init>(RWStrategy.java:137)
at com.bigdata.journal.AbstractJournal.<init>(AbstractJournal.java:1265)
at com.bigdata.journal.Journal.<init>(Journal.java:275)
at com.bigdata.journal.Journal.<init>(Journal.java:268)
at com.bigdata.rdf.sail.webapp.BigdataRDFServletContextListener.openIndexManager(BigdataRDFServletContextListener.java:798)
at com.bigdata.rdf.sail.webapp.BigdataRDFServletContextListener.contextInitialized(BigdataRDFServletContextListener.java:276)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:798)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:789)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1341)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1334)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:497)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:387)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:354)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.bigdata.rdf.sail.webapp.NanoSparqlServer.awaitServerStart(NanoSparqlServer.java:485)
at com.bigdata.rdf.sail.webapp.NanoSparqlServer.main(NanoSparqlServer.java:449)
I was wondering if this could be related to https://jira.blazegraph.com/browse/BLZG-1128 <https://jira.blazegraph.com/browse/BLZG-1128> or if there may be any other issue producing this error.
Any thoughts?
Cheers,
Edgar |
|
From: Bryan T. <br...@sy...> - 2016-03-08 19:02:05
|
We moved to jetty for the blazegraph java client due to bugs in the apache http client. This could be one of them. Suggest using our client or using jetty directly. Bryan ---- Bryan Thompson Chief Scientist & Founder Blazegraph e: br...@bl... w: http://blazegraph.com Blazegraph products help to solve the Graph Cache Thrash to achieve large scale processing for graph and predictive analytics. Blazegraph is the creator of the industry’s first GPU-accelerated high-performance database for large graphs, has been named as one of the “10 Companies and Technologies to Watch in 2016” <http://insideanalysis.com/2016/01/20535/>. Blazegraph Database <https://www.blazegraph.com/> is our ultra-high performance graph database that supports both RDF/SPARQL and Tinkerpop/Blueprints APIs. Blazegraph GPU <https://www.blazegraph.com/product/gpu-accelerated/> andBlazegraph DAS <https://www.blazegraph.com/product/gpu-accelerated/>L are disruptive new technologies that use GPUs to enable extreme scaling that is thousands of times faster and 40 times more affordable than CPU-based solutions. CONFIDENTIALITY NOTICE: This email and its contents and attachments are for the sole use of the intended recipient(s) and are confidential or proprietary to SYSTAP, LLC DBA Blazegraph. Any unauthorized review, use, disclosure, dissemination or copying of this email or its contents or attachments is prohibited. If you have received this communication in error, please notify the sender by reply email and permanently delete all copies of the email and its contents and attachments. On Tue, Mar 8, 2016 at 12:24 PM, Joakim Soderberg < joa...@bl...> wrote: > Hi, > I have a client running org.openrdf.repository.sparql.SPARQLRepository > After some thousands queries I get the following error: > > - *** error closing:org.openrdf.http.client.BackgroundTupleResult@70beb599 > org.openrdf.query.QueryEvaluationException: > org.apache.http.MalformedChunkCodingException: CRLF expected at end of > chunk > at org.openrdf.http.client.BackgroundTupleResult.handleClose( > BackgroundTupleResult.java:79) > at info.aduna.iteration.AbstractCloseableIteration.close( > AbstractCloseableIteration.java:60) > at bwIdEntityClassPopulate.main(bwIdEntityClassPopulate.java:167) > Caused by: org.apache.http.MalformedChunkCodingException: CRLF expected > at end of chunk > at org.apache.http.impl.io.ChunkedInputStream.getChunkSize( > ChunkedInputStream.java:255) > at org.apache.http.impl.io.ChunkedInputStream.nextChunk( > ChunkedInputStream.java:227) > at org.apache.http.impl.io.ChunkedInputStream.read( > ChunkedInputStream.java:186) > at org.apache.http.impl.io.ChunkedInputStream.read( > ChunkedInputStream.java:215) > at org.apache.http.impl.io.ChunkedInputStream.close( > ChunkedInputStream.java:316) > at org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed( > ResponseEntityProxy.java:128) > at org.apache.http.conn.EofSensorInputStream.checkClose( > EofSensorInputStream.java:228) > at org.apache.http.conn.EofSensorInputStream.close( > EofSensorInputStream.java:174) > at org.openrdf.http.client.BackgroundTupleResult.handleClose( > BackgroundTupleResult.java:75) > ... 2 more > > > > And the server is running WDQS: > > blazegraph-service-0.2.0-SNAPSHOT-dist.war > data/ > docs/ > gui/ > jetty-runner-9.2.9.v20150224.jar > jolokia.sh* > lib/ > loadData.sh* > munge.sh* > nohup.out > rules.log > runBlazegraph.sh* > runBlazegraph.sh~* > #runUpdate.sh#* > runUpdate.sh* > runUpdate.sh~* > > I’m not sure if the problem originates from the server or the client? > > > > ------------------------------------------------------------------------------ > Transform Data into Opportunity. > Accelerate data analysis in your applications with > Intel Data Analytics Acceleration Library. > Click to learn more. > http://makebettercode.com/inteldaal-eval > _______________________________________________ > Bigdata-developers mailing list > Big...@li... > https://lists.sourceforge.net/lists/listinfo/bigdata-developers > > |
|
From: Joakim S. <joa...@bl...> - 2016-03-08 17:24:12
|
Hi, I have a client running org.openrdf.repository.sparql.SPARQLRepository After some thousands queries I get the following error: - *** error closing:org.openrdf.http.client.BackgroundTupleResult@70beb599 org.openrdf.query.QueryEvaluationException: org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk at org.openrdf.http.client.BackgroundTupleResult.handleClose(BackgroundTupleResult.java:79) at info.aduna.iteration.AbstractCloseableIteration.close(AbstractCloseableIteration.java:60) at bwIdEntityClassPopulate.main(bwIdEntityClassPopulate.java:167) Caused by: org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255) at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:227) at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186) at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:215) at org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:316) at org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed(ResponseEntityProxy.java:128) at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228) at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:174) at org.openrdf.http.client.BackgroundTupleResult.handleClose(BackgroundTupleResult.java:75) ... 2 more And the server is running WDQS: blazegraph-service-0.2.0-SNAPSHOT-dist.war data/ docs/ gui/ jetty-runner-9.2.9.v20150224.jar jolokia.sh* lib/ loadData.sh* munge.sh* nohup.out rules.log runBlazegraph.sh* runBlazegraph.sh~* #runUpdate.sh#* runUpdate.sh* runUpdate.sh~* I’m not sure if the problem originates from the server or the client? |
|
From: Michael S. <ms...@me...> - 2016-03-03 18:34:05
|
Jeremy, thanks for reporting. I’ve created https://jira.blazegraph.com/browse/BLZG-1792 <https://jira.blazegraph.com/browse/BLZG-1792> and fixed it according to your proposal. As I understand this is not a critical issue though, can’t imagine any scenarios where this one would have any effect (the code is only relevant for single statement pattern queries, so the cardinalities won’t be used). Best, Michael > On 29 Feb 2016, at 23:26, Jeremy J Carroll <jj...@sy...> wrote: > > bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/optimizers/ASTDistinctTermScanOptimizer.java > > line 381 (in master) reads: > final long newCard = (long) (1.0 / arity); > > the comment above says: > > newCard = oldCard * 1.0 / arity(context, sp) > > I suspect the comment is correct and the code is wrong. > > (I have no idea what code this is: I had a case of an obviously incorrect estimate cardinality and was looking for an int/long bug, and found this instead) > > > Jeremy > > > > ------------------------------------------------------------------------------ > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________ > Bigdata-developers mailing list > Big...@li... > https://lists.sourceforge.net/lists/listinfo/bigdata-developers |
27 messages has been excluded from this view by a project administrator.
