[Bigdata-developers] New scale-out architecture (100x => 10, 000x speedup)

[Bryan T. <br...@sy...>]

*We will be introducing a new scale-out **architecture offering speedups of
100x to 10000x.  *

*This will be the fastest graph database processing platform anywhere.*

Today, there are four main deployment models for bigdata. The last of these
(the "bigdata federation") will be replaced by our new scale-out platform.

1. embedded
2. single server
3. highly available replication cluster
4. horizontally scaled database (aka "bigdata federation") <== this will be
replaced.

*We will continue to actively support and develop the following versions of
the bigdata platform:*

1. embedded
2. single server
3. highly available replication cluster

*Upcoming features for these platforms include:*

- Support for openrdf 2.7 (this month)
- Improved query optimization
- Column-wise on the page
- Faster data load times
- Less data footprint on the disk.

*Why MapGraph?*

Our experience with MapGraph (http://mapgraph.io) has shown us how to
create a new horizontally scaled platform that is 100x faster on CPUs and
10,000x faster on GPUs than the existing scale-out architecture. Therefore,
we will be rolling out a new horizontally scaled database platform next
year based on MapGraph and supporting both CPUs and GPUs for outrageous
performance.  If you want a preview, checkout our recent paper at IEEE Big
Data.

*What was wrong with the existing scale-out architecture?*

The existing horizontally scaled architecture (aka the "bigdata
federation") is based on dynamic sharding and was inspired by the Google
bigtable architecture. The bigdata federation has several key innovations
that go beyond the Google bigtable architecture and which provide
significantly better performance than existing attempts to layer RDF/SPARQL
over key-value store.  For example, bigdata makes it possible to map the
query over the data.  This results in significantly less data read than
other approaches such as RYA or CumulousRDF, etc.

However, some aspects of the bigdata federation architecture have limited
how quickly we can evolve the bigdata platform.  In particular bigtable
popularized the notion of a key-value store where the key is an unsigned
byte[] and the value is a byte[].  However, modern high performance
database design uses column-wise (or structure of arrays) layouts in order
to minimize the memory bandwidth and CPU decode overhead associated with
index operations.  Dropping the bigdata federation architecture will allow
us to quickly introduce column-wise storage and new query optimization
techniques and greatly simplify the maintenance of the query engine.


*Look for more news soon.*

Thanks,
Bryan
----
Bryan Thompson
Chief Scientist & Founder
SYSTAP, LLC
4501 Tower Road
Greensboro, NC 27410
br...@sy...
http://bigdata.com
http://mapgraph.io

CONFIDENTIALITY NOTICE:  This email and its contents and attachments are
for the sole use of the intended recipient(s) and are confidential or
proprietary to SYSTAP. Any unauthorized review, use, disclosure,
dissemination or copying of this email or its contents or attachments is
prohibited. If you have received this communication in error, please notify
the sender by reply email and permanently delete all copies of the email
and its contents and attachments.