Re: [dotNetRDF-Develop] Hybrid dataset -- In-memory/SQL

[Rob V. <rv...@do...>]

Simon

This is entirely possible though not well documented, your best bet would be
to look at the source for BrightstarDB
(https://github.com/BrightstarDB/BrightstarDB/tree/develop/src/core/Brightst
arDB/Query) which takes this approach.  A couple of their developers are on
this list/dotNetRDF committers so they may hopefully chime in with advice

The only obvious consequence of your proposed approach is that you might get
quite different performance characteristics for queries depending on whether
a query stays in memory or goes against the SQL database.  Provided your
ISparqlDataset implementation just returns the loaded Triple objects the
rest of the SPARQL engine shouldn't have any problems.

The other thing to be aware of is that the current SPARQL engine is memory
bound in that it does all operations in-memory often materialising large
amounts of data.  Therefore even if you personally aren't keeping all your
data in-memory dotNetRDF may need to pull lots of stuff into memory in order
to successfully evaluate queries.  Long term this will be changed but
realistically it will be a minimum of 3 months before early Alphas of this
functionality are available and 6-12 before it is production ready.

Rob

From:  Simon Ouellette <sou...@he...>
Reply-To:  dotNetRDF Developer Discussion and Feature Request
<dot...@li...>
Date:  Tuesday, 15 July 2014 16:38
To:  "dot...@li..."
<dot...@li...>
Subject:  [dotNetRDF-Develop] Hybrid dataset -- In-memory/SQL

>   
> Hi,  
>    
> We’re interested in potentially using dotNetRdf against a kind of “hybrid
> dataset” where nodes and object properties would be in-memory, as in the
> InMemoryDataset, however there would be the special case of scalar properties
> (a.k.a. data properties), which would be contained in a SQL table (our
> database is too big to hold everything in RAM, but holding everything on disk
> would be too slow).
>    
> I’ve looked a bit at the source code to see how this could be done, and so far
> my understanding is that I could derive my own Dataset class, and implement my
> own GetTriplesWithPredicate, GetTriplesWithSubjectPredicate and
> GetTriplesWithPredicateObject methods? When the predicate exists in memory, I
> would go through the usual InMemoryDataset approach, and when it isn’t I would
> run a SQL query against my database. I’m not sure if I’m on the right path, or
> if there are potentially breaking consequences to what I’m trying to do…
>    
> Any guidance on how to best accomplish this would be appreciated.
>    
> Thanks,  
> Simon Ouellette  
>  <gfidisc.helix-sys.com>
> CONFIDENTIALITY NOTICE
> This message contains confidential information intended only for the use of
> the individual or entity named as recipient. Any dissemination, distribution
> or copying of this communication by anyone other than the intended recipient
> is strictly prohibited. If you have received this message in error, please
> immediately notify us and delete your copy.
> 
> Thank you.
> 
> AVIS DE CONFIDENTIALITÉ
> Les informations contenues aux présentes sont de nature privilégiée et
> confidentielle. Elles ne peuvent être utilisées que par la personne ou
> l'entité dont le nom paraît comme destinataire. Si le lecteur du présent
> message n'est pas le destinataire prévu, il est par les présentes prié de
> noter qu'il est strictement interdit de divulguer, de distribuer ou de copier
> ce message. Si ce message vous a été transmis par mégarde, veuillez nous en
> aviser immédiatement et supprimer votre copie.
> 
> Merci.
> 
>  
>  <gfidisc.helix-sys.com>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your enterprise? Index and search
> up to 200,000 lines of code with a free copy of Black Duck Code Sight - the
> same software that powers the world's largest code search on Ohloh, the Black
> Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds_______________________________________________
> dotNetRDF-develop mailing list dot...@li...
> https://lists.sourceforge.net/lists/listinfo/dotnetrdf-develop