From: Sherman M. <sdm...@gm...> - 2017-12-01 16:32:43
|
After reviewing the dbe_storage_s struct and the code in disk.c, it is apparent that disk.c is the *.db file manager of Virtuoso, and this file manager is sufficiently sophisticated as to elicit an alternative to extracting a programming interface from disk.c. So my preliminary proposal is to leave the disk.c code alone and instead make virtuoso.db a virtual file such that reading from the file actually reads from the stdout of a command and writing to the file actually writes to the stdin of a command. Then the commands in question will interact with the persistent store of choice, including the eos.io smart contract running on the blockchain. Some additional requirements are: - Low level-of-effort, so implementing a custom file system (e.g. fuse <http://fuse.sourceforge.net/>) is out of the question - Must be cross platform, so e.g. linux fifo special files <http://www.gnu.org/software/libc/manual/html_node/FIFO-Special-Files.html> are out of the question - Preserves performance, i.e. the solution cannot adversely impact Virtuoso's current performance metrics After some preliminary research, writing to a named pipe <https://en.wikipedia.org/wiki/Named_pipe> looks very promising. The downside is that you can't have concurrent processes reading/listening to the file, but AFAIK only one process is accessing virtuoso.db at any given time. Once the content is read its gone so disk.c will need to treat it like a buffered stream (e.g. the writer will continuously push the content to the buffer, and the reader will loop the buffered content indefinitely). The command writing to the buffer is what I'll call an adaptor. It's job is to persist the state of the "virtual file" based on what it receives from Virtuoso and to fulfill Virtuoso's reads. This way, the adaptor is agnostic to the contents of virtuoso.db and its logic only needs to handle the details of the persistence provider. What are your thoughts? I'm sure I've overlooked requirements you envision, if so please share those also. On Thu, Nov 30, 2017 at 2:34 PM, Sherman Monroe <sdm...@gm...> wrote: > Correction, dbe_storage_s.dbs_file > > On Thu, Nov 30, 2017 at 1:43 PM, Sherman Monroe <sdm...@gm...> > wrote: > >> I've found dbe_storage_s and dbe_storage_s->dbs_file. Design proposal en >> route... >> >> On Thu, Nov 30, 2017 at 8:24 AM, Sherman Monroe <sdm...@gm...> >> wrote: >> >>> Ok, I will review the codebase and begin the design. I'll post for >>> peer review before I begin coding. >>> >>> On Wed, Nov 29, 2017 at 8:22 PM, Kingsley Idehen <ki...@op... >>> > wrote: >>> >>>> On 11/29/17 6:47 PM, Sherman Monroe wrote: >>>> >>>> So as a starting point, I abstract every single call to .db and also >>>> any dependent functions dealing with in-memory state. Next I implement the >>>> old "file backed" persistence using the new API, and run test cases to >>>> ensure nothing broke. Next I test it on some common alternative types like >>>> the ones you list above. Finally I implement the eos.io backed >>>> persistence. Does this sum up the requirements? >>>> >>>> A 2K feet, yes. >>>> >>>> >>>> Kingsley >>>> >>>> >>>> On Wed, Nov 29, 2017 at 5:40 PM, Sherman Monroe <sdm...@gm...> >>>> wrote: >>>> >>>>> Excellent, that's what I hoped. >>>>> >>>>> On Wed, Nov 29, 2017 at 5:38 PM, Kingsley Idehen < >>>>> ki...@op...> wrote: >>>>> >>>>>> On 11/29/17 6:35 PM, Sherman Monroe wrote: >>>>>> >>>>>> Is Virtuoso loading the contents of "xyz.db" into an internal model, >>>>>> or is it performing transactions against "xyz.db"? >>>>>> >>>>>> >>>>>> Persistence happens against "xyz.db". Other relations related >>>>>> activity happens in memory. >>>>>> >>>>>> Kingsley >>>>>> >>>>>> >>>>>> On Wed, Nov 29, 2017 at 5:18 PM, Sherman Monroe <sdm...@gm...> >>>>>> wrote: >>>>>> >>>>>>> Hi Kingsley, yes I would like to take this on and then implement a >>>>>>> smart contract-based persistence with it. Can you suggest starting points >>>>>>> or what work has been done so far? >>>>>>> >>>>>>> On Wed, Nov 29, 2017 at 4:33 PM, Kingsley Idehen < >>>>>>> ki...@op...> wrote: >>>>>>> >>>>>>>> On 11/29/17 5:09 PM, Sherman Monroe wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I would like to know what is the feasibility of implementing >>>>>>>> Virtuoso's "state" one a blockchain like eos.io? If for instance >>>>>>>> Virt is storing its quad-store state as a linked list, then if I wanted to >>>>>>>> swap that datatype out for a custom, EOSLinkedList, how difficult would >>>>>>>> that be? Does Virtuoso support a swappable persistence API of any sort, or >>>>>>>> interfaces for the data types it uses for persistence? >>>>>>>> >>>>>>>> >>>>>>>> We are looking at loose-coupling of persistent storage i.e., rather >>>>>>>> than being pegged to "xyz.db" you end up being able to use a variety of >>>>>>>> document types (CSV, ORC, Parquet etc..). >>>>>>>> >>>>>>>> Naturally, anyone can pick up this effort to accelerate matters :) >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> >>>>>>>> Kingsley Idehen >>>>>>>> Founder & CEO >>>>>>>> OpenLink Software (Home Page: http://www.openlinksw.com) >>>>>>>> >>>>>>>> Weblogs (Blogs): >>>>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ >>>>>>>> Blogspot Blog: http://kidehen.blogspot.com >>>>>>>> Medium Blog: https://medium.com/@kidehen >>>>>>>> >>>>>>>> Profile Pages: >>>>>>>> Pinterest: https://www.pinterest.com/kidehen/ >>>>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen >>>>>>>> Twitter: https://twitter.com/kidehen >>>>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about >>>>>>>> LinkedIn: http://www.linkedin.com/in/kidehen >>>>>>>> >>>>>>>> Web Identities (WebID): >>>>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i >>>>>>>> : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this >>>>>>>> >>>>>>>> >>>>>>>> ------------------------------------------------------------ >>>>>>>> ------------------ >>>>>>>> Check out the vibrant tech community on one of the world's most >>>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>>>>>>> _______________________________________________ >>>>>>>> Virtuoso-devel mailing list >>>>>>>> Vir...@li... >>>>>>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-devel >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks, >>>>>>> -sherman >>>>>>> >>>>>>> Every good gift and every perfect gift is from above, and cometh >>>>>>> down from the Father of lights, with whom is no variableness, neither >>>>>>> shadow of turning. >>>>>>> (James 1:17) >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks, >>>>>> -sherman >>>>>> >>>>>> Every good gift and every perfect gift is from above, and cometh down >>>>>> from the Father of lights, with whom is no variableness, neither shadow of >>>>>> turning. >>>>>> (James 1:17) >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> >>>>>> Kingsley Idehen >>>>>> Founder & CEO >>>>>> OpenLink Software (Home Page: http://www.openlinksw.com) >>>>>> >>>>>> Weblogs (Blogs): >>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ >>>>>> Blogspot Blog: http://kidehen.blogspot.com >>>>>> Medium Blog: https://medium.com/@kidehen >>>>>> >>>>>> Profile Pages: >>>>>> Pinterest: https://www.pinterest.com/kidehen/ >>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen >>>>>> Twitter: https://twitter.com/kidehen >>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about >>>>>> LinkedIn: http://www.linkedin.com/in/kidehen >>>>>> >>>>>> Web Identities (WebID): >>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i >>>>>> : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks, >>>>> -sherman >>>>> >>>>> Every good gift and every perfect gift is from above, and cometh down >>>>> from the Father of lights, with whom is no variableness, neither shadow of >>>>> turning. >>>>> (James 1:17) >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks, >>>> -sherman >>>> >>>> Every good gift and every perfect gift is from above, and cometh down >>>> from the Father of lights, with whom is no variableness, neither shadow of >>>> turning. >>>> (James 1:17) >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Kingsley Idehen >>>> Founder & CEO >>>> OpenLink Software (Home Page: http://www.openlinksw.com) >>>> >>>> Weblogs (Blogs): >>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ >>>> Blogspot Blog: http://kidehen.blogspot.com >>>> Medium Blog: https://medium.com/@kidehen >>>> >>>> Profile Pages: >>>> Pinterest: https://www.pinterest.com/kidehen/ >>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen >>>> Twitter: https://twitter.com/kidehen >>>> Google+: https://plus.google.com/+KingsleyIdehen/about >>>> LinkedIn: http://www.linkedin.com/in/kidehen >>>> >>>> Web Identities (WebID): >>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i >>>> : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this >>>> >>>> >>> >>> >>> -- >>> >>> Thanks, >>> -sherman >>> >>> Every good gift and every perfect gift is from above, and cometh down >>> from the Father of lights, with whom is no variableness, neither shadow of >>> turning. >>> (James 1:17) >>> >> >> >> >> -- >> >> Thanks, >> -sherman >> >> Every good gift and every perfect gift is from above, and cometh down >> from the Father of lights, with whom is no variableness, neither shadow of >> turning. >> (James 1:17) >> > > > > -- > > Thanks, > -sherman > > Every good gift and every perfect gift is from above, and cometh down from > the Father of lights, with whom is no variableness, neither shadow of > turning. > (James 1:17) > -- Thanks, -sherman Every good gift and every perfect gift is from above, and cometh down from the Father of lights, with whom is no variableness, neither shadow of turning. (James 1:17) |