From: Sherman M. <sdm...@gm...> - 2019-01-03 16:16:18
|
Hi, I would like to retract this issue <https://github.com/openlink/virtuoso-opensource/issues/698>, but I was unable to delete it since I don't own the repository. The approach turned out to be nonviable. Thanks, -sherman On Fri, Dec 1, 2017 at 6:04 PM Sherman Monroe <sdm...@gm...> wrote: > Okay. > > On Fri, Dec 1, 2017 at 3:45 PM, Kingsley Idehen <ki...@op...> > wrote: > >> On 12/1/17 1:22 PM, Sherman Monroe wrote: >> >> As an example of how a disk write will work for the eos.io adaptor, the >> buffer used by disk.c will pass the write onset and the total_bytes to >> the eos blockchain and this transaction (and nothing more) will be recorded >> on the blockchain, respecting its index in the sequence of i/o >> transactions. One a read request, the buffer will pass the read onset >> and length to eos. To derive the current state of the file inside that >> char window, it smart contract will do the equivalent of running the >> transaction history from beginning to end (this will happens using the >> blockchain's built-in parallel processing), but as it runs each >> transaction/state transition, it will only consider (i.e. read and record) >> the bytes that changed within the requested window. It will then return the >> state/content in the window back to the adaptor which passes the content >> back to the buffer. >> >> In eos.io, only the messages (the inputs, which in this case are the >> file write/delete/insert requests) are stored on the chain, not the file >> itself (the output). The state of the state machine at any given moment is >> therefore only implicit (in the message/transaction history). With >> knowledge of the message history and a description of the state transition >> logic, one can derive the state on demand. I propose using the state >> machine (the blockchain) to model a file of infinite length. >> >> The adaptor/virtual file interface allows different approaches to be used >> to take advantage of the characteristics of each persistence strategy, so a >> different approach can be used for Hadoop, etc. >> >> >> Why don't you move this thread to the github repo for VOS. You can open >> an issue and dialog can be built around that. >> >> >> Kingsley >> >> On Fri, Dec 1, 2017 at 11:55 AM, Sherman Monroe <sdm...@gm...> >> wrote: >> >>> fifo special files >>> <http://www.gnu.org/software/libc/manual/html_node/FIFO-Special-Files.html> is >>> a named pipe, I must have missed that. >>> >>> On Fri, Dec 1, 2017 at 10:32 AM, Sherman Monroe <sdm...@gm...> >>> wrote: >>> >>>> After reviewing the dbe_storage_s struct and the code in disk.c, it is >>>> apparent that disk.c is the *.db file manager of Virtuoso, and this >>>> file manager is sufficiently sophisticated as to elicit an alternative to >>>> extracting a programming interface from disk.c. So my preliminary >>>> proposal is to leave the disk.c code alone and instead make virtuoso.db >>>> a virtual file such that reading from the file actually reads from the >>>> stdout of a command and writing to the file actually writes to the >>>> stdin of a command. Then the commands in question will interact with >>>> the persistent store of choice, including the eos.io smart contract >>>> running on the blockchain. Some additional requirements are: >>>> >>>> - Low level-of-effort, so implementing a custom file system (e.g. fuse >>>> <http://fuse.sourceforge.net/>) is out of the question >>>> - Must be cross platform, so e.g. linux fifo special files >>>> <http://www.gnu.org/software/libc/manual/html_node/FIFO-Special-Files.html> >>>> are out of the question >>>> - Preserves performance, i.e. the solution cannot adversely impact >>>> Virtuoso's current performance metrics >>>> >>>> After some preliminary research, writing to a named pipe >>>> <https://en.wikipedia.org/wiki/Named_pipe> looks very promising. The >>>> downside is that you can't have concurrent processes reading/listening to >>>> the file, but AFAIK only one process is accessing virtuoso.db at any >>>> given time. Once the content is read its gone so disk.c will need to >>>> treat it like a buffered stream (e.g. the writer will continuously push the >>>> content to the buffer, and the reader will loop the buffered content >>>> indefinitely). >>>> >>>> The command writing to the buffer is what I'll call an adaptor. It's >>>> job is to persist the state of the "virtual file" based on what it receives >>>> from Virtuoso and to fulfill Virtuoso's reads. This way, the adaptor is >>>> agnostic to the contents of virtuoso.db and its logic only needs to >>>> handle the details of the persistence provider. >>>> >>>> What are your thoughts? I'm sure I've overlooked requirements you >>>> envision, if so please share those also. >>>> >>>> >>>> On Thu, Nov 30, 2017 at 2:34 PM, Sherman Monroe <sdm...@gm...> >>>> wrote: >>>> >>>>> Correction, dbe_storage_s.dbs_file >>>>> >>>>> On Thu, Nov 30, 2017 at 1:43 PM, Sherman Monroe <sdm...@gm...> >>>>> wrote: >>>>> >>>>>> I've found dbe_storage_s and dbe_storage_s->dbs_file. Design >>>>>> proposal en route... >>>>>> >>>>>> On Thu, Nov 30, 2017 at 8:24 AM, Sherman Monroe <sdm...@gm...> >>>>>> wrote: >>>>>> >>>>>>> Ok, I will review the codebase and begin the design. I'll post for >>>>>>> peer review before I begin coding. >>>>>>> >>>>>>> On Wed, Nov 29, 2017 at 8:22 PM, Kingsley Idehen < >>>>>>> ki...@op...> wrote: >>>>>>> >>>>>>>> On 11/29/17 6:47 PM, Sherman Monroe wrote: >>>>>>>> >>>>>>>> So as a starting point, I abstract every single call to .db and >>>>>>>> also any dependent functions dealing with in-memory state. Next I implement >>>>>>>> the old "file backed" persistence using the new API, and run test cases to >>>>>>>> ensure nothing broke. Next I test it on some common alternative types like >>>>>>>> the ones you list above. Finally I implement the eos.io backed >>>>>>>> persistence. Does this sum up the requirements? >>>>>>>> >>>>>>>> A 2K feet, yes. >>>>>>>> >>>>>>>> >>>>>>>> Kingsley >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 29, 2017 at 5:40 PM, Sherman Monroe <sdm...@gm... >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Excellent, that's what I hoped. >>>>>>>>> >>>>>>>>> On Wed, Nov 29, 2017 at 5:38 PM, Kingsley Idehen < >>>>>>>>> ki...@op...> wrote: >>>>>>>>> >>>>>>>>>> On 11/29/17 6:35 PM, Sherman Monroe wrote: >>>>>>>>>> >>>>>>>>>> Is Virtuoso loading the contents of "xyz.db" into an internal >>>>>>>>>> model, or is it performing transactions against "xyz.db"? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Persistence happens against "xyz.db". Other relations related >>>>>>>>>> activity happens in memory. >>>>>>>>>> >>>>>>>>>> Kingsley >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Nov 29, 2017 at 5:18 PM, Sherman Monroe < >>>>>>>>>> sdm...@gm...> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Kingsley, yes I would like to take this on and then >>>>>>>>>>> implement a smart contract-based persistence with it. Can you suggest >>>>>>>>>>> starting points or what work has been done so far? >>>>>>>>>>> >>>>>>>>>>> On Wed, Nov 29, 2017 at 4:33 PM, Kingsley Idehen < >>>>>>>>>>> ki...@op...> wrote: >>>>>>>>>>> >>>>>>>>>>>> On 11/29/17 5:09 PM, Sherman Monroe wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I would like to know what is the feasibility of implementing >>>>>>>>>>>> Virtuoso's "state" one a blockchain like eos.io? If for >>>>>>>>>>>> instance Virt is storing its quad-store state as a linked list, then if I >>>>>>>>>>>> wanted to swap that datatype out for a custom, EOSLinkedList, how difficult >>>>>>>>>>>> would that be? Does Virtuoso support a swappable persistence API of any >>>>>>>>>>>> sort, or interfaces for the data types it uses for persistence? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We are looking at loose-coupling of persistent storage i.e., >>>>>>>>>>>> rather than being pegged to "xyz.db" you end up being able to use a variety >>>>>>>>>>>> of document types (CSV, ORC, Parquet etc..). >>>>>>>>>>>> >>>>>>>>>>>> Naturally, anyone can pick up this effort to accelerate matters >>>>>>>>>>>> :) >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Regards, >>>>>>>>>>>> >>>>>>>>>>>> Kingsley Idehen >>>>>>>>>>>> Founder & CEO >>>>>>>>>>>> OpenLink Software (Home Page: http://www.openlinksw.com) >>>>>>>>>>>> >>>>>>>>>>>> Weblogs (Blogs): >>>>>>>>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ >>>>>>>>>>>> Blogspot Blog: http://kidehen.blogspot.com >>>>>>>>>>>> Medium Blog: https://medium.com/@kidehen >>>>>>>>>>>> >>>>>>>>>>>> Profile Pages: >>>>>>>>>>>> Pinterest: https://www.pinterest.com/kidehen/ >>>>>>>>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen >>>>>>>>>>>> Twitter: https://twitter.com/kidehen >>>>>>>>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about >>>>>>>>>>>> LinkedIn: http://www.linkedin.com/in/kidehen >>>>>>>>>>>> >>>>>>>>>>>> Web Identities (WebID): >>>>>>>>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i >>>>>>>>>>>> : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ------------------------------------------------------------------------------ >>>>>>>>>>>> Check out the vibrant tech community on one of the world's most >>>>>>>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Virtuoso-devel mailing list >>>>>>>>>>>> Vir...@li... >>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-devel >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> -sherman >>>>>>>>>>> >>>>>>>>>>> Every good gift and every perfect gift is from above, and cometh >>>>>>>>>>> down from the Father of lights, with whom is no variableness, neither >>>>>>>>>>> shadow of turning. >>>>>>>>>>> (James 1:17) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> -sherman >>>>>>>>>> >>>>>>>>>> Every good gift and every perfect gift is from above, and cometh >>>>>>>>>> down from the Father of lights, with whom is no variableness, neither >>>>>>>>>> shadow of turning. >>>>>>>>>> (James 1:17) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Kingsley Idehen >>>>>>>>>> Founder & CEO >>>>>>>>>> OpenLink Software (Home Page: http://www.openlinksw.com) >>>>>>>>>> >>>>>>>>>> Weblogs (Blogs): >>>>>>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ >>>>>>>>>> Blogspot Blog: http://kidehen.blogspot.com >>>>>>>>>> Medium Blog: https://medium.com/@kidehen >>>>>>>>>> >>>>>>>>>> Profile Pages: >>>>>>>>>> Pinterest: https://www.pinterest.com/kidehen/ >>>>>>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen >>>>>>>>>> Twitter: https://twitter.com/kidehen >>>>>>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about >>>>>>>>>> LinkedIn: http://www.linkedin.com/in/kidehen >>>>>>>>>> >>>>>>>>>> Web Identities (WebID): >>>>>>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i >>>>>>>>>> : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> -sherman >>>>>>>>> >>>>>>>>> Every good gift and every perfect gift is from above, and cometh >>>>>>>>> down from the Father of lights, with whom is no variableness, neither >>>>>>>>> shadow of turning. >>>>>>>>> (James 1:17) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks, >>>>>>>> -sherman >>>>>>>> >>>>>>>> Every good gift and every perfect gift is from above, and cometh >>>>>>>> down from the Father of lights, with whom is no variableness, neither >>>>>>>> shadow of turning. >>>>>>>> (James 1:17) >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> >>>>>>>> Kingsley Idehen >>>>>>>> Founder & CEO >>>>>>>> OpenLink Software (Home Page: http://www.openlinksw.com) >>>>>>>> >>>>>>>> Weblogs (Blogs): >>>>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ >>>>>>>> Blogspot Blog: http://kidehen.blogspot.com >>>>>>>> Medium Blog: https://medium.com/@kidehen >>>>>>>> >>>>>>>> Profile Pages: >>>>>>>> Pinterest: https://www.pinterest.com/kidehen/ >>>>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen >>>>>>>> Twitter: https://twitter.com/kidehen >>>>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about >>>>>>>> LinkedIn: http://www.linkedin.com/in/kidehen >>>>>>>> >>>>>>>> Web Identities (WebID): >>>>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i >>>>>>>> : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks, >>>>>>> -sherman >>>>>>> >>>>>>> Every good gift and every perfect gift is from above, and cometh >>>>>>> down from the Father of lights, with whom is no variableness, neither >>>>>>> shadow of turning. >>>>>>> (James 1:17) >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks, >>>>>> -sherman >>>>>> >>>>>> Every good gift and every perfect gift is from above, and cometh down >>>>>> from the Father of lights, with whom is no variableness, neither shadow of >>>>>> turning. >>>>>> (James 1:17) >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks, >>>>> -sherman >>>>> >>>>> Every good gift and every perfect gift is from above, and cometh down >>>>> from the Father of lights, with whom is no variableness, neither shadow of >>>>> turning. >>>>> (James 1:17) >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks, >>>> -sherman >>>> >>>> Every good gift and every perfect gift is from above, and cometh down >>>> from the Father of lights, with whom is no variableness, neither shadow of >>>> turning. >>>> (James 1:17) >>>> >>> >>> >>> >>> -- >>> >>> Thanks, >>> -sherman >>> >>> Every good gift and every perfect gift is from above, and cometh down >>> from the Father of lights, with whom is no variableness, neither shadow of >>> turning. >>> (James 1:17) >>> >> >> >> >> -- >> >> Thanks, >> -sherman >> >> Every good gift and every perfect gift is from above, and cometh down >> from the Father of lights, with whom is no variableness, neither shadow of >> turning. >> (James 1:17) >> >> >> -- >> Regards, >> >> Kingsley Idehen >> Founder & CEO >> OpenLink Software (Home Page: http://www.openlinksw.com) >> >> Weblogs (Blogs): >> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ >> Blogspot Blog: http://kidehen.blogspot.com >> Medium Blog: https://medium.com/@kidehen >> >> Profile Pages: >> Pinterest: https://www.pinterest.com/kidehen/ >> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen >> Twitter: https://twitter.com/kidehen >> Google+: https://plus.google.com/+KingsleyIdehen/about >> LinkedIn: http://www.linkedin.com/in/kidehen >> >> Web Identities (WebID): >> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i >> : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this >> >> > > > -- > > Thanks, > -sherman > > Every good gift and every perfect gift is from above, and cometh down from > the Father of lights, with whom is no variableness, neither shadow of > turning. > (James 1:17) > -- Thanks, -sherman Every good gift and every perfect gift is from above, and cometh down from the Father of lights, with whom is no variableness, neither shadow of turning. (James 1:17) |