From: Sherman M. <sdm...@gm...> - 2017-12-02 00:04:47
|
Okay. On Fri, Dec 1, 2017 at 3:45 PM, Kingsley Idehen <ki...@op...> wrote: > On 12/1/17 1:22 PM, Sherman Monroe wrote: > > As an example of how a disk write will work for the eos.io adaptor, the > buffer used by disk.c will pass the write onset and the total_bytes to > the eos blockchain and this transaction (and nothing more) will be recorded > on the blockchain, respecting its index in the sequence of i/o > transactions. One a read request, the buffer will pass the read onset and > length to eos. To derive the current state of the file inside that char > window, it smart contract will do the equivalent of running the transaction > history from beginning to end (this will happens using the blockchain's > built-in parallel processing), but as it runs each transaction/state > transition, it will only consider (i.e. read and record) the bytes that > changed within the requested window. It will then return the state/content > in the window back to the adaptor which passes the content back to the > buffer. > > In eos.io, only the messages (the inputs, which in this case are the file > write/delete/insert requests) are stored on the chain, not the file itself > (the output). The state of the state machine at any given moment is > therefore only implicit (in the message/transaction history). With > knowledge of the message history and a description of the state transition > logic, one can derive the state on demand. I propose using the state > machine (the blockchain) to model a file of infinite length. > > The adaptor/virtual file interface allows different approaches to be used > to take advantage of the characteristics of each persistence strategy, so a > different approach can be used for Hadoop, etc. > > > Why don't you move this thread to the github repo for VOS. You can open an > issue and dialog can be built around that. > > > Kingsley > > On Fri, Dec 1, 2017 at 11:55 AM, Sherman Monroe <sdm...@gm...> > wrote: > >> fifo special files >> <http://www.gnu.org/software/libc/manual/html_node/FIFO-Special-Files.html> is >> a named pipe, I must have missed that. >> >> On Fri, Dec 1, 2017 at 10:32 AM, Sherman Monroe <sdm...@gm...> >> wrote: >> >>> After reviewing the dbe_storage_s struct and the code in disk.c, it is >>> apparent that disk.c is the *.db file manager of Virtuoso, and this >>> file manager is sufficiently sophisticated as to elicit an alternative to >>> extracting a programming interface from disk.c. So my preliminary >>> proposal is to leave the disk.c code alone and instead make virtuoso.db >>> a virtual file such that reading from the file actually reads from the >>> stdout of a command and writing to the file actually writes to the stdin >>> of a command. Then the commands in question will interact with the >>> persistent store of choice, including the eos.io smart contract running >>> on the blockchain. Some additional requirements are: >>> >>> - Low level-of-effort, so implementing a custom file system (e.g. fuse >>> <http://fuse.sourceforge.net/>) is out of the question >>> - Must be cross platform, so e.g. linux fifo special files >>> <http://www.gnu.org/software/libc/manual/html_node/FIFO-Special-Files.html> >>> are out of the question >>> - Preserves performance, i.e. the solution cannot adversely impact >>> Virtuoso's current performance metrics >>> >>> After some preliminary research, writing to a named pipe >>> <https://en.wikipedia.org/wiki/Named_pipe> looks very promising. The >>> downside is that you can't have concurrent processes reading/listening to >>> the file, but AFAIK only one process is accessing virtuoso.db at any >>> given time. Once the content is read its gone so disk.c will need to >>> treat it like a buffered stream (e.g. the writer will continuously push the >>> content to the buffer, and the reader will loop the buffered content >>> indefinitely). >>> >>> The command writing to the buffer is what I'll call an adaptor. It's job >>> is to persist the state of the "virtual file" based on what it receives >>> from Virtuoso and to fulfill Virtuoso's reads. This way, the adaptor is >>> agnostic to the contents of virtuoso.db and its logic only needs to >>> handle the details of the persistence provider. >>> >>> What are your thoughts? I'm sure I've overlooked requirements you >>> envision, if so please share those also. >>> >>> >>> On Thu, Nov 30, 2017 at 2:34 PM, Sherman Monroe <sdm...@gm...> >>> wrote: >>> >>>> Correction, dbe_storage_s.dbs_file >>>> >>>> On Thu, Nov 30, 2017 at 1:43 PM, Sherman Monroe <sdm...@gm...> >>>> wrote: >>>> >>>>> I've found dbe_storage_s and dbe_storage_s->dbs_file. Design proposal >>>>> en route... >>>>> >>>>> On Thu, Nov 30, 2017 at 8:24 AM, Sherman Monroe <sdm...@gm...> >>>>> wrote: >>>>> >>>>>> Ok, I will review the codebase and begin the design. I'll post for >>>>>> peer review before I begin coding. >>>>>> >>>>>> On Wed, Nov 29, 2017 at 8:22 PM, Kingsley Idehen < >>>>>> ki...@op...> wrote: >>>>>> >>>>>>> On 11/29/17 6:47 PM, Sherman Monroe wrote: >>>>>>> >>>>>>> So as a starting point, I abstract every single call to .db and also >>>>>>> any dependent functions dealing with in-memory state. Next I implement the >>>>>>> old "file backed" persistence using the new API, and run test cases to >>>>>>> ensure nothing broke. Next I test it on some common alternative types like >>>>>>> the ones you list above. Finally I implement the eos.io backed >>>>>>> persistence. Does this sum up the requirements? >>>>>>> >>>>>>> A 2K feet, yes. >>>>>>> >>>>>>> >>>>>>> Kingsley >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 29, 2017 at 5:40 PM, Sherman Monroe <sdm...@gm...> >>>>>>> wrote: >>>>>>> >>>>>>>> Excellent, that's what I hoped. >>>>>>>> >>>>>>>> On Wed, Nov 29, 2017 at 5:38 PM, Kingsley Idehen < >>>>>>>> ki...@op...> wrote: >>>>>>>> >>>>>>>>> On 11/29/17 6:35 PM, Sherman Monroe wrote: >>>>>>>>> >>>>>>>>> Is Virtuoso loading the contents of "xyz.db" into an internal >>>>>>>>> model, or is it performing transactions against "xyz.db"? >>>>>>>>> >>>>>>>>> >>>>>>>>> Persistence happens against "xyz.db". Other relations related >>>>>>>>> activity happens in memory. >>>>>>>>> >>>>>>>>> Kingsley >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Nov 29, 2017 at 5:18 PM, Sherman Monroe < >>>>>>>>> sdm...@gm...> wrote: >>>>>>>>> >>>>>>>>>> Hi Kingsley, yes I would like to take this on and then implement >>>>>>>>>> a smart contract-based persistence with it. Can you suggest starting >>>>>>>>>> points or what work has been done so far? >>>>>>>>>> >>>>>>>>>> On Wed, Nov 29, 2017 at 4:33 PM, Kingsley Idehen < >>>>>>>>>> ki...@op...> wrote: >>>>>>>>>> >>>>>>>>>>> On 11/29/17 5:09 PM, Sherman Monroe wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I would like to know what is the feasibility of implementing >>>>>>>>>>> Virtuoso's "state" one a blockchain like eos.io? If for >>>>>>>>>>> instance Virt is storing its quad-store state as a linked list, then if I >>>>>>>>>>> wanted to swap that datatype out for a custom, EOSLinkedList, how difficult >>>>>>>>>>> would that be? Does Virtuoso support a swappable persistence API of any >>>>>>>>>>> sort, or interfaces for the data types it uses for persistence? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> We are looking at loose-coupling of persistent storage i.e., >>>>>>>>>>> rather than being pegged to "xyz.db" you end up being able to use a variety >>>>>>>>>>> of document types (CSV, ORC, Parquet etc..). >>>>>>>>>>> >>>>>>>>>>> Naturally, anyone can pick up this effort to accelerate matters >>>>>>>>>>> :) >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Kingsley Idehen >>>>>>>>>>> Founder & CEO >>>>>>>>>>> OpenLink Software (Home Page: http://www.openlinksw.com) >>>>>>>>>>> >>>>>>>>>>> Weblogs (Blogs): >>>>>>>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ >>>>>>>>>>> Blogspot Blog: http://kidehen.blogspot.com >>>>>>>>>>> Medium Blog: https://medium.com/@kidehen >>>>>>>>>>> >>>>>>>>>>> Profile Pages: >>>>>>>>>>> Pinterest: https://www.pinterest.com/kidehen/ >>>>>>>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen >>>>>>>>>>> Twitter: https://twitter.com/kidehen >>>>>>>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about >>>>>>>>>>> LinkedIn: http://www.linkedin.com/in/kidehen >>>>>>>>>>> >>>>>>>>>>> Web Identities (WebID): >>>>>>>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i >>>>>>>>>>> : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>> ------------------ >>>>>>>>>>> Check out the vibrant tech community on one of the world's most >>>>>>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Virtuoso-devel mailing list >>>>>>>>>>> Vir...@li... >>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-devel >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> -sherman >>>>>>>>>> >>>>>>>>>> Every good gift and every perfect gift is from above, and cometh >>>>>>>>>> down from the Father of lights, with whom is no variableness, neither >>>>>>>>>> shadow of turning. >>>>>>>>>> (James 1:17) >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> -sherman >>>>>>>>> >>>>>>>>> Every good gift and every perfect gift is from above, and cometh >>>>>>>>> down from the Father of lights, with whom is no variableness, neither >>>>>>>>> shadow of turning. >>>>>>>>> (James 1:17) >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Kingsley Idehen >>>>>>>>> Founder & CEO >>>>>>>>> OpenLink Software (Home Page: http://www.openlinksw.com) >>>>>>>>> >>>>>>>>> Weblogs (Blogs): >>>>>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ >>>>>>>>> Blogspot Blog: http://kidehen.blogspot.com >>>>>>>>> Medium Blog: https://medium.com/@kidehen >>>>>>>>> >>>>>>>>> Profile Pages: >>>>>>>>> Pinterest: https://www.pinterest.com/kidehen/ >>>>>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen >>>>>>>>> Twitter: https://twitter.com/kidehen >>>>>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about >>>>>>>>> LinkedIn: http://www.linkedin.com/in/kidehen >>>>>>>>> >>>>>>>>> Web Identities (WebID): >>>>>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i >>>>>>>>> : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks, >>>>>>>> -sherman >>>>>>>> >>>>>>>> Every good gift and every perfect gift is from above, and cometh >>>>>>>> down from the Father of lights, with whom is no variableness, neither >>>>>>>> shadow of turning. >>>>>>>> (James 1:17) >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks, >>>>>>> -sherman >>>>>>> >>>>>>> Every good gift and every perfect gift is from above, and cometh >>>>>>> down from the Father of lights, with whom is no variableness, neither >>>>>>> shadow of turning. >>>>>>> (James 1:17) >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> >>>>>>> Kingsley Idehen >>>>>>> Founder & CEO >>>>>>> OpenLink Software (Home Page: http://www.openlinksw.com) >>>>>>> >>>>>>> Weblogs (Blogs): >>>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ >>>>>>> Blogspot Blog: http://kidehen.blogspot.com >>>>>>> Medium Blog: https://medium.com/@kidehen >>>>>>> >>>>>>> Profile Pages: >>>>>>> Pinterest: https://www.pinterest.com/kidehen/ >>>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen >>>>>>> Twitter: https://twitter.com/kidehen >>>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about >>>>>>> LinkedIn: http://www.linkedin.com/in/kidehen >>>>>>> >>>>>>> Web Identities (WebID): >>>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i >>>>>>> : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks, >>>>>> -sherman >>>>>> >>>>>> Every good gift and every perfect gift is from above, and cometh down >>>>>> from the Father of lights, with whom is no variableness, neither shadow of >>>>>> turning. >>>>>> (James 1:17) >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks, >>>>> -sherman >>>>> >>>>> Every good gift and every perfect gift is from above, and cometh down >>>>> from the Father of lights, with whom is no variableness, neither shadow of >>>>> turning. >>>>> (James 1:17) >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks, >>>> -sherman >>>> >>>> Every good gift and every perfect gift is from above, and cometh down >>>> from the Father of lights, with whom is no variableness, neither shadow of >>>> turning. >>>> (James 1:17) >>>> >>> >>> >>> >>> -- >>> >>> Thanks, >>> -sherman >>> >>> Every good gift and every perfect gift is from above, and cometh down >>> from the Father of lights, with whom is no variableness, neither shadow of >>> turning. >>> (James 1:17) >>> >> >> >> >> -- >> >> Thanks, >> -sherman >> >> Every good gift and every perfect gift is from above, and cometh down >> from the Father of lights, with whom is no variableness, neither shadow of >> turning. >> (James 1:17) >> > > > > -- > > Thanks, > -sherman > > Every good gift and every perfect gift is from above, and cometh down from > the Father of lights, with whom is no variableness, neither shadow of > turning. > (James 1:17) > > > -- > Regards, > > Kingsley Idehen > Founder & CEO > OpenLink Software (Home Page: http://www.openlinksw.com) > > Weblogs (Blogs): > Legacy Blog: http://www.openlinksw.com/blog/~kidehen/ > Blogspot Blog: http://kidehen.blogspot.com > Medium Blog: https://medium.com/@kidehen > > Profile Pages: > Pinterest: https://www.pinterest.com/kidehen/ > Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen > Twitter: https://twitter.com/kidehen > Google+: https://plus.google.com/+KingsleyIdehen/about > LinkedIn: http://www.linkedin.com/in/kidehen > > Web Identities (WebID): > Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i > : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this > > -- Thanks, -sherman Every good gift and every perfect gift is from above, and cometh down from the Father of lights, with whom is no variableness, neither shadow of turning. (James 1:17) |