Re: [Virtuoso-devel] Blockchain-backed "SAIL" for Virt

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

 fifo special files
<http://www.gnu.org/software/libc/manual/html_node/FIFO-Special-Files.html> is
a named pipe, I must have missed that.

On Fri, Dec 1, 2017 at 10:32 AM, Sherman Monroe <sdm...@gm...> wrote:

> After reviewing the dbe_storage_s struct and the code in disk.c, it is
> apparent that disk.c is the *.db file manager of Virtuoso, and this file
> manager is sufficiently sophisticated as to elicit an alternative to
> extracting a programming interface from disk.c. So my preliminary
> proposal is to leave the disk.c code alone and instead make virtuoso.db a
> virtual file such that reading from the file actually reads from the
> stdout of a command and writing to the file actually writes to the stdin
> of a command. Then the commands in question will interact with the
> persistent store of choice, including the eos.io smart contract running
> on the blockchain. Some additional requirements are:
>
> - Low level-of-effort, so implementing a custom file system (e.g. fuse
> <http://fuse.sourceforge.net/>) is out of the question
> - Must be cross platform, so e.g. linux fifo special files
> <http://www.gnu.org/software/libc/manual/html_node/FIFO-Special-Files.html>
> are out of the question
> - Preserves performance, i.e. the solution cannot adversely impact
> Virtuoso's current performance metrics
>
> After some preliminary research, writing to a named pipe
> <https://en.wikipedia.org/wiki/Named_pipe> looks very promising. The
> downside is that you can't have concurrent processes reading/listening to
> the file, but AFAIK only one process is accessing virtuoso.db at any
> given time. Once the content is read its gone so disk.c will need to
> treat it like a buffered stream (e.g. the writer will continuously push the
> content to the buffer, and the reader will loop the buffered content
> indefinitely).
>
> The command writing to the buffer is what I'll call an adaptor. It's job
> is to persist the state of the "virtual file" based on what it receives
> from Virtuoso and to fulfill Virtuoso's reads. This way, the adaptor is
> agnostic to the contents of virtuoso.db and its logic only needs to
> handle the details of the persistence provider.
>
> What are your thoughts? I'm sure I've overlooked requirements you
> envision, if so please share those also.
>
>
> On Thu, Nov 30, 2017 at 2:34 PM, Sherman Monroe <sdm...@gm...>
> wrote:
>
>> Correction, dbe_storage_s.dbs_file
>>
>> On Thu, Nov 30, 2017 at 1:43 PM, Sherman Monroe <sdm...@gm...>
>> wrote:
>>
>>> I've found dbe_storage_s and dbe_storage_s->dbs_file. Design proposal
>>> en route...
>>>
>>> On Thu, Nov 30, 2017 at 8:24 AM, Sherman Monroe <sdm...@gm...>
>>> wrote:
>>>
>>>> Ok, I will review the codebase and begin the design. I'll post for
>>>> peer review before I begin coding.
>>>>
>>>> On Wed, Nov 29, 2017 at 8:22 PM, Kingsley Idehen <
>>>> ki...@op...> wrote:
>>>>
>>>>> On 11/29/17 6:47 PM, Sherman Monroe wrote:
>>>>>
>>>>> So as a starting point, I abstract every single call to .db and also
>>>>> any dependent functions dealing with in-memory state. Next I implement the
>>>>> old "file backed" persistence using the new API, and run test cases to
>>>>> ensure nothing broke. Next I test it on some common alternative types like
>>>>> the ones you list above. Finally I implement the eos.io backed
>>>>> persistence. Does this sum up the requirements?
>>>>>
>>>>> A 2K feet, yes.
>>>>>
>>>>>
>>>>> Kingsley
>>>>>
>>>>>
>>>>> On Wed, Nov 29, 2017 at 5:40 PM, Sherman Monroe <sdm...@gm...>
>>>>> wrote:
>>>>>
>>>>>> Excellent, that's what I hoped.
>>>>>>
>>>>>> On Wed, Nov 29, 2017 at 5:38 PM, Kingsley Idehen <
>>>>>> ki...@op...> wrote:
>>>>>>
>>>>>>> On 11/29/17 6:35 PM, Sherman Monroe wrote:
>>>>>>>
>>>>>>> Is Virtuoso loading the contents of "xyz.db" into an internal model,
>>>>>>> or is it performing transactions against "xyz.db"?
>>>>>>>
>>>>>>>
>>>>>>> Persistence happens against "xyz.db". Other relations related
>>>>>>> activity happens in memory.
>>>>>>>
>>>>>>> Kingsley
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 29, 2017 at 5:18 PM, Sherman Monroe <sdm...@gm...>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Kingsley, yes I would like to take this on and then implement a
>>>>>>>> smart contract-based persistence with it. Can you suggest starting points
>>>>>>>> or what work has been done so far?
>>>>>>>>
>>>>>>>> On Wed, Nov 29, 2017 at 4:33 PM, Kingsley Idehen <
>>>>>>>> ki...@op...> wrote:
>>>>>>>>
>>>>>>>>> On 11/29/17 5:09 PM, Sherman Monroe wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I would like to know what is the feasibility of implementing
>>>>>>>>> Virtuoso's "state" one a blockchain like eos.io? If for instance
>>>>>>>>> Virt is storing its quad-store state as a linked list, then if I wanted to
>>>>>>>>> swap that datatype out for a custom, EOSLinkedList, how difficult would
>>>>>>>>> that be? Does Virtuoso support a swappable persistence API of any sort, or
>>>>>>>>> interfaces for the data types it uses for persistence?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We are looking at loose-coupling of persistent storage i.e.,
>>>>>>>>> rather than being pegged to "xyz.db" you end up being able to use a variety
>>>>>>>>> of document types (CSV, ORC, Parquet etc..).
>>>>>>>>>
>>>>>>>>> Naturally, anyone can pick up this effort to accelerate matters :)
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Kingsley Idehen	
>>>>>>>>> Founder & CEO
>>>>>>>>> OpenLink Software   (Home Page: http://www.openlinksw.com)
>>>>>>>>>
>>>>>>>>> Weblogs (Blogs):
>>>>>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/
>>>>>>>>> Blogspot Blog: http://kidehen.blogspot.com
>>>>>>>>> Medium Blog: https://medium.com/@kidehen
>>>>>>>>>
>>>>>>>>> Profile Pages:
>>>>>>>>> Pinterest: https://www.pinterest.com/kidehen/
>>>>>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen
>>>>>>>>> Twitter: https://twitter.com/kidehen
>>>>>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about
>>>>>>>>> LinkedIn: http://www.linkedin.com/in/kidehen
>>>>>>>>>
>>>>>>>>> Web Identities (WebID):
>>>>>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i
>>>>>>>>>         : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------
>>>>>>>>> ------------------
>>>>>>>>> Check out the vibrant tech community on one of the world's most
>>>>>>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>>>>>>> _______________________________________________
>>>>>>>>> Virtuoso-devel mailing list
>>>>>>>>> Vir...@li...
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -sherman
>>>>>>>>
>>>>>>>> Every good gift and every perfect gift is from above, and cometh
>>>>>>>> down from the Father of lights, with whom is no variableness, neither
>>>>>>>> shadow of turning.
>>>>>>>> (James 1:17)
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -sherman
>>>>>>>
>>>>>>> Every good gift and every perfect gift is from above, and cometh
>>>>>>> down from the Father of lights, with whom is no variableness, neither
>>>>>>> shadow of turning.
>>>>>>> (James 1:17)
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>>
>>>>>>> Kingsley Idehen	
>>>>>>> Founder & CEO
>>>>>>> OpenLink Software   (Home Page: http://www.openlinksw.com)
>>>>>>>
>>>>>>> Weblogs (Blogs):
>>>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/
>>>>>>> Blogspot Blog: http://kidehen.blogspot.com
>>>>>>> Medium Blog: https://medium.com/@kidehen
>>>>>>>
>>>>>>> Profile Pages:
>>>>>>> Pinterest: https://www.pinterest.com/kidehen/
>>>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen
>>>>>>> Twitter: https://twitter.com/kidehen
>>>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about
>>>>>>> LinkedIn: http://www.linkedin.com/in/kidehen
>>>>>>>
>>>>>>> Web Identities (WebID):
>>>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i
>>>>>>>         : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Thanks,
>>>>>> -sherman
>>>>>>
>>>>>> Every good gift and every perfect gift is from above, and cometh down
>>>>>> from the Father of lights, with whom is no variableness, neither shadow of
>>>>>> turning.
>>>>>> (James 1:17)
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>> -sherman
>>>>>
>>>>> Every good gift and every perfect gift is from above, and cometh down
>>>>> from the Father of lights, with whom is no variableness, neither shadow of
>>>>> turning.
>>>>> (James 1:17)
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Kingsley Idehen	
>>>>> Founder & CEO
>>>>> OpenLink Software   (Home Page: http://www.openlinksw.com)
>>>>>
>>>>> Weblogs (Blogs):
>>>>> Legacy Blog: http://www.openlinksw.com/blog/~kidehen/
>>>>> Blogspot Blog: http://kidehen.blogspot.com
>>>>> Medium Blog: https://medium.com/@kidehen
>>>>>
>>>>> Profile Pages:
>>>>> Pinterest: https://www.pinterest.com/kidehen/
>>>>> Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen
>>>>> Twitter: https://twitter.com/kidehen
>>>>> Google+: https://plus.google.com/+KingsleyIdehen/about
>>>>> LinkedIn: http://www.linkedin.com/in/kidehen
>>>>>
>>>>> Web Identities (WebID):
>>>>> Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i
>>>>>         : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> -sherman
>>>>
>>>> Every good gift and every perfect gift is from above, and cometh down
>>>> from the Father of lights, with whom is no variableness, neither shadow of
>>>> turning.
>>>> (James 1:17)
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> -sherman
>>>
>>> Every good gift and every perfect gift is from above, and cometh down
>>> from the Father of lights, with whom is no variableness, neither shadow of
>>> turning.
>>> (James 1:17)
>>>
>>
>>
>>
>> --
>>
>> Thanks,
>> -sherman
>>
>> Every good gift and every perfect gift is from above, and cometh down
>> from the Father of lights, with whom is no variableness, neither shadow of
>> turning.
>> (James 1:17)
>>
>
>
>
> --
>
> Thanks,
> -sherman
>
> Every good gift and every perfect gift is from above, and cometh down from
> the Father of lights, with whom is no variableness, neither shadow of
> turning.
> (James 1:17)
>

-- 

Thanks,
-sherman

Every good gift and every perfect gift is from above, and cometh down from
the Father of lights, with whom is no variableness, neither shadow of
turning.
(James 1:17)