Re: [Relfs-devel] Post-OCaml development. CL-FUSE

[Michael R. <fb0...@ra...>]

Vincenzo Ciancia wrote:
> To anybody reading, that is, to the "casual reader". This project is
> not "abandoned" by itself. It's just that I don't have ANY time for
> it.

I also do not consider RelFS project abandoned. Ocaml version, though,
seems abandoned on any reasonable criteria (nothing worth reusing, no
activity around it etc.)

I hope to submit a paper about my new Lisp branch..

> Such a design inherently does not play well with centralised databases
> like postgresql. After an analysis done this august, we (more or less
> everyone including Michael) agreed that multiple sqlite files are the
> only feasible way.

I currently focus on query interface to the external, so I still use
postgresql because I think of reusing some data.. But in my design using
multiple SQLite DBs will be simple.

> A proper design document will be created if it becomes clear that the
> manpower is really available, that is, if there is someone who really
> has the time to code.

Probably it will partially be documentation of implementation.

> Dear Michael,
> 
> I am glad to hear from you.

I am also glad to receive a reply from you.

>>        As of now, RelFS written in OCaml can be considered abandoned. We
>> decided to switch to some other design. So, what success do you have?
>
> So as you know, the biggest showstopper of relfs is manpower, followed
> / caused also by the lethargic state of ocamlfuse.

And also by some unfortunate things in its general design. And OCamlFuse
is usable right now, the problem is that it is very hard to add
threading to it or improve its speed.

> I don't think I announced anything that you are spoiling already :) So
> no, nothing happened. 5 people want to develop the core relfs (any
> other can just raise their hands) but I didn't ask anything to them,
> since my work and my travels absorb me too much.

If anyone wanted to develop specifically OCaml version, where are the
emails? Where is anything?

I do not consider RelFS as the whole abandoned - or I wouldn't write my
previous email here.

>> http://mtn-host.prjek.net/viewmtn/cl-fuse/revision/browse/23483c4d94f7756d365626b247f82cdd588828e3/example-queries
>>
>>        I would be glad to hear comments about syntax, extension ideas etc.
>>
> 
> I will look at it soon, as you know I am no lisp expert. However, can

There are example-queries. My main question is about your opinion about
these.

> you provide a personal judgment on your binding? How complete is it?
> Did you test efficiency? Is it multi-threaded? Does it use multiple
> cores? (the latter would be a killer feature if compared to ocaml).

My binding is multi-level. On the lowest level, you can use CFFI library
to define C callbacks (these are Lisp functions written by special rules
- related to the way of argument processing). This level is complete
w.r.t. FUSE 2.7.4 release. Actually, you only need a basic
infrastructure there, and the list of fields - there is nothing to wrap
there, user gets raw FUSE interface as it is. About performance: it
boils down to compiled code implementing raw FUSE callbacks. I see no
reason why it should be noticeably slower than C if written carefully.
But SBCL will require more memory for its runtime..

I also allows to implement call manager. By default everything is
single-threaded, but you can instruct my bindings to spawn threads for
requests.

Next level: Lisp-style wrappers. These allow you to implement basic
reading and writing etc. They are easily extensible, though. They
currently force you to have extra memory copying operations (you are
supposed to return Lisp byte sequences, and it will manage foreign
memory interaction). My idea is that you don't care when you feed large
files only via symlinks.

Next level: prototyping bindings with automatic pathname management.
These are even more incomplete and are somewhat hard to extend without
increasing amount of code user has to write. Of course, there is even
more mandatory overhead.

> To you, but probably not really. Try to develop your idea, I will try
> mine (one day, another day, we don't know). We will always be able to
> exchange non-core code. And for now, if your filesystem works, I will

Actually, I currently plan to consider everything except core queries
non-core code. I.e. you can have inotify-listening daemon that tells
filesystem how to update tables. Trivial indexing is implemented via
writing to a special file on my FS - that process is defined in "user
code" in queries.

> probably use it, so perhaps the next step would be to close this
> project and follow you. For now it is better if we remain both  free
> to experiment with our own ideas.

Of course. I do hope to end up with something to officially merge with
RelFS. My project is already in process of consuming previous CL-FUSE
project, so gathering stones is in order.

I guess what we will have to define is the default DB layout for
widespread kinds of indices.

> When you have a prototype alternative design, let me know on this
> list, I will certainly be happy to take a look.

Well, there is currently just layering of definitions.