Re: [Relfs-devel] "LOCAL" #define in Common.h

[Vincenzo a. N. N. <vin...@ya...>]

On Sunday 01 August 2004 22:37, Guido Villa wrote:
> I'm giving a quick look at the source code...
>
> Why not write:
>
> #define LOCAL(path) (path +1)
>
> instead of:
>
> #define LOCAL(path) (string(".")+string(path))
>

First of all, by now we are developing a prototype. C++ is not a good 
language for rapid prototyping, neither is C, (I usually choose ocaml 
when I am in a hurry, and I have many segfault-free programs written in 
a couple of weeks :) ) but I had to choose between cross-language 
interoperability and rapidity of development. Since I want a stable 
interface for plugins, and the ability to write plugins in almost any 
language, I choose C at first. But not having a rich standard library 
convinced me to switch to C++ and here we are.

Since I am prototyping, I don't care about performance too much [and I 
don't care too much about how I code - I don't code that bad 
usually :P ].

My idea is that to make a good design you need to have implemented the 
thing; in other words there's something good in that "I feel like 
rewriting everything". I want to write a _very small_ and modular 
program, and then to audit, and eventually rewrite some part, module by 
module. But the plugin interface, and the data model, must be stable, 
so that people can write plugins while we correct and /or rewrite the 
core, being confident that their plugins will keep on working.

Said this, RelFS.cpp is almost plain C, and it will stay that way or 
even more C, so feel free to audit and correct it. Even the class relfs 
is unuseful and might be removed. In the new code however there is a 
single parameter for all plugin operations (for ease of development), 
of type "class File", so a bit of memory allocation is required. 

The rest of the program will remain in C++ I guess, if not translated 
into a more usable language, because of the need for high level 
libraries.

In fact I had defined LOCAL exactly as you suggested, but then I needed 
strings in the rest of the program and so I opted for using strings 
everywhere. I am not sure about why I choose to add an initial "./", 
feel free to experiment removing it to discover why :) However the most 
important speedups will come from:

1. using the threaded fuse mainloop. This is necessary, since we can't 
block the whole filesystem waiting for a cdrom read. By now this would 
mean implementing either a thread waiting for SQL queries and sending 
it over a protected connection, or making as many SQL connections as 
queries in the program, since libpqxx doesn't accept multiple threads 
on a single opened connection. I would opt for solution 1.

2. asynchronous plugins. This is not as easy as it might seem, since we 
can't just fork a process or a thread for each indexing operation, or 
the system will almost freeze with tons of processes reading from disk. 
It would be better to have a queue of indexers to run (and to remove 
them from the queue if the file is removed in the meantime). Also, a 
good idea would be to try to gather more information about what exactly 
plugins will read from a file: if three different indexers will read a 
huge file from the beginning to the end, it could be better to allow 
them to declare it, and to pass the same data to all of them. Caching 
mitigates this problem but this doesn't mean we can't improve it.

>
> I like the plugin proxy idea, because plugin calls (which could be
> slow) can be moved out of sync from the main process.

Oh, not only they can, but they _will_ be moved out of sync asap :) The 
most important things for now are a stable index plugins interface and 
a good data model. I will think about the data model in the next days. 
After these two steps, we will focus on getting maximum speed and 
stability from relfs.cpp, including a separate thread or process for 
plugins. If other people agree, of course! I am no chief here even if I 
am the only coder right now :)

I would be glad to see patches to code, but I recommend not to take care 
of details at the moment, because it's highly probable for entire files 
to be removed in this phase. It's better to focus on how features can 
be implemented (and to see IF certain features can be implemented at 
all using the current design).

bye

Vincenzo

-- 
I was dressed for success,
but success it never comes
[Pavement - Here]