Re: [Freesql-discuss] Ping?

[Gordan B. <go...@bo...>]

Hi.

> So you want to port FreeSQL to JavaScript, so that the JavaScript "apps"
> can use Freenet as storage?

Yes, that's about the size of it. :-)

> Well, I suppose it's possible--instead of calling Freenet commands on the
> local mahine, the way FreeSQL does in the perl version, you would make
> XML-RPC calls from within the JavaScript to some Freenet node. Assuming JS
> and Freenet support that, I don't see why not.

Well, it wouldn't be XML-RPC to the FreeNet node. I don't think that 
FreeNet nodes support that. It would be more of a case of writing a 
JS library that would talk HTTP to the node in order to upload/download 
data as required.

> One problem I see is that one of the advantages of FreeSQL is performance
> boosting using local caching. If you can't use local files, you are out of
> luck on that one.

It depends on how you look at the problem. The data could be sorted in a 
specific way at insertion time, into different file names and different 
"directories", and then only the relevant data could be retrieved. This 
would be very useful for full-text searching, for example.

In theory, if the application is public, and the data is shared, then the 
more people use it, the less of a problem the performance will become, as 
the nodes start caching the data.

I can certainly see this method working very fast where only one index is 
used on a table. Multiple indices can be implemented, but the there would 
be data duplication. And since most applications spend most of their time 
making use of just one index per table (could be multi-column, of course), 
it isn't really an issue for anything but very large applications with 
very complex tables and relationships.

But if I use the idea of using file naming for fast index searching (say 
convert each indexable key word into an SHA-512 hash), and then use 
cascaded recursive algorithm to perform table joins, it should reduce the 
amount of retrieval work to a bare minimum - even for operations as 
complex as FTS with table joins. Obviously, the less restrictive the 
query, the slower the process.

> The other problem, or question really, is Why do you want to do this in
> JavaScript of all languages? I generally try avoid JavaScript whenever I
> can.

I used to think so until recently, too, but there are a few things that 
have happened since then:

1) Mozilla 1.0+ fully supports DOM.
2) IE 6+ supports DOM sufficiently that the lacking functionality can be 
detected and emulted using wrapper JS libraries living in "user space" (or 
should I say "browser space"?).

Additionally, everybody has a browser nowdays. Not everybody necessarily 
has a Java or .NET VM. JS fits nicely into the web development paradigm, 
and a lot of people know the language. As a language, it is not actually 
that bad - it has a fair amount going for it - especially now there is a 
DOM standard that is addopted.

The FreeNet plug for JS would give the full flexibility of creating a 
secure application that cannot damage the user's machine (because JS is 
bolted down to prevent that), has fully distributed operation (runs in the 
client browser) and has data sharing that is as minimal or extensive as 
required. It would allow for no local disk space requirements - just use 
FreeNet as a big shared, secure and masively redundant file system.

> Another suggestion: Why don't you just use one of the many security holes
> in various JavaScript implementations to gain access to the local file
> system and store the files that way? Just kidding. :)

Because I would rather not depend on users using IE. :-p

> As to your question about indexing and table joins, from a quick look at
> the FreeSQL code, it does do indexing but it doesn't appear to support
> joins as yet.

I've been thinking about this. If a full join is required, then there is 
no choice - at least one table has to be downloaded whole. Slow, but not 
difficult.

However, if some criteria for filtering is specified, i.e.

WHERE MyTable.MyField = 'MyValue'

then MyTable can be selectively retrieved first according to the index on 
MyField.

For the join itself, take the smaller of the two pre-filtered tables 
(according to WHERE statements, if any, defaulting to the one that is 
WHERE filtered, unless some table size metrics can be found), and extract 
unique values of the join fields. Select the values from the table that is 
being joined on, and continue this cascade process until all the joins are 
fullfilled.

Further optimisations can be applied later, but this would deal 
efficiently with a vast majority of cases, and the data storage method can 
help a lot, if for example, you know that all the records that start with 
the letter "a" are in the "a" directory. Of course, the much better way 
would be to isolate key words and use hashes to look for them.

But anyway, this is getting a bit long. I have to look through the FreeSQL 
source code first. :-) Ideally, I would like to see the data 
"storage" formats to be compatible between FreeSQL and what I end up 
implementing in JS...

> Hope that helps, at least a little.

It is always useful to bounce ideas off of someone when deciding on a 
design for an application. :-)

Regards.

Gordan