[Bobs-devel] file index issue again

[Rene <re...@gr...>]

I've been looking at the file index structure. 

Maybe using flat files wasn't such a Great Idea(tm) in the long run.
I've done some research into using db3 (sleepycat software) instead.
Results are nice. It's a really simple database and fairly quick as
well.

I did a test in which I searched for files larger than x bytes, smaller
than y bytes and contains the string "ab" in its name.
It took less than 4 seconds to parse the database. That was with ~130000
files.
Each files has 4 entries "path","name","date" and "size"

The db file is 38 MB in size. The index file I used to fill the database
is only 13 MB in size.

But the search speed is really amazing compared to the way bobs do it.

I tested this on a athlon 1800+ SMP computer. (it only uses on cpu for
searching)

Check this link for more information.
http://dk.php.net/manual/en/ref.dba.php

Redhat 7.2 and 7.3 ships php with db3 enabled.


To make it completely useable there would have to be at least to
databases for each share. One for the files and one for the dirs.
In the dirs database the should be a reference to which index numbers
that dir has in the file database.
eg.
index 0 = /somedir/at/some/path/
index 1 = 1000-1304
which would indicate that there are 304 files in this dir and they are
located between index 1000 and index 1304 in the file database.

This is needed for a really fast search which is needed when we display
files in the browser.

This way a share with 15000 dirs should take less than 1 second to
display a file list in the browser.

The real feature is that this is a 1000 times easier to handle in php.


Let me know what you think.
Is it a good idea to use it?

-Rene