* Andreas Grau (Andreas.Grau@...) wrote:
> The implementation of the new tile storage db is now almost finished. The
> version in the svn (gaia/branches/gaia-tiledb/lib/tiledb), contains:
>
> - put/get/remove(x,y,level)/delete(index_page)
What's delete index page for? I think it isn't for public interface.
I think we'll also need exists(x,y,level) - to check whether specified
tile exists without touching data file (mostly for use with lazy locking in
batch operations)
> - lazy_locking (improve import/export from db without ongoing lock/unlock)
> - support of big/little endianmashines (allowing exchange of db files)
> - support for concurrent put/get/delete operation from different processes
Nice ;)
> - handle unexpected program terminations, keep db consistent
Erm.. There sure should be no exit(3) calls in the library. Functions
should return errors like libwwfetch instead.
> missing features:
> - checksumming, detect corrupted db files (open question: which MDC should
> be used: XOR, CRC32, MD5..., should we protect index and data file)
MD5 is overkill and too slow. Let it be simple 32 bit XOR for now.
Checksums should be only applied to data and be stored in index. Btw, it
will automatically protect us from index corruption, as with corrupted
anything checksum won't match -> data won't get into gaia or other
program.
> - defragment data file (open question also defragment index file?, who calls
> defrag, provide only as function for external tools?)
That's nontrivial task, we should work on it later.
I see it as library function, that does following:
- lock whole index file
- walk index file beginning at (0,0,0), maintaining lists of used
entries (dunno how you name it, when you keep data like that:
(pages 0-10 used, page 11 free, page 12 used, pages 13-99 free,...)
for both index entries and datafile pages. Thus, we'll find gaps in
both data and index files, and it won't consume too much memory.
Along with those lists it should keep last (index_gaps_count + c)
offsets of index pages and last (hmm.. any ideas?) page numbers+counts
of datafile entries.
- next, it should move index and data entries from end of files to gaps.
> - maximum db size (open question, where is the maximum size specified?
> creation of db, on every open? [what if two process opens db with
> different sizes?])
Also, shouldn't concentrate on it for now. I think it should work just
like above, but also we'll just delete (count as gaps) some entries.
(xyl's for those entries will be taken from log file, which contain
(preferred_cache_size/average_data_size) xyl's sorted from least to most
recently accessed. That log file is kept in memory as linked list and
read from disk on database open/written on database close.
If two processes open db with different sizes, nothing bad happens.
First, there's always lock on whole index file.
Next, process with lower threshold will begin to clean cache first, and
second process just won't need to do it.
> - remove old data to fit size limitation (open question: also resize index
> file size? maximum db size has two values?)
See above. Index is automatically resized during cleanup. We may just
count cache size as (data size + index size).
> - log file: (see svn/docs) (open question: log entry per data_entry OR
> index_entry? why not store list in index/or data file?)
hmm... yes, that reminds me that I wanted to use index_entry in the log
file. Well, yes, that'll reduce log size severely and make things
easier/faster.
Log should be stored in separate file, because:
- it's only read/written on cache open/close
- if we store last access time in index, we'll need to do frequent
writes, even when just reading cache. This is bad.
Also we'll need to parse index two times when doing cleanup (first,
find least recently used entries, next count them as gaps)
Storing atime in datafile makes things ever worse.
> - other functions: (open question: what is the interface to get all tiles in a
> specific area? a list of structure{x,y,level,data,size)? what other functions
> are needed in the interface?)
That's beyond library scope. It belongs in gaia-cache utillity, which
should process tiles recursively and check whether each tile is in
needed region itself.
Interface to specify region is what bugs me as well. The best I can come
with is:
--minlevel -l
minimal level of detail to work with
--maxlevel -L
maximal level of detail to work with
--point -p
specify single point within desired square region
Thus, we can specify arbitary box to work with, I think that's quite
enough. About -p, I have non-fully-finished library libgeocoords,
that parses geographical coords in different formats, so points can be
specified as (x,y,level), double value or geographical coords in many forms.
> - handle syscall errors while open/close db or locking pages (open questions:
> what should be done if locking a page fails (should only happens in case
> of coding bugs or corrupted db)? exit(-1)?)
No, absoutely no exit's in the library!
See libwwfetch, I think it's right way to code a library, as from both
points of coding the library, and using it in another program.
- all public functions return tiledb_error (tiledb_open too, by the way,
because if it returns Handle*, we won't be able to know reason of
failure if it happens). All data is returned from functions via
pointers.
- trivial internal functions (in which error just can't happen) can
return value directly (or be defined as void)
- all other internal functions should return tiledb_error, and check
error value with each call of other non-trivial internal function.
I don't know better ways to cleanly write a library in plain C :)
> TODOs:
> - discuss open questions
> - implement missing features
> - test db (check code, test stability, test performance)
Thank you very much, that sure is a great work :)
We'll, my opinion is that top priority targets are:
- checksums (whole concept is based on those)
- working gaia-cache utility with at least import/export support
(it'll be nice test to launch 20 imports + 20 exports of ~1 gb
cache simultaneously). For now we can't test it with gaia, as NASA
server is still down.
--
Best regards,
Dmitry Marakasov mailto:amdmi3@...
|