Re: [Algorithms] High-Speed Flat File System
Brought to you by:
vexxed72
From: <bn...@pl...> - 2007-02-15 19:37:35
|
Hi, This first par is probably not relevant, but just in case... About three years ago I did a port of SQLite to PlayStation 2. The filesystem for the PS2 HDD was extremely limited and basic (for example, no caching, no concept of file-level locking), which made it difficult to get good performance from SQLite for our needs. This was aggravated by a few things in SQLite itself - for example, the page size was something like 1043 bytes, which is very unfriendly to your device drivers (every page read operation had to read 2k or 16k blocks and throw out most of that, worse if the read straddled a block boundary, and even worse still when doing a write). I requested that they change the block size to a power of 2 when they switched from version 2 to 3, but I don't think it was done. Anyway, those problems have little or no bearing on most filesystems, so as I said it's probably not relevant in your situation. What definitely is relevant is that around SQLite we built our media database system. We stored the metadata and directory structure for the media files in an SQL tables, and had wrapper functions to functions to make that appear more like a regular file system. We stored the files directly into the native filesystem, but renamed them to match the database IDs (also allows symbolic links or multiple views), and split them into multiple directories to avoid performance problems with storing large numbers of files in a single directory. This system worked really well overall, despite not picking the optimal database schema and APIs to handle the most common and performance critical operations. We also wrote our own FAT16 filesystem to get around some of the issues with the native filesystem for another project. It worked for us, but I would not recommend that at all for you needs. If you need more info let me know. -Ben John Ratcliff wrote: > Thanks Ivan, and everyone else, for all of the excellent feedback to my > question; both on and off list. > > I was excited to hear about SQLite since it appears designed to solve > exactly the problem I laid out. It is a very modest amount of source code > and is supposed to be very fast, robust, and stable. > > Sometime soon I will write a wrapper around it to emulate basic file access, > I may even emulate a directory structure as well. If I do I will be sure to > post it on my coding website. > > Thanks, > John |