flash filesystem?

2010-10-22
2013-04-06
  • Hi, this isn't directly ChibiOS related, but I thought I would see if anybody here knows of a filesystem module compatible with SPI flash ICs?  I was looking at FatFS and found that it unfortunately doesn't currently work with raw flash devices with arbitrary sized sectors - it appears to be more designed to work with SD cards and other disk-based devices.  Specifically, I have a 64Mbit NOR flash device with sector sizes of 64K - FatFS only supports sector sizes of 512, 1024, 2048 or 4096 bytes.

    Of course, something that could integrate with ChibiOS would be ideal :)  Thanks for any tips!

     
  • Probably this would require a specialized file system, implementing a FAT file system would require a 64K RAM buffer for the read/erase/rewrite cycle.

    We should verify if such a thing (open source flash file system) already exists before discussing other options but probably such a file system would be a good candidate for next developments, may be also transactional and fail safe.

    Giovanni

     
  • The closest things I could find were YAFFS and (maybe?) JFFS2 in the eCos distribution, but it was a bit difficult to understand the state of support, and maturity of these projects.  Otherwise, it seems like there is a bit of a empty niche for an open source flash file system, perhaps?

     
  • alcosar
    alcosar
    2010-10-25

    Take a look at LEAN file system http://freedos-32.sourceforge.net/lean/. It is still in the alpha (unstable) development stage, but it is developed for several years. Initialy it was written in C, now in C++. It claims to be:

    a free, simple, portable, personal, fully featured file system for embedded tasks

    .
    Alexander.

     
  • Hi Alexander - thanks very much for the tip.

    Unfortunately, since this would be for a proprietary project, I would not be able to use LEAN since it is GPL licensed.  It also looks as though it is not designed for very small embedded devices - a lot of the data structures (at http://freedos-32.sourceforge.net/lean/specification.php) are quite large in terms of RAM usage, and it's a little unfortunate that the reference implementation is in C++ instead of C.

    However, there may be some good ideas in there that might be helpful when designing a filesystem.  I'll have a browse through the source package to see if it might be useful in some way.

     
  • I have been thinking about this FS, it is not trivial considering that you cannot manipulate small structures (directory entries etc) without erasing a whole block.

    A FAT-like structure could work because it is "write once" unless you delete a file but a structure holding numeric fieds, like a "file size" would be impossible to handle, we would have to find a "write once" encoding for everything.

    Giovanni

     
  • Liam can you post a link to the datasheet of the SPI chip you are using?

    Giovanni

     
  • Sure thing - http://www.numonyx.com/Documents/Datasheets/M25P64.pdf is the IC.  I'm not a file systems expert, so I'm not familiar with best practices for data structures for different kinds of memory, but I agree that the "write once" nature of flash makes it an interesting problem!

     
  • Probably Enrico is right, the best option would be to use a device more suited for a FS.

    Using the device with large sectors would put constraints on the FS capabilities *or* performance. One possible solution is to reserve a sector as a buffer when a page rewrite is required but this would slow down operations, using this technique even a FAT FS would be usable.

    Giovanni

     
  • Unfortunately the hardware for this project is already finalized on the IC I mentioned previously.  The project currently uses FlashFS from Keil: http://www.keil.com/support/man/docs/rlarm/rlarm_fs_flashfs.htm.  It's completely functional, but the API is very (too) simple in a few cases.  For instance, there's no way to pipeline read access in a callback driven style.

    I suppose that shows there is precedence for file system support for this class of device, if nothing else :)

     
  • Interesting but callback support would be hard to do, the FS would have to live as a state machine on top of ISRs/callbacks. If you need asynchronous operations probably it would be better implemented as a server thread serving transactions feed through pipes/mailboxes.

    Giovanni

     
  • News about this, there is a person willing to work on a transactional file system. We have been collecting some requirements about this "TFFS", Transactional Flash File System.

    http://chibios.sourceforge.net/dokuwiki/doku.php?id=specifications:tffs_requirements

    Our idea is to create an embedded FS able to guarantee the safety of operations ensuring the full reversibility of interrupted transactions.

    Feel free to comment, it is just an early draft, far from being detailed enough.

    Giovanni

     
  • This looks fantastic, and very relevant for my use case.  Can't wait to see more.

    I'm very interested to learn more about how transactions will work - is there basically memory set aside to maintain information about all the ongoing activities within a transaction, such that they can be undone in certain cases?

     
  • That's correct, it would reserve a good part of the memory array as this "transaction buffer", the TB will contain all the info required to eventually restore the state of all the sectors affected by an interrupted transaction. Similar systems are used, as example, into smart cards where the integrity of the internal FS is critical.

    BTW, by transaction I mean a series of operations that must be performed atomically or not performed at all. As example, a transaction could look like:
    Start, write file1, read file 2, write file 2, seek file 1, write file 1, … , Commit.
    Everything between Start and Commit will be performed atomically, in case of an interruption in the middle of the sequence the FS must not be affected at all. The size of the transaction buffer will determine the maximum number of elementary operations that can be encapsulated into a single transaction.

    Of course there will be a trade off between space/performance and safety in such a system. It may look complicated but probably would be pointless to create yet another FS without any specific advantage.

    Giovanni

     
  • More news.

    I think I have a good FS design now, it should allow operations without flash sector erases as long only sequential operations are involved. Only random access to files and file erase operations shall require a large buffer. Devices with small erasable blocks or with internal buffers (modern flash chips have those) will not have this limitation.
    The FS structure will minimize flash wear even without a blocks remapping mechanism, the FS will also handle defective blocks by locking them out.

    This thing will emerge after 2.2.0 and will be the main development during 2.3.x.

    Giovanni

     
  • Great - very much looking forward to this.

     
  • May be it would be worth looking at the new FFS code in FreeBSD 9. They use a technology called softupdates for the data and a transaction buffer only for the metadata.

     
  • Very interesting, probably it is a good idea to study it before proceeding.

    Thanks,
    Giovanni

     
  • Interesting, probably this Coffee could be integrated as-is if somebody is interested in an immediate solution.

    Giovanni