StorageAndLocking

Storage and Locking

Why store data in on-disk XML files?

This decision is likely the one which have most impact on performance and scalability of the server so it warrants some explanation of this unusual decision:

1. No dependency on available of any particular database or even PDO
2. Allows for direct metering of the size of the per-server data (since it is just a directory of files)
3. Allows for easy deployment/backup/cleanup of a server installation - This is probably the biggest single reason since it makes it trivial to run the testing gauntlet against any kind of in-place or clean data
4. Allows for arbitrary data arrangements - This was a large factor in early development since it made the prototyping easier as there was no time required to design a schema for any specific storage philosophy

Obvious problems

1. Scalability - The most obvious problem is in the size of the data structure stored into a single file since every operation on said data needs to read and parse the entire file
2. Data migration - While the lack of a central schema allowed for fast early development this has come at a cost where changing existing data to a new shape requires migrating each instance of the file, individually. This can often be mitigated by offline processing or lazy update code within the server

Why use file system locking?

It is generally uncommon for a PHP-based server to do explicit locking (since most operations are simple DB update or read operations) so there are few options (and many are restricted in deployment).

Since OpenAutonomy functions more like an application server, it required a basic synchronization primitive which could be used as an arbitrary read-write lock. File locking provides this capability and is robust against server crashes or unexpected script termination. The fact that on-disk storage was already used made this locking primitive a natural choice.