#64 Intelligent caching

[James Allen]

Currently the caching mechanism is very simplistic: it stores an entire page at a time, which is a problem for two main reasons:
a) this is incredibly inefficient, as the header, footer and nav bar are stored many times over - I estimate that on 8GBs-worth of cached pages, over 1GB is used to store these static elements that total to a few KB...
2) there is no proper way to store pages with private data, or with AJAX elements - I did implement a workaround (used on the Gene Overview pages) where you create separate cache folders for each group that has access to private data, but this is not at all scalable. It works OK for the few hundred Gene Overview pages where private data is an issue, but is not workable for the Marker Overview pages.

To solve both of these problems, we need a more intelligent caching system, where chunks of the pages are cached, and stitched together as necessary when a page is requested. It may then not be necessary to cache private data, since if most of the page is public, and is thus being served from a cache, then the page might already be quick enough; and if you do have to cache private data, at least you'd only be caching that section, rather than duplicating everything else on the page that is also publicly cached.