Linux Compressed Cache / News: Recent posts

This new major version has two main features: page cache is supported, so from now on all pages from page cache (not only swap cache pages) are compressed; and manual adaptivity is fully implemented. Besides that, this version have shared memory support back, clean pages support in compressed cache, many cleanups, simpler code, much more documented code (mainly vswap) and also a code more rock solid (with allocation failure handling, for instance). Unfortunately the manual adaptivity does not allow us to disable compressed cache, but it's one of our high-priority itens on our todo list. ... read more

New dbench tests have been done with the 2.4.18-0.22pre7 version of Compressed Cache. With our new page cache support, dbench is running much better with compressed caching. Check it out.

http://linuxcompressed.sourceforge.net/statistics/0.22pre7_dbench/

This version fixes some bugs and hopefully the swap shrinkage, so now we can reconfigure compressed cache size again. That way, we have a stable compressed cache with support to the whole page cache, and also manual adaptivity.

Check more details below.

Notes:
http://sourceforge.net/project/shownotes.php?release_id=79353

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.18-0.22pre8.bz2... read more

This version is a first attempt to extend compressed cache support to page cache. It seems to be somewhat stable, but it's likely to have some bugs out there. Virtual Swap adaptivity is _broken_, so please do not resize your compressed cache on-the-fly in this version.

Notes:
http://sourceforge.net/project/shownotes.php?release_id=48824

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.22pre5.bz2

This version have the batch of changes related to vswap adaptivity stabilization. It seems to be actually stable even for heavy concurrency in adaptivity, unlike the last version which was sort of stable only, since it didn't handled failed vm areas lookups.

Notes:
http://sourceforge.net/project/shownotes.php?release_id=48720

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.22pre3.bz2

This version features the adaptable vswap implementation. It's the first version and have to be improved, but it seems stable though. One of the improvements is to deal with many shrink and grows in a row (what I think it's stable but not completely functional). And we still have to deal with a possible find_vma failure, what will cause a BUG() in the current code.

Notes:
http://sourceforge.net/project/shownotes.php?release_id=43957... read more

I ran some tests to have some performance statistics out of
our 0.21 version of Compressed Cache. The selected tests
were: fillmem/mmap001 (from memtest), Linux Kernel
compilation and Dbench.

Kernel compilation is very interesting since this is an CPU
intensive and Dbench an IO intensive application. Besides
that, both are widely used for VM tests and couldn't miss on
our test list.

I tested many Compressed Cache sizes (4, 8, 16 and 24)
and plotted graphics for every test in order to help
understanding the data. Also the CPU usage data (except
for dbench case) was plotted. ... read more

Hash table is adaptable now and it's the first step to conclude adaptivity of our data structures. The next stage will be to make vswap address data structure adaptable as well.

Notes:
http://sourceforge.net/project/shownotes.php?release_id=43804

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.22pre1.bz2

This major version features lots of chances that improved a
lot Compressed Cache code and that make us very glad to
announce.

Adaptivity's been started (it was the main development in
0.21pre1 and 0.21pre2 versions), but the development focus
was changed to performance, what made us perform major
surgeries. The code had been showing a terrible
performance when running with big Compressed Cache
sizes. We hope to make adaptivity work smoothly and to be
fully functional (at least manual adaptivity) in the next
version. Anyway, you can already enjoy this partialy
functional feature introduced here.... read more

Simple changes regarding hash tables and some cleanups.
Now there's no need to enter a power of 2 number for the
maximum number of compressed pages any longer. It's not
increasing the hash table size in a dynamic way, but that
will be done soon. More details in the notes link below.

Notes
http://sourceforge.net/project/shownotes.php?release_id=51703

Download
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.21pre7.bz2

This batch of changes still includes lots of cleanups and code rewrite to make it simpler. Perfomance increase has been noticed too. Comp_cache_free() has been rewritten and does not move fragments to and fro all the time. Vswap does not have functions to lock and unlock its entries since it's not needed. Many other changes in this version and many still to go.

Notes
http://sourceforge.net/project/shownotes.php?release_id=50313... read more

Jeff Dike, author and maintainer of User Mode Linux, came up with good news on Friday. He finally found the bug I reported and has already fixed it.

Check his email regarding the fix here: http://www.geocrawler.com/lists/3/SourceForge/709/25/7517519/

Thus, if you want to run Compressed Cache under UML, please use 2.4.17-5um version or above.

This batch of changes includes cleanups and code rewrites.
The swap_out_fragments() and find_free_swap_buffer() are
much simpler and also more efficient. Some preliminary
tests showed a performance gain due to these
improvements. See details in the notes link below:

Notes
http://sourceforge.net/project/shownotes.php?release_id=48824

Download
http://telia.dl.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.21pre5.bz2... read more

Since Friday this new version is available for download. It
has some changes regarding code optimization, mainly
concerning the fragment hash table. These modifications
have improved a lot some tests (fillmem) we performed to
test our current code. More optimizations to go soon.

Download:
http://telia.dl.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.21pre3.bz2

Notes:
http://sourceforge.net/project/shownotes.php?release_id=48720

A new patch is out there. This patch improves the manual shrinking and growing policies. The changes are very simple and seems to have a great (at least conceptually) impact on the adaptivity. Check details in the changelog below.

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.21pre2.bz2

Notes:
http://sourceforge.net/project/shownotes.php?release_id=43957

New patch is available for download. Maybe the last one
this year. This version introduces a very first prototype for
shrinking and growing the compressed cache. All is done
manually, by echoing the size (in compressed pages) to
/proc/sys/vm/comp_cache/size. Check the changes in
details in the changelog link below.

Happy New Year!

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.21pre1.bz2... read more

The new 0.20 version has been updated to 2.4.17 version of Linux kernel. Below a copy of 2.4.16-0.20 release notes and links to the changelog and to the patch.

2.4.16-0.20
Finally a new major version is out. This version features mostly code cleanup and minor changes. For example, we have a directory in /proc/sys/vm/ where some parameters are (and will be) changed. Now you can choose the compressed cache size by a kernel parameter and also by a configuration option in the kernel menu. ... read more

Finally a new major version is out. This version features mostly code cleanup and minor changes. For example, we have a directory in /proc/sys/vm/ where some parameters are (and will be) changed. Now you can choose the compressed cache size by a kernel parameter and also by a configuration option in the kernel menu.

Some bugs that showed up with the new kernel versions were fixed as well. A major feature is the lru ordering for compressed cache pages, fixing up a conceptual flaw that existed for a long time in our code. That made us remove the second avl tree, used for freeable space, decreasing the complexity of some functions. A good news is that we started the preparation for adaptivity by changing the comp_cache array to slab cache allocation. Adaptivity (only manual) is our next task.... read more

The new 0.20pre5 version of Compressed Cache patch is
out. Now all compressed pages in compressed cache are
swapped out in the exact order they were added to
compressed cache, therefore following the order the
pages were evicted by the VM system (LRU).

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.16-0.20pre5.bz2

A new patch is out there. The 0.20pre4 has the second batch of changes for the removal of #ifdefs in order to make the code cleaner.

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.16-0.20pre4.bz2

Compressed caching is a new level in the virtual memory hierarchy, where pages are stored in some compressed format, decreasing the number of page faults that are serviced by slow hard disks. We aim to implement this idea in the Linux Kernel.

There's a new patch out there which updates the 0.20pre2 code to 2.4.16 version of Linux kernel. No fix whatever. A part of the code was redone in order to work with the new swap file functions (swapfile.c).... read more

The 0.20pre3 version of Compressed Cache is out. This version features the first batch of changes to remove the #ifdefs and thus make the code cleaner. It follows the tips written in Documentation/SubmittingPatches document from Linux source code tree.

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.16-0.20pre3.bz2

The 2.4.16-0.20pre2 version of Compressed Cache is not currently working under User Mode Linux (at least not up to 2.4.16-2um). There's a problem regarding ptes and the virtual swap addresses used by our code. We don't know why some bits are not correctly updated, what messes up everything when a pte faults a not updated address. Check the email I sent to user-mode-linux-devel mailing list.

http://www.geocrawler.com/lists/3/SourceForge/709/0/7304091/... read more

From now on, since all my graduate courses are over, I will work full time on the Compressed Cache project. I think that we will be able to have a greater development and to get much better results from the project.

Linux 2.4.x will have a bigger userbase than 2.5 and will suffer only minor changes, so I think that we should follow this version. Once in a while I will port to 2.5 and make a patch available on the project page to keep track of the changes being done in this kernel version. ... read more

Not sure it's an UML problem, but it has shown up only when running under it. The problem is that we can't run compressed cache with virtual swap address that uml kernel will panic. So, while we can't solve it, if you'd like to run the patches for 2.4.10 under UML, please modify include/linux/comp_cache.h file. Change:

-#define COMP_CACHE_SWP_TYPE MAX_SWAPFILES
+#define COMP_CACHE_SWP_TYPE (MAX_SWAPFILES-1)... read more