Linux Compressed Cache / News: Recent posts

Hi All!

I now got a job and moving to a new city :)

So will not be able to get on with ccache for next 2 weeks - other things are bound to take much of my time!

-- Nitin

Home page now focuses on developement on 2.6.x kernls. All the previous content is intact and accessible through link.

Much water has passed under the bridge since our laste news about the project. I couldn't work on the project for a long time due to my dissertation and two papers that I wrote about this project. Nevertheless, the project is still alive and I want to keep working on it, since I do have some ideas that may be implemented, now that I finally finished writing my master's dissertation. I also think porting it to 2.5/2.6 will be really cool. However, the only difference is that, from now on, I will not be working on it full-time, since my scholarship is over and I got a job. Anyway, if you want to help out in some way, feel free to drop me a note.... read more

Finally a new patch is out for both 2.4.18 and 2.4.19 kernels. This patch features the clean page adaptability, which disables clean page compression when necessary. It also has some bug fixes and cleanups, as always.

Note
http://sourceforge.net/project/shownotes.php?release_id=50313
http://sourceforge.net/project/shownotes.php?release_id=51703

Download
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.18-0.24pre5.bz2?download
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.19-0.24pre5.bz2?download

Finally the home page was updated. Now the directions to install the compressed cache patch matches the options available in the last version. Cache design section has been marked as outdated (and the excerpt on the home page removed). There are links to the patchset that include compressed cache.

It's available a set of performance statistics for the 0.24pre4 version. This set includes:

- DBench
- Fillmem (memtest)
- Linux Kernel Compilation
- MUMmer Scientific Program
- Open Source Database Benchmark

We have very good results with those tests, check them out!

http://linuxcompressed.sourceforge.net/statistics/

Compressed caching is a new level in the virtual memory hierarchy, where pages are stored in some compressed format, decreasing the number of page faults that are serviced by slow hard disks. The same version as 2.4.18-0.24pre4, now for 2.4.19 Linux Kernel.

Details:
http://sourceforge.net/project/shownotes.php?release_id=48824

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.19-0.24pre4.bz2?download

After a long development, a new patch is out with some great new features. In this version, you have adaptivity implementation, which will allow compressed cache to try to resize itself to the size it has the best benefits. You will also find compressed swap, where compressed cache swaps out fragments in compressed format, in order to delay decompression (and even avoid, if not needed). Along with compressed swap, there is compacted swap, feature that writes out many fragments to a single disk block, reducing the number of write submissions. Many other changes, including features, cleanups and bug fixes are in this patch. Give it a try.... read more

New patch is out. Besides many bug fixes, this version features full support for preempted kernels. The last tests run on this kernel (comp cache with page cache support + preempt + lockbreak) have been running for a long time, showing to be stable.

Notes:
http://sourceforge.net/project/shownotes.php?release_id=43957

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.18-0.24pre2.bz2?download... read more

New patch is out, featuring the support for double page sizes. That feature improves efficience of compresed cache even when the compression ratio isn't very good. There is also a bunch of bug fixes, cleanups and the first locking scheme to make compressed cache work with preempted kernels and on SMP systems. So far, there is only support for compressed cache without on demand resizing and without page cache support.... read more

This new version features a much more stable version
of compressed cache code, with many bugs which
have been fixed in the last 9 pre versions.

New online data about compressed cache is provided
with new two /proc entries: comp_cache_hist and
comp_cache_frag. The former shows how many
fragments are stored in compressed cache, so you
can check how it is distributed throught the
several pages of compressed cache. The latter shows
the fragmentation within the compressed cache. It
shows the free space (space that can be used right
away) and the fragmented space (space that is free,
but isn't contiguos, so it needs to compact the
fragments). The already existing comp_cache_stat
proc entry now shows more info about compressed
cache. Its output has been redesigned.... read more

Along with #kernelnewbies, we decided to move our IRC channel to irc.oftc.net server too.

Rodrigo

Cleanups and some important bug fixes needed to release the 0.23 final version. Among the bug fixes, an potential oops and a potential hang that could be caused by submitting errouneously an IO operation to the disk in an allocation code path that wouldn't allow it.

Notes
http://sourceforge.net/project/shownotes.php?release_id=98208

Download
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.18-0.23pre9.bz2?download... read more

There were some rejects with the original patch because of two modified files from vanilla tree. It has been rediffed and now it is supposed to apply cleanly. Download it again if you have had these troubles.

Rodrigo

Besides Linux Kernel compilation, we performed some tests with a scientific application named MUMmer, that shows a very intensive use of CPU and memory to do its computation. In this case, compressed cache performed very well, showing gains up to 45% in the completion time.

http://linuxcompressed.sourceforge.net/statistics/0.23pre8_mummer/

New statistics are available, now for the latest version, the 0.23pre8. There have been many improvements in the implementation since the 0.23pre4, that resulted in very good results (saves up to 36% in the kernel compilation time). Check them out.

http://linuxcompressed.sourceforge.net/statistics/0.23pre8_kernel/

Another unexpected pre is out. Let's hope this is the last one before 0.23 final. This patch features lots of bug fixes as well as fine tunings to make compressed cache performance improve. It also adds an option to resize compressed cache on demand. This means that compressed cache will only grow when actually needed and as soon as it is unneeded, it will shrink. The compsize= parameter, in this case, indicates the maximum size of the compressed cache.... read more

A new patch is available. A simple patch that fixes lots of bugs, most of them reported by Marc-Christian Petersen (thanks for the help). Now it is not supposed to freeze nor end up in an oops (like reported on our development mailing list).

Notes:
http://sourceforge.net/project/shownotes.php?release_id=51703

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.18-0.23pre7.bz2?download... read more

A new patch is out. If no major problem shows up, it will become 0.23 final. It features support for storing pages with buffers in compressed cache, which will avoid part of disk IO we were doing due to the pressure on the uncompressed cache. That feature turns the space reserved for compressed cache into an useful space for pages with buffers too, in spite of not compressing this kind of pages. More details in the link below.... read more

This patch features the support for LZO compression algorithm added back.

There's also a new /proc entry showing what's going on inside the compressed cache. This entry (comp_cache_hist) displays the free space histogram for all compressed cache pages, besides how many fragments are stored in these pages. Very useful data.

Some bugs were fixed. In particular, a bug that was hit under UML which could cause FS corruption and bug in swap buffers management that could hang the machine. About User Mode Linux, its author, Jeff Dike, fixed a bug that I reported concerning swap addresses, so now we can use compressed cache under UML (tested up to version 2.4.18-30um). ... read more

New statistics are available for kernel compilation on 0.23pre4 version of compressed cache. It shows huge improvements in comparison to the last statistics done for kernel compilation. Check it out.

http://linuxcompressed.sourceforge.net/statistics/0.23pre4_kernel/

A new patch featuring some cleanups and performance improvements is out. Its performance features are related to the watermarks used by VM system to decide how many free pages have to be available to the system at any moment. There's also a bug fix and improvements regarding swap buffer.

Notes:
http://sourceforge.net/project/shownotes.php?release_id=45561

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.18-0.23pre4.bz2

New patch is out. It has some bug fixes and the only new feature is the correct accountancy of memory usage of uncompressed and compressed cache. There's even a new output in /proc/meminfo with some compressed cache info.

Notes:
http://sourceforge.net/project/shownotes.php?release_id=48720

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.18-0.23pre3.bz2

This version features a first non-functional version of compressed cache automatic adaptivity to the system behaviour. It also has many changes aiming to fix the performance drop we have in linux kernel compilation test (check statistics for 0.23pre1 on our web site). Our analysis isn't complete and more changes are likely to go since a huge percentage of CPU is still not being used. Anyway, the current changes improve compressed cache a lot, mainly compressed cache support for page cache, and it already works much better in that scenario. ... read more

This patch is an attempt to reduce overhead introduced by Compressed Cache when it's not used (for example, dbench 1 case). It also improves allocation handling, fixes vswap allocation and a couple of other bugs.

Notes:
https://sourceforge.net/project/shownotes.php?release_id=43804

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.18-0.23pre1.bz2