Re: [lc-devel] Dump of my brain real quick

[Rodrigo S. de C. <rc...@im...>]

On Fri, Oct 10, 2003 at 08:47:29PM -0400, John R Moser wrote:
> I'm very interested in this project.  Having no job and 256 MB RAM,
> running Gnome2, and a ton of crap, and having lost a 40 gig HDD
> recently, I'm kind of in need of RAM.  At any rate, here's some of
> the stuff I've been thinking:

Great, you have a great motive to help us making this project work.

> 1) I'm not getting the 2.4.20 patch from CVS to actually work right.
> X won't run (I have nvidia kernel module), although it does on
> 2.4.21 with the same setup (which I slammed the .19 patch into by
> hand).

I´ve been told of instability problems with 2.4.20. I would tell you
to try 2.4.21 if you hadn´t already done it.

How stable 2.4.21 is (when CC is enabled)? Are you testing with normal
workload or with specific tests (kernel compilation and so forth)?

> 2) I'm trying to throw in a configure option to select the default
> compression algorithm, and also trying to set up the thing to allow me to 
> select anywhere from single to 64x pages, in steps of powers of 2, for my 
> page size.  Need a working patch to test it though x.x

There is an option to select compression algorithm, but only at boot
time, as you mention below.

I am not sure a CC with 64Kb pages will turn out to have a good
performance, in particular due to fragments (compressed pages)
compaction.

> 3) I've examined the compalg= code and it looks like you actually
> have to enter the algorithm_idx number for the alg :/ I can rewrite
> that to use strcmp against "lzo", "wkdm", and "wk4x4", and go with
> the default if not.  Don't really like having to know the internals
> to use it, especially since the algorithm indexes can change for no
> apparent reason.  For example, you could add "gzip", and I'd rather
> just type "gzip" than "3".

Great idea, I would love to see it implemented.

> 4) It'd be nice to be able to enable/disable compressed page cache while 
> running, and change algorithms.  The innards would need a slight
> modification; each alg would have to be initialized upon first use,
> and each compressed fragment/page would need to have the idx of the
> algorithm it's using stored within it.  I can try to work on this
> but I can't make any guarenteed commitment to the projet.  The
> code's pretty easy to grok though.

I had this feature working a while back, and the implementation was
actually what you mentioned above. I guess I removed it due to the
swap compression, to avoid the overhead of storing the compression alg
within each swap entry. That´s what I think it would be its major
impact on the code. You can check earlier versions of the code to know
how it was implemented, and we may discuss it further.

> 5) Some form of dynamic algorithm choice should be used.  I'm lacking in 
> ideas for this.  The best I can say is the following:
> 
> - When a page is initially compressed, compress it with each algorithm 
> and compare their relative space/speed tradeoffs
> - Get the slope by 'plotting' X as time and Y as percentage saved
> - Order these slopes from greatest to least
> - Find the last slope in the set that is greater than a minimum
> - Use that algorithm initially
> - Every N compresses (make N a large number like 10240) rerun this check 
> against the block (OPTION)
> - Have N adjust for each block dynamically; make sure they're not getting 
> this check more than once within a certain time interval and if they are, 
> grow the number.  (OPTION)  Similarly, if it's getting run less than once 
> every X time intervals, shrink N.  (OPTION)

I don´t think deciding the algorithm only once is a great idea, nor
compress every page with each algorithm. However, doing this analysis
every N compresses (or time interval) is really great, and that´s
exactly I though when you mentioned the first item. :-) In my opinion,
it is worth implementing it.

> 6) Splitting out ccache's algs into modules would be nice.  Particularly, 
> the modules should be rewritten as library functions (like zlib is),
> and ccache should access those.  That way other compression modules
> can be added dynamically.  The ccache dynalg module described in #5
> should be compiled as a module too to facilitate fast testing of new
> algs.

Indeed that should be tried. We could design a way for modules to
registers themselves in the CC, then you can load up as many modules
as you want, and CC wouldn´t have to be aware of them, as long as they
implemented an API. That would be perfect, but it would require quite
a work. Some compression algorithms would have to be rewritten to
avoid depending on some data passed as parameters, and so forth, ie,
make them compliant to a common compression interface for kernel
pages.

> Well that's about it for me.

Some major changes to add to this list:

- Make it SMP-safe

  o You mention something related to the spinlocks in a later email,
    so you probably gave a thought about it. In order to have any
    chance to be more widespread, we must improve its code to be
    SMP-safe in order to make use of higher CPU power SMP systems do
    have.

- Port it to rmap, preferably 2.6

  o  I am not sure it is so hard to do. I had a brief look at 2.6 code
     and I guess we may be able to do that. The challenge here is to
     make use of rmap in order to gather process information and use
     this information to improve CC behaviour.

- PDAs

  o  Port it a PDA architecture and measure CC benefit on such a
     system. This would require improvement of virtual swap structure
     to make Linux to swap out pages when we don´t have swap
     available.


Best regards,
-- 
Rodrigo