[lc-devel] Brain dump reply

[John R M. <jm...@st...>]

Sorry, i can't send mail from Mozilla . . . my smtp server won't take a 
relay request, so I have to use pine and shell in to the server.  Err.  
Reverse taht.  Shell, then pine.

Anyawy.

All my testing is done with me running.  That's it, no excess workloads 
or stuff.  Although on my laptop i've been testing by passing mem=64M or 
mem=128M (I have more RAM than i can use).  THE PATCH NOW WORKS!! I will 
put it in the patches section.  It's just preliminary separation and 
there's a few crappy hacks, also it seems to occasionally use a random 
algorithm index adn wind up using the compiled-in default instead of what 
compalg= was passed.  I put a simple hack around that in.  FIXME's are 
spewed right before each of these.  YOU WILL SEE IT WHEN CCACHE GETS 
COMPILED.  Guarenteed.  if you're looking at the screen you will see it.

1) 2.4.21 is a bane.  I've never got even the vanilla .21 to run for 24 hours
straight without crashing.  Ccache was no different (no less nor any more
stable).  I've always found .20 to stay up for . ..  well. . . I have to
reboot to take it down.  It won't die by itself.  My laptop is running .20
fine with ccache (WITH my hacks, which now WORK)

2)  Actually I've been using 16x (64KB) pages, and have no problems 
except when I'm under extremely heavy load.  It becomes noticable that 
the system freezes for about 50-100 mS once in a while.  I'll stay away 
from 64x (256KB) pages until I have a 10 Ghz processor ^_~  I use wkdm 
because it gets same or better than wk4x4.  LZO IS CURRENTLY 
UNSTABLE!!!!!!!!!!!!!!!  The box crashes when using LZO.  It can't 
decompress all the time.  Not something I know how to do (maybe something 
I broke, though not likely).

3)  It works.  Unfortunately, there's other bugs that cause a random alg 
to be used once in a while.  I've got bad hacks in to squelch this, and 
there's FIXME's near them, so read the warnings spewed by proc.c at 
compile itme.  Until this is fixed, I can't impliment a sysctl to change 
the alg on the fly, or safely use multiple algs at once.

4)  I've got the preliminaries set up.  It works fine, but there's a bug 
that I've mentioned a ton of times already.  Also I don't know what I'm 
doing much.  I don't know how to store the alg used in the swap page but 
it's TWO BYTES.  You can sacrifice a TWO BYTE overhead.  Yes I'm 
excessive but it's two bytes and you save more than that from the 
compression if you save any (WKDM averages 50% or more, 50% swap and 
70-80% page cache).  You'll have to clean that up behind me.

5)  I'm not implimenting this ^_^;  You can though.  I'd like to see it.

6)  I'm coming closer to that as well.  I've got the stuff divided out a 
bit and i'll put the current patch up (more to guard against a disk crash 
than to show any sort of improvement.  I've added bugs and bad hacks but 
it's stable somehow o.O).  I'm going to next work on dividing out the 
init algs for each algorithm into their source files and headers; and 
creating a registration system.  Ccache will on its own initialization 
request that these 3 default modules register themselves for now.

As for the following:

SMP
spin_lock won't halt other processors?  Hrm.  The problem I see is when 
ccache initializes an algorithm.  Separated data can be used for each 
algorithm easily.  The problem is the linked list I have in there.  
Scanning it should be safe to do twice.  Adding an algorithm to it during 
the scan would be annoying to track, however.  If I was sure that the 
only thing ever initializing algorithms was comp_cache_demand_alg_entry() 
I could probably handle it.  I'm afraid I can't secure it down to a 
single line of code though.  In any case, unless I could halt the other 
CPU from continuing while a function is in use if it calls that function, 
I can't make it perfectly secure.

What I need is a guarentee that two lines of code will be executed only 
once at the same time.  Basicly,

if (alg_registration_entry->enabled)
	return;
alg_registration_entry->enabled = 1;

If that if branches past the return, then I need it to execute the next 
line before any other CPU gets to that exact if statement.  Possibly:

static int skip_out = 0;
if (skip_out)
	return NULL;
skip_out++;
if (alg_registration_entry->enabled || skip_out > 1) {
	skip_out--;
	return NULL;
}
alg_registration_entry->enabled = 1;
<initialize the algorithm>
skip_out--;
return new_alg_entry;

The model illustrated in the above pseudocode may allow me to initialize an
algorithm (i.e.  insert it into the running algorithms list) without needing
a spin_lock and without taking the chance of having the function run twice in
parallel.  Other than that, I can probably set up the compression 
functions to be safe to run multiple times in parallel.  In the event 
that the algorithm can't be initialized (i.e. this skips out), the page 
would simply not be compressed, or would grab a random loaded alg and 
compress with that.

RMAP
UH.  I've just started hacking 2.4 code.  If you're going to go to rmap, 
try to get it stable in 2.4 first, and gimme a week or two to finish the 
modifications I'm doing on the current code.  I'll probably leave a few 
bad hacks and some dirty code around when i'm done so you'll have to 
clean up behind me, but I'll be careful not to do that too much.

PDAs
http://handhelds.org/  I use GPE.  I would love to see WKdm running on 
it, since I have no swap.




That's about it.  My next major step is to separate the init functions 
for lzo, WKdm, and WK4x4 out, set up the registration functions, and have 
ccache's init function request the default set (lzo, wkdm, wk4x4) 
register with it.  Once that's in place, I'll try to rewrite for dynamic 
data allocation and try to make it safe for SMP.  I'm swinging my fists 
blindly here though.  I'll get SOME kind of progress, but it might not go 
all the way.

--Bluefox Icy