linuxcompressed-devel — Linux Compressed Cache Development Mailing List

Re: [lc-devel] Patch for 2.4.18 or newer

[Marcelo T. <ma...@co...>]

On Fri, 1 Mar 2002, Marcelo Tosatti wrote:

> 
> 
> On Fri, 1 Mar 2002 mc...@li... wrote:
> 
> > Hi there,
> > 
> > do you plan to release a new patch for 2.4.18 or newer in the next days?
> > or maybe today/tomorrow ? :-)
> 
> Nope. I've released 2.4.19-pre which contains the missing patch.

Argh, _sorry_ people. I thought this mail was another "are you going to
release another 2.4.18?" addressed to _me_.

I have to stop trying to read things really fast. :)

Re: [lc-devel] Patch for 2.4.18 or newer

[Marcelo T. <ma...@co...>]

On Fri, 1 Mar 2002 mc...@li... wrote:

> Hi there,
> 
> do you plan to release a new patch for 2.4.18 or newer in the next days?
> or maybe today/tomorrow ? :-)

Nope. I've released 2.4.19-pre which contains the missing patch.

Re: [lc-devel] Patch for 2.4.18 or newer

[Rodrigo S. de C. <rc...@im...>]

Hi Marc,

On Fri, Mar 01, 2002 at 02:59:23PM +0100, mc...@li... wrote:
> do you plan to release a new patch for 2.4.18 or newer in the next days?
> or maybe today/tomorrow ? :-)

I updated my pool to 2.4.18 on Wednesday, but had some bugs to solve
first before release a new patch. Once I finally solved a critical bug
that showed up running dbench, I will release a 0.22pre6 version for
2.4.18 later today :-) Wait for updates on our project page.

As soon as I fix the adaptivity stuff I broke when added page cache
support, I will release a 0.22 final.

Best regards,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] Patch for 2.4.18 or newer

[<mc...@li...>]

Hi there,

do you plan to release a new patch for 2.4.18 or newer in the next days?
or maybe today/tomorrow ? :-)

Kind regards,
	Marc

Re: [lc-devel] My first real test and my first real problem :-(

[Rodrigo S. de C. <rc...@im...>]

On Thu, Feb 14, 2002 at 04:02:10PM +0100, CIARROCCHI Paolo wrote:
[snip]

> Then I applied the patch 0.21 (with 128MiB and 16MiB)
> With a simple ./fillmem 400
> malloc/memset 001/240
> malloc/memset 002/240
> malloc/memset 003/240
> malloc/memset 004/240
> malloc/memset 005/240
> malloc/memset 006/240
> malloc/memset 007/240
> [snip]
> malloc/memset 185/240
> malloc/memset 186/240
> mall
> lines 1-24 <1>Unable to handle kernel paging request at virtual address
> 0d77e2ff printing eip:
> c0130060

Could you please decode this oops with ksymoops? Or at least sending
the System.map related to this oops (ie, from the kernel you booted
when oopsed)?

Is it reproducible?

Thanks,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] My first real test and my first real problem :-(

[CIARROCCHI P. <Pao...@om...>]

Hi all,
yesterday I run a script very similar at this:

#/bin/bash
cmd=${0##*/}
kern=$(uname -r)
log=${cmd}-${kern}.log
echo $log

date >$log
echo >>$log 

for i in `seq 20 20 400`;
do echo 'memory ' $i >>$log; 
time ./fillmem $i; 
cat /proc/comp_cache_stat >>$log; 
done 2>>$log

The difference are only related to the output.

No problem with a stock 2.4.17,
here it goes the result:

memory  20
0.15user 0.11system 0:00.27elapsed 95%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+10275minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  40
0.49user 0.39system 0:00.87elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+20535minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  60
0.69user 0.68system 0:01.37elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+30795minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  80
0.68user 0.52system 0:01.20elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+41055minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  100
1.24user 1.13system 0:02.36elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+51315minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  120
0.78user 1.32system 0:02.11elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+61575minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  140
1.63user 1.58system 0:03.20elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+71835minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  160
1.47user 1.45system 0:02.92elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+82095minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  180
2.05user 2.15system 0:04.20elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+92355minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  200
2.20user 2.42system 0:04.61elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+102615minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  220
2.53user 2.49system 0:05.01elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+112875minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  240
1.78user 2.95system 0:07.71elapsed 61%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (403major+130065minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  260
2.18user 2.90system 0:12.80elapsed 39%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (841major+146818minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  280
2.66user 3.25system 0:18.66elapsed 31%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (1337major+165312minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  300
2.45user 3.40system 0:24.72elapsed 23%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (2008major+183771minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  320
2.47user 4.37system 0:31.51elapsed 21%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (2653major+204072minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  340
2.12user 4.17system 0:38.78elapsed 16%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (3359major+224703minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  360
2.94user 5.11system 0:45.81elapsed 17%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (3970major+244526minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  380
2.89user 5.28system 0:51.39elapsed 15%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (4572major+264104minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory
memory  400
3.46user 5.53system 0:58.58elapsed 15%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (5267major+283273minor)pagefaults 0swaps
cat: /proc/comp_cache_stat: No such file or directory

Then I applied the patch 0.21 (with 128MiB and 16MiB)
With a simple ./fillmem 400
malloc/memset 001/240
malloc/memset 002/240
malloc/memset 003/240
malloc/memset 004/240
malloc/memset 005/240
malloc/memset 006/240
malloc/memset 007/240
[snip]
malloc/memset 185/240
malloc/memset 186/240
mall
lines 1-24 <1>Unable to handle kernel paging request at virtual address
0d77e2ff printing eip:
c0130060
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c0130060>]    Not tainted
EFLAGS: 00010206
eax: 0d77e2ff   ebx: 1e1a2d95   ecx: 00001000   edx: c02cca94
esi: 0d77e2ff   edi: 0000003f   ebp: 000000ff   esp: ce8bfd6c
ds: 0018   es: 0018   ss: 0018
Process fillmem (pid: 1092, stackpage=ce8bf000)
Stack: c02c9da0 c0271080 00000005 00000000 00001068 00000000 00001000
1e1a2d95
       ceefe000 00000020 c13bbf80 c012eff0 c13bbf80 00001000 ce8bfdac
ce8be000
              00000000 00000001 00000000 ce8bfe30 20000001 00001068 0000b9de
c011c226
       Call Trace: [<c012eff0>] [<c011c226>] [<c012a2dc>] [<c012a532>]
[<c012a59c>]
          [<c012ae43>] [<c012b0ab>] [<c01223a4>] [<c0121e69>] [<c0122516>]
[<c011245a>]
      [<c010b4fd>] [<c0112fa1>] [<c01122d0>] [<c0106fec>]

      Code: 8b 06 85 c0 75 1a 66 81 7e 06 00 10 0f 84 9f 00 00 00 0f 0b

Please find attached the output of my script with 128MiB and 16MiB of
compressed cache,
as you can see there is something that is not working.

I had also a freez (non ALT+SYS RQ) under X.

do you have any suggestions?

In any case, I was not able to use correctly my machine under X,
segmentation faul using
 a lot of program.

RE: [lc-devel] Statistics: 0.21

[CIARROCCHI P. <Pao...@om...>]

> Just don't test intensively manual adaptivity using 0.22pre2, there a
> number of issues related to concurrency that I already fixed. A new
> 0.22pre3 will be available today or tomorrow.

Ok, I patched my kernel with 0.21, If I can this night I will perform
a test with this kernel and tomorrow I'll post the result.

> > Now, my questions:
> > + How many compressed cache I have to configure and how I can do it?
> 
> You can configure compressed cache initial size in kernel config
> options (in make config/menuconfig/xconfig). Another option is to use
> the kernel parameter in the boot time.
> 
> For example, starting the kernel image with compsize=4096 will boot
> the Linux kernel with a compressed cache of 16 MiB. 

Great, so I can compile the kernel with 128 Mib of compressed cache and
then force at the boot another value.
 
> Notice that the unit used to configure the compressed cache size is
> number of memory pages. Recall that a memory page in i386 is 4096
> bytes.

Ok, thanks.
 
> > + What kind of test can I run? Mmap001 300 should be ok?
> 
> You can test fillmem or mmap001 using a value at least near to fill
> all memory. If you have 256, a range of values from 200-350 would be
> nice to test. You can do it in steps of 10, 20, 30 MiB (it's up to
> you).

I'm gonna use the script I posted yesterday, then I think I'll modified it 
trying to have better report.

 
> It would also be great if you could test something closer to a normal
> workload. For example, loading the same set of programs and then
> measure the time taken to execute a program (eg. to load netscape
> browser). If you have some idea of a usual workload for you, it would
> be great to devise some sort of test to know how well the compressed
> cache could be helping this scenario.

Ok, I prefer test with "numerical" results, but I'll try to measure 
the time taken to execute program like netscape or GIMP.

> Oh, try to gather the compression statistics too (cat
> /proc/comp_cache_stat) after your tests. I forgot to do that when run
> the last tests.

Ok, I'll add it at my script.
 
> > If you could point me in the direction to look, I will perform a few
> > test using my laptop.
> 
> Thank you very much for the help!

My pleasure, 
the only problem if that I have few free time,
the work in Vodafone is very "eating time"... but with the laptop I
can perform the test while I coming back home :-)

Regards,
Paolo Ciarrocchi

Re: [lc-devel] Test

[Rodrigo S. de C. <rc...@im...>]

On Tue, Feb 12, 2002 at 08:47:11AM +0100, CIARROCCHI Paolo wrote:
> Yesterday I wrote a little script in order to try to compare a
>  2.4.17 stock kernel with a patched one.
> This is the script,
> 
> #/bin/sh
> for i in `seq 20 20 400`;do echo 'memory ' $i >>`uname -r`_stock;time
> ./fillmem $i; done 2>>`uname -r`_stock
> 
> Looks good for you?

Yes.

> Should I try this script also with a patched kernel with 128M of
> compressed cache?

IMO that's nice. In the case you have enough time, do it with smaller
caches too (32MiB and 64MiB, for example).

> If you need it, I can provide you the result of the above script on
> my machine.

It would be great since you have a faster machine than the one I used
for my last tests.

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Statistics: 0.21

[Rodrigo S. de C. <rc...@im...>]

Hi Paolo,

On Mon, Feb 11, 2002 at 10:10:12AM +0100, CIARROCCHI Paolo wrote:
> My laptop is a HP Omnibook 6000 with 256 Mb RAM, running a 2.4.17
> stock kernel.

Cool.

> I have downloaded the Fillmem/Mmap001, tomorrow I will apply your
> patch.

Just don't test intensively manual adaptivity using 0.22pre2, there a
number of issues related to concurrency that I already fixed. A new
0.22pre3 will be available today or tomorrow.

> Now, my questions:
> + How many compressed cache I have to configure and how I can do it?

You can configure compressed cache initial size in kernel config
options (in make config/menuconfig/xconfig). Another option is to use
the kernel parameter in the boot time.

For example, starting the kernel image with compsize=4096 will boot
the Linux kernel with a compressed cache of 16 MiB. 

Notice that the unit used to configure the compressed cache size is
number of memory pages. Recall that a memory page in i386 is 4096
bytes.

> + What kind of test can I run? Mmap001 300 should be ok?

You can test fillmem or mmap001 using a value at least near to fill
all memory. If you have 256, a range of values from 200-350 would be
nice to test. You can do it in steps of 10, 20, 30 MiB (it's up to
you).

It would also be great if you could test something closer to a normal
workload. For example, loading the same set of programs and then
measure the time taken to execute a program (eg. to load netscape
browser). If you have some idea of a usual workload for you, it would
be great to devise some sort of test to know how well the compressed
cache could be helping this scenario.

Oh, try to gather the compression statistics too (cat
/proc/comp_cache_stat) after your tests. I forgot to do that when run
the last tests.

> If you could point me in the direction to look, I will perform a few
> test using my laptop.

Thank you very much for the help!

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] Test

[CIARROCCHI P. <Pao...@om...>]

Hi all,
Yesterday I wrote a little script in order to try to compare a
 2.4.17 stock kernel with a patched one.
This is the script,

#/bin/sh
for i in `seq 20 20 400`;do echo 'memory ' $i >>`uname -r`_stock;time
./fillmem $i; done 2>>`uname -r`_stock

Looks good for you?

Should I try this script also with a patched kernel with 128M of compressed
cache?

If you need it, I can provide you the result of the above script on my
machine.

Regards,
Paolo Ciarrocchi

Re: [lc-devel] broken link

[Rodrigo S. de C. <rc...@im...>]

On Mon, Feb 11, 2002 at 10:12:56AM +0100, CIARROCCHI Paolo wrote:
> Hi all,
> in the homepage the download link is broken. 
> 
> (Fev 07th 2002) Patch 2.4.17-0.22pre2 (news, download).

Fixed. Thanks.

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] broken link

[CIARROCCHI P. <Pao...@om...>]

Hi all,
in the homepage the download link is broken. 

(Fev 07th 2002) Patch 2.4.17-0.22pre2 (news, download).

--
		Paolo

RE: [lc-devel] Statistics: 0.21

[CIARROCCHI P. <Pao...@om...>]

Hi all,
I just finished to prepare my laptop.
My laptop is a HP Omnibook 6000 with 256 Mb RAM, running a 2.4.17 stock
kernel.
I have downloaded the Fillmem/Mmap001, tomorrow I will apply your patch.
Now, my questions:
+ How many compressed cache I have to configure and how I can do it?
+ What kind of test can I run? Mmap001 300 should be ok?

If you could point me in the direction to look, I will perform a few test 
using my laptop.

Regards,
Paolo Ciarrocchi  

RE: [lc-devel] Statistics: 0.21

[CIARROCCHI P. <Pao...@om...>]

> Comments are very welcome!
> 
> You can see the statistics at
> 
> http://linuxcompressed.sourceforge.net/statistics/
> 

Hi,
the statistics seems to be really interesting!
Really a good job. Again, how much time do you want to wait 
before post the pathc to lkml?
I think that now you should post the statistics and see what
 these guys think about this.

Just my 2 euro-ct...

			Paolo

[lc-devel] Statistics: 0.21

[Rodrigo S. de C. <rc...@im...>]

Hi there,

I ran some tests to have some performance statistics out of our 0.21
version of Compressed Cache. The selected tests were: fillmem/mmap001
(from memtest), Linux Kernel compilation and Dbench.

Kernel compilation is very interesting since this is an CPU intensive
and Dbench an IO intensive application. Besides that, both are widely
used for VM tests and couldn't miss on our test list.

I tested many Compressed Cache sizes (4, 8, 16 and 24) and plotted
graphics for every test in order to help understanding the data. Also
the CPU usage data (except for dbench case) was plotted.

In Linux Kernel compilation and Dbench tests, we didn't have very good
results. At the bootom of each test page, I tried to write an analysis
of mine on the performance causes and possible code improvements. The
analysis are very similar, stating something I missed so far: the high
importance of buffer/page cache (there's no support for these caches
in Compressed Cache).

Comments are very welcome!

You can see the statistics at

http://linuxcompressed.sourceforge.net/statistics/

Best regards,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] Benchmarks (fwd)

[Rodrigo S. de C. <rc...@im...>]

Hi,

Have anyone ever heard of this AIM benchmark suite? 

----- Forwarded message from Nigel Gamble <ni...@nr...> -----

Date:	Mon, 28 Jan 2002 08:56:25 -0800 (PST)
From: Nigel Gamble <ni...@nr...>
Subject: Re: Don't use dbench for benchmarks
To: "Richard B. Johnson" <ro...@ch...>
Cc: Alex Davis <ale...@ya...>,
	Daniel Phillips <phi...@bo...>,
	<lin...@vg...>

On Mon, 28 Jan 2002, Richard B. Johnson wrote:
> It seems that compiling the Linux Kernel while burning a CDROM gives
> a good check of "acceptable" performance. But, such operations are
> not "benchmarks". The trick is to create a benchmark that performs
> many "simultaneous" independent and co-dependent operations using
> I/O devices that everyone is likely to have. I haven't seen anything
> like this yet.
>
> Such a benchmark might have multiple tasks performing things like:
>
> (1)	Real Math on large arrays.
>
> (2)	Data-base indexed lookups.
>
> (3)	Data-base keys sorting.
>
> (4)	Small file I/O with multiple creations and deletions.
>
> (5)	Large file I/O operations with many seeks.
>
> (6)	Multiple "network" Client/Server tasks through loop-back.
>
> (7)	Simulated compiles by searching directory trees for
> 	"include" files, reading them and closing them, while
> 	performing string-searches to simulate compiler parsing.
>
> (8)	Two or more tasks communicating using shared-RAM. This
> 	can be a "nasty" performance hog, but tests the performance
> 	of threaded applications without having to write those
> 	applications.
>
> (9)	And more....
>
>
> These tasks would be given a "performance weighting value", a heuristic
> that relates to perceived overall performance.

It sounds like you are describing the Aim Benchmark suite, which has
been used for years to compare unix system performancem, and was
recently released under the GPL by Caldera.

See http://caldera.com/developers/community/contrib/aim.html

Nigel Gamble                                    ni...@nr...
Mountain View, CA, USA.                         http://www.nrg.org/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to maj...@vg...
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/



----- End forwarded message -----

-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Next items to be done

[Rodrigo S. de C. <rc...@im...>]

On Fri, Jan 25, 2002 at 10:37:54AM -0500, Dilma DaSilva wrote:
> Your TODO list makes sense. It mentions twice experiments and
> statistics for "workloads". I'm very curious to know which type of
> workload you're planning. I think it's useful to think about this even 
> before going much further with develpment. 

You are right, I think it's very sensible to think about that at this
moment. Most the tests done so far are from memtest suite, which are
not the "best" tests in my opinion (Rik van Riel stated that once on
an irc channel). But I think the main tests from this suite (fillmem
and mmap001) can still be part of the tests we can run to "fell" how
well the cache is going on.

Other tests that I had in mind are the ones used in Swap Compression
paper [1]. Their tests are: fft (fast Fourier transmation), sort,
simulator (simulator of a network of disks from theirs research group)
and xanim. that can replaced by something newer like mplayer to play
dvix, which makes intensive use of CPU and memory. I also think that
the chess program (sjeng) written by Gian-Carlo Pascutto can be a good
option.

The sort test (in randomized inputs) has showed a slowdown in Fred
Douglis' implementation [2]. Another tests that had horrible
performance in Douglis' compressed cache were applications based on
index engine for Gold Mailer (he aimed to show how well main-memory
database could benefit from the compression). His tests were to create
an index and to perform queries. The slowdown was between 20 to 40% in
these cases. Therefore I think we have to test a database application
(maybe even this application Douglis used) and the randomized sort.

To conclude, maybe a complex stuff like linux kernel compilation can
be nice to run and some sort of benchmark kit (lmbench?). 

Comments are welcome.

[1] R. Cerver, T. Cortes, Y. Becerra. Improving Application
Performance through Swap Compression. Universitat Politècnica de
Catalunya - Barcelona. 1999.

[2] Fred Douglis. The Compression Cache: Using On-Line Compression to
Extend Physical Memory. Matsushita Information Technology
Laboratory. 1993.

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] [PATCH] 2.4.17-0.21

[Rodrigo S. de C. <rc...@im...>]

On Fri, Jan 25, 2002 at 12:45:48PM -0200, Rodrigo Souza de Castro wrote:
> Download
> --------
> http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.21.bz2

On Friday I uploaded a buggy version of this patch, what was fixed in
a version available since Saturday afternoon. There was a bogus "if"
in swapin path which would cause a kernel BUG. I found that out when
OOM killer started killing my gcc in a "make -j" linux kernel
compilation in my humble test machine (p133 48Mb RAM) :-)

If somebody tested this version and have any comments, opinions or
suggestions, mainly regarding stability, I'd pleased to
know. Currently I am working on some statistics for this version that
will online soon.

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Next items to be done

[Dilma D. <di...@wa...>]

Rodrigo,

Your TODO list makes sense. It mentions twice experiments and
statistics for "workloads". I'm very curious to know which type of
workload you're planning. I think it's useful to think about this even 
before going much further with develpment. 

dilma

[lc-devel] Next items to be done

[Rodrigo S. de C. <rc...@im...>]

Hi,

There are some things that are on the top of my todo list and that I
want to do them asap. Don't forge that a (complete?) todo list is
always at: http://linuxcompressed.sourceforge.net/todo/.

Adaptivity
----------

I have to make these data structures adaptable along with the
compressed cache. Once these data structures are done, we will be able
to safely change compressed cache size manually and start testing
differente compressed cache sizes for different workloads. And
automatic adaptivity won't be a matter of changing very much the
current code (even though devise which and how to get the correct
parameters in the VM will be a hard work :-).

- Adaptable Hash Table

- Adaptable Vswap Address

- Option to disable compressed cache on-the-fly (something like "echo
0 > /proc/sys/vm/comp_cache/size")

SMP
---

To add SMP support, major surgeries in the code and changes in data
structures will be needed. I think that may take a while to be ready
and stable. It seems User Mode Linux will help us a lot since SMP
support has been heavily developed by UML team.

rmap port
---------

I already make use of some ideas (and some functions :-) present in
Rik van Riel's rmap VM and I think it will be great to check
Compressed Cache performance impact in this new VM.

Statistics
----------

It's time to do some tests and publish statistics on our web page,
trying different workloads and compressed cache sizes. I'd like to
have some help here, if possible.

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Swap lockup in 2.4.17pre7

[Rodrigo S. de C. <rc...@im...>]

On Fri, Jan 25, 2002 at 12:04:05AM +0800, David Chow wrote:
> After download the new pre7 patch for 2.4.17 . It seems I am having the
> same problem. The machine locks up when swap usage go to around 20MB .
> My machine is PIII having 512MB of swap space, runing 16384 pages of
> compressed cache with a total of 256MB of RAM . When under load with
> swap space usage around 10%, the machine will freeze, after freeze the
> harddisk will still scratch for a while and making noise.

David,

Could you please test with 0.21 final version I just released? I
noticed a stupid bug concerning vswap that can make things work over
there.

There is a variable to assure that vswap is conservative when
assigning new vswap address. The point is that we can't have many
vswap address that haven't been compressed, otherwise we may fill up
the compressed cache (since we can't predict how big the fragment will
be) and thus can't store this fragment. Well, this variable was
supposed to be signed, so we would have negative values e when its
value is below 0, we have to stop returning vswap address in some
scenarios. However I declared it as unsigned long (duh), so vswap
would return all available addresses if requested, what's totally
wrong. If many addresses are assigned, the likeliest is to panic (the
panic in mm/comp_cache/swap_out.c), locking up your machine. It's
fixed in 0.21.

I tested this patch for over 12 hours on a real machine and under UML,
swapping out over 90 MiB in the former and 32 MiB in the latter, and
it survived. :-)

Thanks,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] [PATCH] 2.4.17-0.21

[Rodrigo S. de C. <rc...@im...>]

Hi,

This major version features lots of chances that improved a lot
Compressed Cache code and that make us very glad to announce.

Adaptivity's been started (it was the main development in 0.21pre1 and
0.21pre2 versions), but the development focus was changed to
performance (what made us perform major surgeries). The code had been
showing a terrible performance when running with big Compressed Cache
sizes. We hope to make adaptivity work smoothly and to be fully
functional (at least manual adaptivity) in the next version but you
can already enjoy this partialy functional feature introduced here.

It has various cleanups and code rewrite, aiming performance and
clearness. It's likely that this is the version with the best
performance results ever and with the best code.

Swap out path and Swap buffer code were redone. This also applies to
Compressed Cache Free functions.

Hash table are now scaled following the compressed cache size and had
its hash function as well its allocation much improved.

AVL trees were removed, thus getting rid of avl_{insert,remove}
overheads.

OOM kill and allocation finally take into account the Compressed Cache
and the available size in there.

Virtual swap is now portable to other architectures. It makes sure
that we flush the cache and tlb when handling ptes.

Many bugs were also fixed, mainly in virtual swap code.

Check detailed changelog below for each testing patch released.

Release Notes
-------------
http://sourceforge.net/project/shownotes.php?release_id=71931

Download
--------
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.21.bz2

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Swap lockup in 2.4.17pre7

[Rodrigo S. de C. <rc...@im...>]

On Fri, Jan 25, 2002 at 12:04:05AM +0800, David Chow wrote:
> After download the new pre7 patch for 2.4.17 . It seems I am having the
> same problem. The machine locks up when swap usage go to around 20MB .
> My machine is PIII having 512MB of swap space, runing 16384 pages of
> compressed cache with a total of 256MB of RAM . When under load with
> swap space usage around 10%, the machine will freeze, after freeze the
> harddisk will still scratch for a while and making noise.

Please, compile with SysRQ enabled and then, when the machine freeze,
try to get a list of the current tasks (ALT-SysRq-t) and send it to
the list. If you have spare time to debug this problem, you could
still apply kdb [1] patch and check where it's hanging. :-)

PS: I assume this lockup does not happen on 2.4.17 vanilla. Am I
right?

[1] http://oss.sgi.com/projects/kdb/

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] Swap lockup in 2.4.17pre7

[David C. <dav...@rc...>]

After download the new pre7 patch for 2.4.17 . It seems I am having the
same problem. The machine locks up when swap usage go to around 20MB .
My machine is PIII having 512MB of swap space, runing 16384 pages of
compressed cache with a total of 256MB of RAM . When under load with
swap space usage around 10%, the machine will freeze, after freeze the
harddisk will still scratch for a while and making noise.

David

Re: [lc-devel] UML swap bug fixed

[David C. <dav...@rc...>]

On Fri, 2002-01-18 at 03:31, Rodrigo Souza de Castro wrote:
> On Thu, Jan 17, 2002 at 01:00:08PM +0800, David Chow wrote:
> > ?b ?g??, 2002-01-12 23:20, Rodrigo Souza de Castro ?g?D?G
> > > Regarding the bug your pointed out, I'd like you to test 0.21pre5
> > > version I released this week. The swapout path code was rewritten
and
> > > it's much simpler and efficient now. I tested the last code to
trigger
> > > this bug and couldn't make it hang. Neither could Livio. He tested
> > > under UML yesterday and got disappointed that he couldn't make the
> > > code crash. :-) In case you still have problems with the this
newer
> > > version, please let me know.
> > 
> > I have downloaded the latest 0.21pre6 . 
> > 
> > It still have problem nears out of memory when I have not turned on
> > swap. Do you think this is something not related with compressed
cache
> > or a vm bug from Linux?
> 
> Not sure. Can you reproduce this problem without compressed cache
> patch?
> 
My machine is a PIII533 using 16384 compcache pages with a total of
256Mb RAM. I am swapping to a local partition.

I've tested with the pre6, the bug is reproduceable. Everytime when my
swap starts to use up around 48 megs, then I start executed another
application (StarOffice 6.0) then the machine seems lockup, then I can
still hear from the harddrive being heavily loaded (scratching for a
while) after the lock. Then I'll have to press the reset after it does
not make more noise.

> > Usually when really out of memory, the oom killer should be
triggered
> > and will kill that process... my experiecne is the machine freezes.
> 
> Still hard to say. If you can't reproduce with vanilla, please test
> again with 0.21pre7 (there's a small fix for vswap). In the case the
> problem is still there, tell me exactly what's your configuration like
> and what you are trying to do (only allocation, for example, or
> running a normal desktop with browser, etc) so I may be able to
> reproduce it here.
> 
> Regards,
> -- 
> Rodrigo S. de Castro <rc...@im...>

David