linuxcompressed-devel — Linux Compressed Cache Development Mailing List

Re: [lc-devel] UML swap bug fixed

[Rodrigo S. de C. <rc...@im...>]

On Thu, Jan 17, 2002 at 01:00:08PM +0800, David Chow wrote:
> ?b ?g??, 2002-01-12 23:20, Rodrigo Souza de Castro ?g?D?G
> > Regarding the bug your pointed out, I'd like you to test 0.21pre5
> > version I released this week. The swapout path code was rewritten and
> > it's much simpler and efficient now. I tested the last code to trigger
> > this bug and couldn't make it hang. Neither could Livio. He tested
> > under UML yesterday and got disappointed that he couldn't make the
> > code crash. :-) In case you still have problems with the this newer
> > version, please let me know.
> 
> I have downloaded the latest 0.21pre6 . 
> 
> It still have problem nears out of memory when I have not turned on
> swap. Do you think this is something not related with compressed cache
> or a vm bug from Linux?

Not sure. Can you reproduce this problem without compressed cache
patch?

> Usually when really out of memory, the oom killer should be triggered
> and will kill that process... my experiecne is the machine freezes.

Still hard to say. If you can't reproduce with vanilla, please test
again with 0.21pre7 (there's a small fix for vswap). In the case the
problem is still there, tell me exactly what's your configuration like
and what you are trying to do (only allocation, for example, or
running a normal desktop with browser, etc) so I may be able to
reproduce it here.

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Kernel command line for compressed cache size

[Rodrigo S. de C. <rc...@im...>]

On Thu, Jan 17, 2002 at 02:44:56PM -0200, Livio Baldini Soares wrote:
> David Chow writes:
> > I think of a flexible way of using compressed cache in production
> > systems is that allow to overide the default compressed cache size using
> > kernel command parameters. How's that?
> 
>   Well, there has been support for kernel command parameter for some
> time now. You can specify the size of the compressed cache (use 0 to
> disable it), with the "compsize=" parameter.
> 
>   So you can enable the compressed cache with whichever size you
> desire. There is a restriction however, the number of pages has to be
> a power of 2! Rodrigo is currently working on somethings and this
> restriction is something with high priority on his TODO list.

This restriction was due to a simple hash table implementation. I
improved the hash table function and allocation yesterday and now you
can enter any number of pages, but only in menuconfig (I forgot this
old kernel parameter that I don't use often). The new code was
released as 0.21pre7 version. The kernel parameter fix is already in
my pool and I will commit it asap.

>   Another thing, which is a new feature, and not very tested yet, is
> the proc entry: /proc/sys/vm/comp_cache/size
> 
> Try:
> # cat /proc/sys/vm/comp_cache/size
> 
>   And you get the number of pages reserved for the comp_cache. The new
> feature is shrinking/growing the cache with an echo, for example:
> 
> # echo 512 > /proc/sys/vm/comp_cache/size
> 
>   Will change the current size to 512 pages.

This feature is really nice, but I couldn't test a lot (nor improve
it). I noticed lots of performance bottleneck these last weeks and
have been trying to improve cache performance since then. Expect a
much better code for 0.21 that is about to be released.

> Another restriction is that this dynamic change in the cache size
> can never be greater than the size chosen with the "compsize="
> kernel option, or if none is given, the size chosen during the
> configuration of the kernel before compilation.

That's right. The option you enter in kernel parameter or in
menuconfig will be only the initial compressed cache size in a near
future.

>   Rodrigo is also trying to remove this limitation, but some work has
> to be done before this can happen (dynamically changing the sizes of
> the hash tables used, for example).

I think SMP support will be next great feature before get rid of this
limitation. I want to clean up the code (improving performance) and
make all the necessary surgeries now, and only then starting
benchmarking adaptivity and adding parameters for it. Le't hope SMP
support will not be painful since it may need some great changes.
 
>   This is great because you can tweak the size reserved for the
> compressed cache on-line, and try testing it to see which size works
> best for your workload. Try it out!

Anyway, thanks for your suggestion, David.

[]'s
-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Kernel command line for compressed cache size

[Livio B. S. <li...@im...>]

Hello,

David Chow writes:
> Dear all,
> 
> I think of a flexible way of using compressed cache in production
> systems is that allow to overide the default compressed cache size using
> kernel command parameters. How's that?

  Well, there has been support for kernel command parameter for some
time now. You can specify the size of the compressed cache (use 0 to
disable it), with the "compsize=" parameter.

  So you can enable the compressed cache with whichever size you
desire. There is a restriction however, the number of pages has to be
a power of 2! Rodrigo is currently working on somethings and this
restriction is something with high priority on his TODO list.

  Another thing, which is a new feature, and not very tested yet, is
the proc entry: /proc/sys/vm/comp_cache/size

Try:
# cat /proc/sys/vm/comp_cache/size

  And you get the number of pages reserved for the comp_cache. The new
feature is shrinking/growing the cache with an echo, for example:

# echo 512 > /proc/sys/vm/comp_cache/size

  Will change the current size to 512 pages. Another restriction is
that this dynamic change in the cache size can never be greater than
the size chosen with the "compsize=" kernel option, or if none is
given, the size chosen during the configuration of the kernel before
compilation.

  Rodrigo is also trying to remove this limitation, but some work has
to be done before this can happen (dynamically changing the sizes of
the hash tables used, for example).

  This is great because you can tweak the size reserved for the
compressed cache on-line, and try testing it to see which size works
best for your workload. Try it out!

  best regards,

--  
  Livio <li...@im...>

[lc-devel] Kernel command line for compressed cache size

[David C. <dav...@rc...>]

Dear all,

I think of a flexible way of using compressed cache in production
systems is that allow to overide the default compressed cache size using
kernel command parameters. How's that?

David

Re: [lc-devel] UML swap bug fixed

[David C. <dav...@sh...>]

=A6b =B6g=A4=BB, 2002-01-12 23:20, Rodrigo Souza de Castro =BCg=B9D=A1G
> On Sat, Jan 12, 2002 at 05:50:54PM +0800, David Chow wrote:
> > Is this the bug that I point out in my previous mail to the list? Or it=
=20
> > is caused by a different issue? Please clarify, since I don't have=20
> > enough time to understand all the code at the moment. Thanks.
>=20
> I forwarded an email to this list on Dec 12th concerning a bug in
> User-Mode-Linux that made compressed cache not work. To be clear, with
> compressed cache code this bug was triggered easier. And now it's
> finally fixed and we can continue using UML for development.
>=20
> Regarding the bug your pointed out, I'd like you to test 0.21pre5
> version I released this week. The swapout path code was rewritten and
> it's much simpler and efficient now. I tested the last code to trigger
> this bug and couldn't make it hang. Neither could Livio. He tested
> under UML yesterday and got disappointed that he couldn't make the
> code crash. :-) In case you still have problems with the this newer
> version, please let me know.

I have downloaded the latest 0.21pre6 .=20

It still have problem nears out of memory when I have not turned on
swap. Do you think this is something not related with compressed cache
or a vm bug from Linux?

Usually when really out of memory, the oom killer should be triggered
and will kill that process... my experiecne is the machine freezes.

I will do more testing with swap devices.
>=20
> > Rodrigo Souza de Castro wrote:
> >=20
> > >Hi there,
> > >
> > >Finally Jeff seems to have fixed that bug which screwed up our vswap
> > >addresses. I think I can continue developing under UML :-)
>=20
> Best regards,
> --=20
> Rodrigo S. de Castro <rc...@im...>

David

Re: [lc-devel] UML swap bug fixed

[Rodrigo S. de C. <rc...@im...>]

On Sat, Jan 12, 2002 at 05:50:54PM +0800, David Chow wrote:
> Is this the bug that I point out in my previous mail to the list? Or it 
> is caused by a different issue? Please clarify, since I don't have 
> enough time to understand all the code at the moment. Thanks.

I forwarded an email to this list on Dec 12th concerning a bug in
User-Mode-Linux that made compressed cache not work. To be clear, with
compressed cache code this bug was triggered easier. And now it's
finally fixed and we can continue using UML for development.

Regarding the bug your pointed out, I'd like you to test 0.21pre5
version I released this week. The swapout path code was rewritten and
it's much simpler and efficient now. I tested the last code to trigger
this bug and couldn't make it hang. Neither could Livio. He tested
under UML yesterday and got disappointed that he couldn't make the
code crash. :-) In case you still have problems with the this newer
version, please let me know.

> Rodrigo Souza de Castro wrote:
> 
> >Hi there,
> >
> >Finally Jeff seems to have fixed that bug which screwed up our vswap
> >addresses. I think I can continue developing under UML :-)

Best regards,
-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] UML swap bug fixed

[David C. <dav...@rc...>]

Is this the bug that I point out in my previous mail to the list? Or it 
is caused by a different issue? Please clarify, since I don't have 
enough time to understand all the code at the moment. Thanks.

David

Rodrigo Souza de Castro wrote:

>Hi there,
>
>Finally Jeff seems to have fixed that bug which screwed up our vswap
>addresses. I think I can continue developing under UML :-)
>
>----- Forwarded message from Jeff Dike <jd...@ka...> -----
>
>Date: Thu, 10 Jan 2002 22:16:18 -0500
>From: Jeff Dike <jd...@ka...>
>Subject: Re: [uml-devel] swap entry and newpage bit 
>To: Rodrigo Souza de Castro <rc...@im...>
>Cc: use...@li...
>
>rc...@im... said:
>
>>The problem I noticed is that, when setting a pte to a swap entry, the
>>NEWPAGE bit is turned on to tell fix_range() that this address should
>>be unmapped. It turns out that, for swap adresses that are shared by
>>many ptes, the address is not unmapped and the NEWPAGE bit is not
>>turned off soon enough before any pte faults. Then, in fault handlers
>>(handle_pte_fault() and do_swap_page()), the pte has a wrong value, so
>>I can't find the page in swap cache. For example, if the swap cache
>>page is set to 0x3307c and the wrong pte is set to 0x3307e (due to
>>newpage bit), I won't be able to find it.
>>
>
>This is an excellent description of the bug that I just spend 5 days chasing.
>
>Unfortunately, I didn't until recently (yesterday) understand enough of the
>swap code to tell whether you were describing something real or whether
>you were on crack.
>
>It is now fixed.
>
>pte_to_swp_entry now masks off the arch bits.
>
>				Jeff
>
>----- End forwarded message -----
>

[lc-devel] UML swap bug fixed

[Rodrigo S. de C. <rc...@im...>]

Hi there,

Finally Jeff seems to have fixed that bug which screwed up our vswap
addresses. I think I can continue developing under UML :-)

----- Forwarded message from Jeff Dike <jd...@ka...> -----

Date: Thu, 10 Jan 2002 22:16:18 -0500
From: Jeff Dike <jd...@ka...>
Subject: Re: [uml-devel] swap entry and newpage bit 
To: Rodrigo Souza de Castro <rc...@im...>
Cc: use...@li...

rc...@im... said:
> The problem I noticed is that, when setting a pte to a swap entry, the
> NEWPAGE bit is turned on to tell fix_range() that this address should
> be unmapped. It turns out that, for swap adresses that are shared by
> many ptes, the address is not unmapped and the NEWPAGE bit is not
> turned off soon enough before any pte faults. Then, in fault handlers
> (handle_pte_fault() and do_swap_page()), the pte has a wrong value, so
> I can't find the page in swap cache. For example, if the swap cache
> page is set to 0x3307c and the wrong pte is set to 0x3307e (due to
> newpage bit), I won't be able to find it.

This is an excellent description of the bug that I just spend 5 days chasing.

Unfortunately, I didn't until recently (yesterday) understand enough of the
swap code to tell whether you were describing something real or whether
you were on crack.

It is now fixed.

pte_to_swp_entry now masks off the arch bits.

				Jeff

----- End forwarded message -----

-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Lockups in 2.4.17

[David C. <dav...@rc...>]

=A6b =B6g=A4=AD, 2002-01-11 01:54, Rodrigo Souza de Castro =BCg=B9D=A1G
> On Thu, Jan 10, 2002 at 09:32:50AM +0800, David Chow wrote:
> > I happens in 2 situations,
> > 1. The beginning to swap out pages, when the memory getting low and
> > the kernel beginging to swap out pages, this only happened once, not
> > often triggered.
> > 2. Swap is gradually increasing, when its full, the machine
> > lockup. I think this is not out of memory, infact kernel should free
> > up some unused pages or reclaim some swaps. It seems the swap usage
> > never decrease, and it seems the kernel have problem freeing up swap
> > spaces.
> >=20
> > Anyone have experience in this? Thanks.
>=20
> Livio had problems yesterday running a kernel without swap. I think
> that's vswap related and I'll have a look at it.=20
>=20
> Could you please try the 0.21pre5 version I released today to see if
> you can reproce this bug?
>=20
> Regards,
> --=20
> Rodrigo S. de Castro <rc...@im...>

Yes, it also happens without a swap.

David

Re: [lc-devel] Lockups in 2.4.17

[Rodrigo S. de C. <rc...@im...>]

On Thu, Jan 10, 2002 at 09:32:50AM +0800, David Chow wrote:
> I happens in 2 situations,
> 1. The beginning to swap out pages, when the memory getting low and
> the kernel beginging to swap out pages, this only happened once, not
> often triggered.
> 2. Swap is gradually increasing, when its full, the machine
> lockup. I think this is not out of memory, infact kernel should free
> up some unused pages or reclaim some swaps. It seems the swap usage
> never decrease, and it seems the kernel have problem freeing up swap
> spaces.
> 
> Anyone have experience in this? Thanks.

Livio had problems yesterday running a kernel without swap. I think
that's vswap related and I'll have a look at it. 

Could you please try the 0.21pre5 version I released today to see if
you can reproce this bug?

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Lockups in 2.4.17

[Livio B. S. <li...@li...>]

Howdy David!

David Chow writes:
> Dear all,
> 
> I have a couple of lockups in 2.4.17 with compressed cache. Not sure its
> linux related or compressed cache related.
> 
> I happens in 2 situations,
> 1. The beginning to swap out pages, when the memory getting low and the
> kernel beginging to swap out pages, this only happened once, not often
> triggered.

  Humm, this one, I have no idea what is happening.

> 2. Swap is gradually increasing, when its full, the machine lockup. I
> think this is not out of memory, infact kernel should free up some
> unused pages or reclaim some swaps. It seems the swap usage never
> decrease, and it seems the kernel have problem freeing up swap spaces.

  Yes, I've seen this one. This is a bug with the vswap (Virtual Swap)
mechanism. I talked to Rodrigo about it, and he thinks that the variable
'estimated_free_space' (in mm/comp_cache/vswap.c), is being wrongly calculated,
and hinting to the cache that there is still some space left, and giving out
addresses, which in reality wouln't fit into memory.... then when that address
is requested things just explode.

  We are trying to look for a good way to fix it. Maybe, if you have time, you
can take a look at it.

  best regards,

--  
  Livio <li...@li...>

[lc-devel] Lockups in 2.4.17

[David C. <dav...@rc...>]

Dear all,

I have a couple of lockups in 2.4.17 with compressed cache. Not sure its
linux related or compressed cache related.

I happens in 2 situations,
1. The beginning to swap out pages, when the memory getting low and the
kernel beginging to swap out pages, this only happened once, not often
triggered.
2. Swap is gradually increasing, when its full, the machine lockup. I
think this is not out of memory, infact kernel should free up some
unused pages or reclaim some swaps. It seems the swap usage never
decrease, and it seems the kernel have problem freeing up swap spaces.

Anyone have experience in this? Thanks.

David

Re: [lc-devel] Modularising the compressed cache

[David C. <dav...@rc...>]

=A6b =B6g=A4=E9, 2001-12-23 22:19, Rodrigo Souza de Castro =BCg=B9D=A1G
> Hi David,
>=20
> On Sat, Dec 22, 2001 at 01:38:17AM +0800, David Chow wrote:
> > Has anyone think of modularising the compressed cache
> > implementation? I know the current implementation makes changes to
> > the current swap cache.
>=20
> What do you have in mind when talking about modularising this
> implementation?
>=20
> > I already have some ideas to modularise the swap system. This will
> > also allow us to dynamically configure the compressed cache size at
> > run time.
>=20
> Could you tell us something about your ideas to modularise the swap
> system. And in your opinion how that will help us to dynamically
> configure the cache size?
>=20
> Regards,
> --=20
> Rodrigo S. de Castro <rc...@im...>

You know, I've been working on the nfsswap for sometime. The nfsswap
swap code that I am using simply modularise the swap sysstem. When we
want to swap to nfs, we simply insmod the nfsswap.o and call
register_swap_method() . We then patch all swap call to use operations
like VFS. I just think if we define a compressed swap module, and
implement something like a fs to hide all the compression tricks behind
the kernel, then we can make compress swap + cache in our module instead
of the swap_cache. By calling swapon we may add more swap space and
cache just like calling swapon for files and partitions. Also, compress
cache size then can be run time configured due to the separation and may
be pass in as a module option some how. I just thought of this off my
head and I hope this may work.

Of course, there is no easy way to modularising the swap system. I have
done some porting work using the nfsswap code with compressed cache at
the same time and reporting no problems for use in production system
more than 3 months. But I did not implement the compression in the swap
since working on compressed swap require a lot more work.

David

Re: [lc-devel] Modularising the compressed cache

[Rodrigo S. de C. <rc...@im...>]

Hi David,

On Sat, Dec 22, 2001 at 01:38:17AM +0800, David Chow wrote:
> Has anyone think of modularising the compressed cache
> implementation? I know the current implementation makes changes to
> the current swap cache.

What do you have in mind when talking about modularising this
implementation?

> I already have some ideas to modularise the swap system. This will
> also allow us to dynamically configure the compressed cache size at
> run time.

Could you tell us something about your ideas to modularise the swap
system. And in your opinion how that will help us to dynamically
configure the cache size?

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] [PATCH] 2.4.17-0.20

[Rodrigo S. de C. <rc...@im...>]

Hi,

A new major version is out there. After 6 pre versions, I think it's
time to release the 0.20 version before adaptivity development. This
version features mostly code cleanup and minor changes. For example,
we have a directory in /proc/sys/vm/ where some parameters can changed
(some of them, like size, are not working so far). Now you can choose
the compressed cache size by a kernel parameter (compsize=) and also
by a configuration option in the "General Setup" menu.

Some bugs that showed up with the new kernel versions
(2.4.{9,10,16,17}) were fixed as well. A great feature is the lru
ordering for compressed cache pages, fixing up a conceptual flaw that
existed for a long time in our code. This implementation allowed us
remove the second avl tree, indexed by freeable space, decreasing the
complexity of some functions. To conclude, we started the preparation
for adaptivity by changing the comp_cache array to slab cache
allocation. Adaptivity (only manual) is our next task.

This patch is not working under User-Mode-Linux, at least until
2.4.16-2um. And by the way, a version for 2.4.16 is also available on
the downloads page.

Check the detailed changelog and notes in the links below. 

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.17-0.20.bz2

Changelog:
http://sourceforge.net/project/shownotes.php?release_id=66614

final:

- comp_cache array removed. Now all comp_cache struct will be
allocated by slab cache for the adaptivity.

- reverted changes regarding vswap_add_freeable which broke the code.
Only part of the function should have been deleted.
vswap_add_freeable has been renamed to remove_vswap.

pre6:

- removed the references to the freeable avl tree. Currently we search
for a compressed cache page based upon the LRU ordering, no its
freeable space any longer.

pre5:

- compressed pages in compressed cache are swapped out in the order
they were added to compressed cache, therefore following the order the
pages were evicted by the VM system (LRU).

pre4:

- second batch of changes for the removal of #ifdefs that uglify the
code.

- compressed cache code in unuse_pte was removed because it does
not make any sense currently.

pre3:

- first batch of changes to remove #ifdefs and make code cleaner.

- it follows Documentation/SubmittingPatches document.

pre2 (2.4.16):

- 0.20pre2 version updated from 2.4.10 to 2.4.16.

- code was rewritten in swapfile.c to work with the new swap file
functions (swap_free, swap_duplicate, swap_info_get, swap_info_put,
etc)

pre2 (2.4.10):

- In 2.4.10 the MAX_SWAPFILES was changed, what caused some problems
for our code in architectures like UML, which handles swap entry
addresses differently than in i386. That way, we cannot keep using
MAX_SWAPFILES for our swap type unless we decrease this variable. In
this case, we decreased the variable to 31 and so we can use 31 as
COMP_CACHE_SWP_TYPE without worrying about arch-dependent
configuration. Nevertheless, we are limited to 31 swap files, not 32
as in vanilla.

pre1:

- Fix for a bug which would cause an error if you tried to compile
without CONFIG_COMP_SWAP (swap out pages in compressed form).

- Algorithms now are switched by echoing to
/proc/sys/vm/comp_cache/algorithm, not any longer to
/proc/comp_cache. "size" is already present in the directory, but it
does not work so far. In the future it will be able to choose maximum
compressed cache size on the fly.

- By the way, /proc/comp_cache is not available. It has been renamed
to comp_cache_stat and its code is in fs/proc/proc_misc.c with all
other proc entries.

- You can choose the maximum size of compressed size by entering the
value in the kernel configuration or by a kernel parameter
(compsize=).  The latter has priority over the former, in the case
both are define. The parameter is in memory pages unit. Therefore
there's no need to modify a variable in order to change the maximum
size of compressed cache any longer.

- mm/comp_cache/proc.c has been removed. Some remaining functions
that were placed in this file are in aux.c.

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] Modularising the compressed cache

[David C. <dav...@rc...>]

Has anyone think of modularising the compressed cache implementation? I 
know the current implementation makes changes to the current swap cache. 
I already have some ideas to modularise the swap system. This will also 
allow us to dynamically configure the compressed cache size at run time.

David

[lc-devel] 0.20pre2 updated to 2.4.16

[Rodrigo S. de C. <rc...@im...>]

There's a new patch out there which updates the 0.20pre2 code to
2.4.16 version of Linux kernel. No major change, only a part of the
code was rewritten in order to work with the lastest changes in swap
file functions from swapfile.c (swap_free, swap_duplicate,
swap_info_get, etc).

Download:
http://prdownloads.sourceforge.net/linuxcompressed/patch-comp-cache-2.4.16-0.20pre2.bz2

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] 2.4.16-0.20pre2 NOT working under UML

[Rodrigo S. de C. <rc...@im...>]

Hi,

The 2.4.16-0.20pre2 version is not currently working under User Mode
Linux (at least not up to 2.4.16-2um). There's a problem regarding
ptes and the virtual swap addresses used by our code.  We don't know
why some bits are not correctly updated, what messes up everything
when a pte faults a not updated address. Check the email I sent to
user-mode-linux-devel mailing list below (no reply so far).

If somebody feels like digging deep into this problem, it would be
very helpful. I don't know very much of UML internals, so I don't know
exactly what's going on there with respect to pte, tlb, etc.

----- Forwarded message from Rodrigo Souza de Castro <rc...@im...> -----

Date: Wed, 12 Dec 2001 12:07:15 -0200
From: Rodrigo Souza de Castro <rc...@im...>
Subject: [uml-devel] swap entry and newpage bit
To: use...@li...
Delivered-To: rc...@im...

Hi,

[Jeff, I talked to you yesterday about this problem]

In my code, I use a virtual swap address, which is nothing but an
ordinary swap entry, but its type is MAX_SWAPFILES. This address is
assigned to a pte in try_to_swap_out(), like an usual swap entry and
that's _all_ (there some other stuff wrt to these ptes, but I
removed). It should work as usual swap addresses do work, right? But
it does not, at least under UML (on a real machine it does).

The problem I noticed is that, when setting a pte to a swap entry, the
NEWPAGE bit is turned on to tell fix_range() that this address should
be unmapped. It turns out that, for swap adresses that are shared by
many ptes, the address is not unmapped and the NEWPAGE bit is not
turned off soon enough before any pte faults. Then, in fault handlers
(handle_pte_fault() and do_swap_page()), the pte has a wrong value, so
I can't find the page in swap cache. For example, if the swap cache
page is set to 0x3307c and the wrong pte is set to 0x3307e (due to
newpage bit), I won't be able to find it.

In _switch_to, everytime we swith to a new task, you flush its tlb
(flush_tlb_all()), so there shouldn't have this problem. An important
question before bother you more: do you think that it's UML-related or
it's unlikely?

I added a printk to fix_range when fixing my virtual swap entries and
the problem was gone. Will it be a race condition? I am not doing
anything but setting the pte in try_to_swap_out and handling faults in
do_swap_page(), so I have no clue. :-(

The same code was working in the old 2.4.10, when the newpage bit was
not set for swap entries. 

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>


_______________________________________________
User-mode-linux-devel mailing list
Use...@li...
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel



----- End forwarded message -----

-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Statistics: Hint

[Rodrigo S. de C. <rc...@im...>]

On Wed, Dec 12, 2001 at 08:11:24PM +0100, Wolly wrote:
> After reading
> http://linuxcompressed.sourceforge.net/statistics/prototype3-0.html, 
> the first thing I thought at was `and the CPU load?'

Yes, I absolutely agree with you. We didn't think about measuring CPU
usage.

> It would be cool, if you could also add CPU usage statistics. 
> You could do this simply by using /usr/bin/time instead of bash-builtin time, 
> e.g. /usr/bin/time proggy-to-execute
> 0.56user 0.03system 0:01.99elapsed 29%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+0outputs (1428major+137minor)pagefaults 0swaps

Thanks for the suggestion, I already took note and the CPU usage is on
our statistics list. All other comments and suggestions are very
welcome :-)

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] Statistics: Hint

[Wolly <ww...@gm...>]

Hi all, 

After reading
http://linuxcompressed.sourceforge.net/statistics/prototype3-0.html, 
the first thing I thought at was `and the CPU load?'

It would be cool, if you could also add CPU usage statistics. 
You could do this simply by using /usr/bin/time instead of bash-builtin time, 
e.g. /usr/bin/time proggy-to-execute
0.56user 0.03system 0:01.99elapsed 29%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (1428major+137minor)pagefaults 0swaps

Keep on guys, compressing swap is a cool idea!

Regards, 
Wolly

Re: [lc-devel] Let's get back to work!

[David C. <dav...@rc...>]

On Sat, 2001-12-08 at 03:40, Rodrigo Souza de Castro wrote:
> Hi,
> 
> Finally all my graduate courses for my master degree are over. I
> enrolled in many courses this semester and from the middle of the
> semester, I had the need to spend almost all my time on assignments,
> seminars and exams. That's why this list has been so quiet for the
> last two months.
> 
> Anyway, from now on I will work full time on the Compressed Cache
> project. I think that we will be able to have a greater development
> and to get much better results from the project.
> 
> You all know that Linux 2.5 has been opened. Then we can expect minor
> changes from 2.4.x version that is being maintained by Marcelo
> Tosatti, nothing like what happened until 2.4.1X, when VM changed, for
> example. BTW, I talked to him today about whether either I should port
> the current code to 2.5.x and follow this branch or follow 2.4.x. I
> agree with Marcelo that 2.4.x has a bigger userbase and I think that
> we should follow this version. Once in a while I will port to 2.5 and
> make a patch available on the project page to keep track of the
> changes being done in this kernel version. Thus, my first activity
> will be to port to 2.4.16, (or 2.4.17 if Marcelo releases a new
> version until there). Comments about this decision are very welcome.
> 
> Well, let's get back to work. If somebody feels like helping me doing
> something, please let me know.
> 
> Regards,
> -- 
> Rodrigo S. de Castro <rc...@im...>
> 
> 
> _______________________________________________
> linuxcompressed-devel mailing list
> lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxcompressed-devel

I agree staying in 2.4 since from 2.4.16, the vm and other changes seems
to be more stable. I am also studying the change of the vm during 2.4.1x
. I will keep track with you. Thanks.

regards,

David Chow

[lc-devel] GCC __builting_expect

[Rodrigo S. de C. <rc...@im...>]

Hi,

I was wondering what likely() and unlikely() macros defined in
linux/include/linux/compiler.h for GCC > 2.96 did. Two explanations
about that: 

From the compiler.h source code:

/* Somewhere in the middle of the GCC 2.96 development cycle, we
   implemented a mechanism by which the user can annotate likely
   branch directions and expect the blocks to be reordered
   appropriately.  Define __builtin_expect to nothing for earlier
   compilers.  */

I found "Using and Porting the GNU Compiler Collection (GCC): Other
Builtins" (http://www.dis.com/gnu/gcc/gcc_112.html) where I got the
following:

Built-in Function: long __builtin_expect (long exp, long c) 

You may use __builtin_expect to provide the compiler with branch
prediction information. In general, you should prefer to use actual
profile feedback for this (`-fprofile-arcs'), as programmers are
notoriously bad at predicting how their programs actually
perform. However, there are applications in which this data is hard to
collect.

The return value is the value of exp, which should be an integral
expression. The value of c must be a compile-time constant. The
semantics of the built-in are that it is expected that exp == c. For
example:
        
        if (__builtin_expect (x, 0))
          foo ();

would indicate that we do not expect to call foo, since we expect x to
be zero. Since you are limited to integral expressions for exp, you
should use constructions such as
        
        if (__builtin_expect (ptr != NULL, 1))
          error ();

when testing pointer or floating-point values.

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

[lc-devel] Let's get back to work!

[Rodrigo S. de C. <rc...@im...>]

Hi,

Finally all my graduate courses for my master degree are over. I
enrolled in many courses this semester and from the middle of the
semester, I had the need to spend almost all my time on assignments,
seminars and exams. That's why this list has been so quiet for the
last two months.

Anyway, from now on I will work full time on the Compressed Cache
project. I think that we will be able to have a greater development
and to get much better results from the project.

You all know that Linux 2.5 has been opened. Then we can expect minor
changes from 2.4.x version that is being maintained by Marcelo
Tosatti, nothing like what happened until 2.4.1X, when VM changed, for
example. BTW, I talked to him today about whether either I should port
the current code to 2.5.x and follow this branch or follow 2.4.x. I
agree with Marcelo that 2.4.x has a bigger userbase and I think that
we should follow this version. Once in a while I will port to 2.5 and
make a patch available on the project page to keep track of the
changes being done in this kernel version. Thus, my first activity
will be to port to 2.4.16, (or 2.4.17 if Marcelo releases a new
version until there). Comments about this decision are very welcome.

Well, let's get back to work. If somebody feels like helping me doing
something, please let me know.

Regards,
-- 
Rodrigo S. de Castro <rc...@im...>

Re: [lc-devel] Finish up documentation

[David C. <dav...@rc...>]

Rodrigo Souza de Castro wrote:

> On Wed, Oct 31, 2001 at 03:27:49PM +0800, David Chow wrote:
> > For the documentation on the linux compressed cache website. It is
> > imcomplete, I would like to finish up the documentation of the files and
> > functions. But I will need help from you guys to verify and see if it is
> > corret or not. I think this helps me to understand the overall coding
> > and help others to contribute. Thanks.
>
> Besides being incomplete, it's completely outdated. It was done based
> on 2.4.0 or 2.4.1, and lots of changes have been made since then. One
> of the many changes is the VM fork between Linus and Alan branches
> that happened around 2.4.9 IIRC.
>
> I think that first of all we need to update this documentation. If you
> feel like updating those old documentation, it would be even nicer
> that writing new documents. Linus told today on lkml that 2.4.15 is
> likely to be the start point to 2.5.x, so now we can update the
> documentation on the site to have a long-term documentation for 2.4.x.
>
> I'd be pleased to check your documentation. Unfortunately I am really
> busy until the end of this semester (some assignments and finals at
> the university) so I won't be able to help you writing documentation
> while I can't get rid of those stuff (maybe beginning of
> December). The bright side is that when this semester is over, I will
> be finally working full-time on the project. :-)
>
> Best regards,
> --
> Rodrigo S. de Castro <rc...@im...>

Besides about documentation, since the project is in a very seed stage, I
suggest we concentrate on we are doing instead of spending our time keep in
trace with the main stream kernel changes. I suggest pick a typical stable
release and stick to that release, after our work is done, then do the
porting work to the newest ones.  Since the routines that comprssed cache
uses (as I notice) are the swap cache routines, unless Linus make a big
change on the vm, our design is less likely to be affected.  It is very hard
to keep tack of everything, since they are making changes quite often, you
will have to sacrisfy something.  I don't think I can do  the documentation
of the VM without help, but I will do my best for it.

regards,

David

Re: [lc-devel] Finish up documentation

[Rodrigo S. de C. <rc...@im...>]

On Wed, Oct 31, 2001 at 03:27:49PM +0800, David Chow wrote:
> For the documentation on the linux compressed cache website. It is 
> imcomplete, I would like to finish up the documentation of the files and 
> functions. But I will need help from you guys to verify and see if it is 
> corret or not. I think this helps me to understand the overall coding 
> and help others to contribute. Thanks.

Besides being incomplete, it's completely outdated. It was done based
on 2.4.0 or 2.4.1, and lots of changes have been made since then. One
of the many changes is the VM fork between Linus and Alan branches
that happened around 2.4.9 IIRC.

I think that first of all we need to update this documentation. If you
feel like updating those old documentation, it would be even nicer
that writing new documents. Linus told today on lkml that 2.4.15 is
likely to be the start point to 2.5.x, so now we can update the
documentation on the site to have a long-term documentation for 2.4.x.

I'd be pleased to check your documentation. Unfortunately I am really
busy until the end of this semester (some assignments and finals at
the university) so I won't be able to help you writing documentation
while I can't get rid of those stuff (maybe beginning of
December). The bright side is that when this semester is over, I will
be finally working full-time on the project. :-)

Best regards,
-- 
Rodrigo S. de Castro <rc...@im...>