linuxcompressed-devel

[lc-devel] hello

[Bob A. <cu...@pb...>]

hello, i applied latest testing patch to 2.4.20 and it corrupts rw 
filesystems
while running. now trying different combinations (compiling 2.4.20 with 
stable patch right now)

anyone here?
-- 
-- 

Re: [lc-devel] hello

[Rodrigo S. de C. <rc...@im...>]

Hi Bob!

On Fri, May 16, 2003 at 09:20:47PM +0200, Bob Arctor wrote:
> hello, i applied latest testing patch to 2.4.20 and it corrupts rw
> filesystems while running. now trying different combinations

Do you have preempt patch or is your system SMP? The latest testing
patch isn't preempt or SMP-safe, but it is known to be stable for UP
systems without the preempt patch. I am going to release a patch with
the latest code (which is in CVS server) ported to 2.4.20, really soon
(probably this weekend), so you may want to give this patch a try.

> (compiling 2.4.20 with stable patch right now)

The stable patch (0.23) is really old, so I guess you should try to
run the testing ones. Let's try to fix the problems you hit if they
are still there even with the to be released patch for 2.4.20.

Regards,
-- 
Rodrigo

Re: [lc-devel] hello

[curious <cu...@pb...>]

no, i don't have.

latest development patch corrupts filesystem though.
i though it is porting to 21rc2-ac1 then i thought it is -20
fault (because i fall back to -20)
i did all of this on  my perfectly non-smp laptop ofcourse :)
then i concluded -20 is also causing problems or it's your patch
so i tried to use last stable 0.23 on -20
it gave me same results (i tried to execute links and it oopsed)

then i moved to pure 2.4.19 and apllied latest testing patch.
no success.
links caused kernel BUG @ filemap.c :/
i learned to mount all ro, and only /tmp and /var rw after my
first try when 21rc2-ac1+ccache
totally damaged files which were never written to on ext2
filesystem... (like libraries)

then i got a hint and set compcache=0M and it caused kernel only
to hang.

now i am trying to compile 2.4.19 with cvs version, and check.

then i'll start off first stable release with 2.4.19 to check
wheter this is some new feature not working, or .19 mismerge
then i'll try to use .18 branch of patches...

i am reading mail in meantime ofcourse , so feel free to ask
questions/suggest other testing.

i have no problems when i use 2.4.21rc2-ac1 kernel without
ccache, so i exclde any kernel problems/hardware issues


On Fri, 16 May 2003, Rodrigo Souza de Castro wrote:

> Hi Bob!
>
> On Fri, May 16, 2003 at 09:20:47PM +0200, Bob Arctor wrote:
> > hello, i applied latest testing patch to 2.4.20 and it corrupts rw
> > filesystems while running. now trying different combinations
>
> Do you have preempt patch or is your system SMP? The latest testing
> patch isn't preempt or SMP-safe, but it is known to be stable for UP
> systems without the preempt patch. I am going to release a patch with
> the latest code (which is in CVS server) ported to 2.4.20, really soon
> (probably this weekend), so you may want to give this patch a try.
>
> > (compiling 2.4.20 with stable patch right now)
>
> The stable patch (0.23) is really old, so I guess you should try to
> run the testing ones. Let's try to fix the problems you hit if they
> are still there even with the to be released patch for 2.4.20.
>
> Regards,
> --
> Rodrigo
>
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: If flattening out C++ or Java
> code to make your application fit in a relational database is painful,
> don't do it! Check out ObjectStore. Now part of Progress Software.
> http://www.objectstore.net/sourceforge
> _______________________________________________
> linuxcompressed-devel mailing list
> lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxcompressed-devel
>

Re: [lc-devel] hello

[Rodrigo S. de C. <rc...@im...>]

On Sat, May 17, 2003 at 04:24:35AM +0200, curious wrote:
> no, i don't have.
> 
> latest development patch corrupts filesystem though.
> i though it is porting to 21rc2-ac1 then i thought it is -20
> fault (because i fall back to -20)
> i did all of this on  my perfectly non-smp laptop ofcourse :)

It is important to know you are not compiling with preempt patch
before trying to figure out what the problem may be.

> then i concluded -20 is also causing problems or it's your patch
> so i tried to use last stable 0.23 on -20
> it gave me same results (i tried to execute links and it oopsed)

Strange. So I guess only executing a binary oops the kernel, right? If
you do have the oops in your log files (decoded to function names, of
course), please send me it. And some basic information about your
system configuration.

> then i moved to pure 2.4.19 and apllied latest testing patch.
> no success.
> links caused kernel BUG @ filemap.c :/
> i learned to mount all ro, and only /tmp and /var rw after my
> first try when 21rc2-ac1+ccache
> totally damaged files which were never written to on ext2
> filesystem... (like libraries)
> 
> then i got a hint and set compcache=0M and it caused kernel only
> to hang.

You mean "compsize=". I just tested it with 0M here and it
worked. Probably something I fixed since 0.24pre5.

> now i am trying to compile 2.4.19 with cvs version, and check.

I will send you, in a private email, a patch for 2.4.20 that I am
testing before releasing. So you can try it out and give me a feedback
about these corruptions. Maybe it is something related to your tight
memory configuration.

> then i'll start off first stable release with 2.4.19 to check wheter
> this is some new feature not working, or .19 mismerge then i'll try
> to use .18 branch of patches...

Yes, trying 2.4.18 with the latest testing patch is probably a good
idea to check out what is going on.

> i have no problems when i use 2.4.21rc2-ac1 kernel without
> ccache, so i exclde any kernel problems/hardware issues

Indeed it looks like something related to comp cache.

Best regards,
-- 
Rodrigo

Re: [lc-devel] hello

[curious <cu...@pb...>]

On Fri, 16 May 2003, Rodrigo Souza de Castro wrote:

> It is important to know you are not compiling with preempt patch
> before trying to figure out what the problem may be.
unfortunatelly it's not this case (i apply agains vanilla)

> > then i concluded -20 is also causing problems or it's your patch
> > so i tried to use last stable 0.23 on -20
> > it gave me same results (i tried to execute links and it oopsed)
> Strange. So I guess only executing a binary oops the kernel, right? If
> you do have the oops in your log files (decoded to function names, of
> course), please send me it. And some basic information about your
> system configuration.
unfortunatelly i can only reproduce bug and have oops
non-decoded :(

btw. effects are nonpredictable to _some_ extent
i.e. links crash, but sometimes machine just hangs, sometimes
gives oops, sometimes segfault...
it might be linked to corruption of _filesystem_
after running for a while i got seriously damaged filesys, and i
couldn't predict damages.

lunks are compressed with upx, so they decompress to /tmp (which
was ext2 /tmp or /dev/shm a bit later (when i thought it is only
ext2/3 issue) /dev/shm didn't helped though, so i assume it
might be faulty decompression ...

btw, i can't find any params in /proc/sys/vm/ about compcache
(i.e. comp. algorithm mentioned in docs)

> You mean "compsize=". I just tested it with 0M here and it
> worked. Probably something I fixed since 0.24pre5.
no, i mean when i start links machine just hanged :)

ah, an hint - on negative memory system (when swap exceeds free
ram) and/or when swapping out cost (writing to device) is higher
than cpu cost pages should be allways swapped out compressed,
not only when cpu is idle.

> > now i am trying to compile 2.4.19 with cvs version, and check.
>
> I will send you, in a private email, a patch for 2.4.20 that I am
> testing before releasing. So you can try it out and give me a feedback
> about these corruptions. Maybe it is something related to your tight
> memory configuration.
just got it. cancelling cvs compile and compiling .20

> > then i'll start off first stable release with 2.4.19 to check wheter
> > this is some new feature not working, or .19 mismerge then i'll try
> > to use .18 branch of patches...
>
> Yes, trying 2.4.18 with the latest testing patch is probably a good
> idea to check out what is going on.
ok, so it'll be next after .20 i got from you (if it'll not
solve this problem ofcourse :)

thanks for support

-- 

Re: [lc-devel] hello

[curious <cu...@pb...>]

sysinf
http://217.97.12.194/cgi-bin/sysinfo.sh ?
machines are similiar.
all x86

laptop is x86 150mmx, 8M ram , ide disk, ext2 filesys.
gcc 2.95.3

Re: [lc-devel] hello

[Rodrigo S. de C. <rc...@im...>]

On Sat, May 17, 2003 at 05:15:19AM +0200, curious wrote:
> sysinf
> http://217.97.12.194/cgi-bin/sysinfo.sh ?
> machines are similiar.
> all x86
> 
> laptop is x86 150mmx, 8M ram , ide disk, ext2 filesys.
> gcc 2.95.3

Please, send me your .config too, if the patch I just sent you does
not solve the corruption/hangs.

Regards,
-- 
Rodrigo

Re: [lc-devel] hello

[curious <cu...@pb...>]

http://217.97.12.194/Grzybnia/wafel/wafel.cc
is the config file. compilation is in progress, i'll test soon.

> Please, send me your .config too, if the patch I just sent you does
> not solve the corruption/hangs.

Re: [lc-devel] hello

[curious <cu...@pb...>]

i compiled patch from you, and it bug() in filemap.c:66
in process loop0 .
then in concluded it may be associated with mmaped loop0
so i disabled "support for page cache compression"
without this ccache works fine and stable :D

i had swapping via loop0 , and it was on filesys, quite rare
case, but happens.

are you having clue what might be b0rked ?
in meantime i'll apply patch to 21rc2-ac1 and test it with page
cache compression disabled.



On Sat, 17 May 2003, curious wrote:

>
>
> On Fri, 16 May 2003, Rodrigo Souza de Castro wrote:
>
> > It is important to know you are not compiling with preempt patch
> > before trying to figure out what the problem may be.
> unfortunatelly it's not this case (i apply agains vanilla)
>
> > > then i concluded -20 is also causing problems or it's your patch
> > > so i tried to use last stable 0.23 on -20
> > > it gave me same results (i tried to execute links and it oopsed)
> > Strange. So I guess only executing a binary oops the kernel, right? If
> > you do have the oops in your log files (decoded to function names, of
> > course), please send me it. And some basic information about your
> > system configuration.
> unfortunatelly i can only reproduce bug and have oops
> non-decoded :(
>
> btw. effects are nonpredictable to _some_ extent
> i.e. links crash, but sometimes machine just hangs, sometimes
> gives oops, sometimes segfault...
> it might be linked to corruption of _filesystem_
> after running for a while i got seriously damaged filesys, and i
> couldn't predict damages.
>
> lunks are compressed with upx, so they decompress to /tmp (which
> was ext2 /tmp or /dev/shm a bit later (when i thought it is only
> ext2/3 issue) /dev/shm didn't helped though, so i assume it
> might be faulty decompression ...
>
> btw, i can't find any params in /proc/sys/vm/ about compcache
> (i.e. comp. algorithm mentioned in docs)
>
> > You mean "compsize=". I just tested it with 0M here and it
> > worked. Probably something I fixed since 0.24pre5.
> no, i mean when i start links machine just hanged :)
>
> ah, an hint - on negative memory system (when swap exceeds free
> ram) and/or when swapping out cost (writing to device) is higher
> than cpu cost pages should be allways swapped out compressed,
> not only when cpu is idle.
>
> > > now i am trying to compile 2.4.19 with cvs version, and check.
> >
> > I will send you, in a private email, a patch for 2.4.20 that I am
> > testing before releasing. So you can try it out and give me a feedback
> > about these corruptions. Maybe it is something related to your tight
> > memory configuration.
> just got it. cancelling cvs compile and compiling .20
>
> > > then i'll start off first stable release with 2.4.19 to check wheter
> > > this is some new feature not working, or .19 mismerge then i'll try
> > > to use .18 branch of patches...
> >
> > Yes, trying 2.4.18 with the latest testing patch is probably a good
> > idea to check out what is going on.
> ok, so it'll be next after .20 i got from you (if it'll not
> solve this problem ofcourse :)
>
> thanks for support
>
> --
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: If flattening out C++ or Java
> code to make your application fit in a relational database is painful,
> don't do it! Check out ObjectStore. Now part of Progress Software.
> http://www.objectstore.net/sourceforge
> _______________________________________________
> linuxcompressed-devel mailing list
> lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxcompressed-devel
>

Re: [lc-devel] hello

[Rodrigo S. de C. <rc...@im...>]

On Sat, May 17, 2003 at 06:16:59PM +0200, curious wrote:
> i compiled patch from you, and it bug() in filemap.c:66
> in process loop0 .
> then in concluded it may be associated with mmaped loop0
> so i disabled "support for page cache compression"
> without this ccache works fine and stable :D

That is great, although you probably will want support for page cache
compression when we fix this problem :-)

> i had swapping via loop0 , and it was on filesys, quite rare case,
> but happens.

If I understood correctly from your previous email, you have a file
from a FAT partition that you mount in Linux using loop device, right?
I will try to set it up here to try to reproduce the problem if I
notice I don't have clue of what may be wrong. However, let me ask you
something. If you are using a swap file, and mounting it using loop,
why don't you set up Linux to use the swap file directly, since it has
support for it? Is there any issue related to a be a file on a FAT
partition?

> are you having clue what might be b0rked ?

I still don't a clue of what is going on.

> in meantime i'll apply patch to 21rc2-ac1 and test it with page
> cache compression disabled.

Great, tell me if you have problems running comp cache with this
version.

Regards,
-- 
Rodrigo

Re: [lc-devel] hello

[Bob A. <cu...@pb...>]

i patched 2.4.21rc2-ac2 , there were few trivial rejects, i didn't fixed 
only the ones in swapfile.c (i assume they're important when something 
is wrong with swap)

i disabled "support for page cache compression" , compiled and booted.
it works!

i'll install this kernel on my firewall too, which is only non-smp machine 
left :>

btw. what are problems with making patch (or at least parts of it) SMP-able?
i use openmosix patch making cluster out of few single-cpu machines, and on
oM group (ope...@li...) one of programmers got 
information about your patch recently, but never readed it (some other user 
posted question wheter this and few others could be merged into oM)

i think at least swap compression support could be merged in.

btw. performance - my laptop is a bit more responsive, stats show about 50% 
compression ratio and lot of pages compressed. 
one of good things is that when it spins up disk (it is usually off) to swap
something in/out it doesn't 'freeze' system for time disk isn't available.
i think it's because of grouping of swapout requests and compressing them 
before swapping out (so instead of swap out some of memory is just compressed
waiting for disk to spin up, and then -when disk is available-swapped out)

about swapfile - thanks for hint, 
swapon swapfile.img works. 
ofcourse i tested 2.4.21-rc2-ac2-cc with swap via /dev/loop0 before i 
switched to more robust swap method :)



On Saturday 17 May 2003 19:09, Rodrigo Souza de Castro wrote:
> On Sat, May 17, 2003 at 06:16:59PM +0200, curious wrote:
> > i compiled patch from you, and it bug() in filemap.c:66
> > in process loop0 .
> > then in concluded it may be associated with mmaped loop0
> > so i disabled "support for page cache compression"
> > without this ccache works fine and stable :D
>
> That is great, although you probably will want support for page cache
> compression when we fix this problem :-)
>
> > i had swapping via loop0 , and it was on filesys, quite rare case,
> > but happens.
>
> If I understood correctly from your previous email, you have a file
> from a FAT partition that you mount in Linux using loop device, right?
> I will try to set it up here to try to reproduce the problem if I
> notice I don't have clue of what may be wrong. However, let me ask you
> something. If you are using a swap file, and mounting it using loop,
> why don't you set up Linux to use the swap file directly, since it has
> support for it? Is there any issue related to a be a file on a FAT
> partition?
>
> > are you having clue what might be b0rked ?
>
> I still don't a clue of what is going on.
>
> > in meantime i'll apply patch to 21rc2-ac1 and test it with page
> > cache compression disabled.
>
> Great, tell me if you have problems running comp cache with this
> version.
>
> Regards,

-- 
-- 

Re: [lc-devel] hello

[Rodrigo S. de C. <rc...@im...>]

Hi there!

On Sat, May 17, 2003 at 10:06:47PM +0200, Bob Arctor wrote:
> i patched 2.4.21rc2-ac2 , there were few trivial rejects, i didn't
> fixed only the ones in swapfile.c (i assume they're important when
> something is wrong with swap)
> 
> i disabled "support for page cache compression" , compiled and booted.
> it works!

Great!

> i'll install this kernel on my firewall too, which is only non-smp
> machine left :>
> 
> btw. what are problems with making patch (or at least parts of it)
> SMP-able?  i use openmosix patch making cluster out of few
> single-cpu machines, and on oM group
> (ope...@li...) one of programmers got
> information about your patch recently, but never readed it (some
> other user posted question wheter this and few others could be
> merged into oM)

I am very glad to hear that they talked about my patch in the
openmosix mailing list. :-) To answer your question about what are the
problems, I would say that the major problem now is time. As soon as I
can get enough time (or I get help from other programmers), it will be
done. Besides the bug you hit with loop device + swap file and a few
bugs, to check and improve SMP support it one of the top items on my
todo list.

> i think at least swap compression support could be merged in.
> 
> btw. performance - my laptop is a bit more responsive, stats show
> about 50% compression ratio and lot of pages compressed.  one of
> good things is that when it spins up disk (it is usually off) to
> swap something in/out it doesn't 'freeze' system for time disk isn't
> available.  i think it's because of grouping of swapout requests and
> compressing them before swapping out (so instead of swap out some of
> memory is just compressed waiting for disk to spin up, and then
> -when disk is available-swapped out)

A compressed cache that stores only pages backed by swap may improve
your system in a number of scenarios, but not in as many as if we have
page cache support. That is why I suggest you to test comp cache with
page cache support when it is finally stable with swap files.

BTW, I took a little longer to answer because first I wanted to
perform some tests. I found out that there is a bug with swapfile
support, because I guess comp cache code has never been tested with
swap files, only with swap partitions. As soon as I created a swap
file and ran with it, I hit some deadlocks. One is pretty obvious
(shame on me!) and I will fix it asap (now I have to think a little
how to solve it).

> about swapfile - thanks for hint, 
> swapon swapfile.img works. 
> ofcourse i tested 2.4.21-rc2-ac2-cc with swap via /dev/loop0 before i 
> switched to more robust swap method :)

Cool, it is a nicer solution that avoids the loop device overhead.

Regards,
-- 
Rodrigo

Re: [lc-devel] hello

[Bob A. <cu...@pb...>]

> > i patched 2.4.21rc2-ac2 , there were few trivial rejects, i didn't

> > i'll install this kernel on my firewall too, which is only non-smp
> > machine left :>

it works too. 
there is swap over NFS on this firewall , which benefited a _lot_ from 
compression and grouping of requests as it swaps over 10Mbit link.
_here_ i have problems with enabling swap via an file, it is possible 
_only_ via /dev/loop! swapping isn't very intensive, it swaps out some 
inactive data when i log in there and swaps out a bit more when i perform 
administrative tasks, but during normal operation it just swaps out inactive
shells, or inactive services. i felt difference though. compression ratio is 
around 41%!

btw. i lurked into code, and it seems compression algorithm is now determined 
automatically? or it can be changed somehow?

on lowmem systems implementing more complex algorithms would be obsolete, 
but when there is plenty of memory algorithm which would store large book
of possible words could greatly improve caching.
review www.autosophy.com
it is explained well there, and the idea is old and very well known.
if you're worried about legal problems -author (klaus holtz) answers to 
emails and will not have anything against using it, as it isn't really hard
to deduct how to make it by your own, and he just got the idea long time ago, 
before computing was so popular.





> >
> > btw. what are problems with making patch (or at least parts of it)
> > SMP-able?  i use openmosix patch making cluster out of few
> > single-cpu machines, and on oM group
> > (ope...@li...) one of programmers got
> > information about your patch recently, but never readed it (some
> > other user posted question wheter this and few others could be
> > merged into oM)
>
> I am very glad to hear that they talked about my patch in the
> openmosix mailing list. :-) To answer your question about what are the
> problems, I would say that the major problem now is time. As soon as I
> can get enough time (or I get help from other programmers), it will be
> done. Besides the bug you hit with loop device + swap file and a few
> bugs, to check and improve SMP support it one of the top items on my
> todo list.
>
> > i think at least swap compression support could be merged in.
> >
> > btw. performance - my laptop is a bit more responsive, stats show
> > about 50% compression ratio and lot of pages compressed.  one of
> > good things is that when it spins up disk (it is usually off) to
> > swap something in/out it doesn't 'freeze' system for time disk isn't
> > available.  i think it's because of grouping of swapout requests and
> > compressing them before swapping out (so instead of swap out some of
> > memory is just compressed waiting for disk to spin up, and then
> > -when disk is available-swapped out)
>
> A compressed cache that stores only pages backed by swap may improve
> your system in a number of scenarios, but not in as many as if we have
> page cache support. That is why I suggest you to test comp cache with
> page cache support when it is finally stable with swap files.
>
> BTW, I took a little longer to answer because first I wanted to
> perform some tests. I found out that there is a bug with swapfile
> support, because I guess comp cache code has never been tested with
> swap files, only with swap partitions. As soon as I created a swap
> file and ran with it, I hit some deadlocks. One is pretty obvious
> (shame on me!) and I will fix it asap (now I have to think a little
> how to solve it).
>
> > about swapfile - thanks for hint,
> > swapon swapfile.img works.
> > ofcourse i tested 2.4.21-rc2-ac2-cc with swap via /dev/loop0 before i
> > switched to more robust swap method :)
>
> Cool, it is a nicer solution that avoids the loop device overhead.
>
> Regards,

-- 
-- 

Re: [lc-devel] hello

[Rodrigo S. de C. <rc...@im...>]

Hey Bob,

On Mon, May 19, 2003 at 02:08:53AM +0200, Bob Arctor wrote:
> > > i patched 2.4.21rc2-ac2 , there were few trivial rejects, i didn't
> 
> > > i'll install this kernel on my firewall too, which is only
> > > non-smp machine left :>
> 
> it works too. 

It may work, but I don't know how stable it is, since I have never
performed a test on such a system. It is much likelier that you hit a
race condition on a SMP system than on a system with a preempted
kernel, because the former has a more intense concurrency.

> there is swap over NFS on this firewall , which benefited a _lot_
> from compression and grouping of requests as it swaps over 10Mbit
> link. 

Are you using compressed cache and swap compressed with swap over NFS?
It is the first time I heard someone using comp cache with these
systems (although I heard of some development on swap over NFS
sometime ago). That is great.

> _here_ i have problems with enabling swap via an file, it is
> possible _only_ via /dev/loop! 

Oh, yeah, I remember swapon man page warning about that. Actually, the
problem I noticed with compressed cache code is not related to loop
device, but with swap files, so as soon as it works stably with swap
files, I guess you will be able to use NFS over swap.

> swapping isn't very intensive, it swaps out some inactive data when
> i log in there and swaps out a bit more when i perform
> administrative tasks, but during normal operation it just swaps out
> inactive shells, or inactive services. i felt difference
> though. compression ratio is around 41%!

Wow. Since you rely on network transfers, compressing it probably will
help a lot. It is a scenario that I hadn't first taken much into
account. In this particular case the transfer is really important (for
ordinary systems where swap is done on local disk, the access
reduction, rather than transfer, is more important).

> btw. i lurked into code, and it seems compression algorithm is now
> determined automatically? or it can be changed somehow?

The default algorithm, which has achieved the best results, is
LZO. However, you can enable WKdm or WK4x4 changing the "compalg="
kernel parameter.

compalg=0 (WKdm)
compalg=1 (WK4x4)
compalg=2 (LZO, default)

There is no system detection concerning the amount of memory and
compression algorithms that are best suited for each case.

> on lowmem systems implementing more complex algorithms would be obsolete, 
> but when there is plenty of memory algorithm which would store large book
> of possible words could greatly improve caching.

Agreed, that may be an interesting idea. From the experiments I ran,
it is more important the compression ration than the compression
speed. Of course, the speed is important, but if we get a better ratio
with a not very longer time, it would be a better option. That is
something we should consider when thinking about different compression
algoritms for each system.

> review www.autosophy.com
> it is explained well there, and the idea is old and very well known.
> if you're worried about legal problems -author (klaus holtz) answers to 
> emails and will not have anything against using it, as it isn't really hard
> to deduct how to make it by your own, and he just got the idea long time ago, 
> before computing was so popular.

Thanks, I am going to read it carefully later.

Best regards,
-- 
Rodrigo

Re: [lc-devel] hello

[Bob A. <cu...@pb...>]

hmm.. can i enable page cache compression on machine with swap _partition_?
or should i fear of all rw filesystems damage?
i have such machine, and in doubt, it is remote machine, so i fear that i'll
loose it until i'll be able to go there physically ,
if something will go wrong (like /lib/ corruption)

> The default algorithm, which has achieved the best results, is
> LZO. However, you can enable WKdm or WK4x4 changing the "compalg="
> kernel parameter.
yep, WKdm gave me around 61% comp ratio (the lower % of ratio, the better 
compression is, i assume) while LZO around 41%-50% 

> There is no system detection concerning the amount of memory and
> compression algorithms that are best suited for each case.
if CPU is idle cache re-compression could be performed (trying to compress
'better' and re-insert page with 'better' version if possible)
 also, you probably noticed that LZO files can be sometimes compressed twice
(two pass)  with about 5%-20% of further gain (dunno wheter it isn't already 
implemented in LZO)

-- 

Re: [lc-devel] hello

[Rodrigo S. de C. <rc...@im...>]

On Wed, May 21, 2003 at 12:20:47AM +0200, Bob Arctor wrote:
> hmm.. can i enable page cache compression on machine with swap
> _partition_?
> or should i fear of all rw filesystems damage?

That is the sort of system I am used to test and I don't have any bug
report related to FS corruption with the latest code. The latest
testing patch (0.24pre5) has FS corruption only when preempt patch is
enabled. The answer to your question is: there is no warranty, but
given our experience with swap partition, I don't think you will get
FS corruption if using a swap partition.

> i have such machine, and in doubt, it is remote machine, so i fear
> that i'll loose it until i'll be able to go there physically , if
> something will go wrong (like /lib/ corruption)

I hope everything works fine.

> > The default algorithm, which has achieved the best results, is
> > LZO. However, you can enable WKdm or WK4x4 changing the "compalg="
> > kernel parameter.
>
> yep, WKdm gave me around 61% comp ratio (the lower % of ratio, the
> better compression is, i assume) while LZO around 41%-50%

That's right. The lower compression ratio, the better compression you
get. WKdm compression ratio for your data isn't good at all, and this
is the kind of scenario that makes very important the double page size
option (that I suggest to enable).

> > There is no system detection concerning the amount of memory and
> > compression algorithms that are best suited for each case.
>
> if CPU is idle cache re-compression could be performed (trying to
> compress 'better' and re-insert page with 'better' version if
> possible) also, you probably noticed that LZO files can be sometimes
> compressed twice (two pass) with about 5%-20% of further gain (dunno
> wheter it isn't already implemented in LZO)

Using CPU idle time is on todo list too (actually, the "future work"
list), and it may be an interesting idea to recompress. However, you
have to decompress it twice, and that is the performance impact that
worries me.

Regards,
-- 
Rodrigo

Re: [lc-devel] hello

[Bob A. <cu...@pb...>]

> That is the sort of system I am used to test and I don't have any bug
> report related to FS corruption with the latest code. The latest
> testing patch (0.24pre5) has FS corruption only when preempt patch is
> enabled. The answer to your question is: there is no warranty, but
> given our experience with swap partition, I don't think you will get
> FS corruption if using a swap partition.
yes, i don't had any :D

> > i have such machine, and in doubt, it is remote machine, so i fear
> > that i'll loose it until i'll be able to go there physically , if
> > something will go wrong (like /lib/ corruption)
hehe, it turned out that owner of internet link (using winroute to share net, 
an orthodox win user) is not allowing any tcp connections to my machine...
i lost 3 hours trying to 'see antyhing' beyond pings and finding dns names ;)
still no idea how to fix this... but everything worked fine , 
16M of ramdisk, 40M total ram, very slow hdd (1M/sec IDE without dma)
i hope i'll find a way to log in there and stress it further tommorow :>

> Using CPU idle time is on todo list too (actually, the "future work"
> list), and it may be an interesting idea to recompress. However, you
> have to decompress it twice, and that is the performance impact that
> worries me.
hmm. re-compressed (using different algorithm) pages should not take much 
cpu, those compressed twice - indeed. if you worry for performance, there
could be an option to keep some pages both compressed and uncompressed, 
nuking uncompressed pages when doing heavy i/o and in short of memory, then 
possibly recreating most used ones and keeping them uncompressed.
in modern systems CPU's are fastest and cheapest things one can have in 
computer, so i would rather not worry about this (and for users with slow 
cpu's there could be runtime/compile option to disable such features)

i would rather expect gain in performance on high-cached systems (i.e. 1M of 
L1 and 8M of L2 cache which is common on modern x86 systems) where cpu can 
benefit from having compressed data stored in L2 cache (8M is a lot if LZW 
book would be used) and then quickly decompressed to ram/L1 cache . 

if performance is an issue number of accesses of an cached page could be an 
indicator wheter to compress or not compress page . i.e. pages could be not 
cached initially , but only marked in means of frequency of access.
then when idle-mode cache compression would kick in it would compress rarely 
used pages first, then move to more frequently used ones . 
every decompressed page accessed (thresold) times could be also copied as not 
compressed, and pages used by (threshold, i.e. nice=0) processes would be 
considered as 'performance requiring' and not compressed @ all
 

>
> Regards,

-- 
-- 

[lc-devel] oM + cc doesn't mix

[Bob A. <cu...@pb...>]

i tried to mix compressed cache with oM, and no, this is not working :(
i compiled it non-SMP.
it boots, but it panicks @ startx

[lc-devel] Re: [Openmosix-devel] oM + cc doesn't mix

[Moshe B. <mo...@mo...>]

:-) no, the two things won't mix easily.  there's a lot of work 
invovled in making this work.

you have to maintain the cached object structure if you pass them among 
nodes, so you will end up doing lots of compress/decompress which will 
eat up a lot of CPU. Not sure if it makes sense at all.

Moshe


On Thursday, May 22, 2003, at 07:42 Europe/Rome, Bob Arctor wrote:

> i tried to mix compressed cache with oM, and no, this is not working :(
> i compiled it non-SMP.
> it boots, but it panicks @ startx
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: ObjectStore.
> If flattening out C++ or Java code to make your application fit in a
> relational database is painful, don't do it! Check out ObjectStore.
> Now part of Progress Software. http://www.objectstore.net/sourceforge
> _______________________________________________
> Openmosix-devel mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/openmosix-devel
>

Re: [lc-devel] Re: [Openmosix-devel] oM + cc doesn't mixw

[Bob A. <cu...@pb...>]

what about using only compressed swap feature? this souldn't be so hard to 
port and not much cpu-consuming.

On Thursday 22 May 2003 10:31, Moshe Bar wrote:
> :-) no, the two things won't mix easily.  there's a lot of work
>
> invovled in making this work.
>
> you have to maintain the cached object structure if you pass them among
> nodes, so you will end up doing lots of compress/decompress which will
> eat up a lot of CPU. Not sure if it makes sense at all.
>
> Moshe
>
> On Thursday, May 22, 2003, at 07:42 Europe/Rome, Bob Arctor wrote:
> > i tried to mix compressed cache with oM, and no, this is not working :(
> > i compiled it non-SMP.
> > it boots, but it panicks @ startx
> >
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: ObjectStore.
> > If flattening out C++ or Java code to make your application fit in a
> > relational database is painful, don't do it! Check out ObjectStore.
> > Now part of Progress Software. http://www.objectstore.net/sourceforge
> > _______________________________________________
> > Openmosix-devel mailing list
> > Ope...@li...
> > https://lists.sourceforge.net/lists/listinfo/openmosix-devel
>
> -------------------------------------------------------
> This SF.net email is sponsored by: ObjectStore.
> If flattening out C++ or Java code to make your application fit in a
> relational database is painful, don't do it! Check out ObjectStore.
> Now part of Progress Software. http://www.objectstore.net/sourceforge
> _______________________________________________
> linuxcompressed-devel mailing list
> lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxcompressed-devel

-- 
-- 

Re: [lc-devel] Re: [Openmosix-devel] oM + cc doesn't mixw

[Moshe B. <mo...@mo...>]

yes, that makes more sense. Porting it to openMosix should be pretty 
straight forward.

Moshe


On Friday, May 23, 2003, at 04:06 Europe/Rome, Bob Arctor wrote:

> what about using only compressed swap feature? this souldn't be so 
> hard to
> port and not much cpu-consuming.
>
> On Thursday 22 May 2003 10:31, Moshe Bar wrote:
>> :-) no, the two things won't mix easily.  there's a lot of work
>>
>> invovled in making this work.
>>
>> you have to maintain the cached object structure if you pass them 
>> among
>> nodes, so you will end up doing lots of compress/decompress which will
>> eat up a lot of CPU. Not sure if it makes sense at all.
>>
>> Moshe
>>
>> On Thursday, May 22, 2003, at 07:42 Europe/Rome, Bob Arctor wrote:
>>> i tried to mix compressed cache with oM, and no, this is not working 
>>> :(
>>> i compiled it non-SMP.
>>> it boots, but it panicks @ startx
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> This SF.net email is sponsored by: ObjectStore.
>>> If flattening out C++ or Java code to make your application fit in a
>>> relational database is painful, don't do it! Check out ObjectStore.
>>> Now part of Progress Software. http://www.objectstore.net/sourceforge
>>> _______________________________________________
>>> Openmosix-devel mailing list
>>> Ope...@li...
>>> https://lists.sourceforge.net/lists/listinfo/openmosix-devel
>>
>> -------------------------------------------------------
>> This SF.net email is sponsored by: ObjectStore.
>> If flattening out C++ or Java code to make your application fit in a
>> relational database is painful, don't do it! Check out ObjectStore.
>> Now part of Progress Software. http://www.objectstore.net/sourceforge
>> _______________________________________________
>> linuxcompressed-devel mailing list
>> lin...@li...
>> https://lists.sourceforge.net/lists/listinfo/linuxcompressed-devel
>
> -- 
> -- 
>

Re: [lc-devel] Re: [Openmosix-devel] oM + cc doesn't mixw

[Rodrigo S. de C. <rc...@im...>]

Hi Moshe and Bob,

On Fri, May 23, 2003 at 05:52:21AM +0200, Moshe Bar wrote:
> yes, that makes more sense. Porting it to openMosix should be pretty
> straight forward.

Compressed swap is tied to compressed cache implementation, so porting
it (without compressed caching) would require some work. BTW, when I
implemented compressed swap, my focus was mostly swapping on disk,
where the transfer rate isn't a critical matter. I was actually
interested in reducing the number of IO operations, hence checking
this compression impact on the performance. Given the initial design
and implementation, I think there is much room for improvement in this
code, if we came to the conclusion it may be worth adding this feature
to openMosix.

> On Friday, May 23, 2003, at 04:06 Europe/Rome, Bob Arctor wrote:
> 
> >what about using only compressed swap feature? this souldn't be so
> >hard to port and not much cpu-consuming.

-- 
Rodrigo

Re: [lc-devel] Re: [Openmosix-devel] oM + cc doesn't mixw

[Jordi <mu...@wa...>]

> Compressed swap is tied to compressed cache implementation, so porting
> it (without compressed caching) would require some work. BTW, when I
> implemented compressed swap, my focus was mostly swapping on disk,
> where the transfer rate isn't a critical matter. I was actually
> interested in reducing the number of IO operations, hence checking
> this compression impact on the performance. Given the initial design
> and implementation, I think there is much room for improvement in this
> code, if we came to the conclusion it may be worth adding this feature
> to openMosix.
>

It's a long time since i last read code of Openmosix or cc but at that time at 
least it was not difficult to mix both projects. Dealing with compressed 
pages is not more difficult that dealingg with swap space. 
The main problem i see is a philosofical problem.
Openmosix when tryingg to optimice CPU (load balancing processes accordiing to 
CPU usage) assumes that CPU must be optimize but net bandwidth and computer 
I/O is cheap and fast. So it's worthy to sacrify I/O to CPU.
mm-cc is assuming that CPUs doubles their power every 2 years but the rest of 
the computer components advance more slowly. So it's worthy to sacrify CPU to 
I/O.

I don't know  (as i'd make no test) what can happend missing both projects.
Missing both projects correctly seems a different project to me. Where yoou 
can specify if you want to optimize CPU and disk access but no networking or 
the other way round so it manage both openmosix and mm-cc to use more one 
resource or other one.

--
Jordi Polo

Re: [lc-devel] Re: [Openmosix-devel] oM + cc doesn't mixw

[Rodrigo S. de C. <rc...@im...>]

Hi Jordi,

Long time no see!

On Sat, May 24, 2003 at 12:43:27PM +0200, Jordi wrote:
> > Compressed swap is tied to compressed cache implementation, so
> > porting it (without compressed caching) would require some
> > work. BTW, when I implemented compressed swap, my focus was mostly
> > swapping on disk, where the transfer rate isn't a critical
> > matter. I was actually interested in reducing the number of IO
> > operations, hence checking this compression impact on the
> > performance. Given the initial design and implementation, I think
> > there is much room for improvement in this code, if we came to the
> > conclusion it may be worth adding this feature to openMosix.
> >
> 
> It's a long time since i last read code of Openmosix or cc but at
> that time at least it was not difficult to mix both
> projects. Dealing with compressed pages is not more difficult that
> dealingg with swap space.

Great to hear your opinion. I never read OpenMosix code, but my guess
would be the same as yours: supporting compressed pages in OpenMosix
would require much more work than compressed swap.

> The main problem i see is a philosofical problem.  Openmosix when
> tryingg to optimice CPU (load balancing processes accordiing to CPU
> usage) assumes that CPU must be optimize but net bandwidth and
> computer I/O is cheap and fast. So it's worthy to sacrify I/O to
> CPU.  mm-cc is assuming that CPUs doubles their power every 2 years
> but the rest of the computer components advance more slowly. So it's
> worthy to sacrify CPU to I/O.

Yes, we sacrifice CPU to reduce I/O operations. In general, it is
worthwhile for general system performance. 

I don't know much about OpenMosix project, but it still may be worth
to compress a number of pages to reduce the I/O (disk and network) if
the general performance gets some improvement out of it (even if
spending some CPU cycles compressing it). What do you think?

> I don't know (as i'd make no test) what can happend missing both
> projects.  Missing both projects correctly seems a different project
> to me. Where yoou can specify if you want to optimize CPU and disk
> access but no networking or the other way round so it manage both
> openmosix and mm-cc to use more one resource or other one.

Mixing these projects would introduce another variable to compressed
cache code to know when it has to compress some pages. But I don't
know if I would make any distinction between disk and network, since
when you have compressed memory pages, disk may benefit from the
compression, as well as networking (e.g. data migration). I hope it
makes sense.

Regards,
-- 
Rodrigo

Re: [lc-devel] Re: [Openmosix-devel] oM + cc doesn't mix

[Rodrigo S. de C. <rc...@im...>]

On Thu, May 22, 2003 at 10:31:23AM +0200, Moshe Bar wrote:
> :-) no, the two things won't mix easily.  there's a lot of work 
> invovled in making this work.
> 
> you have to maintain the cached object structure if you pass them
> among nodes, so you will end up doing lots of compress/decompress
> which will eat up a lot of CPU. Not sure if it makes sense at all.

I don't know exactly how you pass data among nodes, but wouldn't it
possible to pass pages in compressed cache in the compressed format,
without decompressing them? In this case, it would have to be added to
the compressed cache in the target node, or decompressed if the cc
does not accept it for whatever reason or even if the system doesn't
have a cc. In the latter scenario, the om should be aware that the
data might be received compressed. (I hope this email makes any sense
:-).

> On Thursday, May 22, 2003, at 07:42 Europe/Rome, Bob Arctor wrote:
> >i tried to mix compressed cache with oM, and no, this is not working :(
> >i compiled it non-SMP.
> >it boots, but it panicks @ startx

[]'s
-- 
Rodrigo