Re: [lc-devel] Read the dates on messages.

[Rodrigo S. de C. <rc...@im...>]

On Mon, Oct 13, 2003 at 09:15:06PM -0400, John R Moser wrote:
> Erm, because of my. . . awkward . . . situation, I can't exactly send 
> replies.  I hate butchering the list and breaking the threading but I'm 
> reading mail with Mozilla Thunderbird and sending it by shelling in to 
> the server via ssh and using pine :/  College AIX box won't accept a 
> RELAY request, even with authorization ><.
> 
> Read the dates on the messages.  It seems a few of you are responding to
> my messages in reverse order.
> 
> A quick update as to how things are right now:
> 
> - There's a patch at sf.net/projects/linuxcompressed in patches.  I 
> didn't send it to the mailing list, sorry.  I should have.  I'm a
> little rough to work with since I don't always get proceedure right
> but . . .  erm.  At any rate, it's up there in the patches section.
> RC can take it down if he wants.

I checked the patches sections again, and there is not attached file,
even after logging in. Could you add the patch there or send it to the
list?

> - It works fine, just like the unmodified patch at runtime (except
> for the little sanity check I added when I was having trouble
> tracing the decompression bug).  The bug was an infinite loop I
> introduced.

How have you test stability? What sort of tests, and memory pressure
you forced to check it?

> - LZO IS HOPELESSLY BROKEN.  WKdm is fine.

That's something I am very curious, in particular. What are your
changes that might have broken LZO? It worked find since I first added
it to CC, so I guess a data structure, or a LZO-specific code path
might have been blown it up.

> - Large page sizes (16x for me) will cause a noticable lag in
> high-stress situations, particularly in the fact that spin_lock()
> may occur in 50-200 mS intervals.  I'm trying to buff out the need
> for spin_lock()'s, which will make the lag only be application lag.

Do you have a preemptible kernel? Otherwise, spin_lock()'s don't do
anything, and surely are not the performance bootleneck. How are you
measuring time spent on spinlocks?

Large contiguous pages have a horrible side-effect, which is the
failed allocation attempts. That's awful for us. And it tends to get
worse after some time because memory gets fragmented.

> I haven't confirmed that anything >double has any effect, but I'm
> going to test later with octal and quad. 

That's the same for me. The failed allocations and the inner
management (compaction) have a great cost for the benefits
provided. Another great idea to discuss and implement later consists
of having different page sizes in the CC, so we start allocating, for
example, single pages, and if we detect that the compression ratio
isn't good enough to benefit from compression, we start allocating two
contiguous pages, until we notice it's not useful any longer. (Yes, we
have to discuss it further).

> I'll reflect the results in the Configure.help.  Be aware that
> larger page sizes WILL have an effect for algorithms that are made
> for larger sets of data; my box can do gzip on 256k faster than it
> can read the data from disk, and such higher-intensity algorithms
> will have a better trade-off on higher CPU boxes.  The page size
> should be tailored for the algorithm you intend to use at compile
> time, which for the present is probably going to be WKdm with 2x-8x
> pages.

What's the compression algorithm implemented in gzip? Isn't it an
implementation of Lempel-Ziv?

> - I'm currently doing heavy rewrites of some things in here.  A few
> bad hacks are around in the patch to stave off my breakage of
> things.  They'll probably persist through the next step, but I can
> get some of the things taken care of.

Cool.

Best regards,
-- 
Rodrigo