devil-linux-develop

[Devil-linux-develop] systems with low memory

[Heiko Z. <he...@zu...>]

Hey guys,

I was just thinking if we should modify all the tar commands we use in 
DL (save-config, init system) to work better on low memory systems.
According to 'man bunzip2' the '-s' parameter should help us, but will 
at least take twice as long.

Without this modification, DL 1.2 will not run on 32 MB systems.

What do you think? I think we should do it.

cu
Heiko

Re: [Devil-linux-develop] systems with low memory

[Diego T. <dt...@co...>]

On Sun, Sep 05, 2004 at 10:14:34AM -0400, Heiko Zuerker wrote:

> Without this modification, DL 1.2 will not run on 32 MB systems.
> 
> What do you think? I think we should do it.

is this change fixing also dl 1.3 on 32mb? if so... could you make it configurable?

-- 
-- gnupg keyfingerprint -- 48AF 5BF9 8F54 2966 64CC  2327 7CD0 DD91 B09D 5799
-- Use of a keyboard or mouse may be linked to serious injuries or disorders.
Diego Torres - dtorres at anthalia dot org - Madrid / España

Re: [Devil-linux-develop] systems with low memory

[Heiko Z. <he...@zu...>]

Diego Torres wrote:

>On Sun, Sep 05, 2004 at 10:14:34AM -0400, Heiko Zuerker wrote:
>
>  
>
>>Without this modification, DL 1.2 will not run on 32 MB systems.
>>
>>What do you think? I think we should do it.
>>    
>>
>
>is this change fixing also dl 1.3 on 32mb? if so... could you make it configurable?
>
>  
>
1.2 has a problem too on 32 MB systems.

I'm currently doing some tests, the '-s' flag is really much slower, but 
alone doesn't fix the problem.
I was thinking about checking if the system has less then 64MB and then 
automatically use the '-s' flag.

Additionally I'm right now running a test without the clamav virus 
signatures and without the squidguard blacklist.
I probably remove those completely, since they're always outdated 
anyway. If somebody uses either clamav or squidguard, the files have to 
be replace anyway.

I keep you updated.

cu
Heiko

Re: [Devil-linux-develop] systems with low memory

[Bruce S. <bw...@ar...>]

> I was just thinking if we should modify all the tar commands we use in 
> DL (save-config, init system) to work better on low memory systems.
> According to 'man bunzip2' the '-s' parameter should help us, but will 
> at least take twice as long.
> 
> Without this modification, DL 1.2 will not run on 32 MB systems.
> 
> What do you think? I think we should do it.

Good idea, but I don't think it'll work.  At least not with floppies.

We'd have to remove the "-j" from tar and run bzip2 separately.  
"-s" means something completely different to the tar command.
And that will cause problems because a floppy isn't big enough to 
hold an uncompressed tar config image.

If we do it anyway for USB drives. I agree with Diego that we should
make it an option to run it either way.  Or maybe better yet, if we
could detect the amount of memory and make it default to the correct
method without specifying an option.

What about busybox?  Would it do us any good?  
Does it have tar and/or bzip2 commands?

 - BS

Re: [Devil-linux-develop] systems with low memory

[Heiko Z. <he...@zu...>]

Bruce Smith wrote:

>>I was just thinking if we should modify all the tar commands we use in 
>>DL (save-config, init system) to work better on low memory systems.
>>According to 'man bunzip2' the '-s' parameter should help us, but will 
>>at least take twice as long.
>>
>>Without this modification, DL 1.2 will not run on 32 MB systems.
>>
>>What do you think? I think we should do it.
>>    
>>
>
>Good idea, but I don't think it'll work.  At least not with floppies.
>
>We'd have to remove the "-j" from tar and run bzip2 separately.  
>"-s" means something completely different to the tar command.
>And that will cause problems because a floppy isn't big enough to 
>hold an uncompressed tar config image.
>  
>
tar -c /etc | bzip2 -s -c > etc.tar.bz2
should work, since the uncompressed version of the tar archive gets 
directly piped into bzip2

>If we do it anyway for USB drives. I agree with Diego that we should
>make it an option to run it either way.  Or maybe better yet, if we
>could detect the amount of memory and make it default to the correct
>method without specifying an option.
>  
>
As I wrote before, we could just always use the lowmemory option when we 
detect a system with less then i.e. 40 MB of total RAM.

>What about busybox?  Would it do us any good?  
>Does it have tar and/or bzip2 commands?
>  
>

It does, but I'm not sure about the memory requirements.

Heiko

Re: [Devil-linux-develop] systems with low memory

[Heiko Z. <he...@zu...>]

Heiko Zuerker wrote:

> Hey guys,
>
> I was just thinking if we should modify all the tar commands we use in 
> DL (save-config, init system) to work better on low memory systems.
> According to 'man bunzip2' the '-s' parameter should help us, but will 
> at least take twice as long.
>
> Without this modification, DL 1.2 will not run on 32 MB systems.
>
> What do you think? I think we should do it.


I just finished my testing and checked in the changes into the 1.2 tree. 
1.3 will come later, I have to finish first the test compile.

I didn't not add any intelligence to use he '-s' flag only on low memory 
systems, since the performance impact is much less then I initially thought.
The archives are so small, that it doesn't take much longer to unpack 
them. We still can add this if we have to, but I would suggest to wait 
and see.
Biggest impact was the removal of the clamav virus signatures and the 
squidguard blacklists.

DL can now boot again with only 20 MB of RAM !
I'm sure we could bring it down to 16, but that's not the goal of DL. So 
the official minimum will be 32 MB.

cu
Heiko

Re: [Devil-linux-develop] systems with low memory

[Friedrich L. <fl...@fl...>]

Heiko Zuerker wrote on 06.09.2004 16:51 MET:
> Heiko Zuerker wrote:
> 
>> Hey guys,
>>
>> I was just thinking if we should modify all the tar commands we use in 
>> DL (save-config, init system) to work better on low memory systems.
>> According to 'man bunzip2' the '-s' parameter should help us, but will 
>> at least take twice as long.
>>
>> Without this modification, DL 1.2 will not run on 32 MB systems.
>>
>> What do you think? I think we should do it.
> 
> 
> 
> I just finished my testing and checked in the changes into the 1.2 tree. 
> 1.3 will come later, I have to finish first the test compile.
> 
> I didn't not add any intelligence to use he '-s' flag only on low memory 
> systems, since the performance impact is much less then I initially 
> thought.
> The archives are so small, that it doesn't take much longer to unpack 
> them. We still can add this if we have to, but I would suggest to wait 
> and see.
> Biggest impact was the removal of the clamav virus signatures and the 
> squidguard blacklists.
> 
> DL can now boot again with only 20 MB of RAM !
> I'm sure we could bring it down to 16, but that's not the goal of DL. So 
> the official minimum will be 32 MB.

Another hint for 32MB systems is to disable jailing for bind as that will save 
you about 12 MB of RAM which is "wasted" by the jail.

I had to do this to get 1.2RC1 working correctly with 32 MB.

-- 
MfG / Regards
Friedrich Lobenstock
____________________________________________________________________
Friedrich Lobenstock                       Linux Services Lobenstock
URL: http://www.lsl.at/                         Email: fl...@fl...
____________________________________________________________________

Re: [Devil-linux-develop] systems with low memory

[Heiko Z. <he...@zu...>]

Friedrich Lobenstock wrote:

> Heiko Zuerker wrote on 06.09.2004 16:51 MET:
>
>> Heiko Zuerker wrote:
>>
>>> Hey guys,
>>>
>>> I was just thinking if we should modify all the tar commands we use 
>>> in DL (save-config, init system) to work better on low memory systems.
>>> According to 'man bunzip2' the '-s' parameter should help us, but 
>>> will at least take twice as long.
>>>
>>> Without this modification, DL 1.2 will not run on 32 MB systems.
>>>
>>> What do you think? I think we should do it.
>>
>>
>>
>>
>> I just finished my testing and checked in the changes into the 1.2 
>> tree. 1.3 will come later, I have to finish first the test compile.
>>
>> I didn't not add any intelligence to use he '-s' flag only on low 
>> memory systems, since the performance impact is much less then I 
>> initially thought.
>> The archives are so small, that it doesn't take much longer to unpack 
>> them. We still can add this if we have to, but I would suggest to 
>> wait and see.
>> Biggest impact was the removal of the clamav virus signatures and the 
>> squidguard blacklists.
>>
>> DL can now boot again with only 20 MB of RAM !
>> I'm sure we could bring it down to 16, but that's not the goal of DL. 
>> So the official minimum will be 32 MB.
>
>
> Another hint for 32MB systems is to disable jailing for bind as that 
> will save you about 12 MB of RAM which is "wasted" by the jail.
>
> I had to do this to get 1.2RC1 working correctly with 32 MB.
>
Yes this is a good point, all the chroot jails need a lot of RAM !

-- 

Regards
  Heiko Zuerker
  http://www.devil-linux.org