Re: [OSR-devel] [OSR-users] New Preview RPMs for next Release Candidate of comoonics-bootimage

[Marc G. <gr...@at...>]

On Friday 27 March 2009 21:41:37 Gordan Bobic wrote:
> Marc Grimme wrote:
> > * Updated the lite option in mkinitrd <gordans approch> to be more
> > general and not necessarily need "lsmod".
>
> I'm curious - how do you get away from using lsmod to see what's needed?
> Or are you going to make me chance "yum update" and hope it works just
> to see the code? ;)
It's (at least in my view) very easy. I'm just querying all involved parts to 
give a list of the drivers they require. This is triggered by the function 
get_min_modules(chroot-lib.sh). This function triggers more functions from 
the dependent modules like i.e. storage_get_drivers, ${clutype}_get_drivers, 
${rootfs}_get_drivers .. (but I didn't integrated drbd, glusterfs and iscsi 
yet, but I'll do it today). Then all drivers found in cluster.conf are 
included. These are all drivers found as a driver attribute in any path in 
the initrd. Finally you can define more drivers either by commandline (-M) or 
in /etc/comoonics/bootimage.cfg.
To validate which modules get loaded into initrd you can invoke mkinitrd 
with -V.
>
> > * Implemented filters to globally remove files from initrd (Gordan's
> > idea)
>
> Is this due to (or coupled with) the approach of dumping all of the
> required files from the rpm database in one pass, rather than querying
> each packake in a separate rpm call?
It were my action points taken from the discussion. One was the single pass 
rpm and the other was having global filters.
I tried to implement it with single pass rpm but didn't succeed with gaining 
any performance from it. So I revoked the changed for the single pass rpm 
(they were too ugly) and stayed with the "additional" global filters. So we 
now have rpm filters that are given per rpm and filters applied globally 
(like *.pyc).
>
> > * Implemented an update feature in the initrd (see
> > https://bugzilla.atix.de/show_bug.cgi?id=340). That's a cool one.
> > Updating an initrd within seconds.
>
> Why is this faster than rebuilding from scratch? What gets skipped?
> Compression stage has to remain, so the other big win can only be the
> rpm extraction query stage. If this was implemented in the single pass,
> then is it really that big a win?
First yes. The rpm single pass is no big win. Both setups take 1.11min 
(aprox.). The problem is bash but I recognized it later.
I broke the single pass rpm because rpm single pass is nearly exactly the same 
speed then the multi pass.

But when playing with the update functionality I implemented a copy based on a 
filelist for all files in the initrd with modification timestamps saved 
(see .index.list in initrd). The copy ran throught the pregenerated list and 
only copied the files that have changed. At this time I had ca. 2000 files in 
the initrd. The virtual machine took 40 secs to run throught this process 
(with 2000 files). Then I implemented the same functions with python and it 
took 4 seconds (see std-lib.sh vs. stdlib.py). So my finding is that bash is 
the bottleneck.

So right now the python implementation is only used in the update process and 
that takes the time it needs to unpack the initrd copy the files and kernel 
if need be (-A, -D) and then pack it again. That's like up to 10 times faster 
then building the initrd.

On the long run I have to think about implementing hot spots in the mkinitrd 
and initrd process in high level languages (python) as they seem to be much 
faster.
>
> > Some thoughts to sum up:
> > 1. It is now possible to use the lite version of mkinitrd. It should
> > reduce the size of the initrd by 50%.
> > Use -L if you want to include all loaded modules or if you need more
> > modules use either -M <modulename> or specify
> > in /etc/comoonics/comoonics-bootimage.cfg
>
> How do you determine what modules are necessary vs what modules are
> currently loaded? Some of them are likely to be ambiguos, depending on
> the use case and rootfs type. Relying on what's loaded in the running
> system seems safer (except for the initial migration from standalone to
> shared root, but that might as well be a full fat initrd to get the
> system bootstrapped).
See above.

All the same when using lite initrd I reduced the size  (one kernel in the 
initrd) from 45MB to 24MB (including the global filters). That's at least 
something ;-) . And that can still be optimized cause right now only *.py[co] 
and /usr/share/doc /usr/share/man are excluded.

Marc.

-- 
Gruss / Regards,

Marc Grimme
http://www.atix.de/               http://www.open-sharedroot.org/