Menu
▾
▴
open-sharedroot-devel — Mailinglist for developer discussions
You can subscribe to this list here.
| 2006 |
Jan
(1) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2007 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(105) |
Nov
(10) |
Dec
(7) |
| 2008 |
Jan
|
Feb
(31) |
Mar
(13) |
Apr
(7) |
May
|
Jun
(2) |
Jul
(1) |
Aug
|
Sep
(4) |
Oct
|
Nov
(23) |
Dec
|
| 2009 |
Jan
(25) |
Feb
(24) |
Mar
(10) |
Apr
(8) |
May
(4) |
Jun
(6) |
Jul
(27) |
Aug
(1) |
Sep
|
Oct
(2) |
Nov
(7) |
Dec
(25) |
| 2010 |
Jan
|
Feb
(7) |
Mar
|
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
|
Dec
|
| 2011 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(3) |
Dec
(1) |
|
From: Gordan B. <go...@bo...> - 2009-04-20 20:46:32
|
What's this about: /opt/atix/comoonics-bootimage/boot-scripts/etc/clusterfs-lib.sh: line 1180: typeset: `fuse.glusterfs_chroot_needed': not a valid identifier I haven't seen that one before. Other than that and the ambiguous redirect mentioned previously, the latest version in preview seems to "just work" (tm). :) I've not had that happen in a while with my weird bleeding edge clusters. :) Oh, and the fact that shutdown seems to hang half way through on glusterfs, but that's always been the case with glusterfs - from what I can tell killall seems to kill something important (I excluded glusterfs daemons). But I can live with that at the moment because it's quite hard to debug. ;) Gordan |
|
From: Marc G. <gr...@at...> - 2009-04-14 11:22:42
|
On Tuesday 14 April 2009 11:06:49 Gordan Bobic wrote: > On Tue, 14 Apr 2009 09:59:37 +0200, Marc Grimme <gr...@at...> wrote: > > On Sunday 12 April 2009 13:08:51 Gordan Bobic wrote: > >> Marc Grimme wrote: > >> > On Saturday 04 April 2009 12:59:00 Gordan Bobic wrote: > >> >> This comes up when building the initrd with the latest preview > >> >> version: > >> >> > >> >> Extracting > >> >> rpms.../opt/atix/comoonics-bootimage/boot-scripts/etc/chroot-lib.sh: > >> >> line 292: $filename: ambiguous redirect > >> > > >> > Looks like there is something weird come up in the filelist. Means one > >> > file is misinterpreted. Hmm. > >> > What does file-list.lst (or something like this) look like in the > >> > initrd? > >> > >> file-list.txt is attached. I'm guessing it's the line that says "@map" > >> that's causing a problem. No idea where that came from, though... > > > > No the @map is perfectly normal. This direction tells the mkinitrd to > > copy > > > all > > from next token to next token and ignore the path prefixed. Means: > > cp -a /opt/atix/comoonics-bootimage/boot-scripts/* / > > relative to the initrd. > > So this is very important. > > I see. So which file in the list is causing the problem? I don't know ;-( . Everything looks ok. I don't think it's a big problem and it might go away with newer versions. I would say we leave it there and see if it is still existant with newer version and then look into it in more detail. Ok? > > Gordan > > --------------------------------------------------------------------------- >--- This SF.net email is sponsored by: > High Quality Requirements in a Collaborative Environment. > Download a free trial of Rational Requirements Composer Now! > http://p.sf.net/sfu/www-ibm-com > _______________________________________________ > Open-sharedroot-devel mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/open-sharedroot-devel -- Gruss / Regards, Marc Grimme http://www.atix.de/ http://www.open-sharedroot.org/ |
|
From: Gordan B. <go...@bo...> - 2009-04-14 09:07:15
|
On Tue, 14 Apr 2009 09:59:37 +0200, Marc Grimme <gr...@at...> wrote: > On Sunday 12 April 2009 13:08:51 Gordan Bobic wrote: >> Marc Grimme wrote: >> > On Saturday 04 April 2009 12:59:00 Gordan Bobic wrote: >> >> This comes up when building the initrd with the latest preview >> >> version: >> >> >> >> Extracting >> >> rpms.../opt/atix/comoonics-bootimage/boot-scripts/etc/chroot-lib.sh: >> >> line 292: $filename: ambiguous redirect >> > >> > Looks like there is something weird come up in the filelist. Means one >> > file is misinterpreted. Hmm. >> > What does file-list.lst (or something like this) look like in the >> > initrd? >> >> file-list.txt is attached. I'm guessing it's the line that says "@map" >> that's causing a problem. No idea where that came from, though... > No the @map is perfectly normal. This direction tells the mkinitrd to copy > all > from next token to next token and ignore the path prefixed. Means: > cp -a /opt/atix/comoonics-bootimage/boot-scripts/* / > relative to the initrd. > So this is very important. I see. So which file in the list is causing the problem? Gordan |
|
From: Marc G. <gr...@at...> - 2009-04-14 07:59:45
|
On Sunday 12 April 2009 13:08:51 Gordan Bobic wrote:
> Marc Grimme wrote:
> > On Saturday 04 April 2009 12:59:00 Gordan Bobic wrote:
> >> This comes up when building the initrd with the latest preview version:
> >>
> >> Extracting
> >> rpms.../opt/atix/comoonics-bootimage/boot-scripts/etc/chroot-lib.sh:
> >> line 292: $filename: ambiguous redirect
> >
> > Looks like there is something weird come up in the filelist. Means one
> > file is misinterpreted. Hmm.
> > What does file-list.lst (or something like this) look like in the initrd?
>
> file-list.txt is attached. I'm guessing it's the line that says "@map"
> that's causing a problem. No idea where that came from, though...
No the @map is perfectly normal. This direction tells the mkinitrd to copy all
from next token to next token and ignore the path prefixed. Means:
cp -a /opt/atix/comoonics-bootimage/boot-scripts/* /
relative to the initrd.
So this is very important.
>
> >> Doesn't seem to be terminal, but it might be breaking something.
> >>
> >> Also this:
> >> Copying kernelmodules
> >> (2.6.24.7).../opt/atix/comoonics-bootimage/boot-scripts/etc/clusterfs-li
> >>b.s h: line 1181: fuse_get_drivers: command not found
> >>
> >> I suspect this is caused by GlusterFS coming up as a fuse FS (which it
> >> is), but there's an assumption being made about the function name based
> >> on the FS. I'm not sure what the best solution for that might be. Any
> >> ideas?
> >
> > Yes that's due to the "new" modules list. But it should be
> > glusterfs_get_drivers shouldn't it. In my version I already added this
> > function. It just returns fuse.
>
> Yes, that could be related to the fact that any fuse based FS will show
> up as type fuse.
>
> Here's a line that mount reports:
>
> glusterfs on /home type fuse
> (rw,user_id=0,group_id=0,default_permissions,allow_other,max_read=1048576)
>
> I can see that clusterfs-lib.sh calls ${rootfs}_get_drivers, but which
> file should I add this function to? etc/rhel5/glusterfs-lib.sh?
No you can just add a function called glusterfs_get_driver
to /etc/glusterfs-lib.sh. You'll find my version attached.
But still rootfs should be glusterfs as specified in the cluster.conf as
rootfs. It shouldn't query the mounted filesystems but use the one specified
in cluster.conf.
So still the question is where did rootfs=fuse come from.
Do you only see it when /etc/init.d/bootsr is called or also during boot?
>
> >> I also had an idea for reducing the module count using my old lsmod
> >> method. There is a reference count column, and modules with ref-count of
> >> 0 obviously aren't needed by the logic used. I'll write a patch that
> >> takes this into account when pruning the modules.
> >
> > Ok. That's right.
> > But still there are modules being loaded later that do not necessarily
> > need to get into the initrd ;-) .
>
> Interestingly enough, fuse.ko wasn't being bundled into the initrd by
> default with the latest update. It seems to be an edge case that got
> missed. I can add it explicitly by adding kernel-module-fuse, but this
> seems like a bit of an inconsistency on the module auto-detection. Any
> ideas on how to reconcile that?
What does "get_min_modules" tell you if you call it on comandline:
[root@axqad108-1 ~]#
source /opt/atix/comoonics-bootimage/boot-scripts/etc/std-lib.sh
[root@axqad108-1 ~]# sourceLibs /opt/atix/comoonics-bootimage/boot-scripts
[root@axqad108-1 ~]#
sourceRootfsLibs /opt/atix/comoonics-bootimage/boot-scripts
[root@axqad108-1 ~]# get_min_modules
dlm
lock[_-]dlm
gfs
gfs2
configfs
lock[_-]nolock
sunrpc
nfs
lockd
fscache
nfs[_-]acl
>
> Gordan
--
Gruss / Regards,
Marc Grimme
http://www.atix.de/ http://www.open-sharedroot.org/
|
|
From: Gordan B. <go...@bo...> - 2009-04-12 11:09:27
|
Marc Grimme wrote:
> On Saturday 04 April 2009 12:59:00 Gordan Bobic wrote:
>> This comes up when building the initrd with the latest preview version:
>>
>> Extracting
>> rpms.../opt/atix/comoonics-bootimage/boot-scripts/etc/chroot-lib.sh:
>> line 292: $filename: ambiguous redirect
> Looks like there is something weird come up in the filelist. Means one file is
> misinterpreted. Hmm.
> What does file-list.lst (or something like this) look like in the initrd?
file-list.txt is attached. I'm guessing it's the line that says "@map"
that's causing a problem. No idea where that came from, though...
>> Doesn't seem to be terminal, but it might be breaking something.
>>
>> Also this:
>> Copying kernelmodules
>> (2.6.24.7).../opt/atix/comoonics-bootimage/boot-scripts/etc/clusterfs-lib.s
>> h: line 1181: fuse_get_drivers: command not found
>>
>> I suspect this is caused by GlusterFS coming up as a fuse FS (which it
>> is), but there's an assumption being made about the function name based
>> on the FS. I'm not sure what the best solution for that might be. Any
>> ideas?
> Yes that's due to the "new" modules list. But it should be
> glusterfs_get_drivers shouldn't it. In my version I already added this
> function. It just returns fuse.
Yes, that could be related to the fact that any fuse based FS will show
up as type fuse.
Here's a line that mount reports:
glusterfs on /home type fuse
(rw,user_id=0,group_id=0,default_permissions,allow_other,max_read=1048576)
I can see that clusterfs-lib.sh calls ${rootfs}_get_drivers, but which
file should I add this function to? etc/rhel5/glusterfs-lib.sh?
>> I also had an idea for reducing the module count using my old lsmod
>> method. There is a reference count column, and modules with ref-count of
>> 0 obviously aren't needed by the logic used. I'll write a patch that
>> takes this into account when pruning the modules.
>
> Ok. That's right.
> But still there are modules being loaded later that do not necessarily need to
> get into the initrd ;-) .
Interestingly enough, fuse.ko wasn't being bundled into the initrd by
default with the latest update. It seems to be an edge case that got
missed. I can add it explicitly by adding kernel-module-fuse, but this
seems like a bit of an inconsistency on the module auto-detection. Any
ideas on how to reconcile that?
Gordan
|
|
From: Marc G. <gr...@at...> - 2009-04-04 11:54:09
|
On Saturday 04 April 2009 12:59:00 Gordan Bobic wrote: > This comes up when building the initrd with the latest preview version: > > Extracting > rpms.../opt/atix/comoonics-bootimage/boot-scripts/etc/chroot-lib.sh: > line 292: $filename: ambiguous redirect Looks like there is something weird come up in the filelist. Means one file is misinterpreted. Hmm. What does file-list.lst (or something like this) look like in the initrd? > > Doesn't seem to be terminal, but it might be breaking something. > > Also this: > Copying kernelmodules > (2.6.24.7).../opt/atix/comoonics-bootimage/boot-scripts/etc/clusterfs-lib.s >h: line 1181: fuse_get_drivers: command not found > > I suspect this is caused by GlusterFS coming up as a fuse FS (which it > is), but there's an assumption being made about the function name based > on the FS. I'm not sure what the best solution for that might be. Any > ideas? Yes that's due to the "new" modules list. But it should be glusterfs_get_drivers shouldn't it. In my version I already added this function. It just returns fuse. > > I also had an idea for reducing the module count using my old lsmod > method. There is a reference count column, and modules with ref-count of > 0 obviously aren't needed by the logic used. I'll write a patch that > takes this into account when pruning the modules. Ok. That's right. But still there are modules being loaded later that do not necessarily need to get into the initrd ;-) . > > Gordan > > --------------------------------------------------------------------------- >--- _______________________________________________ > Open-sharedroot-devel mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/open-sharedroot-devel I think on Monday I'll upload a new bunch of rpms. -- Gruss / Regards, Marc Grimme http://www.atix.de/ http://www.open-sharedroot.org/ |
|
From: Gordan B. <go...@bo...> - 2009-04-04 10:59:28
|
This comes up when building the initrd with the latest preview version: Extracting rpms.../opt/atix/comoonics-bootimage/boot-scripts/etc/chroot-lib.sh: line 292: $filename: ambiguous redirect Doesn't seem to be terminal, but it might be breaking something. Also this: Copying kernelmodules (2.6.24.7).../opt/atix/comoonics-bootimage/boot-scripts/etc/clusterfs-lib.sh: line 1181: fuse_get_drivers: command not found I suspect this is caused by GlusterFS coming up as a fuse FS (which it is), but there's an assumption being made about the function name based on the FS. I'm not sure what the best solution for that might be. Any ideas? I also had an idea for reducing the module count using my old lsmod method. There is a reference count column, and modules with ref-count of 0 obviously aren't needed by the logic used. I'll write a patch that takes this into account when pruning the modules. Gordan |
|
From: Gordan B. <go...@bo...> - 2009-03-30 12:13:59
|
On Mon, 30 Mar 2009 08:48:54 +0100, Marc Grimme <gr...@at...> wrote: >> > * Implemented filters to globally remove files from initrd (Gordan's >> > idea) >> >> Is this due to (or coupled with) the approach of dumping all of the >> required files from the rpm database in one pass, rather than querying >> each packake in a separate rpm call? > It were my action points taken from the discussion. One was the single pass > rpm and the other was having global filters. > I tried to implement it with single pass rpm but didn't succeed with > gaining > any performance from it. So I revoked the changed for the single pass rpm > (they were too ugly) and stayed with the "additional" global filters. So we > now have rpm filters that are given per rpm and filters applied globally > (like *.pyc). Can you send me the single-pass implementation you tried? In my preliminary tests the single pass was MASSIVELY faster. Here are my figures: # rpm -qa > packages.list Current equivalent: # time for package in `cat packages.list`; do rpm -q --list $package >> files.list; done real 1m4.470s user 0m13.644s sys 0m4.498s Single pass: # time rpm -q --list `cat packages.list` > files.list real 0m2.501s user 0m1.853s sys 0m0.582s Theoretical best case: # time rpm -qa --list > files.list real 0m2.220s user 0m1.588s sys 0m0.568s The above list included about 700 packages. While this is way more than the initrd will ever contain, it does illustrate that there are potential savings to be made here. Whatever other bottlenecks may affect this, at least rpm time will still get cut by about 25-fold according to these figures. Gordan |
|
From: Marc G. <gr...@at...> - 2009-03-30 07:49:09
|
On Friday 27 March 2009 21:41:37 Gordan Bobic wrote:
> Marc Grimme wrote:
> > * Updated the lite option in mkinitrd <gordans approch> to be more
> > general and not necessarily need "lsmod".
>
> I'm curious - how do you get away from using lsmod to see what's needed?
> Or are you going to make me chance "yum update" and hope it works just
> to see the code? ;)
It's (at least in my view) very easy. I'm just querying all involved parts to
give a list of the drivers they require. This is triggered by the function
get_min_modules(chroot-lib.sh). This function triggers more functions from
the dependent modules like i.e. storage_get_drivers, ${clutype}_get_drivers,
${rootfs}_get_drivers .. (but I didn't integrated drbd, glusterfs and iscsi
yet, but I'll do it today). Then all drivers found in cluster.conf are
included. These are all drivers found as a driver attribute in any path in
the initrd. Finally you can define more drivers either by commandline (-M) or
in /etc/comoonics/bootimage.cfg.
To validate which modules get loaded into initrd you can invoke mkinitrd
with -V.
>
> > * Implemented filters to globally remove files from initrd (Gordan's
> > idea)
>
> Is this due to (or coupled with) the approach of dumping all of the
> required files from the rpm database in one pass, rather than querying
> each packake in a separate rpm call?
It were my action points taken from the discussion. One was the single pass
rpm and the other was having global filters.
I tried to implement it with single pass rpm but didn't succeed with gaining
any performance from it. So I revoked the changed for the single pass rpm
(they were too ugly) and stayed with the "additional" global filters. So we
now have rpm filters that are given per rpm and filters applied globally
(like *.pyc).
>
> > * Implemented an update feature in the initrd (see
> > https://bugzilla.atix.de/show_bug.cgi?id=340). That's a cool one.
> > Updating an initrd within seconds.
>
> Why is this faster than rebuilding from scratch? What gets skipped?
> Compression stage has to remain, so the other big win can only be the
> rpm extraction query stage. If this was implemented in the single pass,
> then is it really that big a win?
First yes. The rpm single pass is no big win. Both setups take 1.11min
(aprox.). The problem is bash but I recognized it later.
I broke the single pass rpm because rpm single pass is nearly exactly the same
speed then the multi pass.
But when playing with the update functionality I implemented a copy based on a
filelist for all files in the initrd with modification timestamps saved
(see .index.list in initrd). The copy ran throught the pregenerated list and
only copied the files that have changed. At this time I had ca. 2000 files in
the initrd. The virtual machine took 40 secs to run throught this process
(with 2000 files). Then I implemented the same functions with python and it
took 4 seconds (see std-lib.sh vs. stdlib.py). So my finding is that bash is
the bottleneck.
So right now the python implementation is only used in the update process and
that takes the time it needs to unpack the initrd copy the files and kernel
if need be (-A, -D) and then pack it again. That's like up to 10 times faster
then building the initrd.
On the long run I have to think about implementing hot spots in the mkinitrd
and initrd process in high level languages (python) as they seem to be much
faster.
>
> > Some thoughts to sum up:
> > 1. It is now possible to use the lite version of mkinitrd. It should
> > reduce the size of the initrd by 50%.
> > Use -L if you want to include all loaded modules or if you need more
> > modules use either -M <modulename> or specify
> > in /etc/comoonics/comoonics-bootimage.cfg
>
> How do you determine what modules are necessary vs what modules are
> currently loaded? Some of them are likely to be ambiguos, depending on
> the use case and rootfs type. Relying on what's loaded in the running
> system seems safer (except for the initial migration from standalone to
> shared root, but that might as well be a full fat initrd to get the
> system bootstrapped).
See above.
All the same when using lite initrd I reduced the size (one kernel in the
initrd) from 45MB to 24MB (including the global filters). That's at least
something ;-) . And that can still be optimized cause right now only *.py[co]
and /usr/share/doc /usr/share/man are excluded.
Marc.
--
Gruss / Regards,
Marc Grimme
http://www.atix.de/ http://www.open-sharedroot.org/
|
|
From: Gordan B. <go...@bo...> - 2009-03-27 20:42:11
|
Marc Grimme wrote: > * Updated the lite option in mkinitrd <gordans approch> to be more general and > not necessarily need "lsmod". I'm curious - how do you get away from using lsmod to see what's needed? Or are you going to make me chance "yum update" and hope it works just to see the code? ;) > * Implemented filters to globally remove files from initrd (Gordan's idea) Is this due to (or coupled with) the approach of dumping all of the required files from the rpm database in one pass, rather than querying each packake in a separate rpm call? > * Implemented an update feature in the initrd (see > https://bugzilla.atix.de/show_bug.cgi?id=340). That's a cool one. Updating an > initrd within seconds. Why is this faster than rebuilding from scratch? What gets skipped? Compression stage has to remain, so the other big win can only be the rpm extraction query stage. If this was implemented in the single pass, then is it really that big a win? > Some thoughts to sum up: > 1. It is now possible to use the lite version of mkinitrd. It should reduce > the size of the initrd by 50%. > Use -L if you want to include all loaded modules or if you need more modules > use either -M <modulename> or specify > in /etc/comoonics/comoonics-bootimage.cfg How do you determine what modules are necessary vs what modules are currently loaded? Some of them are likely to be ambiguos, depending on the use case and rootfs type. Relying on what's loaded in the running system seems safer (except for the initial migration from standalone to shared root, but that might as well be a full fat initrd to get the system bootstrapped). Gordan |
|
From: Marc G. <gr...@at...> - 2009-03-27 16:18:14
|
Hello *, I finished the last implementations and bugfixes of the (nearly) final release candidate of the new comoonics tools (Version 4.5 including comoonics-bootimage). Here are the last relevant changes: * Some bugfixes in hardware detection specially with udev renaming nics and therefore "deadlocking" udevd (timeout gets called) * Bugfixes in powering up nics. They might be called multiple times (i.e. when using bridging and bonding and vlans at once ;-) ) * Verified ocfs2 with RHEL5 and ocfs2 with SLES10 * Updated the lite option in mkinitrd <gordans approch> to be more general and not necessarily need "lsmod". * Implemented filters to globally remove files from initrd (Gordan's idea) * Implemented an update feature in the initrd (see https://bugzilla.atix.de/show_bug.cgi?id=340). That's a cool one. Updating an initrd within seconds. * Fixed bugs (338, 331, ..) Some thoughts to sum up: 1. It is now possible to use the lite version of mkinitrd. It should reduce the size of the initrd by 50%. Use -L if you want to include all loaded modules or if you need more modules use either -M <modulename> or specify in /etc/comoonics/comoonics-bootimage.cfg 2. The mkinitrd -U updates an existing initrd an therefore also speeds up the initrd building. With -A <kernelversion> and -D <kernelversion> even kernels can be exchanged in the same initrd. So that's it. Open issues before having a proper RC is: Retest with SLES10(ocfs2/nfs) and test with FC10(nfs/gfs2). That's it, let me know what you think and have fun. NOTE: Those packages are NOT YET production ready. And they might include bugs as they did not undergo the Q&A. So use them at your own risc. P.S. And may the root always be shared! ;-) On Tuesday 10 February 2009 20:53:18 Marc Grimme wrote: > Hello *, > finally we got the first stage of the new version of the osr cluster suite > in the preview channel. This will be the first release candidate for the > next productive channel. > > NOTE: Those packages are NOT YET production ready. And they might include > bugs as they did not undergo the Q&A. So use them at your own risc. > > 2nd NOTE: Those packages were only tested with OSR<RHEL5> clusters based on > GFS or NFS. OSR<SLES10/ocfs2>,OSR<Fedora/gfs2|nfs> and OSR<RHEL5/ocfs2> > will be tested next. This means you shouldn't use those packages with ocfs2 > as rootfilesystem. > > As this is only a preview update I will sum up the most important changes > not automated: > > - Rewritten hardwaredetection: https://bugzilla.atix.de/show_bug.cgi?id=325 > - Usability review: https://bugzilla.atix.de/show_bug.cgi?id=323 > - Support for breakpoints in bootprocess > - Break shell will always continue the bootprocess when exited > - Bootparameters may be overwritten in Breakshell > - Rootfs Check with bootparameter rootfsck (use with care and up to now > only with GFS) > - NFS OSR will not start scsi detection > - Preview support for OSR on fedora (NFS) > - Preview support for OSR<NFS4> without authentication > - Preview support for OSR<RHEL5 with localfs> > - Many small bugfixes > Patches from Gordan Bobic (all preview status): > - glusterfs support > - Software RAID with md support > - diet patch (new option to mkinitrd <-l> to make initrd smaller) > > Please let me know what you think > > Thanks > Marc > > P.S. May the root always be shared! ;-) > > -- > Gruss / Regards, > > Marc Grimme > http://www.atix.de/ http://www.open-sharedroot.org/ > > > --------------------------------------------------------------------------- >--- Create and Deploy Rich Internet Apps outside the browser with > Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing > skills and code to build responsive, highly engaging applications that > combine the power of local resources and data with the reach of the web. > Download the Adobe AIR SDK and Ajax docs to start building applications > today-http://p.sf.net/sfu/adobe-com > _______________________________________________ > Open-sharedroot-users mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/open-sharedroot-users -- Gruss / Regards, Marc Grimme http://www.atix.de/ http://www.open-sharedroot.org/ |
|
From: Gordan B. <go...@bo...> - 2009-03-16 15:24:00
|
On Mon, 16 Mar 2009 09:45:11 +0100, Marc Grimme <gr...@at...> wrote: >> >> A new parameter (-w) has been added to the fence_tool utility in >> >> RHEL5.3. In the latest open-sharedroot preview packages the -w option >> >> is >> >> used. If your fence_tool version does not support -w, the join command >> >> will fail. The rpm changelog shows the following: >> >> # rpm -q --changelog cman >> >> * Wed Dec 03 2008 Chris Feist <cf...@re...> - 2.0.84-2_el5_2.3 >> >> - Added missing patch to allow delaying fence_tool joins. >> >> - Resolves rhbz#474467 > > Sometimes patches look quite the same. But I was first ;-) . > https://bugzilla.atix.de/show_bug.cgi?id=335 Hehe! :) I didn't know, it wasn't (and still isn't!) in the preview version. :p Gordan |
|
From: Marc G. <gr...@at...> - 2009-03-16 08:45:31
|
On Thursday 12 March 2009 14:36:13 Gordan Bobic wrote: > Taking this thread here since it's not strictly a RHCS issue. > > The same way -c is being detected, would it be OK to auto-detect -m? > > Attached is a patch (currently untested!). > > Does that seem reasonable? It accounts for the fact that there are > versions of cman that can handle -c but not -m. > > Gordan > > Mark Hlawatschek wrote: > > The new parameter is -m. Sorry for the confusion. > > > > -Mark > > > >> A new parameter (-w) has been added to the fence_tool utility in > >> RHEL5.3. In the latest open-sharedroot preview packages the -w option is > >> used. If your fence_tool version does not support -w, the join command > >> will fail. The rpm changelog shows the following: > >> # rpm -q --changelog cman > >> * Wed Dec 03 2008 Chris Feist <cf...@re...> - 2.0.84-2_el5_2.3 > >> - Added missing patch to allow delaying fence_tool joins. > >> - Resolves rhbz#474467 Sometimes patches look quite the same. But I was first ;-) . https://bugzilla.atix.de/show_bug.cgi?id=335 -- Gruss / Regards, Marc Grimme http://www.atix.de/ http://www.open-sharedroot.org/ |
|
From: Gordan B. <go...@bo...> - 2009-03-12 13:39:16
|
Taking this thread here since it's not strictly a RHCS issue. The same way -c is being detected, would it be OK to auto-detect -m? Attached is a patch (currently untested!). Does that seem reasonable? It accounts for the fact that there are versions of cman that can handle -c but not -m. Gordan Mark Hlawatschek wrote: > The new parameter is -m. Sorry for the confusion. > > -Mark > >> A new parameter (-w) has been added to the fence_tool utility in RHEL5.3. >> In the latest open-sharedroot preview packages the -w option is used. If >> your fence_tool version does not support -w, the join command will fail. >> The rpm changelog shows the following: >> # rpm -q --changelog cman >> * Wed Dec 03 2008 Chris Feist <cf...@re...> - 2.0.84-2_el5_2.3 >> - Added missing patch to allow delaying fence_tool joins. >> - Resolves rhbz#474467 |
|
From: Gordan B. <go...@bo...> - 2009-03-12 08:48:48
|
Marc Grimme wrote: > On Thursday 12 March 2009 01:50:41 Gordan Bobic wrote: >> Hi, >> >> I can't seem to find any docs on this, and I've forgotten where it >> should go. Can anybody jog my memory on where I need to put lvm_sup=0 to >> disable lvm support? > > I think there is no lvm_sup=0 anywhere. It is automatically detected. This > means whenever your rootdevice is either 8,ca (scsi) as major it will not > start lvm (see hardware-lib.sh function lvm_check and linux.generic.sh:315). > But if you want to introduce a new bootparameter like lvmsup or the like feel > free to patch it and send over the patch. I'll be happy to apply it. Ah, OK. :) I'll add it as a kernel boot parameter. Gordan |
|
From: Marc G. <gr...@at...> - 2009-03-12 08:19:23
|
On Thursday 12 March 2009 01:50:41 Gordan Bobic wrote: > Hi, > > I can't seem to find any docs on this, and I've forgotten where it > should go. Can anybody jog my memory on where I need to put lvm_sup=0 to > disable lvm support? I think there is no lvm_sup=0 anywhere. It is automatically detected. This means whenever your rootdevice is either 8,ca (scsi) as major it will not start lvm (see hardware-lib.sh function lvm_check and linux.generic.sh:315). But if you want to introduce a new bootparameter like lvmsup or the like feel free to patch it and send over the patch. I'll be happy to apply it. Regards Marc. > > Many thanks. > > Gordan > > --------------------------------------------------------------------------- >--- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and > easily build your RIAs with Flex Builder, the Eclipse(TM)based development > software that enables intelligent coding and step-through debugging. > Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com > _______________________________________________ > Open-sharedroot-devel mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/open-sharedroot-devel -- Gruss / Regards, Marc Grimme http://www.atix.de/ http://www.open-sharedroot.org/ |
|
From: Gordan B. <go...@bo...> - 2009-03-12 00:51:55
|
Hi, I can't seem to find any docs on this, and I've forgotten where it should go. Can anybody jog my memory on where I need to put lvm_sup=0 to disable lvm support? Many thanks. Gordan |
|
From: Gordan B. <go...@bo...> - 2009-02-25 15:22:59
|
On Wed, 25 Feb 2009 16:16:49 +0100, Marc Grimme <gr...@at...> wrote:
> On Wednesday 25 February 2009 15:54:13 Gordan Bobic wrote:
>> First of all, I'd like to say that the latest update that only probes
the
>> @driver listed modules seems to be working great. It has made the
startup
>> process much more stable and much, much faster on the clusters that I
>> tested it on. :)
>>
>> One thing that doesn't seem to have made it into the update, though, is
>> the parallel gzip mod. Shall I write a patch for auto-detecting the
presence
>> of pigz on the system and using that instead of gzip, if available?
>
> Did you try the version uploaded today?
Yes, the one I tried was yum updated at about 1pm GMT.
> There should be an option to change
> the binary to use gzip.
>
> It's the compression_cmd and opts. I didn't test it but I think it should
> do ;-) .
>
> Take the version comoonics-bootimage-1.4-6 (should be the most recent,
> uploaded today).
>
> [root@axqad102_2 boot-scripts]# cat
/etc/comoonics/comoonics-bootimage.cfg
> #****h* comoonics-bootimage/comoonics-bootimage.cfg
> # NAME
> # comoonics-bootimage.cfg
> # $id$
> # DESCRIPTION
> # Configurationfile for the Comoonics bootimage
> # AUTHOR
> # Marc Grimme
> #
> #*******
>
> # Size of the ramdisk defaults to 85536
> # size=85536
>
> # mountpoint to mount the loop-dev to. Defaults to
> ${TMPDIR}/initrd.mnt.XXXXXX
> # mountpoint=$(mktemp -d /tmp/initrd.mnt.XXXXXX)
>
> # force to overwrite the image. Default is to ask
> # force=0
>
> # kernel what kernel to take. Default is $(uname -r)
> # kernel=$(uname -r)
>
> # dep_filename is the file where all dependent files are to be taken from
> # default is /etc/comoonics/bootimage/files-$(uname -r).list
> dep_filename=/etc/comoonics/bootimage/basefiles.list
>
> #
> # rpm_filename is the file where all dependent files are to be taken from
> # default is /etc/comoonics/bootimage/rpms.list
> rpm_filename=/etc/comoonics/bootimage/rpms.list
>
> #
> # what compression tool should be used
> compression_cmd=gzip
> # what options will be passed to command in order to create archive
> compression_opts="-c -9"
>
> # default lockfile points to
> lockfile="/var/lock/mkinitrd.lck"
Ah, OK, I didn't check for that, thanks. :)
Gordan
|
|
From: Marc G. <gr...@at...> - 2009-02-25 15:16:57
|
On Wednesday 25 February 2009 15:54:13 Gordan Bobic wrote: > First of all, I'd like to say that the latest update that only probes the > @driver listed modules seems to be working great. It has made the startup > process much more stable and much, much faster on the clusters that I > tested it on. :) > > One thing that doesn't seem to have made it into the update, though, is the > parallel gzip mod. Shall I write a patch for auto-detecting the presence of > pigz on the system and using that instead of gzip, if available? > > Gordan > > --------------------------------------------------------------------------- >--- Open Source Business Conference (OSBC), March 24-25, 2009, San > Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing > the Enterprise -Strategies to boost innovation and cut costs with open > source participation -Receive a $600 discount off the registration fee with > the source code: SFAD http://p.sf.net/sfu/XcvMzF8H > _______________________________________________ > Open-sharedroot-devel mailing list > Ope...@li... > https://lists.sourceforge.net/lists/listinfo/open-sharedroot-devel Did you try the version uploaded today? There should be an option to change the binary to use gzip. It's the compression_cmd and opts. I didn't test it but I think it should do ;-) . Take the version comoonics-bootimage-1.4-6 (should be the most recent, uploaded today). [root@axqad102_2 boot-scripts]# cat /etc/comoonics/comoonics-bootimage.cfg #****h* comoonics-bootimage/comoonics-bootimage.cfg # NAME # comoonics-bootimage.cfg # $id$ # DESCRIPTION # Configurationfile for the Comoonics bootimage # AUTHOR # Marc Grimme # #******* # Size of the ramdisk defaults to 85536 # size=85536 # mountpoint to mount the loop-dev to. Defaults to ${TMPDIR}/initrd.mnt.XXXXXX # mountpoint=$(mktemp -d /tmp/initrd.mnt.XXXXXX) # force to overwrite the image. Default is to ask # force=0 # kernel what kernel to take. Default is $(uname -r) # kernel=$(uname -r) # dep_filename is the file where all dependent files are to be taken from # default is /etc/comoonics/bootimage/files-$(uname -r).list dep_filename=/etc/comoonics/bootimage/basefiles.list # # rpm_filename is the file where all dependent files are to be taken from # default is /etc/comoonics/bootimage/rpms.list rpm_filename=/etc/comoonics/bootimage/rpms.list # # what compression tool should be used compression_cmd=gzip # what options will be passed to command in order to create archive compression_opts="-c -9" # default lockfile points to lockfile="/var/lock/mkinitrd.lck" -- Gruss / Regards, Marc Grimme http://www.atix.de/ http://www.open-sharedroot.org/ |
|
From: Gordan B. <go...@bo...> - 2009-02-25 14:54:23
|
First of all, I'd like to say that the latest update that only probes the @driver listed modules seems to be working great. It has made the startup process much more stable and much, much faster on the clusters that I tested it on. :) One thing that doesn't seem to have made it into the update, though, is the parallel gzip mod. Shall I write a patch for auto-detecting the presence of pigz on the system and using that instead of gzip, if available? Gordan |
|
From: Gordan B. <go...@bo...> - 2009-02-18 15:00:23
|
First of all, let me say that the resumable on-error shell in the latest preview is made of purest awesome. It makes troubleshooting so much easier. FTW! :D But this got me thinking - when something goes wrong with the rootfs, it's still hard to debug. The machine will just stop, even if the initroot is still hapilly ticking away underneath, albeit with a broken finalroot somewhere off it. So, how about having a getty running on, say, VT12, pointing straight at the initroot strap? The extension of this would also be, for ease of use, having sshd listening on an alternate port in the initroot. Granted, sshd would add to the undesirable bloat of the initrd so it should be optional, but it would be useful for those of us who are both lazy enough to not want to walk to the rack when things go wrong and at the same time poor enough to not have remote console cards in all the servers. I know the feature creep inevitably ends up with thoughts of "Wouldn't it be cool to also add X and [KDE | Gnome]", but is the getty at least deemed reasonable? I'm happy to look into implementing such a thing, but will need some pre-emptive pointers on where best to hook it and suchlike. I would guess it would best be fired up just before chrooting into the final root. The main reason I'm thinking about all this is because I have a really annoying lock-up and shutdown with glusterfs root - it all stops just after "Sending all processes the TERM signal...", even though I have added exclusions for the glusterfs daemon, and it's impossible to debug without a lower level handle to see what is running and what isn't. On a separate note - at what point does the initroot get dumped to the designated disk partition? The reason I ask is because I can't seem to find the logs I expect in /var/comoonics/chroot/var/log/gluster so I'm guessing that it gets shifted after the glusterfs daemon starts up, but I'm not sure. Again, having an initroot getty might help get to the bottom of the problem... Gordan |
|
From: Gordan B. <go...@bo...> - 2009-02-15 14:36:25
|
Marc Grimme wrote: >> So, if I may be so bold to ask - any chance of including pigz in the yum >> repository and adding a dependency on it for the comoonics-bootimage? >> >> I'd submit a create-gfs-initrd-lib.sh patch, but I can't help but feel >> that a patch as small as 2 lines would be a bit lame. :^) > > That's no problem. I will make a variable for the compression program that can > be overwritten in /etc/comoonics/bootimage/bootimage.cfg. > And yes that's only two lines. Awesome, thanks. :) > The more dramatic speed up I'm thinking about is the one with using only one > rpm process and dieting wherever possible and wherever needed. This reduces > the size of initrd and consequently the time for compressing it. Indeed, that was the other thing I was thinking about. It is possible to list multiple packages on the same query, as per what is being done in get_filelist_from_installed_rpm(): rpm -q <package1> <package2> ... <packageN> --dump > But I need to check if the filters are compatible with that concept. If so > that should be pretty easy if not the filters have to be rewritten. Which > should also be possible. I can't see how the current filtering could work with this approach. Only one big list is returned, so all filtering would have to operate on that unified list. This means that filtering would also have to be unified, rather than per-package, which is what I was referring to in the previous post. Filtering out everything under /usr/share/[doc | man], /usr/include, etc. is simple enough, but it isn't as flexible as per-package filtering. I can't see a way around that, though. Gordan |
|
From: Marc G. <gr...@at...> - 2009-02-15 10:14:06
|
On Friday 13 February 2009 19:42:42 Gordan Bobic wrote: > Gordan Bobic wrote: > > 2) Compression speed > > > > 2.1) Using a parallel gzip (http://www.zlib.net/pigz/) compressor, which > > should scale pretty much linearly with the number of CPU cores. A > > (source) RPM seems to be available, but only for SuSE > > (http://rpm.pbone.net/index.php3/stat/4/idpl/11044884/com/pigz-2.1.4-5.1. > >x86_64.rpm.html), so until it is more common, it may have to be made > > available via the comoonics yum repository. > > OK, that was 100% pain-free. The SuSE src.rpm compiles cleanly on > RHEL/CentOS 5.x. The performance scaling is completely linear (32 > seconds with gzip on a quad core Core2, 8 seconds using pigz). ;-) > > The resulting compressed files aren't identical (pigz one is actually a > tiny bit smaller), but when they are ungzipped (ungzipping works using > standard ungzip), the decompressed files are the same, so it seems safe > enough. :) > > Even without any further boosts in compression speed, this seems pretty > worthwhile, and it has the advantage of being 100% clean, OSS, and > extremely painless to implement (whereas ICC will involve opening a > whole new can of worms for a much smaller benefit). > > So, if I may be so bold to ask - any chance of including pigz in the yum > repository and adding a dependency on it for the comoonics-bootimage? > > I'd submit a create-gfs-initrd-lib.sh patch, but I can't help but feel > that a patch as small as 2 lines would be a bit lame. :^) That's no problem. I will make a variable for the compression program that can be overwritten in /etc/comoonics/bootimage/bootimage.cfg. And yes that's only two lines. The more dramatic speed up I'm thinking about is the one with using only one rpm process and dieting wherever possible and wherever needed. This reduces the size of initrd and consequently the time for compressing it. But I need to check if the filters are compatible with that concept. If so that should be pretty easy if not the filters have to be rewritten. Which should also be possible. I'll give feetback. Marc. -- Gruss / Regards, Marc Grimme http://www.atix.de/ http://www.open-sharedroot.org/ |
|
From: Gordan B. <go...@bo...> - 2009-02-13 18:42:49
|
Gordan Bobic wrote: > 2) Compression speed > > 2.1) Using a parallel gzip (http://www.zlib.net/pigz/) compressor, which > should scale pretty much linearly with the number of CPU cores. A (source) > RPM seems to be available, but only for SuSE > (http://rpm.pbone.net/index.php3/stat/4/idpl/11044884/com/pigz-2.1.4-5.1.x86_64.rpm.html), > so until it is more common, it may have to be made available via the > comoonics yum repository. OK, that was 100% pain-free. The SuSE src.rpm compiles cleanly on RHEL/CentOS 5.x. The performance scaling is completely linear (32 seconds with gzip on a quad core Core2, 8 seconds using pigz). The resulting compressed files aren't identical (pigz one is actually a tiny bit smaller), but when they are ungzipped (ungzipping works using standard ungzip), the decompressed files are the same, so it seems safe enough. :) Even without any further boosts in compression speed, this seems pretty worthwhile, and it has the advantage of being 100% clean, OSS, and extremely painless to implement (whereas ICC will involve opening a whole new can of worms for a much smaller benefit). So, if I may be so bold to ask - any chance of including pigz in the yum repository and adding a dependency on it for the comoonics-bootimage? I'd submit a create-gfs-initrd-lib.sh patch, but I can't help but feel that a patch as small as 2 lines would be a bit lame. :^) Gordan |
|
From: Gordan B. <go...@bo...> - 2009-02-13 16:12:00
|
I'm pondering what could be done to speed up mkinitrd, so thought I'd share some thoughts. Apart from removing the unnecessary files from it (omitting .pyo/.pyc files (is this filter included in the current preview), omitting unused kernel modules (diet patch)), there are two things that I can see as making a big difference to the initrd build speed. 1) Extracting file lists from the RPM DB This involces invoking rpm -q for each package, which is slow (lots of process startup latency and churn and not much CPU used, most of the time is spent starting up and tearing down processes. If this could somehow be combined it might just yield a signifficant speed-up. I'll test this theory over the weekend and report back. The one problem with this approach is that there would be no sensible way to apply per-package filtering. 2) Compression speed 2.1) Using a parallel gzip (http://www.zlib.net/pigz/) compressor, which should scale pretty much linearly with the number of CPU cores. A (source) RPM seems to be available, but only for SuSE (http://rpm.pbone.net/index.php3/stat/4/idpl/11044884/com/pigz-2.1.4-5.1.x86_64.rpm.html), so until it is more common, it may have to be made available via the comoonics yum repository. 2.2) Using a decent compiler (Intel's ICC) to squeeze more performance out of the compressor. Intel do provide an optimized gzip library sample (http://www.intel.com/cd/software/products/asmo-na/eng/219967.htm) which according to the docs seems to also be multi-threaded (will have to double check that). My previous tests on Pentium III indicated that ICC built gzip is about 20% faster than the GCC built one. Since IPP includes a highly optimized gzip module, it should do even better, and still stack with the multi-processor scaling. The only problem I can see with 2.2) is that ICC is only free for non-commercial use (OSS is mentioned as an example, and IIRC MySQL used to distribute an ICC built version of their community DB), so that part is something for you guys at Atix to figure out. :) I'll look into this and post some performance results over the weekend, but in the meantime, has anyone got any thoughts on this? Any reason why this might be deemed a bad idea? Gordan |
12 messages has been excluded from this view by a project administrator.
