open-sharedroot-devel

[OSR-devel] Bug Report in rhel5/hardware-lib.sh

[Gordan B. <go...@bo...>]

Hi,

It would appear that 
/opt/atix/comoonics-bootimage/boot-scripts/etc/rhel5/hardware-lib.sh has 
gone through a few changes in the past few months, which, unfortunately, 
break it for me.

The problem is in the ordering of the detected NICs. On one of my 
systems I have a dual e1000 built into the mobo, and an e100 as an 
add-in card. /etc/modprobe.conf lists eth0 and eth1 as the e1000s, and 
eth2 as e100. This works fine with hardware-lib.sh v1.5, but with v1.7 
the ordering seems to be both unstable (about 1/10 of the time it'll 
actually get the NIC ordering as expected and specified in cluster.conf 
and the rest of the time it'll do something different) and inconsistent 
with what is in cluster.conf and modprobe.conf.

The last version that works for me is v1.5, and the latest released 
version (I'm talking about CVS version numbers here) appears to be v1.7 
for this file (in the comoonics-bootimage-1.3-40.noarch.rpm release).

Needless to say, trying to boot off an iSCSI shared root with the NIC 
not starting because eth designation doesn't match the MAC doesn't get 
very far. :-/

On a separate note, would it perhaps be a good idea to also have an 
updinitrd script? After a few versions of the clustering tools and OSR 
tools, it's impossible to tell what bugs could be introduced that break 
things. Granted, indiscriminately doing "yum update" is a bad idea, but 
it happens to the best of us that we miss something that we really ought 
to exclude. But what could be done instead is to have an updinitrd 
script that opens the current initrd and just modifies the handful of 
files that need changing (e.g. adding a service to cluster.conf) before 
re-cpio-ing it. Any thoughts on this idea? I know that in the ideal 
world it shouldn't be needed, but this is exactly what I ended up having 
to do yesterday because new initrds just wouldn't boot (there was an 
additional problem _just_ before mounting the GFS off DRBD where it 
fails and drops to the prompt - I haven't gotten to the bottom of that 
one yet). Interestingly, the latest tools work just fine for GlusterFS.

Gordan

Re: [OSR-devel] Bug Report in rhel5/hardware-lib.sh

[Reiner R. <rot...@at...>]

Hi Gordan,

First of all I have great respect for your work and would like to thank you 
for your effort!

Regarding the update of the initrd you may have a look at the comoonics 
enterprise copy tool (com-ec) which is able to inject a new cluster.conf in 
the existing initrd by re-cpio'ing it.

The usage is quite simple. You need to install the following RPMs:

comoonics-ec-py-0.1-36
comoonics-cs-xsl-ec-0.5-19

Then in /etc/comoonics/enterprisecopy/ there is a XML file called 
updateinitrd.xml. There you need to specify where your boot-device resides 
and what to mount.

For your requirements you may need to adapt the XML file. e.g. to skip the 
mounting of boot and simply stating the location of your initrd.

Best regards,

Reiner

-- 
Gruss / Regards,
Dipl.-Ing. (FH) Reiner Rottmann

ATIX Informationstechnologie und Consulting AG
Einsteinstr. 10 
85716 Unterschleissheim
Deutschland/Germany

Phone: +49-89 452 3538-12     Fax: +49-89 990 1766-0
Email: rot...@at...       PGP Key-ID: 0xCA67C5A6
       www.atix.de  |  www.open-sharedroot.org

Vorstaende: Marc Grimme, Mark Hlawatschek, Thomas Merz (Vors.)
Vorsitzender des Aufsichtsrats: Dr. Martin Buss
Registergericht: Amtsgericht Muenchen
Registernummer: HRB 168930
USt.-Id.: DE209485962

----------------------------------------------------------------------
*** Besuchen Sie uns auf dem ATIX IT Solution Day:
Linux Cluster-Technolgien, am 05.02.2009 in Neuss b. Koeln/Duesseldorf!
www.atix.de/event-archiv/atix-it-solution-day-linux-neuss  ***
----------------------------------------------------------------------

On Wednesday 21 January 2009 02:52:57 am Gordan Bobic wrote:
> Hi,
>
> It would appear that
> /opt/atix/comoonics-bootimage/boot-scripts/etc/rhel5/hardware-lib.sh has
> gone through a few changes in the past few months, which, unfortunately,
> break it for me.
>
> The problem is in the ordering of the detected NICs. On one of my
> systems I have a dual e1000 built into the mobo, and an e100 as an
> add-in card. /etc/modprobe.conf lists eth0 and eth1 as the e1000s, and
> eth2 as e100. This works fine with hardware-lib.sh v1.5, but with v1.7
> the ordering seems to be both unstable (about 1/10 of the time it'll
> actually get the NIC ordering as expected and specified in cluster.conf
> and the rest of the time it'll do something different) and inconsistent
> with what is in cluster.conf and modprobe.conf.
>
> The last version that works for me is v1.5, and the latest released
> version (I'm talking about CVS version numbers here) appears to be v1.7
> for this file (in the comoonics-bootimage-1.3-40.noarch.rpm release).
>
> Needless to say, trying to boot off an iSCSI shared root with the NIC
> not starting because eth designation doesn't match the MAC doesn't get
> very far. :-/
>
> On a separate note, would it perhaps be a good idea to also have an
> updinitrd script? After a few versions of the clustering tools and OSR
> tools, it's impossible to tell what bugs could be introduced that break
> things. Granted, indiscriminately doing "yum update" is a bad idea, but
> it happens to the best of us that we miss something that we really ought
> to exclude. But what could be done instead is to have an updinitrd
> script that opens the current initrd and just modifies the handful of
> files that need changing (e.g. adding a service to cluster.conf) before
> re-cpio-ing it. Any thoughts on this idea? I know that in the ideal
> world it shouldn't be needed, but this is exactly what I ended up having
> to do yesterday because new initrds just wouldn't boot (there was an
> additional problem _just_ before mounting the GFS off DRBD where it
> fails and drops to the prompt - I haven't gotten to the bottom of that
> one yet). Interestingly, the latest tools work just fine for GlusterFS.
>
> Gordan
>
> ---------------------------------------------------------------------------
>--- This SF.net email is sponsored by:
> SourcForge Community
> SourceForge wants to tell your story.
> http://p.sf.net/sfu/sf-spreadtheword
> _______________________________________________
> Open-sharedroot-devel mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/open-sharedroot-devel

Re: [OSR-devel] Bug Report in rhel5/hardware-lib.sh

[Marc G. <gr...@at...>]

Hi Gordan,
On Wednesday 21 January 2009 02:52:57 Gordan Bobic wrote:
> Hi,
>
> It would appear that
> /opt/atix/comoonics-bootimage/boot-scripts/etc/rhel5/hardware-lib.sh has
> gone through a few changes in the past few months, which, unfortunately,
> break it for me.
>
> The problem is in the ordering of the detected NICs. On one of my
> systems I have a dual e1000 built into the mobo, and an e100 as an
> add-in card. /etc/modprobe.conf lists eth0 and eth1 as the e1000s, and
> eth2 as e100. This works fine with hardware-lib.sh v1.5, but with v1.7
> the ordering seems to be both unstable (about 1/10 of the time it'll
> actually get the NIC ordering as expected and specified in cluster.conf
> and the rest of the time it'll do something different) and inconsistent
> with what is in cluster.conf and modprobe.conf.
That's strange. I have the same problems on one cluster like you describe it. 
One time everything works and the other time it doesn't. But all other 
clusters work.

The reason why I changed the hw detection for rhel5 is because it didn't work 
for VMs (especially kvm) and I didn't find any problems on all the other 
clusters (except for the one me and the one from you).

I think I have to look deeper into that matter.

So what you say is if you just change hardware-lib.sh from 1.7 to 1.5 
everything works fine?

Cause I thought it was due to the order (that's what I've changed) of udevd 
and kudzu/modprobe eth* being called. Older versions first called kudzu then 
probed for the nics and then started udevd. 

Now I'm first starting udevd then - if appropriate - kudzu and then probe for 
the NICs. I always thought that it was because of the order. But if the new 
order works with hardware-lib.sh (v1.5) but not for 1.7 it isn't because of 
the order. As the order is defined by linuxrc.generic.sh.

Can you acknowledge that it's only the version of hardware-lib.sh?

>
> The last version that works for me is v1.5, and the latest released
> version (I'm talking about CVS version numbers here) appears to be v1.7
> for this file (in the comoonics-bootimage-1.3-40.noarch.rpm release).
>
> Needless to say, trying to boot off an iSCSI shared root with the NIC
> not starting because eth designation doesn't match the MAC doesn't get
> very far. :-/
Very needless. It's the same for non iscsi clusters ;-) . So this needs to be 
fixed.
>
> On a separate note, would it perhaps be a good idea to also have an
> updinitrd script? After a few versions of the clustering tools and OSR
> tools, it's impossible to tell what bugs could be introduced that break
> things. Granted, indiscriminately doing "yum update" is a bad idea, but
> it happens to the best of us that we miss something that we really ought
> to exclude. But what could be done instead is to have an updinitrd
> script that opens the current initrd and just modifies the handful of
> files that need changing (e.g. adding a service to cluster.conf) before
> re-cpio-ing it. Any thoughts on this idea? I know that in the ideal
> world it shouldn't be needed, but this is exactly what I ended up having
> to do yesterday because new initrds just wouldn't boot (there was an
> additional problem _just_ before mounting the GFS off DRBD where it
> fails and drops to the prompt - I haven't gotten to the bottom of that
> one yet). Interestingly, the latest tools work just fine for GlusterFS.
That's it. It's working for most clusters but some make problems so I need to 
elaborate on this.

The updateinitrd was answered by Reiner already.

Thanks and sorry about that ugly bug.

-- 
Gruss / Regards,

Marc Grimme
http://www.atix.de/               http://www.open-sharedroot.org/

Re: [OSR-devel] Bug Report in rhel5/hardware-lib.sh

[Gordan B. <go...@bo...>]

On Wed, 21 Jan 2009 13:19:45 +0100, Marc Grimme <gr...@at...> wrote:

>> It would appear that
>> /opt/atix/comoonics-bootimage/boot-scripts/etc/rhel5/hardware-lib.sh has
>> gone through a few changes in the past few months, which, unfortunately,
>> break it for me.
>>
>> The problem is in the ordering of the detected NICs. On one of my
>> systems I have a dual e1000 built into the mobo, and an e100 as an
>> add-in card. /etc/modprobe.conf lists eth0 and eth1 as the e1000s, and
>> eth2 as e100. This works fine with hardware-lib.sh v1.5, but with v1.7
>> the ordering seems to be both unstable (about 1/10 of the time it'll
>> actually get the NIC ordering as expected and specified in cluster.conf
>> and the rest of the time it'll do something different) and inconsistent
>> with what is in cluster.conf and modprobe.conf.
> That's strange. I have the same problems on one cluster like you describe
> it. 
> One time everything works and the other time it doesn't. But all other 
> clusters work.
> 
> The reason why I changed the hw detection for rhel5 is because it didn't
> work 
> for VMs (especially kvm) and I didn't find any problems on all the other 
> clusters (except for the one me and the one from you).
> 
> I think I have to look deeper into that matter.

I made a rudimentary attempt at rectifying it by explicitly sorting the
module list, but that didn't fix it. The problem is that the eth* binding
ends up being done in the order the drivers are loaded (i.e. if I load the
e100 driver before the e1000 driver, e100 ends up being eth0). This seems
to override and ignore any settings listed in modprobe.conf, and more
disturbingly, it seems to ignore the by-MAC bindings in cluster.conf which
should really have the highest precedence (but either way they should agree
with modprobe.conf if everything is set up right).

> So what you say is if you just change hardware-lib.sh from 1.7 to 1.5 
> everything works fine?

Yes. Note, however, that it could just be that the failure in 1.5 is always
consistent with my hardware so it always comes up the right way around.
1.7, however, definitely doesn't come up right, and more importantly, it
doesn't come up consistently.

> Cause I thought it was due to the order (that's what I've changed) of
udevd
> and kudzu/modprobe eth* being called. Older versions first called kudzu
> then probed for the nics and then started udevd. 
> 
> Now I'm first starting udevd then - if appropriate - kudzu and then probe
> for 
> the NICs. I always thought that it was because of the order. But if the
new
> 
> order works with hardware-lib.sh (v1.5) but not for 1.7 it isn't because
of
> 
> the order. As the order is defined by linuxrc.generic.sh.
> 
> Can you acknowledge that it's only the version of hardware-lib.sh?

Yes, it's the only file I copied across from the older package. Note,
however, the caveat above - it could just be that it makes things work on
this one system where I observed it. In other words, just because 1.5 makes
it work doesn't mean that the bug is in hardware-lib.sh. It could just be
covering up a problem elsewhere. It could be some kind of a weird kudzu
problem, too - I've found it to be unreliable and break things in the past,
albeit not recently (having said that, it's the first thing I switch off on
a new system, so maybe I just didn't notice before).

>> The last version that works for me is v1.5, and the latest released
>> version (I'm talking about CVS version numbers here) appears to be v1.7
>> for this file (in the comoonics-bootimage-1.3-40.noarch.rpm release).
>>
>> Needless to say, trying to boot off an iSCSI shared root with the NIC
>> not starting because eth designation doesn't match the MAC doesn't get
>> very far. :-/
>
> Very needless. It's the same for non iscsi clusters ;-) . So this needs
to
> be fixed.

Indeed. DRBD is even worse, as it has extra scope for split-brain,
particularly if IP addresses are fail-over resources and they happen to
live on an interface that does end up coming up correctly.

> Thanks and sorry about that ugly bug.

The fact that you observed it, too, is rather a relief, actually. It took
me a fair while and a number of initrd rebuilds and a bit of digging to
make sure that I was seeing what I _thought_ I was seeing, and not a weird
side-effect of something I'd done to the configuration. Please, do post
when you have a fix. :-)

Gordan

Re: [OSR-devel] Bug Report in rhel5/hardware-lib.sh

[Marc G. <gr...@at...>]

Hi,
I opened a bug for this problem
https://bugzilla.atix.de/show_bug.cgi?id=325

I will describe/discuss my findings there.

I think I have a solution already.

Gordan, you might want to add you to this Bug.

Regards Marc.

BTW: I would not use the current comoonics-bootimage from preview for clusters 
with multiple nics (means ones with different drivers)!
Wait a day or two and I'll come up with a new version.
On Wednesday 21 January 2009 16:52:44 Gordan Bobic wrote:
> On Wed, 21 Jan 2009 13:19:45 +0100, Marc Grimme <gr...@at...> wrote:
> >> It would appear that
> >> /opt/atix/comoonics-bootimage/boot-scripts/etc/rhel5/hardware-lib.sh has
> >> gone through a few changes in the past few months, which, unfortunately,
> >> break it for me.
> >>
> >> The problem is in the ordering of the detected NICs. On one of my
> >> systems I have a dual e1000 built into the mobo, and an e100 as an
> >> add-in card. /etc/modprobe.conf lists eth0 and eth1 as the e1000s, and
> >> eth2 as e100. This works fine with hardware-lib.sh v1.5, but with v1.7
> >> the ordering seems to be both unstable (about 1/10 of the time it'll
> >> actually get the NIC ordering as expected and specified in cluster.conf
> >> and the rest of the time it'll do something different) and inconsistent
> >> with what is in cluster.conf and modprobe.conf.
> >
> > That's strange. I have the same problems on one cluster like you describe
> > it.
> > One time everything works and the other time it doesn't. But all other
> > clusters work.
> >
> > The reason why I changed the hw detection for rhel5 is because it didn't
> > work
> > for VMs (especially kvm) and I didn't find any problems on all the other
> > clusters (except for the one me and the one from you).
> >
> > I think I have to look deeper into that matter.
>
> I made a rudimentary attempt at rectifying it by explicitly sorting the
> module list, but that didn't fix it. The problem is that the eth* binding
> ends up being done in the order the drivers are loaded (i.e. if I load the
> e100 driver before the e1000 driver, e100 ends up being eth0). This seems
> to override and ignore any settings listed in modprobe.conf, and more
> disturbingly, it seems to ignore the by-MAC bindings in cluster.conf which
> should really have the highest precedence (but either way they should agree
> with modprobe.conf if everything is set up right).
>
> > So what you say is if you just change hardware-lib.sh from 1.7 to 1.5
> > everything works fine?
>
> Yes. Note, however, that it could just be that the failure in 1.5 is always
> consistent with my hardware so it always comes up the right way around.
> 1.7, however, definitely doesn't come up right, and more importantly, it
> doesn't come up consistently.
>
> > Cause I thought it was due to the order (that's what I've changed) of
>
> udevd
>
> > and kudzu/modprobe eth* being called. Older versions first called kudzu
> > then probed for the nics and then started udevd.
> >
> > Now I'm first starting udevd then - if appropriate - kudzu and then probe
> > for
> > the NICs. I always thought that it was because of the order. But if the
>
> new
>
> > order works with hardware-lib.sh (v1.5) but not for 1.7 it isn't because
>
> of
>
> > the order. As the order is defined by linuxrc.generic.sh.
> >
> > Can you acknowledge that it's only the version of hardware-lib.sh?
>
> Yes, it's the only file I copied across from the older package. Note,
> however, the caveat above - it could just be that it makes things work on
> this one system where I observed it. In other words, just because 1.5 makes
> it work doesn't mean that the bug is in hardware-lib.sh. It could just be
> covering up a problem elsewhere. It could be some kind of a weird kudzu
> problem, too - I've found it to be unreliable and break things in the past,
> albeit not recently (having said that, it's the first thing I switch off on
> a new system, so maybe I just didn't notice before).
>
> >> The last version that works for me is v1.5, and the latest released
> >> version (I'm talking about CVS version numbers here) appears to be v1.7
> >> for this file (in the comoonics-bootimage-1.3-40.noarch.rpm release).
> >>
> >> Needless to say, trying to boot off an iSCSI shared root with the NIC
> >> not starting because eth designation doesn't match the MAC doesn't get
> >> very far. :-/
> >
> > Very needless. It's the same for non iscsi clusters ;-) . So this needs
>
> to
>
> > be fixed.
>
> Indeed. DRBD is even worse, as it has extra scope for split-brain,
> particularly if IP addresses are fail-over resources and they happen to
> live on an interface that does end up coming up correctly.
>
> > Thanks and sorry about that ugly bug.
>
> The fact that you observed it, too, is rather a relief, actually. It took
> me a fair while and a number of initrd rebuilds and a bit of digging to
> make sure that I was seeing what I _thought_ I was seeing, and not a weird
> side-effect of something I'd done to the configuration. Please, do post
> when you have a fix. :-)
>
> Gordan
>
> ---------------------------------------------------------------------------
>--- This SF.net email is sponsored by:
> SourcForge Community
> SourceForge wants to tell your story.
> http://p.sf.net/sfu/sf-spreadtheword
> _______________________________________________
> Open-sharedroot-devel mailing list
> Ope...@li...
> https://lists.sourceforge.net/lists/listinfo/open-sharedroot-devel



-- 
Gruss / Regards,

Marc Grimme
http://www.atix.de/               http://www.open-sharedroot.org/

Re: [OSR-devel] Bug Report in rhel5/hardware-lib.sh

[Gordan B. <go...@bo...>]

Hi,

Replying here because I thought it was too discussiony for a bugzilla
comment.

# A new attribute driver per nic per clusternode was introduced.
# <eth name=".." mac=".." driver=".."/>

I can see that this is useful for heterogenous clusters, but if "driver"
isn't specified, the NIC driver "probing" shouldn't really occur at all. It
should be done according to the content of modprobe.conf. I think this
should be deemed authoritative unless specifically overriden by the driver
parameter in the NIC spec.

Also, what happens if we have an alternating NIC driver setup, e.g.

eth0 e1000
eth1 e100
eth2 e1000

Will this work correctly, or will loading the e1000 driver wrongly make the
two e1000 NICs eth0 and eth1?

If udev configuration is dynamically generated from cluster.conf by MAC
address using a line like:
KERNEL=="eth*", SYSFS{address}=="00:11:22:33:44:55", NAME="eth0"
that should probably suffice.
Unfortunately, AFAIK this is not redundant with modprobe.conf stuff (need
the driver loaded before we can read the MAC). Still, I feel there is a
strong argument for making modprobe.conf the default.

Or, as a potentially easier-to-implement alternative, maybe it would be
better to make the driver parameter mandatory (assuming it isn't at the
moment) and abort mkinitrd if it isn't provided.

Gordan

On Thu, 29 Jan 2009 11:21:07 +0100, Marc Grimme <gr...@at...> wrote:
> Hi,
> I opened a bug for this problem
> https://bugzilla.atix.de/show_bug.cgi?id=325
> 
> I will describe/discuss my findings there.
> 
> I think I have a solution already.
> 
> Gordan, you might want to add you to this Bug.
> 
> Regards Marc.
> 
> BTW: I would not use the current comoonics-bootimage from preview for
> clusters 
> with multiple nics (means ones with different drivers)!
> Wait a day or two and I'll come up with a new version.
> On Wednesday 21 January 2009 16:52:44 Gordan Bobic wrote:
>> On Wed, 21 Jan 2009 13:19:45 +0100, Marc Grimme <gr...@at...> wrote:
>> >> It would appear that
>> >> /opt/atix/comoonics-bootimage/boot-scripts/etc/rhel5/hardware-lib.sh
>> >> has
>> >> gone through a few changes in the past few months, which,
>> >> unfortunately,
>> >> break it for me.
>> >>
>> >> The problem is in the ordering of the detected NICs. On one of my
>> >> systems I have a dual e1000 built into the mobo, and an e100 as an
>> >> add-in card. /etc/modprobe.conf lists eth0 and eth1 as the e1000s,
and
>> >> eth2 as e100. This works fine with hardware-lib.sh v1.5, but with
v1.7
>> >> the ordering seems to be both unstable (about 1/10 of the time it'll
>> >> actually get the NIC ordering as expected and specified in
>> >> cluster.conf
>> >> and the rest of the time it'll do something different) and
>> >> inconsistent
>> >> with what is in cluster.conf and modprobe.conf.
>> >
>> > That's strange. I have the same problems on one cluster like you
>> > describe
>> > it.
>> > One time everything works and the other time it doesn't. But all other
>> > clusters work.
>> >
>> > The reason why I changed the hw detection for rhel5 is because it
>> > didn't
>> > work
>> > for VMs (especially kvm) and I didn't find any problems on all the
>> > other
>> > clusters (except for the one me and the one from you).
>> >
>> > I think I have to look deeper into that matter.
>>
>> I made a rudimentary attempt at rectifying it by explicitly sorting the
>> module list, but that didn't fix it. The problem is that the eth*
binding
>> ends up being done in the order the drivers are loaded (i.e. if I load
>> the
>> e100 driver before the e1000 driver, e100 ends up being eth0). This
seems
>> to override and ignore any settings listed in modprobe.conf, and more
>> disturbingly, it seems to ignore the by-MAC bindings in cluster.conf
>> which
>> should really have the highest precedence (but either way they should
>> agree
>> with modprobe.conf if everything is set up right).
>>
>> > So what you say is if you just change hardware-lib.sh from 1.7 to 1.5
>> > everything works fine?
>>
>> Yes. Note, however, that it could just be that the failure in 1.5 is
>> always
>> consistent with my hardware so it always comes up the right way around.
>> 1.7, however, definitely doesn't come up right, and more importantly, it
>> doesn't come up consistently.
>>
>> > Cause I thought it was due to the order (that's what I've changed) of
>>
>> udevd
>>
>> > and kudzu/modprobe eth* being called. Older versions first called
kudzu
>> > then probed for the nics and then started udevd.
>> >
>> > Now I'm first starting udevd then - if appropriate - kudzu and then
>> > probe
>> > for
>> > the NICs. I always thought that it was because of the order. But if
the
>>
>> new
>>
>> > order works with hardware-lib.sh (v1.5) but not for 1.7 it isn't
>> > because
>>
>> of
>>
>> > the order. As the order is defined by linuxrc.generic.sh.
>> >
>> > Can you acknowledge that it's only the version of hardware-lib.sh?
>>
>> Yes, it's the only file I copied across from the older package. Note,
>> however, the caveat above - it could just be that it makes things work
on
>> this one system where I observed it. In other words, just because 1.5
>> makes
>> it work doesn't mean that the bug is in hardware-lib.sh. It could just
be
>> covering up a problem elsewhere. It could be some kind of a weird kudzu
>> problem, too - I've found it to be unreliable and break things in the
>> past,
>> albeit not recently (having said that, it's the first thing I switch off
>> on
>> a new system, so maybe I just didn't notice before).
>>
>> >> The last version that works for me is v1.5, and the latest released
>> >> version (I'm talking about CVS version numbers here) appears to be
>> >> v1.7
>> >> for this file (in the comoonics-bootimage-1.3-40.noarch.rpm release).
>> >>
>> >> Needless to say, trying to boot off an iSCSI shared root with the NIC
>> >> not starting because eth designation doesn't match the MAC doesn't
get
>> >> very far. :-/
>> >
>> > Very needless. It's the same for non iscsi clusters ;-) . So this
needs
>>
>> to
>>
>> > be fixed.
>>
>> Indeed. DRBD is even worse, as it has extra scope for split-brain,
>> particularly if IP addresses are fail-over resources and they happen to
>> live on an interface that does end up coming up correctly.
>>
>> > Thanks and sorry about that ugly bug.
>>
>> The fact that you observed it, too, is rather a relief, actually. It
took
>> me a fair while and a number of initrd rebuilds and a bit of digging to
>> make sure that I was seeing what I _thought_ I was seeing, and not a
>> weird
>> side-effect of something I'd done to the configuration. Please, do post
>> when you have a fix. :-)
>>
>> Gordan
>>
>>
---------------------------------------------------------------------------
>>--- This SF.net email is sponsored by:
>> SourcForge Community
>> SourceForge wants to tell your story.
>> http://p.sf.net/sfu/sf-spreadtheword
>> _______________________________________________
>> Open-sharedroot-devel mailing list
>> Ope...@li...
>> https://lists.sourceforge.net/lists/listinfo/open-sharedroot-devel

Re: [OSR-devel] Bug Report in rhel5/hardware-lib.sh

[Marc G. <gr...@at...>]

Hi Gordan,
On Thursday 29 January 2009 12:21:52 Gordan Bobic wrote:
> Hi,
>
> Replying here because I thought it was too discussiony for a bugzilla
> comment.
I see ;-)
>
> # A new attribute driver per nic per clusternode was introduced.
> # <eth name=".." mac=".." driver=".."/>
>
> I can see that this is useful for heterogenous clusters, but if "driver"
> isn't specified, the NIC driver "probing" shouldn't really occur at all. It
> should be done according to the content of modprobe.conf. I think this
> should be deemed authoritative unless specifically overriden by the driver
> parameter in the NIC spec.
Yes an no. See below.
>
> Also, what happens if we have an alternating NIC driver setup, e.g.
>
> eth0 e1000
> eth1 e100
> eth2 e1000
>
> Will this work correctly, or will loading the e1000 driver wrongly make the
> two e1000 NICs eth0 and eth1?
Yes I think it will. See below.
>
> If udev configuration is dynamically generated from cluster.conf by MAC
> address using a line like:
> KERNEL=="eth*", SYSFS{address}=="00:11:22:33:44:55", NAME="eth0"
> that should probably suffice.
> Unfortunately, AFAIK this is not redundant with modprobe.conf stuff (need
> the driver loaded before we can read the MAC). Still, I feel there is a
> strong argument for making modprobe.conf the default.
>
> Or, as a potentially easier-to-implement alternative, maybe it would be
> better to make the driver parameter mandatory (assuming it isn't at the
> moment) and abort mkinitrd if it isn't provided.
What we do is start udevd then implicitly let it autoload the mods. Save the 
MACs (note the first udevtrigger is only used to detect the MAC). Then the 
modules are unloaded again. 

And now they are (newly) loaded in the following order.
1. If a driver is defined in cluster.conf (eth@driver) it has precedence
2. Load the eth* which are defined in modprobe.conf
3. Trigger udev
This should still be stable.

Note: The first triggering of udev is just to cause the modules to be loaded 
once to save all MAC addresses. Then they are unloaded again.

But in order to detect the nics I can also change the process like as follows:
1. load drivers as specified in modprobe.conf
2. Save the drivers
3. Trigger udev
4. Save macs
5. Unload all newly loaded drivers
But I think this wouldn't be too universal. As It does not really work stable 
for currupt modprobe.confs. The other way would do the same and additionally 
would also work with corrupt modprobe.confs and @driver in the cluster.conf.

I still think this is a general way which is quite stable and most universal.

-- 
Gruss / Regards,

Marc Grimme
http://www.atix.de/               http://www.open-sharedroot.org/

Re: [OSR-devel] Bug Report in rhel5/hardware-lib.sh

[Gordan B. <go...@bo...>]

On Thu, 29 Jan 2009 16:39:30 +0100, Marc Grimme <gr...@at...> wrote:

[...]
> Note: The first triggering of udev is just to cause the modules to be
> loaded once to save all MAC addresses. Then they are unloaded again.
> 
> But in order to detect the nics I can also change the process like as
> follows:
> 1. load drivers as specified in modprobe.conf
> 2. Save the drivers
> 3. Trigger udev
> 4. Save macs
> 5. Unload all newly loaded drivers
> But I think this wouldn't be too universal. As It does not really work
> stable for currupt modprobe.confs.

Is "it must work with broken/corrupt modprobe.conf" really a reasonable
requirement?

> The other way would do the same and
> additionally would also work with corrupt modprobe.confs and @driver in
the
> cluster.conf.

What happens when @driver and modprobe.conf contradict each other? Which
takes precedence?

> I still think this is a general way which is quite stable and most
> universal.

I agree that it should be stable, but I'm still in two minds about
supporting the case of corrupt modprobe.conf. I can see that different
cluster nodes could have different hardware and thus have different
modprobe.conf-s, which means that there are only two options:

1) Specify drivers in cluster.conf and ignore modprobe.conf completely.
Note that this also applies to storage drivers - if the nodes are different
they could also have different disk controllers (very relevant for
shared-SCSI-bus, DRBD and GlusterFS solutions), which would also cause
similar problems.

2) Load each node's modprobe.conf (hopefully /cdsl.local is mounted off the
shared file system and not a local partition on each disk - not a
requirement (not enforced at least) at the moment!) into the initrd and
analyze at runtime based on which node we are running on. The
scsi_controller drivers would have to be loaded after the NICs are set up
since until we get MACs we don't know which node we're running on.

I can see advantages to both approaches. 1) is more elegant from the
implementation point of view since we only have to configure storage and
NICs for all nodes in one file. 2) is more elegant because there are no
redundant configuration entries between cluster.conf and modprobe.conf.
Having said that, we need at least some of the NIC setup in cluster.conf,
so that already makes the configuration redundancy necessary anyway.

OK, I think I'm convinced - maybe it would be better to ignore
modprobe.conf alltogether.

Does that mean something similar is required for disk controller drivers in
cluster.conf? :)

Gordan

Re: [OSR-devel] Bug Report in rhel5/hardware-lib.sh

[Marc G. <gr...@at...>]

On Thursday 29 January 2009 17:30:49 Gordan Bobic wrote:
> On Thu, 29 Jan 2009 16:39:30 +0100, Marc Grimme <gr...@at...> wrote:
>
> [...]
>
> > Note: The first triggering of udev is just to cause the modules to be
> > loaded once to save all MAC addresses. Then they are unloaded again.
> >
> > But in order to detect the nics I can also change the process like as
> > follows:
> > 1. load drivers as specified in modprobe.conf
> > 2. Save the drivers
> > 3. Trigger udev
> > 4. Save macs
> > 5. Unload all newly loaded drivers
> > But I think this wouldn't be too universal. As It does not really work
> > stable for currupt modprobe.confs.
>
> Is "it must work with broken/corrupt modprobe.conf" really a reasonable
> requirement?
I've seen it not only once. Case 1: mixed hardware. Case 2: Cloned clusters 
forgot to change the modprobe.conf.

I don't know if this is reasonable. I would say it's a positive side effect of 
better supporting mixed hardware and that's the real reason.

There are customers who are using our initrd to being able to boot guest on 
real physical hardware and vice versa if need be. Then that @driver concept 
is quite a nice thing to have.
>
> > The other way would do the same and
> > additionally would also work with corrupt modprobe.confs and @driver in
>
> the
>
> > cluster.conf.
>
> What happens when @driver and modprobe.conf contradict each other? Which
> takes precedence?
@driver. If you leave it out it won't be used. So everything works as before 
but you can add such a thing.
>
> > I still think this is a general way which is quite stable and most
> > universal.
>
> I agree that it should be stable, but I'm still in two minds about
> supporting the case of corrupt modprobe.conf. I can see that different
> cluster nodes could have different hardware and thus have different
> modprobe.conf-s, which means that there are only two options:
>
> 1) Specify drivers in cluster.conf and ignore modprobe.conf completely.
> Note that this also applies to storage drivers - if the nodes are different
> they could also have different disk controllers (very relevant for
> shared-SCSI-bus, DRBD and GlusterFS solutions), which would also cause
> similar problems.
>
> 2) Load each node's modprobe.conf (hopefully /cdsl.local is mounted off the
> shared file system and not a local partition on each disk - not a
> requirement (not enforced at least) at the moment!) into the initrd and
> analyze at runtime based on which node we are running on. The
> scsi_controller drivers would have to be loaded after the NICs are set up
> since until we get MACs we don't know which node we're running on.
>
> I can see advantages to both approaches. 1) is more elegant from the
> implementation point of view since we only have to configure storage and
> NICs for all nodes in one file. 2) is more elegant because there are no
> redundant configuration entries between cluster.conf and modprobe.conf.
> Having said that, we need at least some of the NIC setup in cluster.conf,
> so that already makes the configuration redundancy necessary anyway.
>
> OK, I think I'm convinced - maybe it would be better to ignore
> modprobe.conf alltogether.
>
> Does that mean something similar is required for disk controller drivers in
> cluster.conf? :)
I knew you would come over with something like this and the answer is yes but 
not know.

I want to implement and see the @driver scenario then we can easily add the 
same thing for storage. But there you normally don't have that order problem. 
But still I think it's a good idea to later have it there too.

Marc.

-- 
Gruss / Regards,

Marc Grimme
http://www.atix.de/               http://www.open-sharedroot.org/

Re: [OSR-devel] Bug Report in rhel5/hardware-lib.sh

[Gordan B. <go...@bo...>]

On Thu, 29 Jan 2009 18:03:17 +0100, Marc Grimme <gr...@at...> wrote:

>> > Note: The first triggering of udev is just to cause the modules to be
>> > loaded once to save all MAC addresses. Then they are unloaded again.
>> >
>> > But in order to detect the nics I can also change the process like as
>> > follows:
>> > 1. load drivers as specified in modprobe.conf
>> > 2. Save the drivers
>> > 3. Trigger udev
>> > 4. Save macs
>> > 5. Unload all newly loaded drivers
>> > But I think this wouldn't be too universal. As It does not really work
>> > stable for currupt modprobe.confs.
>>
>> Is "it must work with broken/corrupt modprobe.conf" really a reasonable
>> requirement?
>
> I've seen it not only once. Case 1: mixed hardware. Case 2: Cloned
clusters
> forgot to change the modprobe.conf.

Of course - I have done it more than once myself. :^)
Anyway, as I already said, I'm now fully convinced that your original idea
(@driver) is the best solution. Sorry I doubted it. :)

> I don't know if this is reasonable. I would say it's a positive side
effect
> of better supporting mixed hardware and that's the real reason.
>
> There are customers who are using our initrd to being able to boot guest
on
> real physical hardware and vice versa if need be. Then that @driver
concept
> is quite a nice thing to have.

Agreed.

>> > The other way would do the same and
>> > additionally would also work with corrupt modprobe.confs and @driver
in
>> > the cluster.conf.
>>
>> What happens when @driver and modprobe.conf contradict each other? Which
>> takes precedence?
>
> @driver. If you leave it out it won't be used. So everything works as
> before but you can add such a thing.

OK. What about making @driver mandatory? It would mean there is less scope
for a mistake? Perhaps for now backward compatibility is important for
those who blindly "yum update" and expect everything to still work, but I
think mkinitrd should at least throw a warning if @driver isn't present,
saying that non-use of it is deprecated?

>> > I still think this is a general way which is quite stable and most
>> > universal.
>>
>> I agree that it should be stable, but I'm still in two minds about
>> supporting the case of corrupt modprobe.conf. I can see that different
>> cluster nodes could have different hardware and thus have different
>> modprobe.conf-s, which means that there are only two options:
>>
>> 1) Specify drivers in cluster.conf and ignore modprobe.conf completely.
>> Note that this also applies to storage drivers - if the nodes are
>> different
>> they could also have different disk controllers (very relevant for
>> shared-SCSI-bus, DRBD and GlusterFS solutions), which would also cause
>> similar problems.
>>
>> 2) Load each node's modprobe.conf (hopefully /cdsl.local is mounted off
>> the shared file system and not a local partition on each disk - not a
>> requirement (not enforced at least) at the moment!) into the initrd and
>> analyze at runtime based on which node we are running on. The
>> scsi_controller drivers would have to be loaded after the NICs are set
up
>> since until we get MACs we don't know which node we're running on.
>>
>> I can see advantages to both approaches. 1) is more elegant from the
>> implementation point of view since we only have to configure storage and
>> NICs for all nodes in one file. 2) is more elegant because there are no
>> redundant configuration entries between cluster.conf and modprobe.conf.
>> Having said that, we need at least some of the NIC setup in
cluster.conf,
>> so that already makes the configuration redundancy necessary anyway.
>>
>> OK, I think I'm convinced - maybe it would be better to ignore
>> modprobe.conf alltogether.
>>
>> Does that mean something similar is required for disk controller drivers
>> in cluster.conf? :)
>
> I knew you would come over with something like this and the answer is yes
> but not know.

Sorry. :-( I didn't mean to be difficult, just trying to cover an extra
base that seemed like a logical extension.

> I want to implement and see the @driver scenario then we can easily add
the
> same thing for storage. But there you normally don't have that order
> problem. 
> But still I think it's a good idea to later have it there too.

Great, thanks for clearing it up. Please post when the updated package is
in preview and I'll test it on the cluster that I found to be affected by
the issue. :)

Gordan