open-sharedroot-users

[OSR-users] Sorting of eth ports inside initrd

[Klaus S. <kla...@Ph...>]

Hi,

we did run into trouble with the new production version of Openshared 
Root. We do have 6 Ethernet ports on our servers, 2 of them with the 
internal ports of the system board (broadcomm, bnx2 module), and 4 
additional ports on a add on PCI Express Card (Intel, e1000e Module).

With the old initrd we got the following ordering:

bnx2 Module:  eth0 and eth1
e1000e Module:  eth2 - eth5

Now with the new initrd, something changed with loading modules inside 
the initrd, and we got them exchanged, so e1000e is first (eth0 - eth3), 
and second bnx2  (eth4, eth5).

Of course that didn't work, as our device assignements did not fit any more.


Before we reorder our eth Ports in the Config, is there any way to 
change this behaviour, and fix the ordering of the module loading?

Sincerly,
Klaus Steinberger

Re: [OSR-users] Sorting of eth ports inside initrd

[Marc G. <gr...@at...>]

On Monday 03 August 2009 15:19:41 Klaus Steinberger wrote:
> Hi,
>
> we did run into trouble with the new production version of Openshared
> Root. We do have 6 Ethernet ports on our servers, 2 of them with the
> internal ports of the system board (broadcomm, bnx2 module), and 4
> additional ports on a add on PCI Express Card (Intel, e1000e Module).
>
> With the old initrd we got the following ordering:
>
> bnx2 Module:  eth0 and eth1
> e1000e Module:  eth2 - eth5
>
> Now with the new initrd, something changed with loading modules inside
> the initrd, and we got them exchanged, so e1000e is first (eth0 - eth3),
> and second bnx2  (eth4, eth5).
>
> Of course that didn't work, as our device assignements did not fit any
> more.
>
>
> Before we reorder our eth Ports in the Config, is there any way to
> change this behaviour, and fix the ordering of the module loading?
>
> Sincerly,
> Klaus Steinberger

Yes there is. With latest version of productive/preview. You can define the 
driver per nic.

This means if you specify:
<eth name="eth0" ... driver="bnx2"/>
For this nic bnx2 will be loaded first. 

Do think that could help?

-- 
Gruss / Regards,

Marc Grimme
http://www.atix.de/               http://www.open-sharedroot.org/

Re: [OSR-users] Sorting of eth ports inside initrd

[Klaus S. <kla...@Ph...>]

Hi,


>> Before we reorder our eth Ports in the Config, is there any way to
>> change this behaviour, and fix the ordering of the module loading?

> Yes there is. With latest version of productive/preview. You can define the 
> driver per nic.
> 
> This means if you specify:
> <eth name="eth0" ... driver="bnx2"/>
> For this nic bnx2 will be loaded first. 
> 
> Do think that could help?

It seems to not work. If I specify driver="bnx2" it runs into the 
comoonics debug shell as it seems to not find parameters correctly.

Now I tried resorting interfaces, but it got more bad, seems like 
loading order is really unpredictable.

Sincerly,
Klaus

Re: [OSR-users] Sorting of eth ports inside initrd

[Marc G. <gr...@at...>]

On Tuesday 04 August 2009 11:19:58 Klaus Steinberger wrote:
> Hi,
>
> >> Before we reorder our eth Ports in the Config, is there any way to
> >> change this behaviour, and fix the ordering of the module loading?
> >
> > Yes there is. With latest version of productive/preview. You can define
> > the driver per nic.
> >
> > This means if you specify:
> > <eth name="eth0" ... driver="bnx2"/>
> > For this nic bnx2 will be loaded first.
> >
> > Do think that could help?
>
> It seems to not work. If I specify driver="bnx2" it runs into the
> comoonics debug shell as it seems to not find parameters correctly.
>
> Now I tried resorting interfaces, but it got more bad, seems like
> loading order is really unpredictable.
>
> Sincerly,
> Klaus

Did you specify the driver at every nodes eth element?
If you don't specify the @driver at every clusternode in the cluster.conf udev 
is started for detecting the nic. If you want to have a static configuration 
use the @driver in every eth-tag you are using. Even if you use bridges.

My config looks as follows and it works very much predictable:
<?xml version='1.0' encoding='UTF-8'?>
<cluster config_version='35' name='clu_generix' type='gfs'>
  <clusternodes>
    <clusternode votes='1' nodeid='2' name='generix2.local'>
      <com_info>
        <eth mac='...' name='eth1' bridge="clusterbr0" driver="bridge"/>
	<eth type="bridge" name="clusterbr0" ip="10.10.10.82" mask='255.255.255.0' 
driver="tg3"/>
        <rootvolume name='/dev/vg_clu_generix_s/lv_sharedroot'  
mountopts="noatime"/>
        <fenceackserver passwd='XXX' user='root'/>
        <chrootenv mountpoint="/var/comoonics/chroot" fstype="ext3" 
device="/dev/vg_local/lv_chroot" chrootdir="/var/comoonics/chroot" 
driver="sata_mv libata sata_svw sata_mv"/>
        <syslog name="10.10.10.1"/>
        <scsi failover="mapper"/>
      </com_info>
      <fence>
       <method name="1">
	       <device name="fence_apc" port="2"/>
       </method>
       <method name="2">	
	       <device name="fence_sanbox2" port="6"/>
       </method>
      </fence>
    </clusternode>
    <clusternode votes='1' nodeid='3' name='generix3.local'>
      <com_info>
        <eth mac='...' name='eth1' bridge="clusterbr0" driver="bridge"/>
	<eth type="bridge" name="clusterbr0" ip="10.10.10.83" mask='255.255.255.0' 
driver="tg3"/>
        <rootvolume name='/dev/vg_clu_generix_s/lv_sharedroot' 
mountopts="defaults"/>
        <fenceackserver passwd='XXX' user='root'/>
        <chrootenv mountpoint="/var/comoonics/chroot" fstype="ext3" 
device="/dev/vg_local/lv_chroot" chrootdir="/var/comoonics/chroot" 
driver="sata_mv libata sata_svw sata_mv"/>
        <syslog name="10.10.10.1"/>
        <scsi failover="mapper"/>
      </com_info>
      <fence>
        <method name="1">
		<device name="fence_apc" port="3"/>
        </method>
        <method name="2">
                <device name="fence_ipmilan" ipaddr="192.168.3.90"/>
	</method>
      </fence>
    </clusternode>
    <clusternode votes='1' nodeid='4' name='generix4.local'>
      <com_info>
        <eth name='eth0' mac="..." driver="r8169"/>
        <eth name='eth1' mac="..." driver="e1000" bridge="clusterbr0"/>
	<eth type="bridge" name="clusterbr0" ip="10.10.10.84" mask='255.255.255.0' 
driver="bridge"/>
        <rootvolume name='/dev/vg_clu_generix_s/lv_sharedroot' 
mountopts="defaults"/>
        <fenceackserver passwd='XXX' user='root'/>
        <chrootenv mountpoint="/var/comoonics/chroot" fstype="ext3" 
device="/dev/vg_local/lv_chroot" chrootdir="/var/comoonics/chroot" 
driver="sata_mv libata sata_svw sata_mv ata_piix"/>
        <syslog name="10.10.10.1"/>
        <scsi failover="mapper"/>
      </com_info>
      <fence>
        <method name="1">
		<device name="fence_apc" port="4"/>
        </method>
       <method name="2">	
	       <device name="fence_sanbox2" port="7"/>
       </method>
      </fence>
    </clusternode>
  </clusternodes>
...

-- 
Gruss / Regards,

Marc Grimme
Phone: +49-89 452 3538-14
http://www.atix.de/               http://www.open-sharedroot.org/

ATIX Informationstechnologie und Consulting AG | Einsteinstrasse 10 |
85716 Unterschleissheim | www.atix.de | www.open-sharedroot.org

Registergericht: Amtsgericht Muenchen, Registernummer: HRB 168930, USt.-Id.: 
DE209485962 | Vorstand: Marc Grimme, Mark Hlawatschek, Thomas Merz (Vors.) |
Vorsitzender des Aufsichtsrats: Dr. Martin Buss

Re: [OSR-users] Sorting of eth ports inside initrd

[Klaus S. <kla...@Ph...>]

Hi,

> Did you specify the driver at every nodes eth element?

ahh, I needed the driver parameter also in the bonding entry. Now it works.

Thanks a lot!

Sincerly,
Klaus

Re: [OSR-users] Sorting of eth ports inside initrd

[Klaus S. <kla...@Ph...>]

Hi,


>> Did you specify the driver at every nodes eth element?
> 
> ahh, I needed the driver parameter also in the bonding entry. Now it works.

There is now one problem left:

Our bonding device has to be set to the following options.
options bond0 miimon=100 mode=1
This in modprobe.conf, but the new initrd seems to not use that?

How can I specify that parameters?


Btw. where can I find the complete documentation for the com_info 
section? There are many howto's for different types of clusters, but is 
there any central point of documentation for com_info?

Sincerly,
Klaus

Re: [OSR-users] Sorting of eth ports inside initrd

[Marc G. <gr...@at...>]

On Tuesday 04 August 2009 12:29:34 Klaus Steinberger wrote:
> Hi,
>
> >> Did you specify the driver at every nodes eth element?
> >
> > ahh, I needed the driver parameter also in the bonding entry. Now it
> > works.
>
> There is now one problem left:
>
> Our bonding device has to be set to the following options.
> options bond0 miimon=100 mode=1
> This in modprobe.conf, but the new initrd seems to not use that?
It should be "options bonding miimon=100 mode=1". Shouldn't it?
>
> How can I specify that parameters?
The /etc/modprobe.conf should end up in the initrd.
>
>
> Btw. where can I find the complete documentation for the com_info
> section? There are many howto's for different types of clusters, but is
> there any central point of documentation for com_info?
Not yet. We are going to have a consolidated Howto with also reference to 
com_info (see 
http://www.open-sharedroot.org/documentation/general-osr-howto). Some bits 
are also to be found in the Administrator Handbook 
(http://www.open-sharedroot.org/documentation/administrators-handbook).
Sorry about that.
>
> Sincerly,
> Klaus

-- 
Gruss / Regards,

Marc Grimme
http://www.atix.de/               http://www.open-sharedroot.org/

Re: [OSR-users] Sorting of eth ports inside initrd

[Klaus S. <kla...@Ph...>]

Hi,

> It should be "options bonding miimon=100 mode=1". Shouldn't it?

yep, that was some skeleton in the closet. In the old version the 
options bond0 did work .....

Now it works completly.

Sincerly,
Klaus