OpenSSI Clusters for Linux

Hi,

I found some Assert failures in the logs; post Openssi-1.9.1-Fedora here.=
=20
Looks like another creepy bug is creeping up...

I've included some context, maybe this will give some initial ideas.

Sep 1 21:57:05 node2 kernel: drbd0: drbd0_worker [131474]: cstate SyncTarge=
t=20
--> Connected
Sep 1 21:57:05 node2 kernel: drbd1: Syncer continues.
Sep 1 21:57:05 node2 kernel: drbd1: drbd0_worker [131474]: cstate=20
PausedSyncT --> SyncTarget
Sep 1 21:58:02 node2 kernel: IPVS: Scheduler module ip_vs_seduster/node2/=
=20
not found
?
Sep 1 18:59:32 node2 kernel: Assertion failed! svr && !inval(1,svr),=20
cluster/ssi/util/rmtfb.c, rmtfb_newcli, line=3D822
Sep 1 18:59:32 node2 kernel: ACPI: Power Button (FF) [PWRF]
Sep 1 18:59:32 node2 kernel: usbcore: registered new driver usbfs
.............
Sep 1 18:59:33 node2 kernel: eth0: link up, 100Mbps, full-duplex, lpa 0x41E=
1
Sep 1 18:59:40 node2 kernel: IPVS: sync thread started: state =3D BACKUP,=
=20
mcast_ifn =3D eth1, syncid =3D 32
Sep 1 18:59:40 node2 kernel: IPVS: [sed] scheduler registered.
Sep 1 21:21:29 node2 kernel: Assertion failed! svr && !inval(1,svr),=20
cluster/ssi/util/rmtfb.c, rmtfb_newcli, line=3D822
Sep 1 21:23:02 node2 kernel: kjournald starting. Commit interval 5 seconds
Sep 1 21:23:02 node2 kernel: EXT3 FS on hdc1, internal journal
Sep 1 21:23:02 node2 kernel: EXT3-fs: mounted filesystem with ordered data=
=20
mode.
Sep 1 21:23:31 node2 kernel: kjournald starting. Commit interval 5 seconds
.............
Sep 1 21:57:00 node2 kernel: drbd2: drbd2_receiver [131530]: cstate=20
SyncTarget --> PausedSyncT
Sep 1 21:57:00 node2 kernel: drbd2: Syncer waits for sync group.
Sep 1 21:57:00 node2 kernel: drbd2: Secondary/Secondary -->=20
Secondary/Primary
Sep 1 21:57:00 node2 kernel: IPVS: [sed] scheduler registered.
Sep 1 21:57:00 node2 kernel: Assertion failed! svr && !inval(1,svr),=20
cluster/ssi/util/rmtfb.c, rmtfb_newcli, line=3D822
Sep 1 21:57:00 node2 kernel: usbcore: registered new driver usbfs
Sep 1 21:57:00 node2 kernel: usbcore: registered new driver hub
Sep 1 21:57:00 node2 kernel: USB Universal Host Controller Interfa
...........
Sep 1 23:38:36 node2 kernel: This Cluster Node: 2
Sep 1 23:38:36 node2 kernel: Potential Initnode(s): 1:10.xxxxxxxxxxxxxxxxxx=
x
Sep 1 23:38:36 node2 kernel:=20
Sep 1 23:38:36 node2 kernel: Name server registered with clms
Sep 1 23:38:36 node2 kernel: ipcname_read completed
Sep 1 23:38:36 node2 kernel: Assertion failed! svr && !inval(1,svr),=20
cluster/ssi/util/rmtfb.c, rmtfb_newcli, line=3D822
Sep 1 23:38:36 node2 kernel: usbcore: registered new driver usbfs
Sep 1 23:38:36 node2 kernel: usbcore: registered new driver hub
Sep 1 23:38:36 node2 kernel: USB Universal Host Controller Interface driver=
=20
v2.2
Sep 1 23:38:36 node2 kernel: ACPI: PCI interrupt 0000:00:04.3[D] -> GSI 7=
=20
(level, low) -> IRQ 7
Sep 1 23:38:36 node2 kernel: uhci_h
...........
Sep 2 00:22:14 node2 kernel:=20
Sep 2 00:22:14 node2 kernel: Name server registered with clms
Sep 2 00:22:14 node2 kernel: ipcname_read completed
Sep 2 00:22:14 node2 kernel: Assertion failed! svr && !inval(1,svr),=20
cluster/ssi/util/rmtfb.c, rmtfb_newcli, line=3D822
Sep 2 00:22:14 node2 kernel: usbcore: registered new driver usbfs
Sep 2 00:22:14 node2 kernel: usbcore: registered new driver hub
...........
Sep 2 02:58:05 node2 kernel: drbd2: drbd2_receiver [131506]: cstate=20
SyncTarget --> PausedSyncT
Sep 2 02:58:05 node2 kernel: drbd2: Syncer waits for sync group.
Sep 2 02:58:05 node2 kernel: Assertion failed! svr && !inval(1,svr),=20
cluster/ssi/util/rmtfb.c, rmtfb_newcli, line=3D822
Sep 2 02:58:05 node2 kernel: usbcore: registered new driver usbfs
Sep 2 02:58:05 node2 kernel: usbcore: registered=20
..........
Sep 2 03:58:22 node2 kernel: drbd2: Resync done (total 1 sec; paused 0 sec;=
=20
0 K/sec)
Sep 2 03:58:22 node2 kernel: drbd2: drbd2_receiver [131506]: cstate=20
SyncTarget --> Connected
Sep 2 03:58:22 node2 kernel: Assertion failed! svr && !inval(1,svr),=20
cluster/ssi/util/rmtfb.c, rmtfb_newcli, line=3D822
Sep 2 03:58:22 node2 kernel: drbd0: Resync done (total 7 sec; paused 0 sec;=
=20
3720 K/sec)
Sep 2 03:58:22 node2 kernel: drbd0: drbd0_worker [131450]: cstate SyncTarge=
t=20
--> Connected
Sep 2 03:58:22 node2 kernel: drbd1: Syncer continu
..........
ep 2 05:38:34 node2 kernel: drbd2: Resync done (total 1 sec; paused 0 sec; =
0=20
K/sec)
Sep 2 05:38:34 node2 kernel: drbd2: drbd2_receiver [131506]: cstate=20
SyncTarget --> Connected
Sep 2 05:38:34 node2 kernel: Assertion failed! svr && !inval(1,svr),=20
cluster/ssi/util/rmtfb.c, rmtfb_newcli, line=3D822
Sep 2 05:38:34 node2 kernel: usbcore: registered new driver usbfs
Sep 2 05:38:34 node2 kernel: usbcore: registered new driver hub
Sep 2 05:38:34 node2 kernel: US
....


Roger

Iain Young wrote:

>However, I did try to install the packages on a pure debian sid
>machine, and I had dependency issues, but it may just be revision
>numbers, so I might try modifying the depends, but there's been
>a fair few major migrations in sid post the sarge release.
>  
>
As far as I can tell the big problems in trying to run OpenSSI 1.9 on 
sid at the moment are caused by the libc6 migration process (new g++ 
abi) and udev:

If you try to use anything that needs the new g++4.0 libraries it'll 
ruin your mkinitrd-tools and e2fsprogs.

(for mkinitrd tools I just grabbed the OpenSSI version, made the one 
patch needed to mkinitrd for libc6 and made a new version with a bigger 
version number.  For e2fsprogs it's a bit harder, you need to copy 
OpenSSI version of fsck into the debian source).

(I'll try and put the debs & source on my website soon).

You can't use hald 'cos it needs udev >= 0.063, OpenSSI has 0.056-2.ss1 
(I toyed with the idea of making a 0.063-ssi version, but udev 0.063 
needs kernel >= 2.6.12).

(I'm doing all this 'cos my new Dell Optiplex 280's need the X.org xserver).

I decided to go give git a try.=20

http://debian.openssi.org/ci-linux.git/

BTW git clone won't work with http:// so you may need to use cogito=20


cg clone http://debian.openssi.org/ci-linux.git/  test-ci-linux.git

That is the complete project history.=20

-aneesh

Hi,

I haven't really been able to reproduce this 100% of the time, but with the=
=20
current development version CFS sometimes reads the old file if you try to=
=20
read immediately after overwriting. I've had a few occasions where ssi-ksyn=
c=20
right after mkinitrd after make install would sync the old initrd left over=
=20
from kernel make install. I can tell because the initrd would fail to boot.

Roger

I can reproduce the syn flood pretty much everytime.

1. Loadbalance DR to more than one realserver on the ICS (Apache port 80=20
this case).
2. Establish connections until a Routed realserver gets hit.
3. Stop all traffic to the balancer.
4. tcpdump port 80 on the ICS and you will see an endless flood of duplicat=
e=20
syn packets.
5. tcpdump port 80 on the external interface and you will see no traffic.
6. These packets correspond to the stale SYN_RECV entries in the IPVS table=
.

The only way to stop the flooding before the cluster freezes up, is stop=20
Apache on port 80. Or kill ha-lvs.

DR loadbalancing works for the client, but you don't expect it to trigger=
=20
continuous syn floods within the cluster. Looks like IPVS fails to=20
transition some entries from SYN_RECV to FIN, etc.

Roger


On 9/1/05, Roger Tsang <roger.tsang@...> wrote:
>=20
> Hi Aneesh,
>=20
> In OpenSSI-FC (OpenSSI-1.9.1) the weights assigned to realservers get=20
> reset to 1 if one were to restart ha-lvs service. This is a problem. The=
=20
> expected behavior is all entries in IPVS table keep the same weights as=
=20
> before the restart - ie. same as in cvip.conf.
>=20
> We have to restart all services before ha-lvs will re-assign them proper=
=20
> realserver weights. It's kinda annoying.
>=20
> Also sometimes the director will flood the realserver with syn packets=20
> even when there is no real external traffic... I'm looking into that.
>=20
>=20
> Roger
>