- priority: 5 --> 8
- assigned_to: bhojas --> krishmnc
(A) Setup:
- Configured 3 JBODs on SN5428-2
- mounted /dev/sdb and /dev/sdc on /mnt/sdb
and /mnt/sdc respectively and ran I/O using mkfile/verfile
- Executed raw I/O on the 3rd disk:
dd if=/dev/zero of=/dev/sdd bs=512
- Executed all this I/O for overnight
- Also executed iscsi-ls in a loop with 3 seconds sleep:
while true; do iscsi-ls -l; echo ""; echo ""; echo ""; sleep
3; done
- In addition to this, enabled the discovery address
in /etc/iscsi.conf for the configuration on MDS and
executed a "iscsi reload"
- On MDS 2 JBODs and one MSA Array LUN were
configured
- iscsi-ls -l displayed these devices
- I/Os in the devices configured on SN5428-2 continued
- Morning found all the I/Os were running fine
- Stopped the I/Os and also executed sync for any
pending I/Os to finish
- Stopped the driver
- Checked with iscsi-ls -l, the driver was not running
- Restarted the driver, could not see the devices from
another portal on MDS
- So restarted the driver and the system crashed
- Important thing to notice is that raw IO was done
on /dev/sdd and /etc/fstab.iscsi (for the sake of
automounting) & /etc/fstab (for the sake of running I/Os
using fcreate script) contained entries:
/dev/sdb /mnt/sdb ext2 defaults
0 0
/dev/sdc /mnt/sdc ext2 defaults
0 0
/dev/sdd /mnt/sdd ext2 defaults
0 0
(B) Output of iscsi-ls -l which was run in a loop:
******************************************
*************************************
SFNet iSCSI Driver Version ...4:0.1.11(12-Jan-2005)
******************************************
*************************************
TARGET NAME : iqn.1987-
05.com.cisco:00.1bb5ca1be023.cnitin-t1
TARGET ALIAS : cnitin-t1
HOST NO : 10
TARGET ADDRESS : 10.1.1.67
SESSION STATUS : CONNECTION ON 10.1.1.67
ESTABLISHED AT Thu Feb 3 17:05:04
IST 2005
SESSION ID : ISID 00023d000001
TSIH 3
DEVICE DETAILS:
---------------
LUN NO: 0
Vendor: SEAGATE
Model: ST318452FC
Rev: 0004
-------------------------------------------------------
------------------------
TARGET NAME : iqn.2004-12.com.cisco.cnitin-
jbod2-mds2-fc1-567c
TARGET ALIAS : <NULL>
HOST NO : 11
TARGET ADDRESS : 10.1.2.61
SESSION STATUS : CONNECTION ON 10.1.2.61
ESTABLISHED AT Thu Feb 3 18:29:05
IST 2005
SESSION ID : ISID 00023d000001
TSIH 86
DEVICE DETAILS:
---------------
LUN NO: 0
Vendor: SEAGATE
Model: ST318452FC
Rev: 0004
-------------------------------------------------------
------------------------
TARGET NAME : iqn.2004-12.com.cisco.cnitin-
jbod2-mds2-fc1-5594
TARGET ALIAS : <NULL>
HOST NO : 12
TARGET ADDRESS : 10.1.2.61
SESSION STATUS : CONNECTION ON 10.1.2.61
ESTABLISHED AT Thu Feb 3 18:29:07
IST 2005
SESSION ID : ISID 00023d000001
TSIH 86
DEVICE DETAILS:
---------------
LUN NO: 0
Vendor: SEAGATE
Model: ST318452FC
Rev: 0004
-------------------------------------------------------
------------------------
cat: /sys/class/iscsi_transport_class/target13:0:0/target
_name: No such file or directory
TARGET NAME :
cat: /sys/class/iscsi_transport_class/target13:0:0/target
_alias: No such file or directory
TARGET ALIAS :
HOST NO : 13
cat: /sys/class/iscsi_transport_class/target13:0:0/ip_add
ress: No such file or directory
TARGET ADDRESS :
cat: /sys/class/iscsi_transport_class/target13:0:0/ip_add
ress: No such file or directory
SESSION STATUS : CONNECTION ON
ESTABLISHED AT Thu Feb 3 18:29:08
IST 2005
cat: /sys/class/iscsi_transport_class/target13:0:0/isid:
No such file or directory
SESSION ID : ISID
cat: /sys/class/iscsi_transport_class/target13:0:0/tsih:
No such file or directory
TSIH
DEVICE DETAILS:
---------------
LUN NO: 0*
cat: /sys/class/iscsi_transport_class/target13:0:0/device
/13:0:0*/vendor: No such file or directory
Vendor:
cat: /sys/class/iscsi_transport_class/target13:0:0/device
/13:0:0*/model: No such file or directory
Model:
cat: /sys/class/iscsi_transport_class/target13:0:0/device
/13:0:0*/rev: No such file or directory
Rev:
-------------------------------------------------------
------------------------
TARGET NAME : iqn.1987-
05.com.cisco:00.7d30f143aacd.cnitin-t2
TARGET ALIAS : cnitin-t2
HOST NO : 8
TARGET ADDRESS : 10.1.1.67
SESSION STATUS : CONNECTION ON 10.1.1.67
ESTABLISHED AT Thu Feb 3 16:50:10
IST 2005
SESSION ID : ISID 00023d000001
TSIH 4
DEVICE DETAILS:
---------------
LUN NO: 0
Vendor: SEAGATE
Model: ST318452FC
Rev: 0004
-------------------------------------------------------
------------------------
TARGET NAME : iqn.1987-
05.com.cisco:00.0ea42193ba50.cnitin-t3
TARGET ALIAS : cnitin-t3
HOST NO : 9
TARGET ADDRESS : 10.1.1.67
SESSION STATUS : CONNECTION ON 10.1.1.67
ESTABLISHED AT Thu Feb 3 16:50:11
IST 2005
SESSION ID : ISID 00023d000001
TSIH 2
DEVICE DETAILS:
---------------
LUN NO: 0
Vendor: SEAGATE
Model: ST318452FC
Rev: 0004
-------------------------------------------------------
------------------------
(C) Output of dmesg 5 minutes before the crash
happened:
[root@localhost tools]#
[root@localhost tools]# dmesg
with errors, running e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs error (device sdb): ext2_free_blocks: Freeing
blocks not in datazone - block = 6285312, count = 1024
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
printk: 65149 messages suppressed.
swapper: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154e01>] __kmalloc+0xb1/0xe0
[<c0454e98>] alloc_skb+0x48/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0101100>] cpu_idle+0x40/0x50
[<c06568ac>] start_kernel+0x16c/0x1b0
[<c010019f>] 0xc010019f
swapper: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154cd2>] kmem_cache_alloc+0x82/0xa0
[<c0454e73>] alloc_skb+0x23/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0101100>] cpu_idle+0x40/0x50
[<c06568ac>] start_kernel+0x16c/0x1b0
[<c010019f>] 0xc010019f
swapper: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154cd2>] kmem_cache_alloc+0x82/0xa0
[<c0454e73>] alloc_skb+0x23/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0101100>] cpu_idle+0x40/0x50
[<c06568ac>] start_kernel+0x16c/0x1b0
[<c010019f>] 0xc010019f
swapper: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154cd2>] kmem_cache_alloc+0x82/0xa0
[<c0454e73>] alloc_skb+0x23/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0101100>] cpu_idle+0x40/0x50
[<c06568ac>] start_kernel+0x16c/0x1b0
[<c010019f>] 0xc010019f
swapper: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154cd2>] kmem_cache_alloc+0x82/0xa0
[<c0454e73>] alloc_skb+0x23/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0101100>] cpu_idle+0x40/0x50
[<c06568ac>] start_kernel+0x16c/0x1b0
[<c010019f>] 0xc010019f
swapper: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154cd2>] kmem_cache_alloc+0x82/0xa0
[<c0454e73>] alloc_skb+0x23/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0101100>] cpu_idle+0x40/0x50
[<c06568ac>] start_kernel+0x16c/0x1b0
[<c010019f>] 0xc010019f
swapper: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154cd2>] kmem_cache_alloc+0x82/0xa0
[<c0454e73>] alloc_skb+0x23/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0101100>] cpu_idle+0x40/0x50
[<c06568ac>] start_kernel+0x16c/0x1b0
[<c010019f>] 0xc010019f
swapper: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154cd2>] kmem_cache_alloc+0x82/0xa0
[<c0454e73>] alloc_skb+0x23/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0101100>] cpu_idle+0x40/0x50
[<c06568ac>] start_kernel+0x16c/0x1b0
[<c010019f>] 0xc010019f
swapper: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154cd2>] kmem_cache_alloc+0x82/0xa0
[<c0454e73>] alloc_skb+0x23/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0101100>] cpu_idle+0x40/0x50
[<c06568ac>] start_kernel+0x16c/0x1b0
[<c010019f>] 0xc010019f
swapper: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154cd2>] kmem_cache_alloc+0x82/0xa0
[<c0454e73>] alloc_skb+0x23/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0101100>] cpu_idle+0x40/0x50
[<c06568ac>] start_kernel+0x16c/0x1b0
[<c010019f>] 0xc010019f
printk: 110 messages suppressed.
kswapd0: page allocation failure. order:0, mode:0x20
[<c010414e>] dump_stack+0x1e/0x30
[<c014d63c>] __alloc_pages+0x1cc/0x3a0
[<c014d837>] __get_free_pages+0x27/0x40
[<c0152262>] kmem_getpages+0x22/0xc0
[<c0153ce7>] cache_grow+0x137/0x360
[<c01544fe>] cache_alloc_refill+0x15e/0x360
[<c0154e01>] __kmalloc+0xb1/0xe0
[<c0454e98>] alloc_skb+0x48/0xf0
[<c0303fa2>] tg3_rx+0x332/0x400
[<c0304211>] tg3_poll+0x1a1/0x450
[<c045cfea>] net_rx_action+0x6a/0xf0
[<c0124f55>] __do_softirq+0x95/0xa0
[<c0124f8b>] do_softirq+0x2b/0x30
[<c01058ec>] do_IRQ+0x3c/0x70
[<c0103c12>] common_interrupt+0x1a/0x20
[<c0158979>] shrink_cache+0x219/0x720
[<c0159a4c>] shrink_zone+0xbc/0xf0
[<c0159f26>] balance_pgdat+0x286/0x330
[<c015a0b3>] kswapd+0xe3/0xf0
[<c01012f5>] kernel_thread_helper+0x5/0x10
EXT2-fs warning: mounting fs with errors, running e2fsck
is recommended
EXT2-fs warning: maximal mount count reached, running
e2fsck is recommended
(D) When the driver was stopped, at the end of dmesg
following messages were displayed - in addition to above
message:
iscsi-sfnet:host12: lun0: Session terminating, failing to
queue cdb 0x35 and any following commands
iscsi-sfnet:host12: lun0: Session terminating, failing to
queue cdb 0x35 and any following commands
iscsi-sfnet:host12: lun0: Session terminating, failing to
queue cdb 0x35 and any following commands
FAILED
status = 0, message = 00, host = 1, driver = 08
Current sd: sense = 70 2
ASC= 8 ASCQ= 0
Raw sense data:0x70 0x00 0x02 0x00 0x00 0x00 0x00
0x06 0x00 0x00 0x00 0x00 0x08 0x00
iscsi-sfnet:host13: Session dropped
[root@localhost tools]#
(E) Output of iscsi-ls after the driver stopped:
[root@localhost tools]# iscsi-ls
######################################
######################################
###
iSCSI driver is not loaded
######################################
######################################
###
(F) Started the driver, but could not see the devices
getting mounted /dev/sdb, /dev/sdc, /dev/sdd
[root@localhost tools]# /etc/init.d/iscsi start
Starting iSCSI: iscsi iscsid fsck/mount
[root@localhost tools]# fsck.ext2: Bad magic number in
super-block while trying to open /dev/sdb
*** device not yet available, 9 retries remaining
fsck.ext2: Bad magic number in super-block while trying
to open /dev/sdb
*** device not yet available, 8 retries remaining
fsck.ext2: Bad magic number in super-block while trying
to open /dev/sdb
*** device not yet available, 7 retries remaining
fsck.ext2: Bad magic number in super-block while trying
to open /dev/sdb
*** device not yet available, 6 retries remaining
fsck.ext2: Bad magic number in super-block while trying
to open /dev/sdb
*** device not yet available, 5 retries remaining
fsck.ext2: Bad magic number in super-block while trying
to open /dev/sdb
*** device not yet available, 4 retries remaining
fsck.ext2: Bad magic number in super-block while trying
to open /dev/sdb
*** device not yet available, 3 retries remaining
fsck.ext2: Bad magic number in super-block while trying
to open /dev/sdb
*** device not yet available, 2 retries remaining
fsck.ext2: Bad magic number in super-block while trying
to open /dev/sdb
*** device not yet available, 1 retries remaining
fsck.ext2: Bad magic number in super-block while trying
to open /dev/sdb
*** device not yet available, 0 retries remaining
[root@localhost tools]#
[root@localhost tools]# mount
/dev/sda9 on / type ext3 (rw)
none on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
/dev/sda10 on /usr/local type ext3 (rw)
none on /sys type sysfs (rw)
(G) iscsi-ls -l also showed the devices configured on
SN5428-2 and not the ones on MDS.
But the discovery address on MDS was ping-able.
[root@localhost tools]# ping dingdong =====>
DiscoveryAddress on MDS
PING dingdong (10.1.2.61) 56(84) bytes of data.
64 bytes from dingdong (10.1.2.61): icmp_seq=1 ttl=255
time=0.175 ms
64 bytes from dingdong (10.1.2.61): icmp_seq=2 ttl=255
time=0.202 ms
..
[root@localhost tools]# iscsi-ls
******************************************
*************************************
SFNet iSCSI Driver Version ...4:0.1.11(12-Jan-2005)
******************************************
*************************************
TARGET NAME : iqn.1987-
05.com.cisco:00.1bb5ca1be023.cnitin-t1
TARGET ALIAS : cnitin-t1
HOST NO : 14
TARGET ADDRESS : 10.1.1.67
SESSION STATUS : CONNECTION ON 10.1.1.67
ESTABLISHED AT Fri Feb 4 10:27:32
IST 2005
SESSION ID : ISID 00023d000001
TSIH 2
-------------------------------------------------------
------------------------
TARGET NAME : iqn.1987-
05.com.cisco:00.7d30f143aacd.cnitin-t2
TARGET ALIAS : cnitin-t2
HOST NO : 15
TARGET ADDRESS : 10.1.1.67
SESSION STATUS : CONNECTION ON 10.1.1.67
ESTABLISHED AT Fri Feb 4 10:27:32
IST 2005
SESSION ID : ISID 00023d000001
TSIH 3
-------------------------------------------------------
------------------------
TARGET NAME : iqn.1987-
05.com.cisco:00.0ea42193ba50.cnitin-t3
TARGET ALIAS : cnitin-t3
HOST NO : 16
TARGET ADDRESS : 10.1.1.67
SESSION STATUS : CONNECTION ON 10.1.1.67
ESTABLISHED AT Fri Feb 4 10:27:32
IST 2005
SESSION ID : ISID 00023d000001
TSIH 4
-------------------------------------------------------
------------------------
[root@localhost tools]#
(H) Restarted the driver and the system crashed:
[root@localhost tools]# /etc/init.d/iscsi restart
Stopping iSCSI: sync umount sync iscsid
/dev/sdd: UNEXPECTED INCONSISTENCY; RUN fsck
MANUALLY.
(i.e., without -a or -p options)
(I) The system was no more accessible. Noted down the
trace shown on the console:
[<c0121bdb>] do_exit+0xab/0x8a0
[<c0104466e>] die+0x28e/0x290
[<c0116922>] do_page_fault+0x3b2/0x6f2
[<c0103c4b>] error_code+0x2b/0x30
[<c0121bdb>] do_exit+0xab/0x8a0
[<c0104466e>] die+0x28e/0x290
[<c0116922>] do_page_fault+0x3b2/0x6f2
[<c0103c4b>] error_code+0x2b/0x30
[<c0121bdb>] do_exit+0xab/0x8a0
(J) Attaching the /var/log/messages file - picked up
after the system came up after the reboot - renamed as
messages.crash. Crash happened at around "Feb 4
10:30:16"
(K) At this system, when the system is up and running, I
can see all the devices from SN5428-2 as well as MDS.
Output of iscsi-ls -l:
[root@localhost log]# iscsi-ls -l
******************************************
*************************************
SFNet iSCSI Driver Version ...4:0.1.11(12-Jan-2005)
******************************************
*************************************
TARGET NAME : iqn.2004-12.com.cisco.cnitin-
jbod2-mds2-fc1-567c
TARGET ALIAS : <NULL>
HOST NO : 4
TARGET ADDRESS : 10.1.2.61
SESSION STATUS : CONNECTION ON 10.1.2.61
ESTABLISHED AT Fri Feb 4 11:22:53
IST 2005
SESSION ID : ISID 00023d000001
TSIH 86
DEVICE DETAILS:
---------------
LUN NO: 0
Vendor: SEAGATE
Model: ST318452FC
Rev: 0004
-------------------------------------------------------
------------------------
TARGET NAME : iqn.2004-12.com.cisco.cnitin-
jbod2-mds2-fc1-5594
TARGET ALIAS : <NULL>
HOST NO : 5
TARGET ADDRESS : 10.1.2.61
SESSION STATUS : CONNECTION ON 10.1.2.61
ESTABLISHED AT Fri Feb 4 11:22:53
IST 2005
SESSION ID : ISID 00023d000001
TSIH 86
DEVICE DETAILS:
---------------
LUN NO: 0
Vendor: SEAGATE
Model: ST318452FC
Rev: 0004
-------------------------------------------------------
------------------------
cat: /sys/class/iscsi_transport_class/target6:0:0/target_
name: No such file or directory
TARGET NAME :
cat: /sys/class/iscsi_transport_class/target6:0:0/target_
alias: No such file or directory
TARGET ALIAS :
HOST NO : 6
cat: /sys/class/iscsi_transport_class/target6:0:0/ip_addr
ess: No such file or directory
TARGET ADDRESS :
cat: /sys/class/iscsi_transport_class/target6:0:0/ip_addr
ess: No such file or directory
SESSION STATUS : CONNECTION ON
ESTABLISHED AT Fri Feb 4 11:22:53
IST 2005
cat: /sys/class/iscsi_transport_class/target6:0:0/isid:
No such file or directory
SESSION ID : ISID
cat: /sys/class/iscsi_transport_class/target6:0:0/tsih:
No such file or directory
TSIH
DEVICE DETAILS:
---------------
LUN NO: 0*
cat: /sys/class/iscsi_transport_class/target6:0:0/device/
6:0:0*/vendor: No such file or directory
Vendor:
cat: /sys/class/iscsi_transport_class/target6:0:0/device/
6:0:0*/model: No such file or directory
Model:
cat: /sys/class/iscsi_transport_class/target6:0:0/device/
6:0:0*/rev: No such file or directory
Rev:
-------------------------------------------------------
------------------------
TARGET NAME : iqn.1987-
05.com.cisco:00.1bb5ca1be023.cnitin-t1
TARGET ALIAS : cnitin-t1
HOST NO : 7
TARGET ADDRESS : 10.1.1.67
SESSION STATUS : CONNECTION ON 10.1.1.67
ESTABLISHED AT Fri Feb 4 11:25:28
IST 2005
SESSION ID : ISID 00023d000001
TSIH 2
DEVICE DETAILS:
---------------
LUN NO: 0
Vendor: SEAGATE
Model: ST318452FC
Rev: 0004
-------------------------------------------------------
------------------------
TARGET NAME : iqn.1987-
05.com.cisco:00.7d30f143aacd.cnitin-t2
TARGET ALIAS : cnitin-t2
HOST NO : 8
TARGET ADDRESS : 10.1.1.67
SESSION STATUS : CONNECTION ON 10.1.1.67
ESTABLISHED AT Fri Feb 4 11:25:28
IST 2005
SESSION ID : ISID 00023d000001
TSIH 4
DEVICE DETAILS:
---------------
LUN NO: 0
Vendor: SEAGATE
Model: ST318452FC
Rev: 0004
-------------------------------------------------------
------------------------
TARGET NAME : iqn.1987-
05.com.cisco:00.0ea42193ba50.cnitin-t3
TARGET ALIAS : cnitin-t3
HOST NO : 9
TARGET ADDRESS : 10.1.1.67
SESSION STATUS : CONNECTION ON 10.1.1.67
ESTABLISHED AT Fri Feb 4 11:25:28
IST 2005
SESSION ID : ISID 00023d000001
TSIH 3
DEVICE DETAILS:
---------------
LUN NO: 0
Vendor: SEAGATE
Model: ST318452FC
Rev: 0004
-------------------------------------------------------
------------------------
(K) Also mount command shows:
[root@localhost log]# mount
/dev/sda9 on / type ext3 (rw)
none on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
/dev/sda10 on /usr/local type ext3 (rw)
none on /sys type sysfs (rw)
/dev/sdb on /mnt/sdb type ext2 (rw)
/dev/sdc on /mnt/sdc type ext2 (rw)
==> Since earlier raw I/O was done on /dev/sdd, the
disk may not be in a shape where it can be mounted.
The filesystem might have to be created before
mounting it.