[Scst-devel] SRP and Persistent Reservations

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hello,

I am seeing inconsistent results using persistent reservations with SRP, it
seems to take a number of attempts to register, reserve, release and
unregister from the LUN and writing to the
LUN from the initiator that is the reservation holder does not always work.

I have two initiators connecting to one target and would like to make use
of SCSI-3 persistent reservations to make sure only one host can write to
the LUN at a time.

Hopefully this is just a configuration issue with SRP, when I tried this
with iSCSI it works without a problem.

I'm currently running:

ib_srp-backport - v2.0.35
scst - trunk 6553 (debug build)
OS: Ubuntu 14.04 - kernel 3.18.22

with the following ib_srp module parameters:
options ib_srp fast_io_fail_tmo=5 dev_loss_tmo=15 reconnect_delay=1
cmd_sg_entries=255 ch_count=8

The first problem I am seeing is trying to register the initiators, it
usually take a couple of attempts to register:

sg_persist -n --out --register --param-sark=c0a80014 /dev/sdf

persistent reserve out:  Fixed format, current;  Sense key: Unit Attention
 Additional sense: Power on, reset, or bus device reset occurred
PR out (Register): Unit attention

After a couple of attempts I can register the initiators on the target:

  PR generation=0x2
    Key=0xc0a80014
      All target ports bit clear
      Relative port address: 0x33
      not reservation holder
      Transport Id of initiator:
        RDMA initiator port identifier:
        91 ad 35 00 03 c9 02 00  00 02 c9 03 00 f9 75 b1
    Key=0xc0a80015
      All target ports bit clear
      Relative port address: 0x33
      not reservation holder
      Transport Id of initiator:
        RDMA initiator port identifier:
        91 ad 35 00 03 c9 02 00  00 02 c9 03 00 f9 78 81

When I try to reserve the LUN:

sg_persist -n --out --reserve --param-rk=c0a80014 --prout-type=1 /dev/sdf

I start seeing a reservation conflict:

Console output:
PR out (Reserve): Reservation conflict

Initiator log entry:
Oct 28 11:47:56 client1 kernel: sd 1318:0:0:0: reservation conflict

Target log entry:
Oct 28 11:47:54 san1 kernel: [8665]: scst:
scst_set_cmd_error_status:1749:Reservation conflict (dev sata1, initiator
fe80:0000:0000:0000:0002:c903:00f9:75b1, tgt_id 51)

It took four attempts to finally get the reservation:

    sg_persist -ns /dev/sdf

      PR generation=0x2
        Key=0xc0a80014
          All target ports bit clear
          Relative port address: 0x33
          << Reservation holder >>
          scope: LU_SCOPE,  type: Write Exclusive
          Transport Id of initiator:
            RDMA initiator port identifier:
    91 ad 35 00 03 c9 02 00  00 02 c9 03 00 f9 75 b1
        Key=0xc0a80015
          All target ports bit clear
          Relative port address: 0x33
          not reservation holder
          Transport Id of initiator:
            RDMA initiator port identifier:
    91 ad 35 00 03 c9 02 00  00 02 c9 03 00 f9 78 81

When I try writing data to the LUN from the initiators I get unexpected
results:

using dd to write data from host: 0xc0a80015 results in the following log
entry, which is expected:

    Oct 28 11:54:44 client2 kernel: sd 1307:0:0:0: reservation conflict
    Oct 28 11:54:44 client2 kernel: sd 1307:0:0:0: [sdc]
    Oct 28 11:54:44 client2 kernel: Result: hostbyte=DID_OK
driverbyte=DRIVER_OK
    Oct 28 11:54:44 client2 kernel: sd 1307:0:0:0: [sdc] CDB:
    Oct 28 11:54:44 client2 kernel: Write(10): 2a 00 00 00 00 00 00 00 08 00
    Oct 28 11:54:44 client2 kernel: blk_update_request: critical nexus
error, dev sdc, sector 0
    Oct 28 11:54:44 client2 kernel: Buffer I/O error on dev sdc, logical
block 0, lost async page write

using dd to write data from host: 0xc0a80014 sometimes blocks the write
which results in the following log entry, which is unexpected:

    Oct 28 11:57:13 client1 kernel: sd 1318:0:0:0: reservation conflict
    Oct 28 11:57:13 client1 kernel: sd 1318:0:0:0: [sdf]
    Oct 28 11:57:13 client1 kernel: Result: hostbyte=DID_OK
driverbyte=DRIVER_OK
    Oct 28 11:57:13 client1 kernel: sd 1318:0:0:0: [sdf] CDB:
    Oct 28 11:57:13 client1 kernel: Write(10): 2a 00 00 00 00 00 00 00 08 00
    Oct 28 11:57:13 client1 kernel: blk_update_request: critical nexus
error, dev sdf, sector 0
    Oct 28 11:57:13 client1 kernel: Buffer I/O error on dev sdf, logical
block 0, lost async page write

I also get the following entry on the target side:

Oct 28 11:57:11 san1 kernel: [8698]: scst:
scst_set_cmd_error_status:1749:Reservation conflict (dev sata1, initiator
fe80:0000:0000:0000:0002:c903:00f9:75b1, tgt_id 51)

Just to check the persistent reservations as the target sees them:

echo 1 > /sys/kernel/scst_tgt/handlers/vdisk_blockio/sata1/dump_prs

which results in the following log entries on the target:

Oct 28 11:58:09 san1 kernel: [2765]: scst: scst_pr_dump_prs:239:Persistent
reservations for device sata1:
Oct 28 11:58:09 san1 kernel: [2765]: scst: scst_pr_dump_prs:253:  [0]
registrant 91:ad:35:00:03:c9:02:00:00:02:c9:03:00:f9:75:b1/51, key
00000000c0a80014 (reg ffff8800a5088f00, tgt_dev ffff880092ba0000)
Oct 28 11:58:09 san1 kernel: [2765]: scst: scst_pr_dump_prs:253:  [1]
registrant 91:ad:35:00:03:c9:02:00:00:02:c9:03:00:f9:78:81/51, key
00000000c0a80015 (reg ffff8800a5088d80, tgt_dev ffff880092ba12c0)
Oct 28 11:58:09 san1 kernel: [2765]: scst: scst_pr_dump_prs:266:Reservation
holder is 91:ad:35:00:03:c9:02:00:00:02:c9:03:00:f9:75:b1/51 (key
00000000c0a80014, scope 0, type 1, reg ffff8800a5088f00, tgt_dev
ffff880092ba0000)

When trying to release the reservation it took ~35 attempts:

sg_persist -n --out --release --param-rk=c0a80014 --prout-type=1 /dev/sdf

Console output:
PR out (Release): Reservation conflict

Target log entry:
Oct 28 12:33:20 cube1 kernel: [8665]: scst:
scst_set_cmd_error_status:1749:Reservation conflict (dev sata1, initiator
fe80:0000:0000:0000:0002:c903:00f9:75b1, tgt_id 51)

Please let me know if there is any more information I can provide.

Regards
Ed