[Scst-devel] ZFS on iscsi: GET_LBA_STATUS fails with INVALID_FIELD_IN_CDB(0x2400) / KVM/QEMU drive-mirror fails

[Alexander S. <al...@mo...>]

Hello,
i`m running into some trouble with QEMU/KVM, especially when using 
drive-mirror from an iscsi-volume, for example by writing directly to 
the socket:

socat - UNIX-CLIENT:/var/run/qemu-server/108.qmp

{"QMP": {"version": {"qemu": {"micro": 1, "minor": 5, "major": 2}, 
"package": "pve-qemu-kvm_2.5-19"}, "capabilities": []}}
{ "execute": "qmp_capabilities" }
{"return": {}}
{ "execute": "drive-mirror", "arguments": { "device": "drive-virtio1", 
"target": "/tmp/myImage", "sync": "full", "format": "raw", 
"on-source-error": "report"} }
{"return": {}}
{"timestamp": {"seconds": 1469457158, "microseconds": 105888}, "event": 
"BLOCK_JOB_ERROR", "data": {"device": "drive-virtio1", "operation": 
"read", "action": "report"}}
{"timestamp": {"seconds": 1469457158, "microseconds": 129351}, "event": 
"BLOCK_JOB_COMPLETED", "data": {"device": "drive-virtio1", "len": 
2147483648, "offset": 0, "speed": 0, "type": "mirror", "error": "Invalid 
argument"}}

Unfortunately, QEMU is not very detailed about the error that ocurred.

The backstore of the volume in question is a ZFS Volume.
If i do the same with a Volume that is not backed by ZFS drive-mirror 
works like expected.

{ "execute": "drive-mirror", "arguments": { "device": "drive-virtio0", 
"target": "/tmp/myImage", "sync": "full", "format": "raw", 
"on-source-error": "report"} }
{"return": {}}
{"timestamp": {"seconds": 1469457364, "microseconds": 493040}, "event": 
"BLOCK_JOB_READY", "data": {"device": "drive-virtio0", "len": 
8589934592, "offset": 8589934592, "speed": 0, "type": "mirror"}}

Both volumes come from the same SCST-Server, on different targets, 
devices are handled by vdisk_blockio.
The second volume currently is backed by a RAID1.

Kernel is: 4.4.13

Also, when i move the volume when the VM is offline from the ZFS-Backed 
storage to another storage (i.e. localstorage or even the iscsi-target 
backed by RAID1) i get the following errors a few times at startup and 
sometimes also during and/or after the transfer:

transferred: 0 bytes remaining: 2147483648 bytes total: 2147483648 bytes 
progression: 0.00 %
qemu-img: iSCSI Failure: SENSE KEY:ILLEGAL_REQUEST(5) 
ASCQ:INVALID_FIELD_IN_CDB(0x2400) qemu-img: iSCSI Failure: SENSE 
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400) qemu-img: iSCSI 
Failure: SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)[ 
... ] transferred: 22978075 bytes remaining: 2124505573 bytes total: 
2147483648 bytes progression: 1.07 %

[....]

transferred: 2147483648 bytes remaining: 0 bytes total: 2147483648 bytes 
progression: 100.00 % qemu-img: iSCSI Failure: SENSE 
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400) transferred: 
2147483648 bytes remaining: 0 bytes total: 2147483648 bytes progression: 
100.00 %

In the end, the volume is moved away and also works on it`s new 
Location. On other Targets, not backed by ZFS, these errors / failures 
don't show up.

So i tried to figure-out what might be wrong and used iscsi-test-cu from 
libiscsi-bin. This is part of the result, what shows the 
INVALID_FIELD_IN_CDB(0x2400) i also see when moving a volume away from 
the target:

      CUnit - A unit testing framework for C - Version 2.1-2      
http://cunit.sourceforge.net/ Suite: CompareAndWrite   Test: Simple 
...passed   Test: Miscompare ...passed Suite: GetLBAStatus   Test: 
Simple ...    [FAILED] GET_LBA_STATUS command: failed with sense. SENSE 
KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)     [FAILED] 
GET_LBA_STATUS command: failed with sense. SENSE KEY:ILLEGAL_REQUEST(5) 
ASCQ:INVALID_FIELD_IN_CDB(0x2400)     [FAILED] GET_LBA_STATUS command: 
failed with sense. SENSE KEY:ILLEGAL_REQUEST(5) 
ASCQ:INVALID_FIELD_IN_CDB(0x2400)     [FAILED] GET_LBA_STATUS command: 
failed with sense. SENSE KEY:ILLEGAL_REQUEST(5) 
ASCQ:INVALID_FIELD_IN_CDB(0x2400)

[....]

   Test: UnmapSingle ...    [FAILED] GET_LBA_STATUS command: LBA offset 
in first descriptor does not match request (0xa000000002400 != 0x0). 
FAILED     1. test_get_lba_status_unmap_single.c:94  - CU_FAIL("[FAILED] 
GET_LBA_STATUS command: " "LBA offset in first descriptor does not match 
" "the LBA in the CDB.")

I wonder if this is an issue with SCST and ZFS or if the root cause is 
located in KVM / QEMU and how to deal with this and make drive-mirror 
from qpm work. Any help / hint would be appreciated. If it helps i can 
supply more information from configs, logfiles etc. Thanks, Alexander 
Schmid