Re: [Scst-devel] ZFS on iscsi: GET_LBA_STATUS fails with INVALID_FIELD_IN_CDB(0x2400) / KVM/QEMU drive-mirror fails

[Vladislav B. <vs...@vl...>]

Hi,

SCST does not implement GET LBA STATUS, because there is no known way to get this info
from the block layer. Since GET LBA STATUS command is subcommand of SERVICE ACTION IN
(16) multiplexing command, INVALID_FIELD_IN_CDB pointing to the SERVICE ACTION field in
the CDB supposed to specify that GET LBA STATUS subcommand is not supported.

Looks like KVM/QEMU does not obey this SCSI provision and confused by receiving
INVALID_FIELD_IN_CDB response.

Vlad

Alexander Schmid wrote on 07/25/2016 07:59 AM:
> Hello,
> i`m running into some trouble with QEMU/KVM, especially when using 
> drive-mirror from an iscsi-volume, for example by writing directly to 
> the socket:
> 
> socat - UNIX-CLIENT:/var/run/qemu-server/108.qmp
> 
> {"QMP": {"version": {"qemu": {"micro": 1, "minor": 5, "major": 2}, 
> "package": "pve-qemu-kvm_2.5-19"}, "capabilities": []}}
> { "execute": "qmp_capabilities" }
> {"return": {}}
> { "execute": "drive-mirror", "arguments": { "device": "drive-virtio1", 
> "target": "/tmp/myImage", "sync": "full", "format": "raw", 
> "on-source-error": "report"} }
> {"return": {}}
> {"timestamp": {"seconds": 1469457158, "microseconds": 105888}, "event": 
> "BLOCK_JOB_ERROR", "data": {"device": "drive-virtio1", "operation": 
> "read", "action": "report"}}
> {"timestamp": {"seconds": 1469457158, "microseconds": 129351}, "event": 
> "BLOCK_JOB_COMPLETED", "data": {"device": "drive-virtio1", "len": 
> 2147483648, "offset": 0, "speed": 0, "type": "mirror", "error": "Invalid 
> argument"}}
> 
> Unfortunately, QEMU is not very detailed about the error that ocurred.
> 
> The backstore of the volume in question is a ZFS Volume.
> If i do the same with a Volume that is not backed by ZFS drive-mirror 
> works like expected.
> 
> { "execute": "drive-mirror", "arguments": { "device": "drive-virtio0", 
> "target": "/tmp/myImage", "sync": "full", "format": "raw", 
> "on-source-error": "report"} }
> {"return": {}}
> {"timestamp": {"seconds": 1469457364, "microseconds": 493040}, "event": 
> "BLOCK_JOB_READY", "data": {"device": "drive-virtio0", "len": 
> 8589934592, "offset": 8589934592, "speed": 0, "type": "mirror"}}
> 
> Both volumes come from the same SCST-Server, on different targets, 
> devices are handled by vdisk_blockio.
> The second volume currently is backed by a RAID1.
> 
> Kernel is: 4.4.13
> 
> Also, when i move the volume when the VM is offline from the ZFS-Backed 
> storage to another storage (i.e. localstorage or even the iscsi-target 
> backed by RAID1) i get the following errors a few times at startup and 
> sometimes also during and/or after the transfer:
> 
> transferred: 0 bytes remaining: 2147483648 bytes total: 2147483648 bytes 
> progression: 0.00 %
> qemu-img: iSCSI Failure: SENSE KEY:ILLEGAL_REQUEST(5) 
> ASCQ:INVALID_FIELD_IN_CDB(0x2400) qemu-img: iSCSI Failure: SENSE 
> KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400) qemu-img: iSCSI 
> Failure: SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)[ 
> ... ] transferred: 22978075 bytes remaining: 2124505573 bytes total: 
> 2147483648 bytes progression: 1.07 %
> 
> [....]
> 
> transferred: 2147483648 bytes remaining: 0 bytes total: 2147483648 bytes 
> progression: 100.00 % qemu-img: iSCSI Failure: SENSE 
> KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400) transferred: 
> 2147483648 bytes remaining: 0 bytes total: 2147483648 bytes progression: 
> 100.00 %
> 
> In the end, the volume is moved away and also works on it`s new 
> Location. On other Targets, not backed by ZFS, these errors / failures 
> don't show up.
> 
> So i tried to figure-out what might be wrong and used iscsi-test-cu from 
> libiscsi-bin. This is part of the result, what shows the 
> INVALID_FIELD_IN_CDB(0x2400) i also see when moving a volume away from 
> the target:
> 
>       CUnit - A unit testing framework for C - Version 2.1-2      
> http://cunit.sourceforge.net/ Suite: CompareAndWrite   Test: Simple 
> ...passed   Test: Miscompare ...passed Suite: GetLBAStatus   Test: 
> Simple ...    [FAILED] GET_LBA_STATUS command: failed with sense. SENSE 
> KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)     [FAILED] 
> GET_LBA_STATUS command: failed with sense. SENSE KEY:ILLEGAL_REQUEST(5) 
> ASCQ:INVALID_FIELD_IN_CDB(0x2400)     [FAILED] GET_LBA_STATUS command: 
> failed with sense. SENSE KEY:ILLEGAL_REQUEST(5) 
> ASCQ:INVALID_FIELD_IN_CDB(0x2400)     [FAILED] GET_LBA_STATUS command: 
> failed with sense. SENSE KEY:ILLEGAL_REQUEST(5) 
> ASCQ:INVALID_FIELD_IN_CDB(0x2400)
> 
> [....]
> 
>    Test: UnmapSingle ...    [FAILED] GET_LBA_STATUS command: LBA offset 
> in first descriptor does not match request (0xa000000002400 != 0x0). 
> FAILED     1. test_get_lba_status_unmap_single.c:94  - CU_FAIL("[FAILED] 
> GET_LBA_STATUS command: " "LBA offset in first descriptor does not match 
> " "the LBA in the CDB.")
> 
> I wonder if this is an issue with SCST and ZFS or if the root cause is 
> located in KVM / QEMU and how to deal with this and make drive-mirror 
> from qpm work. Any help / hint would be appreciated. If it helps i can 
> supply more information from configs, logfiles etc. Thanks, Alexander 
> Schmid
> 
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity planning
> reports.http://sdm.link/zohodev2dev
> _______________________________________________
> Scst-devel mailing list
> https://lists.sourceforge.net/lists/listinfo/scst-devel
>