Re: [Scst-devel] ZFS on iscsi: GET_LBA_STATUS fails with INVALID_FIELD_IN_CDB(0x2400) / KVM/QEMU dr

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi Vlad,

Am 26.07.16 um 04:42 schrieb Vladislav Bolkhovitin:
> Hi,
>
> SCST does not implement GET LBA STATUS, because there is no known way to get this info
> from the block layer.
Ok, this makes sense.
> Since GET LBA STATUS command is subcommand of SERVICE ACTION IN
> (16) multiplexing command, INVALID_FIELD_IN_CDB pointing to the SERVICE ACTION field in
> the CDB supposed to specify that GET LBA STATUS subcommand is not supported.
>
> Looks like KVM/QEMU does not obey this SCSI provision and confused by receiving
> INVALID_FIELD_IN_CDB response.

I came across this issue while adding support for SCST to the Proxmox 
ZFS over iSCSI options:
https://pve.proxmox.com/wiki/Storage:_ZFS_over_iSCSI

What is done so far works fine, besides the issues with the 
INVALID_FIELD_IN_CDB. For my observations this don't hurt when moving a 
volume away from the target (offline) but breaks mirroring (online) 
completely.

I wonder why a volume on a RAID1 target, served by the same instance of 
SCST, does not emit INVALID_FIELD_IN_CDB to QEMU in the mirroring 
process and other situations.
This also doesn't happen on a quite fast all-flash Target on the same 
SCST (8x SSD in RAID 10).

So there must be something different going on with ZFS backed 
targets/LUNs either in QEMU or SCST what causes this confusion.

Any idea how to solve this and make drive-mirror, and therefore 
online-migration of a volume possible, would help.

Thanks,
Alex
>
> Vlad
>
> Alexander Schmid wrote on 07/25/2016 07:59 AM:
>> Hello,
>> i`m running into some trouble with QEMU/KVM, especially when using
>> drive-mirror from an iscsi-volume, for example by writing directly to
>> the socket:
>>
>> socat - UNIX-CLIENT:/var/run/qemu-server/108.qmp
>>
>> {"QMP": {"version": {"qemu": {"micro": 1, "minor": 5, "major": 2},
>> "package": "pve-qemu-kvm_2.5-19"}, "capabilities": []}}
>> { "execute": "qmp_capabilities" }
>> {"return": {}}
>> { "execute": "drive-mirror", "arguments": { "device": "drive-virtio1",
>> "target": "/tmp/myImage", "sync": "full", "format": "raw",
>> "on-source-error": "report"} }
>> {"return": {}}
>> {"timestamp": {"seconds": 1469457158, "microseconds": 105888}, "event":
>> "BLOCK_JOB_ERROR", "data": {"device": "drive-virtio1", "operation":
>> "read", "action": "report"}}
>> {"timestamp": {"seconds": 1469457158, "microseconds": 129351}, "event":
>> "BLOCK_JOB_COMPLETED", "data": {"device": "drive-virtio1", "len":
>> 2147483648, "offset": 0, "speed": 0, "type": "mirror", "error": "Invalid
>> argument"}}
>>
>> Unfortunately, QEMU is not very detailed about the error that ocurred.
>>
>> The backstore of the volume in question is a ZFS Volume.
>> If i do the same with a Volume that is not backed by ZFS drive-mirror
>> works like expected.
>>
>> { "execute": "drive-mirror", "arguments": { "device": "drive-virtio0",
>> "target": "/tmp/myImage", "sync": "full", "format": "raw",
>> "on-source-error": "report"} }
>> {"return": {}}
>> {"timestamp": {"seconds": 1469457364, "microseconds": 493040}, "event":
>> "BLOCK_JOB_READY", "data": {"device": "drive-virtio0", "len":
>> 8589934592, "offset": 8589934592, "speed": 0, "type": "mirror"}}
>>
>> Both volumes come from the same SCST-Server, on different targets,
>> devices are handled by vdisk_blockio.
>> The second volume currently is backed by a RAID1.
>>
>> Kernel is: 4.4.13
>>
>> Also, when i move the volume when the VM is offline from the ZFS-Backed
>> storage to another storage (i.e. localstorage or even the iscsi-target
>> backed by RAID1) i get the following errors a few times at startup and
>> sometimes also during and/or after the transfer:
>>
>> transferred: 0 bytes remaining: 2147483648 bytes total: 2147483648 bytes
>> progression: 0.00 %
>> qemu-img: iSCSI Failure: SENSE KEY:ILLEGAL_REQUEST(5)
>> ASCQ:INVALID_FIELD_IN_CDB(0x2400) qemu-img: iSCSI Failure: SENSE
>> KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400) qemu-img: iSCSI
>> Failure: SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)[
>> ... ] transferred: 22978075 bytes remaining: 2124505573 bytes total:
>> 2147483648 bytes progression: 1.07 %
>>
>> [....]
>>
>> transferred: 2147483648 bytes remaining: 0 bytes total: 2147483648 bytes
>> progression: 100.00 % qemu-img: iSCSI Failure: SENSE
>> KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400) transferred:
>> 2147483648 bytes remaining: 0 bytes total: 2147483648 bytes progression:
>> 100.00 %
>>
>> In the end, the volume is moved away and also works on it`s new
>> Location. On other Targets, not backed by ZFS, these errors / failures
>> don't show up.
>>
>> So i tried to figure-out what might be wrong and used iscsi-test-cu from
>> libiscsi-bin. This is part of the result, what shows the
>> INVALID_FIELD_IN_CDB(0x2400) i also see when moving a volume away from
>> the target:
>>
>>        CUnit - A unit testing framework for C - Version 2.1-2
>> http://cunit.sourceforge.net/ Suite: CompareAndWrite   Test: Simple
>> ...passed   Test: Miscompare ...passed Suite: GetLBAStatus   Test:
>> Simple ...    [FAILED] GET_LBA_STATUS command: failed with sense. SENSE
>> KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)     [FAILED]
>> GET_LBA_STATUS command: failed with sense. SENSE KEY:ILLEGAL_REQUEST(5)
>> ASCQ:INVALID_FIELD_IN_CDB(0x2400)     [FAILED] GET_LBA_STATUS command:
>> failed with sense. SENSE KEY:ILLEGAL_REQUEST(5)
>> ASCQ:INVALID_FIELD_IN_CDB(0x2400)     [FAILED] GET_LBA_STATUS command:
>> failed with sense. SENSE KEY:ILLEGAL_REQUEST(5)
>> ASCQ:INVALID_FIELD_IN_CDB(0x2400)
>>
>> [....]
>>
>>     Test: UnmapSingle ...    [FAILED] GET_LBA_STATUS command: LBA offset
>> in first descriptor does not match request (0xa000000002400 != 0x0).
>> FAILED     1. test_get_lba_status_unmap_single.c:94  - CU_FAIL("[FAILED]
>> GET_LBA_STATUS command: " "LBA offset in first descriptor does not match
>> " "the LBA in the CDB.")
>>
>> I wonder if this is an issue with SCST and ZFS or if the root cause is
>> located in KVM / QEMU and how to deal with this and make drive-mirror
>> from qpm work. Any help / hint would be appreciated. If it helps i can
>> supply more information from configs, logfiles etc. Thanks, Alexander
>> Schmid
>>
>> ------------------------------------------------------------------------------
>> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
>> patterns at an interface-level. Reveals which users, apps, and protocols are
>> consuming the most bandwidth. Provides multi-vendor support for NetFlow,
>> J-Flow, sFlow and other flows. Make informed decisions using capacity planning
>> reports.http://sdm.link/zohodev2dev
>> _______________________________________________
>> Scst-devel mailing list
>> https://lists.sourceforge.net/lists/listinfo/scst-devel
>>

Re: [Scst-devel] ZFS on iscsi: GET_LBA_STATUS fails with INVALID_FIELD_IN_CDB(0x2400) / KVM/QEMU dr

Re: [Scst-devel] ZFS on iscsi: GET_LBA_STATUS fails with INVALID_FIELD_IN_CDB(0x2400) / KVM/QEMU drive-mirror fails