Re: [Scst-devel] Thin Provisioning and Ceph RBD's

[Alex G. <ag...@is...>]

RBD illustration showing RBD ignoring discard until a certain
threshold - why is that?  This behavior is unfortunately incompatible
with ESXi discard (UNMAP) behavior.

Is there a way to lower the discard sensitivity on RBD devices?



root@e1:/var/log# rbd diff spin1/testdis|awk '{ SUM += $2 } END {
print SUM/1024 " KB" }'
819200 KB

root@e1:/var/log# blkdiscard -o 0 -l 4096 /dev/rbd28
root@e1:/var/log# rbd diff spin1/testdis|awk '{ SUM += $2 } END {
print SUM/1024 " KB" }'
819200 KB

root@e1:/var/log# blkdiscard -o 0 -l 40960 /dev/rbd28
root@e1:/var/log# rbd diff spin1/testdis|awk '{ SUM += $2 } END {
print SUM/1024 " KB" }'
819200 KB

root@e1:/var/log# blkdiscard -o 0 -l 409600 /dev/rbd28
root@e1:/var/log# rbd diff spin1/testdis|awk '{ SUM += $2 } END {
print SUM/1024 " KB" }'
819200 KB

root@e1:/var/log# blkdiscard -o 0 -l 4096000 /dev/rbd28
root@e1:/var/log# rbd diff spin1/testdis|awk '{ SUM += $2 } END {
print SUM/1024 " KB" }'
819200 KB

root@e1:/var/log# blkdiscard -o 0 -l 40960000 /dev/rbd28
root@e1:/var/log# rbd diff spin1/testdis|awk '{ SUM += $2 } END {
print SUM/1024 " KB" }'
782336 KB
--
Alex Gorbachev
Storcium


On Sat, Jul 30, 2016 at 9:11 PM, Alex Gorbachev <ag...@is...> wrote:
>>
>> On Wednesday, July 27, 2016, Vladislav Bolkhovitin <vs...@vl...> wrote:
>>>
>>>
>>> Alex Gorbachev wrote on 07/27/2016 10:33 AM:
>>> > One other experiment: just running blkdiscard against the RBD block
>>> > device completely clears it, to the point where the rbd-diff method
>>> > reports 0 blocks utilized.  So to summarize:
>>> >
>>> > - ESXi sending UNMAP via SCST does not seem to release storage from
>>> > RBD (BLOCKIO handler that is supposed to work with UNMAP)
>>> >
>>> > - blkdiscard does release the space
>>>
>>> How did you run blkdiscard? It might be that blkdiscard discarded big
>>> areas, while ESXi
>>> sending UNMAP commands for areas smaller, than min size, which could be
>>> discarded, or
>>> not aligned as needed, so those discard requests just ignored.
>
> Here is the output of the debug, many more of these statements before
> and after.  Is it correct to state then that SCST is indeed executing
> the discard and the RBD device is ignoring it (since the used size in
> ceph is not diminishing)?
>
> Jul 30 21:08:46 e1 kernel: [ 3032.199972] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570716160, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.202622] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570724352, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.207214] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570732544, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.210395] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570740736, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.212951] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570748928, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.216187] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570757120, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.219299] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570765312, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.222658] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570773504, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.225948] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570781696, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.230092] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570789888, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.234153] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570798080, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.238001] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570806272, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.240876] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570814464, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.242771] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570822656, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.244943] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570830848, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.247506] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570839040, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.250090] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570847232, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.253229] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570855424, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.256001] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570863616, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.259204] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570871808, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.261368] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570880000, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.264025] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570888192, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.266737] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570896384, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.270143] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570904576, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.273975] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570912768, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.278163] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570920960, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.282250] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570929152, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.285932] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570937344, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.289736] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570945536, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.292506] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570953728, nr_sects
> 8192)
> Jul 30 21:08:46 e1 kernel: [ 3032.294706] [22016]:
> vdisk_unmap_range:3830:Discarding (start_sector 570961920, nr_sects
> 8192)
>
>
> Thank you,
> Alex
>
>>
>>
>> I indeed ran blkdiscard on the whole device.  So the question to the Ceph
>> list is below what length discard is ignored? I saw at least one other user
>> post a similar issue with ESXi-SCST-RBD.
>>
>>>
>>>
>>> For completely correct test you need to run blkdiscard for exactly the
>>> same areas, both
>>> start and size, as the ESXi UNMAP requests you are seeing on the SCST
>>> traces.
>>
>>
>> I am running a test with the debug settings you provided, and will keep this
>> thread updated with results.  Much appreciate the guidance.
>>
>> Alex
>>
>>>
>>>
>>> Vlad
>>>
>>
>>
>> --
>> --
>> Alex Gorbachev
>> Storcium
>>