Menu
▾
▴
[Scst-devel] Strange performance behavior on top of encrypted raid
|
From: Göran B. <goe...@gm...> - 2019-06-06 20:17:34
|
Dear developers,
I experience a strange performance issue.
Setup is a software raid5 encrypted via dm-crypt with logical volumes.
One logical volumne is exported via scst-iscsi and used by a windows
computer.
I had this setup working without encryption for years and there were
never any problems.
Since I switched to encryption the write performance when transfering
large files (few GBs) is bad ...about 30MB/s.
It looks like this (iostat -m 10):
avg-cpu: %user %nice %system %iowait %steal %idle
0.05 0.00 30.71 0.00 0.00 69.24
Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 146.60 2.86 16.41 28 164
sdb 144.30 2.79 16.62 27 166
sdc 141.40 2.66 16.42 26 164
md1 114.20 0.00 28.31 0 283
dm-2 114.20 0.00 28.31 0 283
dm-5 114.20 0.00 28.31 0 283
sdx are the harddrives, md1 is raid5, dm-2 ist the encryted volume and
dm-5 the logical volume used as block device.
When I copy smaller files eg. bunch of audio files it looks like this:
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 56.49 0.00 0.00 43.51
Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 210.20 6.00 37.28 59 372
sdb 203.80 5.95 37.28 59 372
sdc 204.80 6.18 37.45 61 374
md1 262.50 0.00 63.91 0 639
dm-2 262.40 0.00 63.88 0 638
dm-5 262.40 0.00 63.88 0 638
When switching to file io it gets even faster, with the usual buffer
effects.
The cpu is only dual core but encryption performance does not seem to be
the problem.
My scst.conf is pretty standard:
HANDLER vdisk_blockio {
DEVICE backup-goeran {
filename /dev/raid/backup-goeran
nv_cache 1
blocksize 4096
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2013-04.local.brunsnetz:backup.goeran {
IncomingUser "iqn.1991-05.com.microsoft:goeran xxxxxxxxxxx"
enabled 1
LUN 0 backup-goeran
HeaderDigest CRC32C
DataDigest CRC32C
}
}
The only unusal is the blocksize. I did change it for tests but it does
not make a difference.
Start up log entries.
2019-06-06 21:37:20 warning /etc/init.d/scst[11662]: /etc/init.d/scst
uses runscript, please convert to openrc-run.
2019-06-06 21:37:20 info kernel: scst: Attached to scsi0, channel 0, id
0, lun 0, type 0
2019-06-06 21:37:20 info kernel: scst: Attached to scsi1, channel 0, id
0, lun 0, type 0
2019-06-06 21:37:20 info kernel: scst: Attached to scsi2, channel 0, id
0, lun 0, type 0
2019-06-06 21:37:20 info kernel: scst: Attached to scsi6, channel 0, id
0, lun 0, type 0
2019-06-06 21:37:20 info kernel: scst: Attached to scsi7, channel 0, id
0, lun 0, type 0
2019-06-06 21:37:20 info kernel: scst: User interface thread started
2019-06-06 21:37:20 info kernel: scst: Init thread started
2019-06-06 21:37:20 info kernel: scst: Task management thread started
2019-06-06 21:37:20 info kernel: scst: Target template copy_manager
registered successfully
2019-06-06 21:37:20 info kernel: scst: Target copy_manager_tgt for
template copy_manager registered successfully
2019-06-06 21:37:20 info kernel: scst: Using security group
"copy_manager_tgt" for initiator "copy_manager_sess" (target
copy_manager_tgt)
2019-06-06 21:37:20 info kernel: scst: SCST version 3.4.0-pre1 loaded
successfully (global max mem for commands ignored, per device 788MB)
2019-06-06 21:37:20 info kernel: scst: Management thread started
2019-06-06 21:37:20 info kernel: scst: Virtual device handler
vdisk_fileio for type 0 registered successfully
2019-06-06 21:37:20 info kernel: scst: Virtual device handler
vdisk_blockio for type 0 registered successfully
2019-06-06 21:37:20 info kernel: scst: Virtual device handler
vdisk_nullio for type 0 registered successfully
2019-06-06 21:37:20 info kernel: scst: Virtual device handler vcdrom for
type 5 registered successfully
2019-06-06 21:37:20 info kernel: iscsi-scst: iSCSI SCST Target - version
3.4.0-pre1
2019-06-06 21:37:20 info kernel: iscsi-scst: Registered iSCSI transport:
iSCSI-TCP
2019-06-06 21:37:20 info kernel: scst: Target template iscsi registered
successfully
2019-06-06 21:37:20 info kernel: iscsi-scst: Read thread for pool
000000001d6a42f4 started
2019-06-06 21:37:20 info kernel: iscsi-scst: Read thread for pool
000000001d6a42f4 started
2019-06-06 21:37:20 info kernel: iscsi-scst: Write thread for pool
000000001d6a42f4 started
2019-06-06 21:37:20 info kernel: iscsi-scst: Write thread for pool
000000001d6a42f4 started
2019-06-06 21:37:20 debug iscsi-scstd: max_data_seg_len 1048576,
max_queued_cmds 2048
2019-06-06 21:37:21 info kernel: scst: Removed all devices from group
copy_manager_tgt
2019-06-06 21:37:21 info kernel: dev_vdisk: Registering virtual
vdisk_blockio device backup-goeran (NV_CACHE, BLOCKIO, ROTATIONAL)
2019-06-06 21:37:21 info kernel: dev_vdisk: Attached SCSI target virtual
disk backup-goeran (file="/dev/raid/backup-goeran", fs=1048576MB,
bs=4096, nblocks=268435456, cyln=131072)
2019-06-06 21:37:21 info kernel: scst: Added device backup-goeran to
group copy_manager_tgt (LUN 0, flags 0x4) to target copy_manager_tgt
2019-06-06 21:37:21 info kernel: scst: Attached to virtual device
backup-goeran (id 1)
2019-06-06 21:37:21 info kernel: scst: Target
iqn.2013-04.local.brunsnetz:backup.goeran for template iscsi registered
successfully
2019-06-06 21:37:21 info kernel: scst: Added device backup-goeran to
group iqn.2013-04.local.brunsnetz:backup.goeran (LUN 0, flags 0x2) to
target iqn.2013-04.local.brunsnetz:backup.goeran
2019-06-06 21:37:21 info kernel: scst: Using autogenerated relative
target id 1 for target iqn.2013-04.local.brunsnetz:backup.goeran
2019-06-06 21:37:22 info iscsi-scstd: Connect from 192.168.2.2:54649 to
192.168.2.201:3260
2019-06-06 21:37:22 info kernel: scst: Using security group
"iqn.2013-04.local.brunsnetz:backup.goeran" for initiator
"iqn.1991-05.com.microsoft:goeran" (target
iqn.2013-04.local.brunsnetz:backup.goeran)
2019-06-06 21:37:22 info kernel: iscsi-scst: Negotiated parameters:
InitialR2T No, ImmediateData Yes, MaxConnections 1,
MaxRecvDataSegmentLength 1048576, MaxXmitDataSegmentLength 65536,
2019-06-06 21:37:22 info kernel: iscsi-scst: MaxBurstLength 262144,
FirstBurstLength 65536, DefaultTime2Wait 0, DefaultTime2Retain 0,
2019-06-06 21:37:22 info kernel: iscsi-scst: MaxOutstandingR2T 16,
DataPDUInOrder Yes, DataSequenceInOrder Yes, ErrorRecoveryLevel 0,
2019-06-06 21:37:22 info kernel: iscsi-scst: HeaderDigest CRC32C,
DataDigest CRC32C, OFMarker No, IFMarker No, OFMarkInt 2048, IFMarkInt
2048, RDMAExtensions No
2019-06-06 21:37:22 info kernel: iscsi-scst: Target parameters set for
session 1000037010040: QueuedCommands 32, Response timeout 90, Nop-In
interval 30, Nop-In timeout 30
I updated to latest svn commit r8418 but no changes. I am out of ideas,
why the performance for large files is so bad. I hope you can give me a
hint.
regards
Göran
PS:
https://sourceforge.net/p/scst/svn/HEAD/tree/trunk/iscsi-scst/Makefile#l174
causes an make error. $(INFINIBAND_ENABLED) is a string not a bool value.
|
