[Iscsitarget-devel] Iscsi Read Consumes 100% CPU Problem

[rae l <cr...@gm...>]

A simple nullio configuration:

Target iqn.2001-04.com.example:storage.disk2.sys1.xyzz
	Lun 0 Type=nullio

A windows client initiator logon this target and use iometer to test it,
While iometer is reading, this cpu load is observed on the target server:

gektop@tux ~ $ dstat -M cpu,net 5
----total-cpu-usage---- -net/total-
usr sys idl wai hiq siq| recv  send
 30   4  66   0   0   0|   0     0
 23  41   0   0  17  19| 246k   10M
 22  36   0   0  21  21| 264k   11M
 24  38   0   0  17  21| 261k   11M
 24  36   0   0  21  19| 268k   11M

on the desktop computer the only one nic is a 100Mb network adapter,
so the network is near the limit,

but the problem is: why it consumes so high CPU?

on another server hardware with the same ietd.conf and dual-core CPU
and 1000Mb nic, it was observed that "sys,hiq,siq" consumes all the
CPU resources, too.

with systemtap debugging, I found the sendpage (really tcp_sendpage)
would very likely return -EAGAIN, I think this loop consumes so high
CPU,

but I have done a simple test, sendpage without MSG_DONTWAIT, in this
situation, the target consumed very low CPU resources, average under
1%, with the same nullio performance, 110MB, nearly the 1000Mb limit,

I'm not very clear about the meangings of tcp_sendpage with or without
MSG_DONTWAIT, so please review this patch:

Index: iscsitarget-r168/kernel/nthread.c
===================================================================
--- iscsitarget-r168.orig/kernel/nthread.c
+++ iscsitarget-r168/kernel/nthread.c
@@ -294,7 +294,7 @@ static int write_data(struct iscsi_conn
 	struct iovec *iop;
 	int saved_size, size, sendsize;
 	int offset, idx;
-	int flags, res;
+	int res;

 	file = conn->file;
 	saved_size = size = conn->write_size;
@@ -351,12 +351,11 @@ static int write_data(struct iscsi_conn

 	sock = conn->sock;
 	sendpage = sock->ops->sendpage ? : sock_no_sendpage;
-	flags = MSG_DONTWAIT;

 	while (1) {
 		sendsize = PAGE_CACHE_SIZE - offset;
 		if (size <= sendsize) {
-			res = sendpage(sock, tio->pvec[idx], offset, size, flags);
+			res = sendpage(sock, tio->pvec[idx], offset, size, 0);
 			dprintk(D_DATA, "%s %#Lx:%u: %d(%lu,%u,%u)\n",
 				sock->ops->sendpage ? "sendpage" : "writepage",
 				(unsigned long long ) conn->session->sid, conn->cid,
@@ -377,7 +376,7 @@ static int write_data(struct iscsi_conn
 			continue;
 		}

-		res = sendpage(sock, tio->pvec[idx], offset,sendsize, flags | MSG_MORE);
+		res = sendpage(sock, tio->pvec[idx], offset,sendsize, MSG_MORE);
 		dprintk(D_DATA, "%s %#Lx:%u: %d(%lu,%u,%u)\n",
 			sock->ops->sendpage ? "sendpage" : "writepage",
 			(unsigned long long ) conn->session->sid, conn->cid,


Thanks.

-- 
Denis Cheng
Linux Application Developer

"One of my most productive days was throwing away 1000 lines of code."
 - Ken Thompson.