RE: [UDT] UDT Resource Usage (again) + Bug in socket creation

[Chen X. <xd...@es...>]

Hi Yunhong,

I spent more time debugging sndHandler. Now I know how exactly the problem
occurs: the sndHandler thread is waiting for ACK, but because it is actively
running in a loop, ACK does not get chance to come in (possible due to both
sides are on the same machine).

(I did not notice the problem during connection, but saw it seriously during
sending. Maybe you got different thread scheduling so you observed
differently.)

Here is the situation:
- When start to send short text (say 10 bytes), CSndBuffer::addBuffer
creates a single block kept by m_pCurrSendBlk, and m_iCurrBufSize is 10.
- Then inside the while loop of sndHandler, the call to CSndBuffer::readData
successfully obtains the only block. CSndBuffer::m_pCurrSendBlk is then set
to NULL, but CSndBuffer::m_iCurrBufSize remains 10.
- sndHandler then comes to another iteration of the while loop.
CSndBuffer::readData is called again, but this time it returns 0 since
CSndBuffer::m_pCurrSendBlk has been set to NULL during last iteration.
- CSndBuffer::getCurrBufSize is then called. It returns 10 because
CSndBuffer::m_iCurrBufSize is still 10 (not yet ACKed). It thus loses a
chance to wait in the background for more data.
- Eventually the sndHandler thread does not wait for anything but continue
to the next while iteration. This goes on and on and the CPU is occupied by
this active thread.

Here again are the key lines in core.cpp:
0972:   while (!self->m_bClosing)
1047:         if (0 == (payload =
self->m_pSndBuffer->readData(&(datapkt.m_pcData), self->m_iPayloadSize,
datapkt.m_iMsgNo)))
/// above "if" is false for first time but true subsequently
1050:            if (0 == self->m_pSndBuffer->getCurrBufSize())
/// above "if" is always false because ACK does not get chance to come in;
this causes the thread lose a chance to become background
1075:            continue;

Possible solution:
Need a proper way to put sndHandler thread into background in such case.
Maybe to wait for ACK if no data to send. Such wait will put sndHandler
thread into sleep.

Workaround:
I took suggestions from Pascal (Pas...@gn...) by adding Sleep(1)
inside the loop (just before the continue that is after
self->m_pSndBuffer->getCurrBufSize). It did work. I think the Sleep puts the
sndHandler thread into background and thus ACK can come in.

Regards,
Xudong

PS: You're right on overlapped IO. Pardon me I did not study on its usage as
in your documentation.

-----Original Message-----
From: Yunhong Gu [mailto:yu...@la...] 
Sent: Wednesday, March 15, 2006 1:33 PM
To: Chen Xudong
Cc: udt...@li...
Subject: RE: [UDT] UDT Resource Usage (again) + Bug in socket creation


I checked and tested your code.

1. I noticed the high CPU problem before connection (and a little after 
connection). The reason is that it seems SO_RCVTIMEO does not work on 
Windows when receiving from a local address. Adding a "Sleep(1)" at line 
665 of core.cpp can solve the problem.

2. I didn't notice any high CPU problem after the connection is 
established.

3. when using overlapped IO, you cannot reuse the same buffer for sending. 
Because in overlapped IO, UDT does not duplicate user buffer but read data 
from the user buffer directly. If the user buffer is written before the 
sending is finished, unexpected problem can happen.

4. the const char* problem is fixed, thanks.

Yunhong


On Tue, 14 Mar 2006, Chen Xudong wrote:

> Hi Yunhong,
>
> Forgot to mention that the high CPU usage problem was observed for
> rendezvous connection and both sides were on the same machine.
> I hence created a standalone rendezvous connection test program as
attached.
> I tried it with UDT 3.0 with observations below. Hasn't had time to try
with
> UDT 2.3 yet.
>
> Usage: peer self_ip self_port peer_ip peer_port
> For example:
> Start peer A as "peer localhost 10000 localhost 20000".
> Start peer B as "peer localhost 20000 localhost 10000".
> Type some text at one peer and each line is supposed to appear at the
other
> peer.
> Type an empty line to exit the program.
>
> In peer.cpp two compilation switches are defined:
> MODE: 0 means using SOCK_DGRAM and message IO (sendmsg/recvmsg);
>      1 means using SOCK_STREAM and regular IO (send/recv without handle);
>      2 means using SOCK_STREAM and overlapped IO (send/recv with handle
> given).
> MSGSIZE: 0 means using default UDT_MSS.
>         1 means using customized UDT_MSS as in original appclient test
> program.
>
> Observations:
>
> For MODE 0 & 1 (message IO & regular IO):
> - Sending peer occupies high CPU (more than 90%) for several seconds
before
> the text appears at the other peer,
> - and it continuously occupies CPU for more seconds.
> - Sometimes it occupies even longer until the socket is broken.
>
> For MODE 2 (overlapped IO):
> - Sending peer occupies CPU in the same pattern,
> - but more seriously the text does never appear at the other peer.
>
> For all MODE:
> - Setting UDT_MSS or not leads to the same behavior.
> - Compiling in Debug or Release leads to the same behavior.
>
> Also found another issue:
> The sendmsg API is defined with "char *" parameter. "const char *" should
be
> more appropriate.
>
> Thanks.
>
> Regards,
> Xudong
>
> -----Original Message-----
> From: Yunhong Gu [mailto:yu...@la...]
> Sent: Tuesday, March 14, 2006 1:45 PM
> To: Chen Xudong
> Cc: udt...@li...
> Subject: Re: [UDT] UDT Resource Usage (again) + Bug in socket creation
>
>
>
> Hi, Xudong,
>
> Thanks for the information, especially the silly bug in 3.0. I have
> updated the CVS.
>
> As to the first problem, I just couldn't reproduce it on my machine
> (windows XP pro, dual xeon, VS 2003). I attached the source code I am
> using on my machine in this email, can you give them a try and let me know
> if the problem is still there. Please use release version when you compile
> them.
>
> Please let me know the throughput, CPU percentage and CPU type.
>
> Thanks,
> Yunhong
>
>
> On Tue, 14 Mar 2006, Chen Xudong wrote:
>
>> Hi,
>>
>>
>>
>> I also observed high CPU usage at the sending side. This occurred for
both
>> UDT 2.3 and UDT 3.0 on Windows and it happened every time. I had tried to
>> step into UDT 3.0 while it was busy and found out the code was iterating
>> inside the while loop in CUDT::sndHandler. Following shows the important
>> lines in core.cpp that execution looped around.
>>
>> 0972:   while (!self->m_bClosing)
>>
>> 1047:         if (0 == (payload =
>> self->m_pSndBuffer->readData(&(datapkt.m_pcData), self->m_iPayloadSize,
>> datapkt.m_iMsgNo)))
>>
>> 1050:            if (0 == self->m_pSndBuffer->getCurrBufSize())
>>
>> 1072:               self->m_ullTimeDiff = 0;
>>
>> 1075:            continue;
>>
>> I compiled the UDT library using NO_BUSY_WAITING but without CUSTOM_CC.
>>
>>
>>
>> BTW, there is a bug in your UDT 3.0 code in CUDTUnited::newSocket in
>> api.cpp.
>>
>> 0155:   if ((type != SOCK_STREAM) || (type != SOCK_DGRAM))
>>
>> 0156:      throw CUDTException(5, 3, 0);
>>
>> "||" should be changed to "&&". This bug causes all programs to fail
> because
>> a socket can never be created.
>>
>>
>>
>> Thanks & Regards,
>>
>> Xudong
>>
>>
>