RE: [UDT] UDT Resource Usage (again) + Bug in socket creation

[Chen X. <xd...@es...>]

Hi Yunhong,

Yes, it does resolve the high CPU usage problem. Thank you very much!

Regards,
Xudong

-----Original Message-----
From: Yunhong Gu [mailto:yu...@la...] 
Sent: Thursday, March 16, 2006 11:14 AM
To: Chen Xudong
Cc: udt...@li...
Subject: RE: [UDT] UDT Resource Usage (again) + Bug in socket creation



Xudong,

Thank you very much for tracing down the problem. I modified the 
sndHandler to let it wait until an ACK comes in that special situation.

The updated core.cpp is attached. I will update the CVS in the next 
several hours.

Let me know if this fixes the problem.

Yunhong


On Wed, 15 Mar 2006, Chen Xudong wrote:

> Hi Yunhong,
>
> I spent more time debugging sndHandler. Now I know how exactly the problem
> occurs: the sndHandler thread is waiting for ACK, but because it is
actively
> running in a loop, ACK does not get chance to come in (possible due to
both
> sides are on the same machine).
>
> (I did not notice the problem during connection, but saw it seriously
during
> sending. Maybe you got different thread scheduling so you observed
> differently.)
>
> Here is the situation:
> - When start to send short text (say 10 bytes), CSndBuffer::addBuffer
> creates a single block kept by m_pCurrSendBlk, and m_iCurrBufSize is 10.
> - Then inside the while loop of sndHandler, the call to
CSndBuffer::readData
> successfully obtains the only block. CSndBuffer::m_pCurrSendBlk is then
set
> to NULL, but CSndBuffer::m_iCurrBufSize remains 10.
> - sndHandler then comes to another iteration of the while loop.
> CSndBuffer::readData is called again, but this time it returns 0 since
> CSndBuffer::m_pCurrSendBlk has been set to NULL during last iteration.
> - CSndBuffer::getCurrBufSize is then called. It returns 10 because
> CSndBuffer::m_iCurrBufSize is still 10 (not yet ACKed). It thus loses a
> chance to wait in the background for more data.
> - Eventually the sndHandler thread does not wait for anything but continue
> to the next while iteration. This goes on and on and the CPU is occupied
by
> this active thread.
>
> Here again are the key lines in core.cpp:
> 0972:   while (!self->m_bClosing)
> 1047:         if (0 == (payload =
> self->m_pSndBuffer->readData(&(datapkt.m_pcData), self->m_iPayloadSize,
> datapkt.m_iMsgNo)))
> /// above "if" is false for first time but true subsequently
> 1050:            if (0 == self->m_pSndBuffer->getCurrBufSize())
> /// above "if" is always false because ACK does not get chance to come in;
> this causes the thread lose a chance to become background
> 1075:            continue;
>
> Possible solution:
> Need a proper way to put sndHandler thread into background in such case.
> Maybe to wait for ACK if no data to send. Such wait will put sndHandler
> thread into sleep.
>
> Workaround:
> I took suggestions from Pascal (Pas...@gn...) by adding Sleep(1)
> inside the loop (just before the continue that is after
> self->m_pSndBuffer->getCurrBufSize). It did work. I think the Sleep puts
the
> sndHandler thread into background and thus ACK can come in.
>
> Regards,
> Xudong
>
> PS: You're right on overlapped IO. Pardon me I did not study on its usage
as
> in your documentation.
>
> -----Original Message-----
> From: Yunhong Gu [mailto:yu...@la...]
> Sent: Wednesday, March 15, 2006 1:33 PM
> To: Chen Xudong
> Cc: udt...@li...
> Subject: RE: [UDT] UDT Resource Usage (again) + Bug in socket creation
>
>
> I checked and tested your code.
>
> 1. I noticed the high CPU problem before connection (and a little after
> connection). The reason is that it seems SO_RCVTIMEO does not work on
> Windows when receiving from a local address. Adding a "Sleep(1)" at line
> 665 of core.cpp can solve the problem.
>
> 2. I didn't notice any high CPU problem after the connection is
> established.
>
> 3. when using overlapped IO, you cannot reuse the same buffer for sending.
> Because in overlapped IO, UDT does not duplicate user buffer but read data
> from the user buffer directly. If the user buffer is written before the
> sending is finished, unexpected problem can happen.
>
> 4. the const char* problem is fixed, thanks.
>
> Yunhong
>
>
> On Tue, 14 Mar 2006, Chen Xudong wrote:
>
>> Hi Yunhong,
>>
>> Forgot to mention that the high CPU usage problem was observed for
>> rendezvous connection and both sides were on the same machine.
>> I hence created a standalone rendezvous connection test program as
> attached.
>> I tried it with UDT 3.0 with observations below. Hasn't had time to try
> with
>> UDT 2.3 yet.
>>
>> Usage: peer self_ip self_port peer_ip peer_port
>> For example:
>> Start peer A as "peer localhost 10000 localhost 20000".
>> Start peer B as "peer localhost 20000 localhost 10000".
>> Type some text at one peer and each line is supposed to appear at the
> other
>> peer.
>> Type an empty line to exit the program.
>>
>> In peer.cpp two compilation switches are defined:
>> MODE: 0 means using SOCK_DGRAM and message IO (sendmsg/recvmsg);
>>      1 means using SOCK_STREAM and regular IO (send/recv without handle);
>>      2 means using SOCK_STREAM and overlapped IO (send/recv with handle
>> given).
>> MSGSIZE: 0 means using default UDT_MSS.
>>         1 means using customized UDT_MSS as in original appclient test
>> program.
>>
>> Observations:
>>
>> For MODE 0 & 1 (message IO & regular IO):
>> - Sending peer occupies high CPU (more than 90%) for several seconds
> before
>> the text appears at the other peer,
>> - and it continuously occupies CPU for more seconds.
>> - Sometimes it occupies even longer until the socket is broken.
>>
>> For MODE 2 (overlapped IO):
>> - Sending peer occupies CPU in the same pattern,
>> - but more seriously the text does never appear at the other peer.
>>
>> For all MODE:
>> - Setting UDT_MSS or not leads to the same behavior.
>> - Compiling in Debug or Release leads to the same behavior.
>>
>> Also found another issue:
>> The sendmsg API is defined with "char *" parameter. "const char *" should
> be
>> more appropriate.
>>
>> Thanks.
>>
>> Regards,
>> Xudong
>>
>> -----Original Message-----
>> From: Yunhong Gu [mailto:yu...@la...]
>> Sent: Tuesday, March 14, 2006 1:45 PM
>> To: Chen Xudong
>> Cc: udt...@li...
>> Subject: Re: [UDT] UDT Resource Usage (again) + Bug in socket creation
>>
>>
>>
>> Hi, Xudong,
>>
>> Thanks for the information, especially the silly bug in 3.0. I have
>> updated the CVS.
>>
>> As to the first problem, I just couldn't reproduce it on my machine
>> (windows XP pro, dual xeon, VS 2003). I attached the source code I am
>> using on my machine in this email, can you give them a try and let me
know
>> if the problem is still there. Please use release version when you
compile
>> them.
>>
>> Please let me know the throughput, CPU percentage and CPU type.
>>
>> Thanks,
>> Yunhong
>>
>>
>> On Tue, 14 Mar 2006, Chen Xudong wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> I also observed high CPU usage at the sending side. This occurred for
> both
>>> UDT 2.3 and UDT 3.0 on Windows and it happened every time. I had tried
to
>>> step into UDT 3.0 while it was busy and found out the code was iterating
>>> inside the while loop in CUDT::sndHandler. Following shows the important
>>> lines in core.cpp that execution looped around.
>>>
>>> 0972:   while (!self->m_bClosing)
>>>
>>> 1047:         if (0 == (payload =
>>> self->m_pSndBuffer->readData(&(datapkt.m_pcData), self->m_iPayloadSize,
>>> datapkt.m_iMsgNo)))
>>>
>>> 1050:            if (0 == self->m_pSndBuffer->getCurrBufSize())
>>>
>>> 1072:               self->m_ullTimeDiff = 0;
>>>
>>> 1075:            continue;
>>>
>>> I compiled the UDT library using NO_BUSY_WAITING but without CUSTOM_CC.
>>>
>>>
>>>
>>> BTW, there is a bug in your UDT 3.0 code in CUDTUnited::newSocket in
>>> api.cpp.
>>>
>>> 0155:   if ((type != SOCK_STREAM) || (type != SOCK_DGRAM))
>>>
>>> 0156:      throw CUDTException(5, 3, 0);
>>>
>>> "||" should be changed to "&&". This bug causes all programs to fail
>> because
>>> a socket can never be created.
>>>
>>>
>>>
>>> Thanks & Regards,
>>>
>>> Xudong
>>>
>>>
>>
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting
language
> that extends applications into web and mobile media. Attend the live
webcast
> and join the prime developer group breaking into this new coding
territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> UDT-general mailing list
> UDT...@li...
> https://lists.sourceforge.net/lists/listinfo/udt-general
>