RE: [UDT] UDT Resource Usage (again) + Bug in socket creation

[Yunhong Gu <yu...@la...>]

Xudong,

Thank you very much for tracing down the problem. I modified the sndHandler 
to let it wait until an ACK comes in that special situation.

The updated member function is attached. I will update the CVS in the next 
several hours.

Let me know if this fixes the problem.
Yunhong


> On Wed, 15 Mar 2006, Chen Xudong wrote:
>
>> Hi Yunhong,
>> 
>> I spent more time debugging sndHandler. Now I know how exactly the problem
>> occurs: the sndHandler thread is waiting for ACK, but because it is 
>> actively
>> running in a loop, ACK does not get chance to come in (possible due to both
>> sides are on the same machine).
>> 
>> (I did not notice the problem during connection, but saw it seriously 
>> during
>> sending. Maybe you got different thread scheduling so you observed
>> differently.)
>> 
>> Here is the situation:
>> - When start to send short text (say 10 bytes), CSndBuffer::addBuffer
>> creates a single block kept by m_pCurrSendBlk, and m_iCurrBufSize is 10.
>> - Then inside the while loop of sndHandler, the call to 
>> CSndBuffer::readData
>> successfully obtains the only block. CSndBuffer::m_pCurrSendBlk is then set
>> to NULL, but CSndBuffer::m_iCurrBufSize remains 10.
>> - sndHandler then comes to another iteration of the while loop.
>> CSndBuffer::readData is called again, but this time it returns 0 since
>> CSndBuffer::m_pCurrSendBlk has been set to NULL during last iteration.
>> - CSndBuffer::getCurrBufSize is then called. It returns 10 because
>> CSndBuffer::m_iCurrBufSize is still 10 (not yet ACKed). It thus loses a
>> chance to wait in the background for more data.
>> - Eventually the sndHandler thread does not wait for anything but continue
>> to the next while iteration. This goes on and on and the CPU is occupied by
>> this active thread.
>> 
>> Here again are the key lines in core.cpp:
>> 0972:   while (!self->m_bClosing)
>> 1047:         if (0 == (payload =
>> self->m_pSndBuffer->readData(&(datapkt.m_pcData), self->m_iPayloadSize,
>> datapkt.m_iMsgNo)))
>> /// above "if" is false for first time but true subsequently
>> 1050:            if (0 == self->m_pSndBuffer->getCurrBufSize())
>> /// above "if" is always false because ACK does not get chance to come in;
>> this causes the thread lose a chance to become background
>> 1075:            continue;
>> 
>> Possible solution:
>> Need a proper way to put sndHandler thread into background in such case.
>> Maybe to wait for ACK if no data to send. Such wait will put sndHandler
>> thread into sleep.
>> 
>> Workaround:
>> I took suggestions from Pascal (Pas...@gn...) by adding Sleep(1)
>> inside the loop (just before the continue that is after
>> self->m_pSndBuffer->getCurrBufSize). It did work. I think the Sleep puts 
>> the
>> sndHandler thread into background and thus ACK can come in.
>> 
>> Regards,
>> Xudong
>> 
>> PS: You're right on overlapped IO. Pardon me I did not study on its usage 
>> as
>> in your documentation.
>> 
>> -----Original Message-----
>> From: Yunhong Gu [mailto:yu...@la...]
>> Sent: Wednesday, March 15, 2006 1:33 PM
>> To: Chen Xudong
>> Cc: udt...@li...
>> Subject: RE: [UDT] UDT Resource Usage (again) + Bug in socket creation
>> 
>> 
>> I checked and tested your code.
>> 
>> 1. I noticed the high CPU problem before connection (and a little after
>> connection). The reason is that it seems SO_RCVTIMEO does not work on
>> Windows when receiving from a local address. Adding a "Sleep(1)" at line
>> 665 of core.cpp can solve the problem.
>> 
>> 2. I didn't notice any high CPU problem after the connection is
>> established.
>> 
>> 3. when using overlapped IO, you cannot reuse the same buffer for sending.
>> Because in overlapped IO, UDT does not duplicate user buffer but read data
>> from the user buffer directly. If the user buffer is written before the
>> sending is finished, unexpected problem can happen.
>> 
>> 4. the const char* problem is fixed, thanks.
>> 
>> Yunhong
>> 
>> 
>> On Tue, 14 Mar 2006, Chen Xudong wrote:
>> 
>>> Hi Yunhong,
>>> 
>>> Forgot to mention that the high CPU usage problem was observed for
>>> rendezvous connection and both sides were on the same machine.
>>> I hence created a standalone rendezvous connection test program as
>> attached.
>>> I tried it with UDT 3.0 with observations below. Hasn't had time to try
>> with
>>> UDT 2.3 yet.
>>> 
>>> Usage: peer self_ip self_port peer_ip peer_port
>>> For example:
>>> Start peer A as "peer localhost 10000 localhost 20000".
>>> Start peer B as "peer localhost 20000 localhost 10000".
>>> Type some text at one peer and each line is supposed to appear at the
>> other
>>> peer.
>>> Type an empty line to exit the program.
>>> 
>>> In peer.cpp two compilation switches are defined:
>>> MODE: 0 means using SOCK_DGRAM and message IO (sendmsg/recvmsg);
>>>      1 means using SOCK_STREAM and regular IO (send/recv without handle);
>>>      2 means using SOCK_STREAM and overlapped IO (send/recv with handle
>>> given).
>>> MSGSIZE: 0 means using default UDT_MSS.
>>>         1 means using customized UDT_MSS as in original appclient test
>>> program.
>>> 
>>> Observations:
>>> 
>>> For MODE 0 & 1 (message IO & regular IO):
>>> - Sending peer occupies high CPU (more than 90%) for several seconds
>> before
>>> the text appears at the other peer,
>>> - and it continuously occupies CPU for more seconds.
>>> - Sometimes it occupies even longer until the socket is broken.
>>> 
>>> For MODE 2 (overlapped IO):
>>> - Sending peer occupies CPU in the same pattern,
>>> - but more seriously the text does never appear at the other peer.
>>> 
>>> For all MODE:
>>> - Setting UDT_MSS or not leads to the same behavior.
>>> - Compiling in Debug or Release leads to the same behavior.
>>> 
>>> Also found another issue:
>>> The sendmsg API is defined with "char *" parameter. "const char *" should
>> be
>>> more appropriate.
>>> 
>>> Thanks.
>>> 
>>> Regards,
>>> Xudong
>>> 
>>> -----Original Message-----
>>> From: Yunhong Gu [mailto:yu...@la...]
>>> Sent: Tuesday, March 14, 2006 1:45 PM
>>> To: Chen Xudong
>>> Cc: udt...@li...
>>> Subject: Re: [UDT] UDT Resource Usage (again) + Bug in socket creation
>>> 
>>> 
>>> 
>>> Hi, Xudong,
>>> 
>>> Thanks for the information, especially the silly bug in 3.0. I have
>>> updated the CVS.
>>> 
>>> As to the first problem, I just couldn't reproduce it on my machine
>>> (windows XP pro, dual xeon, VS 2003). I attached the source code I am
>>> using on my machine in this email, can you give them a try and let me know
>>> if the problem is still there. Please use release version when you compile
>>> them.
>>> 
>>> Please let me know the throughput, CPU percentage and CPU type.
>>> 
>>> Thanks,
>>> Yunhong
>>> 
>>> 
>>> On Tue, 14 Mar 2006, Chen Xudong wrote:
>>> 
>>>> Hi,
>>>> 
>>>> 
>>>> 
>>>> I also observed high CPU usage at the sending side. This occurred for
>> both
>>>> UDT 2.3 and UDT 3.0 on Windows and it happened every time. I had tried to
>>>> step into UDT 3.0 while it was busy and found out the code was iterating
>>>> inside the while loop in CUDT::sndHandler. Following shows the important
>>>> lines in core.cpp that execution looped around.
>>>> 
>>>> 0972:   while (!self->m_bClosing)
>>>> 
>>>> 1047:         if (0 == (payload =
>>>> self->m_pSndBuffer->readData(&(datapkt.m_pcData), self->m_iPayloadSize,
>>>> datapkt.m_iMsgNo)))
>>>> 
>>>> 1050:            if (0 == self->m_pSndBuffer->getCurrBufSize())
>>>> 
>>>> 1072:               self->m_ullTimeDiff = 0;
>>>> 
>>>> 1075:            continue;
>>>> 
>>>> I compiled the UDT library using NO_BUSY_WAITING but without CUSTOM_CC.
>>>> 
>>>> 
>>>> 
>>>> BTW, there is a bug in your UDT 3.0 code in CUDTUnited::newSocket in
>>>> api.cpp.
>>>> 
>>>> 0155:   if ((type != SOCK_STREAM) || (type != SOCK_DGRAM))
>>>> 
>>>> 0156:      throw CUDTException(5, 3, 0);
>>>> 
>>>> "||" should be changed to "&&". This bug causes all programs to fail
>>> because
>>>> a socket can never be created.
>>>> 
>>>> 
>>>> 
>>>> Thanks & Regards,
>>>> 
>>>> Xudong
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> 
>> -------------------------------------------------------
>> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
>> that extends applications into web and mobile media. Attend the live 
>> webcast
>> and join the prime developer group breaking into this new coding territory!
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
>> _______________________________________________
>> UDT-general mailing list
>> UDT...@li...
>> https://lists.sourceforge.net/lists/listinfo/udt-general
>