Re: [Lksctp-developers] blocking sctp_sendmsg & internal socket buffering

[Neil H. <nh...@re...>]

On Thu, Mar 24, 2005 at 10:00:34AM -0800, Brad Penoff wrote:
> Greetings,
>=20
> We have an application whose behavior is raising some questions regarding=
=20
> how the socket buffers act in lksctp;  hopefully I can get some clarity=
=20
> and maybe even some pointers!
>=20
> Our application can run on any number of nodes, and each node is=20
> constantly doing as much reading and writing as it can.  We use the=20
> one-to-many style with a single socket.  The psuedo code of the portion i=
n=20
> question would look something like this:
>=20
>=20
> //magically passed in variables
> struct thing_to_write_to *have_stuff_left_to_write;
> int number_of_possible_pending_reads
>=20
> //local variable
> int have_read_something =3D 0;
>=20
> //write loop
> while (have_stuff_left_to_write){
>   if( write_to_the_appropriate_stream(write_data)  =3D=3D -1)
>     if(! (this_is_a_non_blocking_operation && errno =3D=3D EAGAIN))
>       return ERROR;
>=20
>   have_stuff_left_to_write =3D have_stuff_left_to_write->next;
> }
>=20
> //read loop
> while(number_of_possible_pending_reads) {
>   if( read_from_whatever_is_next(read_data) =3D=3D -1)
>     if(! (this_is_a_non_blocking_operation && errno =3D=3D EAGAIN))
>       return ERROR;
>=20
>   have_read_something++;
>   put_data_in_the_appropriate_userspace_buffer(read_data);
>   number_of_possible_pending_reads--;
> }
>=20
>=20
>=20
> From this, I'd like to ask a few questions:
>=20
> 1)  When write_data is large (relative to the send socket buffer), we spl=
it=20
> it into smaller messages, so at the level shown=20
> "write_to_the_appropriate_stream" is atomic.  But what happens is, say fo=
r=20
> a write_data of 8 MB, the internal
> sctp_sendmsg call inside write_to_the_appropriate stream will block about=
=20
> 2 MB in, regardless of whether the socket itself is set to blocking or=20
> non-blocking.  Why might this be?
>=20
Not sure, but it sounds like a bug to me.

>    a) If my socket is set to non-blocking, I was hoping in the write loop=
=20
> that it'd return EAGAIN so I'd eventually make it to my read loop.=20
> However, it blocks... why, and how can I avoid this?
>=20
I'm not sure why you're not getting EAGAIN.  I can get that return in sctp_=
darn
quite when I set it to non-blocking and back up the send queue.  Can you be
a little more detailed in your test case.  I force this error by sending se=
tting
up an sctp_darn listener and sender, suspending the listener (via ctrl-z) a=
nd
flooding data through the sender.

>    b) MSG_DONTWAIT isn't recognized by sctp_sendmsg; is this correct?
>=20
It should be.  Its tested for in sctp_sendmsg, when we retrieve the socket =
send
timeout value.  If MSG_DONTWAIT is set on the socket, this test should set =
the
timeout to be zero.

>=20
> 2)  Are the socket send and receive buffers the total size per socket or=
=20
> the size per association?  I've read where HAVE_SCTP_MULTIBUF tells of=20
> this but I'm having trouble understanding if the socket size is simply=20
> divided by the number of (active?) associations or if the total actual=20
> size for a socket is multiplied by number of associations.
>=20
Currently each association on a socket get sk_sendbuf bytes to allocate
(effectively implying that each socket can allocate up to n times the memory
specified in the sendbuf limit for the socket, where n is the number of soc=
kets.
I've got a patch pending on this list that allows the mode of a sctp socket=
 to
be switched such that sendbuffer accounting is done in such a manner that
respects the sctp spec (the current operational mode), or respects socket
accounting rules, or a hybrid thereof.  You should be able to find it in the
archives if your interested.  Its only about a week or two old.

>=20
> So my main thing is I can't get sctp_sendmsg to return EAGAIN instead of=
=20
> blocking.  I'm guessing this has to do with internal socket buffering, so=
=20
> I'm just trying to understand what limitations I have to work with.
>=20
> Any suggestions!??
I'd welcome some additional testing of my patch :).  Its here:
http://sourceforge.net/mailarchive/message.php?msg_id=3D11135419
Also, are you certain that you've maxed out the sendbuffer queue?  How are =
you
determining that its full, and should be returning EAGAIN to the applicatio=
n?

Regards
Neil
>=20
> Thanks,
> brad
>=20
>=20
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=3D6595&alloc_id=3D14396&op=3Dclick
> _______________________________________________
> Lksctp-developers mailing list
> Lks...@li...
> https://lists.sourceforge.net/lists/listinfo/lksctp-developers

--=20
/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *nh...@re...
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/