The setsockopt doesn't seem to change the buffer lower after you call it. When the first socket is created for incoming connections UDT opens it with a big buffer and after that it doesn't change it lower with a setsockopt call.
I know because if I open it, call setsockopt (as usual) and then close it and then re-open it at that point UDT uses less memory (from 30M ro 10M, which is still too much, my god, what the hell is it doing?)
In this program we clear the buffer pretty fast so I don't know if it needs a big buffer.
The temp fix is to set the default values directly in the code so it always starts out small. (if I can find them all)
It would be nice to have a obvious place in your code that the initial buffer size could be set so that we don't get a big buffer created for 1 second and then reduce that down to small in the next second. Could cause problems in memory management, like going to swap for no reason etc..