Hi, after digging through the code, it seems to me like there would be a really good use case for some codec changes.
In particular, I think it would be important to support different codec parameters going to the server and coming from the server. Almost all "at home" Internet connections feature a significantly higher download capacity as opposed to the upload capacity, and a good transparent mix requires more bandwidth than the individual instruments it has been created from.
Jamulus does support one case of asymmetry: Mono-In/Stereo-Out, but while it is logically present, it is absent at the communication layer: the mono instrument is expanded to stereo at the client side and transferred with a stereo codec, using the same fixed bandwidth that is used for receiving the stereo mix. Also, if you do server-side recordings, a stereo track with identical channels will be laid down. Complete waste of bandwidth and disk space.
Now one quirk of Opus is that the decoder does not actually need to have any clue about the quality setting or stereo setting of the encoder: a custom mode (like those used by Jamulus) is defined via sample rate and packet duration (currently 64 or 128 samples). The quality information is conveyed via packet size which may be variable for VBR modes. Jamulus does not keep track of network packet size (or sends packets in the size signaled by the encoder), so it cannot even do VBR. That seems like bad programming.
Once you nail down the sampling interval (in Jamulus, currently OPUS or OPUS64 formats), any decoder can deal with any encoder if it is given the actual packet size arriving, assuming I read the documentation correctly. When needed, stereo is downmixed to mono, or mono is expanded to stereo. So assuming the server is programmed correctly (namely passing received packet sizes on), it only needs to get informed about the quality and stereosity of the departing packets, and the arriving packets can be whatever (it would still make sense to inform the server about tracks that would record well enough in mono).
So steps I see as sensible: make server and client pass actual network packet sizes on everywhere. The server needs to flag to a client (protocol version number or something?) that it is capable of dealing with packets in different sizes. A client that has this information can choose to send stuff upstream with lower quality than it receives it, particularly relevant for mono-in/stereo-out. It also can choose to send stuff with VBR though I am not sure there would be much of a point to that.
Basically git grep iUnused gives a good approximation of the locations that want fixing in order to correspond with the documented Opus lib API (as far as I understand reading the code, with the current use patterns and modes stuff is likely to happen to work but there seems little point in relying on that).
Then the particular case of mono-in/stereo-out would warrant informing the server so that it has the option of recording in mono (and probably decoding a bit more efficiently).
For a server supporting different quality settings up/downstream, the client should adapt the upstream stereosity to the need and possibly offer to adjust the upstream parameters/quality in other respects.
I am currently looking at the code, but the protocol for extending the protocol (and the requisite versioning) is not clear to me, and of course a solid implementation that can be used in the official servers/clients would be desired eventually.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi, after digging through the code, it seems to me like there would be a really good use case for some codec changes.
In particular, I think it would be important to support different codec parameters going to the server and coming from the server. Almost all "at home" Internet connections feature a significantly higher download capacity as opposed to the upload capacity, and a good transparent mix requires more bandwidth than the individual instruments it has been created from.
Jamulus does support one case of asymmetry: Mono-In/Stereo-Out, but while it is logically present, it is absent at the communication layer: the mono instrument is expanded to stereo at the client side and transferred with a stereo codec, using the same fixed bandwidth that is used for receiving the stereo mix. Also, if you do server-side recordings, a stereo track with identical channels will be laid down. Complete waste of bandwidth and disk space.
Now one quirk of Opus is that the decoder does not actually need to have any clue about the quality setting or stereo setting of the encoder: a custom mode (like those used by Jamulus) is defined via sample rate and packet duration (currently 64 or 128 samples). The quality information is conveyed via packet size which may be variable for VBR modes. Jamulus does not keep track of network packet size (or sends packets in the size signaled by the encoder), so it cannot even do VBR. That seems like bad programming.
Once you nail down the sampling interval (in Jamulus, currently OPUS or OPUS64 formats), any decoder can deal with any encoder if it is given the actual packet size arriving, assuming I read the documentation correctly. When needed, stereo is downmixed to mono, or mono is expanded to stereo. So assuming the server is programmed correctly (namely passing received packet sizes on), it only needs to get informed about the quality and stereosity of the departing packets, and the arriving packets can be whatever (it would still make sense to inform the server about tracks that would record well enough in mono).
So steps I see as sensible: make server and client pass actual network packet sizes on everywhere. The server needs to flag to a client (protocol version number or something?) that it is capable of dealing with packets in different sizes. A client that has this information can choose to send stuff upstream with lower quality than it receives it, particularly relevant for mono-in/stereo-out. It also can choose to send stuff with VBR though I am not sure there would be much of a point to that.
Basically
git grep iUnusedgives a good approximation of the locations that want fixing in order to correspond with the documented Opus lib API (as far as I understand reading the code, with the current use patterns and modes stuff is likely to happen to work but there seems little point in relying on that).Then the particular case of mono-in/stereo-out would warrant informing the server so that it has the option of recording in mono (and probably decoding a bit more efficiently).
For a server supporting different quality settings up/downstream, the client should adapt the upstream stereosity to the need and possibly offer to adjust the upstream parameters/quality in other respects.
I am currently looking at the code, but the protocol for extending the protocol (and the requisite versioning) is not clear to me, and of course a solid implementation that can be used in the official servers/clients would be desired eventually.