Hi folks and thanks again to Volker for this amazing software.
I was experimenting some more today with Jamulus and a question came into my mind - could latency be reduced by essentially running the server in each client... avoiding the need to have everything go via an external server? If I run a server locally and friend A connects to that then we have approximated that, but if a third person B joins then the round trip latency from A to B via my server won't be so good and it would be better to have the server running in a datacentre with better connectivity than my broadband. But running a server/VPS hosted in a datacentre is way beyond my technical skill limit (running a server on my laptop and setting up port forwarding I can just about cope with). However in my crazy concept if all 3 of us have "servers" running then could latency be reduced that bit further?
Please bear in mind that this is the thought process of a newbie and might not make much sense or have much merit..! And refactoring Jamulus to work in this way is a long way beyond my skill...... I was simply wondering if this concept might be a way in the future of both reducing latency and making private sessions easier.... but it might not be the way the project or community is currently heading. Clearly it would work best for small groups, if at all.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Giles, not really sure if I got your point: How should one connect to each other with the central connection point missing, as everyone is running a server on his own? You will certainly bring down your latency to your own server, as you will probably be even on the same machine, but what good is it for if nobody else connects, as everybody is "talking" to his/her own server?
Last edit: Harald 2020-04-29
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Harald, yes interconnection is needed clearly (which would be a non-trivial feature request). We just 2 people we can already do it - I tried with a friend connecting to a server on my laptop last week and we concluded that the performance was better connecting to a good public server. Probably the connection between ISPs isn't good enough for this to work well as someone noted in another thread
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ok, I am getting it. Yes, indeed, we just terminated our session - was great. I am running a server on a linux machine below my desk and have installed the client on my (Win10) notebook. My server adress is privatly shared with my folks here in Munich only, and by that, we are getting good latency allowing us to play in sync. Should be related to the short distances, our good network connectivity and up-to-date CPU power allowing for small buffers (64k), with only a few folks connected.
Thanks to Volker for this great piece of software!!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry for the delay. Had to first ask around my team mates about their values.
With my private server running on my separate box at home, of course, I am getting best values, for I am in the same home network; here ping times are some 1-2ms and overall delay around 20 ms. No wonder why :-) More interesting are the times of my Munich mates. Here the average ping time is some 12-15 ms, and overall delay between 35 and 45 ms, which still allows for a good common playing experience.
Just for the sake of doing it and for comparison, I created a MS Azure cloud server with Ubuntu 18.04, according to MS located in western Europe (a more specific location would have come at an unaffordable price :-). Here, my own ping time was 19ms and overall delay 40 ms, not so bad, as along as i was alone. However the values of my mates were mostly worse and mines even got also when more people connected - obvioulsy the load of the server increased and had an influence. Overall, we decided to drop that configuration and continue with my home box.
May be I will give it a try and setup a server at the Frankfurt based provider that Volker suggests in another conversation. Does not require me to start my own box all the time. Let's see when I find some time.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Giles, I don't think the idea of everyone running their own server makes sense. Then there would need to be additional software connecting those servers together. It would just add more complexity and latency.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have been looking into peer to peer communication. It actually slashes latency as the buffering in the server (os) is the biggest delay in the network! We got less than 30 ms mouth to ear across The Netherlands.
The idea is as follows; If you leave the server just to introduce the parties in the session and share addresses and ports where media must be streamed (google UDP hole-punching) you have a very efficient solution.
With some friends I am trying to make a demonstrator of this concept. But perhaps adding this to jamulus might be a faster route....
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The problem with peer to peer is that you lose the synchronization that a
central server provides. The value of a central server is in effectively
providing a common clock source that everyone's audio is synchronized to.
If you can find another way to synchronize all the audio streams together
at the recipient, more power to you.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
which synchronization? By having a low enough latency you have the same synchronization as if you were together live. You can alway include a metronome tick in the session but is it no longer necissary...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Elliot is right. It isn't obvious, but the central server synchronizes the signals implicitly: every receiver of a mixed signal has the same deviation in the delay times amongst all senders. So if somebody compensates his deviation (via his own mix), there isn't any deviation in the mixes of all other listeners, either.
Example of 3 Parties A,B,C via central server:
A > Server: 3ms
B > Server: 5ms
C > Server: 8ms
Server > A: 11ms
Server > B: 18 ms
Server > C: 20 ms
results in:
A > A: 14ms
B > A: 16ms
C > A: 19ms
A > B: 21ms
B > B: 23ms
C > B: 26ms
A > C: 23ms
B > C: 25ms
C > C: 28ms
=> all three hear, That B is 2ms too late and C is 5ms too late compared to A
Similar numbers, but now for direct connection:
A > B: 3ms
A > C: 5ms
B > A: 8ms
B > C: 11ms
C > A: 18ms
C > B: 20ms
=> A hears, that C is 10ms behind B
B hears, that C is 17ms behind A
C hears, that B is 6ms behind A
=> according to the latter 2 results, C would be 11ms behind B. That's an error of 1 ms, which can not be compensated by a player. It's not about the actual numbers (1ms seems neglectable), but to show, that there will be a problem in princible, which you do not have with Jamulus due to the implicit synchronization done by the server.
So, in addition to the compensation done by each individual player, you have to measure the transport times amongst all routes and delay single routes individually, such that the deviations between all players are consistent to each listener. So, a dedicated server might be required to control the individual delays. As the signals need not to be routed via that server, the overall time lag of the audio signals might be lower, though.
A huge disadvantage of that concept would be, that the more persons are joining a session or the worse single time delays are, this will affect the overall delay for all users in that session, which is not the case for the current centralized Jamulus architecture.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
what you are missing in this caclulation is the delay incurred by the server itself. Even just forwarding packets adds quite significant numbers of millisends to the overall latency. (I measured it)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No, I did not miss it. I was just talking about synchronization. Of course, decoding each signal takes time on the server, as well as mixing and encoding. But add the decoding time to the input transport time and time for mixing/encoding to the output transport time: the audible delay deviation will still be the same for each listener.
The decentralized approach is a great idea and might be a huge improvement in terms of overall delay under certain conditions, but not in general. So, it might be not more or less than a good alternative to Jamulus with many pros and cons.
Just another thing to think about is bandwidth: In Jamulus 2 streams are required for each musician, with a decentralized concept it's 2(n-1) with n being the number of musicians. Let's assume that 500kbps are required for each stream (just another example), then each musician needs to have 500kbps upload and 500kbps download, independent from the overall number of musicians. With a decentralized concept, the requirements increase with each additional musician. For 4 people its 1500 kbps each, for a choir with 30 people it's 14500kbps down- and upstream each! In Jamulus, only the server must provide such a bandwidth.
Another topic would be recording: With Jamulus, Multitrack-Recording would be possible on the server only (or by providing a special multitrack-stream), whereas with a decentralized concept each musician might do a multitrack-recording.
I really like the idea of a decentralized concept, but it's just another approach...
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sonobus looked so promising for high quality P2P that I did some testing. Ultimately it was a great disappointment. Taking a P2P approach made the P2P latency hugely more critical than with the Jamulus centralised approach, where the server acts as the focal point for sync. With the same clients the delay (max. 60mS for us) made Sonobus unusable whilst Jamulus was fine. On this basis we didn't do any scale-up testing with more clients.
If I were doing a small number of clients, all of whom had <30mS delay between all of them, then I'd consider Sonobus - but mainly because it offers lossless coding. Even so, if I was one of those clients, then I'd host a local/private Jamulus server and get better results overall.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I also have doubts as to the quality of the code in Sonobus. I have tried it on 2 occasions. Both times it crashed after a while. Jamulus has never, I repeat never, crashed on me during many, many hours of use in the mean time.
And I agree with Andrew, as there is no synchronization the delays must be much smaller with Sonobus. Two uses was enough to see that it is not worth the time spent.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well, that saves me some work (I am doing a talk about Jamulus in about 3 weeks and don't have a lot of time for testing alternatives myself).
With regard to lossless coding: one could put some appropriate codec into Jamulus if wanted. It might make some sense for in-house applications (like the Intranet of a socially distanced music school where everybody is there but in separate rooms). Basically you'd do it when the network overhead ends up a net win in latency (there is no real expectation of quality gains when you drive up the Opus quality to silly levels). I don't think there would be a point in using, say, FLAC to stay lossless when the resulting latency is not an improvement over OPUS. I think it's either (lossy) OPUS compression or uncompressed. Stuff in between does not seem to help a lot.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi folks and thanks again to Volker for this amazing software.
I was experimenting some more today with Jamulus and a question came into my mind - could latency be reduced by essentially running the server in each client... avoiding the need to have everything go via an external server? If I run a server locally and friend A connects to that then we have approximated that, but if a third person B joins then the round trip latency from A to B via my server won't be so good and it would be better to have the server running in a datacentre with better connectivity than my broadband. But running a server/VPS hosted in a datacentre is way beyond my technical skill limit (running a server on my laptop and setting up port forwarding I can just about cope with). However in my crazy concept if all 3 of us have "servers" running then could latency be reduced that bit further?
Please bear in mind that this is the thought process of a newbie and might not make much sense or have much merit..! And refactoring Jamulus to work in this way is a long way beyond my skill...... I was simply wondering if this concept might be a way in the future of both reducing latency and making private sessions easier.... but it might not be the way the project or community is currently heading. Clearly it would work best for small groups, if at all.
Hi Giles, not really sure if I got your point: How should one connect to each other with the central connection point missing, as everyone is running a server on his own? You will certainly bring down your latency to your own server, as you will probably be even on the same machine, but what good is it for if nobody else connects, as everybody is "talking" to his/her own server?
Last edit: Harald 2020-04-29
Hi Harald, yes interconnection is needed clearly (which would be a non-trivial feature request). We just 2 people we can already do it - I tried with a friend connecting to a server on my laptop last week and we concluded that the performance was better connecting to a good public server. Probably the connection between ISPs isn't good enough for this to work well as someone noted in another thread
Ok, I am getting it. Yes, indeed, we just terminated our session - was great. I am running a server on a linux machine below my desk and have installed the client on my (Win10) notebook. My server adress is privatly shared with my folks here in Munich only, and by that, we are getting good latency allowing us to play in sync. Should be related to the short distances, our good network connectivity and up-to-date CPU power allowing for small buffers (64k), with only a few folks connected.
Thanks to Volker for this great piece of software!!
That sounds great. What ping times do your band achieve, and what overall delay?
Sorry for the delay. Had to first ask around my team mates about their values.
With my private server running on my separate box at home, of course, I am getting best values, for I am in the same home network; here ping times are some 1-2ms and overall delay around 20 ms. No wonder why :-) More interesting are the times of my Munich mates. Here the average ping time is some 12-15 ms, and overall delay between 35 and 45 ms, which still allows for a good common playing experience.
Just for the sake of doing it and for comparison, I created a MS Azure cloud server with Ubuntu 18.04, according to MS located in western Europe (a more specific location would have come at an unaffordable price :-). Here, my own ping time was 19ms and overall delay 40 ms, not so bad, as along as i was alone. However the values of my mates were mostly worse and mines even got also when more people connected - obvioulsy the load of the server increased and had an influence. Overall, we decided to drop that configuration and continue with my home box.
May be I will give it a try and setup a server at the Frankfurt based provider that Volker suggests in another conversation. Does not require me to start my own box all the time. Let's see when I find some time.
Giles, I don't think the idea of everyone running their own server makes sense. Then there would need to be additional software connecting those servers together. It would just add more complexity and latency.
I have been looking into peer to peer communication. It actually slashes latency as the buffering in the server (os) is the biggest delay in the network! We got less than 30 ms mouth to ear across The Netherlands.
The idea is as follows; If you leave the server just to introduce the parties in the session and share addresses and ports where media must be streamed (google UDP hole-punching) you have a very efficient solution.
With some friends I am trying to make a demonstrator of this concept. But perhaps adding this to jamulus might be a faster route....
The problem with peer to peer is that you lose the synchronization that a
central server provides. The value of a central server is in effectively
providing a common clock source that everyone's audio is synchronized to.
If you can find another way to synchronize all the audio streams together
at the recipient, more power to you.
which synchronization? By having a low enough latency you have the same synchronization as if you were together live. You can alway include a metronome tick in the session but is it no longer necissary...
I think Elliot is confusing Jamulus with programs like Jamtaba or Jammr, those two programs are based on intervals and synchronisation.
Elliot is right. It isn't obvious, but the central server synchronizes the signals implicitly: every receiver of a mixed signal has the same deviation in the delay times amongst all senders. So if somebody compensates his deviation (via his own mix), there isn't any deviation in the mixes of all other listeners, either.
Example of 3 Parties A,B,C via central server:
A > Server: 3ms
B > Server: 5ms
C > Server: 8ms
Server > A: 11ms
Server > B: 18 ms
Server > C: 20 ms
results in:
A > A: 14ms
B > A: 16ms
C > A: 19ms
A > B: 21ms
B > B: 23ms
C > B: 26ms
A > C: 23ms
B > C: 25ms
C > C: 28ms
=> all three hear, That B is 2ms too late and C is 5ms too late compared to A
Similar numbers, but now for direct connection:
A > B: 3ms
A > C: 5ms
B > A: 8ms
B > C: 11ms
C > A: 18ms
C > B: 20ms
=> A hears, that C is 10ms behind B
B hears, that C is 17ms behind A
C hears, that B is 6ms behind A
=> according to the latter 2 results, C would be 11ms behind B. That's an error of 1 ms, which can not be compensated by a player. It's not about the actual numbers (1ms seems neglectable), but to show, that there will be a problem in princible, which you do not have with Jamulus due to the implicit synchronization done by the server.
So, in addition to the compensation done by each individual player, you have to measure the transport times amongst all routes and delay single routes individually, such that the deviations between all players are consistent to each listener. So, a dedicated server might be required to control the individual delays. As the signals need not to be routed via that server, the overall time lag of the audio signals might be lower, though.
A huge disadvantage of that concept would be, that the more persons are joining a session or the worse single time delays are, this will affect the overall delay for all users in that session, which is not the case for the current centralized Jamulus architecture.
what you are missing in this caclulation is the delay incurred by the server itself. Even just forwarding packets adds quite significant numbers of millisends to the overall latency. (I measured it)
No, I did not miss it. I was just talking about synchronization. Of course, decoding each signal takes time on the server, as well as mixing and encoding. But add the decoding time to the input transport time and time for mixing/encoding to the output transport time: the audible delay deviation will still be the same for each listener.
The decentralized approach is a great idea and might be a huge improvement in terms of overall delay under certain conditions, but not in general. So, it might be not more or less than a good alternative to Jamulus with many pros and cons.
Just another thing to think about is bandwidth: In Jamulus 2 streams are required for each musician, with a decentralized concept it's 2(n-1) with n being the number of musicians. Let's assume that 500kbps are required for each stream (just another example), then each musician needs to have 500kbps upload and 500kbps download, independent from the overall number of musicians. With a decentralized concept, the requirements increase with each additional musician. For 4 people its 1500 kbps each, for a choir with 30 people it's 14500kbps down- and upstream each! In Jamulus, only the server must provide such a bandwidth.
Another topic would be recording: With Jamulus, Multitrack-Recording would be possible on the server only (or by providing a special multitrack-stream), whereas with a decentralized concept each musician might do a multitrack-recording.
I really like the idea of a decentralized concept, but it's just another approach...
You could check out Sonobus and report back your experiences...
Sonobus looked so promising for high quality P2P that I did some testing. Ultimately it was a great disappointment. Taking a P2P approach made the P2P latency hugely more critical than with the Jamulus centralised approach, where the server acts as the focal point for sync. With the same clients the delay (max. 60mS for us) made Sonobus unusable whilst Jamulus was fine. On this basis we didn't do any scale-up testing with more clients.
If I were doing a small number of clients, all of whom had <30mS delay between all of them, then I'd consider Sonobus - but mainly because it offers lossless coding. Even so, if I was one of those clients, then I'd host a local/private Jamulus server and get better results overall.
I also have doubts as to the quality of the code in Sonobus. I have tried it on 2 occasions. Both times it crashed after a while. Jamulus has never, I repeat never, crashed on me during many, many hours of use in the mean time.
And I agree with Andrew, as there is no synchronization the delays must be much smaller with Sonobus. Two uses was enough to see that it is not worth the time spent.
Well, that saves me some work (I am doing a talk about Jamulus in about 3 weeks and don't have a lot of time for testing alternatives myself).
With regard to lossless coding: one could put some appropriate codec into Jamulus if wanted. It might make some sense for in-house applications (like the Intranet of a socially distanced music school where everybody is there but in separate rooms). Basically you'd do it when the network overhead ends up a net win in latency (there is no real expectation of quality gains when you drive up the Opus quality to silly levels). I don't think there would be a point in using, say, FLAC to stay lossless when the resulting latency is not an improvement over OPUS. I think it's either (lossy) OPUS compression or uncompressed. Stuff in between does not seem to help a lot.
There are a number of projects that take a P2P approach as well as hybrid I believe: https://en.wikipedia.org/wiki/Comparison_of_Remote_Music_Performance_Software
Reminder: Please post new threads on GitHub Discussions. We will be shutting down SourceForge forums next month.