I’ve spent a great deal of time trying to figure out why many US users can never see the server list for ‘All Genres” from the connect Dialog, and why when we do it can disappear for minute or hours. I believe it’s because the PROTMESSID_CLM_SERVER_LIST getting sent from Germany to the US is fragmented into too many UDP packets. The routers along the way (at least 5 hops for me), are NOT obligated to pass along these fragmented packets. The more Servers on the list the greater the odds that we will not be able to get a whole PROTMESSID_CLM_SERVER_LIST intact. Having been a communications programmer for the last (let just say many) years, here are some thoughts:
1. A Compressed SERVER LIST message.
From the code here’s the PROTMESSID_CLM_SERVER_LIST:
4/* IP address */+2/* port number */+2/* country */+1/* maximum number of connected clients */+1/* is permanent flag */+2/* name utf-8 string size */+namesizeaMAXof202/* empty string */+2/* city utf-8 string size */+citysizeaMAXof20
The PROTMESSID_CLM_SERVER_LIST message has a fixed length per server of
16 + up to 20 chars for the name and 20 for the city. So for the All Genres server list which averages 110-150 servers, assuming an average name size of 12 and an average city size of 8, you get 36 * 110 = 3960 to 36 * 150 = 5400 bytes per UDP message thus ensuring message fragmentation.
What if we reduced the message to its essentials? Since the client limits name and city size entries to 20 bytes, why not take advantage of that? Since the permanent flag is only 1 bit and the max clients field allows up to 255, why not limit the max clients to 128 and use the high order bit as the permanent flag. I propose a compressed server format that looks like this:
4/* IP address */+2/* port number */+2/* country */+1/* permanent plus max connected clients */1/* name utf-8 string size */+namesizeaMAXof201/* city utf-8 string size */+citysizeaMAXof20
This reduces the fixed message size to 11 and will save (5110) = 560 to (5150) = 750 bytes per message.
Or if you changed the max city name and name sizes to 16 and packed the lengths into 1 byte you’d save another byte.
On most systems this should eliminate 1 UDP fragment per message. It’s easy and just requires two new messages:
define PROTMESSID_CLM_REQ_COM_COMP_SERVER_LIST 1018 // request compressed server list
define PROTMESSID_CLM_REQ_SERVER_LIST 1019 // compressed server list
2. Even More Compression:
I thought about compressing the Unicode text using the standard Unicode compression algorithm (https://www.unicode.org/reports/tr6/tr6-4.html) but that really won’t save that much for most small texts. Other types of compression will use far less data and could potentially fit in 1 UDP message (1460 bytes), but would require a few more additional messages and more CPU time on the central server to process. While I haven’t designed the actual scheme, the outline would be that the server list message would provide only indices into a server table, and the clients would have to request server information in the same manner as the PROTMESSID_CLM_PING_MS_WITHNUMCLIENTS message. The downside is that the connect dialog would take at least a minute to load and the central server would be sending two orders of magnitude more messages.
3. Have More Central Servers:
Many of the permanent servers on the ‘All Genres’ server list are ALWAYS there.
Have Permanent Servers query the main Central Server once per minute for a current server list and keep it.
Add them as “backup servers” to your Central server list in the client, or publish a list (on the website).
The client could use this server list if it could not get to the one in Germany for a minute. This would reduce the load on the Central server because it wouldn’t have to respond to endless requests for connect messages that will not arrive intact.
If I were the programmer I’d go with solution #3 because that’s a pretty standard way of doing things in the communications world. Companies have multiple redundant servers to deal with just such problems as well as outages. These are just my thoughts.
❤️
1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I’ve spent a great deal of time trying to figure out why many US users can never see the server list for ‘All Genres” from the connect Dialog, and why when we do it can disappear for minute or hours. I believe it’s because the PROTMESSID_CLM_SERVER_LIST getting sent from Germany to the US is fragmented into too many UDP packets. The routers along the way (at least 5 hops for me), are NOT obligated to pass along these fragmented packets. The more Servers on the list the greater the odds that we will not be able to get a whole PROTMESSID_CLM_SERVER_LIST intact. Having been a communications programmer for the last (let just say many) years, here are some thoughts:
1. A Compressed SERVER LIST message.
From the code here’s the PROTMESSID_CLM_SERVER_LIST:
The PROTMESSID_CLM_SERVER_LIST message has a fixed length per server of
16 + up to 20 chars for the name and 20 for the city. So for the All Genres server list which averages 110-150 servers, assuming an average name size of 12 and an average city size of 8, you get 36 * 110 = 3960 to 36 * 150 = 5400 bytes per UDP message thus ensuring message fragmentation.
What if we reduced the message to its essentials? Since the client limits name and city size entries to 20 bytes, why not take advantage of that? Since the permanent flag is only 1 bit and the max clients field allows up to 255, why not limit the max clients to 128 and use the high order bit as the permanent flag. I propose a compressed server format that looks like this:
This reduces the fixed message size to 11 and will save (5110) = 560 to (5150) = 750 bytes per message.
Or if you changed the max city name and name sizes to 16 and packed the lengths into 1 byte you’d save another byte.
On most systems this should eliminate 1 UDP fragment per message. It’s easy and just requires two new messages:
define PROTMESSID_CLM_REQ_COM_COMP_SERVER_LIST 1018 // request compressed server list
define PROTMESSID_CLM_REQ_SERVER_LIST 1019 // compressed server list
2. Even More Compression:
I thought about compressing the Unicode text using the standard Unicode compression algorithm (https://www.unicode.org/reports/tr6/tr6-4.html) but that really won’t save that much for most small texts. Other types of compression will use far less data and could potentially fit in 1 UDP message (1460 bytes), but would require a few more additional messages and more CPU time on the central server to process. While I haven’t designed the actual scheme, the outline would be that the server list message would provide only indices into a server table, and the clients would have to request server information in the same manner as the PROTMESSID_CLM_PING_MS_WITHNUMCLIENTS message. The downside is that the connect dialog would take at least a minute to load and the central server would be sending two orders of magnitude more messages.
3. Have More Central Servers:
Many of the permanent servers on the ‘All Genres’ server list are ALWAYS there.
Have Permanent Servers query the main Central Server once per minute for a current server list and keep it.
Add them as “backup servers” to your Central server list in the client, or publish a list (on the website).
The client could use this server list if it could not get to the one in Germany for a minute. This would reduce the load on the Central server because it wouldn’t have to respond to endless requests for connect messages that will not arrive intact.
If I were the programmer I’d go with solution #3 because that’s a pretty standard way of doing things in the communications world. Companies have multiple redundant servers to deal with just such problems as well as outages. These are just my thoughts.
wouldnt a better solution be to use TCP/IP for these types of data?
Hi Bart - you may want to raise this (as one or several separate) issues on Github I think:
https://github.com/corrados/jamulus/issues
Engineering stuff tends to gravitate over there.