Just wanted to share a recent experience using a Raspberry Pi 4 as a server. In the past I've been using AWS to host our Jamulus jams, but I wanted to try hosting a server at home since I have fiber internet.
I set up my Raspberry Pi 4 (4 GB) with standard Raspberry Pi OS and built Jamulus from the github source. We did a jam with 8 people and it worked flawlessly. During the jam the CPU usage never went above 17% and was transferring about 6 Mb/s for a few hours.
Nice work everybody :)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello,
I also have a Raspi 4B4G with Debian Buster. There are irregular, occasional hangs of several seconds (periodic repeats of the same short sound packet) with more than 8 people connected via 1Gbit fiber. The average CPU load was around 25% for each of the 4 cores but when the hangs occurred, it went up to above 80% on alternating cores. As to my knowledge, nothing else runs on the RPi, just the Jamulus server, the standard VNC server and the SSH server.
RAM usage was always below 50%.
Might there be any unnecessary system services that I could deactivate?
Thank you in advance for any ideas / experiences.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Might there be any unnecessary system services that I could deactivate?
I believe if you run the systemctl command on it's own then that'll give you a list of services. Use systemctl stop <service> then systemctl disable <service> to stop any running on boot.
Also worth doing systemctl list-timers to check if there are any scheduled things that might be causing load spikes that you could instead do manually.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thank you, I tried systemctl which produced a long list of services that all seem essential to the functioning of the kernel.
However I took a snapshot of the cpu load chart in htop (taskmanager).
This was with 10 connected clients (all on the same but different raspi).
The interesting thing is that the CPU load changes that much and jumps from one core to another.
I didn't expect this behaviour, what I would finf logical is an even distribution over the cores and a smooth graph.
Another interesting thing is that the max value of cpu load for the cores concerned does not change with the number of connected clients down to two clients. Only the density of the "saturated" core slots reduces when closing clients.
Are there any hints on the limit of clients the raspi 4B can safely carry in an optimal installation?
In terms of memory consumption, even the most elementary raspi 4B (2GB) should work, since the RAM usage never exceeded 250MB and seemed fairly independent on the number of clients.
Can this be? Does it make a difference that all connections were made from the same hardware?
There is always the possibility that the Rpi simply isn't able to service the load if you're seeing averages over 1 (on a single core system) on average (Linux load averages are conglomeration of CPU, disk and memory I think). 10 clients seems pretty heavy to me.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Jean,
You didn't say, are you using the -T multithreading command line option?
On my Odroid-N2, which is not that much different from a P4, I see the processor load spread quite evenly over the 4 cores that the multithreading uses (there are 6 cores total).
I am surprised by your results.
Does the P4 have variable CPU-speed ? I see the load of the 4 cores going up to about 40% and then the clock speed is upped a notch. So it looks like the load hardly rises with more clients, but it does as the clock speed rises. At 25 clients I have still not hit full speed (but my router is starting to get hot :) ).
I am running Ubuntu 20.04
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi DonC,
yes I'm using the commandline option -T.
I think otherwise Jamulus would run just on one core?
Yes the Pi must have variable frequency, since the command
sudo cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
gives me 800 when no client connected, 900 when one and 1500 for two clients.
Debian 10 and Jamulus 3.6.1
I will do some more experiments and connect a screen to omit ssh.
On the odroid, why doesn't Jamulus use all six cores?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
On the odroid, why doesn't Jamulus use all six cores?
As I understand it there are 2 low power cores and 4 high performance cores.
They have different architectures.
Jamulus only uses the high performance cores.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think our setups are pretty similar -- I am also using buster and don't have anything else running except for jamulus and telegraf for metrics collection. My systemd startup script is based on https://jamulus.io/wiki/Server-Linux. I am not using multithreaded mode. I did disable the GUI mode of the OS but that's about all the changes I made from a stock install.
Attached are some CPU graphs of cpu mean and each individual cpu. Mean stayed below 15% the individual CPUs peaked at around 30%.
skrul, thank you for this input.
I've been trying to understand the business model of telegraf, they say it's open source, how can I get it onto the RPI?
I use the same startup script (the link doesn't work because of the extra dot).
Sometimes I would like to invoke the server GUI but how can I do it, the server running from the startup script in no gui mode?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello, first of all thank you guys for all the good work on Jamulus, now and in the past, and stay healthy !
We're using Jamulus a few months now .
Installed a linux private server on a 4 core RPI4 with 1,5 Ghz 4GB with the option -T multithreading enabled, and a maximum number of channels 24. No Gui mode chosen.
No other progams then Jamulus run on the server.
I've attached a screenshot of the Jamulus configuration.
The servers got a glass-fiber internet connection 100Mbps up and down.
With this configuration the server can handle 21 clients.
The bigger the group makes audio-quality getting gradually less.
Most of the clients don't live further than 20 miles from the server.
The overall latency depending on the hardware and internetconnections of course varies between 30 and 80 ms.
21 clients seems to be the limit.
Now we were wondering if the CPU is fast enough for the option -F --fastupdate ?
Or would it be to slow for that ?
When first trying to set up a Jamulus server on a Raspi 4, I tried on the ubuntu 64 bit OS. Although Jamulus built and seemed to run OK, after an hour or so, the Raspi totally hung - no console, nothing. Only a power reboot would revive it.
So I rebuilt everything to to run on the 32 bit debian based OS and all seemed to work fine, except for one glitch that happens once or twice a day.
Our setup is as follows: one Raspi 4 attached to internet link A with 4 instances of the server running on it. Another Pi on a separate internet link B running the server as a custom central server.
The servers are started with the following exec statements in the systemctl service files:
Jamulus -s -n -p 22120 -e ourccs.ddns.net -o Jam-Room-1;city;38 -T -w /usr/local/bin/j1.txt
Jamulus -s -n -p 22121 -e ourccs.ddns.net -o Jam-Room-1;city;38 -T -w /usr/local/bin/j2.txt
Jamulus -s -n -p 22122 -e ourccs.ddns.net -o Jam-Room-1;city;38 -T -w /usr/local/bin/j3txt
Jamulus -s -n -p 22123 -e ourccs.ddns.net -o Jam-Room-1;city;38 -T -w /usr/local/bin/j4.txt
The clients are able to access the central server, see which rooms are available or being used by other clients.
Typically the central server log shows a registration request for each jamulus server instance every 15 minutes like this:
Feb 4 03:37:06 ourCCS jamulus[399]: Requested to register entry for xx.yyy.209.81:22121 (192.168.86.24:22121): Jam-Room-2
But periodically there is a session expiry message followed by a change of port number in the log:
Feb 4 04:10:27 ourCCS jamulus[399]: Expired entry for xx.yyy.209.81:22121
Feb 4 04:22:06 ourCCS jamulus[399]: Requested to register entry for xx.yyy.209.81:1061 (192.168.86.24:22121): Jam-Room-2
At this point the Room disappears from the Central Server even though the log on the Server making the registration request shows that it is registered (probably because the changed port numbers are not open on the firewall).
If I restart the server service, it continues to use the incorrect port. I have to stop the service, wait for up to 15 minutes, then start the server service again and the correct port gets used and the Jam Room shows in the custom central server.
When this has occurred, it doesn't happen to all the instances of Jamulus clients, just some, but when it does occur it hits 2 to 3 of the 4 at the same time.
Just wanted to share a recent experience using a Raspberry Pi 4 as a server. In the past I've been using AWS to host our Jamulus jams, but I wanted to try hosting a server at home since I have fiber internet.
I set up my Raspberry Pi 4 (4 GB) with standard Raspberry Pi OS and built Jamulus from the github source. We did a jam with 8 people and it worked flawlessly. During the jam the CPU usage never went above 17% and was transferring about 6 Mb/s for a few hours.
Nice work everybody :)
Hello,
I also have a Raspi 4B4G with Debian Buster. There are irregular, occasional hangs of several seconds (periodic repeats of the same short sound packet) with more than 8 people connected via 1Gbit fiber. The average CPU load was around 25% for each of the 4 cores but when the hangs occurred, it went up to above 80% on alternating cores. As to my knowledge, nothing else runs on the RPi, just the Jamulus server, the standard VNC server and the SSH server.
RAM usage was always below 50%.
Might there be any unnecessary system services that I could deactivate?
Thank you in advance for any ideas / experiences.
I believe if you run the systemctl command on it's own then that'll give you a list of services. Use
systemctl stop <service>thensystemctl disable <service>to stop any running on boot.Also worth doing
systemctl list-timersto check if there are any scheduled things that might be causing load spikes that you could instead do manually.Thank you, I tried systemctl which produced a long list of services that all seem essential to the functioning of the kernel.
However I took a snapshot of the cpu load chart in htop (taskmanager).
This was with 10 connected clients (all on the same but different raspi).
The interesting thing is that the CPU load changes that much and jumps from one core to another.
I didn't expect this behaviour, what I would finf logical is an even distribution over the cores and a smooth graph.
Another interesting thing is that the max value of cpu load for the cores concerned does not change with the number of connected clients down to two clients. Only the density of the "saturated" core slots reduces when closing clients.
Are there any hints on the limit of clients the raspi 4B can safely carry in an optimal installation?
In terms of memory consumption, even the most elementary raspi 4B (2GB) should work, since the RAM usage never exceeded 250MB and seemed fairly independent on the number of clients.
Can this be? Does it make a difference that all connections were made from the same hardware?
There is always the possibility that the Rpi simply isn't able to service the load if you're seeing averages over 1 (on a single core system) on average (Linux load averages are conglomeration of CPU, disk and memory I think). 10 clients seems pretty heavy to me.
... I forgot to mention that I was watching the server over an SSH connection. Could the CPU load of the SSH server be fooling me?
the load average over 1 didn't worry my too much since the raspi has 4 cores...
Oh OK. In which case not a problem. The question then is whether the clients are getting decent audio sessions. If they are, then job done.
Hi Jean,
You didn't say, are you using the -T multithreading command line option?
On my Odroid-N2, which is not that much different from a P4, I see the processor load spread quite evenly over the 4 cores that the multithreading uses (there are 6 cores total).
I am surprised by your results.
Does the P4 have variable CPU-speed ? I see the load of the 4 cores going up to about 40% and then the clock speed is upped a notch. So it looks like the load hardly rises with more clients, but it does as the clock speed rises. At 25 clients I have still not hit full speed (but my router is starting to get hot :) ).
I am running Ubuntu 20.04
Hi DonC,
yes I'm using the commandline option -T.
I think otherwise Jamulus would run just on one core?
Yes the Pi must have variable frequency, since the command
sudo cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
gives me 800 when no client connected, 900 when one and 1500 for two clients.
Debian 10 and Jamulus 3.6.1
I will do some more experiments and connect a screen to omit ssh.
On the odroid, why doesn't Jamulus use all six cores?
On the odroid, why doesn't Jamulus use all six cores?As I understand it there are 2 low power cores and 4 high performance cores.
They have different architectures.
Jamulus only uses the high performance cores.
I think our setups are pretty similar -- I am also using buster and don't have anything else running except for jamulus and telegraf for metrics collection. My systemd startup script is based on https://jamulus.io/wiki/Server-Linux. I am not using multithreaded mode. I did disable the GUI mode of the OS but that's about all the changes I made from a stock install.
Attached are some CPU graphs of cpu mean and each individual cpu. Mean stayed below 15% the individual CPUs peaked at around 30%.
skrul, thank you for this input.
I've been trying to understand the business model of telegraf, they say it's open source, how can I get it onto the RPI?
I use the same startup script (the link doesn't work because of the extra dot).
Sometimes I would like to invoke the server GUI but how can I do it, the server running from the startup script in no gui mode?
Hello, first of all thank you guys for all the good work on Jamulus, now and in the past, and stay healthy !
We're using Jamulus a few months now .
Installed a linux private server on a 4 core RPI4 with 1,5 Ghz 4GB with the option -T multithreading enabled, and a maximum number of channels 24. No Gui mode chosen.
No other progams then Jamulus run on the server.
I've attached a screenshot of the Jamulus configuration.
The servers got a glass-fiber internet connection 100Mbps up and down.
With this configuration the server can handle 21 clients.
The bigger the group makes audio-quality getting gradually less.
Most of the clients don't live further than 20 miles from the server.
The overall latency depending on the hardware and internetconnections of course varies between 30 and 80 ms.
21 clients seems to be the limit.
Now we were wondering if the CPU is fast enough for the option -F --fastupdate ?
Or would it be to slow for that ?
When first trying to set up a Jamulus server on a Raspi 4, I tried on the ubuntu 64 bit OS. Although Jamulus built and seemed to run OK, after an hour or so, the Raspi totally hung - no console, nothing. Only a power reboot would revive it.
So I rebuilt everything to to run on the 32 bit debian based OS and all seemed to work fine, except for one glitch that happens once or twice a day.
Our setup is as follows: one Raspi 4 attached to internet link A with 4 instances of the server running on it. Another Pi on a separate internet link B running the server as a custom central server.
The servers are started with the following exec statements in the systemctl service files:
Jamulus -s -n -p 22120 -e ourccs.ddns.net -o Jam-Room-1;city;38 -T -w /usr/local/bin/j1.txt
Jamulus -s -n -p 22121 -e ourccs.ddns.net -o Jam-Room-1;city;38 -T -w /usr/local/bin/j2.txt
Jamulus -s -n -p 22122 -e ourccs.ddns.net -o Jam-Room-1;city;38 -T -w /usr/local/bin/j3txt
Jamulus -s -n -p 22123 -e ourccs.ddns.net -o Jam-Room-1;city;38 -T -w /usr/local/bin/j4.txt
The clients are able to access the central server, see which rooms are available or being used by other clients.
Typically the central server log shows a registration request for each jamulus server instance every 15 minutes like this:
Feb 4 03:37:06 ourCCS jamulus[399]: Requested to register entry for xx.yyy.209.81:22121 (192.168.86.24:22121): Jam-Room-2
But periodically there is a session expiry message followed by a change of port number in the log:
Feb 4 04:10:27 ourCCS jamulus[399]: Expired entry for xx.yyy.209.81:22121
Feb 4 04:22:06 ourCCS jamulus[399]: Requested to register entry for xx.yyy.209.81:1061 (192.168.86.24:22121): Jam-Room-2
At this point the Room disappears from the Central Server even though the log on the Server making the registration request shows that it is registered (probably because the changed port numbers are not open on the firewall).
If I restart the server service, it continues to use the incorrect port. I have to stop the service, wait for up to 15 minutes, then start the server service again and the correct port gets used and the Jam Room shows in the custom central server.
When this has occurred, it doesn't happen to all the instances of Jamulus clients, just some, but when it does occur it hits 2 to 3 of the 4 at the same time.
Any suggestions appreciated.
Last edit: Bill Colvin 2021-02-05