If you want to allow only a IP whitelist to access to your jamulus private server, you can do it with a firewall.

A simple firewall for Jamulus OS/Ubuntu is ufw with a nice GUI named gufw.

sudo apt install gufw

Deny access to your computer and for each IP address (musician IPv4 addresses) add a rule to allow input on 22124/UDP.