Hello all. I'm running a Jamulus server (3.5.9git, latest commit a708a53c)
on a Raspberry pi. I've successfully managed to register it to the
Classical/Folk/Choir Central Server.
In my router I have ports 22124 and 22125 forwarded to the Raspberry. If I
start the server on these ports, I can see the server in the Jamulus client
running on a different network, and connect to it. So far, so good.
However, if I start the server on port 22126 (which is not forwarded), it
registers to the central server (I can see it in http://jamulus.softins.co.uk/), but it doesn't show in the Jamulus client
running on a different network (I can't try in the same network right now,
but my guess is it would work there).
If I run the client with the --showallservers option, then my server does
show. But when I try to connect to it, it stays "trying to connect" forever.
I have a tp-link Archer C20 router with upnp enabled. Any idea of what
could be happening? I don't need this for myself because I can forward the
ports, but I need it for some colleagues of mine who can't do it (or who
find it too complicated).
I'll be mentioning @Gronaz, @Gilgongo and @corrados as I've seen in other posts you
may know what might be going on or help me diagnose it.
BTW, server name is "rpi" in "Ramos Mejía, Argentina". I will try leave it running so you can help me :)
Thanks!!
Last edit: Diego de la Hera 2020-08-11
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Edit: This connection succeeds even if I don't have the server running. It does not succeed however -with the server turned off- on ports 22124 and 22125, which are forwarded in the router (it succeeds only if I have the server listening to those ports).
Last edit: Diego de la Hera 2020-08-11
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Firstly, if you always intend to have your server registered to a central server, you don't need any port forwarding, due to the magic I described to you on github. But if you want your server to be private and unregistered, so that you just give the IP and port number to your friends to connect, then you do need the port forwarding, as the magic doesn't happen.
The Jamulus Explorer back-end behaves just like a Jamulus client when it fetches the server list and the details of each server. If it is unable to contact a particular server, the ping time, version and OS fields will be blank. In the case of your rpi, it can see that rpi registered with a source port of 22126, and it can successfully fetch the info from the rpi. That is because of the magic: the central server tells your rpi to send a message to the Jamulus Explorer back-end, which means your router will then allow requests from the back-end on that port.
So if your Jamulus client on another network can't see the rpi server, it has a restriction that the Explorer back-end does not. How "other" is the network of this client? Is it completely elsewhere, or on a different lan behind the same router?
I'm not sure what nc means by a connection to a UDP port succeeding, because UDP is connectionless. And with -z, it doesn't send any data, so I'm not sure what it actually does. Maybe it sends a zero length packet and then waits a short while to see if it receives an ICMP Port Unreachable?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
tl;dr: public servers behind NATs seem to be unreachable from clients behind NATs which randomize source port numbers.
To recap:
- Jamulus server is running on computer A. Computer A is behind an NAT with ports 22124 and 22125 forwarded (but not 22126). The WAN-facing side of the NAT has a public WAN IP. Jamulus server successfully registers itself to the "Classical" genre central server, regardless of what port (22124-22126) it is set to listen to.
- Jamulus client is running on computer B. Computer B is also behind an NAT. The WAN side of the NAT does not have a public WAN IP. The client can see the server at computer A if it's listening to forwarded ports 22124 or 22125, but not if it's listening to non-forwarded port 22126 (it can see it if the --showallservers option is used, but it can't connect to it).
The key to solve this problem came from Tony Mountifield's response to a seemingly unrelated question here. I noticed that if I run a server on computer B instead, it shows up with a "port2" in Tony's list. This means that the server on computer B registers itself with a port number, but when Tony's site tries to retrieve information from it, it uses another port. Apparently, the NAT behind which computer B is, is randomising source port numbers.
So, back to server on computer A and client on computer B:
1. Client at computer B calls the central server. The NAT provides a randomised source port, say X.
2. The central server provides the server in computer A with computer B's IP number, and port X.
3. Server at computer A sends a ping to computer B's IP number port X. This should open the door at computer A's NAT for incoming packets from computer B.
4. Computer B tries to reach server at computer A, but the NAT uses another randomised source port, say Y.
5. Computer A's NAT gets the packet from computer B's NAT, but source port Y doesn't match source port X. Therefore, the NAT blocks the packet.
I've pretty much confirmed this is the case using tcpdump in computer B and an independent central server.
Unfortunately, it doesn't seem to be possible to disable source port randomization at computer B's NAT. I don't even know if maybe that's imposed by the ISP itself (ISP is Telecentro in Buenos Aires, Argentina).
As far as I know, only solution in these cases seems to be forwarding the port at the server's NAT.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Good detective work. The only bit I wanted clarification on is "The WAN side of [computer B's] NAT does not have a public WAN IP." Do you mean the WAN side has another private address, such as 10.x.x.x or 172.16.x.x, etc.? And when you run a server on computer B, and view it in Explorer, what IP does it show? That must be a public IP. You probably also find that both the port and port2 values are different from the port assigned to the Jamulus server on computer B. My guess is that computer B's ISP is actually doing carrier NAT for all its subscribers, so that many of them share a public IP, and therefore will always get allocated random source ports.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Do you mean the WAN side has another private address, such as 10.x.x.x or 172.16.x.x, etc.?
Yes.
And when you run a server on computer B, and view it in Explorer, what IP does it show?
The carrier's NAT shared public IP.
You probably also find that both the port and port2 values are different from the port assigned to the Jamulus server on computer B.
Exactly!
My guess is that computer B's ISP is actually doing carrier NAT for all its subscribers
It is indeed :(
and therefore will always get allocated random source ports.
I didn't know carrier NAT came with this.
So, generally speaking, would it be accurate to say that public servers behind NATs, listening to non-forwarded ports, are unreachable from clients behind carrier-grade NAT?
Maybe that's related to what it says here, that "being behind a Carrier-grade NAT may mean not being able to see server lists in the client".
Thanks Tony for helping me figure this out!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I can't think of a reason for a client not to work when it is behind double-NAT (e.g. CGN or a router behind a router). It should be able to connect to a server that is elsewhere, even if that server is behind a single NAT (without port randomisation). But being behind double-NAT would definitely make it difficult or impossible to run a server locally and have it reachable from clients elsewhere. It may be possible for such a server to register with a central server, but the other clients will probably be unable to connect to it.
Last edit: Tony Mountifield 2020-08-11
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I can't think of a reason for a client not to work when it is behind double-NAT (e.g. CGN or a router behind a router). It should be able to connect to a server that is elsewhere, even if that server is behind a single NAT (without port randomisation).
But that's exactly what seems to be failing here. As described in my previous message:
The client behind a CGN calls the central server.
The central server gets the call from cgn-ip : random-port-x, and sends this info to the servers registered at this central server.
Servers send a ping to cgn-ip : random-port-x. If the server is behind a NAT, this would allow packets coming back from cgn-ip : random-port-x to go through.
The original client tries to reach one of the servers retrieved from the central server. Packets originate from cgn-ip : random-port-y.
a. If the server is not behind an NAT, connection succeeds.
b. If the server is behind an NAT (my case) connection fails, because cgn-ip : random-port-x does not match cgn-ip : random-port-y
I'm not a network expert. Does this make any sense?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, you're absolutely right. I hadn't thought through that particular scenario completely, and your explanation makes perfect sense.
So if the server is behind either a NAT, or a non-translating firewall that only allows inbound packets for established sessions, a client with randomised source ports will only be able to connect if that server also has the inbound port directly enabled and forwarded.
If a server is behind a firewall that randomises source ports, it's even worse, as even if it has a port forwarding enabled, the port registered on the central server will be the randomised port, not the one the server is really listening on. So no-one else will be able to connect, as they won't know the real listening port. I don't think a regular Jamulus client, particularly behind a NAT or firewall, has the same "port2" detection that Jamulus Explorer has.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello all. I'm running a Jamulus server (3.5.9git, latest commit a708a53c)
on a Raspberry pi. I've successfully managed to register it to the
Classical/Folk/Choir Central Server.
In my router I have ports 22124 and 22125 forwarded to the Raspberry. If I
start the server on these ports, I can see the server in the Jamulus client
running on a different network, and connect to it. So far, so good.
However, if I start the server on port 22126 (which is not forwarded), it
registers to the central server (I can see it in
http://jamulus.softins.co.uk/), but it doesn't show in the Jamulus client
running on a different network (I can't try in the same network right now,
but my guess is it would work there).
If I run the client with the --showallservers option, then my server does
show. But when I try to connect to it, it stays "trying to connect" forever.
I have a tp-link Archer C20 router with upnp enabled. Any idea of what
could be happening? I don't need this for myself because I can forward the
ports, but I need it for some colleagues of mine who can't do it (or who
find it too complicated).
I'll be mentioning @Gronaz, @Gilgongo and @corrados as I've seen in other posts you
may know what might be going on or help me diagnose it.
BTW, server name is "rpi" in "Ramos Mejía, Argentina". I will try leave it running so you can help me :)
Thanks!!
Last edit: Diego de la Hera 2020-08-11
Nothing in Jamulus makes any use of UPNP, so it doesn't matter for Jamulus whether UPNP is enabled or not.
I guess it may be worth noting that the 22126 UDP port that the server is listening to is somehow reachable from the outside:
Edit: This connection succeeds even if I don't have the server running. It does not succeed however -with the server turned off- on ports 22124 and 22125, which are forwarded in the router (it succeeds only if I have the server listening to those ports).
Last edit: Diego de la Hera 2020-08-11
Hi Diego, softins here...
Firstly, if you always intend to have your server registered to a central server, you don't need any port forwarding, due to the magic I described to you on github. But if you want your server to be private and unregistered, so that you just give the IP and port number to your friends to connect, then you do need the port forwarding, as the magic doesn't happen.
The Jamulus Explorer back-end behaves just like a Jamulus client when it fetches the server list and the details of each server. If it is unable to contact a particular server, the ping time, version and OS fields will be blank. In the case of your rpi, it can see that rpi registered with a source port of 22126, and it can successfully fetch the info from the rpi. That is because of the magic: the central server tells your rpi to send a message to the Jamulus Explorer back-end, which means your router will then allow requests from the back-end on that port.
So if your Jamulus client on another network can't see the rpi server, it has a restriction that the Explorer back-end does not. How "other" is the network of this client? Is it completely elsewhere, or on a different lan behind the same router?
I'm not sure what
ncmeans by a connection to a UDP port succeeding, because UDP is connectionless. And with-z, it doesn't send any data, so I'm not sure what it actually does. Maybe it sends a zero length packet and then waits a short while to see if it receives an ICMP Port Unreachable?I think I may have found the root of the problem.
tl;dr: public servers behind NATs seem to be unreachable from clients behind NATs which randomize source port numbers.
To recap:
- Jamulus server is running on computer A. Computer A is behind an NAT with ports 22124 and 22125 forwarded (but not 22126). The WAN-facing side of the NAT has a public WAN IP. Jamulus server successfully registers itself to the "Classical" genre central server, regardless of what port (22124-22126) it is set to listen to.
- Jamulus client is running on computer B. Computer B is also behind an NAT. The WAN side of the NAT does not have a public WAN IP. The client can see the server at computer A if it's listening to forwarded ports 22124 or 22125, but not if it's listening to non-forwarded port 22126 (it can see it if the --showallservers option is used, but it can't connect to it).
The key to solve this problem came from Tony Mountifield's response to a seemingly unrelated question here. I noticed that if I run a server on computer B instead, it shows up with a "port2" in Tony's list. This means that the server on computer B registers itself with a port number, but when Tony's site tries to retrieve information from it, it uses another port. Apparently, the NAT behind which computer B is, is randomising source port numbers.
So, back to server on computer A and client on computer B:
1. Client at computer B calls the central server. The NAT provides a randomised source port, say X.
2. The central server provides the server in computer A with computer B's IP number, and port X.
3. Server at computer A sends a ping to computer B's IP number port X. This should open the door at computer A's NAT for incoming packets from computer B.
4. Computer B tries to reach server at computer A, but the NAT uses another randomised source port, say Y.
5. Computer A's NAT gets the packet from computer B's NAT, but source port Y doesn't match source port X. Therefore, the NAT blocks the packet.
I've pretty much confirmed this is the case using tcpdump in computer B and an independent central server.
Unfortunately, it doesn't seem to be possible to disable source port randomization at computer B's NAT. I don't even know if maybe that's imposed by the ISP itself (ISP is Telecentro in Buenos Aires, Argentina).
As far as I know, only solution in these cases seems to be forwarding the port at the server's NAT.
Good detective work. The only bit I wanted clarification on is "The WAN side of [computer B's] NAT does not have a public WAN IP." Do you mean the WAN side has another private address, such as 10.x.x.x or 172.16.x.x, etc.? And when you run a server on computer B, and view it in Explorer, what IP does it show? That must be a public IP. You probably also find that both the port and port2 values are different from the port assigned to the Jamulus server on computer B. My guess is that computer B's ISP is actually doing carrier NAT for all its subscribers, so that many of them share a public IP, and therefore will always get allocated random source ports.
Yes.
The carrier's NAT shared public IP.
Exactly!
It is indeed :(
I didn't know carrier NAT came with this.
So, generally speaking, would it be accurate to say that public servers behind NATs, listening to non-forwarded ports, are unreachable from clients behind carrier-grade NAT?
Maybe that's related to what it says here, that "being behind a Carrier-grade NAT may mean not being able to see server lists in the client".
Thanks Tony for helping me figure this out!
I can't think of a reason for a client not to work when it is behind double-NAT (e.g. CGN or a router behind a router). It should be able to connect to a server that is elsewhere, even if that server is behind a single NAT (without port randomisation). But being behind double-NAT would definitely make it difficult or impossible to run a server locally and have it reachable from clients elsewhere. It may be possible for such a server to register with a central server, but the other clients will probably be unable to connect to it.
Last edit: Tony Mountifield 2020-08-11
But that's exactly what seems to be failing here. As described in my previous message:
a. If the server is not behind an NAT, connection succeeds.
b. If the server is behind an NAT (my case) connection fails, because cgn-ip : random-port-x does not match cgn-ip : random-port-y
I'm not a network expert. Does this make any sense?
Yes, you're absolutely right. I hadn't thought through that particular scenario completely, and your explanation makes perfect sense.
So if the server is behind either a NAT, or a non-translating firewall that only allows inbound packets for established sessions, a client with randomised source ports will only be able to connect if that server also has the inbound port directly enabled and forwarded.
If a server is behind a firewall that randomises source ports, it's even worse, as even if it has a port forwarding enabled, the port registered on the central server will be the randomised port, not the one the server is really listening on. So no-one else will be able to connect, as they won't know the real listening port. I don't think a regular Jamulus client, particularly behind a NAT or firewall, has the same "port2" detection that Jamulus Explorer has.