STAF communication error between servers with IPV4_IPV6 and IPV4 protocol

[Titus]

I just removed the initial post and simplified my question because i did not get an answer for 6 months.

I have two linux servers:
- Server1 with ipv4 interfaces
- Server2 with ipv4_ipv6 interfaces

From server1: staf server2 ping ping -> works.
From server2: staf server1 ping ping -> does not work.

Note that the command is using the hostname, and the DNS will first return the IPv6 address, then the IPv4 one.

When using a hostname, does STAF know to fallback to ipv4 address if ipv6 address is not reachable?

In other words does STAF know to do this?

[root@tv-145 ~]# ssh -vvv tv-355.ta.opsware.com
...
debug1: Connecting to tv-355.ta.opsware.com [fc00:132:1::224] port     22.
debug1: connect to address fc00:132:1::224 port 22: No route to host
debug1: Connecting to tv-355.ta.opsware.com [192.168.178.224] port 22.
debug1: Connection established.

[Titus]

Edit my initial question.

[Sharon Lucas]

Yes, when the STAF TCP Connection Provider is working in a IPv4 and IPv6 mixed mode, the code appears to try to connect via IPv6 first and then tries to connect via IPv4. But, there may be a bug in how it is handling an error when trying to connect via IPv6 which may be preventing it from then trying to connect via IPv4. What is the complete result (error message) that you get when server2 runs "staf server1 ping ping"?

Also, have you tried specifying ip4_ip6 for the PROTOCOL value on server1 (so that both your systems are configured using the ip4_ip6 protocol) or do you get an error when trying to do this? If you can specify ip4_ip6 for the PROTOCOL value on both server1 and server2, do you get the same or different result then when server 2 runs "staf server1 ping ping"?

[Titus]

For the purpose of these tests, I have installed the latest version of STAF 3.4.19. The majority of the servers used are 3.4.15.

I'm getting this error when pinging from IPv4_IPv6 to IPv4 :

[root@tv-356 ~]# staf tv-183.ta.opsware.com ping ping
Error submitting request, RC: 16
Additional info
---------------
STAFConnectionProviderConnect: Error performing test read on connected endpoint: recv() RC=113: 22, Endpoint: tcp://tv-183.ta.opsware.com

The log reported by the trace is:

20141024-10:05:31;139998813980416;00000001;Remote ping Service     Request - To: tv-183.ta.opsware.com, from Handle: 6, Process: STAF/Client, Request: ping
20141024-10:05:35;139998813980416;00000400;STAFConnectionManager::makeConnection - Attempt #1 of 2 (Delay 383 milliseconds), RC: 16, Result: STAFConnectionProviderConnect: Error performing test read on connected endpoint: recv() RC=113: 22, Endpoint: tcp://tv-183.ta.opsware.com
20141024-10:05:38;139998813980416;00000400;STAFConnectionManager::makeConnection - Attempt #2 of 2 (Delay 0 milliseconds), RC: 16, Result: STAFConnectionProviderConnect: Error performing test read on connected endpoint: recv() RC=113: 22, Endpoint: tcp://tv-183.ta.opsware.com
20141024-10:05:38;139998813980416;00000002;Remote ping Service Result (16) - To: tv-183.ta.opsware.com, from Handle: 6, Process: STAF/Client, Request: ping, Result: STAFConnectionProviderConnect: Error performing test read on connected endpoint: recv() RC=113: 22, Endpoint: tcp://tv-183.ta.opsware.com
20141024-10:05:38;139998813980416;00000200;Unregistered Handle: 6, PID: 4402

Server1 is ATM installed without -DUSE_TCP_VERSION=IPV4_IPV6 option. Following your suggestion I installed STAF with IPv6 support on this server. Note that the server has default IPv6 settings (link and global), so no manual configuration for ipv6 address and is not ping6 reachable from other systems.
The result is the same:

[root@tv-356 staf]# staf tv-183.ta.opsware.com ping ping
Error submitting request, RC: 16
Additional info
---------------
STAFConnectionProviderConnect: Error performing test read on connected endpoint:     recv() RC=113: 22, Endpoint: tcp://tv-183.ta.opsware.com

On other servers with ipv4 only, I'm not able to start STAF with ipv4_ipv6 protocol, because ipv6 might be disabled from the system and then I get the error bellow, but I assume this works as expected:

Error starting ssl interface
Error code: 10
Reason    : STAFConnectionProviderStart: No socket available: IPv6, socket() RC=97

The workaround I had in place for server2, until ssl certificates expired on the 17th, was:

interface ssl library STAFTCP option Secure=Yes option Port=6550 option Protocol=IPv4
interface tcp library STAFTCP option Secure=No  option Port=6500 option Protocol=IPv4_IPv6

SSL was used to communicate with IPv4 only servers and tcp with IPv6 ones. I could again workaround the ssl problem with another interface, but it seems to me all this communication should be internally handled.

Thanks,
-Titus

[Sharon Lucas]

Yes, I agree that the STAF TCP/IP Connection Provider should handle this internally. However, there appears to be a bug. I'll see if I can set up a similar environment and debug this. If I can't, I may need to provide you with a debug version of STAF. Are you using 32-bit or 64-bit Linux? Also, it may take me some time until I can get to this.

Note that as another workaround you could define another non-secure tcp interface in your STAF.cfg files to use for IPv4 communication instead of using the ssl interface to avoid the SSL certificate issue. For example:

interface tcp2 library STAFTCP option Secure=No option Port=6501 option Protocol=IPv4

But, I agree that this should not be needed and that the STAF TCP/IP Connection Provider should be handling this. It just may take some time until I have a fix for this issue.

Can you open a bug, or let me know and I'll open one for you.

[Titus]

The servers tried here here are x86_64, but I'm able to reproduce it also on x86 or other unix architectures. This was the installer used: STAF3419-setup-linux-amd64.bin.

If you don't have the environment, you can send me a debug version and I can send you back all the info. Happy to help.

In my original post, I've stated this type of communication works if the IPv4 machine runs windows, but I need to get back to you on this one.

Ticket #1533 created.

Thanks,
-Titus