#4231 PMR 03372,077,724:xCAT needs to support new ssh ECDSA host key.

[Lissa Valletta]

Need to investigate the support of new levels of openssh which generate a ECDSA hostkey.
This new key affects our remoteshell setup of the nodes with command hostkeys. Currently we do not generate and transfer a ECDSA key.

Service Saul
Customer LZ

[Lissa Valletta]

I find that openssh will use the rsa key in the known hosts file and not add the ecdsa key.
This is on my MN (service-02) and node (service-07)

First regenerate you hostkeys on the MN just to be safe

xcatconfig -s on the MN.

I then distributed the new keys to the node

updatenode service-07 -k

I then removed /root/.ssh/known_hosts on the MN
and ran the following and it added ECDSA to the known_hosts

[root@service-02 .ssh]# xdsh service-07 date
service-07: Thu Jul 17 08:43:48 PDT 2014
service-07: Warning: Permanently added 'service-07,9.114.34.52' (ECDSA) to the list of known hosts.

I then removed the /root/.ssh/known_hosts again. If you do not want to remove it make sure you at least run makeknownhosts <nodename> -r to get rid of all entries for the node name</nodename>

I then ran
makeknownhosts service-07 to generate an rsa entry for service-07.

[root@service-02 .ssh]# makeknownhosts service-07
[root@service-02 .ssh]# cat known_hosts
service-07,service-07.clusters.com,9.114.34.52 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFUNjaTT9l2+HYF8XcgiA/32jHWMav4Qlkb39Ba47xdpBRKfzdzXFqCzkT3N7WN/k1IqlMNH7i1coDEMSNXW2g8WtVJ1XxfNNypyqHVFZ3RFxkHrSbL3JUTeSuvUE+QFwgiQCGGm6DEkgGq1eHL1yDy4TKEnOixa0w8anWyUUMgD9hFhVSp4FQ2n3dJud/OhKOPbZXT0G8KlCwfgAzIS2yP85qqkLLZxTqzqjGM2k7TLgVANlr29tvq9UZBO3ildXEOI05VDKApyClkhiMDJievw3gvMyYD9h60TM4e/ehlE/ys3If0aDmz9BIBhmRVrnGMayBPfYQmvALvh5qsvT5

When I ran the ssh command it did not add the ECDSA file, it accepted the rsa key, which is sort of what I would expect.
[root@service-02 .ssh]# xdsh service-07 date
service-07: Thu Jul 17 08:48:43 PDT 2014

[root@service-02 .ssh]# ssh -v service-07 date
OpenSSH_6.4, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 50: Applying options for *
debug1: Connecting to service-07 [9.114.34.52] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.4
debug1: match: OpenSSH_6.4 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: RSA a9:04:da:6d:d6🆎f8:a7:ae:04:7d:19:30:41:99:8e
debug1: Host 'service-07' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1

.
.
..
.

[Lissa Valletta]

Decided we do need to generate and transfer the new ssh host key. For diskless install, there are two many scenaios where it will cause problems if we do not. I will implement for 2.8.5
xcatconfig -s
remoteshell
credentials.pm
It will be generated if the MN supports the new ECDSA key and transfered to the node, if there is an /etc/ssh/ecdsa key on the node. There is the scenaio that the MN may have an ssh level that does not support ecdsa, but the node does. In that case a warning will be give that we cannot update the ecdsa key on the node. In that case they can still use the rsa key, but maintenance of the known_hosts file is a little more difficult. I think the usual case is the MN and SN are at OS level equal to or greater that the nodes. But this in not required.

[Lissa Valletta]

2.8.5
commit b47d73efec1e5e27e78537f3b28f9a201de00016
commit 616b6fbc6de175c80f6be8d95ea5f57bc4a8050b
commit b42741e5a15e3f1aa89455c20e7d17323e750eda

2.9
commit 946664e1a10d97b9cdc75b239f9caf03240c2162
commit 22d0dd5eb779f8f49510199d62f4be797b9de07a
commit ba99e7cef1b5ecc3bc882eac1f7496e8d4ebc9e1

[Lissa Valletta]

2.8.5
aixremoteshell
commit 5ad5eb1b1eca2787f3f7a1dc66d8f781d9ef6a15
2.9
commit dad35bb816b14cde64b782718e13ed9666437814

[Lissa Valletta]

Three new files attached here must be added to the system. You should be at xCAT 2.8.4 to apply these files. Best to do both the Management Node and Service nodes.
/opt/xcat/lib/perl/xCAT_plugin/credentials.pm
/opt/xcat/sbin/xcatconfig
/install/postscripts/remoteshell

Backup the original versions.
Replace with the new files.
service xcatd restart/reload

I would remove /root/.ssh/know_hosts on the Management Node and Service Nodes to start fresh.

To generate the new ecdsa host key on the Management Node run
xcatconfig -s

Distribute to service nodes first
updatenode <servicenodes> -k or updatenode <servicenodes> -P remoteshell</servicenodes></servicenodes>

The distribute to your nodes
updatenode <computenodes> -P remoteshell</computenodes>

[Lissa Valletta]

Checking the PMR it is still open. Service provided a the patch but no indication they have tried it. This needs to stay pending until the PMR is closed.