#3559 RHEL 6.4 NFS based statelite (on both ppc64 and x86_64) issue - tmpfs files can't be written to

[yangsong]

this defect is committed in the xcat-user list.

The defect is caused by a rhels6.4 bug. I have opened a bug to LTC:

https://bugzilla.linux.ibm.com/show_bug.cgi?id=93048

There is no work around yet, so I will add a restriction in xcat 2.8.1 release note.

---

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi there,

After more playing with statelite I've got an issue with the NFS based
version that doesn't occur with the ramdisk version.

xCAT: Version 2.8 (svn r15205, built Mon Feb 18 17:01:27 EST 2013)

RHEL 6.4 with updates on management node, node built from straight 6.4.

On boot of an NFS statelite I see (for example):

Tue Apr 30 10:01:36 EDT 2013 Running postscript: syslog
./syslog: line 78: /etc/rsyslog.conf.XCATORIG: Read-only file system
./syslog: line 87: /etc/rsyslog.conf: Read-only file system
./syslog: line 105: /etc/rsyslog.conf: Read-only file system
./syslog: line 107: /etc/rsyslog.conf: Read-only file system
./syslog: line 108: /etc/rsyslog.conf: Read-only file system

If I use the RAMDISK version instead by doing:

[root@barcoo-m ~]# chdef -t osimage -o rhel64nophi rootfstype=ramdisk

[root@barcoo-m ~]# genimage rhel64nophi && liteimg rhel64nophi && nodeset nophi osimage=rhel64nophi && rpower barcoo001 reset

then the node boots fine and the postscripts work:

Tue Apr 30 10:09:06 EDT 2013 Running postscript: syslog
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Postscript: syslog exited with code 0

If I investigate further:

[root@barcoo-m ~]# chdef -t osimage -o rhel64nophi rootfstype=nfs
[root@barcoo-m ~]# genimage rhel64nophi && liteimg rhel64nophi && nodeset nophi osimage=rhel64nophi && rpower barcoo001 reset

I see that the tmpfs mounts occur as defined in the litefile table
(taken from the xCAT statelite wiki page):

[root@barcoo-m ~]# tabdump litefile

image,file,options,comments,disable

"ALL","/etc/adjtime","tmpfs",,
"ALL","/etc/securetty","tmpfs",,
"ALL","/etc/lvm/","tmpfs",,
"ALL","/etc/ntp.conf","tmpfs",,
"ALL","/etc/rsyslog.conf","tmpfs",,
"ALL","/etc/rsyslog.conf.XCATORIG","tmpfs",,
"ALL","/etc/udev/","tmpfs",,
"ALL","/etc/ntp.conf.predhclient","tmpfs",,
"ALL","/etc/resolv.conf","tmpfs",,
"ALL","/etc/yp.conf","tmpfs",,
"ALL","/etc/resolv.conf.predhclient","tmpfs",,
"ALL","/etc/sysconfig/","tmpfs",,
"ALL","/etc/ssh/","tmpfs",,
"ALL","/etc/inittab","tmpfs",,
"ALL","/tmp/","tmpfs",,
"ALL","/var/","tmpfs",,
"ALL","/opt/xcat/","tmpfs",,
"ALL","/xcatpost/","tmpfs",,
"ALL","/root/.ssh/","tmpfs",,

[root@barcoo001 ~]# mount
rootfs on / type rootfs (rw)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=132316976k,nr_inodes=33079244,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
10.3.0.58:/install/netboot/rhels6/x86_64/compute/rootimg on / type nfs (ro,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.3.0.58,mountvers=3,mountport=55734,mountproto=udp,local_lock=all,addr=10.3.0.58)
none on /selinux type selinuxfs (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=132316976k,nr_inodes=33079244,mode=755)
rw on /.statelite type tmpfs (rw,relatime)
10.3.0.58:/state on /.statelite/persistent type nfs (rw,relatime,vers=3,rsize=32768,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=14,retrans=2,sec=sys,mountaddr=10.3.0.58,mountvers=3,mountport=52253,mountproto=tcp,local_lock=all,addr=10.3.0.58)
rw on /etc/adjtime type tmpfs (rw,relatime)
rw on /etc/inittab type tmpfs (rw,relatime)
rw on /etc/lvm type tmpfs (rw,relatime)
rw on /etc/ntp.conf type tmpfs (rw,relatime)
rw on /etc/ntp.conf.predhclient type tmpfs (rw,relatime)
rw on /etc/resolv.conf type tmpfs (rw,relatime)
rw on /etc/resolv.conf.predhclient type tmpfs (rw,relatime)
rw on /etc/rsyslog.conf type tmpfs (rw,relatime)
rw on /etc/rsyslog.conf.XCATORIG type tmpfs (rw,relatime)
rw on /etc/securetty type tmpfs (rw,relatime)
rw on /etc/ssh type tmpfs (rw,relatime)
rw on /etc/sysconfig type tmpfs (rw,relatime)
rw on /etc/udev type tmpfs (rw,relatime)
rw on /etc/yp.conf type tmpfs (rw,relatime)
rw on /opt/xcat type tmpfs (rw,relatime)
rw on /root/.ssh type tmpfs (rw,relatime)
rw on /tmp type tmpfs (rw,relatime)
rw on /var type tmpfs (rw,relatime)
rw on /xcatpost type tmpfs (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=132316976k,nr_inodes=33079244,mode=755)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
/proc/bus/usb on /proc/bus/usb type usbfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime,size=10240k)
tmpfs on /var/tmp type tmpfs (rw,relatime,size=10240k)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)

The issue is that those rw mounts that are files are not writeable
whereas directories are.

[root@barcoo001 ~]# cat /etc/yp.conf
[root@barcoo001 ~]# echo foo > /etc/yp.conf

• -bash: /etc/yp.conf: Read-only file system

[root@barcoo001 ~]# ls /etc/ssh
moduli sshd_config ssh_host_dsa_key ssh_host_key ssh_host_rsa_key
ssh_config sshd_config.ORIG ssh_host_dsa_key.pub ssh_host_key.pub ssh_host_rsa_key.pub

[root@barcoo001 ~]# echo bar > /etc/ssh/foo
[root@barcoo001 ~]# cat /etc/ssh/foo
bar
[root@barcoo001 ~]# rm /etc/ssh/foo

When booted as a ramdisk image the fstab looks almost identical, except
(of course) that the root filesystem is a tmpfs one which is read/write
rather than NFS which is mounted read-only.

Now to me it looks like an NFS root is the default option for statelite
and so I'd assume it's pretty widely used so I'm wondering if the
functionality it relies on has been broken in a recent RHEL release and
others haven't upgraded to it yet?

All the best,
Chris

[Chris Samuel]

Hi there,

Thanks for looking into this - given that the BG/Q also uses bind mounts in RHEL 6.3 for its ionodes it might be useful to give the BG/Q developers a heads up as they are looking to implement support for RHEL 6.4..

The URL you've got seems to be NXDOMAIN (i.e not resolvable) - is that a private bugzilla?

All the best,
Chris

[yangsong]

According to the feedback from LTC, Patch(es) are available on kernel-2.6.32-375.el6. however,the latest RHEL 6.4 kernel I could find on both FTP3 (https://ftp3.linux.ibm.com/files/redhat/yum/6/server/updates/ppc/RPMS/) and RHN is kernel-2.6.32-358.6.2.el6.ppc64.rpm.

[Chris Samuel]

I have a suspicion that kernel-2.6.32-375.el6 will be the base kernel for RHEL 6.5.

[yangsong]

I verified that this defect had been fixed in kernel-2.6.32-431.el6.ppc64.rpm shipped in rhels6.5.

Can we simply use kernel-2.6.32-431.el6.ppc64.rpm as a fix for rhels6.4?

[yangsong]

Since kernel-2.6.32-431.el6.ppc64.rpm shipped in rhels6.5 is a kernel package, simply updating kernel rpm in rhels6.4 with it might introduce some other problems.

I will add an limitation in xCAT 2.8.4 release notes and recommend user to use rhels6.5 instead if possible.