Menu
▾
▴
#4526 [FVT] Bash script /xcatpost/otherpkgs stuck at apt-get forever while installing ubuntu service node
The management node is a PowerKVM guest, and runs Ubuntu Linux 14.04.1. I was trying to install a diskfull service node with xCAT. The service node is a PowerKVM guest as well.
Below is the detail information about the osimage definition. The file pkglist, otherpkglist, template are all copied from the directory, /opt/xcat/share/xcat/install/ubuntu, and left as the default shape.
root@c910f03c04k07:~# lsdef -t osimage service_1503c
Object name: service_1503c
imagetype=linux
osarch=ppc64el
osname=Linux
osvers=ubuntu14.04.1
otherpkgdir=http://ports.ubuntu.com/ubuntu-ports trusty main,http://ports.ubuntu.com/ubuntu-ports trusty-updates main,http://ports.ubuntu.com/ubuntu-ports trusty universe,http://10.3.4.7/install/post/otherpkgs/ubuntu14.04.1/ppc64el/xcat-core/ trusty main,http://10.3.4.7/install/post/otherpkgs/ubuntu14.04.1/ppc64el/xcat-dep/ trusty main
otherpkglist=/install/custom/install/ubuntu/service.ubuntu.otherpkgs.pkglist
pkgdir=/install/ubuntu14.04.1/ppc64el
pkglist=/install/custom/install/ubuntu/service.pkglist
profile=service
provmethod=install
template=/install/custom/install/ubuntu/service.tmpl
The service is c910f03c04k06. It is the only node in the service group.
root@c910f03c04k07:~# lsdef -t group service
Object name: service
members=c910f03c04k06
postscripts=servicenode
Run the command nodeset.
root@c910f03c04k07:~# nodeset service osimage=service_1503c
This is the node definition of the service node after the nodeset command was run.
root@c910f03c04k07:~# lsdef service
Object name: c910f03c04k06
arch=ppc64el
currchain=boot
currstate=boot
groups=all,service
id=6
initrd=xcat/osimage/service_1503c/initrd.img
installnic=mac
kcmdline=nofb utf8 auto url=http://!myipfn!/install/autoinst/c910f03c04k06 xcatd=!myipfn! mirror/http/hostname=!myipfn! netcfg/choose_interface=42:a7:88:6e:27:ab console=tty0 console=hvc0,115200 locale=en_US priority=critical hostname=c910f03c04k06 live-installer/net-image=http://!myipfn!/install/ubuntu14.04.1/ppc64el/install/filesystem.squashfs
kernel=xcat/osimage/service_1503c/vmlinuz
mac=42:a7:88:6e:27:ab|42:3b:0a:03:04:06!*NOIP*|42:71:0a:03:04:06!*NOIP*
mgt=kvm
netboot=grub2
os=ubuntu14.04.1
postbootscripts=otherpkgs
postscripts=syslog,remoteshell,syncfiles,servicenode
profile=service
provmethod=service_1503c
serialport=0
serialspeed=115200
status=booting
statustime=01-18-2015 22:45:44
updatestatus=failed
updatestatustime=11-14-2014 01:54:39
vmcpus=4
vmhost=c910f03c04
vmmemory=8192
vmnics=brenP3p5s0f0,brenP3p5s0f1,brenP3p5s0f2
vmstorage=phy:/dev/mapper/vdiskvg-vdisk00n06
After the target service node was rebooted and installed, while the postscript was running, it stuck somewhere in the middle. The serial console and VNC remote console remained blank for 24+ hours. But the service node can be accessed with ssh. As you can see below, it stuck in the script otherpkgs. And an apt-get was running.
root@c910f03c04k06:~# ps axfww
PID TTY STAT TIME COMMAND
<<< ... omit ... >>>
1 ? Ss 0:03 /sbin/init
346 ? S 0:00 /sbin/ureadahead --daemon
357 ? S 0:00 @sbin/plymouthd --mode=boot --attach-to-session
362 ? Ss 0:00 plymouth-upstart-bridge
580 ? Ss 0:00 dbus-daemon --system --fork
626 ? Ss 0:00 rpc.idmapd
913 ? Ss 0:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
1048 ? Ss 0:00 /bin/sh -e /proc/self/fd/9
1066 ? S 0:00 \_ /bin/sh /etc/init.d/rc 2
1165 ? S 0:00 \_ /bin/sh /etc/rc2.d/S84xcatpostinit1 start
1174 ? S 0:00 \_ /bin/bash /opt/xcat/xcatinstallpost
1210 ? S 0:00 \_ /bin/bash /opt/xcat/xcatinstallpost
1222 ? S 0:00 \_ /bin/bash ./otherpkgs
13447 ? S 0:00 | \_ /bin/bash ./otherpkgs
13448 ? S 0:47 | \_ apt-get -q -y --force-yes install mariadb-client mariadb-common mariadb-server xcatsn conserver-xcat
16996 ? S 0:00 | \_ /usr/bin/dpkg --status-fd 19 --configure libaio1:ppc64el libapr1:ppc64el libaprutil1:ppc64el liblua5.2-0:ppc64el libmariadbclient18:ppc64el libtcl8.6:ppc64el mariadb-client-core-5.5:ppc64el mariadb-client-5.5:ppc64el libjemalloc1:ppc64el mariadb-server-core-5.5:ppc64el mariadb-server-5.5:ppc64el libaprutil1-dbd-sqlite3:ppc64el libaprutil1-ldap:ppc64el apache2-bin:ppc64el apache2-data:all apache2:ppc64el bind9utils:ppc64el bind9:ppc64el expect:ppc64el ipmitool-xcat:ppc64el isc-dhcp-server:ppc64el libblas3:ppc64el libclass-inspector-perl:all libcommon-sense-perl:ppc64el libconvert-binhex-perl:all libcrypt-rijndael-perl:ppc64el libcrypt-cbc-perl:all libcrypt-ssleay-perl:ppc64el libdbd-sqlite3-perl:ppc64el libdigest-hmac-perl:all libencode-locale-perl:all libio-stty-perl:all libio-pty-perl:ppc64el libexpect-perl:all libfcgi-perl:ppc64el libhttp-date-perl:all libfile-listing-perl:all libfont-afm-perl:all liburi-perl:all libhtml-tagset-perl:all libhtml-parser-perl:ppc64el libio-html-perl:all liblwp-mediatypes-perl:all libhttp-message-perl:all libhtml-form-perl:all libhtml-tree-perl:all libhtml-format-perl:all libhtml-template-perl:all libhttp-cookies-perl:all libhttp-daemon-perl:all libhttp-negotiate-perl:all libio-sessiondata-perl:all libsocket6-perl:ppc64el libio-socket-inet6-perl:all libnet-ssleay-perl:ppc64el libio-socket-ssl-perl:all libio-stringy-perl:all libjson-perl:all libjson-xs-perl:ppc64el liblinear1:ppc64el liblinear-tools:ppc64el libnet-http-perl:all libwww-robotrules-perl:all libwww-perl:all liblwp-protocol-https-perl:all libnet-smtp-ssl-perl:all libmailtools-perl:all libmime-tools-perl:all libnet-ip-perl:all libnet-dns-perl:ppc64el libnet-libidn-perl:ppc64el libnet-telnet-perl:all libossp-uuid16:ppc64el libossp-uuid-perl:ppc64el libsnmp-perl:ppc64el libtask-weaken-perl:all libxml-parser-perl:ppc64el libsoap-lite-perl:all libxml-namespacesupport-perl:all libxml-sax-base-perl:all libxml-sax-perl:all libxml-libxml-perl:ppc64el libxml-sax-expat-perl:all libxml-simple-perl:all libxmlrpc-lite-perl:all make:ppc64el mariadb-client:all mariadb-server:all nfs-kernel-server:ppc64el ssl-cert:all conserver-xcat:ppc64el debootstrap:all elilo-xcat:all grub2-xcat:all libauthen-sasl-perl:all nmap:ppc64el perl-xcat:all syslinux-xcat:all tftp-hpa:ppc64el tftpd-hpa:ppc64el xcat-client:all xcat-genesis-base-ppc64:all xcat-genesis-scripts:ppc64el xcat-server:all xnba-undi:all xcatsn:all
1223 ? S 0:00 \_ tee -a /var/log/xcat/xcat.log
<<< ... omit ... >>>
And here is the details of xCAT log file.
root@c910f03c04k06:~# cat /var/log/xcat/xcat.log Sun Jan 18 18:33:39 EST 2015 Running postscript: syslog initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused Sun Jan 18 18:33:39 EST 2015 Running postscript: remoteshell Sun Jan 18 18:33:41 EST 2015 Running postscript: syncfiles Did not sync any files. Sun Jan 18 18:33:41 EST 2015 Running postscript: servicenode Sun Jan 18 18:34:28 EST 2015 Running postscript: otherpkgs pkgsarray: mariadb-client mariadb-common mariadb-server xcat-core/xcatsn xcat-dep/conserver-xcat, 5 yum: 0, apt: 1, zypper: 0
Addtional information, see below for the detail version informaiton of the xCAT.
root@c910f03c04k07:~# dpkg -l | grep -i xcat ii conserver-xcat 8.1.16-3 ppc64el Serial console server daemon/client ii elilo-xcat 3.14-4 all Open source implementation of Service Location Protocol V2 ii grub2-xcat 1.0-2 all grub2-xcat provides some grub2 resources generated by grub2-mknetdir,including grub2 modules and grub2 image. ii ipmitool-xcat 1.8.11-1 ppc64el utility for IPMI control with kernel driver or LAN interface ii perl-xcat 2.9-snap20150112 all xCAT perl libraries ii xcat 2.9-snap20150112 ppc64el Server and configuration utilities of the xCAT management project ii xcat-buildkit 2.9-snap20150112 all xCAT-buildkit provides the buildkit tool and sample kit files to build an xCAT kit. ii xcat-client 2.9-snap20150112 all Core executables and data of the xCAT management project ii xcat-genesis-base-ppc64 2.9-snap201410130243 all xCAT Genesis netboot image ii xcat-genesis-scripts 2.9-snap20150112 ppc64el xCAT genesis ii xcat-server 2.9-snap20150112 all Server and configuration utilities of the xCAT management project ii xnba-undi 1.0.3-7 all xCAT Network Boot Agent for x86 PXE hosts
After looking into the problem, the postbootscript "otherpkgs" will run in the first reboot after installation, it will try to upgrade all the packages if the higher version packages can be found in the online ubuntu mirror with the following code block:
elif [ $hasapt -eq 1 ]; then apt_get_update_if_repos_changed $REPOFILE if [ $VERBOSE ]; then echo "$envlist DEBIAN_FRONTEND=noninteractive apt-get -y --allow-unauthenticated --force-yes -o Dpkg:: Options::='--force-confold' -o Dpkg::Options::='--force-confdef' upgrade" fi result=`eval $envlist DEBIAN_FRONTEND=noninteractive apt-get -y --allow-unauthenticated --force-yes -o Dpkg::Options::='--force-confold' -o Dpkg::Options::='--force-confdef' upgrade 2>&1` R=$? if [ $R -ne 0 ]; then RETURNVAL=$R fi logger -p local4.info -t xcat "$result" if [ $VERBOSE ]; then echo "$result" fi fiThe following packages will be upgraded:
As show in above many upstart related packages are updated, the start condition of hvc0 cannot be triggered:
start on stopped rc RUNLEVEL=[2345] and ( not-container or container CONTAINER=lxc or container CONTAINER=lxc-libvirt)if the "upgrade" line is commented out, the problem disappeared
After we added the local mirror support, the os updates mirrors should be specified with pkgdir, the otherpkgs will not update the os packages again, so this should not be a problem any more.
We could verify and close this bug.
specifying the ubuntu official mirror in "pkgdir" instead of "otherpkgdir" can fix this problem, the document has been updated.
After installation,as shown in the process tree listed below, the login prompt in the console is started successfully :
Verified on the latest xCAT build. This bug was already fixed.