[Ltp-cvs] ltp/testcases/kdump runkdump.conf, NONE, 1.1 runkdump.sh, NONE, 1.1 Makefile, 1.3, 1.4 README, 1.3, 1.4 TEST_PLAN.txt, 1.2, NONE crash_cmds, 1.3, NONE distro, 1.4, NONE kdump_propagate, 1.1, NONE master, 1.8, NONE setup, 1.14, NONE summary, 1.3, NONE sysinfo, 1.3, NONE test, 1.6, NONE tests, 1.3, NONE verify, 1.3, NONE verify_dump, 1.1, NONE

[Subrata <sub...@us...>]

Update of /cvsroot/ltp/ltp/testcases/kdump
In directory sc8-pr-cvs6.sourceforge.net:/tmp/cvs-serv29537/ltp/testcases/kdump

Modified Files:
	Makefile README 
Added Files:
	runkdump.conf runkdump.sh 
Removed Files:
	TEST_PLAN.txt crash_cmds distro kdump_propagate master setup 
	summary sysinfo test tests verify verify_dump 
Log Message:
Huge Changes from Cai Qian <qc...@re...>. Please see the Description below:

The updated version
has the following improvments,

- simplify maintenance

- fix several bugs

- better automation

- extend support on Debian and Fedora

- merge Poornima Nayak <Poo...@in...> 's latest patch

- other changes

The new version simplifies code layout, and all tools under lib/
directory are standalone and reusable, all configurable variables are
under the file runkdump.conf. There are also included sample
configuration files for different distros and test components. In
addition, it avoided code duplication by merging RHEL and SLES specified
code in lkdtm module together, and add a checking before hand. The
working flow in main script has also been tidied up. verify and
verify_dump have been unified. unused file "summary" has been deleted.

In previous version,
* The part of comparing kernel version in setup will not work proper 
  if the kernel grows to something like 2.7.1 or 2.8.1.
* verify_dump will not able to find previous vmcore if vmcore creation
  and vmcore verfication happen at different hours.
* Other random bugs because of code complexity.

In the new version, once you define everything in a config file, it will
run in an automatically fashion on some of systems, if there is proper
tools installed.

The code has been rewritten in POSIX shell syntax, and made more
portable, and attempted to remove hard-coded stuff as much as
possible. It has been tested fairly on RHEL5.1, SLES10sp1, Debian
unstable, and Fedora 8. If needed I can send you result logs. Sachin
has done a quick test on PPC64 machine running RHEL5.1 and the script
run fine. Poornima has done some code review.

Poornima's patch has been merged,
1. Crashkerenl reservation check is modified to support any craskernel
value.
2. A message before system reboots
3. Added one more line to 'status' if dump verification passed.
4. Modified 'README' file
5. If kdump start fails for invalid arguments in kdump.conf is fixed
6. Replaced MAKE_OPTS with MAKE_OPTIONS.
7. Removed some unwanted statements.

Other changes included that all tests descriptions have been documented
in doc/ALL_TEST.txt. KNLD (dump over scp with link delay) test case has
been merged into KNSCP (dump over scp). Once LINK_DELAY has been defined
somewhere in configuration file, all test cases (KNSCP and KNNFS)
dumping to a network destination will take advantage of it. If
LINK_DELAY is not defined or equals to 0, KNSCP and KNNFS will just the
same as in previous version.

 Cai Qian


--- crash_cmds DELETED ---

--- tests DELETED ---

Index: README
===================================================================
RCS file: /cvsroot/ltp/ltp/testcases/kdump/README,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** README	22 Nov 2007 14:28:14 -0000	1.3
--- README	18 Jan 2008 09:20:25 -0000	1.4
***************
*** 2,8 ****
  ----------------------------
  
! The kdump test automation suite helps run the kdump tests and report results. 
! The testscripts cycle through a series of crash scenarios. Each test cycle does
!  the following:
  
  1.  Sets up a crash scenario.
--- 2,8 ----
  ----------------------------
  
! The kdump test automation suite helps run the kdump tests and report
! results.  The testscripts cycle through a series of crash
! scenarios. Each test cycle does the following:
  
  1.  Sets up a crash scenario.
***************
*** 11,21 ****
  4.  System reboots to 1st kernel.
  5.  vmcore is validated and results are saved.
! 6.  After a 1 to 2 minute delay, the next crash scenario is setup and run.
  
! The scripts make use of the crasher module for basic testing of kdump and the 
! new Linux Kernel Dump Test Module (LKDTM) for more involved testing. LKDTM makes
! use of the kprobes infrastructure for inserting crashpoints into the kernel at 
! run-time. Thus the kernel need not be patched and rebuilt.This script also tests kdump
! dumping on different destinations like ext3, raw and network dump.
  
  KDUMP TEST INSTRUCTION
--- 11,23 ----
  4.  System reboots to 1st kernel.
  5.  vmcore is validated and results are saved.
! 6.  After a 1 to 2 minute delay, the next crash scenario is setup and
!     run. 
  
! The scripts make use of the crasher module for basic testing of kdump
! and the new Linux Kernel Dump Test Module (LKDTM) for more involved
! testing. LKDTM makes use of the kprobes infrastructure for inserting
! crashpoints into the kernel at run-time. Thus the kernel need not be
! patched and rebuilt.This script also tests kdump dumping on different
! destinations like ext3, raw and network dump.
  
  KDUMP TEST INSTRUCTION
***************
*** 24,35 ****
  Follow the steps to setup kdump test automation suite.
  
! The tests are written for SuSE Linux Enterprise Server 10 (and onward releases)
! as well as RedHat Enterprise Linux 5. Since KDUMP is supported by the above 
! mentioned distro's the tests were written and tested on them. Contribution 
! towards supporting more distributions are welcome.
  
  1. Install these additional packages:
  
! For SLES10 Distro :
     
     * kernel-kdump
--- 26,38 ----
  Follow the steps to setup kdump test automation suite.
  
! The tests are written for SuSE Linux Enterprise Server 10 (and onward
! releases), OpenSUSE, Fedora, Debian, as well as RedHat Enterprise Linux
! 5. Since KDUMP is supported by the above mentioned distro's the tests
! were written and tested on them. Contribution towards supporting more
! distributions are welcome.
  
  1. Install these additional packages:
  
! For SLES10 or OpenSUSE Distro :
     
     * kernel-kdump
***************
*** 37,97 ****
     * kexec-tools
     * zlib-64bit-<xxx> (ppc64 only)
!    * expect
  
! For RHEL5 distro :
  
     * kexec-tools
!    * kernel-debuginfo rpm
     * kernel-kdump ( only for ppc64 )
!    * expect
! 
! 2.  Make sure the partition where the tests are running has space for the tests
! results and one vmcore file (size of physical memory).
! 
! 3. Setup script is enhanced to update menu.lst file with 'crashkernel=' parameter based
! on architecture.
  
! 4.  Edit the bootloader configuration file to add appropriate 
! 'nmi_watchdog=' parameter.
!  
! 6.  'cd' to the test suite directory and run "make". Carefully check for any 
! errors. If at some point you need to restart the tests from the beginning, 
! simply run "make clean" followed by "make" again.
  
! 7.  To enable usage of 'crash' utility to test the integrity of the dump 
! captured, path to vmlinux with debug info needs to be provided at the time of 
! setup (the path would generally be 
! /usr/lib/debug/lib/modules/<kernel version>/vmlinux).
  
! 8.  To start the kdump test run './setup test'. Inputs to the script can be provided interactively
! or through the data file, format of data in input file should be like exampledata file bundled 
! along with ltp kdump code.
!  
! 9.  Setup script can also be executed as './setup enable' to enable 
! kdump to dump on network dump server, information required for dumping i.e network dump server 
! and password should be provided in 'enable_data'. 
  
  Few Important points to remember:
        
! * If you need to stop the tests before all tests have run, run "crontab -r" and 
! "killall master" within 1 minute after the 1st kernel reboots.
  
! * A failure is likely to occur when booting the kdump kernel. If this happens, 
! you'll need to manually reset the system so it reboots back to the 1st kernel 
! and continues on to the next test. For this reason, it's best to monitor the 
! tests from a console. If possible, setup a serial console (not a must, any type
! of console setup will do). If using minicom, enable saving of kernel messages 
! displayed on minicom into a file, by pressing ctrl+a+l on the console. Else, 
! when it is observed that the kdump kernel has failed to boot, manually copy the
! boot message into a file to enable the debugging the cause of the hang.
  
! * The results are saved in <kdump-test-dir>/results/<hostname>.<date-time>. The
! "status" file in that directory shows where you are in the test run. When the 
! "Test run complete" entry appears in that file, you're done.
  
! * The test machine would be unavailabe for any other work during the period of 
! the test run. 
  
! * System will hang if incorrect partition information is provided for dumping, like specifying
! a partition which does not exist, specifying a parition label which does not exist. This is not
! ltp kdump bug.
--- 40,90 ----
     * kexec-tools
     * zlib-64bit-<xxx> (ppc64 only)
!    * expect (if dump to a network destination)
  
! For RHEL5 or Fedora distro :
  
     * kexec-tools
!    * kernel-devel
!    * kernel-debuginfo rpm (if using crash to verify vmcore)
     * kernel-kdump ( only for ppc64 )
!    * expect (if dump to a network destination)
  
! 2.  Make sure the partition where the tests are running has space for
! the tests results and one vmcore file (size of physical memory).
  
! 3. Modify configuration file runkdump.conf or copy a existing one from
! sample/.
  
! 4.  Run "make". Carefully check for any errors.
  
  Few Important points to remember:
        
! * If you need to stop the tests before all tests have run, run "crontab
! -r" and "killall runkdump.sh" within 1 minute after the 1st kernel
! reboots. Then, if you'd like to carry on tests from the point on, run
! "crontab kdump.cron", and "./runkdump.sh". If you'd like to start tests
! from the beginning, modify the configuration file, and set
! "REBOOT=0". Then, "./runkdump.sh"
  
! * A failure is likely to occur when booting the kdump kernel. If this
! happens, you'll need to manually reset the system so it reboots back to
! the 1st kernel and continues on to the next test. For this reason, it's
! best to monitor the tests from a console. If possible, setup a serial
! console (not a must, any type of console setup will do). If using
! minicom, enable saving of kernel messages displayed on minicom into a
! file, by pressing ctrl+a+l on the console. Else, when it is observed
! that the kdump kernel has failed to boot, manually copy the boot message
! into a file to enable the debugging the cause of the hang.
  
! * The results are saved in
! <kdump-test-dir>/log/<hostname>.<date-time>. The "status" file in that
! directory shows where you are in the test run. When the "Test run
! complete" entry appears in that file, you're done. Verbose log can be
! found at /tmp/kdump-<date>.log.
  
! * The test machine would be unavailabe for any other work during the
! period of the test run.
  
! * System may hang if incorrect partition information is provided for
! dumping, like specifying a partition which does not exist, specifying a
! parition label which does not exist. This is not ltp kdump bug.

--- verify DELETED ---

--- setup DELETED ---

Index: Makefile
===================================================================
RCS file: /cvsroot/ltp/ltp/testcases/kdump/Makefile,v
retrieving revision 1.3
retrieving revision 1.4
diff -C2 -d -r1.3 -r1.4
*** Makefile	22 Nov 2007 14:28:14 -0000	1.3
--- Makefile	18 Jan 2008 09:20:24 -0000	1.4
***************
*** 1,12 ****
! CFLAGS = -Wall
! 
! all:
! 	chmod 755 crash_cmds distro master setup summary sysinfo test tests verify
! 	./distro
! 	make -C tools all
! 
! clean:
! 	make -C tools clean
! 	rm -f config control crontab testlists/lkdtm testfile
! 	mv -f /etc/sysconfig/kdump.orig /etc/sysconfig/kdump 2> /dev/null
! 	crontab -r
--- 1,3 ----
! all: runkdump.conf
! 	chmod a+x runkdump.sh lib/*.sh
! 	./runkdump.sh

--- NEW FILE: runkdump.conf ---
# Repeation of test.
ITERATION=1

# Vmcore verification using crash command.
# Require kernel-debuginfo and crash packages.
CRASH=0

# Reboot count. New test should start from 0.
REBOOT=0

# Kernel debuginfo.
VMLINUX="/usr/lib/debug/lib/modules/$(uname -r)/vmlinux"

# Where vmcore will be saved.
# /var/crash for Red Hat.
# /var/log/dump for SUSE.
COREDIR=/var/crash

# Enable dump with filtering.
FILTER=0
# makedumpfile options.
MAKE_OPTIONS=

# Delay in seconds when dumping to a network destination.
LINK_DELAY=

# The list of testcases. Please see doc for the overview of 
# those tests.

# ACS ACP ACB ACE ACL
CRASHER="ACS ACP ACB ACE ACL"

# KPIDB KPIEB KPTEE KPBL
BASIC_LKDTM="KPIDB KPIEB KPTEE KPBL"

#       KPIDE KPIDL KPIDP KPIDO
#       KPIEE KPIEL KPIEP KPIEO
# KPTEB       KPTEL KPTEP KPTEO
# KPBB  KPBE         KPBP  KPBO
# KPMSB KPMSE KPMSL KPMSP KPMSO
# KPTB  KPTE  KPTL  KPTP  KPTO
# KPSB  KPSE  KPSL  KPSP  KPSO
# KPIB  KPIE  KPIL  KPIP  KPIO
EXTRA_LKDTM="
KPIDE KPIDL KPIDP KPIDO
KPIEE KPIEL KPIEP KPIEO
KPTEB KPTEL KPTEP KPTEO
KPBB KPBE KPBP KPBO
KPMSB KPMSE KPMSL KPMSP KPMSO
KPTB KPTE KPTL KPTP KPTO
KPSB KPSE KPSL KPSP KPSO
KPIB KPIE KPIL KPIP KPIO
"

# KLEXT KLLBL KLUID KLRAW KNSCP KNNFS KDENB
EXTRA_DUMP="KLEXT KLLBL KLUID KLRAW KNSCP KNNFS KDENB"

# Dump vmcore on an EXT3 partition.
EXT3_PART=

# Dump vmcore on an EXT3 partition with a LABEL.
EXT3_LABEL=

# Dump vmcore on an EXT3 partition with a partition UID.
EXT3_UID=

# Dump vmcore on a RAW partition.
RAW_PART=

# Dump vmcore on network.
SCP_PATH=
# Password for user@server.
SCP_PASS=

# Dump vmcore on a mounted NFS filesystem.
NFS_PATH=

--- summary DELETED ---

--- NEW FILE: runkdump.sh ---
#!/bin/sh -xe

SetupCrontab ()
{
    echo "Setup crontab."

    set +e
    crontab -r
    set -e
   
    # crontab in some distros will not read from STDIN.

    cat <<EOF >kdump.cron
SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin
MAILTO=root
@reboot cd "$(pwd)"; cd ..; ${0} >>/tmp/kdump-$(date +%F-%T).log 2>&1
EOF

    crontab kdump.cron

    echo "Enable cron daemon by default."
    
    if [ -f /etc/init.d/crond ]; then
        cron=crond
    else
        # SUSE
        cron=cron
    fi

    # Red Hat and SUSE.
    if [ -x "/sbin/chkconfig" ]; then
        /sbin/chkconfig "${cron}" on
 
    # Debian and Ubuntu.
    elif [ -x "/sbin/update-rc.d" ]; then
        /sbin/update-rc.d "${cron}" defaults
    fi
}

SetupKdump ()
{
    echo "Start kdump daemon."
    /etc/init.d/kdump restart
    
    echo "Enable kdump daemon by default."
    # Red Hat and SUSE.
    if [ -x "/sbin/chkconfig" ]; then
        /sbin/chkconfig kdump on
        
    # Debian and Ubuntu.
    elif [ -x "/sbin/update-rc.d" ]; then
        /sbin/update-rc.d kdump defaults
    fi    
}


PrepareVerify ()
{
    if [ "${last}" = "KLEXT" ]; then
        # If not mountable, skip it, and continue doing the test.
        set +e
        mount "${EXT3_PART}" /mnt
        set -e

        COREDIR=/mnt"${COREDIR}"

    elif [ "${last}" = "KLLBL" ]; then
        # If not mountable, skip it, and continue doing the test.
        set +e
        mount -L "${EXT3_LABEL}" /mnt
        set -e

        COREDIR=/mnt"${COREDIR}"

    elif [ "${last}" = "KLUID" ]; then
        # If not mountable, skip it, and continue doing the test.
        set +e
        mount "/dev/disk/by-uuid/${EXT3_UID}" /mnt
        set -e

        COREDIR=/mnt"${COREDIR}"

    elif [ "${last}" = "KLRAW" ]; then
        mkdir -p "${COREDIR}/${last}"

        # If not dumpable, skip it, and continue doing the test.
        set +e
        dd if="${RAW_PART}" of="${COREDIR}/${last}/vmcore" bs=1024
        set -e

    elif [ "${last}" = "KNSCP" ]; then
        if [ -z "${SCP_PATH}" ]; then
            echo "Fail: network destination not defined."
            exit 1
        fi

        file=$(ssh "${SCP_PATH}" "ls -t ${COREDIR}/*/vmcore \
         2>/dev/null | head -1")

        mkdir -p "${COREDIR}/${last}"

        if [ "${file}" ]; then
            # Not fatal error.
            set +e
            scp "${SCP_PATH}:${file}" "${COREDIR}/${last}"
            set -e
        fi

    elif [ "${last}" = "KNNFS" ]; then
        # Not fatal error.
        set +e
        mount "${NFS_PATH}" /mnt
        set -e

        COREDIR=/mnt"${COREDIR}"
    fi

    vmcore=$(ls -t "${COREDIR}"/*/vmcore 2>/dev/null | head -1)
}

VerifyTest ()
{
    # Should not be here.
    if [ -z "${last}" ]; then
        echo "Should not be here!"
	echo "There must be something wrong with the test setup."
	exit 1
    fi

    echo "Verifying the result of previous test ${last}."
    ldir=$(ls -td "../${log}/$(hostname)."* | head -1)
    
    if [ -f "${vmcore}" ]; then
        echo "$(date +%F-%T): verification of test ${last} passed." \
        >>"${ldir}/status"
        
        ./verify.sh "../${conf}" "${vmcore}" "${CRASH}" \
        >>"${ldir}/${ITERATION}.${last}.$(date +%F-%T)"

        # Be careful to define COREDIR.
        rm -rf "${COREDIR}"/*

    else
        echo "$(date +%F-%T): verification of test ${last} failed:\
 vmcore NOT FOUND." >>"${ldir}/status"
        echo "vmcore NOT FOUND." \
        >>"${ldir}/${ITERATION}.${last}.$(date +%F-%T)"
    fi
}

RunTest ()
{

    sed -i "s/\(^REBOOT\)=.*/\1=$((count + 1))/" \
     "../${conf}"

    echo "Running current test ${i}."
            
    echo "$(date +%F-%T): running current test ${i}." \
    >> "${ldir}/status"

    # Save STDIO buffers.
    sync
    ./test.sh "../${conf}" "${i}" "../${log}"
}

# Start test.
conf="runkdump.conf"
lib="lib"
log="log"

# Read test configuration file.
. "${conf}"

# Check mandatory variables.
if [ -z "${ITERATION}" ] || [ -z "${REBOOT}" ] || [ -z "${COREDIR}" ]
then
    echo "Fail: some mandatory variables are missing from\
 configuration file."
    exit 1
fi

cd "${lib}"

while [ "${ITERATION}" -ge 1 ]; do

# Reboot the machine first to take advantage of boot parameter 
# changes.
if [ -z "${REBOOT}" ] || [ "${REBOOT}" -eq 0 ]; then
    echo "Setup test environment."

    SetupCrontab

    ./setup.sh "../${conf}"

    sed -i 's/\(^REBOOT\)=.*/\1=1/' "../${conf}"

    echo "System is going to reboot."
    /sbin/shutdown -r now
    sleep 60

else
    count=1

    for i in ${CRASHER} ${BASIC_LKDTM} ${EXTRA_LKDTM} ${EXTRA_DUMP} \
             END; do

        if [ "${count}" -eq "${REBOOT}" ]; then
            # Wait for machine fully booted.
            sleep 60

            # First Test.
            if [ "${REBOOT}" -eq 1 ]; then
                echo "First test..."
                echo "Verify Boot Loader."
                if ! grep 'crashkernel=' /proc/cmdline; then
                    echo "Fail: error changing Boot Loader, no crashkernel=."
                    exit 1
                fi

                SetupKdump

                # Creat log directory.
                mkdir -p "../${log}/$(hostname).$(date +%F-%T)"

                echo "Gather system information."

                ldir=$(ls -td "../${log}/$(hostname)."* | head -1)
                ./sysinfo.sh >"${ldir}/system.info"

            else
                PrepareVerify

                VerifyTest

		if [ "${i}" = END ]; then
		    # We are done.
		    break
		fi

            fi

            RunTest

            # Some tests could not reboot target. They can hung up
            # machine or leave it working. But we need to do all
            # tests. So we are going to reboot if we are in wrong 
            # place.

            sleep 3600
            echo "$(date +%F-%T): manually reboot for test ${i}." >>"${ldir}/status"
            /sbin/shutdown -r now
            sleep 60
        fi

        # No test is scheduled to run.
        count=$((count + 1))
	last=${i}
    done
fi

if [ "${ITERATION}" -eq 1 ]; then
    # We are done.
    break

else
    # Run the next iteration.
    sed -i "s/\(^ITERATION\)=.*/\1=$((ITERATION - 1))/" \
     "../${conf}"    
fi

done

# We are done.
# Reset.
sed -i "s/\(^REBOOT\)=.*/\1=0/" "../${conf}"
crontab -r
ldir=$(ls -td "../${log}/$(hostname)."* | head -1)
echo "$(date +%F-%T): test run complete." >>"${ldir}/status"

exit 0

--- master DELETED ---

--- TEST_PLAN.txt DELETED ---

--- kdump_propagate DELETED ---

--- verify_dump DELETED ---

--- test DELETED ---

--- sysinfo DELETED ---

--- distro DELETED ---