Menu ▾ ▴

startcrash

Alex Sidorenko

Startcrash

Sometimes we need to execute PyKdump program for several vmcores. For example, customer sends us several vmcores for the same host and we would like to run crashinfo for each of them. Normally we do the following:

  • start crash specifying Sysmap/vmlinux/vmcore
  • load the extension (this can be automated using .crashrc)
  • run crashinfo redirecting output to a file

After that we repeate the process for another vmcore.

In most cases, this process can be reasonably automated. If we are using x86_64 host to analyze dumps, we can analyze both 32-it and 64-bit vmcores (we need two copies of crash executables for that, let us call them for convenience crash32 and crash64). Usually vmcore and Sysmap files are in the same directory. If we specify this directory, our script can do the following:

  • make a reasonable guess about the name of vmcore. For example, if there are vmcore.gz and vmcore.04_04_2008 files in our directory, we are probably interested in vmcore.04_04_2008
  • find whether vmcore file is 32-bit or 64-bit
  • search for vmlinux file in predefined places (such as are used while installing kernel-debuginfo packages) if we are running on a live kernel
  • if we are not running on a live kernel, it is difficult (though possible) to find the versions of the kernel looking at 'strings vmcore' output.
  • if there are Sysmap* and vmlinux* files in the dump directory, we can try to use them
  • search for extension (we use the contents of .crashrc)
  • when we have found all the needed filenames, we can prepare an input file for crash containing the needed commands to run crashinfo and starts he needed crash executable (crash32 or crash64) passing this file to it via '-i' crash option.

After unpacking the files, do the following:
+ find the version of he kernel
+ either put the needed vmlinux files in the dump directory, or create symlinks
+ we expect vmcore files to match 'vmcore*' pattern. If this is different, e.g. 28-03-vmcore, either rename it or create a symlink (e.g. 'ln -s 28-03-vmcore vmcore')
+ we expect Sysmap files to match 'Sysmap*' pattern. Once again, rename/create symlinks if necessary.

Examples of 'standardized' directories:

{alexs 15:32:56} ls -l
total 7970652
-rwxr-xr-x 1 alexs users 12884897792 2007-03-07 18:17 vmcore*
lrwxrwxrwx 1 alexs users          60 2008-01-03 15:40 vmlinux-2.4.21-47.ELsmp -> /kernels/x86/vmlinux-2.4.21-47.ELsmp/vmlinux-2.4.21-47.ELsmp*
lrwxrwxrwx 1 alexs users          66 2008-01-03 15:40 vmlinux-2.4.21-47.ELsmp.debug -> /kernels/x86/vmlinux-2.4.21-47.ELsmp/vmlinux-2.4.21-47.ELsmp.debug*
{alexs 15:34:32} ls -l
total 422580
-rw------- 1 alexs users     720969 2008-06-20 09:40 System.map-2.6.16.46-0.12-bigsmp
-rw-r--r-- 1 alexs users     743694 2008-06-20 09:40 System.map-2.6.16.46-0.12-bigsmp.customer
-rw-r--r-- 1 alexs users     720969 2008-06-20 09:40 System.map-2.6.16.46-0.12-bigsmp.ori
-rw-r--r-- 1 alexs users 4225721432 2008-06-20 10:23 vmcore
-rw-r--r-- 1 alexs users   36288595 2008-06-20 10:31 vmlinux-2.6.16.46-0.12-bigsmp

Using startcrash

If a dump directory is structured as described above, we can do the following from the shell prompt:

$ cd /path/to/dumpdir
$ startcrash crashinfo [opts] [args]

This will be equivalent to starting the crash manually, loading the extension, doing 'epython crashifo [opts] [args]' and then quitting.
It is not necessary to cd to the dump directory - you can specify it as th 1st argument, e.g.

$ startcrash /path/to/dmpdir crashinfo [opts] [args]

In this case startcrash will not chdir but rather search for the needed files in the directory specified as the 1st argument.
Finally, if you are using crashinfo often, you can make a symlink in the following manner:

$ ln -s /my/directory/startcrash /usr/local/bin/crashinfo

After that, you can just do:

$ crashinfo [dumpdir] [opts] [args]

and this will be equivalent to

$ startcrash [dumpdir] crashinfo [opts] [args]

Overriding Filenames Manually

In case you don't use the recommended directory layout, you can specify filenames manually using the following options:

  • --vmlinux=filename
  • --sysmap=filename
  • --vmcore=filename
  • --crash=filename
  • --pythonso=extensionfilename

These options will be processed before starting crash and will be removed from the options passed to epython.

Related

Wiki: Home