Menu ▾ ▴

Building From GIT - old Python2 based

Alex Sidorenko

This document describes building from old sources, Python-2.7 based. All new development is now done using Python3.6.x. If you need old sources, use python2 branch

If you still need the old Python2 based version (e.g. for transition period), build from python2 branch
You can retrieve the sources with GIT

$ git clone git://git.code.sf.net/p/pykdump/code pykdump

Even though all popular Linux distrubutions have Python already packaged, you need to Python build yourself, from sources. BThe reasoning for that is as follows.

It is possible to package everything (C-module, Python intepreter, pure Python code) into a single file. In this case, the dependencies are minimal - the created file depends just on glibc and its related libraries only (such as libm). If we build on a host with specific GLIBC version, the extension should be usable on any GLIBC-compatible host (the same or newer version of GLIBC). For example, if we build on RHEL4, the built extension should be usable not only on RHEL4 but on RHEL5, Ubuntu, SLES9/10 and so on. There will be no need to install any additional packages on the target system - just copy 'crash' and extension file from your build host to target host and start using them (that is, no need for Python to be installed on the target host).

It always makes sense to use recent versions of Python. But please note that even though the dev branch supports both Python-2 and Python-3, the support for Python-3 has been added recently and is not tested as thoroughly as Python-2.

Before building the extension itself, we need to build Python and Crash.

Compiling Python

There are two major versions of Python - 2.x and 3.x. Python version 3 is not backwards compatible with Python version 2 and most applications have not been ported to Python 3 yet. At this moment, it is recommended to build with Python 2 (it will be maintained for at least another 7 years). It is recommended to use the latest version of Python-2.7. Most probably, PyKdump will build fine with Python-2.6, but we do not test with this version anymore. Python-3.2.2 already works but this is still a bit experimental. So if you are not adventurous, use Python-2.7!

Unpack the sources and go to the top Python source directory

$ tar xvfj /ar/new/Python-2.7.5.tar.bz2
$ cd Python-2.7.5

We need an archive library, not .so - but with position-independent code, so we use the following command:

$ configure CFLAGS='-fPIC'

Obtain the customized modules configuration file Setup.local. You can get it from PyKdump sourcetree, Extension/ subdirectory (unfortunately, it cannot be included here - MediaWiki supports only png/jpg file attachements). You can get the whole tree using GIT, or you can download the needed file using GIT WEB-Browser, https://sourceforge.net/p/pykdump/code. Go to the tree matching the needed branch and then to Extension directory. There are three versions of this file - Setup.local-2.6, Setup.local-2.7 for Python-2.7 and Setup.local-3.2 for Python-3.2.2. Copy the needed version to Modules/ subdirectory of the Python source tree, for example:

$ mv /tmp/Setup.local-2.7 Modules/Setup.local

Please note that you need to rename it to just 'Setup.local' as demonstrated above.

Compile Python

$ make

By default, Python C-code is compiled with -g flag (debugging). If you want to decrease the size of the built extension, strip the library from debugging symbols

{alexs 9:47:08} ls -l libpython2.7.a
 -rw-r--r-- 1 alexs alexs 5508896 2012-02-06 09:47 libpython2.7.a
{alexs 9:47:16} strip -d libpython2.7.a
{alexs 9:47:23} ls -l libpython2.7.a
 -rw-r--r-- 1 alexs alexs 2110152 2012-02-06 09:47 libpython2.7.a

Compiling ''crash''

The extension can be built both with crash-5.X and crash-6.X. But beware: extensions built for different major versions of crash are not compatible.

  1. Obtain the sources of ''crash'' from http://people.redhat.com/anderson/
  2. Unpack the sources and run 'make'

Install ''zip'' command

To build the extension you need ZIP utility. Most distributions include ''zip'' it by default, otherwise please install the needed package.

Build the extension

Obtain the sources of Pykdump from GIT

$ git clone git://git.code.sf.net/p/pykdump/code pykdump

Change to the top directory of ''pykdump'' sourcetree, then change to Extension/ subdirectory.

{alexs 9:58:52} cd pykdump/Extension

Configure Makefiles specifying the location of Python and Crash you have compiled from sources. The configuration script requires full pathnames (those starting from '/'). For example:

{alexs 9:58:56} ./configure -p /src/Python/Python-2.7.3 -c /src/kerntools/crash-6.0.8

If everything worked normally, there should be 2 files created: crash.mk, and slocal.mk

Run 'make':

{alexs 10:02:14} make

If everything worked as expected, the file ''mpykdump.so'' should be created. This file is both a dynamically-loadable library and ZIP:

{alexs 10:03:52} ldd mpykdump.so linux-gate.so.1
        linux-gate.so.1 =>  (0xffffe000)
        libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7e75000)
        libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e71000)
        libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e6c000)
        libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7e47000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7cfd000)
        /lib/ld-linux.so.2 (0x80000000)
 {alexs 10:04:31} zipinfo mpykdump.so
   Archive:  mpykdump.so   1899550 bytes   79 files
   -rw-r--r--  2.3 unx    18162 bx defN  4-Feb-08 14:08 pylib/string.pyc
   ...
   -rwxr-xr-x  2.3 unx     2679 tx defN  3-Jan-08 10:39 progs/task.py
   79 files, 908886 bytes uncompressed, 336646 bytes compressed:  63.0%

You can start using this file by doing 'extend ...' in crash.

Analyzing 32-bit and 64-bit dumps on AMD64 host running 64 bits

If you have an AMD64 host running 64 bits, you can analyze both 32-bit dumps and 64-bit dumps on it. But you will need two pairs of ''crash'' and ''mpykdump.so'', the first pair compiled on a 32-bit host and another one on a 64-bit host. I usually name them differently, here is what I have on my dump-analysis AMD64 host:

-rwxr-xr-x 1 alexs users 3490964 2008-01-23 15:16 /usr/local/bin/crash32*
-rwxr-xr-x 1 root  root  4258536 2008-01-31 11:13 /usr/local/bin/crash64*
-rwxrwxr-x 1 alexs users 1842151 2008-01-31 11:32 /usr/local/lib/mpykdump32.so*
-rwxr-xr-x 1 alexs users 2176607 2008-02-05 11:30 /usr/local/lib/mpykdump64.so*

The Details Of The Build Process

This section is mainly of interest for developers.

Pykdump logically consists of three parts:

An extension written in C which is linked against Python library and 'crash' to create a DLL - a file that can be loaded by 'extend' command from crash. The compilation stage needs headers from ''crash'' (including ''gdb''-headers distributed with ''crash'') and Python headers (from the source tree). The extension is created by compiling C-sources specific to PyKdump and linking with Python C-library (e.g. libpython2.7.a). As we want to build a dynamically loadable extension, all C-code (including libpython2.7.a) needs to be compiled as ''position-indepent code'', with '''gcc''' this is either '-fpic' or '-fPIC' flag.

Python runtime environment, in paritcular the Python Standard Library, consists of many
subdirectories and contains many hundreds files. Not all of them are needed for our
purposes - e.g. we don't need to work with XML or produce sounds :-) Most of them are in
Python, but there are some modules that are usually DLLs. If we do not want to distribute
them with pykdump extension, we need to build Python with these modules compiled statically
(in this case their PIC code is present in libpython2.7.a). This is why a custom
'Setup.local' file is needed (it lists modules to be built statically).

In addition to these modules we need to copy parts of the Python Standard Library that we need (e.g. regular expressions) that are written in Python. The list of these files is Python version specific and there are several versions in the Extension directory, e.g. minpylib-2.7.lst lists these files for Python-2.7.

Python programs build on top of this - e.g. xportshow. They use more Python files implementing various things for different kernel subsystems:

  {alexs 15:38:04} wc {LinuxDump,pykdump}/**/*py
     475   1533  13639 LinuxDump/BTstack.py
      54    250   1732 LinuxDump/Files.py
      28     84    767 LinuxDump/flock.py
      ...
     237    888   6983 pykdump/obsolete.py
     615   2101  18575 pykdump/tparser.py
    1180   3944  32794 pykdump/wrapcrash.py
    8200  28712 249081 total

So how do we combine all this into a single file? The idea (borrowed from cx_Freeze Python packager) is based on two facts:

a. Python libraries can be packed into a ZIP-file instead of using real directories and Python has API that lets us use these ZIP-files.
b. ZIP-archive can be prepended by a stub (usually for Self-Extracting archives).

So we ZIP all needed Python libraries (both pieces of standard library and pykdump/LinuxDump stuff) and append them to the end of our shared library.
The resulting file, mpykdump.so, is both a DLL and ZIP-archive!

{alexs 9:24:10} ldd mpykdump.so
        linux-gate.so.1 =>  (0xffffe000)
        libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7d8b000)
        libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7d87000)
        libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7d82000)
        libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7d5d000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7c13000)
        /lib/ld-linux.so.2 (0x80000000)

{alexs 9:24:32} zipinfo mpykdump.so
Archive:  mpykdump.so   4230169 bytes   81 files
-rw-r--r--  2.3 unx    18162 bx defN 26-Feb-08 09:57 pylib/string.pyc
-rw-r--r--  2.3 unx    14581 bx defN 26-Feb-08 09:57 pylib/weakref.pyc
...
-rwxr-xr-x  2.3 unx    14005 tx defN  3-Mar-08 16:28 progs/crashinfo.py
-rwxr-xr-x  2.3 unx    25439 tx defN  5-Mar-08 16:19 progs/xportshow.py
-rwxr-xr-x  2.3 unx     1921 tx defN 28-Feb-08 16:48 progs/task.py
-rw-r--r--  2.3 unx     1125 tx defN 15-Feb-08 13:37 PyKdumpInit.py
81 files, 922536 bytes uncompressed, 342701 bytes compressed:  62.9%

Related

Wiki: Home