Building From GIT
This document describes building from new sources, Python3.6.x base. At this moment it is in dev branch but soon will be merged with master.
- This document describes building from new sources, Python3.6.x base. At this moment it is in dev branch but soon will be merged with master.
- Compiling Python
- Compiling ''crash''
- Install ''zip'' command
- Build the extension
- Analyzing 32-bit and 64-bit dumps on AMD64 host running 64 bits
- The Details Of The Build Process
You can retrieve the sources with GIT
$ git clone git://git.code.sf.net/p/pykdump/code pykdump
Even though all popular Linux distrubutions have Python already packaged, you need to Python build yourself, from sources. The reasoning for that is as follows.
It is possible to package everything (C-module, Python intepreter, pure Python code) into a single file. In this case, the dependencies are minimal - the created file depends just on glibc and its related libraries only (such as libm). If we build on a host with specific GLIBC version, the extension should be usable on any GLIBC-compatible host (the same or newer version of GLIBC). For example, if we build on RHEL5, the built extension should be usable not only on RHEL5 but on RHEL6-7, Ubuntu, SLES10-12 and so on. There will be no need to install any additional packages on the target system - just copy 'crash' and extension file from your build host to target host and start using them (that is, no need for Python to be installed on the target host).
This document describes build based on Python-3.6.x, using 'python3' branch. In the nearest future it will be merged with 'dev' and 'master' branches. Old code based on Python-2.7 will be frozen and moved to 'python2' branch
Before building the extension itself, we need to build Python and Crash.
Compiling Python
There are two major versions of Python - 2.x and 3.x. Python version 3 is not totally backwards compatible with Python version 2. All new PyKdump development is now done using Python-3.6, this document describes build process for this Python version.
- Obtain Python sources from http://www.python.org. At this moment the latest 3.X release is 3.6.1
Unpack the sources and go to the top Python source directory (we use 3.6.2.rc2 but the same should work for any 3.6.x)
{alexs 9:47:45} tar xvf Python-3.6.2rc2.tar.xz
{alexs 9:48:49} cd Python-3.6.2rc2
We need an archive library, not .so - but with position-independent code, so we use the following command:
{alexs 9:51:23} ./configure CFLAGS=-fPIC --disable-shared
Obtain the customized modules configuration file Setup.local. You can get it from PyKdump sourcetree, Extension/ subdirectory (unfortunately, it cannot be included here - MediaWiki supports only png/jpg file attachements). You can get the whole tree using GIT, or you can download the needed file using GIT WEB-Browser, https://sourceforge.net/p/pykdump/code. Go to the tree matching the needed branch and then to Extension directory. There can be multiple versions of this file for different Python revisions, you need Setup.local-3.6. Copy the needed version to Modules/ subdirectory of the Python source tree, for example:
$ mv /tmp/Setup.local-3.6 Modules/Setup.local
Please note that you need to rename it to just 'Setup.local' as demonstrated above.
Compile Python
$ make
By default, Python C-code is compiled with -g flag (debugging). If you want to decrease the size of the built extension, strip the library from debugging symbols
{alexs 10:02:31} ls -l libpython3.6m.a
-rw-rw-r--. 1 alexs debuginfo 17620634 Jul 11 09:58 libpython3.6m.a
Compiling ''crash''
Extensions built for different major versions of crash are not compatible. We recommend to use crash-7.X
- Obtain the sources of ''crash'' from http://people.redhat.com/anderson/
- Unpack the sources and run 'make'
Install ''zip'' command
To build the extension you need ZIP utility. Most distributions include ''zip'' it by default, otherwise please install the needed package.
Build the extension
Obtain the sources of Pykdump from GIT
$ git clone git://git.code.sf.net/p/pykdump/code pykdump
Change to the top directory of ''pykdump'' sourcetree, then change to Extension/ subdirectory.
{alexs 9:58:52} cd pykdump/Extension
Configure Makefiles specifying the location of Python and Crash you have compiled from sources. The configuration script requires full pathnames (those starting from '/'). For example:
{alexs 9:58:56} ./configure -p /src/Python/Python-3.6.2rc2 -c /src/kerntools/crash-7.1.9
If everything worked normally, there should be 2 files created: crash.mk, and slocal.mk
Run 'make':
{alexs 10:02:14} make
If everything worked as expected, the file ''mpykdump.so'' should be created. This file is both a dynamically-loadable library and ZIP:
{alexs 10:08:13} ldd mpykdump.so
linux-vdso.so.1 => (0x00007ffe133d8000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5c51530000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f5c51328000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f5c51120000)
librt.so.1 => /lib64/librt.so.1 (0x00007f5c50f18000)
libm.so.6 => /lib64/libm.so.6 (0x00007f5c50c90000)
libc.so.6 => /lib64/libc.so.6 (0x00007f5c508f8000)
/lib64/ld-linux-x86-64.so.2 (0x0000003b03e00000)
{alexs 10:09:21} zipinfo mpykdump.so
Archive: mpykdump.so
Zip file size: 11670011 bytes, number of entries: 140
drwxrwxr-x 3.0 unx 0 bx stor 17-Jul-11 10:08 pylib/
-rw-r--r-- 3.0 unx 14027 bx defN 17-Jul-11 10:08 pylib/gettext.pyc
drwxrwxr-x 3.0 unx 0 bx stor 17-Jul-11 10:08 pylib/importlib/
-rw-r--r-- 3.0 unx 3625 bx defN 17-Jul-11 10:08 pylib/importlib/__init__.pyc
-rw-r--r-- 3.0 unx 3751 bx defN 17-Jul-11 10:08 pylib/genericpath.pyc
-rw-r--r-- 3.0 unx 8226 bx defN 17-Jul-11 10:08 pylib/types.pyc
...
-rw-rw-r-- 3.0 unx 770 tx defN 17-Apr-13 14:27 progs/tslog.py
-rw-rw-r-- 3.0 unx 7355 tx defN 17-Apr-11 11:29 progs/scsi.py
-rw-rw-r-- 3.0 unx 3481 tx defN 17-Mar-28 15:09 PyKdumpInit.py
140 files, 1500162 bytes uncompressed, 629550 bytes compressed: 58.0%
You can start using this file by doing 'extend ...' in crash.
Analyzing 32-bit and 64-bit dumps on AMD64 host running 64 bits
If you have an AMD64 host running 64 bits, you can analyze both 32-bit dumps and 64-bit dumps on it. But you will need two pairs of ''crash'' and ''mpykdump.so'', the first pair compiled on a 32-bit host and another one on a 64-bit host. I usually name them differently, here is what I have on my dump-analysis AMD64 host:
-rwxr-xr-x 1 alexs users 3490964 2008-01-23 15:16 /usr/local/bin/crash32* -rwxr-xr-x 1 root root 4258536 2008-01-31 11:13 /usr/local/bin/crash64* -rwxrwxr-x 1 alexs users 1842151 2008-01-31 11:32 /usr/local/lib/mpykdump32.so* -rwxr-xr-x 1 alexs users 2176607 2008-02-05 11:30 /usr/local/lib/mpykdump64.so*
The Details Of The Build Process
This section is mainly of interest for developers.
Pykdump logically consists of three parts:
An extension written in C which is linked against Python library and 'crash' to create a DLL - a file that can be loaded by 'extend' command from crash. The compilation stage needs headers from ''crash'' (including ''gdb''-headers distributed with ''crash'') and Python headers (from the source tree). The extension is created by compiling C-sources specific to PyKdump and linking with Python C-library (e.g. libpython2.7.a). As we want to build a dynamically loadable extension, all C-code (including libpython2.7.a) needs to be compiled as ''position-indepent code'', with '''gcc''' this is either '-fpic' or '-fPIC' flag.
Python runtime environment, in paritcular the Python Standard Library, consists of many
subdirectories and contains many hundreds files. Not all of them are needed for our
purposes - e.g. we don't need to work with XML or produce sounds :-) Most of them are in
Python, but there are some modules that are usually DLLs. If we do not want to distribute
them with pykdump extension, we need to build Python with these modules compiled statically
(in this case their PIC code is present in libpython3.6m.a). This is why a custom
'Setup.local' file is needed (it lists modules to be built statically).
In addition to these modules we need to copy parts of the Python Standard Library that we need (e.g. regular expressions) that are written in Python. The list of these files is Python version specific and there are several versions in the Extension directory, e.g. minpylib-3.6.lst lists these files for Python-3.6.
Python programs build on top of this - e.g. xportshow. They use more Python files implementing various things for different kernel subsystems:
{alexs 15:38:04} wc {LinuxDump,pykdump}/**/*py
475 1533 13639 LinuxDump/BTstack.py
54 250 1732 LinuxDump/Files.py
28 84 767 LinuxDump/flock.py
...
237 888 6983 pykdump/obsolete.py
615 2101 18575 pykdump/tparser.py
1180 3944 32794 pykdump/wrapcrash.py
8200 28712 249081 total
So how do we combine all this into a single file? The idea (borrowed from cx_Freeze Python packager) is based on two facts:
a. Python libraries can be packed into a ZIP-file instead of using real directories and Python has API that lets us use these ZIP-files.
b. ZIP-archive can be prepended by a stub (usually for Self-Extracting archives).
So we ZIP all needed Python libraries (both pieces of standard library and pykdump/LinuxDump stuff) and append them to the end of our shared library.
The resulting file, mpykdump.so, is both a DLL and ZIP-archive!
{alexs 9:24:10} ldd mpykdump.so
linux-gate.so.1 => (0xffffe000)
libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7d8b000)
libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7d87000)
libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7d82000)
libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7d5d000)
libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7c13000)
/lib/ld-linux.so.2 (0x80000000)
{alexs 9:24:32} zipinfo mpykdump.so
Archive: mpykdump.so 4230169 bytes 81 files
-rw-r--r-- 2.3 unx 18162 bx defN 26-Feb-08 09:57 pylib/string.pyc
-rw-r--r-- 2.3 unx 14581 bx defN 26-Feb-08 09:57 pylib/weakref.pyc
...
-rwxr-xr-x 2.3 unx 14005 tx defN 3-Mar-08 16:28 progs/crashinfo.py
-rwxr-xr-x 2.3 unx 25439 tx defN 5-Mar-08 16:19 progs/xportshow.py
-rwxr-xr-x 2.3 unx 1921 tx defN 28-Feb-08 16:48 progs/task.py
-rw-r--r-- 2.3 unx 1125 tx defN 15-Feb-08 13:37 PyKdumpInit.py
81 files, 922536 bytes uncompressed, 342701 bytes compressed: 62.9%