crash64 doesn't work on RHEL8.x

[Yoonsuk Cho]

Hi. I've tried to open vmcore with 3.5.0/3.5.1 both on RHEL 7.x/8.x.
I got error message like this.

error message : crash64: cannot determine length of symbol: log_end

and crash64 doesn't work anymore. Couldn't get any 'crash>' shell.
Just back to normal $SHELL after i got error.

There is no problem if i run crash binary which is provided by OS default.

Please, let me know how to fix this issue .

# crash64 /usr/lib/debug/lib/modules/4.18.0-372.9.1.el8.x86_64/vmlinux ./vmcore

crash64 7.2.8
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [914MB]: patching 105260 gdb minimal_symbol values

      KERNEL: /usr/lib/debug/lib/modules/4.18.0-372.9.1.el8.x86_64/vmlinux
    DUMPFILE: ./vmcore  [PARTIAL DUMP]
        CPUS: 8
        DATE: Wed May 29 23:30:48 2024
      UPTIME: 56 days, 10:35:09
LOAD AVERAGE: 17.97, 17.86, 15.12
       TASKS: 51772
    NODENAME: xxx
     RELEASE: 4.18.0-372.9.1.el8.x86_64
     VERSION: #1 SMP Fri Apr 15 22:12:19 EDT 2022
     MACHINE: x86_64  (3800 Mhz)
      MEMORY: 127.5 GB
       PANIC:

crash64: cannot determine length of symbol: log_end

-> Just i got bash shell from here. Not get in to 'crash> ' shell.

[Martin Moore]

Can you try with the latest PyKdump version, which is 3.9.1? You can download the .so file from https://sourceforge.net/projects/pykdump/files/mpykdump-x86_64/. There are separate versions for crash7 and crash8:

mpykdump-3.9.1-crash7.so
mpykdump-3.9.1-crash8.so

Use the one that matches your version of crash.

There are also many improvements and bug fixes in PyKdump 3.9 compared to 3.5.

[Yoonsuk Cho]

Thank you for your reply. Yes. It works~!. Crash binary version is 7.x under RHEL8 and I include mpykdump-3.9.1-crash7.so with 'extend' option inside 'crash>'.

I thought mpykdump works only with crash64 binary which has included and uploaded mpykdump-3.5.1.tar.gz. That's why I asked full binary like crash64 for 3.9.1 earlier. Now I know that mpykdump-xx.so can be used with OS default 'crash' binary. :)

Thank you so much.

IMHO, debug mode would be better for someone who wants to look inside what happen and want to see how it collect the information in each step. :)

I'm trying to find mutex owner from one structure to another. It was very easy to get it from 'crashinfo --mutex=0xAA..' option. However, not easy to find by manual.

I am looking at KernLocks.py. It seems contains the way how to collect and parse from results.
But, not easy to follow it. I'm not expert yet. :)

BTW, I've got another error again after issue this command 'crashinfo' as below.

   print_blkreq is not supported on this kernel. Use 'scsishow -r' to see SCSI I/O requests.
Traceback (most recent call last):
  File "/home/martin/pykdump-code/pykdump/memocaches.py", line 76, in newfunc
KeyError: ('bdev_map', 'gdb_typeinfo')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/CORES/20240610_rhel86_kworker_cgroup/kdumptool/mpykdump-3.9.1-crash7.so/progs/crashinfo.py", line 1673, in <module>
  File "/home/martin/pykdump-code/LinuxDump/Dev.py", line 556, in print_gendisk
  File "/home/martin/pykdump-code/LinuxDump/Dev.py", line 496, in get_gendisks
  File "/home/martin/pykdump-code/pykdump/datatypes.py", line 94, in __init__
  File "/home/martin/pykdump-code/pykdump/memocaches.py", line 79, in newfunc
  File "/home/martin/pykdump-code/pykdump/datatypes.py", line 44, in gdb_typeinfo
crash.error: PyKdump/GDB error

******************************************************************************
************************ A Summary Of Problems Found *************************
******************************************************************************
-------------------- A list of all +++WARNING+++ messages --------------------
    PARTIAL DUMP with size(vmcore) < 25% size(RAM)
    There are 6 threads running in their own namespaces.
        Use 'taskinfo --ns' to get more details.
------------------------------------------------------------------------------

[Martin Moore]

That's very strange...I can reproduce the error with crash7, but it works fine on crash8. I'll look into this, but in the meantime you could try switching to crash8 (which I would recommend anyway, as it has many improvements and is much faster to start up on large vmcores).