Journaled File System / Bugs / #263 fsck.jfs segfaults

David Ford - 2008-03-24

Logged In: YES
user_id=2710
Originator: YES

Unfortunately, rebuilding with the only difference being adding -g to CFLAGS, results in a working fsck. I suspect a hidden memory corruption inside fsck.jfs

Colt tmp # sbin/fsck.jfs -v /dev/sda7
sbin/fsck.jfs version 1.1.12, 24-Aug-2007
processing started: 3/24/2008 8.53.38
Using default parameter: -p
The current device is: /dev/sda7
Open(...READ/WRITE EXCLUSIVE...) returned rc = 0
Primary superblock is valid.
The type of file system for the device is JFS.
Block size in bytes: 4096
Filesystem size in blocks: 52707249
**Phase 0 - Replay Journal Log
LOGREDO: Allocating for ReDoPage: (d) 4096 bytes
LOGREDO: Allocating for NoDoFile: (d) 4096 bytes
LOGREDO: Allocating for BMap: (d) 179232 bytes
LOGREDO: Allocating for IMap: (d) 19648 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Log record for Sync Point at: 0x01a37928
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for DoBLk: (d) 4096 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Allocating for IMap: (d) 2048 bytes
LOGREDO: Beginning to update the Inode Allocation Map.
LOGREDO: Done updating the Inode Allocation Map.
LOGREDO: Beginning to update the Block Map.
LOGREDO: Done updating the Block Map.
LOGREDO: End of log found at logend = 0x01a44fe0
LOGREDO: Synch point record number: 0x01a37928
LOGREDO: Synch point record address: 0x01a1bd70
LOGREDO: Number of log records: (d) 1068
LOGREDO: Number of Do blocks: (d) 89
LOGREDO: Number of NoDo blocks: (d) 48
logredo returned rc = 0
Filesystem is clean.
All observed inconsistencies have been repaired.
Filesystem has been marked clean.
**** Filesystem was modified. ****
processing terminated: 3/24/2008 8:53:44 with return code: 0 exit code: 0.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nobody/Anonymous - 2008-03-24

Logged In: NO

Darn it. Before seeing the last comment, I was going to ask you run it under gdb to see where it was failing.

It looks like it's crashing in two places. Somewhere in phase 0, and then again in phase 1 if replaying the journal is skipped. I'm not sure where to look.

Since the "good" fsck only replayed the journal, you might want to run "fsck -n" against the partition to make sure it's really okay.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Kleikamp - 2008-03-24

Logged In: YES
user_id=422440
Originator: NO

Forgot to log in. That last comment was from me.

Shaggy

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Kleikamp - 2008-03-24

assigned_to: nobody --> shaggyk
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Ford - 2008-03-24

Logged In: YES
user_id=2710
Originator: YES

Colt ~ # /sbin/fsck.jfs -n /dev/sda7
fsck.jfs version 1.1.12, 24-Aug-2007
processing started: 3/24/2008 10.3.6
The current device is: /dev/sda7
Block size in bytes: 4096
Filesystem size in blocks: 52707249
**Phase 1 - Check Blocks, Files/Directories, and Directory Entries
Segmentation fault

Colt ~ # /root/jfs-utils/sbin/fsck.jfs -n /dev/sda7
/root/jfs-utils/sbin/fsck.jfs version 1.1.12, 24-Aug-2007
processing started: 3/24/2008 10.3.31
The current device is: /dev/sda7
Block size in bytes: 4096
Filesystem size in blocks: 52707249
**Phase 1 - Check Blocks, Files/Directories, and Directory Entries
**Phase 2 - Count links
**Phase 3 - Duplicate Block Rescan and Directory Connectedness
**Phase 4 - Report Problems
**Phase 5 - Check Connectivity
**Phase 6 - Perform Approved Corrections
**Phase 7 - Verify File/Directory Allocation Maps
**Phase 8 - Verify Disk Allocation Maps
210828996 kilobytes total disk space.
47997 kilobytes in 6442 directories.
6453410 kilobytes in 228828 user files.
4 kilobytes in extended attributes
167091 kilobytes reserved for system use.
204256488 kilobytes are available for use.
File system checked READ ONLY.
Filesystem is clean.

Colt ~ # /root/jfs-utils/sbin/fsck.jfs -v -f /dev/sda7
/root/jfs-utils/sbin/fsck.jfs version 1.1.12, 24-Aug-2007
processing started: 3/24/2008 10.8.14
The current device is: /dev/sda7
Open(...READ/WRITE EXCLUSIVE...) returned rc = 0
Primary superblock is valid.
The type of file system for the device is JFS.
Block size in bytes: 4096
Filesystem size in blocks: 52707249
**Phase 0 - Replay Journal Log
LOGREDO: Log already redone!
logredo returned rc = 0
**Phase 1 - Check Blocks, Files/Directories, and Directory Entries
**Phase 2 - Count links
**Phase 3 - Duplicate Block Rescan and Directory Connectedness
**Phase 4 - Report Problems
**Phase 5 - Check Connectivity
**Phase 6 - Perform Approved Corrections
**Phase 7 - Rebuild File/Directory Allocation Maps
**Phase 8 - Rebuild Disk Allocation Maps
Filesystem Summary:
Blocks in use for inodes: 30728
Inode count: 245824
File count: 228828
Directory count: 6442
Block count: 52707249
Free block count: 51064278
210828996 kilobytes total disk space.
47997 kilobytes in 6442 directories.
6453410 kilobytes in 228828 user files.
4 kilobytes in extended attributes
0 kilobytes in access control lists
166467 kilobytes reserved for system use.
204257112 kilobytes are available for use.
Filesystem is clean.
All observed inconsistencies have been repaired.
Filesystem has been marked clean.
**** Filesystem was modified. ****
processing terminated: 3/24/2008 10:10:01 with return code: 0 exit code: 0.

Colt ~ # fsck.jfs -v -n /dev/sda7
fsck.jfs version 1.1.12, 24-Aug-2007
processing started: 3/24/2008 10.10.39
The current device is: /dev/sda7
Open(...READONLY...) returned rc = 0
Primary superblock is valid.
The type of file system for the device is JFS.
Block size in bytes: 4096
Filesystem size in blocks: 52707249
**Phase 1 - Check Blocks, Files/Directories, and Directory Entries
Segmentation fault

FYI, I was errant previously. The build I was using had the following compile flags:

x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I.. -I../include -I../libfs -O2 -pipe -fforce-addr -D_FORTIFY_SOURCE=2 -fstack-protector -MT fsck_message.o -MD -MP -MF

It appears to malfunction based on hardened compile options. I'll post which one specifically after I've rebuilt, omitting each in turn.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Ford - 2008-03-24

Logged In: YES
user_id=2710
Originator: YES

And the winner isn't as expected. Adding -static to the linking flags is doing the breaking. It works fine with the previously mentioned hardening options.

Colt jfsutils-1.1.12 # CFLAGS="-g" LDFLAGS="-static" ./configure
[...]

Colt jfsutils-1.1.12 # gdb fsck/jfs_fsck
GNU gdb 6.7.1
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu"...
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) r -n -v /dev/sda7
Starting program: /root/jfs-utils/jfsutils-1.1.12/fsck/jfs_fsck -n -v /dev/sda7
Failed to read a valid object file image from memory.
/root/jfs-utils/jfsutils-1.1.12/fsck/jfs_fsck version 1.1.12, 24-Aug-2007
processing started: 3/24/2008 10.38.2
Filesystem is currently mounted.
WARNING: Checking a mounted filesystem does not produce dependable results.
The current device is: /dev/sda7
Open(...READONLY...) returned rc = 0
Primary superblock is valid.
The type of file system for the device is JFS.
Block size in bytes: 4096
Filesystem size in blocks: 52707249
**Phase 1 - Check Blocks, Files/Directories, and Directory Entries
|........
Program received signal SIGSEGV, Segmentation fault.
0x000000000042ef21 in fsck_hbeat (unused=0) at fsckruns.c:118
118 }
(gdb) bt
#0 0x000000000042ef21 in fsck_hbeat (unused=0) at fsckruns.c:118
#1 0x00002b6251466000 in ?? ()
#2 0x00002b6251466000 in ?? ()
#3 0x00002b6251466000 in ?? ()
#4 0x00002b6251466000 in ?? ()
#5 0x00002b6251466000 in ?? ()
#6 0x00002b6251466000 in ?? ()
#7 0x00002b6251466000 in ?? ()
#8 0x00002b6251467000 in ?? ()
#9 0x0000000000000000 in ?? ()
(gdb) l
113 current_heartbeat = 1;
114 }
115
116 fflush(stdout);
117 alarm(1);
118 }
119
120 /*****************************************************************************
121 * NAME: fsck_hbeat_start
122 *
(gdb) info local
current_heartbeat = 1
(gdb)

Rather odd. I've attached a valgrind log.

File Added: jfs.vlog

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Ford - 2008-03-24

valgrind log

jfs.vlog

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Kleikamp - 2008-03-25

Logged In: YES
user_id=422440
Originator: NO

Interesting. I don't really have a clue. Building jfsutils statically on x86 seems to work. I'll try tomorrow on an amd64 machine.

I probably won't find a machine with gentoo, but maybe I'll be able to recreate it. What versions of gcc and glibc are you running?

Thanks,
Shaggy

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Kleikamp - 2008-03-25

Logged In: YES
user_id=422440
Originator: NO

I tried to reproduce the problem on an amd64 box running ubuntu gutsy, but I don't get a trap. It's using gcc 4.1.3 and glibc 2.6.1.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Ford - 2008-03-25

Logged In: YES
user_id=2710
Originator: YES

Colt ~ # gcc -v
Using built-in specs.
Target: x86_64-pc-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-4.2.3/work/gcc-4.2.3/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/4.2.3 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.2.3/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.2.3 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.2.3/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.2.3/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.2.3/include/g++-v4 --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --disable-altivec --enable-nls --without-included-gettext --with-system-zlib --disable-checking --disable-werror --enable-secureplt --disable-libunwind-exceptions --enable-multilib --disable-libmudflap --disable-libssp --disable-libgcj --enable-languages=c,c++,treelang --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu
Thread model: posix
gcc version 4.2.3 (Gentoo 4.2.3 p1.0)

Colt ~ # emerge --info
Portage 2.1.4.4 (hardened/amd64/multilib, gcc-4.2.3, glibc-2.7-r1, 2.6.24-gentoo-r3 x86_64)
=================================================================
System uname: 2.6.24-gentoo-r3 x86_64 Dual-Core AMD Opteron(tm) Processor 2212
Timestamp of tree: Mon, 24 Mar 2008 19:00:01 +0000
ccache version 2.4 [enabled]
app-shells/bash: 3.2_p33
dev-lang/python: 2.4.3-r4, 2.5.1-r5
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache: 2.4-r7
sys-apps/baselayout: 1.12.11.1
sys-apps/sandbox: 1.2.18.1-r2
sys-devel/autoconf: 2.13, 2.61-r1
sys-devel/automake: 1.5, 1.7.9-r1, 1.9.6-r2, 1.10.1
sys-devel/binutils: 2.16.1-r3, 2.18-r1
sys-devel/gcc-config: 1.4.0-r4
sys-devel/libtool: 1.5.26
virtual/os-headers: 2.6.24
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -fforce-addr -D_FORTIFY_SOURCE=2 -fstack-protector"
CHOST="x86_64-pc-linux-gnu"

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Steve Costaras - 2008-06-05

Logged In: YES
user_id=1898120
Originator: NO

Just want to add here, that I'm seeing the same problem under ubuntu 8.04 LTS (64bit). (that comes w/ jfs_fsck 1.1.11 by default) I've compiled 1.1.12 from here same problem. fsck will pass through all steps when in -n mode but not in -f. I also tried without -O2 for CFLAGS and am not using -static for linking.

----
root@loki:/usr/src/jfs/jfsutils-1.1.12/fsck# ./jfs_fsck -f /dev/sda
./jfs_fsck version 1.1.12, 24-Aug-2007
processing started: 6/5/2008 6.39.55
The current device is: /dev/sda
Block size in bytes: 4096
Filesystem size in blocks: 5371093376
**Phase 0 - Replay Journal Log
Segmentation fault

----- (dmesg)

[ 121.927733] jfs_fsck[6124]: segfault at 7544fc rip 427846 rsp 7fff003b9890 error 4
[ 1844.412077] jfs_fsck[9940]: segfault at 75735e rip 4289b2 rsp 7fffad8bbd60 error 4
--------

Also due to a power failure

---------
stcost@loki:~$ uname -a
Linux loki 2.6.24-17-server #1 SMP Thu May 1 14:28:06 UTC 2008 x86_64 GNU/Linux
stcost@loki:~$ gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Kleikamp - 2008-06-05

Logged In: YES
user_id=422440
Originator: NO

Steve,
Can you build jfsutils with CFLAGS=-g and run it under gdb to see if it is trapping in the same place?

CFLAGS=-g ./configure
make
gdb fsck/jfs_fsck
(gdb) run -f /dev/sda

If it is trapping in the same place, you may try redirecting stdout to a file as a workaround. In that case, jfs_fsck won't run the heartbeat code.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Steve Costaras - 2008-06-05

Logged In: YES
user_id=1898120
Originator: NO

Yes, here's the output:

----------
(gdb) run -f /dev/sda
Starting program: /usr/src/jfs/jfsutils-1.1.12/fsck/jfs_fsck -f /dev/sda
/usr/src/jfs/jfsutils-1.1.12/fsck/jfs_fsck version 1.1.12, 24-Aug-2007
processing started: 6/5/2008 12.1.44
The current device is: /dev/sda
Block size in bytes: 4096
Filesystem size in blocks: 5371093376
**Phase 0 - Replay Journal Log
|........
Program received signal SIGSEGV, Segmentation fault.
0x0000000000438ae4 in updatePage (ld=0x7fff34e95550, logaddr=8274056) at log_work.c:2679
2679 ln = __le16_to_cpu(*--segdata); /* get length */
(gdb)
----------

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Kleikamp - 2008-06-05

Logged In: YES
user_id=422440
Originator: NO

This is a different problem, and one that I believe is fixed in cvs.

I'm attaching a patch that should fix it.

It's probably time to release jfsutils-1.1.13

Please let me know if this fixes the problem.

Thanks,
Shaggy
File Added: updatePage_sanity_check.patch

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Kleikamp - 2008-06-05

Patch to updatePage()

updatePage_sanity_check.patch

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Steve Costaras - 2008-06-06

Logged In: YES
user_id=1898120
Originator: NO

ok, I tried the patch, it would not apply against 1.1.12, seems that log_work.c already had those changes implemented. I manually added the define for UPDATEPAGE_BAD_RANGEto the logredo.h header file. jfs compiled and I ran it with some trepidation against the array. It seemed to have worked a little, it cleared up errors for the various files that were in use when it crashed but then a new error cropped up:

----
root@loki:/usr/src/jfs/jfsutils-1.1.12# ./fsck/jfs_fsck -n /dev/sda
./fsck/jfs_fsck version 1.1.12, 24-Aug-2007
processing started: 6/5/2008 19.8.54
The current device is: /dev/sda
Block size in bytes: 4096
Filesystem size in blocks: 5371093376
**Phase 1 - Check Blocks, Files/Directories, and Directory Entries
**Phase 2 - Count links
**Phase 3 - Duplicate Block Rescan and Directory Connectedness
**Phase 4 - Report Problems
File system object DF4 is linked as: /lost+found
Errors detected in Directory Index Table.
**Phase 5 - Check Connectivity
**Phase 6 - Perform Approved Corrections
**Phase 7 - Verify File/Directory Allocation Maps
**Phase 8 - Verify Disk Allocation Maps
21484373504 kilobytes total disk space.
250323 kilobytes in 64822 directories.
18818402755 kilobytes in 908414 user files.
148 kilobytes in extended attributes
3835588 kilobytes reserved for system use.
2662385336 kilobytes are available for use.
File system checked READ ONLY.
ERRORS HAVE BEEN DETECTED. Run fsck with the -f parameter to repair.
Filesystem is dirty.
Filesystem is dirty but is marked clean. In its present state,
the results of accessing /dev/sda (except by this utility) are undefined.
------

I was not getting the DF4 & Directory Index Table errors before. wanted to post results before running again in read/write mode to avoid making matters worse. ;)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

David Kleikamp - 2008-06-06

Logged In: YES
user_id=422440
Originator: NO

That's not a very serious error. Rerunning with -f won't be a problem.

I'll have to take a closer look to see why it messed up lost+found.

As far as I can tell the patch should have applied to 1.1.12. I made the patch hastily from cvs and it applies with patch -p0 rather than -p1.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Nobody/Anonymous - 2011-05-13

Sourceforge.. Peachy :)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

fsck.jfs segfaults

Group

Searches

Help

#263 fsck.jfs segfaults

Discussion