#277 Filesystem Expand Clobbered filesystem

[rwk]

I attempted to expand a JFS filesystem from 21TB to 43TB using EVMS.
It failed and now my filesystem will not mount. Expand errors:

device-mapper: table: 254:5: mirror: unknown target type
device-mapper: ioctl: error adding target to table
jfs_extendfs: mapSize (0x16448c000) > t64 (0xbacfe000)
ERROR: (device dm-2): jfs_extendfs
JFS: metapage_get_blocks failed
JFS: metapage_get_blocks failed
BUG: Bad page state in process umount pfn:c7c55
page:ffffea0002bb3298 flags:4000000000000808 count:0 mapcount:0 mapping:(null) index:aeadd
Pid: 7487, comm: umount Not tainted 2.6.31-gentoo-r5 #3
Call Trace:
[<ffffffff8108130f>] ? bad_page+0x102/0x113
[<ffffffff810818d9>] ? __pagevec_free+0x29/0x3b
[<ffffffff81084c6b>] ? release_pages+0x14b/0x199
[<ffffffff81085280>] ? __pagevec_release+0x19/0x22
[<ffffffff810857ef>] ? truncate_inode_pages_range+0xe4/0x2e1
[<ffffffff811891a4>] ? dbUnmount+0x3c/0x4b
[<ffffffff81180018>] ? jfs_umount+0x95/0xe5
[<ffffffff8117cafb>] ? jfs_put_super+0x1d/0x76
[<ffffffff810a8b15>] ? generic_shutdown_super+0x4f/0xdf
[<ffffffff810a8bc7>] ? kill_block_super+0x22/0x3a
[<ffffffff810a8e9c>] ? deactivate_super+0x43/0x5a
[<ffffffff810ba77a>] ? sys_umount+0x2a9/0x2d6
[<ffffffff8100adeb>] ? system_call_fastpath+0x16/0x1b
Disabling lock debugging due to kernel taint

fsck.jfs now reports:

fsck.jfs version 1.1.14, 06-Apr-2009
processing started: 1/21/2010 7.55.6
The current device is: /dev/evms/dvds
Block size in bytes: 4096
Filesystem size in blocks: 5855592447
**Phase 0 - Replay Journal Log
**Phase 1 - Check Blocks, Files/Directories, and Directory Entries
Errors detected in the Primary File/Directory Allocation Table.
Errors detected in the Secondary File/Directory Allocation Table.
CANNOT CONTINUE.

EVMS now reports a filesystem size of 43TB, but will not mount saying:

mount: wrong fs type, bad option, bad superblock on /dev/evms/dvds,
missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

Is there any way to recover, or revert back to the 21TB filesystem
before the attempted but failed expand operation?

[Tahoe]

I have found that the problem occurs going from 32 to 33 TB. Has anyone else
experienced this? I have a 200TB storage array and am currently having to slice it into
32TB chunks. The following is are recreation of the problem.

#######################################################################################
# Ubuntu Version 10.04 LTS
#######################################################################################

hal:~$ uname -a
Linux hal 2.6.32-28-server #55-Ubuntu SMP Mon Jan 10 23:57:16 UTC 2011 x86_64 GNU/Linux

#######################################################################################
# JFS Utils Version
#######################################################################################
hal:~$ mkfs.jfs
mkfs.jfs version 1.1.14, 06-Apr-2009

Error: Device not specified or command format error

Usage: mkfs.jfs [-cOqV] [-j log_device] [-J options] [-L vol_label] [-s log_size] device [ blocks ]

Emergency help:
-c Check device for bad blocks before building file system.
-O Provide case-insensitive support for OS/2 compatability.
-q Quiet execution.
-V Print version information only.
-j log_device Set external journal device.
-J options Set external journal options.
-L vol_label Set volume label for the file system.
-s log_size Set log size (in megabytes).

NOTE: -j and -J cannot be used at the same time.

#######################################################################################
# Create a 32 TB Logical Volume
#######################################################################################
hal:~$ sudo lvcreate -L32T -nbackup datavg
Logical volume "backup" created

#######################################################################################
# Create a 32 TB JFS Filesystem
#######################################################################################
hal:~$ sudo mkfs.jfs /dev/datavg/backup
mkfs.jfs version 1.1.14, 06-Apr-2009
Warning! All data on device /dev/datavg/backup will be lost!

Continue? (Y/N) y
|

Format completed successfully.

34359738368 kilobytes total disk space.

#######################################################################################
# Do a filesystem check
#######################################################################################
hal:~$ sudo fsck.jfs /dev/datavg/backup
fsck.jfs version 1.1.14, 06-Apr-2009
processing started: 2/28/2011 20.27.1
Using default parameter: -p
The current device is: /dev/datavg/backup
Block size in bytes: 4096
Filesystem size in blocks: 8589934592
**Phase 0 - Replay Journal Log
Filesystem is clean.

#######################################################################################
# Mount the filesystem
#######################################################################################
hal:~$ sudo mount /dev/datavg/backup /backup

#######################################################################################
# Extend the logical volume by 1 TB
#######################################################################################
hal:~$ sudo lvextend -L+1T /dev/datavg/backup
Extending logical volume backup to 33.00 TiB
Logical volume backup successfully resized

#######################################################################################
# Extend JFS
#######################################################################################
hal:~$ sudo mount -o remount,resize /backup
mount: /dev/mapper/datavg-backup: can't read superblock

#######################################################################################
# dmesg
#######################################################################################
hal:~$ dmesg | tail
[ 22.701014] eth0: no IPv6 routers present
[ 29.512256] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 29.515458] NFSD: starting 90-second grace period
[ 149.346690] IPMI System Interface driver.
[ 149.346695] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
[ 151.941824] ipmi: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20)
[ 151.941872] IPMI kcs interface initialized
[ 973.381244] JFS: nTxBlock = 8192, nTxLock = 65536
[11894.047996] ERROR: (device dm-1): jfs_extendfs
[19701.575221] ERROR: (device dm-1): jfs_extendfs

#######################################################################################
# Unmount the filesystem and do an fsck
#######################################################################################
hal:~$ sudo umount /backup
hal:~$ sudo fsck.jfs /dev/datavg/backup
fsck.jfs version 1.1.14, 06-Apr-2009
processing started: 2/28/2011 20.31.10
Using default parameter: -p
The current device is: /dev/datavg/backup
Block size in bytes: 4096
Filesystem size in blocks: 8589934592
**Phase 0 - Replay Journal Log
**Phase 1 - Check Blocks, Files/Directories, and Directory Entries
Errors detected in the Primary File/Directory Allocation Table.
Errors detected in the Secondary File/Directory Allocation Table.
CANNOT CONTINUE.

#######################################################################################
# Try to remount the filesystem
#######################################################################################
hal:~$ sudo mount /dev/datavg/backup /backup
mount: wrong fs type, bad option, bad superblock on /dev/mapper/datavg-backup,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

#######################################################################################
# Mount the filesystem read only
#######################################################################################
hal:~$ sudo mount -oro /dev/datavg/backup /backup