#277 Filesystem Expand Clobbered filesystem

open
nobody
utilities (15)
5
2010-01-21
2010-01-21
rwk
No

I attempted to expand a JFS filesystem from 21TB to 43TB using EVMS.
It failed and now my filesystem will not mount. Expand errors:

device-mapper: table: 254:5: mirror: unknown target type
device-mapper: ioctl: error adding target to table
jfs_extendfs: mapSize (0x16448c000) > t64 (0xbacfe000)
ERROR: (device dm-2): jfs_extendfs
JFS: metapage_get_blocks failed
JFS: metapage_get_blocks failed
BUG: Bad page state in process umount pfn:c7c55
page:ffffea0002bb3298 flags:4000000000000808 count:0 mapcount:0 mapping:(null) index:aeadd
Pid: 7487, comm: umount Not tainted 2.6.31-gentoo-r5 #3
Call Trace:
[<ffffffff8108130f>] ? bad_page+0x102/0x113
[<ffffffff810818d9>] ? __pagevec_free+0x29/0x3b
[<ffffffff81084c6b>] ? release_pages+0x14b/0x199
[<ffffffff81085280>] ? __pagevec_release+0x19/0x22
[<ffffffff810857ef>] ? truncate_inode_pages_range+0xe4/0x2e1
[<ffffffff811891a4>] ? dbUnmount+0x3c/0x4b
[<ffffffff81180018>] ? jfs_umount+0x95/0xe5
[<ffffffff8117cafb>] ? jfs_put_super+0x1d/0x76
[<ffffffff810a8b15>] ? generic_shutdown_super+0x4f/0xdf
[<ffffffff810a8bc7>] ? kill_block_super+0x22/0x3a
[<ffffffff810a8e9c>] ? deactivate_super+0x43/0x5a
[<ffffffff810ba77a>] ? sys_umount+0x2a9/0x2d6
[<ffffffff8100adeb>] ? system_call_fastpath+0x16/0x1b
Disabling lock debugging due to kernel taint

fsck.jfs now reports:

fsck.jfs version 1.1.14, 06-Apr-2009
processing started: 1/21/2010 7.55.6
The current device is: /dev/evms/dvds
Block size in bytes: 4096
Filesystem size in blocks: 5855592447
**Phase 0 - Replay Journal Log
**Phase 1 - Check Blocks, Files/Directories, and Directory Entries
Errors detected in the Primary File/Directory Allocation Table.
Errors detected in the Secondary File/Directory Allocation Table.
CANNOT CONTINUE.

EVMS now reports a filesystem size of 43TB, but will not mount saying:

mount: wrong fs type, bad option, bad superblock on /dev/evms/dvds,
missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

Is there any way to recover, or revert back to the 21TB filesystem
before the attempted but failed expand operation?

Discussion

  • Tahoe
    Tahoe
    2011-02-28

    I have found that the problem occurs going from 32 to 33 TB. Has anyone else
    experienced this? I have a 200TB storage array and am currently having to slice it into
    32TB chunks. The following is are recreation of the problem.

    #######################################################################################
    # Ubuntu Version 10.04 LTS
    #######################################################################################

    hal:~$ uname -a
    Linux hal 2.6.32-28-server #55-Ubuntu SMP Mon Jan 10 23:57:16 UTC 2011 x86_64 GNU/Linux

    #######################################################################################
    # JFS Utils Version
    #######################################################################################
    hal:~$ mkfs.jfs
    mkfs.jfs version 1.1.14, 06-Apr-2009

    Error: Device not specified or command format error

    Usage: mkfs.jfs [-cOqV] [-j log_device] [-J options] [-L vol_label] [-s log_size] device [ blocks ]

    Emergency help:
    -c Check device for bad blocks before building file system.
    -O Provide case-insensitive support for OS/2 compatability.
    -q Quiet execution.
    -V Print version information only.
    -j log_device Set external journal device.
    -J options Set external journal options.
    -L vol_label Set volume label for the file system.
    -s log_size Set log size (in megabytes).

    NOTE: -j and -J cannot be used at the same time.

    #######################################################################################
    # Create a 32 TB Logical Volume
    #######################################################################################
    hal:~$ sudo lvcreate -L32T -nbackup datavg
    Logical volume "backup" created

    #######################################################################################
    # Create a 32 TB JFS Filesystem
    #######################################################################################
    hal:~$ sudo mkfs.jfs /dev/datavg/backup
    mkfs.jfs version 1.1.14, 06-Apr-2009
    Warning! All data on device /dev/datavg/backup will be lost!

    Continue? (Y/N) y
    |

    Format completed successfully.

    34359738368 kilobytes total disk space.

    #######################################################################################
    # Do a filesystem check
    #######################################################################################
    hal:~$ sudo fsck.jfs /dev/datavg/backup
    fsck.jfs version 1.1.14, 06-Apr-2009
    processing started: 2/28/2011 20.27.1
    Using default parameter: -p
    The current device is: /dev/datavg/backup
    Block size in bytes: 4096
    Filesystem size in blocks: 8589934592
    **Phase 0 - Replay Journal Log
    Filesystem is clean.

    #######################################################################################
    # Mount the filesystem
    #######################################################################################
    hal:~$ sudo mount /dev/datavg/backup /backup

    #######################################################################################
    # Extend the logical volume by 1 TB
    #######################################################################################
    hal:~$ sudo lvextend -L+1T /dev/datavg/backup
    Extending logical volume backup to 33.00 TiB
    Logical volume backup successfully resized

    #######################################################################################
    # Extend JFS
    #######################################################################################
    hal:~$ sudo mount -o remount,resize /backup
    mount: /dev/mapper/datavg-backup: can't read superblock

    #######################################################################################
    # dmesg
    #######################################################################################
    hal:~$ dmesg | tail
    [ 22.701014] eth0: no IPv6 routers present
    [ 29.512256] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
    [ 29.515458] NFSD: starting 90-second grace period
    [ 149.346690] IPMI System Interface driver.
    [ 149.346695] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
    [ 151.941824] ipmi: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20)
    [ 151.941872] IPMI kcs interface initialized
    [ 973.381244] JFS: nTxBlock = 8192, nTxLock = 65536
    [11894.047996] ERROR: (device dm-1): jfs_extendfs
    [19701.575221] ERROR: (device dm-1): jfs_extendfs

    #######################################################################################
    # Unmount the filesystem and do an fsck
    #######################################################################################
    hal:~$ sudo umount /backup
    hal:~$ sudo fsck.jfs /dev/datavg/backup
    fsck.jfs version 1.1.14, 06-Apr-2009
    processing started: 2/28/2011 20.31.10
    Using default parameter: -p
    The current device is: /dev/datavg/backup
    Block size in bytes: 4096
    Filesystem size in blocks: 8589934592
    **Phase 0 - Replay Journal Log
    **Phase 1 - Check Blocks, Files/Directories, and Directory Entries
    Errors detected in the Primary File/Directory Allocation Table.
    Errors detected in the Secondary File/Directory Allocation Table.
    CANNOT CONTINUE.

    #######################################################################################
    # Try to remount the filesystem
    #######################################################################################
    hal:~$ sudo mount /dev/datavg/backup /backup
    mount: wrong fs type, bad option, bad superblock on /dev/mapper/datavg-backup,
    missing codepage or helper program, or other error
    In some cases useful info is found in syslog - try
    dmesg | tail or so

    #######################################################################################
    # Mount the filesystem read only
    #######################################################################################
    hal:~$ sudo mount -oro /dev/datavg/backup /backup