Re: [Next3-users] Kernel bug..
Brought to you by:
amir73il
Menu
▾
▴
Re: [Next3-users] Kernel bug..
|
From: Kim H. <kh...@mu...> - 2012-04-27 09:46:30
|
Hi Amir, The crash happens every day we create a snapshot it has happened for 3 days straight. The size of our volume is 10TB. /dev/mapper/BackupData-BackupData 9.9T 478G 8.9T 5% --- Logical volume --- LV Name /dev/BackupData/BackupData VG Name BackupData LV UUID jndfk9-GYHb-6aqu-3MKV-fLml-OFd4-3qHJNg LV Write Access read/write LV Status available # open 1 LV Size 10.00 TiB Current LE 2621440 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:1 We have used this volume for 5 days, first 2 days were for testing (without rsync) we also installed next3 at the same time we started using the volume. The output of the fsck... (i had already removed the snapshot yesterday and removed the cron job that takes the snapshots): # fsck.next3 -nf -C0 /dev/mapper/BackupData-BackupData e2fsck 1.41.14-next3-1.0.13-1 (22-Feb-2011) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information BackupData: 519307/671088640 files (3.7% non-contiguous), 167287708/2684354560 blocks This is what is in the logs... (sorry for the length): Apr 26 07:00:02 nl-gouda-07 kernel: [71796.800005] snapshot: next3_snapshot_enable() called with snapshot file (ino=21061634) not on list Apr 26 07:00:02 nl-gouda-07 kernel: [71796.858621] snapshot: snapshot (36) created Apr 26 07:00:02 nl-gouda-07 kernel: [71797.094522] snapshot: snapshot (36) activated Apr 26 07:00:02 nl-gouda-07 kernel: [71797.096916] snapshot: snapshot (36) has been taken Apr 26 07:16:47 nl-gouda-07 kernel: [72802.630978] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:16:48 nl-gouda-07 kernel: [72803.712051] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:16:48 nl-gouda-07 kernel: [72803.773532] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:16:56 nl-gouda-07 kernel: [72810.923815] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:18:17 nl-gouda-07 kernel: [72892.729274] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:28:55 nl-gouda-07 kernel: [73530.588936] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:28:56 nl-gouda-07 kernel: [73530.839443] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:28:56 nl-gouda-07 kernel: [73531.517396] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:28:57 nl-gouda-07 kernel: [73532.444935] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:28:57 nl-gouda-07 kernel: [73532.491089] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:28:57 nl-gouda-07 kernel: [73532.758984] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:28:58 nl-gouda-07 kernel: [73533.479672] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:28:58 nl-gouda-07 kernel: [73533.747038] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:29:00 nl-gouda-07 kernel: [73535.182968] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:41:59 nl-gouda-07 kernel: [74314.708945] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:42:13 nl-gouda-07 kernel: [74328.253262] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:43:40 nl-gouda-07 kernel: [74415.359183] snapshot: next3_snapshot_get_undo_access(b:0/67162) failed! h_ref=1, err=-5 Apr 26 07:43:40 nl-gouda-07 kernel: [74415.359233] next3_try_to_allocate_with_rsv: aborting transaction: IO failure in __next3_journal_get_undo_access Apr 26 07:43:40 nl-gouda-07 kernel: [74415.359283] NEXT3-fs error (device dm-1) in next3_new_blocks: IO failure Apr 26 07:43:40 nl-gouda-07 kernel: [74415.361688] NEXT3-fs error (device dm-1) in next3_reserve_inode_write: Readonly filesystem Apr 26 07:43:40 nl-gouda-07 kernel: [74415.363617] NEXT3-fs error (device dm-1) in next3_dirty_inode: IO failure Apr 26 07:43:40 nl-gouda-07 kernel: [74415.365512] NEXT3-fs error (device dm-1) in next3_inode_list_add: Readonly filesystem Apr 26 07:43:40 nl-gouda-07 kernel: [74415.367397] NEXT3-fs error (device dm-1) in next3_write_begin: IO failure Apr 26 07:43:40 nl-gouda-07 rsyncd[19142]: rsync: write failed on "/de-duisburg-02/file.pdf.restore" (in BackupData): Input/output error (5) Apr 26 07:43:40 nl-gouda-07 rsyncd[19142]: rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.7] Apr 26 07:43:40 nl-gouda-07 kernel: [74415.471732] snapshot: next3_snapshot_get_undo_access(b:0/67162) failed! h_ref=2, err=-5 Apr 26 07:43:40 nl-gouda-07 kernel: [74415.471781] __next3_free_blocks_sb_inode: aborting transaction: IO failure in __next3_journal_get_undo_access Apr 26 07:43:40 nl-gouda-07 kernel: [74415.471831] NEXT3-fs error (device dm-1) in __next3_free_blocks_sb_inode: IO failure Apr 26 07:43:40 nl-gouda-07 kernel: [74415.474006] NEXT3-fs error (device dm-1) in next3_reserve_inode_write: Readonly filesystem Apr 26 07:43:40 nl-gouda-07 kernel: [74415.475918] NEXT3-fs error (device dm-1) in next3_truncate: IO failure Apr 26 07:43:40 nl-gouda-07 kernel: [74415.477827] NEXT3-fs error (device dm-1) in next3_reserve_inode_write: Readonly filesystem Apr 26 07:43:40 nl-gouda-07 kernel: [74415.479708] NEXT3-fs error (device dm-1) in next3_inode_list_del: Readonly filesystem Apr 26 07:43:40 nl-gouda-07 kernel: [74415.481625] NEXT3-fs error (device dm-1) in next3_reserve_inode_write: Readonly filesystem Apr 26 07:43:40 nl-gouda-07 kernel: [74415.483537] NEXT3-fs error (device dm-1) in next3_delete_inode: IO failure Apr 26 07:43:40 nl-gouda-07 rsyncd[19142]: rsync: connection unexpectedly closed (1491 bytes received so far) [generator] Apr 26 07:43:40 nl-gouda-07 rsyncd[19142]: rsync error: error in rsync protocol data stream (code 12) at io.c(601) [generator=3.0.7] Apr 26 07:46:59 nl-gouda-07 kernel: [74614.638221] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:46:59 nl-gouda-07 kernel: [74614.638233] attempt to access beyond end of device Apr 26 07:46:59 nl-gouda-07 kernel: [74614.638237] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:46:59 nl-gouda-07 kernel: [74614.638240] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:46:59 nl-gouda-07 kernel: [74614.638273] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:46:59 nl-gouda-07 kernel: [74614.638319] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:46:59 nl-gouda-07 kernel: [74614.638380] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:46:59 nl-gouda-07 kernel: [74614.640734] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:00 nl-gouda-07 kernel: [74615.254942] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:00 nl-gouda-07 kernel: [74615.254953] attempt to access beyond end of device Apr 26 07:47:00 nl-gouda-07 kernel: [74615.254956] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:00 nl-gouda-07 kernel: [74615.254963] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:47:00 nl-gouda-07 kernel: [74615.254995] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:47:00 nl-gouda-07 kernel: [74615.255041] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:00 nl-gouda-07 kernel: [74615.255089] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:00 nl-gouda-07 kernel: [74615.257408] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:00 nl-gouda-07 kernel: [74615.563027] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:00 nl-gouda-07 kernel: [74615.563037] attempt to access beyond end of device Apr 26 07:47:00 nl-gouda-07 kernel: [74615.563041] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:00 nl-gouda-07 kernel: [74615.563044] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:47:00 nl-gouda-07 kernel: [74615.563077] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:47:00 nl-gouda-07 kernel: [74615.563123] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:00 nl-gouda-07 kernel: [74615.563171] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:00 nl-gouda-07 kernel: [74615.566488] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:01 nl-gouda-07 kernel: [74615.839271] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:01 nl-gouda-07 kernel: [74615.839285] attempt to access beyond end of device Apr 26 07:47:01 nl-gouda-07 kernel: [74615.839288] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:01 nl-gouda-07 kernel: [74615.839293] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:47:01 nl-gouda-07 kernel: [74615.839326] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:47:01 nl-gouda-07 kernel: [74615.839372] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:01 nl-gouda-07 kernel: [74615.839421] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:01 nl-gouda-07 kernel: [74615.843553] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:03 nl-gouda-07 rsyslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="1841" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'. Apr 26 07:47:05 nl-gouda-07 kernel: [74620.302879] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:05 nl-gouda-07 kernel: [74620.302888] attempt to access beyond end of device Apr 26 07:47:05 nl-gouda-07 kernel: [74620.302892] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:05 nl-gouda-07 kernel: [74620.302896] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:47:05 nl-gouda-07 kernel: [74620.302928] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:47:05 nl-gouda-07 kernel: [74620.302975] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:05 nl-gouda-07 kernel: [74620.303023] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:05 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/......etc.Forum.pdf.fkslcK" (in BackupData) failed: Input/output error (5) Apr 26 07:47:05 nl-gouda-07 kernel: [74620.305387] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:05 nl-gouda-07 kernel: [74620.523632] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:05 nl-gouda-07 kernel: [74620.523641] attempt to access beyond end of device Apr 26 07:47:05 nl-gouda-07 kernel: [74620.523645] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:05 nl-gouda-07 kernel: [74620.523649] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:47:05 nl-gouda-07 kernel: [74620.523681] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:47:05 nl-gouda-07 kernel: [74620.523727] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:05 nl-gouda-07 kernel: [74620.523776] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:05 nl-gouda-07 kernel: [74620.525947] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:05 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/....etc.pdf.9RoRu9" (in BackupData) failed: Input/output error (5) Apr 26 07:47:06 nl-gouda-07 kernel: [74621.355528] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:06 nl-gouda-07 kernel: [74621.355537] attempt to access beyond end of device Apr 26 07:47:06 nl-gouda-07 kernel: [74621.355541] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:06 nl-gouda-07 kernel: [74621.355545] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:47:06 nl-gouda-07 kernel: [74621.355577] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:47:06 nl-gouda-07 kernel: [74621.355632] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:06 nl-gouda-07 kernel: [74621.355681] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:06 nl-gouda-07 kernel: [74621.358001] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:06 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.fUXwdB" (in BackupData) failed: Input/output error (5) Apr 26 07:47:06 nl-gouda-07 kernel: [74621.451537] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:06 nl-gouda-07 kernel: [74621.451545] attempt to access beyond end of device Apr 26 07:47:06 nl-gouda-07 kernel: [74621.451548] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:06 nl-gouda-07 kernel: [74621.451552] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:47:06 nl-gouda-07 kernel: [74621.451583] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:47:06 nl-gouda-07 kernel: [74621.451630] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:06 nl-gouda-07 kernel: [74621.451678] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:06 nl-gouda-07 kernel: [74621.453915] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:06 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.pmPwd3" (in BackupData) failed: Input/output error (5) Apr 26 07:47:07 nl-gouda-07 kernel: [74621.899588] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:07 nl-gouda-07 kernel: [74621.899598] attempt to access beyond end of device Apr 26 07:47:07 nl-gouda-07 kernel: [74621.899601] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:07 nl-gouda-07 kernel: [74621.899605] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:47:07 nl-gouda-07 kernel: [74621.899637] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:47:07 nl-gouda-07 kernel: [74621.899683] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:07 nl-gouda-07 kernel: [74621.899731] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:07 nl-gouda-07 kernel: [74621.903637] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:07 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.zWyoww" (in BackupData) failed: Input/output error (5) Apr 26 07:47:07 nl-gouda-07 kernel: [74622.571597] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:07 nl-gouda-07 kernel: [74622.571607] attempt to access beyond end of device Apr 26 07:47:07 nl-gouda-07 kernel: [74622.571610] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:07 nl-gouda-07 kernel: [74622.571614] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:47:07 nl-gouda-07 kernel: [74622.571646] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:47:07 nl-gouda-07 kernel: [74622.571692] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:07 nl-gouda-07 kernel: [74622.571740] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:07 nl-gouda-07 kernel: [74622.574126] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:07 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.ZeByM1" (in BackupData) failed: Input/output error (5) Apr 26 07:47:07 nl-gouda-07 kernel: [74622.739001] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:07 nl-gouda-07 kernel: [74622.739009] attempt to access beyond end of device Apr 26 07:47:07 nl-gouda-07 kernel: [74622.739013] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:07 nl-gouda-07 kernel: [74622.739017] snapshot: failed to read COW bitmap #67168 of snapshot (36) Apr 26 07:47:07 nl-gouda-07 kernel: [74622.745017] snapshot: next3_snapshot_get_write_access(b:1/67168) failed! h_ref=1, err=-5 Apr 26 07:47:07 nl-gouda-07 kernel: [74622.745064] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:07 nl-gouda-07 kernel: [74622.745112] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:07 nl-gouda-07 kernel: [74622.747357] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:07 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.1uWVwx" (in BackupData) failed: Input/output error (5) Apr 26 07:47:08 nl-gouda-07 kernel: [74623.289794] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:08 nl-gouda-07 kernel: [74623.289802] attempt to access beyond end of device Apr 26 07:47:08 nl-gouda-07 kernel: [74623.289805] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:08 nl-gouda-07 kernel: [74623.289809] snapshot: failed to read COW bitmap #67169 of snapshot (36) Apr 26 07:47:08 nl-gouda-07 kernel: [74623.289841] snapshot: next3_snapshot_get_write_access(b:1/67169) failed! h_ref=1, err=-5 Apr 26 07:47:08 nl-gouda-07 kernel: [74623.289888] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:08 nl-gouda-07 kernel: [74623.289936] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:08 nl-gouda-07 kernel: [74623.292109] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:08 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.dwNJO4" (in BackupData) failed: Input/output error (5) Apr 26 07:47:10 nl-gouda-07 kernel: [74625.547394] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:10 nl-gouda-07 kernel: [74625.547405] attempt to access beyond end of device Apr 26 07:47:10 nl-gouda-07 kernel: [74625.547409] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:10 nl-gouda-07 kernel: [74625.547413] snapshot: failed to read COW bitmap #67169 of snapshot (36) Apr 26 07:47:10 nl-gouda-07 kernel: [74625.547445] snapshot: next3_snapshot_get_write_access(b:1/67169) failed! h_ref=1, err=-5 Apr 26 07:47:10 nl-gouda-07 kernel: [74625.547491] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:10 nl-gouda-07 kernel: [74625.547540] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:10 nl-gouda-07 kernel: [74625.550711] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:10 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.xjB0JI" (in BackupData) failed: Input/output error (5) Apr 26 07:47:13 nl-gouda-07 kernel: [74628.131379] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:13 nl-gouda-07 kernel: [74628.131389] attempt to access beyond end of device Apr 26 07:47:13 nl-gouda-07 kernel: [74628.131393] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:13 nl-gouda-07 kernel: [74628.131397] snapshot: failed to read COW bitmap #67169 of snapshot (36) Apr 26 07:47:13 nl-gouda-07 kernel: [74628.131429] snapshot: next3_snapshot_get_write_access(b:1/67169) failed! h_ref=1, err=-5 Apr 26 07:47:13 nl-gouda-07 kernel: [74628.131476] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:13 nl-gouda-07 kernel: [74628.131533] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:13 nl-gouda-07 kernel: [74628.134510] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:13 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.loyFbu" (in BackupData) failed: Input/output error (5) Apr 26 07:47:15 nl-gouda-07 kernel: [74630.386694] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:15 nl-gouda-07 kernel: [74630.386703] attempt to access beyond end of device Apr 26 07:47:15 nl-gouda-07 kernel: [74630.386707] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:15 nl-gouda-07 kernel: [74630.386711] snapshot: failed to read COW bitmap #67169 of snapshot (36) Apr 26 07:47:15 nl-gouda-07 kernel: [74630.386743] snapshot: next3_snapshot_get_write_access(b:1/67169) failed! h_ref=1, err=-5 Apr 26 07:47:15 nl-gouda-07 kernel: [74630.386789] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:15 nl-gouda-07 kernel: [74630.386837] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:15 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.L8Mocm" (in BackupData) failed: Input/output error (5) Apr 26 07:47:15 nl-gouda-07 kernel: [74630.389213] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:20 nl-gouda-07 kernel: [74635.584440] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:20 nl-gouda-07 kernel: [74635.584452] attempt to access beyond end of device Apr 26 07:47:20 nl-gouda-07 kernel: [74635.584455] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:20 nl-gouda-07 kernel: [74635.584460] snapshot: failed to read COW bitmap #67169 of snapshot (36) Apr 26 07:47:20 nl-gouda-07 kernel: [74635.584492] snapshot: next3_snapshot_get_write_access(b:1/67169) failed! h_ref=1, err=-5 Apr 26 07:47:20 nl-gouda-07 kernel: [74635.584539] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:20 nl-gouda-07 kernel: [74635.584587] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:20 nl-gouda-07 kernel: [74635.587525] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:20 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.DjEhlt" (in BackupData) failed: Input/output error (5) Apr 26 07:47:23 nl-gouda-07 kernel: [74638.203074] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:47:23 nl-gouda-07 kernel: [74638.203085] attempt to access beyond end of device Apr 26 07:47:23 nl-gouda-07 kernel: [74638.203088] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:47:23 nl-gouda-07 kernel: [74638.203093] snapshot: failed to read COW bitmap #67169 of snapshot (36) Apr 26 07:47:23 nl-gouda-07 kernel: [74638.203126] snapshot: next3_snapshot_get_write_access(b:1/67169) failed! h_ref=1, err=-5 Apr 26 07:47:23 nl-gouda-07 kernel: [74638.203172] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:47:23 nl-gouda-07 kernel: [74638.203221] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:47:23 nl-gouda-07 kernel: [74638.207136] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:47:23 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.BdBO6H" (in BackupData) failed: Input/output error (5) Apr 26 07:48:57 nl-gouda-07 kernel: [74732.278241] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:48:57 nl-gouda-07 kernel: [74732.278249] attempt to access beyond end of device Apr 26 07:48:57 nl-gouda-07 kernel: [74732.278253] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:48:57 nl-gouda-07 kernel: [74732.278257] snapshot: failed to read COW bitmap #67171 of snapshot (36) Apr 26 07:48:57 nl-gouda-07 kernel: [74732.278289] snapshot: next3_snapshot_get_write_access(b:1/67171) failed! h_ref=1, err=-5 Apr 26 07:48:57 nl-gouda-07 kernel: [74732.278335] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:48:57 nl-gouda-07 kernel: [74732.278383] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:48:57 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.ls.Lj38Ce" (in BackupData) failed: Input/output error (5) Apr 26 07:48:57 nl-gouda-07 kernel: [74732.281008] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:50:10 nl-gouda-07 kernel: [74805.589644] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:50:10 nl-gouda-07 kernel: [74805.589654] attempt to access beyond end of device Apr 26 07:50:10 nl-gouda-07 kernel: [74805.589657] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:50:10 nl-gouda-07 kernel: [74805.589661] snapshot: failed to read COW bitmap #67174 of snapshot (36) Apr 26 07:50:10 nl-gouda-07 kernel: [74805.589694] snapshot: next3_snapshot_get_write_access(b:1/67174) failed! h_ref=1, err=-5 Apr 26 07:50:10 nl-gouda-07 kernel: [74805.589740] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:50:10 nl-gouda-07 kernel: [74805.589788] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:50:10 nl-gouda-07 kernel: [74805.592176] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:50:10 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc (in BackupData) failed: Input/output error (5) Apr 26 07:50:10 nl-gouda-07 kernel: [74805.603200] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:50:10 nl-gouda-07 kernel: [74805.603207] attempt to access beyond end of device Apr 26 07:50:10 nl-gouda-07 kernel: [74805.603210] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:50:10 nl-gouda-07 kernel: [74805.603214] snapshot: failed to read COW bitmap #67174 of snapshot (36) Apr 26 07:50:10 nl-gouda-07 kernel: [74805.603246] snapshot: next3_snapshot_get_write_access(b:1/67174) failed! h_ref=1, err=-5 Apr 26 07:50:10 nl-gouda-07 kernel: [74805.603298] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:50:10 nl-gouda-07 kernel: [74805.603347] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:50:10 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.ls.NiNvWQ" (in BackupData) failed: Input/output error (5) Apr 26 07:50:10 nl-gouda-07 kernel: [74805.605316] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:50:43 nl-gouda-07 kernel: [74838.565081] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:50:43 nl-gouda-07 kernel: [74838.565091] attempt to access beyond end of device Apr 26 07:50:43 nl-gouda-07 kernel: [74838.565094] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:50:43 nl-gouda-07 kernel: [74838.565098] snapshot: failed to read COW bitmap #67173 of snapshot (36) Apr 26 07:50:43 nl-gouda-07 kernel: [74838.565131] snapshot: next3_snapshot_get_write_access(b:1/67173) failed! h_ref=1, err=-5 Apr 26 07:50:43 nl-gouda-07 kernel: [74838.565177] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:50:43 nl-gouda-07 kernel: [74838.565225] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:50:43 nl-gouda-07 kernel: [74838.567610] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:50:43 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.pdf.9NvFJ9" (in BackupData) failed: Input/output error (5) Apr 26 07:50:44 nl-gouda-07 kernel: [74838.797944] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:50:44 nl-gouda-07 kernel: [74838.797952] attempt to access beyond end of device Apr 26 07:50:44 nl-gouda-07 kernel: [74838.797955] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:50:44 nl-gouda-07 kernel: [74838.797959] snapshot: failed to read COW bitmap #67173 of snapshot (36) Apr 26 07:50:44 nl-gouda-07 kernel: [74838.797991] snapshot: next3_snapshot_get_write_access(b:1/67173) failed! h_ref=1, err=-5 Apr 26 07:50:44 nl-gouda-07 kernel: [74838.798037] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:50:44 nl-gouda-07 kernel: [74838.798085] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:50:44 nl-gouda-07 kernel: [74838.800157] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:50:44 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc failed: Input/output error (5) Apr 26 07:50:44 nl-gouda-07 kernel: [74838.802769] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 07:50:44 nl-gouda-07 kernel: [74838.802777] attempt to access beyond end of device Apr 26 07:50:44 nl-gouda-07 kernel: [74838.802780] dm-1: rw=0, want=23732295768, limit=21474836480 Apr 26 07:50:44 nl-gouda-07 kernel: [74838.802784] snapshot: failed to read COW bitmap #67173 of snapshot (36) Apr 26 07:50:44 nl-gouda-07 kernel: [74838.802816] snapshot: next3_snapshot_get_write_access(b:1/67173) failed! h_ref=1, err=-5 Apr 26 07:50:44 nl-gouda-07 kernel: [74838.802862] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 07:50:44 nl-gouda-07 kernel: [74838.802910] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 07:50:44 nl-gouda-07 kernel: [74838.805242] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 07:50:44 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc.ppt.hcFvdt" (in BackupData) failed: Input/output error (5) Apr 26 08:01:42 nl-gouda-07 kernel: [75497.760927] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:06:20 nl-gouda-07 kernel: [75774.795683] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:08:33 nl-gouda-07 kernel: [75907.799568] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:08:58 nl-gouda-07 kernel: [75933.145117] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:09:05 nl-gouda-07 kernel: [75939.889330] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:09:12 nl-gouda-07 kernel: [75947.223080] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:09:25 nl-gouda-07 kernel: [75960.531557] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:09:43 nl-gouda-07 kernel: [75978.034688] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:09:50 nl-gouda-07 kernel: [75985.114594] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:09:50 nl-gouda-07 kernel: [75985.296441] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497242] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497254] ------------[ cut here ]------------ Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497275] WARNING: at /usr/src/next3-1.0.13/next3/snapshot.c:486 next3_snapshot_test_cow_bitmap+0x208/0x415 [next3]() Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497278] Hardware name: ProLiant DL380 G5 Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497279] Modules linked in: hcpdriver(P) cpufreq_conservative cpufreq_powersave cpufreq_userspace cpufreq_stats dm_round_robin dm_multipath scsi_dh loop radeon ttm drm_kms _helper drm i2c_algo_bit i2c_core i5000_edac snd_pcm snd_timer snd soundcore snd_page_alloc ipmi_si pcspkr psmouse ipmi_msghandler edac_core shpchp hpilo serio_raw hpwdt i5k_amb pci_hotplug joydev rng_core evdev c ontainer button processor ext4 jbd2 crc16 dm_mod raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 raid0 multipath linear md_mod ext3 next3 mbcache jbd ses enclosure sd_m od crc_t10dif usbhid hid sg sr_mod cdrom ata_generic hpsa thermal ata_piix uhci_hcd mptspi scsi_transport_spi mptsas cciss libata bnx2 mptscsih mptbase ehci_hcd scsi_transport_sas scsi_mod thermal_sys usbcore nls_ base [last unloaded: scsi_wait_scan] Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497345] Pid: 27670, comm: rsync Tainted: P 2.6.32-5-amd64 #1 Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497347] Call Trace: Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497355] [<ffffffffa018fd2b>] ? next3_snapshot_test_cow_bitmap+0x208/0x415 [next3] Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497362] [<ffffffffa018fd2b>] ? next3_snapshot_test_cow_bitmap+0x208/0x415 [next3] Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497368] [<ffffffff8104df9c>] ? warn_slowpath_common+0x77/0xa3 Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497375] [<ffffffffa018fd2b>] ? next3_snapshot_test_cow_bitmap+0x208/0x415 [next3] Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497381] [<ffffffffa01693c9>] ? do_get_write_access+0x3e3/0x441 [jbd] Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497389] [<ffffffffa01907ba>] ? next3_snapshot_test_and_cow+0x2ac/0x616 [next3] Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497397] [<ffffffffa018f336>] ? __next3_journal_get_write_access_inode+0x40/0x8f [next3] Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497404] [<ffffffffa017e33d>] ? next3_new_inode+0x4f5/0x9f1 [next3] Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497409] [<ffffffffa0169ba0>] ? journal_start+0x96/0xc6 [jbd] Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497416] [<ffffffffa01874a4>] ? next3_create+0x96/0x113 [next3] Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497421] [<ffffffff810f6448>] ? generic_permission+0xe/0x8a Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497424] [<ffffffff810f7f44>] ? vfs_create+0x6d/0x89 Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497427] [<ffffffff810f8e6d>] ? do_filp_open+0x31e/0x94b Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497432] [<ffffffff810ed563>] ? do_sys_open+0x55/0xfc Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497436] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497438] ---[ end trace 90d840e343cd0075 ]--- Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497441] snapshot: failed to read COW bitmap #75360 of snapshot (36) Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497474] snapshot: next3_snapshot_get_write_access(b:1/75360) failed! h_ref=1, err=-5 Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497520] next3_new_inode: aborting transaction: IO failure in __next3_journal_get_write_access_inode Apr 26 08:09:52 nl-gouda-07 kernel: [75987.497568] NEXT3-fs error (device dm-1) in next3_new_inode: IO failure Apr 26 08:09:52 nl-gouda-07 kernel: [75987.499968] NEXT3-fs error (device dm-1) in next3_create: IO failure Apr 26 08:09:52 nl-gouda-07 rsyncd[27657]: rsync: mkstemp "/de-duisburg-02/.....etc failed: Input/output error (5) Apr 26 08:09:52 nl-gouda-07 kernel: [75987.723775] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:09:53 nl-gouda-07 kernel: [75987.809208] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:10:03 nl-gouda-07 kernel: [75998.282167] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.925187] NEXT3-fs warning (device dm-1): next3_block_to_path: block < 0 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.970342] ------------[ cut here ]------------ Apr 26 08:10:05 nl-gouda-07 kernel: [75999.970376] kernel BUG at /usr/src/next3-1.0.13/next3/snapshot.c:1063! Apr 26 08:10:05 nl-gouda-07 kernel: [75999.970405] invalid opcode: 0000 [#1] SMP Apr 26 08:10:05 nl-gouda-07 kernel: [75999.970434] last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:09:00.0/0000:0a:01.0/0000:0e:00.0/host3/port-3:0/end_device-3:0/target3:0:0/3:0:0:1/state Apr 26 08:10:05 nl-gouda-07 kernel: [75999.970504] CPU 1 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.970526] Modules linked in: hcpdriver(P) cpufreq_conservative cpufreq_powersave cpufreq_userspace cpufreq_stats dm_round_robin dm_multipath scsi_dh loop radeon ttm drm_kms _helper drm i2c_algo_bit i2c_core i5000_edac snd_pcm snd_timer snd soundcore snd_page_alloc ipmi_si pcspkr psmouse ipmi_msghandler edac_core shpchp hpilo serio_raw hpwdt i5k_amb pci_hotplug joydev rng_core evdev c ontainer button processor ext4 jbd2 crc16 dm_mod raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 raid0 multipath linear md_mod ext3 next3 mbcache jbd ses enclosure sd_m od crc_t10dif usbhid hid sg sr_mod cdrom ata_generic hpsa thermal ata_piix uhci_hcd mptspi scsi_transport_spi mptsas cciss libata bnx2 mptscsih mptbase ehci_hcd scsi_transport_sas scsi_mod thermal_sys usbcore nls_ base [last unloaded: scsi_wait_scan] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.970974] Pid: 27670, comm: rsync Tainted: P W 2.6.32-5-amd64 #1 ProLiant DL380 G5 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971020] RIP: 0010:[<ffffffffa0190335>] [<ffffffffa0190335>] next3_snapshot_test_and_move+0x8b/0x264 [next3] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971081] RSP: 0018:ffff88012c541888 EFLAGS: 00010202 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971108] RAX: ffff88012b839000 RBX: 0000000000000000 RCX: 0000000082200883 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971138] RDX: ffff88010e39cba8 RSI: ffff8801289fc6f0 RDI: ffffffffa0197af0 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971168] RBP: ffff88010e39cba8 R08: 0000000000000001 R09: 0000000000000001 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971198] R10: 0000000000000004 R11: dead000000100100 R12: 0000000000000001 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971228] R13: ffff8801289fc6f0 R14: ffff88010e39cba8 R15: 0000000000000001 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971258] FS: 00007f82d6344700(0000) GS:ffff880005440000(0000) knlGS:0000000000000000 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971304] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971331] CR2: 00007fff137bd428 CR3: 000000012c534000 CR4: 00000000000006e0 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971361] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971391] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971422] Process rsync (pid: 27670, threadinfo ffff88012c540000, task ffff88012b918000) Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971475] Stack: Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971499] ffffffff81065058 0000000000001000 0000000082200883 ffffffffa0197af0 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971533] <0> ffff88012b838000 ffff88012c5419e8 0000000000000000 ffff880075670e00 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971585] <0> ffff880075670070 ffff88012c5419e8 ffff88010e39cba8 0000000000000004 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971653] Call Trace: Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971679] [<ffffffff81065058>] ? wake_bit_function+0x0/0x23 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971713] [<ffffffffa0181cf3>] ? next3_get_blocks_handle+0xc06/0x1127 [next3] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971762] [<ffffffffa01825f3>] ? next3_getblk+0x53/0x172 [next3] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971793] [<ffffffff8110da89>] ? __find_get_block+0x176/0x186 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971824] [<ffffffff810168f3>] ? read_tsc+0xa/0x20 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971855] [<ffffffffa018fd09>] ? next3_snapshot_test_cow_bitmap+0x1e6/0x415 [next3] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971904] [<ffffffffa01693c9>] ? do_get_write_access+0x3e3/0x441 [jbd] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971936] [<ffffffff812fbf38>] ? __wait_on_bit+0x62/0x70 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.971967] [<ffffffffa01907ba>] ? next3_snapshot_test_and_cow+0x2ac/0x616 [next3] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972019] [<ffffffffa018f336>] ? __next3_journal_get_write_access_inode+0x40/0x8f [next3] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972069] [<ffffffffa017e33d>] ? next3_new_inode+0x4f5/0x9f1 [next3] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972101] [<ffffffffa0169ba0>] ? journal_start+0x96/0xc6 [jbd] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972133] [<ffffffffa01874a4>] ? next3_create+0x96/0x113 [next3] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972163] [<ffffffff810f6448>] ? generic_permission+0xe/0x8a Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972192] [<ffffffff810f7f44>] ? vfs_create+0x6d/0x89 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972220] [<ffffffff810f8e6d>] ? do_filp_open+0x31e/0x94b Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972250] [<ffffffff810e6f20>] ? __slab_alloc+0xd3/0x39b Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972278] [<ffffffff810f9502>] ? getname+0x23/0x1a0 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972306] [<ffffffff810ed563>] ? do_sys_open+0x55/0xfc Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972334] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972361] Code: 1e 44 89 0c 24 48 8b 54 24 20 45 31 c0 4c 8b 4c 24 10 48 8b 7c 24 18 48 89 e9 e8 02 f1 ff ff 41 f6 45 14 08 75 05 4c 39 f5 75 04 <0f> 0b eb fe 4c 89 ef e8 8 3 f1 ff ff 31 c0 48 85 ed 74 68 0f b7 Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972563] RIP [<ffffffffa0190335>] next3_snapshot_test_and_move+0x8b/0x264 [next3] Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972613] RSP <ffff88012c541888> Apr 26 08:10:05 nl-gouda-07 kernel: [75999.972909] ---[ end trace 90d840e343cd0076 ]--- Apr 26 08:10:05 nl-gouda-07 rsyncd[27657]: rsync: connection unexpectedly closed (318 bytes received so far) [generator] Apr 26 08:10:05 nl-gouda-07 rsyncd[27657]: rsync error: error in rsync protocol data stream (code 12) at io.c(601) [generator=3.0.7] Regards, Kim. From: Amir Goldstein <ami...@gm...> To: Kim Hagen <kh...@mu...>, Cc: nex...@li... Date: 04/26/2012 06:56 PM Subject: Re: [Next3-users] Kernel bug.. Hi Kim, Did you see any other snapshot related errors/warning in the kernel log prior to this crash? Is this crash reproducible? What is the size of the volume/fs you are using? Have you been using this volume for a long time with snapshots? Can you please run fsck.next3 -nf to see if there are snapshot consistency errors on your fs. Thanks, Amir. On Thu, Apr 26, 2012 at 3:06 PM, Kim Hagen <kh...@mu...> wrote: Hi Amir, The kernel version is: root@nl-gouda-07:~# uname -a Linux nl-gouda-07 2.6.32-5-amd64 #1 SMP Thu Mar 22 17:26:33 UTC 2012 x86_64 GNU/Linux next3 version is: root@nl-gouda-07:~# next3 version snapshot-version = next3 snapshot v1.0.13-6 (2-May-2010) Kim From: Amir Goldstein <ami...@gm...> To: Kim Hagen <kh...@mu...>, Cc: nex...@li... Date: 04/26/2012 12:30 PM Subject: Re: [Next3-users] Kernel bug.. Hi Kim, Can you please specify kernel version (uname -a) and next3 version by Running 'next3 version ' command Thanks, Amir Sent from my private persona, enabled by Cellrox On Apr 26, 2012 12:45 PM, "Kim Hagen" <kh...@mu...> wrote: Hi, Im experiencing a kernel bug on making a snapshot, I really like to use next3 so i hope it can be fixed. >From what I can see rsync has something to do with it. I am using Debian Squeeze on a ProLiant DL380 G5, the file system is on top of an LVM volume on a multipath MSA2012sa device. [75999.970342] ------------[ cut here ]------------ [75999.970376] kernel BUG at /usr/src/next3-1.0.13/next3/snapshot.c:1063! [75999.970405] invalid opcode: 0000 [#1] SMP [75999.970434] last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:09:00.0/0000:0a:01.0/0000:0e:00.0/host3/port-3:0/end_device-3:0/target3:0:0/3:0:0:1/state [75999.970504] CPU 1 [75999.970526] Modules linked in: hcpdriver(P) cpufreq_conservative cpufreq_powersave cpufreq_userspace cpufreq_stats dm_round_robin dm_multipath scsi_dh loop radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core i5000_edac snd_pcm snd_timer snd soundcore snd_page_alloc ipmi_si pcspkr psmouse ipmi_msghandler edac_core shpchp hpilo serio_raw hpwdt i5k_amb pci_hotplug joydev rng_core evdev container button processor ext4 jbd2 crc16 dm_mod raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 raid0 multipath linear md_mod ext3 next3 mbcache jbd ses enclosure sd_mod crc_t10dif usbhid hid sg sr_mod cdrom ata_generic hpsa thermal ata_piix uhci_hcd mptspi scsi_transport_spi mptsas cciss libata bnx2 mptscsih mptbase ehci_hcd scsi_transport_sas scsi_mod thermal_sys usbcore nls_base [last unloaded: scsi_wait_scan] [75999.970974] Pid: 27670, comm: rsync Tainted: P W 2.6.32-5-amd64 #1 ProLiant DL380 G5 [75999.971020] RIP: 0010:[<ffffffffa0190335>] [<ffffffffa0190335>] next3_snapshot_test_and_move+0x8b/0x264 [next3] [75999.971081] RSP: 0018:ffff88012c541888 EFLAGS: 00010202 [75999.971108] RAX: ffff88012b839000 RBX: 0000000000000000 RCX: 0000000082200883 [75999.971138] RDX: ffff88010e39cba8 RSI: ffff8801289fc6f0 RDI: ffffffffa0197af0 [75999.971168] RBP: ffff88010e39cba8 R08: 0000000000000001 R09: 0000000000000001 [75999.971198] R10: 0000000000000004 R11: dead000000100100 R12: 0000000000000001 [75999.971228] R13: ffff8801289fc6f0 R14: ffff88010e39cba8 R15: 0000000000000001 [75999.971258] FS: 00007f82d6344700(0000) GS:ffff880005440000(0000) knlGS:0000000000000000 [75999.971304] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [75999.971331] CR2: 00007fff137bd428 CR3: 000000012c534000 CR4: 00000000000006e0 [75999.971361] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [75999.971391] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [75999.971422] Process rsync (pid: 27670, threadinfo ffff88012c540000, task ffff88012b918000) [75999.971475] Stack: [75999.971499] ffffffff81065058 0000000000001000 0000000082200883 ffffffffa0197af0 [75999.971533] <0> ffff88012b838000 ffff88012c5419e8 0000000000000000 ffff880075670e00 [75999.971585] <0> ffff880075670070 ffff88012c5419e8 ffff88010e39cba8 0000000000000004 [75999.971653] Call Trace: [75999.971679] [<ffffffff81065058>] ? wake_bit_function+0x0/0x23 [75999.971713] [<ffffffffa0181cf3>] ? next3_get_blocks_handle+0xc06/0x1127 [next3] [75999.971762] [<ffffffffa01825f3>] ? next3_getblk+0x53/0x172 [next3] [75999.971793] [<ffffffff8110da89>] ? __find_get_block+0x176/0x186 [75999.971824] [<ffffffff810168f3>] ? read_tsc+0xa/0x20 [75999.971855] [<ffffffffa018fd09>] ? next3_snapshot_test_cow_bitmap+0x1e6/0x415 [next3] [75999.971904] [<ffffffffa01693c9>] ? do_get_write_access+0x3e3/0x441 [jbd] [75999.971936] [<ffffffff812fbf38>] ? __wait_on_bit+0x62/0x70 [75999.971967] [<ffffffffa01907ba>] ? next3_snapshot_test_and_cow+0x2ac/0x616 [next3] [75999.972019] [<ffffffffa018f336>] ? __next3_journal_get_write_access_inode+0x40/0x8f [next3] [75999.972069] [<ffffffffa017e33d>] ? next3_new_inode+0x4f5/0x9f1 [next3] [75999.972101] [<ffffffffa0169ba0>] ? journal_start+0x96/0xc6 [jbd] [75999.972133] [<ffffffffa01874a4>] ? next3_create+0x96/0x113 [next3] [75999.972163] [<ffffffff810f6448>] ? generic_permission+0xe/0x8a [75999.972192] [<ffffffff810f7f44>] ? vfs_create+0x6d/0x89 [75999.972220] [<ffffffff810f8e6d>] ? do_filp_open+0x31e/0x94b [75999.972250] [<ffffffff810e6f20>] ? __slab_alloc+0xd3/0x39b [75999.972278] [<ffffffff810f9502>] ? getname+0x23/0x1a0 [75999.972306] [<ffffffff810ed563>] ? do_sys_open+0x55/0xfc [75999.972334] [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b [75999.972361] Code: 1e 44 89 0c 24 48 8b 54 24 20 45 31 c0 4c 8b 4c 24 10 48 8b 7c 24 18 48 89 e9 e8 02 f1 ff ff 41 f6 45 14 08 75 05 4c 39 f5 75 04 <0f> 0b eb fe 4c 89 ef e8 83 f1 ff ff 31 c0 48 85 ed 74 68 0f b7 [75999.972563] RIP [<ffffffffa0190335>] next3_snapshot_test_and_move+0x8b/0x264 [next3] [75999.972613] RSP <ffff88012c541888> [75999.972909] ---[ end trace 90d840e343cd0076 ]--- Kind regards, Kim Hagen This message contains information that may be priviliged or confidential and is intended only for use by the addressee(s). You are not authorized to use or distribute this message or any part of it in any form, unless you are the intended recipient. If you receive this communication unintentionally or in error, please inform the sender immediately by return e-mail and delete all copies of this message. If you do not want to receive any e-mail from Multi Corporation and/or its affiliates in the future, please send an e-mail to: no...@mu... Multi Corporation B.V., Gouda, is registered with the trade register in the Netherlands under no. 34237857. ------------------------------------------------------------------------------------------------------------------------------- Dit bericht bevat informatie die vertrouwelijk kan zijn en is uitsluitend bestemd voor de geadresseerde(n). Het is u niet toegestaan dit bericht of delen ervan te gebruiken of te verspreiden in welke vorm dan ook, tenzij u de beoogde ontvanger bent. Indien u dit bericht abusievelijk of onterecht ontvangt, wordt u verzocht de afzender direkt te informeren door het bericht te retourneren en alle kopieen van dit bericht direkt te verwijderen. Indien u geen email meer wilt ontvangen van Multi Corporation of haar groepsvennootschappen kunt u dit kenbaar maken door een email te sturen naar : no...@mu... Multi Corporation B.V., Gouda, staat ingeschreven in de Kamer van Koophandel onder nr. 34237857. ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Next3-users mailing list Nex...@li... https://lists.sourceforge.net/lists/listinfo/next3-users This message contains information that may be priviliged or confidential and is intended only for use by the addressee(s). You are not authorized to use or distribute this message or any part of it in any form, unless you are the intended recipient. If you receive this communication unintentionally or in error, please inform the sender immediately by return e-mail and delete all copies of this message. If you do not want to receive any e-mail from Multi Corporation and/or its affiliates in the future, please send an e-mail to: no...@mu... Multi Corporation B.V., Gouda, is registered with the trade register in the Netherlands under no. 34237857. ------------------------------------------------------------------------------------------------------------------------------- Dit bericht bevat informatie die vertrouwelijk kan zijn en is uitsluitend bestemd voor de geadresseerde(n). Het is u niet toegestaan dit bericht of delen ervan te gebruiken of te verspreiden in welke vorm dan ook, tenzij u de beoogde ontvanger bent. Indien u dit bericht abusievelijk of onterecht ontvangt, wordt u verzocht de afzender direkt te informeren door het bericht te retourneren en alle kopieen van dit bericht direkt te verwijderen. Indien u geen email meer wilt ontvangen van Multi Corporation of haar groepsvennootschappen kunt u dit kenbaar maken door een email te sturen naar : no...@mu... Multi Corporation B.V., Gouda, staat ingeschreven in de Kamer van Koophandel onder nr. 34237857. ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Next3-users mailing list Nex...@li... https://lists.sourceforge.net/lists/listinfo/next3-users This message contains information that may be priviliged or confidential and is intended only for use by the addressee(s). You are not authorized to use or distribute this message or any part of it in any form, unless you are the intended recipient. If you receive this communication unintentionally or in error, please inform the sender immediately by return e-mail and delete all copies of this message. If you do not want to receive any e-mail from Multi Corporation and/or its affiliates in the future, please send an e-mail to: no...@mu... Multi Corporation B.V., Gouda, is registered with the trade register in the Netherlands under no. 34237857. ------------------------------------------------------------------------------------------------------------------------------- Dit bericht bevat informatie die vertrouwelijk kan zijn en is uitsluitend bestemd voor de geadresseerde(n). Het is u niet toegestaan dit bericht of delen ervan te gebruiken of te verspreiden in welke vorm dan ook, tenzij u de beoogde ontvanger bent. Indien u dit bericht abusievelijk of onterecht ontvangt, wordt u verzocht de afzender direkt te informeren door het bericht te retourneren en alle kopieen van dit bericht direkt te verwijderen. Indien u geen email meer wilt ontvangen van Multi Corporation of haar groepsvennootschappen kunt u dit kenbaar maken door een email te sturen naar : no...@mu... Multi Corporation B.V., Gouda, staat ingeschreven in de Kamer van Koophandel onder nr. 34237857. |
