From: SourceForge.net <no...@so...> - 2006-10-19 19:10:58
|
Bugs item #1569947, was opened at 2006-10-03 05:48 Message generated for change (Comment added) made by nobody You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=622063&aid=1569947&group_id=98788 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Markus Müller (privi) Assigned to: Nobody/Anonymous (nobody) Summary: Data corruption on md/raid5 under 0.6.3 and also 0.7.1-hn14 Initial Comment: There is somewhere a problem in the coLinux System and/or kernel about md and/or raid5. There is data corruption and also filesystemerrors. Sometimes also kernel panics occure. If you, for example, write 1 GB it is in most cases damaged. The data corruption increases highly, if the raid is rebuilding; if there is no rebuild, every second file of 1 GB is not damaged. If there is rebuild running, there are kernel panics and data corruptions on any files you write which are more than 20 MB. The panics mostly occures if you then read the file, on writing there is no hint at all that something bad is going on. On reading there are often messages like "attempt to access beyond end of device md0: rw=0, want=18446744061337158144, limit=4367491072" The problem occures if you map the cobds via file (cobd1=M:\data.img\ncobd2=N:\data.img\n...), and also via block device (cobd1=\Device\HarddiskVolume1,...): The problem occures in 0.6.3 as well as in 0.7.1-hn14, I reproduced it on both. The Problem does NOT occure if you simply create files via dd on a mounted (cobd-)device. The problem is also NOT reproduceable if you run the raid under native linux, without colinux but in real linux. So there is NOT a Hardware problem! I have here hardware to test, I simply need assistence what to test to determine the problem. Please give input to col...@pr.... Produced with: ============== chieftec:/mnt/md0# while true; do for i in 1 2 3; do dd_rescue -m 1G /dev/zero ./f.$i; done; for i in 1 2 3; do ls -la ./f.$i; md5sum ./f.$i; done; for i in 1 2 3; do rm ./f.$i; done; done dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 5564kB/s, avg.rate: 8975kB/s, avg.load: 18.5% Summary for /dev/zero -> ./f.1: dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 0kB/s, avg.rate: 8975kB/s, avg.load: 18.5% dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 5759kB/s, avg.rate: 7320kB/s, avg.load: 15.7% Summary for /dev/zero -> ./f.2: dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 0kB/s, avg.rate: 7320kB/s, avg.load: 15.7% dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 5602kB/s, avg.rate: 7194kB/s, avg.load: 15.9% Summary for /dev/zero -> ./f.3: dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 0kB/s, avg.rate: 7194kB/s, avg.load: 15.9% -rw-r----- 1 root root 1073741824 Oct 3 14:17 ./f.1 md5sum: ./f.1: Input/output error -rw-r----- 1 root root 1073741824 Oct 3 14:20 ./f.2 2d261173f8d9f9b8d4257df01d51be48 ./f.2 -rw-r----- 1 root root 1073741824 Oct 3 14:22 ./f.3 md5sum: ./f.3: Input/output error Segmentation fault Kernel Panic: ============= md: using 128k window, over a total of 311963648 blocks. md: md0: sync done. md: md0 stopped. md: unbind<cobd8> md: export_rdev(cobd8) md: unbind<cobd7> md: export_rdev(cobd7) md: unbind<cobd6> md: export_rdev(cobd6) md: unbind<cobd5> md: export_rdev(cobd5) md: unbind<cobd4> md: export_rdev(cobd4) md: unbind<cobd3> md: export_rdev(cobd3) md: unbind<cobd2> md: export_rdev(cobd2) md: unbind<cobd1> md: export_rdev(cobd1) md: bind<cobd1> md: bind<cobd2> md: bind<cobd3> md: bind<cobd4> md: bind<cobd5> md: bind<cobd6> md: bind<cobd7> md: bind<cobd8> raid5: device cobd7 operational as raid disk 6 raid5: device cobd6 operational as raid disk 5 raid5: device cobd5 operational as raid disk 4 raid5: device cobd4 operational as raid disk 3 raid5: device cobd3 operational as raid disk 2 raid5: device cobd2 operational as raid disk 1 raid5: device cobd1 operational as raid disk 0 raid5: allocated 8374kB for md0 raid5: raid level 5 set md0 active with 7 out of 8 devices, algorithm 2 RAID5 conf printout: --- rd:8 wd:7 fd:1 disk 0, o:1, dev:cobd1 disk 1, o:1, dev:cobd2 disk 2, o:1, dev:cobd3 disk 3, o:1, dev:cobd4 disk 4, o:1, dev:cobd5 disk 5, o:1, dev:cobd6 disk 6, o:1, dev:cobd7 RAID5 conf printout: --- rd:8 wd:7 fd:1 disk 0, o:1, dev:cobd1 disk 1, o:1, dev:cobd2 disk 2, o:1, dev:cobd3 disk 3, o:1, dev:cobd4 disk 4, o:1, dev:cobd5 disk 5, o:1, dev:cobd6 disk 6, o:1, dev:cobd7 disk 7, o:1, dev:cobd8 .<6>md: syncing RAID array md0 md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc. md: using maximum available idle IO bandwith (but not more than 200000 KB/sec) for reconstruction. md: using 128k window, over a total of 311963648 blocks. ReiserFS: dm-9: found reiserfs format "3.6" with standard journal ReiserFS: dm-9: using ordered data mode ReiserFS: dm-9: journal params: device dm-9, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30 ReiserFS: dm-9: checking transaction log (dm-9) ReiserFS: dm-9: Using r5 hash to sort names attempt to access beyond end of device dm-9: rw=0, want=11015784704, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=18446744068373508432, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=6890187672, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=16134801512, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=18446744062966200928, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=18446744068915172520, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=4661400440, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=11234349824, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=18446744070176263304, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=13845667976, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=18446744070797588552, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=8587809376, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=13814691040, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=8825944912, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=15052302704, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=6184617992, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=18446744061014929280, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=15635940952, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=18446744073305828400, limit=4367491072 attempt to access beyond end of device dm-9: rw=0, want=11015784704, limit=4367491072 Buffer I/O error on device dm-9, logical block 1376973087 attempt to access beyond end of device dm-9: rw=0, want=11015784704, limit=4367491072 Buffer I/O error on device dm-9, logical block 1376973087 ReiserFS: md0: found reiserfs format "3.6" with standard journal ReiserFS: md0: using ordered data mode ReiserFS: md0: journal params: device md0, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30 ReiserFS: md0: checking transaction log (md0) ReiserFS: md0: Using r5 hash to sort names attempt to access beyond end of device md0: rw=0, want=8341820328, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744063147890160, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744063115474816, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744069584729480, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744063052644416, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=16766954656, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744059358993104, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744059733663912, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744063091516640, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744061337158144, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=12829122048, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=12804994112, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744058686830464, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=8341820328, limit=4367491072 Buffer I/O error on device md0, logical block 1042727540 attempt to access beyond end of device md0: rw=0, want=8341820328, limit=4367491072 Buffer I/O error on device md0, logical block 1042727540 attempt to access beyond end of device md0: rw=0, want=8341820328, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744063147890160, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744063115474816, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744069584729480, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744063052644416, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=16766954656, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744059358993104, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744059733663912, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744063091516640, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744061337158144, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=12829122048, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=12804994112, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=18446744058686830464, limit=4367491072 attempt to access beyond end of device md0: rw=0, want=8341820328, limit=4367491072 Buffer I/O error on device md0, logical block 1042727540 attempt to access beyond end of device md0: rw=0, want=8341820328, limit=4367491072 Buffer I/O error on device md0, logical block 1042727540 ReiserFS: md0: warning: vs-4075: reiserfs_free_block: block 2417127151 is out of range on md0 ReiserFS: md0: warning: vs-4075: reiserfs_free_block: block 1600624263 is out of range on md0 ReiserFS: md0: warning: vs-4075: reiserfs_free_block: block 1603640255 is out of range on md0 ReiserFS: md0: warning: vs-4075: reiserfs_free_block: block 2748418111 is out of range on md0 ReiserFS: md0: warning: vs-4075: reiserfs_free_block: block 2967712923 is out of range on md0 ReiserFS: md0: warning: vs-4080: reiserfs_free_block: free_block (md0:255068793)[dev:blocknr]: bit already cleared ReiserFS: md0: warning: vs-4075: reiserfs_free_block: block 2547981332 is out of range on md0 ReiserFS: md0: warning: vs-4075: reiserfs_free_block: block 2501147481 is out of range on md0 ReiserFS: md0: warning: vs-4080: reiserfs_free_block: free_block (md0:255815333)[dev:blocknr]: bit already cleared ReiserFS: md0: warning: vs-4080: reiserfs_free_block: free_block (md0:532900159)[dev:blocknr]: bit already cleared Unable to handle kernel NULL pointer dereference at virtual address 00000db0 printing eip: c01a9c51 *pde = 00000000 Oops: 0002 [#1] PREEMPT Modules linked in: CPU: 0 EIP: 0060:[<c01a9c51>] Not tainted VLI EFLAGS: 00010286 (2.6.12-co-default) EIP is at set_bit_in_list_bitmap+0x31/0x60 eax: 00000000 ebx: 0000f9d8 ecx: d6918c00 edx: e0a3c724 esi: 00006d93 edi: e09c80f4 ebp: d5867b4c esp: d5867b40 ds: 007b es: 007b ss: 0068 Process rm (pid: 1414, threadinfo=d5866000 task=deb3ba40) Stack: d5867f0c 7cec6d93 d6918c00 d5867b78 c01aeafc d6918c00 7cec6d93 e09c80f4 0006a402 00000000 e09c8000 d5867f0c d6918c00 7cec6d93 d5867ba0 c018a54b d5867f0c d6918c00 7cec6d93 00000001 dde1107c d5867bec 000000fa c3cc6c14 Call Trace: [<c0103c15>] show_stack+0x75/0x90 [<c0103d6b>] show_registers+0x11b/0x190 [<c0103f55>] die+0xd5/0x170 [<c010ee02>] do_page_fault+0x422/0x70c [<c010381f>] error_code+0x4f/0x60 [<c01aeafc>] journal_mark_freed+0xac/0x230 [<c018a54b>] reiserfs_free_block+0x2b/0x70 [<c01a6dcc>] prepare_for_delete_or_cut+0x55c/0x750 [<c01a7f68>] reiserfs_cut_from_item+0xb8/0x500 [<c01a8671>] reiserfs_do_truncate+0x231/0x560 [<c01a799b>] reiserfs_delete_object+0x2b/0x80 [<c01912ff>] reiserfs_delete_inode+0x6f/0xd0 [<c016d6f3>] generic_delete_inode+0x93/0x140 [<c016358c>] sys_unlink+0xbc/0x110 [<c0102849>] syscall_call+0x7/0xb Code: 56 53 8b 4d 08 8b 7d 10 8b 41 0c 8d 1c c5 00 00 00 00 8b 45 0c f7 f3 89 c3 8b 47 04 89 d6 8b 14 98 85 d2 74 15 8b 04 98 8b 40 04 <0f> ab 30 31 c0 8d 65 f4 5b 5e 5f 5d c3 89 f6 51 e8 aa fe ff ff ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2006-10-19 12:10 Message: Logged In: NO I don't know if this is a bug, and I'm not going to try to figure it out. I just have one question for you. You said (and I'm pulling this from multiple parts of the same comment): cobd2=M:\linux.img cobd3=M:\linux2.img cobd4=M:\linux3.img mdadm -C /dev/md0 /dev/cobd2 /dev/cobd3 /dev/cobd4 --level 5 --raid-devices 3 Why are you even considering doing RAID on images on the same disk? Are you trying to kill it? Ignoring any form of disk caching, every access to this RAID device is going to have to touch three places on the same disk. This is a giant performance penalty. Furthermore, you gain no redundancy because it's all on the same disk. If M: fails, you lose. ---------------------------------------------------------------------- Comment By: Markus Müller (privi) Date: 2006-10-19 11:16 Message: Logged In: YES user_id=1612112 VERY INTERESTING: If I do raid6, the problem does NOT occure if: - The raid is rebuilding with all drives - The raid is not rebuilding with all drives The problem occures with raid6 if - the raid is rebuilding with one drive missing - the raid is not rebuilding with one drive missing - the raid is not rebuilding with two drives missing It think this certified herewith that the routines which handles the restore sums are involved and/or causes this problem. I am anxious to read your idears how to locate the problem. ---------------------------------------------------------------------- Comment By: Markus Müller (privi) Date: 2006-10-19 10:13 Message: Logged In: YES user_id=1612112 I found something new out: The Problem occures extensive if - raid5 is rebuilding - raid5 is started with one missing drive, so that the index must read/written to get the data The Problem occures NOT if you use raid 0 or 1. Also there are only few bytes fliping if the raid is synced (16-64 Bytes every 1 GB instead of 16-64 every 4 MB), so there IS a connection to this problem and the index of raid5. -> The The problem must be located in the routines which handle the sum for the ability to rebuild. Any idear how to debug this? ---------------------------------------------------------------------- Comment By: Markus Müller (privi) Date: 2006-10-17 13:54 Message: Logged In: YES user_id=1612112 I made some more tests. The Problem is realy on block device level; with the following code I reproduced the problem with the only following 3 commands: First: Init RAID, so that it rebuilds: mdadm -C /dev/md0 /dev/cobd2 /dev/cobd3 /dev/cobd4 --level 5 --raid-devices 3 Second: Fill the raid (/dev/md0) with 1kbyte of 0x00, 1kbyte of 0x01, 1k of 0x02, ...: perl -e 'open(A, ">/dev/md0")||die; $a=chr(0) x 1024; binmode(A)||die; while (syswrite(A, $a, 1024) == 1024) { $a = chr($i % 255) x 1024; print "Writing testdata to md0: ".$i++."k\r"; last if ($i > 512*1024) }' Third, read and verify that we get what we wrote: perl -e '$i = -1; $b = chr(0) x 1024; open(A, "</dev/md0")||die; binmode(A)||die; while (sysread(A, $a, 1024) == 1024) { print "Verifying testdata on md0: ".$i++."k \r"; if ($a ne $b) { my $x = 0; foreach (split(//, $a)) { $x++ if ($_ ne chr(($i-1) % 255)); } print "\n:".$x.":\n"; } $b = chr($i % 255) x 1024; last if ($i > 512*1024)}' cobd2,3,4 is 9 GB on the same hard disk: cobd2=M:\linux.img cobd3=M:\linux2.img cobd4=M:\linux3.img In this case the problem is only significant, if the raid rebuilds. In this case I get the following output: The Values between the ":" are the number of BYTES which are Flipped. As you can see, it is every time a multiply of 16 and max 64 Bytes: chieftec:~# mdadm -C /dev/md0 /dev/cobd2 /dev/cobd3 /dev/cobd4 --level 5 --raid-devices 3 && perl -e 'open(A, ">/dev/md0")||die; $a=chr(0) x 1024; binmode(A)||die; while (syswrite(A, $a, 1024) == 1024) { $a = chr($i % 255) x 1024; print "Writing testdata to md0: ".$i++."k\r"; last if ($i > 512*1024) }; print "\n";' && perl -e '$i = -1; $b = chr(0) x 1024; open(A, "</dev/md0")||die; binmode(A)||die; while (sysread(A, $a, 1024) == 1024) { print "Verifying testdata on md0: ".$i++."k \r"; last if ($i > 512*1024); if ($a ne $b) { my $x = 0; foreach (split(//, $a)) { $x++ if ($_ ne chr(($i-1) % 255)); } print "\nFOUND ".$x." BAD BYTES!\n"; } $b = chr($i % 255) x 1024; }' mdadm: /dev/cobd2 appears to be part of a raid array: level=5 devices=3 ctime=Tue Oct 17 22:40:25 2006 mdadm: /dev/cobd3 appears to be part of a raid array: level=5 devices=3 ctime=Tue Oct 17 22:40:25 2006 mdadm: /dev/cobd4 appears to be part of a raid array: level=5 devices=3 ctime=Tue Oct 17 22:40:25 2006 Continue creating array? y mdadm: array /dev/md0 started. Writing testdata to md0: 524288k Verifying testdata on md0: 7045k FOUND 16 BAD BYTES! Verifying testdata on md0: 16484k FOUND 64 BAD BYTES! Verifying testdata on md0: 20522k FOUND 16 BAD BYTES! Verifying testdata on md0: 23012k FOUND 64 BAD BYTES! Verifying testdata on md0: 23188k FOUND 64 BAD BYTES! Verifying testdata on md0: 31059k FOUND 16 BAD BYTES! Verifying testdata on md0: 36232k FOUND 16 BAD BYTES! Verifying testdata on md0: 36821k FOUND 64 BAD BYTES! Verifying testdata on md0: 39114k FOUND 16 BAD BYTES! Verifying testdata on md0: 40115k FOUND 16 BAD BYTES! Verifying testdata on md0: 47783k FOUND 64 BAD BYTES! Verifying testdata on md0: 56592k FOUND 64 BAD BYTES! Verifying testdata on md0: 56830k FOUND 64 BAD BYTES! Verifying testdata on md0: 62960k FOUND 64 BAD BYTES! Verifying testdata on md0: 70445k FOUND 64 BAD BYTES! Verifying testdata on md0: 80037k FOUND 32 BAD BYTES! Verifying testdata on md0: 87117k FOUND 64 BAD BYTES! Verifying testdata on md0: 93279k FOUND 64 BAD BYTES! Verifying testdata on md0: 100958k FOUND 32 BAD BYTES! Verifying testdata on md0: 112262k FOUND 64 BAD BYTES! Verifying testdata on md0: 114235k FOUND 16 BAD BYTES! Verifying testdata on md0: 128816k FOUND 64 BAD BYTES! Verifying testdata on md0: 131847k FOUND 64 BAD BYTES! Verifying testdata on md0: 146295k FOUND 64 BAD BYTES! Verifying testdata on md0: 168715k FOUND 64 BAD BYTES! Verifying testdata on md0: 171008k FOUND 16 BAD BYTES! Verifying testdata on md0: 183503k FOUND 64 BAD BYTES! Verifying testdata on md0: 186394k FOUND 64 BAD BYTES! Verifying testdata on md0: 202113k FOUND 64 BAD BYTES! Verifying testdata on md0: 207550k FOUND 64 BAD BYTES! Verifying testdata on md0: 209415k FOUND 64 BAD BYTES! Verifying testdata on md0: 215558k FOUND 16 BAD BYTES! Verifying testdata on md0: 234408k FOUND 16 BAD BYTES! Verifying testdata on md0: 257269k FOUND 48 BAD BYTES! Verifying testdata on md0: 272044k FOUND 64 BAD BYTES! Verifying testdata on md0: 290077k FOUND 64 BAD BYTES! Verifying testdata on md0: 302931k FOUND 64 BAD BYTES! Verifying testdata on md0: 317717k FOUND 16 BAD BYTES! Verifying testdata on md0: 351499k FOUND 64 BAD BYTES! Verifying testdata on md0: 361887k FOUND 16 BAD BYTES! Verifying testdata on md0: 367266k FOUND 64 BAD BYTES! Verifying testdata on md0: 391654k FOUND 32 BAD BYTES! Verifying testdata on md0: 392590k FOUND 32 BAD BYTES! Verifying testdata on md0: 413564k FOUND 16 BAD BYTES! Verifying testdata on md0: 446928k FOUND 64 BAD BYTES! Verifying testdata on md0: 455422k FOUND 64 BAD BYTES! Verifying testdata on md0: 467882k FOUND 48 BAD BYTES! Verifying testdata on md0: 471113k FOUND 64 BAD BYTES! Verifying testdata on md0: 476918k FOUND 64 BAD BYTES! Verifying testdata on md0: 480915k FOUND 16 BAD BYTES! Verifying testdata on md0: 497273k FOUND 48 BAD BYTES! Verifying testdata on md0: 497803k FOUND 64 BAD BYTES! Verifying testdata on md0: 510137k FOUND 64 BAD BYTES! Verifying testdata on md0: 523328k FOUND 64 BAD BYTES! chieftec:~# If I run this Test to cobd2/3/4, also if I do it simultan, there is no problem. So there MUST be a Bug in md/raid5?! The problem ALSO occures if I make a test on files I create files on a 80 GB cofs5, which I use as device via /dev/loop: chieftec:/mnt/cobd5# dd if=/dev/zero of=./fs.1 bs=1k count=1k seek=9kxk 1024+0 records in 1024+0 records out 1048576 bytes transferred in 0.007837 seconds (133796836 bytes/sec) chieftec:/mnt/cobd5# dd if=/dev/zero of=./fs.2 bs=1k count=1k seek=9kxk 1024+0 records in 1024+0 records out 1048576 bytes transferred in 0.008237 seconds (127299801 bytes/sec) chieftec:/mnt/cobd5# dd if=/dev/zero of=./fs.3 bs=1k count=1k seek=9kxk 1024+0 records in 1024+0 records out 1048576 bytes transferred in 0.008029 seconds (130598315 bytes/sec) chieftec:/mnt/cobd5# ls -la total 3140 drwxr-xr-x 3 root root 4096 Oct 17 22:53 . drwxr-xr-x 11 root root 264 Oct 17 22:46 .. -rw-r--r-- 1 root root 9664724992 Oct 17 22:53 fs.1 -rw-r--r-- 1 root root 9664724992 Oct 17 22:53 fs.2 -rw-r--r-- 1 root root 9664724992 Oct 17 22:53 fs.3 drwx------ 2 root root 16384 Oct 17 22:45 lost+found chieftec:/mnt/cobd5# losetup /dev/loop0 fs.1 chieftec:/mnt/cobd5# losetup /dev/loop1 fs.2 chieftec:/mnt/cobd5# losetup /dev/loop2 fs.3 chieftec:/mnt/cobd5# mdadm -C /dev/md0 /dev/loop[0-2] --level 5 --raid-devices 3 && perl -e 'open(A, ">/dev/md0")||die; $a=chr(0) x 1024; binmode(A)||die; while (syswrite(A, $a, 1024) == 1024) { $a = chr($i % 255) x 1024; print "Writing testdata to md0: ".$i++."k\r"; last if ($i > 512*1024) }; print "\n";' && perl -e '$i = -1; $b = chr(0) x 1024; open(A, "</dev/md0")||die; binmode(A)||die; while (sysread(A, $a, 1024) == 1024) { print "Verifying testdata on md0: ".$i++."k \r"; last if ($i > 512*1024); if ($a ne $b) { my $x = 0; foreach (split(//, $a)) { $x++ if ($_ ne chr(($i-1) % 255)); } print "\nFOUND ".$x." BAD BYTES!\n"; } $b = chr($i % 255) x 1024; }' mdadm: array /dev/md0 started. Writing testdata to md0: 524288k Verifying testdata on md0: 11705k FOUND 64 BAD BYTES! Verifying testdata on md0: 27259k FOUND 16 BAD BYTES! Verifying testdata on md0: 37423k FOUND 64 BAD BYTES! Verifying testdata on md0: 61035k FOUND 64 BAD BYTES! Verifying testdata on md0: 66007k FOUND 16 BAD BYTES! Verifying testdata on md0: 75998k FOUND 64 BAD BYTES! Verifying testdata on md0: 101151k FOUND 64 BAD BYTES! Verifying testdata on md0: 114212k FOUND 64 BAD BYTES! Verifying testdata on md0: 116723k FOUND 64 BAD BYTES! Verifying testdata on md0: 122819k FOUND 64 BAD BYTES! Verifying testdata on md0: 143748k FOUND 64 BAD BYTES! Verifying testdata on md0: 154159k FOUND 64 BAD BYTES! Verifying testdata on md0: 154524k FOUND 64 BAD BYTES! Verifying testdata on md0: 158178k FOUND 32 BAD BYTES! Verifying testdata on md0: 182343k FOUND 64 BAD BYTES! Verifying testdata on md0: 189643k FOUND 64 BAD BYTES! Verifying testdata on md0: 193499k FOUND 64 BAD BYTES! Verifying testdata on md0: 203508k FOUND 64 BAD BYTES! Verifying testdata on md0: 207295k FOUND 64 BAD BYTES! Verifying testdata on md0: 213834k FOUND 64 BAD BYTES! Verifying testdata on md0: 214980k FOUND 16 BAD BYTES! Verifying testdata on md0: 220744k FOUND 64 BAD BYTES! Verifying testdata on md0: 227660k FOUND 64 BAD BYTES! Verifying testdata on md0: 231489k FOUND 32 BAD BYTES! Verifying testdata on md0: 250708k FOUND 64 BAD BYTES! Verifying testdata on md0: 256079k FOUND 64 BAD BYTES! Verifying testdata on md0: 261490k FOUND 16 BAD BYTES! Verifying testdata on md0: 267834k FOUND 64 BAD BYTES! Verifying testdata on md0: 343249k FOUND 64 BAD BYTES! Verifying testdata on md0: 381681k FOUND 64 BAD BYTES! Verifying testdata on md0: 382792k FOUND 32 BAD BYTES! Verifying testdata on md0: 398557k FOUND 64 BAD BYTES! Verifying testdata on md0: 399295k FOUND 16 BAD BYTES! Verifying testdata on md0: 406247k FOUND 64 BAD BYTES! Verifying testdata on md0: 412928k FOUND 64 BAD BYTES! Verifying testdata on md0: 419488k FOUND 64 BAD BYTES! Verifying testdata on md0: 419490k FOUND 16 BAD BYTES! Verifying testdata on md0: 424853k FOUND 16 BAD BYTES! Verifying testdata on md0: 427715k FOUND 64 BAD BYTES! Verifying testdata on md0: 444089k FOUND 64 BAD BYTES! Verifying testdata on md0: 463286k FOUND 16 BAD BYTES! Verifying testdata on md0: 475531k FOUND 64 BAD BYTES! Verifying testdata on md0: 485371k FOUND 48 BAD BYTES! Verifying testdata on md0: 516807k FOUND 16 BAD BYTES! chieftec:/mnt/cobd5# ---------------------------------------------------------------------- Comment By: Markus Müller (privi) Date: 2006-10-05 10:09 Message: Logged In: YES user_id=1612112 Latest demo what happens if you write ONE TIME to a md/raid5, and then make md5 sums on the same files all the time: chieftec:/mnt/md0# while true; do for i in 1 2 3 4; do ls -la ./f.$i; md5sum ./f.$i; done; done 6fd7390af52ebb80271d0ee2fe4cdbc3 ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 131fdbeffccba787dd1fb92242f1669a ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 f6a7ef548daf9f64e6837d1979a123ab ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 eed5da3983811b9fbc5584b04ef445c0 ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 8f61d6ac4089c3a4cb68483d9822c776 ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 194f88a6e7bb718cad10205be70dee8b ./f.1 442d3d7d484c4c4440a504f59f2ff016 ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 c9aa18b6e06788fd8fbbcd5f9b6aca9a ./f.1 27f20fcf912b93021a99fd2737610838 ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 4d377495f479afdc0c7d14a09116b805 ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 ab1c51a8b048502a3f29d3c13caa4b18 ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 95728ddbd9ca1a3fe6ca5606e71ec347 ./f.2 faeda9466e16f7635ff3b78bf7712832 ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.2 db9834fca270b58ebbcd38e5f4b923a0 ./f.3 ---------------------------------------------------------------------- Comment By: Markus Müller (privi) Date: 2006-10-05 10:08 Message: Logged In: YES user_id=1612112 This problem - AT ALL - is not related to reiserfs or the md tools. The problem also occures if you use the block device, /dev/md0, itself, but is easier to show with a filesystem. Of course the Problem occurse same at all on ext2, cause it is a problem ON BLOCK DEVICE LEVEL, and not filesystem. There is data corruption if you read data from a md/raid5, at least on reading and maybe also on writing. md is implemented in kernel, and mdtools is only there to start up the raid. Then it doesn't affect the work of the raid at all! Please talk to me in IRC; I wait for you from 10:00 o'clock toll 24 o'clock german time. I have multiple hardware to reproduce and debug, I only need to know how to use this resources. There is a basic i/o error in md and/or raid5 on cobd, which is NOT reproduceable with the only use of cobd as block device but device as raid5 device. I think it is a good thing for the whole project to fix such rudimentary block-device problems, at least to see if this problem is located on other places. Thanks! ---------------------------------------------------------------------- Comment By: Henry N. (henryn) Date: 2006-10-05 10:00 Message: Logged In: YES user_id=579204 Hello, I'm tends to not trust the reiserfs + md tools, and this all on a cobd. Is this also, if you using ext2 as file system? What does "dd-rescue" ? I have this not under SuSE. What can I use for it? A file with some random data, for sample my memory? Henry ---------------------------------------------------------------------- Comment By: Markus Müller (privi) Date: 2006-10-04 11:01 Message: Logged In: YES user_id=1612112 Mistake!! Please ignore the "NOT" in my last post! Correct is: The problem IS also reproduceable if I write to a single drive, for example to cobd1=M:\data.img. ... ---------------------------------------------------------------------- Comment By: Markus Müller (privi) Date: 2006-10-04 11:00 Message: Logged In: YES user_id=1612112 I now reproduced the problem on simply one cobd, mapped into the colinux via "cobd1=M:\linux.img.a". I created there 3 files, 20 gigs per file, and joined them via "mdadm -C /dev/md0 /dev/loop[0-2] --level 5 --raid-disks 3; mkreiserfs /dev/md0; mount /dev/md0 /dev/md0; for i in 1 2 3 4; do dd_rescue -m 1G /dev/zero ./f.$i; done; for i in 1 2 3 4; do ls -la ./f.$i; md5sum ./f.$i; done": -rw-r----- 1 root root 1073741824 Oct 4 19:47 ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.1 -rw-r----- 1 root root 1073741824 Oct 4 19:49 ./f.2 8bb182ea4a691e880808700df7a8c6cd ./f.2 -rw-r----- 1 root root 1073741824 Oct 4 19:50 ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.3 -rw-r----- 1 root root 1073741824 Oct 4 19:51 ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.4 Then I did not remove the files and made again the md5 sums; I saw that the sum of file2 is no longer wrong, but the 3. file is wrong: chieftec:/mnt/md0# for i in 1 2 3 4; do ls -la ./f.$i; md5sum ./f.$i; done -rw-r----- 1 root root 1073741824 Oct 4 19:47 ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.1 -rw-r----- 1 root root 1073741824 Oct 4 19:49 ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.2 -rw-r----- 1 root root 1073741824 Oct 4 19:50 ./f.3 7f2c9cba45d23fd0c9c8f9cb82125df6 ./f.3 -rw-r----- 1 root root 1073741824 Oct 4 19:51 ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.4 chieftec:/mnt/md0# Ok, guy, I think this is, at least in this test, a problem on reading from a raid5... Please help me to find the problem!!! chieftec:/mnt/md0# uname -a Linux chieftec.priv.de 2.6.12-co-default #2 Sun May 28 02:50:44 CEST 2006 i686 GNU/Linux chieftec:/mnt/md0# ---------------------------------------------------------------------- Comment By: Markus Müller (privi) Date: 2006-10-03 09:15 Message: Logged In: YES user_id=1612112 The problem is also NOT reproduceable if I write to a single drive, for example to cobd1=M:\data.img. A further idicator that this is not a hardware or driver caused problem but the cause is into coLinux. chieftec:/mnt/sda1# for i in 1 2 3 4 5; do dd_rescue -m 1G /dev/zero ./f.$i; done; for i in 1 2 3 4 5; do ls -la ./f.$i; md5sum ./f.$i; done; for i in 1 2 3 4 5; do rm ./f.$i; done dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 43279kB/s, avg.rate: 45956kB/s, avg.load: 85.9% Summary for /dev/zero -> ./f.1: dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 0kB/s, avg.rate: 45954kB/s, avg.load: 85.9% dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 37371kB/s, avg.rate: 40409kB/s, avg.load: 86.6% Summary for /dev/zero -> ./f.2: dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 0kB/s, avg.rate: 40406kB/s, avg.load: 86.6% dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 41206kB/s, avg.rate: 40830kB/s, avg.load: 85.4% Summary for /dev/zero -> ./f.3: dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 0kB/s, avg.rate: 40828kB/s, avg.load: 85.4% dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 43374kB/s, avg.rate: 38770kB/s, avg.load: 85.3% Summary for /dev/zero -> ./f.4: dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 0kB/s, avg.rate: 38768kB/s, avg.load: 85.3% dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 40382kB/s, avg.rate: 40590kB/s, avg.load: 84.4% Summary for /dev/zero -> ./f.5: dd_rescue: (info): ipos: 1048576.0k, opos: 1048576.0k, xferd: 1048576.0k errs: 0, errxfer: 0.0k, succxfer: 1048576.0k +curr.rate: 0kB/s, avg.rate: 40587kB/s, avg.load: 84.4% -rw-r----- 1 root root 1073741824 Oct 3 18:10 ./f.1 cd573cfaace07e7949bc0c46028904ff ./f.1 -rw-r----- 1 root root 1073741824 Oct 3 18:11 ./f.2 cd573cfaace07e7949bc0c46028904ff ./f.2 -rw-r----- 1 root root 1073741824 Oct 3 18:11 ./f.3 cd573cfaace07e7949bc0c46028904ff ./f.3 -rw-r----- 1 root root 1073741824 Oct 3 18:12 ./f.4 cd573cfaace07e7949bc0c46028904ff ./f.4 -rw-r----- 1 root root 1073741824 Oct 3 18:12 ./f.5 cd573cfaace07e7949bc0c46028904ff ./f.5 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=622063&aid=1569947&group_id=98788 |