Problems setting up my first snapraid + aufs which contains 3x3TB discs at the moment, 1 parity and 2 data + content configured in snapraid (running OMV) also created an aufs partition from the 2 data discs.
I filled up the aufs partition with about 1.5TB of data and restarted the box (power outage and UPS installation) now after restart I receive error messages when booting and doing fs check as below:
Log of fsck -C -R -A -a
Wed Apr 22 18:34:07 2015
fsck from util-linux 2.20.1
3TB1 contains a file system with errors, check forced.
3TB2: clean, 51317/183148544 files, 358176784/732566385 blocks
3TB3: clean, 50603/183148544 files, 358201716/732566385 blocks
3TB1: Duplicate or bad block in use!
3TB1: Multiply-claimed block(s) in inode 23592961: 94371842
3TB1: Multiply-claimed block(s) in inode 26476545: 105906179
3TB1: (There are 2 inodes containing multiply-claimed blocks.)
3TB1: File /.wh..wh.orph (inode #23592961, mod time Mon Apr 20 22:51:50 2015)
has 1 multiply-claimed block(s), shared with 1 file(s):
3TB1: <filesystem metadata="">
3TB1:</filesystem>
3TB1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
fsck died with exit status 4
OK, I managed to circumvent that with ctrl+d since I did not want to mess it up more with fsck so I booted and did a snapraid sync and this follows:
root@mediavault:~# snapraid sync
Self test...
Loading state from /media/fae60755-feec-49f0-bf97-b884c09d9f3b/snapraid.content...
Scanning disk Data1...
Scanning disk Data2...
Using 1144 MiB of memory.
Saving state to /media/fae60755-feec-49f0-bf97-b884c09d9f3b/snapraid.content...
Saving state to /media/d4873045-bedf-416c-b228-245599be44dc/snapraid.content...
after this follows about 100k lines of outofparity messages as below and which ends with:
outofparity /media/d4873045-bedf-416c-b228-245599be44dc/DIGICAM_ARCH/_16gig_rec/1_FAT32/xxx/Tagged Image File/601-664/file661.TIF
WARNING! Without an accessible Parity file, it isn't possible to sync.
So now I have no idea what to do, I have tried googling but I'm having a had time finding someone who has the same problem, so now I turn to these forums, could someone enlighten me to where I am failing?
Thank you all in advance.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Do you have all the important data you want in a safe place? If yes, you can proceed - if not you need some other strategy for RECOVERING first everything you might need (and lose otherwise if you do any changes).
Now, assuming all your important data is stashed somewhere (somewhere ELSE, not in the array you want to fix!) and you want to bring back the array to consistency.
First you need to fix all your disks that fail fsck. Unmount them and do fsck, fix everything until they are all clean. This includes your parity disk(s). After that mount them. Go through the folders, maybe open some files, see if everything looks right. For parity file(s) see if the file is there. See if it is accessible (run "file" on it for example). Maybe you can run a snapraid diff or check or scrub first but it won't make much of a difference. In the end you have to run snapraid sync - if it still fails with the same message you need to make sure the files is there and accessible. Maybe run md5sum on it just to make sure (it will take a LONG time)?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have all relevant data backed up on external discs and also on another computer so I am ok with even "wiping" all the 3TB discs, just hope I dont have to since it took quite some time to copy the data there.
But I will umount them and run fsck on them all now to see if the error persists.
Will update later with information, but thank you all so far!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Could someone enlighten me to what this means, and how come it has happened on a brand new disc that just got the parity file on it?
3TB1: Duplicate or bad block in use!
3TB1: Multiply-claimed block(s) in inode 23592961: 94371842
3TB1: Multiply-claimed block(s) in inode 26476545: 105906179
3TB1: (There are 2 inodes containing multiply-claimed blocks.)
3TB1: File /.wh..wh.orph (inode #23592961, mod time Mon Apr 20 22:51:50 2015)
has 1 multiply-claimed block(s), shared with 1 file(s):
3TB1: <filesystem metadata="">
3TB1:</filesystem>
3TB1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
fsck died with exit status 4
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am running fsck now on the unmounted 3TB1 drive, and it's been running for 2 hours now, can I expect this to take long? Is it ok that I answered yes to the questions posed?
Is it normal that the snapraid.parity file has "multiply-claimed blocks" ?
~~~~root@mediavault:~# fsck -v /dev/sdb1
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
3TB1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 15 has an invalid extent node (blk 199753733, lblk 187242496)
Clear<y>? yes
Inode 15 has an invalid extent node (blk 9605, lblk 4294967295)
Clear<y>? yes
Inode 15, i_blocks is 1596588200, should be 1496891544. Fix<y>? yes</y></y></y>
Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks
Multiply-claimed block(s) in inode 15: 144179201
Multiply-claimed block(s) in inode 23592961: 94371842
Multiply-claimed block(s) in inode 26476545: 105906179
Pass 1C: Scanning directories for inodes with multiply-claimed blocks
Pass 1D: Reconciling multiply-claimed blocks
(There are 3 inodes containing multiply-claimed blocks.)
File /snapraid.parity (inode #15, mod time Wed Apr 22 19:34:31 2015)
has 1 multiply-claimed block(s), shared with 1 file(s):
<filesystem metadata="">
Clone multiply-claimed blocks<y>? yes
~~~~</y></filesystem>
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am actually still waiting for fsck to finish, its been almost 4 days now.
Is this normal? I mean can I cancel it or will it eventually finish with "Clone multiply-claimed blocks<y>?"</y>
What to do?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Did you run SMART long test on it before using it?
What is SMART saying about the drive?
Brand new HD:s are not error free, ie you want to test them before putting in "production".
If it is the parity disk wouldn't it be safer to reformat and recreate the parity (after the SMART validation)? As long as you have the content file you can at least checksum validate the datafiles even if you can't recover the errors. Remember that snapraid do not touch your datafiles but add file validation using the content file and additional recovery using the parity.
/X
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Problems setting up my first snapraid + aufs which contains 3x3TB discs at the moment, 1 parity and 2 data + content configured in snapraid (running OMV) also created an aufs partition from the 2 data discs.
I filled up the aufs partition with about 1.5TB of data and restarted the box (power outage and UPS installation) now after restart I receive error messages when booting and doing fs check as below:
Log of fsck -C -R -A -a
Wed Apr 22 18:34:07 2015
fsck from util-linux 2.20.1
3TB1 contains a file system with errors, check forced.
3TB2: clean, 51317/183148544 files, 358176784/732566385 blocks
3TB3: clean, 50603/183148544 files, 358201716/732566385 blocks
3TB1: Duplicate or bad block in use!
3TB1: Multiply-claimed block(s) in inode 23592961: 94371842
3TB1: Multiply-claimed block(s) in inode 26476545: 105906179
3TB1: (There are 2 inodes containing multiply-claimed blocks.)
3TB1: File /.wh..wh.orph (inode #23592961, mod time Mon Apr 20 22:51:50 2015)
has 1 multiply-claimed block(s), shared with 1 file(s):
3TB1: <filesystem metadata="">
3TB1:</filesystem>
3TB1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
fsck died with exit status 4
OK, I managed to circumvent that with ctrl+d since I did not want to mess it up more with fsck so I booted and did a snapraid sync and this follows:
root@mediavault:~# snapraid sync
Self test...
Loading state from /media/fae60755-feec-49f0-bf97-b884c09d9f3b/snapraid.content...
Scanning disk Data1...
Scanning disk Data2...
Using 1144 MiB of memory.
Saving state to /media/fae60755-feec-49f0-bf97-b884c09d9f3b/snapraid.content...
Saving state to /media/d4873045-bedf-416c-b228-245599be44dc/snapraid.content...
after this follows about 100k lines of outofparity messages as below and which ends with:
outofparity /media/d4873045-bedf-416c-b228-245599be44dc/DIGICAM_ARCH/_16gig_rec/1_FAT32/xxx/Tagged Image File/601-664/file661.TIF
WARNING! Without an accessible Parity file, it isn't possible to sync.
So now I have no idea what to do, I have tried googling but I'm having a had time finding someone who has the same problem, so now I turn to these forums, could someone enlighten me to where I am failing?
Thank you all in advance.
What does your configuration file and fstab look like?
I used the below links to setup my Snapraid + AUFS server.
http://zackreed.me/articles/72-snapraid-on-ubuntu-12-04
http://zackreed.me/articles/89-compile-aufs-with-3-18-6-kernel-hnotify-and-nfs-exportability
Do you have all the important data you want in a safe place? If yes, you can proceed - if not you need some other strategy for RECOVERING first everything you might need (and lose otherwise if you do any changes).
Now, assuming all your important data is stashed somewhere (somewhere ELSE, not in the array you want to fix!) and you want to bring back the array to consistency.
First you need to fix all your disks that fail fsck. Unmount them and do fsck, fix everything until they are all clean. This includes your parity disk(s). After that mount them. Go through the folders, maybe open some files, see if everything looks right. For parity file(s) see if the file is there. See if it is accessible (run "file" on it for example). Maybe you can run a snapraid diff or check or scrub first but it won't make much of a difference. In the end you have to run snapraid sync - if it still fails with the same message you need to make sure the files is there and accessible. Maybe run md5sum on it just to make sure (it will take a LONG time)?
And my fstab looks like this:
<file system=""> <mount point=""> <type> <options> <dump> <pass>
/ was on /dev/sda1 during installation
UUID=ced3f516-b21e-422f-a30f-c237a81f0040 / ext4 errors=remount-ro 0 1
swap was on /dev/sda5 during installation
UUID=a599cc7d-d3db-4510-bbef-09770a192b74 none swap sw 0 0
/dev/sdb1 /media/usb0 auto rw,user,noauto 0 0
tmpfs /tmp tmpfs defaults 0 0</pass></dump></options></type></mount></file>
I have all relevant data backed up on external discs and also on another computer so I am ok with even "wiping" all the 3TB discs, just hope I dont have to since it took quite some time to copy the data there.
But I will umount them and run fsck on them all now to see if the error persists.
Will update later with information, but thank you all so far!
Could someone enlighten me to what this means, and how come it has happened on a brand new disc that just got the parity file on it?
3TB1: Duplicate or bad block in use!
3TB1: Multiply-claimed block(s) in inode 23592961: 94371842
3TB1: Multiply-claimed block(s) in inode 26476545: 105906179
3TB1: (There are 2 inodes containing multiply-claimed blocks.)
3TB1: File /.wh..wh.orph (inode #23592961, mod time Mon Apr 20 22:51:50 2015)
has 1 multiply-claimed block(s), shared with 1 file(s):
3TB1: <filesystem metadata="">
3TB1:</filesystem>
3TB1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
fsck died with exit status 4
I am running fsck now on the unmounted 3TB1 drive, and it's been running for 2 hours now, can I expect this to take long? Is it ok that I answered yes to the questions posed?
Is it normal that the snapraid.parity file has "multiply-claimed blocks" ?
~~~~root@mediavault:~# fsck -v /dev/sdb1
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
3TB1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 15 has an invalid extent node (blk 199753733, lblk 187242496)
Clear<y>? yes
Inode 15 has an invalid extent node (blk 9605, lblk 4294967295)
Clear<y>? yes
Inode 15, i_blocks is 1596588200, should be 1496891544. Fix<y>? yes</y></y></y>
Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks
Multiply-claimed block(s) in inode 15: 144179201
Multiply-claimed block(s) in inode 23592961: 94371842
Multiply-claimed block(s) in inode 26476545: 105906179
Pass 1C: Scanning directories for inodes with multiply-claimed blocks
Pass 1D: Reconciling multiply-claimed blocks
(There are 3 inodes containing multiply-claimed blocks.)
File /snapraid.parity (inode #15, mod time Wed Apr 22 19:34:31 2015)
has 1 multiply-claimed block(s), shared with 1 file(s):
<filesystem metadata="">
Clone multiply-claimed blocks<y>? yes
~~~~</y></filesystem>
I am actually still waiting for fsck to finish, its been almost 4 days now.
Is this normal? I mean can I cancel it or will it eventually finish with "Clone multiply-claimed blocks<y>?"</y>
What to do?
Did you run SMART long test on it before using it?
What is SMART saying about the drive?
Brand new HD:s are not error free, ie you want to test them before putting in "production".
If it is the parity disk wouldn't it be safer to reformat and recreate the parity (after the SMART validation)? As long as you have the content file you can at least checksum validate the datafiles even if you can't recover the errors. Remember that snapraid do not touch your datafiles but add file validation using the content file and additional recovery using the parity.
/X