I almost thought I was going to hit the go button today. Alas, I considered yet another scenario while in the shower and now im pulling back yet again.
the short of it is my needing to understand how (if) additional parity drives protect against changed data.
so my parity drives need to be 10TB. not cheap. However im in a place right now where im running two pools. one is set up, wide open, able to be backed up with snapraid in full without concern.
the other houses everything that is emulation. All my roms will not change (maybe location or name, but not data itself) however I have artwork/cover media folders that are WIP that will do who knows what, I have tool/app/script folders with softwares that have databases and have logs, and i have emulators and frontend folders that will update and all that entails.
needless to say I initially excluded all but the games themselves. I was going to back up everything else via robocopy script to a local drive and that would be that. i was quite eager to finally move forward with my first sync.
Then i thought, wait a minute - if a drive dies.. its only going to restore the games. Even if i have everything else backed up, how on earth will i know what was missing? im using drive pool so i have no idea what and where anything is and the amount of data (especially small files) that could be present on a random 10tb drive that just up and failed is staggering.
I considered doing like a bulk copy/paste of everything i have backed up locally in that scenario on top of the pool, then just skipping the files that were already present and while it would work (i think) it would certainly be an imperfect solution.
that leads me to where I am now. Someone had explained to me previously that changed or missing data will result in a restore missing data from those blocks in some other location. they suggested that this can be mitigated with additional parity drives. (completely?)
Just so i understand correctly, would buying more parity drives (adding 2 to my existing 2) ensure that should some random smaller files sprinkled about change, id still be able to restore? aka i could include the directories im excluding at the moment and move on/rest easy?
Im covering 24 disks so i need to add more than 2 just from a "what if several fail at once" standpoint, but i was going to put it off for awhile. Since its currently preventing my ideal long term setup and me from even syncing the first time yet, i'll do it now if it would provide protection against changed files within reason.
sorry for the book.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
snapraid diff will return differences between current state of the array and last successful sync. It will also return path of changed or missing files. If files are silently corrupted, snapraid scrub -p 100 should find them.
Restore depends on failure itself - if entire drive dies, single parity drive and all remaining data drives are required to recover.
If two drives die at once - double parity and all remaining drives must be heatlhy.
Generally speaking, if You don't miss something in "exclude " options or another drive die You should be able to recover everything from single drive failure. Except situation where replacement drive is too small and except situation where snapraid SYNC was issued after failure and many new files were added. If You accidentaly delete data You may be not able to recover after snapraid sync.
For 24 disks I would recommend 3 parity drives at least.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
perfect, snapraid info is sometimes hard to come by outside of a seasoned user with a good understanding posting/replying here. I appreciate youre having took the time to reply!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Use --log switch and collect these files to track all changes, my sync script looks like this:
#!/bin/bash#varsdata=$(date +'%Y%m%d')#echo Today is $data !
snapraid diff --log $data.diff; snapraid status --log $data.status; snapraid sync --log $data.sync; snapraid scrub -p new --log $data.scrub; snapraid touch --log $data.touch;snapraid status --log $data.status
after revieving them I just gzip them.
For data recovery usually procedure is:
1. replace failed drive, note which one was that (d1, d2, d3... p1 etc)
1a. if it was data drive:
snapraid -d disk1 -l /tmp/snapraid-disk1-fix.log fix
snapraid -d disk1 -a check
1b. if it was parity:
snapraid sync to compute parity again or
snapraid fix -d PARITY_NAME
2. For undelete a dir:
snapraid fix -m -f Dir_Example/
I almost thought I was going to hit the go button today. Alas, I considered yet another scenario while in the shower and now im pulling back yet again.
the short of it is my needing to understand how (if) additional parity drives protect against changed data.
so my parity drives need to be 10TB. not cheap. However im in a place right now where im running two pools. one is set up, wide open, able to be backed up with snapraid in full without concern.
the other houses everything that is emulation. All my roms will not change (maybe location or name, but not data itself) however I have artwork/cover media folders that are WIP that will do who knows what, I have tool/app/script folders with softwares that have databases and have logs, and i have emulators and frontend folders that will update and all that entails.
needless to say I initially excluded all but the games themselves. I was going to back up everything else via robocopy script to a local drive and that would be that. i was quite eager to finally move forward with my first sync.
Then i thought, wait a minute - if a drive dies.. its only going to restore the games. Even if i have everything else backed up, how on earth will i know what was missing? im using drive pool so i have no idea what and where anything is and the amount of data (especially small files) that could be present on a random 10tb drive that just up and failed is staggering.
I considered doing like a bulk copy/paste of everything i have backed up locally in that scenario on top of the pool, then just skipping the files that were already present and while it would work (i think) it would certainly be an imperfect solution.
that leads me to where I am now. Someone had explained to me previously that changed or missing data will result in a restore missing data from those blocks in some other location. they suggested that this can be mitigated with additional parity drives. (completely?)
Just so i understand correctly, would buying more parity drives (adding 2 to my existing 2) ensure that should some random smaller files sprinkled about change, id still be able to restore? aka i could include the directories im excluding at the moment and move on/rest easy?
Im covering 24 disks so i need to add more than 2 just from a "what if several fail at once" standpoint, but i was going to put it off for awhile. Since its currently preventing my ideal long term setup and me from even syncing the first time yet, i'll do it now if it would provide protection against changed files within reason.
sorry for the book.
snapraid diff will return differences between current state of the array and last successful sync. It will also return path of changed or missing files. If files are silently corrupted, snapraid scrub -p 100 should find them.
Restore depends on failure itself - if entire drive dies, single parity drive and all remaining data drives are required to recover.
If two drives die at once - double parity and all remaining drives must be heatlhy.
Generally speaking, if You don't miss something in "exclude " options or another drive die You should be able to recover everything from single drive failure. Except situation where replacement drive is too small and except situation where snapraid SYNC was issued after failure and many new files were added. If You accidentaly delete data You may be not able to recover after snapraid sync.
For 24 disks I would recommend 3 parity drives at least.
perfect, snapraid info is sometimes hard to come by outside of a seasoned user with a good understanding posting/replying here. I appreciate youre having took the time to reply!
Use --log switch and collect these files to track all changes, my sync script looks like this:
after revieving them I just gzip them.
For data recovery usually procedure is:
1. replace failed drive, note which one was that (d1, d2, d3... p1 etc)
1a. if it was data drive:
snapraid -d disk1 -l /tmp/snapraid-disk1-fix.log fix
snapraid -d disk1 -a check
1b. if it was parity:
snapraid sync to compute parity again or
snapraid fix -d PARITY_NAME
2. For undelete a dir:
snapraid fix -m -f Dir_Example/
Many questions were already anserwered here:
https://www.snapraid.it/faq
Golden rule is to recover from any damages before next SYNC.