I think we should have some kind of "best practices" small document (or small chapter in one of the existing documentes like FAQ or Manual) to document some of the more unknown caveats of a snapraid setup.
For example (I'm including more explanations than needed for a small and concise document):
one basic thing that isn't snapraid specific but otherwise next to impossible to find is the "-T largefile4" option for ext4 (or other). This is good not only for the parity filesystem but also for any other filesystem where you don't store more than a few million files (if used as "-T largefile"). The option doesn't even exist in man mkfs.ext4! And if you don't use it you'll lose about 30GB for each 2TB disk (for example)! This is not snapraid specific, I am not yet talking about the fact that the parity will be a bit bigger than the biggest disk you have.
snapraid is for data that you'll have "later". It is not for temporary unpredictable data (with the important examples: virtual machines or temporary/incomplete downloads). The problem here is not only that you don't have redundancy for the data that changed since the last "snapraid sync" run but also the data that changes is needed to recover some OTHER static data from some OTHER disk in case that disk fails. You might think deleting data is bad for redundancy but it isn't as long as the file deleted still exists somewhere (for example it is some public download you can get again). The bad part is including a file that nobody else has and then changing it (so you don't have it either). Of course multiple redundancy always helps but this is another discussion.
the "parity full" problem can be easily solved just by parking some folders you don't want included in snapraid somewhere in the biggest data drives (and excluding them from snapraid in some way). You can use for that precisely the files you shouldn't include in snapraid! Temporary files, virtual machines, stuff you don't care much about, etc!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I mentioned this in another thread, and didn't get much of a response -
but why not use the wiki that comes with the Sourceforge project? That
way, multiple people can contribute, without forcing the entire workload
onto Andrea?
I personally would like to see a chapter on the technical nuts & bolts
of how SR operates - something more detailed than what's currently in
the FAQ.
And a really thorough page on how to calculate the max size of the
parity file would probably cut traffic on the email list in half.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Don't really want to hijack my own thread ... but I learned that calculating the wastage in snapraid parity file is only half of the story - there is wastage in the data drives too (this "helps", in the sense that you can't put that much in the data disks, therefore making it a tiny bit easy not to overflow the parity), making things next to impossible to estimate correctly based just on the number of files and snapraid block size.
In real life it might be possible (and much easier) to just "eat away" some space from the biggest data drives. Reduce the partition and add some swap partition or add some ignored_by_snapraid folders in snapraid.conf and on each "biggest" drive and then drop there something that changes (like temp download folder or virtual machines as I mentioned), etc. Whatever, just use some (preferably 4-5+ for 2TB drives let's say) tens of gigs on the biggest data drives and you're done.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've got one drive that's pushing 3% in estimated wasted space: 2.3TB
data, ~790,000 files. If that drive were full, I'd be looking at
something like 90GB. I'm not sure it would occur to a new user to
reserve that much space (nor would I want to, if I didn't have to).
Which is a longwinded way of saying that it seems like it's worth a
page. Probably mentioning "safety partitioning" like you mentioned.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I do not think it is a good idea to suggest putting a temporary download directory or anything changeable in size like that on the data drives. The problem is that a temporary directory should usually have a nice chunk of space available for unexpected usage. But filler for a SnapRAID data drive should ideally be of a fixed size. So, a fixed size VM file (excluded from SnapRAID) would be good, but not a temporary directory. A single large file (for example, a copy of a video file or just a junk file created for the purpose) that does not need protection would also be good.
Last edit: jwill42 2014-11-03
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
"The problem is that a temporary directory should usually have a nice chunk of space available for unexpected usage."
YEP, and if your parity disks are as big as some of the data drives (a very common situation and the root of all "parity size" discussions) you HAVE a nice chunk of space that you can't include in snapraid anyway (90GB as per previous example from Mitchell I guess it qualifies). Is a match from heaven.
In fact there are people (on this forum) ALREADY doing that (keeping the incomplete to**ent downloads together with the more permanent files on snapraid data disks), compromising the chances for recovery for OTHER data disks. Just ignoring those files in snapraid.conf would fix both problems (data integrity and what to use some 50-100GB+ for).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If you already did there would be no reason to come up with "stupid" labels which is at best impolite on a public forum. If you aren't willing or able to explain what you meant it is downright rude and trollish. I see that you destroyed at least one other thread, please refrain from doing it again here.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am willing and able to explain, and I ALREADY DID EXPLAIN IN THIS VERY THREAD. Are you having difficulty reading? Or are you just trolling because you cannot handle being corrected?
Last edit: jwill42 2014-11-05
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
jwill42 would you please turn down the condescension and close to instant insulting? It is constant and it getting old fast.
If someone asks you for a cite, just please provide a cite.
If someone asks you for an explanation, please provide an explanation or if you have already provide that explanation just reply with a link to that message.
Once someone provides a cite that generally ends the argument.
It's really that easy. You obviously have a pretty good handle on SR so many people are looking to you for advice and insight.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You really need to stop telling other people what to do when you so obviously have no clue what is going on.
Are you as incapable of reading this thread as Oblivious John? Maybe you should be named Oblivious David.
Maybe you just need to shut up. It really is that easy. Or if it is so important to you to explain something again that has already been explained in this thread AND REPEATEDLY REFERENCED, then why don't you explain it rather than telling me what to do.
Last edit: jwill42 2014-11-05
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Is it possible that whenever there is a flame you are involved? Doesn't matter who is right or wrong. Just express your opinions without upsetting other users. I would really appreciate that.
Thanks,
Andrea
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I cannot control whether other people get upset. I just state the facts and correct mistakes.
And it very much does matter what is right or wrong.
ETA: I will point out that in both of the threads in question, I stopped posting as soon as the person posting incorrect or misleading information stopped posting such things. Cause and effect, with a straightforward conclusion.
Last edit: jwill42 2014-11-05
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
In MY case putting a temporary download folder on one of the data drives that is excluded from Snapraid is a perfect good and fine thing. Should it be best practice? Don't think so. Is it stupid beyond belief? Don't think so.
The main reason why it's not: It's using the space that should not be used since my data drives and parity drives are the same size and I won't like to split them into different partitions or something. And my VM files are stored on SSDs and are not protected by SR, so that's not an option. It's just simple and plain the only thing I can use it for. And space isn't a problem in this case, so I have a nice chunk of space available there.
Imho this should be up to the user, but if someone knows what he is doing there I don't think it's stupid.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You seem to have missed the point. The suggestion which is stupid is putting a temporary directory (or anything of changeable size) on a data drive IN ORDER TO RESERVE SPACE and keep SnapRAID from having parity drive out of space issues.
If you are not concerned about reserving space, then my comment has no bearing on how you utilize unused space on your data drives.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think jwill was trying to say that a temp dir wouldn't maintain a consistent size, and thus wouldn't strictly enforce a reduced size on the snapraid protected part of the disk. But if you want to enforce the limit that strictly, surely shrinking the SR partition would accomplish that even more strictly, allowing you to use the remaining part for temp files or leaked NSA docs or child porn or whatever.
Next strictest would be a large statically size file(s), per jwill. (But if they're truly static, rather than just static-sized then why not include them in SR? I'm not sure how many things are of a constant size, but change internally - other than the VM example given.) More flexible than partitioning, but stricter than a temp dir.
Least strict would be a temp dir or whatever, and then you just remember the target size of the SR area, and enforce it mentally. Or I guess you could write a little script that monitors the size of the temp dir, and writes / deletes junk files to maintain a constant size. This would probably be my choice.
As jmc said, the stupidity of any of these solutions depends on the user & context.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Why do you say that putting a large static file on a data drive is "less strict" than using a partition that is smaller by the size of the large static file?
I think either one is just as effective. The advantage of the static file technique is that you do not need to shrink or grow active partitions in the event that you later decide to change the amount of reserved space. The advantage of partitioning is that you could then use the extra space for something of changeable size, since the partition itself enforces the reserved space.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Your second paragraph below answers your first. Repartitioning is
harder than deleting a large static file - so if we're grading solutions
(and hence, users) on a scale of "strictness" (and the users by
"irresponsibility"), the former is more strict than the latter, as it
prevents someone (or someone else) from deciding they need the space to
download Transformers XXXIV now now now, and then forgetting about it or
otherwise getting themselves in a pickle.
The semantics hardly matter, though. The right solution for the user in
question is the perfect amount of "strict".
mitch
On 11/5/2014 12:07 PM, jwill42 wrote:
Why do you say that putting a large static file on a data drive is "less strict" than using a partition that is smaller by the size of the large static file?
I think either one is just as effective. The advantage of the static file technique is that you do not need to shrink or grow active partitions in the event that you later decide to change the amount of reserved space.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think semantics matter a bit, since I misunderstood your previous point until you elaborated. I think you meant difficulty rather than strictness. For strictness, if the reserve file had the permissions set so that only root could modify or delete it, then it is just as strict as partitioning since you would presumably need root access to change the partition size.
Anyway, I think we agree that either a fixed-size reserve file or partitioning can be best practice for reserving data drive space, depending on the specific requirements of an installation.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think we should have some kind of "best practices" small document (or small chapter in one of the existing documentes like FAQ or Manual) to document some of the more unknown caveats of a snapraid setup.
For example (I'm including more explanations than needed for a small and concise document):
one basic thing that isn't snapraid specific but otherwise next to impossible to find is the "-T largefile4" option for ext4 (or other). This is good not only for the parity filesystem but also for any other filesystem where you don't store more than a few million files (if used as "-T largefile"). The option doesn't even exist in man mkfs.ext4! And if you don't use it you'll lose about 30GB for each 2TB disk (for example)! This is not snapraid specific, I am not yet talking about the fact that the parity will be a bit bigger than the biggest disk you have.
snapraid is for data that you'll have "later". It is not for temporary unpredictable data (with the important examples: virtual machines or temporary/incomplete downloads). The problem here is not only that you don't have redundancy for the data that changed since the last "snapraid sync" run but also the data that changes is needed to recover some OTHER static data from some OTHER disk in case that disk fails. You might think deleting data is bad for redundancy but it isn't as long as the file deleted still exists somewhere (for example it is some public download you can get again). The bad part is including a file that nobody else has and then changing it (so you don't have it either). Of course multiple redundancy always helps but this is another discussion.
the "parity full" problem can be easily solved just by parking some folders you don't want included in snapraid somewhere in the biggest data drives (and excluding them from snapraid in some way). You can use for that precisely the files you shouldn't include in snapraid! Temporary files, virtual machines, stuff you don't care much about, etc!
I mentioned this in another thread, and didn't get much of a response -
but why not use the wiki that comes with the Sourceforge project? That
way, multiple people can contribute, without forcing the entire workload
onto Andrea?
I personally would like to see a chapter on the technical nuts & bolts
of how SR operates - something more detailed than what's currently in
the FAQ.
And a really thorough page on how to calculate the max size of the
parity file would probably cut traffic on the email list in half.
Yes, the wiki is a good idea!
Don't really want to hijack my own thread ... but I learned that calculating the wastage in snapraid parity file is only half of the story - there is wastage in the data drives too (this "helps", in the sense that you can't put that much in the data disks, therefore making it a tiny bit easy not to overflow the parity), making things next to impossible to estimate correctly based just on the number of files and snapraid block size.
In real life it might be possible (and much easier) to just "eat away" some space from the biggest data drives. Reduce the partition and add some swap partition or add some ignored_by_snapraid folders in snapraid.conf and on each "biggest" drive and then drop there something that changes (like temp download folder or virtual machines as I mentioned), etc. Whatever, just use some (preferably 4-5+ for 2TB drives let's say) tens of gigs on the biggest data drives and you're done.
I've got one drive that's pushing 3% in estimated wasted space: 2.3TB
data, ~790,000 files. If that drive were full, I'd be looking at
something like 90GB. I'm not sure it would occur to a new user to
reserve that much space (nor would I want to, if I didn't have to).
Which is a longwinded way of saying that it seems like it's worth a
page. Probably mentioning "safety partitioning" like you mentioned.
I do not think it is a good idea to suggest putting a temporary download directory or anything changeable in size like that on the data drives. The problem is that a temporary directory should usually have a nice chunk of space available for unexpected usage. But filler for a SnapRAID data drive should ideally be of a fixed size. So, a fixed size VM file (excluded from SnapRAID) would be good, but not a temporary directory. A single large file (for example, a copy of a video file or just a junk file created for the purpose) that does not need protection would also be good.
Last edit: jwill42 2014-11-03
"The problem is that a temporary directory should usually have a nice chunk of space available for unexpected usage."
YEP, and if your parity disks are as big as some of the data drives (a very common situation and the root of all "parity size" discussions) you HAVE a nice chunk of space that you can't include in snapraid anyway (90GB as per previous example from Mitchell I guess it qualifies). Is a match from heaven.
In fact there are people (on this forum) ALREADY doing that (keeping the incomplete to**ent downloads together with the more permanent files on snapraid data disks), compromising the chances for recovery for OTHER data disks. Just ignoring those files in snapraid.conf would fix both problems (data integrity and what to use some 50-100GB+ for).
No, it is NOT "a match from heaven". It is stupid beyond belief.
Would you care to explain what or who is so stupid here?
Already explained.
NOW you explained indeed. However, this might not be the best forum to vent or resolve your psihological issues.
No, I already explained in this thread. This is not rocket science. But it seems even spelling and reading pose a problem.
If you already did there would be no reason to come up with "stupid" labels which is at best impolite on a public forum. If you aren't willing or able to explain what you meant it is downright rude and trollish. I see that you destroyed at least one other thread, please refrain from doing it again here.
I am willing and able to explain, and I ALREADY DID EXPLAIN IN THIS VERY THREAD. Are you having difficulty reading? Or are you just trolling because you cannot handle being corrected?
Last edit: jwill42 2014-11-05
jwill42 would you please turn down the condescension and close to instant insulting? It is constant and it getting old fast.
If someone asks you for a cite, just please provide a cite.
If someone asks you for an explanation, please provide an explanation or if you have already provide that explanation just reply with a link to that message.
Once someone provides a cite that generally ends the argument.
It's really that easy. You obviously have a pretty good handle on SR so many people are looking to you for advice and insight.
David:
You really need to stop telling other people what to do when you so obviously have no clue what is going on.
Are you as incapable of reading this thread as Oblivious John? Maybe you should be named Oblivious David.
Maybe you just need to shut up. It really is that easy. Or if it is so important to you to explain something again that has already been explained in this thread AND REPEATEDLY REFERENCED, then why don't you explain it rather than telling me what to do.
Last edit: jwill42 2014-11-05
I didn't tell you what to do. I used "would" and I asked you. Whatever dude. I'm done with you and your instant abusive attitude in here.
Go take out your frustrations somewhere else.
Whatever you say, Oblivious David. Just as long as you stop telling other people what to do when really you are the one who needs to shut up.
Really jwill42,
Is it possible that whenever there is a flame you are involved? Doesn't matter who is right or wrong. Just express your opinions without upsetting other users. I would really appreciate that.
Thanks,
Andrea
I cannot control whether other people get upset. I just state the facts and correct mistakes.
And it very much does matter what is right or wrong.
ETA: I will point out that in both of the threads in question, I stopped posting as soon as the person posting incorrect or misleading information stopped posting such things. Cause and effect, with a straightforward conclusion.
Last edit: jwill42 2014-11-05
In MY case putting a temporary download folder on one of the data drives that is excluded from Snapraid is a perfect good and fine thing. Should it be best practice? Don't think so. Is it stupid beyond belief? Don't think so.
The main reason why it's not: It's using the space that should not be used since my data drives and parity drives are the same size and I won't like to split them into different partitions or something. And my VM files are stored on SSDs and are not protected by SR, so that's not an option. It's just simple and plain the only thing I can use it for. And space isn't a problem in this case, so I have a nice chunk of space available there.
Imho this should be up to the user, but if someone knows what he is doing there I don't think it's stupid.
therealjmc:
You seem to have missed the point. The suggestion which is stupid is putting a temporary directory (or anything of changeable size) on a data drive IN ORDER TO RESERVE SPACE and keep SnapRAID from having parity drive out of space issues.
If you are not concerned about reserving space, then my comment has no bearing on how you utilize unused space on your data drives.
I think jwill was trying to say that a temp dir wouldn't maintain a consistent size, and thus wouldn't strictly enforce a reduced size on the snapraid protected part of the disk. But if you want to enforce the limit that strictly, surely shrinking the SR partition would accomplish that even more strictly, allowing you to use the remaining part for temp files or leaked NSA docs or child porn or whatever.
Next strictest would be a large statically size file(s), per jwill. (But if they're truly static, rather than just static-sized then why not include them in SR? I'm not sure how many things are of a constant size, but change internally - other than the VM example given.) More flexible than partitioning, but stricter than a temp dir.
Least strict would be a temp dir or whatever, and then you just remember the target size of the SR area, and enforce it mentally. Or I guess you could write a little script that monitors the size of the temp dir, and writes / deletes junk files to maintain a constant size. This would probably be my choice.
As jmc said, the stupidity of any of these solutions depends on the user & context.
Why do you say that putting a large static file on a data drive is "less strict" than using a partition that is smaller by the size of the large static file?
I think either one is just as effective. The advantage of the static file technique is that you do not need to shrink or grow active partitions in the event that you later decide to change the amount of reserved space. The advantage of partitioning is that you could then use the extra space for something of changeable size, since the partition itself enforces the reserved space.
Your second paragraph below answers your first. Repartitioning is
harder than deleting a large static file - so if we're grading solutions
(and hence, users) on a scale of "strictness" (and the users by
"irresponsibility"), the former is more strict than the latter, as it
prevents someone (or someone else) from deciding they need the space to
download Transformers XXXIV now now now, and then forgetting about it or
otherwise getting themselves in a pickle.
The semantics hardly matter, though. The right solution for the user in
question is the perfect amount of "strict".
mitch
On 11/5/2014 12:07 PM, jwill42 wrote:
I think semantics matter a bit, since I misunderstood your previous point until you elaborated. I think you meant difficulty rather than strictness. For strictness, if the reserve file had the permissions set so that only root could modify or delete it, then it is just as strict as partitioning since you would presumably need root access to change the partition size.
Anyway, I think we agree that either a fixed-size reserve file or partitioning can be best practice for reserving data drive space, depending on the specific requirements of an installation.