Could someone who's had time to read the source sanity check my understanding of the "wasted" block space calculation? The following is speculation based on the manual, so feel free to correct me.
I take it that files are diced into blocks of a certain size - meaning that the last block of each file is somewhere between 0-99% empty. Hence the section in the manual that indicates that a reasonable approximation for wasted block space is 50% of a block size * number of files. Presumably, this "wasted" space is in the parity file (i.e. the data files themselves don't take up more space), and represents the difference in the size of the parity file from the size on disk of the actual data.
So when calculating parity, SR grabs a block from each data drive. If each of these blocks corresponds to the middle of a file, then it's all data & nothing is wasted. If any block happens to come from the end of a file, that block has some padding in it, and thus the associated file has just "wasted" some parity room that could have been used for actual data.
However, in the case where one of the drives is fuller than the rest, it's likely that you haven't actually lost anything, because there's already a big chunk of the parity file that's calculated against padding. For the less-full drives, the wasted block space is just pushing whatever data it sits in further out into the already existing zone of "padded parity".
So in order to calculate how much headroom you need for your parity file, you need to do the "wasted" block calculation on each data drive individually, and make sure that drive maintains at least that much empty space.
So for my setup of about 7.6TB and 1.3 million total files, I'm losing ~160GB, or about 2.1% of my storage.
Comparing this against the RAM calculation: (5 disks * 3 TB * 28) / 256KB = 1.6GB of RAM needed. It looks like I'm better off shrinking my block size to 128KB, getting back 80GB of disk, and paying an additional 1.6GB RAM for it, yes?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hmmm, I'm not sure if that is quite right (and I wonder if the manual is misleading in its formula), although it sounds close. I would say to compute how many blocks each individual data drive uses, and whichever data drive has the most blocks, multiply that number of blocks by the block size to get the size of the parity file. If you want "wasted" space in the parity file, then subtract the actual data usage on that drive with the most blocks from the parity file size. Perhaps Andrea can confirm if this is correct.
As for your proposed change, since (as far as I know) snapraid does not support in-place shrinking of the block size, you will need to completely regenerate your parity and content files. If you have enough spare storage to keep a copy of the old parity and content files until the new ones are generated, great. If not, then you are going to be taking a risk during the generation of the new parity and content files.
I do not know what your 1.3 million files consist of, but if a lot of them are small, unimportant files, then you might consider just excluding them. My SnapRAID server is a media fileserver, and there are a lot of small metadata files that go along with the video files. I exclude all the metadata files from my snapraid data, since I can easily regenerate the metadata if necessary.
Last edit: jwill42 2014-02-17
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I would say to compute how many blocks each individual data drive uses, and whichever data drive has the most blocks, multiply that number of blocks by the block size to get the size of the parity file.
Isn't that roughly equivalent to what's in the manual / my post? The "compute how many blocks" part would be: data size / block size + # of files * 50%, no?
I do not know what your 1.3 million files consist of, but if a lot of them are small, unimportant files
For me the smaller files are probably the more important ones. Incremental system backups, project file backups, etc. 1.3mil over 7.6TB is ~= 6MB avg. file size, which feels about right for the majority of my important data. Video files are definitely the exception, and fall into two classes: 1) important (raw footage that I've shot personally), and 2) totally unimportant (entertainment b.s. that can be re-ripped or re-downloaded).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No, your post talked about the most full drive, by which I assume you meant the most bytes. That is not the same as the drive that uses the most blocks (it also depends on the number of files and the exact file sizes). Also, you talked about 7.6TB and 1.3 million files, but how many are on each drive?
For the block computation on each drive, in addition to the estimate already discussed, I suggest also computing the worst case.
The estimate (which will be close on average but could be
wrong in certain degenerate cases) is
est_num_blocks = KiB_used_on_drive / block_size_KiB
+ 0.5 * num_files_on_drive
The worst case is
wc_num_blocks = KiB_used_on_drive / block_size_KiB
+ num_files_on_drive
As for your small files, are they frequently modified? If so, I hope you run sync very frequently, because when many of your files are modified, it can create a corresponding "hole" in your restored data if you have a drive failure.
Last edit: jwill42 2014-02-17
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If you're referring to paragraph 4 in the OP, what I meant was: using the total # of files on the array is not correct, if you're trying to figure out how big your parity file will be. The following paragraph referenced all the drives individually, and the next one talked about all the files together. Apologies if it wasn't clear that these were three related-but-distinct ideas.
w.r.t. your question, all of the files are of the library/archive/backup flavor, in that they change when I decide to change them, not continuously on their own. My "live" data is segregated into its own area (for which I'll probably have a separate SR config).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I’m very confused by all this text, so I’m not sure if I am repeating something that has been said already, but the calculations for “space lost to blocksize” should work like this:
For each drive calculate parity_required = sum_of_filesizes + num_files * block_size/2.
Take the drive with the maximum parity_required and call that parity_required_max
Find the drive with the maximum sum_of_filesizes and call that parity_required_min
The parity space you loose due to the given blocksize is now parity_required_max - parity_required_min.
If you know you have a lot of files with size << 256kb, the estimate block_size/2 for wasted block space is not accurate anymore, though. You might want to use the full block_size for the first calculation instead. As jwill42 noted, that will give you a worst case estimate. But unless you know that most of your files are smaller than 100kb, this estimate is also way of. But now you at least have a lower and upper bound.
Are you running on linux or windows? Those 1.3 million total files are across all disks, right?
Last edit: Chronial 2014-02-17
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sounds about right to me. Of course, the actual value of parity_required is whatever the size of the parity file is. But it's useful to be able to compare this to the predicted size, so you know how much headroom to maintain.
A: I'm running linux, and yes, 1.3mil across 7.6TB data, all disks.
btw - file counts can be gotten thusly (linux):
df --inodes
or
find <directory> -type f | wc -l
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Your initial estimation of 160GB wasted for 1.3 million of files is correct. But if you are using linux, you can reduce it a lot formatting the parity partition with the -T largefile4 option of mkfs.
This will result in a lot bigger partition, reducing the wasted space. For each extra GB you get, you will reduce the wasted space of the same amount for each data disk. For example, with 20 extra GB in the parity partition and 8 data disks, you'll get back the 160 GB.
Also, any excluded file and the content files stored in the data disks, are reducing the wasted space. So, in the end, I suppose that this could be a problem only in some corner cases.
Ciao,
Andrea
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Andrea - thanks for the reply. Yup, I'm already using -m 0 and setting the number of inodes manually. I forget the value I used, I think it was around 11 million, which is a lot fewer than the default.
I seem to remember super_large_file and huge_file were both default options as well, though I'd have to check the man page to know what effect those have, if any.
And I think "problem" isn't black-and-white, anyway. 2% isn't a gigantic number, really. On the other hand, if you've got datasets that just happen to fit onto a nearly full drive, it could be a bummer. I believe I've got one drive that has about 30GB free - but it happens to be full of very large files, so it probably isn't going to overflow my parity headroom.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Could someone who's had time to read the source sanity check my understanding of the "wasted" block space calculation? The following is speculation based on the manual, so feel free to correct me.
I take it that files are diced into blocks of a certain size - meaning that the last block of each file is somewhere between 0-99% empty. Hence the section in the manual that indicates that a reasonable approximation for wasted block space is 50% of a block size * number of files. Presumably, this "wasted" space is in the parity file (i.e. the data files themselves don't take up more space), and represents the difference in the size of the parity file from the size on disk of the actual data.
So when calculating parity, SR grabs a block from each data drive. If each of these blocks corresponds to the middle of a file, then it's all data & nothing is wasted. If any block happens to come from the end of a file, that block has some padding in it, and thus the associated file has just "wasted" some parity room that could have been used for actual data.
However, in the case where one of the drives is fuller than the rest, it's likely that you haven't actually lost anything, because there's already a big chunk of the parity file that's calculated against padding. For the less-full drives, the wasted block space is just pushing whatever data it sits in further out into the already existing zone of "padded parity".
So in order to calculate how much headroom you need for your parity file, you need to do the "wasted" block calculation on each data drive individually, and make sure that drive maintains at least that much empty space.
So for my setup of about 7.6TB and 1.3 million total files, I'm losing ~160GB, or about 2.1% of my storage.
Comparing this against the RAM calculation: (5 disks * 3 TB * 28) / 256KB = 1.6GB of RAM needed. It looks like I'm better off shrinking my block size to 128KB, getting back 80GB of disk, and paying an additional 1.6GB RAM for it, yes?
Hmmm, I'm not sure if that is quite right (and I wonder if the manual is misleading in its formula), although it sounds close. I would say to compute how many blocks each individual data drive uses, and whichever data drive has the most blocks, multiply that number of blocks by the block size to get the size of the parity file. If you want "wasted" space in the parity file, then subtract the actual data usage on that drive with the most blocks from the parity file size. Perhaps Andrea can confirm if this is correct.
As for your proposed change, since (as far as I know) snapraid does not support in-place shrinking of the block size, you will need to completely regenerate your parity and content files. If you have enough spare storage to keep a copy of the old parity and content files until the new ones are generated, great. If not, then you are going to be taking a risk during the generation of the new parity and content files.
I do not know what your 1.3 million files consist of, but if a lot of them are small, unimportant files, then you might consider just excluding them. My SnapRAID server is a media fileserver, and there are a lot of small metadata files that go along with the video files. I exclude all the metadata files from my snapraid data, since I can easily regenerate the metadata if necessary.
Last edit: jwill42 2014-02-17
Thanks for the reply. Comments:
Isn't that roughly equivalent to what's in the manual / my post? The "compute how many blocks" part would be: data size / block size + # of files * 50%, no?
For me the smaller files are probably the more important ones. Incremental system backups, project file backups, etc. 1.3mil over 7.6TB is ~= 6MB avg. file size, which feels about right for the majority of my important data. Video files are definitely the exception, and fall into two classes: 1) important (raw footage that I've shot personally), and 2) totally unimportant (entertainment b.s. that can be re-ripped or re-downloaded).
No, your post talked about the most full drive, by which I assume you meant the most bytes. That is not the same as the drive that uses the most blocks (it also depends on the number of files and the exact file sizes). Also, you talked about 7.6TB and 1.3 million files, but how many are on each drive?
For the block computation on each drive, in addition to the estimate already discussed, I suggest also computing the worst case.
The estimate (which will be close on average but could be wrong in certain degenerate cases) is est_num_blocks = KiB_used_on_drive / block_size_KiB + 0.5 * num_files_on_drive The worst case is wc_num_blocks = KiB_used_on_drive / block_size_KiB + num_files_on_driveAs for your small files, are they frequently modified? If so, I hope you run sync very frequently, because when many of your files are modified, it can create a corresponding "hole" in your restored data if you have a drive failure.
Last edit: jwill42 2014-02-17
If you're referring to paragraph 4 in the OP, what I meant was: using the total # of files on the array is not correct, if you're trying to figure out how big your parity file will be. The following paragraph referenced all the drives individually, and the next one talked about all the files together. Apologies if it wasn't clear that these were three related-but-distinct ideas.
w.r.t. your question, all of the files are of the library/archive/backup flavor, in that they change when I decide to change them, not continuously on their own. My "live" data is segregated into its own area (for which I'll probably have a separate SR config).
I’m very confused by all this text, so I’m not sure if I am repeating something that has been said already, but the calculations for “space lost to blocksize” should work like this:
parity_required = sum_of_filesizes + num_files * block_size/2.parity_requiredand call thatparity_required_maxsum_of_filesizesand call thatparity_required_minparity_required_max - parity_required_min.If you know you have a lot of files with size << 256kb, the estimate
block_size/2for wasted block space is not accurate anymore, though. You might want to use the fullblock_sizefor the first calculation instead. As jwill42 noted, that will give you a worst case estimate. But unless you know that most of your files are smaller than 100kb, this estimate is also way of. But now you at least have a lower and upper bound.Are you running on linux or windows? Those 1.3 million total files are across all disks, right?
Last edit: Chronial 2014-02-17
Sounds about right to me. Of course, the actual value of parity_required is whatever the size of the parity file is. But it's useful to be able to compare this to the predicted size, so you know how much headroom to maintain.
A: I'm running linux, and yes, 1.3mil across 7.6TB data, all disks.
btw - file counts can be gotten thusly (linux):
Hi Mitchell,
Your initial estimation of 160GB wasted for 1.3 million of files is correct. But if you are using linux, you can reduce it a lot formatting the parity partition with the -T largefile4 option of mkfs.
Something like: mkfs.ext4 -m 0 -T largefile4 DEVICE
This will result in a lot bigger partition, reducing the wasted space. For each extra GB you get, you will reduce the wasted space of the same amount for each data disk. For example, with 20 extra GB in the parity partition and 8 data disks, you'll get back the 160 GB.
Also, any excluded file and the content files stored in the data disks, are reducing the wasted space. So, in the end, I suppose that this could be a problem only in some corner cases.
Ciao,
Andrea
Andrea - thanks for the reply. Yup, I'm already using -m 0 and setting the number of inodes manually. I forget the value I used, I think it was around 11 million, which is a lot fewer than the default.
I seem to remember super_large_file and huge_file were both default options as well, though I'd have to check the man page to know what effect those have, if any.
And I think "problem" isn't black-and-white, anyway. 2% isn't a gigantic number, really. On the other hand, if you've got datasets that just happen to fit onto a nearly full drive, it could be a bummer. I believe I've got one drive that has about 30GB free - but it happens to be full of very large files, so it probably isn't going to overflow my parity headroom.