i would like to know if this unit ICY BOX IB-3810U3 will work with snapraid. this case is a JBOD-case with 10-bays, connected through USB-3 on Windows 7 64-bit.
i want to put 10 HDD, with 3 or 4 parity drives. all 10 drives will have the same size (6TB or 8TB).
the purpose is to use this combination as backup-storage. everyday i would like to backup 3 GB of data to this combination. the 3 GB data are 90% small files (about 30-40kb, mostly WORD-documents).
maybe the first question should be: is snapraid the right option for my backup-plans? because i will make a backup everyday of 3 GB and the files are small size (30kb) files. this means everyday 3 GB is coming by, so my backup is everyday growing with 3 GB....thanks in advance
oh yeah...i am totally new to snapraid.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The USB connection on the icy box is limited to 5 Gigabits/s, which means that if all disks are access simultatiously you would be looking at maximum speeds in the area of 50 MiB/s.
The default configuration of snapraid uses 256 kiB parity blocks, which means that you would need an 8 TB parity disk to protect ~1 TB data if all the data files are tiny ~30 kB files.
Microsoft office documents are backup-unfriendly by design. Each time you open and close an office document it is modified even if you don't do any changes and don't save.
None of these issues need to be deal-breakers by themself.
You could live with the slow performance.
You could change the block size in snapraid to 16 kiB to better match the file sizes but you would then also increase the amount of RAM needed to 1 GiB per TB data stored (maybe even more to account for millions of file namnes and related things to keep track of).
You could archive the files in larger zip or rar archives before you put them into backup to avoid them being secretly modified when opened and at the same time make the tiny sizes of individual files a non.-issue.
TLDR: It doesn't seem like an ideal backup plan.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The only remaining issue would be the slow speed per disk, but since you only plan to add a few GB per day that shouldn't really be an issue. The most noticable consequence would be that recovering a lost 8 TB disk will probably take ~2 days instead of 1 day.
Please note that I have absolutely no experience or knowledge on how reliable the icy box itself is.
You may want to google some on that topic before going all in.
Last edit: Leifi Plomeros 2017-09-23
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
i neither do know the reliability of icy box. but i want to put enterprise hard drives with URE 10^15. but as i understand the combination with this JBOD-case and snapraid will work? becaus JBOD-case is a DAS, like a regular external hard drive
Last edit: ari 2017-09-23
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Snapraid will happily work with USB drives, so as long as Windows can partition them as 10 different disks it would not be any problem.
But I really don't see any need for enterprise drives in this setup.
If the backup data is so important that the already overkill in parity drives you are planning isn't enough, then it would be better to spend the extra cost on an additional separate backup (preferably off site), to avoid single point of failure (collective overheating, lightning strike, etc).
Last edit: Leifi Plomeros 2017-09-23
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Snapraid will happily work with USB drives, so as long as Windows can partition them as 10 different disks it would not be any problem.
......ok, this is not a probrem for this icy box case. but this not what i want. i want to see one big partition in my windows managed by snapraid. is this posibble with snapraid? or snapraid will treat each of those 10 drives separatly as a 1 partition. or let me ask this way....if i put 10 drives in icy-box, do i need in snapraid to assing some drive as data and some drives as parity? if the answer is YES, than i have configure the icy-box in a way so the windows can see them as 10 different drives. am i right?
> If the backup data is so important that the already overkill in parity drives you are planning isn't enough, then it would be better to spend the extra cost on an additional separate backup (preferably off site), to avoid single point of failure (collective overheating, lightning strike, etc).
......yes, i know that. i am also planning offsite storage of 2nd copy of backup. more clrearly explained. ==> i make a zip/rar archive of my all files on the main desktop, from main desktop i copy this zip/rar archive (3 GB) to icy-box (managed by snapraid) as my 1st backup. then i copy once more to my mobile ssd-drive for transportation to an offsite place. in the offsite place i put the 2nd copy of 3 GB zip-archive. in the mean time i have 3 copies of the zip-archive, icy-box (snapraid), mobile-ssd and offsite place. rocket enough?
Last edit: ari 2017-09-23
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
i want to see one big partition in my windows managed by snapraid. is this posibble with snapraid?
No, Snapraid has a pooling option but it does not give you a single partition. Instead it gives you a folder with symbolic links to all the files in the snapraid array (which is still a collection of separate disks).
if i put 10 drives in icy-box, do i need in snapraid to assing some drive as data and some drives as parity? if the answer is YES, than i have configure the icy-box in a way so the windows can see them as 10 different drives. am i right?
Yes, Right.
rocket enough?
Yes
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
There is a Windows program called Drive Bender that can be used to create a virtual unified file system from the individual drives. It duplicates the directory structure across all the drives and then selects one of the drives as the target for each new file. Each file will physically exist on one drive only but you'll see the entire collection of drives as one single directory structure.
While I've not used SnapRAID with Drive Bender I'm quite certain it will work just fine. You'll want SnapRAID to know about the individual drives, and this shouldn't be a problem with Drive Bender. You'll view things through Drive Bender as a unified file space and SnapRAID will operate one level lower, working with the actual drives individually.
One of the benefits of Drive Bender is you can move the drives to another system if needed and your files will still be accessible. They're scattered about over all the drives, but you can still get at them if need be independent of Drive Bender. I did this just recently as I migrated from a Drive Bender based Windows setup to a Linux server. I mounted the drives and copied everything off, then reformatted them with a file system better for use with Linux. (They had been NTFS. Incidentally, my move away from Drive Bender had little to do with Drive Bender and a lot to do with a large lightning strike very close by.)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You can use both Stablebit DrivePool and Drive Bender together with snapraid.
I personally have almost no expericene with drive bender. I tried it and ran into issues with folder permissions, tried drivepool instead and found it less troublesome.
Both products have auto balancing activated by default, which you absolutely must get rid of.
Spreading out files evenly on multiple disks is perfectly OK, randomly moving them around from one disk to another is not.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I looked at both Drive Bender and Drive Pool. For reasons I don't now remember, I chose Drive Bender over Drive Pool. As near as I can tell, the basic functionality of the two products is similar. I just mentioned Drive Bender because it was the one I used and was reasonably happy with.
Now that I've used mergerFS under Linux, which provides a similar capability, I greatly prefer the mergerFS approach to directories. It always bothered me a little that Drive Bender duplicated the entire directory structure across all drives in the pool. mergerFS can be configured to duplicate directories only when needed due to drives filling up. That approach gives you much greater control over where things go and also allows for keeping files associated by directory physically together. With Drive Bender I'd frequently end up with the odd numbered TV episodes on one drive and the even numbered ones on another. It doesn't matter if you always view through the Drive Bender unified file space, but if you look at the physical drives it can be a bit disconcerting. With mergerFS the episodes of a TV series stay together on one drive unless that drive fills up, then the new ones are put on a single other drive (until that one fills up). I much prefer the mergerFS approach on this.
At any rate, I tihnk either Drive Bender or Drive Pool would be a viable solution for ari.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You've said what your data capacity requirements are but nothing about operational requirements. Will you ever be changing any of the existing data? What kind of protection do you need for the data? Will the data ever be accessed in anything other than a backup capacity? Will you ever delete any of the existing data? Might you want protection against erroneous deletions?
Based on what you've said -- small daily changes done all at once -- SnapRAID would seem to be a good data protection solution. That assumes running a SnapRAID sync once a day won't interfere with things on the hosting computer.
Given the resources you seem willing to throw at this, the only alternative that comes to mind is putting together a real server running BSD and using ZFS. Some assembly required, and the learning curve would be fairly steep if you don't already know ZFS. I just recently started using SnapRAID and I have to say it was far easier to get started than I thought it would be. I've even recovered a lost disk already and that was very easy as well.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
everyday i will make a full backup of my files (a zip-archive of 3 GB). then i will copy this 3 GB ZIP to the external usb which is managed by snapraid. i am also planning to do a full disk image (VHD-file) copy, then put this VHD-file on external usb. all this means that for 1 month of all bakcups will be around 350 GB.
the backup-data will only be used in case of neccesity, e.g. drive failure in the main desktop, we are not working from or on backup-data. protection against errorneous deletion is not obligatio, it is more wanted (would be nice to have it)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Sorry to revive this old discussion, but I'm thinking about gettting this 10-bay box to use with snapraid + drivepool/scanner.
The one thing I'm concerned about is the speed while scanning. Right now using an HBA (M1015) I get up to about 150MB/s scanning speed on each of my 10 HDDs using Scanner. So with this box, am I right that scanning speed using Stablebit Scanner would be drastically lower than this over the USB 3.0 connection?
Last edit: pespn2000 2019-01-24
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, a single USB3 connection can at best handle 625 MB/s.
Consider building your own DAS instead:
LSI SAS 9200-16e or SAS 9201-16e (used ones are very cheap on ebay)
Any computer chassis + power supply with jumper or switch for on/off
Some SFF-8088 to 4 SATA-cables.
The 16e card goes into your current computer and the SAS cables goes directly to the HDD's in the external chassis.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
i would like to know if this unit ICY BOX IB-3810U3 will work with snapraid. this case is a JBOD-case with 10-bays, connected through USB-3 on Windows 7 64-bit.
http://www.raidsonic.de/en/standards/searchresults.php?we_objectID=3410
i want to put 10 HDD, with 3 or 4 parity drives. all 10 drives will have the same size (6TB or 8TB).
the purpose is to use this combination as backup-storage. everyday i would like to backup 3 GB of data to this combination. the 3 GB data are 90% small files (about 30-40kb, mostly WORD-documents).
maybe the first question should be: is snapraid the right option for my backup-plans? because i will make a backup everyday of 3 GB and the files are small size (30kb) files. this means everyday 3 GB is coming by, so my backup is everyday growing with 3 GB....thanks in advance
oh yeah...i am totally new to snapraid.
The USB connection on the icy box is limited to 5 Gigabits/s, which means that if all disks are access simultatiously you would be looking at maximum speeds in the area of 50 MiB/s.
The default configuration of snapraid uses 256 kiB parity blocks, which means that you would need an 8 TB parity disk to protect ~1 TB data if all the data files are tiny ~30 kB files.
Microsoft office documents are backup-unfriendly by design. Each time you open and close an office document it is modified even if you don't do any changes and don't save.
None of these issues need to be deal-breakers by themself.
You could live with the slow performance.
You could change the block size in snapraid to 16 kiB to better match the file sizes but you would then also increase the amount of RAM needed to 1 GiB per TB data stored (maybe even more to account for millions of file namnes and related things to keep track of).
You could archive the files in larger zip or rar archives before you put them into backup to avoid them being secretly modified when opened and at the same time make the tiny sizes of individual files a non.-issue.
TLDR: It doesn't seem like an ideal backup plan.
i am sorry if i was not clear....i will maken one big zip-file (3GB) and then put it on this case/combination. would be this e better plan?
Yes, it would make it a perfectly good plan.
The only remaining issue would be the slow speed per disk, but since you only plan to add a few GB per day that shouldn't really be an issue. The most noticable consequence would be that recovering a lost 8 TB disk will probably take ~2 days instead of 1 day.
Please note that I have absolutely no experience or knowledge on how reliable the icy box itself is.
You may want to google some on that topic before going all in.
Last edit: Leifi Plomeros 2017-09-23
i neither do know the reliability of icy box. but i want to put enterprise hard drives with URE 10^15. but as i understand the combination with this JBOD-case and snapraid will work? becaus JBOD-case is a DAS, like a regular external hard drive
Last edit: ari 2017-09-23
Snapraid will happily work with USB drives, so as long as Windows can partition them as 10 different disks it would not be any problem.
But I really don't see any need for enterprise drives in this setup.
If the backup data is so important that the already overkill in parity drives you are planning isn't enough, then it would be better to spend the extra cost on an additional separate backup (preferably off site), to avoid single point of failure (collective overheating, lightning strike, etc).
Last edit: Leifi Plomeros 2017-09-23
......ok, this is not a probrem for this icy box case. but this not what i want. i want to see one big partition in my windows managed by snapraid. is this posibble with snapraid? or snapraid will treat each of those 10 drives separatly as a 1 partition. or let me ask this way....if i put 10 drives in icy-box, do i need in snapraid to assing some drive as data and some drives as parity? if the answer is YES, than i have configure the icy-box in a way so the windows can see them as 10 different drives. am i right?
> If the backup data is so important that the already overkill in parity drives you are planning isn't enough, then it would be better to spend the extra cost on an additional separate backup (preferably off site), to avoid single point of failure (collective overheating, lightning strike, etc).
......yes, i know that. i am also planning offsite storage of 2nd copy of backup. more clrearly explained. ==> i make a zip/rar archive of my all files on the main desktop, from main desktop i copy this zip/rar archive (3 GB) to icy-box (managed by snapraid) as my 1st backup. then i copy once more to my mobile ssd-drive for transportation to an offsite place. in the offsite place i put the 2nd copy of 3 GB zip-archive. in the mean time i have 3 copies of the zip-archive, icy-box (snapraid), mobile-ssd and offsite place. rocket enough?
Last edit: ari 2017-09-23
No, Snapraid has a pooling option but it does not give you a single partition. Instead it gives you a folder with symbolic links to all the files in the snapraid array (which is still a collection of separate disks).
Yes, Right.
Yes
There is a Windows program called Drive Bender that can be used to create a virtual unified file system from the individual drives. It duplicates the directory structure across all the drives and then selects one of the drives as the target for each new file. Each file will physically exist on one drive only but you'll see the entire collection of drives as one single directory structure.
While I've not used SnapRAID with Drive Bender I'm quite certain it will work just fine. You'll want SnapRAID to know about the individual drives, and this shouldn't be a problem with Drive Bender. You'll view things through Drive Bender as a unified file space and SnapRAID will operate one level lower, working with the actual drives individually.
One of the benefits of Drive Bender is you can move the drives to another system if needed and your files will still be accessible. They're scattered about over all the drives, but you can still get at them if need be independent of Drive Bender. I did this just recently as I migrated from a Drive Bender based Windows setup to a Linux server. I mounted the drives and copied everything off, then reformatted them with a file system better for use with Linux. (They had been NTFS. Incidentally, my move away from Drive Bender had little to do with Drive Bender and a lot to do with a large lightning strike very close by.)
I thought drive bender was discontinued.
https://blog.division-m.com/2015/11/19/all-good-things-must-come-to-an-end/
But apparently that is not the case any more.
https://www.division-m.com/downloads/drive-bender-change-log.html
You can use both Stablebit DrivePool and Drive Bender together with snapraid.
I personally have almost no expericene with drive bender. I tried it and ran into issues with folder permissions, tried drivepool instead and found it less troublesome.
Both products have auto balancing activated by default, which you absolutely must get rid of.
Spreading out files evenly on multiple disks is perfectly OK, randomly moving them around from one disk to another is not.
I looked at both Drive Bender and Drive Pool. For reasons I don't now remember, I chose Drive Bender over Drive Pool. As near as I can tell, the basic functionality of the two products is similar. I just mentioned Drive Bender because it was the one I used and was reasonably happy with.
Now that I've used mergerFS under Linux, which provides a similar capability, I greatly prefer the mergerFS approach to directories. It always bothered me a little that Drive Bender duplicated the entire directory structure across all drives in the pool. mergerFS can be configured to duplicate directories only when needed due to drives filling up. That approach gives you much greater control over where things go and also allows for keeping files associated by directory physically together. With Drive Bender I'd frequently end up with the odd numbered TV episodes on one drive and the even numbered ones on another. It doesn't matter if you always view through the Drive Bender unified file space, but if you look at the physical drives it can be a bit disconcerting. With mergerFS the episodes of a TV series stay together on one drive unless that drive fills up, then the new ones are put on a single other drive (until that one fills up). I much prefer the mergerFS approach on this.
At any rate, I tihnk either Drive Bender or Drive Pool would be a viable solution for ari.
Another alternative worth looking into could be flexraid pool.
You've said what your data capacity requirements are but nothing about operational requirements. Will you ever be changing any of the existing data? What kind of protection do you need for the data? Will the data ever be accessed in anything other than a backup capacity? Will you ever delete any of the existing data? Might you want protection against erroneous deletions?
Based on what you've said -- small daily changes done all at once -- SnapRAID would seem to be a good data protection solution. That assumes running a SnapRAID sync once a day won't interfere with things on the hosting computer.
Given the resources you seem willing to throw at this, the only alternative that comes to mind is putting together a real server running BSD and using ZFS. Some assembly required, and the learning curve would be fairly steep if you don't already know ZFS. I just recently started using SnapRAID and I have to say it was far easier to get started than I thought it would be. I've even recovered a lost disk already and that was very easy as well.
everyday i will make a full backup of my files (a zip-archive of 3 GB). then i will copy this 3 GB ZIP to the external usb which is managed by snapraid. i am also planning to do a full disk image (VHD-file) copy, then put this VHD-file on external usb. all this means that for 1 month of all bakcups will be around 350 GB.
the backup-data will only be used in case of neccesity, e.g. drive failure in the main desktop, we are not working from or on backup-data. protection against errorneous deletion is not obligatio, it is more wanted (would be nice to have it)
Sorry to revive this old discussion, but I'm thinking about gettting this 10-bay box to use with snapraid + drivepool/scanner.
The one thing I'm concerned about is the speed while scanning. Right now using an HBA (M1015) I get up to about 150MB/s scanning speed on each of my 10 HDDs using Scanner. So with this box, am I right that scanning speed using Stablebit Scanner would be drastically lower than this over the USB 3.0 connection?
Last edit: pespn2000 2019-01-24
Yes, a single USB3 connection can at best handle 625 MB/s.
Consider building your own DAS instead:
LSI SAS 9200-16e or SAS 9201-16e (used ones are very cheap on ebay)
Any computer chassis + power supply with jumper or switch for on/off
Some SFF-8088 to 4 SATA-cables.
The 16e card goes into your current computer and the SAS cables goes directly to the HDD's in the external chassis.