I currently pursue the plan to secure a fleet of 16 to 20 8 TB HDDs with the help of SnapRAID.
Each HDD is encrypted through BitLocker (AES) and contains up to 3 million files.
The computer in which the HDDs will be gathered is meant as a pure file server with the following specification:
Supermicro X11, 8 SATA ports, IPMI
16 GB ECC RAM
LSI 9206 SAS controller, 16 ports
Windows 8.1 OS on SSD
A mixture of WD Blue, Seagate Archive 8 TB HDDs
With regards to the workload BitLocker and the myriads of files imposes on the CPU, the crucial question for me is, which ECC capable processor is powerful enough to match my criteria?
Thanks for any idea!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If you have so many files/folders/disks the initial disk scan (at sync start) could take very long (~1h) and the .content files will become huge (~40GB). I think Snapraid performs best with few big files.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
What is the expected workload? The total number of files aren't really going to impact the choice of CPU. What kind of data access rate are you expecting? If a dozen users are hitting it up constantly for data I assume you're going to need more CPU than if one user transfers a couple GB per day.
I recently built a pure file server running SnapRAID and mergerFS on Debian. It has a Celeron G3930 CPU. I was trying to keep costs down and this CPU cost me just $40. It's pretty meager by today's CPU standards, though newly released by Intel when I bought it, but it's perfectly adequate for this usage. Of course, I don't have my data encrypted and I'm not running Windows. The system is usually cruising at less than 1% CPU usage and under 200MB RAM usage. When SnapRAID is scrubbing at ~1200MB/s it's using about 65% of both CPUs.
I also feel compelled to ask why you intend to use Windows on a pure server. That choice alone is likely bringing a lot of unnecessary overhead.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I already run an almost identically equipped file server with an Intel i3-6300 for a 1-user-scenario.
The major difference is that it contains 23 4 TB HDDs that aren't encrypted at all and the number of files is dramatically lower.
Here I see syncing speeds of up to 2800 MB/s and a CPU load that not seldomly remains under 90 %.
That is the reason why I am sceptical about an i3 being able to drive the 2nd server with the extra burdens of encryption and creating parity for millions of files.
I have opted for Windows because I am bound to utilize particular file managing applications that are restricted to the Windows platform.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Given the batch processing nature of SnapRAID, would it be critical that the CPU stay under 100% utilization during syncs? I would venture that if such were the case, SnapRAID might not be the best solution. If a SnapRAID sync is regularly taking more than a couple hours in the middle of the night, you either have a grossly inadequate CPU or you have a use case where conventional RAID is probably a better solution. So if SnapRAID is the right choice, is the difference between two hours of sync and three hours of sync in the middle of the night actually something that's a concern?
Under normal use where data is likely coming from one or two disks at any given time, most any modern CPU is likely to be able to keep up with BitLocker. If you want sync operations to proceed at the kind of speed you're seeing now, my brief research would suggest you're going to need a lot more processing power. So the question becomes, how much are you willing to spend to speed up sync operations? In my personal situation the answer would be nothing because sync happens when I'm sleeping and it matters not a bit to me if it takes five minutes or five hours. As long as it doesn't seriously hinder normal usage of the server, I don't care how long it takes. Obviously, your answer here may quite legitimately be very different from mine.
You might poke around online and see if you can find any data rates from BitLocker tests with SSDs. That could give you a ballpark on what's possible. In my reading I found very little that was remotely applicable to reading from 20 drives simultaneously. All the tests I found were of single disk read speeds, and most of those were from seven years ago when BitLocker first got attention and people were concerned about how much it would impact performance. Most everything was handwaving about not changing boot time or being barely noticeable when loading application X, both situations with no connection to the data rates we're talking about.
Sorry I can't be of more help. If you find anything that would lead you to determining how much CPU you really need to support your very high data rate, please come back and let us know. I don't see myself ever needing to use encryption on that scale but it's still nice to know.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Personally I use the G4560 for my home NAS as of 2016 which is similar to an i7 860. It even supports VT-D:
Intel Pentium G4560 @ 3.50GHz
4917 (my current cpu)
Last edit: Magistar 2018-01-08
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have been using a few for at least a year. Basically these things come with a +- 20 GB write cache in which they are fast (even 3x faster than regular hdd for small files writes) and once they fill up the back-end starts to slow down, meaning you notice the write cache faster. As long as you don't write 100+ GB per disk per day they are great.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
i have no problems with this drives, but in that paper, he used these drives in a NAS for RAID 1. That operation mode is NOT supported by Seagate Archive drives. There special NAS drives for RAIDs.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have a total of seven archive drives, co-located with 9 WD Red and they work well for me too. Some of the archive drives are almost 3 years old and have been running 24/7.
My best guess to why 2 of 2 drives failed in short time for the author of the review is that he has extremely bad luck, a cat or clumpsy feets.
Last edit: Leifi Plomeros 2018-01-11
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I currently pursue the plan to secure a fleet of 16 to 20 8 TB HDDs with the help of SnapRAID.
Each HDD is encrypted through BitLocker (AES) and contains up to 3 million files.
The computer in which the HDDs will be gathered is meant as a pure file server with the following specification:
Supermicro X11, 8 SATA ports, IPMI
16 GB ECC RAM
LSI 9206 SAS controller, 16 ports
Windows 8.1 OS on SSD
A mixture of WD Blue, Seagate Archive 8 TB HDDs
With regards to the workload BitLocker and the myriads of files imposes on the CPU, the crucial question for me is, which ECC capable processor is powerful enough to match my criteria?
Thanks for any idea!
If you have so many files/folders/disks the initial disk scan (at sync start) could take very long (~1h) and the .content files will become huge (~40GB). I think Snapraid performs best with few big files.
What is the expected workload? The total number of files aren't really going to impact the choice of CPU. What kind of data access rate are you expecting? If a dozen users are hitting it up constantly for data I assume you're going to need more CPU than if one user transfers a couple GB per day.
I recently built a pure file server running SnapRAID and mergerFS on Debian. It has a Celeron G3930 CPU. I was trying to keep costs down and this CPU cost me just $40. It's pretty meager by today's CPU standards, though newly released by Intel when I bought it, but it's perfectly adequate for this usage. Of course, I don't have my data encrypted and I'm not running Windows. The system is usually cruising at less than 1% CPU usage and under 200MB RAM usage. When SnapRAID is scrubbing at ~1200MB/s it's using about 65% of both CPUs.
I also feel compelled to ask why you intend to use Windows on a pure server. That choice alone is likely bringing a lot of unnecessary overhead.
Thank you for your reply.
I already run an almost identically equipped file server with an Intel i3-6300 for a 1-user-scenario.
The major difference is that it contains 23 4 TB HDDs that aren't encrypted at all and the number of files is dramatically lower.
Here I see syncing speeds of up to 2800 MB/s and a CPU load that not seldomly remains under 90 %.
That is the reason why I am sceptical about an i3 being able to drive the 2nd server with the extra burdens of encryption and creating parity for millions of files.
I have opted for Windows because I am bound to utilize particular file managing applications that are restricted to the Windows platform.
Given the batch processing nature of SnapRAID, would it be critical that the CPU stay under 100% utilization during syncs? I would venture that if such were the case, SnapRAID might not be the best solution. If a SnapRAID sync is regularly taking more than a couple hours in the middle of the night, you either have a grossly inadequate CPU or you have a use case where conventional RAID is probably a better solution. So if SnapRAID is the right choice, is the difference between two hours of sync and three hours of sync in the middle of the night actually something that's a concern?
Under normal use where data is likely coming from one or two disks at any given time, most any modern CPU is likely to be able to keep up with BitLocker. If you want sync operations to proceed at the kind of speed you're seeing now, my brief research would suggest you're going to need a lot more processing power. So the question becomes, how much are you willing to spend to speed up sync operations? In my personal situation the answer would be nothing because sync happens when I'm sleeping and it matters not a bit to me if it takes five minutes or five hours. As long as it doesn't seriously hinder normal usage of the server, I don't care how long it takes. Obviously, your answer here may quite legitimately be very different from mine.
You might poke around online and see if you can find any data rates from BitLocker tests with SSDs. That could give you a ballpark on what's possible. In my reading I found very little that was remotely applicable to reading from 20 drives simultaneously. All the tests I found were of single disk read speeds, and most of those were from seven years ago when BitLocker first got attention and people were concerned about how much it would impact performance. Most everything was handwaving about not changing boot time or being barely noticeable when loading application X, both situations with no connection to the data rates we're talking about.
Sorry I can't be of more help. If you find anything that would lead you to determining how much CPU you really need to support your very high data rate, please come back and let us know. I don't see myself ever needing to use encryption on that scale but it's still nice to know.
The processing power of low-end cpu's has drastically increased the past few years. Check https://www.cpubenchmark.net/cpu_list.php
Intel Core i3-2100 @ 3.10GHz
3673
Intel Core i3-3210 @ 3.20GHz
4028
Intel Core i3-4150 @ 3.50GHz
4890
Intel Core i3-6100 @ 3.70GHz
5493
Intel Core i3-7100 @ 3.90GHz
5854
Intel Core i3-8100 @ 3.60GHz
8119
Personally I use the G4560 for my home NAS as of 2016 which is similar to an i7 860. It even supports VT-D:
Intel Pentium G4560 @ 3.50GHz
4917 (my current cpu)
Last edit: Magistar 2018-01-08
My next PC will have a i3-8100 since it's now a real quad core.
Seagate Archive 8 TB HDDs seem to have poor write performance and reliability: http://www.guru3d.com/articles_pages/seagate_archive_8tb_hdd_review,13.html
I have been using a few for at least a year. Basically these things come with a +- 20 GB write cache in which they are fast (even 3x faster than regular hdd for small files writes) and once they fill up the back-end starts to slow down, meaning you notice the write cache faster. As long as you don't write 100+ GB per disk per day they are great.
i have no problems with this drives, but in that paper, he used these drives in a NAS for RAID 1. That operation mode is NOT supported by Seagate Archive drives. There special NAS drives for RAIDs.
I have a total of seven archive drives, co-located with 9 WD Red and they work well for me too. Some of the archive drives are almost 3 years old and have been running 24/7.
My best guess to why 2 of 2 drives failed in short time for the author of the review is that he has extremely bad luck, a cat or clumpsy feets.
Last edit: Leifi Plomeros 2018-01-11