Menu
▾
▴
moosefs-users
|
From: Michael T. <mic...@ho...> - 2015-07-15 07:24:06
|
Hi. Is anybody using btrfs for moosefs? I currently have a 2-chunkserver (using jbod, ext4 fs, and goal=2 for all files) setup running that has recently experienced an hd failure in one of the chunkservers. While replacing the failed hdd was an almost trivial task, there was a long span of time where a big number of chunks only had 1 copy. While I did not lose sleep over it, I had this concern of potentially losing chunks should there be another hd failure in the other chunkserver while moosefs is still replicating the 1-copy chunks. So now I'm entertaining the idea of moving from just-a-bunch-of-ext4-disks to one-big-btrfs-raid1-fs as the underlying storage provider -- using Ubuntu 14.04 distro plus the 4.1.x LTS kernel from kernel-ppa. I'm thinking that with such a setup I would still be up and running with 2 copies for all chunks even if both chunkservers have 1 disk failure each at the same time. And the setup would survive a failed chunkserver with 1 failed disk in the remaining chunkserver. I would like to hear your thoughts on this. Regards. --- mike t. |
|
From: Aleksander W. <ale...@mo...> - 2015-07-15 13:16:04
|
Hi Michael We had bad experience with BTFS so we are not advising to use this fs as backend for chunkserver HDD. We recommend XFS because we achieved the best performance with disk of 90% usage. We have extra questions: How big your MooseFS is - how many files you have? What is your LAN speed? What is the MooseFS version? Best regards Aleksander Wieliczko Technical Support Engineer MooseFS.com <moosefs.com> On 15.07.2015 09:24, Michael Tinsay wrote: > Hi. > > Is anybody using btrfs for moosefs? > > I currently have a 2-chunkserver (using jbod, ext4 fs, and goal=2 for > all files) setup running that has recently experienced an hd failure > in one of the chunkservers. While replacing the failed hdd was an > almost trivial task, there was a long span of time where a big number > of chunks only had 1 copy. While I did not lose sleep over it, I had > this concern of potentially losing chunks should there be another hd > failure in the other chunkserver while moosefs is still replicating > the 1-copy chunks. > > So now I'm entertaining the idea of moving from > just-a-bunch-of-ext4-disks to one-big-btrfs-raid1-fs as the underlying > storage provider -- using Ubuntu 14.04 distro plus the 4.1.x LTS > kernel from kernel-ppa. > > I'm thinking that with such a setup I would still be up and running > with 2 copies for all chunks even if both chunkservers have 1 disk > failure each at the same time. And the setup would survive a failed > chunkserver with 1 failed disk in the remaining chunkserver. > > I would like to hear your thoughts on this. > > Regards. > > > --- mike t. > > > ------------------------------------------------------------------------------ > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that > you need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Ricardo J. B. <ric...@do...> - 2015-07-15 17:39:34
|
Maybe you could consider another chunkserver and a goal of 3? It's one more chunkserver but only 1.5x disk space compared to 2x with 2 chunkservers and RAID1 (whether by software or hardware). El Miércoles 15/07/2015, Aleksander Wieliczko escribió: > Hi Michael > > We had bad experience with BTFS so we are not advising to use this fs as > backend for chunkserver HDD. > We recommend XFS because we achieved the best performance with disk of > 90% usage. > > We have extra questions: > How big your MooseFS is - how many files you have? > What is your LAN speed? > What is the MooseFS version? > > Best regards > Aleksander Wieliczko > Technical Support Engineer > MooseFS.com <moosefs.com> > > On 15.07.2015 09:24, Michael Tinsay wrote: > > Hi. > > > > Is anybody using btrfs for moosefs? > > > > I currently have a 2-chunkserver (using jbod, ext4 fs, and goal=2 for > > all files) setup running that has recently experienced an hd failure > > in one of the chunkservers. While replacing the failed hdd was an > > almost trivial task, there was a long span of time where a big number > > of chunks only had 1 copy. While I did not lose sleep over it, I had > > this concern of potentially losing chunks should there be another hd > > failure in the other chunkserver while moosefs is still replicating > > the 1-copy chunks. > > > > So now I'm entertaining the idea of moving from > > just-a-bunch-of-ext4-disks to one-big-btrfs-raid1-fs as the underlying > > storage provider -- using Ubuntu 14.04 distro plus the 4.1.x LTS > > kernel from kernel-ppa. > > > > I'm thinking that with such a setup I would still be up and running > > with 2 copies for all chunks even if both chunkservers have 1 disk > > failure each at the same time. And the setup would survive a failed > > chunkserver with 1 failed disk in the remaining chunkserver. > > > > I would like to hear your thoughts on this. > > > > Regards. > > > > > > --- mike t. > > > > > > ------------------------------------------------------------------------- > >----- Don't Limit Your Business. Reach for the Cloud. > > GigeNET's Cloud Solutions provide you with the tools and support that > > you need to offload your IT needs and focus on growing your business. > > Configured For All Businesses. Start Your Cloud Today. > > https://www.gigenetcloud.com/ > > > > > > _________________________________________ > > moosefs-users mailing list > > moo...@li... > > https://lists.sourceforge.net/lists/listinfo/moosefs-users -- Ricardo J. Barberis Senior SysAdmin / IT Architect DonWeb La Actitud Es Todo www.DonWeb.com |
|
From: Michael T. <mic...@ho...> - 2015-07-16 02:09:43
|
Is there a performance boost in increasing the chunkservers and goal? Adding an additional chunkserver would be more expensive than doubling the current storage space; and I still have enough drive bays in each chunkserver for this. So at the moment, I can only justify spending for an additional server if there is a significant performance improvement. It is very gratifying to see that upgrading to 2.x would greatly reduce my concern. So I guess I'll have to stick it out with ext4 until btrfs is tested more -- I had several bad experiences (fs/data corruption) with xfs in the past that made me stay away from it for good. Thanks for the insights Aleksander and Ricardo. Best Regards. Mike Tinsay Subject: Re: [MooseFS-Users] BTRFS From: ale...@mo... Date: Wed, 15 Jul 2015 23:39:33 +0200 CC: moo...@li... To: mic...@ho... Hi. In your situation the best idea is to add another chunk server and change GOAL to 3 as it was mentioned by Ricardo J. Barberis.Replication mechanism in MooseFS 2.0 and 3.0 is redesigned, and it works much more efficient than in old 1.6 version.So for example if you loose one disk(full 1TB) it will replicate in less than 2 hours - 1TB RAID1 rebuild will take much much more time than this. Another problems:- RAID1 is slower than JBOD in MooseFS- Available space is lower when you are using RAID1 for chunkserver disk than JBOD. About BTRFS - we have tested older version of BTFS an we had kernel panic during tests.We didn’t test the latest available release but we will in the nearest future.At this moment we recommend XFS as backend FS for chunkserver disks. Best regardsAleksander WieliczkoTechnical Support EngineerMooseFS.com <moosefs.com> On 15 Jul 2015, at 19:17, Ricardo J. Barberis <ric...@do...> wrote:Maybe you could consider another chunkserver and a goal of 3?It's one more chunkserver but only 1.5x disk space compared to 2x with 2 chunkservers and RAID1 (whether by software or hardware).El Miércoles 15/07/2015, Aleksander Wieliczko escribió:Hi Michael We had bad experience with BTFS so we are not advising to use this fs as backend for chunkserver HDD. We recommend XFS because we achieved the best performance with disk of 90% usage. We have extra questions: How big your MooseFS is - how many files you have? What is your LAN speed? What is the MooseFS version? Best regards Aleksander Wieliczko Technical Support Engineer MooseFS.com <moosefs.com> On 15.07.2015 09:24, Michael Tinsay wrote: Hi. Is anybody using btrfs for moosefs? I currently have a 2-chunkserver (using jbod, ext4 fs, and goal=2 for all files) setup running that has recently experienced an hd failure in one of the chunkservers. While replacing the failed hdd was an almost trivial task, there was a long span of time where a big number of chunks only had 1 copy. While I did not lose sleep over it, I had this concern of potentially losing chunks should there be another hd failure in the other chunkserver while moosefs is still replicating the 1-copy chunks. So now I'm entertaining the idea of moving from just-a-bunch-of-ext4-disks to one-big-btrfs-raid1-fs as the underlying storage provider -- using Ubuntu 14.04 distro plus the 4.1.x LTS kernel from kernel-ppa. I'm thinking that with such a setup I would still be up and running with 2 copies for all chunks even if both chunkservers have 1 disk failure each at the same time. And the setup would survive a failed chunkserver with 1 failed disk in the remaining chunkserver. I would like to hear your thoughts on this. Regards. --- mike t. ------------------------------------------------------------------------- ----- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ _________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users -- Ricardo J. BarberisSenior SysAdmin / IT ArchitectDonWebLa Actitud Es Todowww.DonWeb.com------------------------------------------------------------------------------Don't Limit Your Business. Reach for the Cloud.GigeNET's Cloud Solutions provide you with the tools and support thatyou need to offload your IT needs and focus on growing your business.Configured For All Businesses. Start Your Cloud Today.https://www.gigenetcloud.com/_________________________________________moosefs-users mailing lis...@li...://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: F. O. O. <oz...@gm...> - 2015-07-16 15:00:41
|
On 07/15/2015 10:09 PM, Michael Tinsay wrote: > > Is there a performance boost in increasing the chunkservers and goal? > Adding an additional chunkserver would be more expensive than doubling > the current storage space; and I still have enough drive bays in each > chunkserver for this. So at the moment, I can only justify spending for > an additional server if there is a significant performance improvement. > > It is very gratifying to see that upgrading to 2.x would greatly reduce > my concern. So I guess I'll have to stick it out with ext4 until btrfs > is tested more -- I had several bad experiences (fs/data corruption) > with xfs in the past that made me stay away from it for good. We have used xfs on several servers with couple hundred TBs of actual data with no problems. I don't know how long ago you had bad experiences with xfs but it is probably worth trying again. Even in the case of an underlying filesystem corruption, keep in mind that MooseFS has built-in error detection and self healing capabilities. If it detects an error with a chunk it will recreate it from a safe copy of that chunk. > > Thanks for the insights Aleksander and Ricardo. > > Best Regards. > > Mike Tinsay > > > ------------------------------------------------------------------------ > Subject: Re: [MooseFS-Users] BTRFS > From: ale...@mo... > Date: Wed, 15 Jul 2015 23:39:33 +0200 > CC: moo...@li... > To: mic...@ho... > > Hi. > > In your situation the best idea is to add another chunk server and > change GOAL to 3 as it was mentioned by Ricardo J. Barberis. > Replication mechanism in MooseFS 2.0 and 3.0 is redesigned, and it works > much more efficient than in old 1.6 version. > So for example if you loose one disk(full 1TB) it will replicate in less > than 2 hours - 1TB RAID1 rebuild will take much much more time than this. > > Another problems: > - RAID1 is slower than JBOD in MooseFS > - Available space is lower when you are using RAID1 for chunkserver disk > than JBOD. > > About BTRFS - we have tested older version of BTFS an we had kernel > panic during tests. > We didn’t test the latest available release but we will in the nearest > future. > At this moment we recommend XFS as backend FS for chunkserver disks. > > Best regards > Aleksander Wieliczko > Technical Support Engineer > MooseFS.com <http://MooseFS.com> <moosefs.com <http://moosefs.com>> > > On 15 Jul 2015, at 19:17, Ricardo J. Barberis > <ric...@do... <mailto:ric...@do...>> > wrote: > > Maybe you could consider another chunkserver and a goal of 3? > > It's one more chunkserver but only 1.5x disk space compared to 2x with 2 > chunkservers and RAID1 (whether by software or hardware). > > El Miércoles 15/07/2015, Aleksander Wieliczko escribió: > > Hi Michael > > We had bad experience with BTFS so we are not advising to use > this fs as > backend for chunkserver HDD. > We recommend XFS because we achieved the best performance with > disk of > 90% usage. > > We have extra questions: > How big your MooseFS is - how many files you have? > What is your LAN speed? > What is the MooseFS version? > > Best regards > Aleksander Wieliczko > Technical Support Engineer > MooseFS.com <http://moosefs.com/><moosefs.com <http://moosefs.com/>> > > On 15.07.2015 09:24, Michael Tinsay wrote: > > Hi. > > Is anybody using btrfs for moosefs? > > I currently have a 2-chunkserver (using jbod, ext4 fs, and > goal=2 for > all files) setup running that has recently experienced an hd > failure > in one of the chunkservers. While replacing the failed hdd > was an > almost trivial task, there was a long span of time where a > big number > of chunks only had 1 copy. While I did not lose sleep over > it, I had > this concern of potentially losing chunks should there be > another hd > failure in the other chunkserver while moosefs is still > replicating > the 1-copy chunks. > > So now I'm entertaining the idea of moving from > just-a-bunch-of-ext4-disks to one-big-btrfs-raid1-fs as the > underlying > storage provider -- using Ubuntu 14.04 distro plus the 4.1.x LTS > kernel from kernel-ppa. > > I'm thinking that with such a setup I would still be up and > running > with 2 copies for all chunks even if both chunkservers have > 1 disk > failure each at the same time. And the setup would survive > a failed > chunkserver with 1 failed disk in the remaining chunkserver. > > I would like to hear your thoughts on this. > > Regards. > > > --- mike t. > > > ------------------------------------------------------------------------- > ----- Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and > support that > you need to offload your IT needs and focus on growing your > business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > > > -- > Ricardo J. Barberis > Senior SysAdmin / IT Architect > DonWeb > La Actitud Es Todo > www.DonWeb.com <http://www.donweb.com/> > > ------------------------------------------------------------------------------ > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that > you need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > _________________________________________ > moosefs-users mailing list > moo...@li... > <mailto:moo...@li...> > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > > > ------------------------------------------------------------------------------ > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that > you need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
|
From: Aleksander W. <ale...@mo...> - 2015-07-15 21:40:11
|
Hi. In your situation the best idea is to add another chunk server and change GOAL to 3 as it was mentioned by Ricardo J. Barberis. Replication mechanism in MooseFS 2.0 and 3.0 is redesigned, and it works much more efficient than in old 1.6 version. So for example if you loose one disk(full 1TB) it will replicate in less than 2 hours - 1TB RAID1 rebuild will take much much more time than this. Another problems: - RAID1 is slower than JBOD in MooseFS - Available space is lower when you are using RAID1 for chunkserver disk than JBOD. About BTRFS - we have tested older version of BTFS an we had kernel panic during tests. We didn’t test the latest available release but we will in the nearest future. At this moment we recommend XFS as backend FS for chunkserver disks. Best regards Aleksander Wieliczko Technical Support Engineer MooseFS.com <moosefs.com> > On 15 Jul 2015, at 19:17, Ricardo J. Barberis <ric...@do...> wrote: > > Maybe you could consider another chunkserver and a goal of 3? > > It's one more chunkserver but only 1.5x disk space compared to 2x with 2 > chunkservers and RAID1 (whether by software or hardware). > > El Miércoles 15/07/2015, Aleksander Wieliczko escribió: >> Hi Michael >> >> We had bad experience with BTFS so we are not advising to use this fs as >> backend for chunkserver HDD. >> We recommend XFS because we achieved the best performance with disk of >> 90% usage. >> >> We have extra questions: >> How big your MooseFS is - how many files you have? >> What is your LAN speed? >> What is the MooseFS version? >> >> Best regards >> Aleksander Wieliczko >> Technical Support Engineer >> MooseFS.com <http://moosefs.com/> <moosefs.com <http://moosefs.com/>> >> >> On 15.07.2015 09:24, Michael Tinsay wrote: >>> Hi. >>> >>> Is anybody using btrfs for moosefs? >>> >>> I currently have a 2-chunkserver (using jbod, ext4 fs, and goal=2 for >>> all files) setup running that has recently experienced an hd failure >>> in one of the chunkservers. While replacing the failed hdd was an >>> almost trivial task, there was a long span of time where a big number >>> of chunks only had 1 copy. While I did not lose sleep over it, I had >>> this concern of potentially losing chunks should there be another hd >>> failure in the other chunkserver while moosefs is still replicating >>> the 1-copy chunks. >>> >>> So now I'm entertaining the idea of moving from >>> just-a-bunch-of-ext4-disks to one-big-btrfs-raid1-fs as the underlying >>> storage provider -- using Ubuntu 14.04 distro plus the 4.1.x LTS >>> kernel from kernel-ppa. >>> >>> I'm thinking that with such a setup I would still be up and running >>> with 2 copies for all chunks even if both chunkservers have 1 disk >>> failure each at the same time. And the setup would survive a failed >>> chunkserver with 1 failed disk in the remaining chunkserver. >>> >>> I would like to hear your thoughts on this. >>> >>> Regards. >>> >>> >>> --- mike t. >>> >>> >>> ------------------------------------------------------------------------- >>> ----- Don't Limit Your Business. Reach for the Cloud. >>> GigeNET's Cloud Solutions provide you with the tools and support that >>> you need to offload your IT needs and focus on growing your business. >>> Configured For All Businesses. Start Your Cloud Today. >>> https://www.gigenetcloud.com/ >>> >>> >>> _________________________________________ >>> moosefs-users mailing list >>> moo...@li... >>> https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > > -- > Ricardo J. Barberis > Senior SysAdmin / IT Architect > DonWeb > La Actitud Es Todo > www.DonWeb.com <http://www.donweb.com/> > > ------------------------------------------------------------------------------ > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that > you need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
