moosefs-users — Mailing list for MooseFS Users

Re: [Moosefs-users] Solution of small file store

[<ro...@mm...>]

If you're on linux,

i'd try a filesystem-in-a-file which would be used to store the photo and
this file itselfs stored in a MooseFS volume.

For example something along these lines:

dd if=/dev/zero of=FILENAME bs=1024 count=1 seek=$((2*1024*1024-1))
mkfs -t ext3 FILENAME
mount -t ext3 -o loop FILENAME MOUNTPOINT

where MOUNTPOINT is a MooseFS volume.

-Stathis

Wang Jian wrote:
> TFS has strict and random URL scheme, so it's difficult to do URL based
> tuning.
>
> 于 2012/4/24 15:11, Davies Liu 写道:
>> On Tue, Apr 24, 2012 at 2:29 PM, Ken <ken...@gm...
>> <mailto:ken...@gm...>> wrote:
>>
>>     On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm...
>> <mailto:dav...@gm...>> wrote:
>>     > Maybe leveldb + MooseFS is better.
>>     Is this your mean?
>>      leveldb store url
>>      moosefs store photo
>>
>> No, store url and photo in leveldb, which use MooseFS as disks.
>> the performance may be not perfect, but enough.
>> If not, TFS from taobao.com <http://taobao.com> may be the better
>> choice.
>>
>>     >
>>     > On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm...
>> <mailto:ken...@gm...>> wrote:
>>     >>
>>     >> We need to store tons of small files(photo), as noticed in faq,
>> file
>>     >> count is limited in moosefs, I think bundle small files to a huge
>> file
>>     >> maybe work.
>>     >>
>>     >> Write photos procedure like:
>>     >>   allocate a huge file
>>     >>   write head and photo content
>>     >>   return (huge file, position, size)
>>     >>   write another head and another photo
>>     >>   return (huge file, position, size)
>>     >>   ...
>>     >>
>>     >> Before read a photo, we should have enough information: huge
>> file,
>>     >> position and length, the reading procedure is expected normally.
>>     >>
>>     >> To read a photo, we should provide an URL, like
>>     >> 'http://xxx.com/prefix/huge file/offset/size.jpg'
>>     >>
>>     >> And to be useful in WEB, build a fastcgi program for read/write
>> access
>>     >> to the huge file.
>>     >>
>>     >> ps:
>>     >> *  The matchup information photo and url should store outside of
>> mooosefs.
>>     >> *  Huge file size limited in 2G.
>>     >> *  Huge file maybe cause race condition.
>>     >>
>>     >> Is there anyone interested in this?
>>     >> or better solution?
>>     >>
>>     >> -Ken
>>     >>
>>     >>
>>     >> ------------------------------------------------------------------------------
>>     >> Live Security Virtual Conference
>>     >> Exclusive live event will cover all the ways today's security and
>>     >> threat landscape has changed and how IT managers can respond.
>> Discussions
>>     >> will include endpoint security, mobile security and the latest in
>> malware
>>     >> threats.
>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>     >> _______________________________________________
>>     >> moosefs-users mailing list
>>     >> moo...@li...
>> <mailto:moo...@li...>
>>     >> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>     >
>>     >
>>     >
>>     >
>>     > --
>>     >  - Davies
>>
>>
>>
>>
>> --
>>  - Davies
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond.
>> Discussions
>> will include endpoint security, mobile security and the latest in
>> malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>
>>
>> _______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

Re: [Moosefs-users] Solution of small file store

[Wang J. <jia...@re...>]

TFS has strict and random URL scheme, so it's difficult to do URL based tuning.

于 2012/4/24 15:11, Davies Liu 写道:
> On Tue, Apr 24, 2012 at 2:29 PM, Ken <ken...@gm... <mailto:ken...@gm...>> wrote:
>
>     On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm... <mailto:dav...@gm...>> wrote:
>     > Maybe leveldb + MooseFS is better.
>     Is this your mean?
>      leveldb store url
>      moosefs store photo
>
> No, store url and photo in leveldb, which use MooseFS as disks.
> the performance may be not perfect, but enough.
> If not, TFS from taobao.com <http://taobao.com> may be the better choice.
>
>     >
>     > On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm... <mailto:ken...@gm...>> wrote:
>     >>
>     >> We need to store tons of small files(photo), as noticed in faq, file
>     >> count is limited in moosefs, I think bundle small files to a huge file
>     >> maybe work.
>     >>
>     >> Write photos procedure like:
>     >>   allocate a huge file
>     >>   write head and photo content
>     >>   return (huge file, position, size)
>     >>   write another head and another photo
>     >>   return (huge file, position, size)
>     >>   ...
>     >>
>     >> Before read a photo, we should have enough information: huge file,
>     >> position and length, the reading procedure is expected normally.
>     >>
>     >> To read a photo, we should provide an URL, like
>     >> 'http://xxx.com/prefix/huge file/offset/size.jpg'
>     >>
>     >> And to be useful in WEB, build a fastcgi program for read/write access
>     >> to the huge file.
>     >>
>     >> ps:
>     >> *  The matchup information photo and url should store outside of mooosefs.
>     >> *  Huge file size limited in 2G.
>     >> *  Huge file maybe cause race condition.
>     >>
>     >> Is there anyone interested in this?
>     >> or better solution?
>     >>
>     >> -Ken
>     >>
>     >>
>     >> ------------------------------------------------------------------------------
>     >> Live Security Virtual Conference
>     >> Exclusive live event will cover all the ways today's security and
>     >> threat landscape has changed and how IT managers can respond. Discussions
>     >> will include endpoint security, mobile security and the latest in malware
>     >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>     >> _______________________________________________
>     >> moosefs-users mailing list
>     >> moo...@li... <mailto:moo...@li...>
>     >> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>     >
>     >
>     >
>     >
>     > --
>     >  - Davies
>
>
>
>
> -- 
>  - Davies
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Solution of small file store

[Davies L. <dav...@gm...>]

On Tue, Apr 24, 2012 at 2:29 PM, Ken <ken...@gm...> wrote:

> On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm...> wrote:
> > Maybe leveldb + MooseFS is better.
> Is this your mean?
>  leveldb store url
>  moosefs store photo


No, store url and photo in leveldb, which use MooseFS as disks.
the performance may be not perfect, but enough.
If not, TFS from taobao.com may be the better choice.

>
> > On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm...> wrote:
> >>
> >> We need to store tons of small files(photo), as noticed in faq, file
> >> count is limited in moosefs, I think bundle small files to a huge file
> >> maybe work.
> >>
> >> Write photos procedure like:
> >>   allocate a huge file
> >>   write head and photo content
> >>   return (huge file, position, size)
> >>   write another head and another photo
> >>   return (huge file, position, size)
> >>   ...
> >>
> >> Before read a photo, we should have enough information: huge file,
> >> position and length, the reading procedure is expected normally.
> >>
> >> To read a photo, we should provide an URL, like
> >> 'http://xxx.com/prefix/huge file/offset/size.jpg'
> >>
> >> And to be useful in WEB, build a fastcgi program for read/write access
> >> to the huge file.
> >>
> >> ps:
> >> *  The matchup information photo and url should store outside of
> mooosefs.
> >> *  Huge file size limited in 2G.
> >> *  Huge file maybe cause race condition.
> >>
> >> Is there anyone interested in this?
> >> or better solution?
> >>
> >> -Ken
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Live Security Virtual Conference
> >> Exclusive live event will cover all the ways today's security and
> >> threat landscape has changed and how IT managers can respond.
> Discussions
> >> will include endpoint security, mobile security and the latest in
> malware
> >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> >> _______________________________________________
> >> moosefs-users mailing list
> >> moo...@li...
> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> >
> >
> >
> >
> > --
> >  - Davies
>



-- 
 - Davies

Re: [Moosefs-users] Solution of small file store

[Ken <ken...@gm...>]

On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm...> wrote:
> Maybe leveldb + MooseFS is better.
Is this your mean?
  leveldb store url
  moosefs store photo


>
> On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm...> wrote:
>>
>> We need to store tons of small files(photo), as noticed in faq, file
>> count is limited in moosefs, I think bundle small files to a huge file
>> maybe work.
>>
>> Write photos procedure like:
>>   allocate a huge file
>>   write head and photo content
>>   return (huge file, position, size)
>>   write another head and another photo
>>   return (huge file, position, size)
>>   ...
>>
>> Before read a photo, we should have enough information: huge file,
>> position and length, the reading procedure is expected normally.
>>
>> To read a photo, we should provide an URL, like
>> 'http://xxx.com/prefix/huge file/offset/size.jpg'
>>
>> And to be useful in WEB, build a fastcgi program for read/write access
>> to the huge file.
>>
>> ps:
>> *  The matchup information photo and url should store outside of mooosefs.
>> *  Huge file size limited in 2G.
>> *  Huge file maybe cause race condition.
>>
>> Is there anyone interested in this?
>> or better solution?
>>
>> -Ken
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
>
>
> --
>  - Davies

Re: [Moosefs-users] Solution of small file store

[Davies L. <dav...@gm...>]

Maybe leveldb + MooseFS is better.

On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm...> wrote:

> We need to store tons of small files(photo), as noticed in faq, file
> count is limited in moosefs, I think bundle small files to a huge file
> maybe work.
>
> Write photos procedure like:
>   allocate a huge file
>   write head and photo content
>   return (huge file, position, size)
>   write another head and another photo
>   return (huge file, position, size)
>   ...
>
> Before read a photo, we should have enough information: huge file,
> position and length, the reading procedure is expected normally.
>
> To read a photo, we should provide an URL, like
> 'http://xxx.com/prefix/huge file/offset/size.jpg'
>
> And to be useful in WEB, build a fastcgi program for read/write access
> to the huge file.
>
> ps:
> *  The matchup information photo and url should store outside of mooosefs.
> *  Huge file size limited in 2G.
> *  Huge file maybe cause race condition.
>
> Is there anyone interested in this?
> or better solution?
>
> -Ken
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>



-- 
 - Davies

[Moosefs-users] Solution of small file store

[Ken <ken...@gm...>]

We need to store tons of small files(photo), as noticed in faq, file
count is limited in moosefs, I think bundle small files to a huge file
maybe work.

Write photos procedure like:
  allocate a huge file
  write head and photo content
  return (huge file, position, size)
  write another head and another photo
  return (huge file, position, size)
  ...

Before read a photo, we should have enough information: huge file,
position and length, the reading procedure is expected normally.

To read a photo, we should provide an URL, like
'http://xxx.com/prefix/huge file/offset/size.jpg'

And to be useful in WEB, build a fastcgi program for read/write access
to the huge file.

ps:
*  The matchup information photo and url should store outside of mooosefs.
*  Huge file size limited in 2G.
*  Huge file maybe cause race condition.

Is there anyone interested in this?
or better solution?

-Ken

Re: [Moosefs-users] Hardware choice

[Allen, B. S <bs...@la...>]

I agree with this approach as well for most general purpose deployments.

Unless you're trying to optimize for single stream performance. In which case I'd suggest going back to large storage node approach, with RAID sets instead of showing MFS each spindle. For example I'm using a handful of SuperMicro's 36 drive chassis, 10GbE, with Illumos based OS, and ZFS RAIDZ2 sets. Ending up with 43TB / node. I present MFS with a single HD that is comprised of 4x 8 Drive RAIDZ2 per node. It's going to be extremely painful when I need to migrate data off a node, but I get good single stream performance. 

I could likely better balance single stream performance with time to migrate by going to 16 drive chassis or similar, however overall cost / TB would increase as you're now buying more CPUs, etc. Although you could drop down to a single socket, less RAM per node, and so on to displace this extra cost.

Like everything it's a trade off.

Ben

 
On Apr 21, 2012, at 5:09 AM, Steve Thompson wrote:

> On Fri, 20 Apr 2012, Atom Powers wrote:
> 
>> Because Moose is so good at dealing with system failure but slow to 
>> re-balance chunks I would recommend several "smaller" capacity servers 
>> over a few very large ones. Even at 10TB per server it takes a very long 
>> time to re-balance when I add or remove a system from the cluster; I 
>> would avoid going over about 10TB per server. Less is more in this case.
> 
> I have to agree with this. I have two chunkservers with 25TB storage, as 
> well as several smaller chunkservers, and I recently removed 20TB of disk 
> from one of them (about 70% full). It took a little over 10 days to 
> replicate the removed chunks.
> 
> Steve
> 
> ------------------------------------------------------------------------------
> For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You.
> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] IPv6 support

[Scott D. <sd...@cl...>]

On Sat, Apr 21, 2012 at 6:17 AM, Fabien Germain <fab...@gm...>wrote:

> Hi Scott, hi list !
>
> 2012/4/20 Scott Duckworth <sd...@cl...>:
> > IPv6 is coming whether we want it or not.  Here's some evidence:
>
> I agree with your arguments, and that IPv6 is a need in MooseFS. But
> just one question :
>
> > Many parts of the world have exhausted their IPv4 address blocks, meaning
> > you can only get new IPv4 addresses if they are already allocated to you
> or
> > from IPv4 scavengers [
> http://en.wikipedia.org/wiki/IPv4_address_exhaustion]
>
> Why would someone use *public* IPv4 addresses on chunkservers ??? In
> our case, we use private IP on a dedicated VLAN for all the cluster,
> and several clients have a public IP (to "export" the data), and a
> private IP in the MooseFS VLAN.
>

If somebody wanted to use MooseFS over a WAN, I'm assuming that the
chunkservers and master server would need to be reachable from each other
and from the clients.  This could either be accomplished with a VPN using
private IP addresses or directly over the Internet using public IP
addresses (hopefully encrypted with IPsec).  This would be the case with
either IPv4 or IPv6, only that IPv6 doesn't exactly have private address
ranges.

Regardless of where MooseFS is being used, LAN or WAN, IPv6 will eventually
become a consideration.  Even though it is WAN that is driving IPv6, I
expect it will eventually overcome IPv4 even for internal network traffic.
If network administrators are going to have to support IPv6 anyways, why
also support IPv4 once everything is reachable via IPv6?  At that point
IPv4 is just that old rusty protocol that we used to have to use which is
now getting in the way.  Anybody remember IPX/SPX? [
http://en.wikipedia.org/wiki/IPX/SPX]

Re: [Moosefs-users] Hardware choice

[Steve T. <sm...@cb...>]

On Fri, 20 Apr 2012, Atom Powers wrote:

> Because Moose is so good at dealing with system failure but slow to 
> re-balance chunks I would recommend several "smaller" capacity servers 
> over a few very large ones. Even at 10TB per server it takes a very long 
> time to re-balance when I add or remove a system from the cluster; I 
> would avoid going over about 10TB per server. Less is more in this case.

I have to agree with this. I have two chunkservers with 25TB storage, as 
well as several smaller chunkservers, and I recently removed 20TB of disk 
from one of them (about 70% full). It took a little over 10 days to 
replicate the removed chunks.

Steve

Re: [Moosefs-users] Hardware choice

[Quenten G. <QG...@on...>]

Very true, ECC memory has dropped in price considerably over the years.

Also I think the embedded CPUs really still aren't up to scratch in the way of performance which is what I forgot to mention before however I think building a system in a truly brick by brick (modular) maybe another way to look at a solution.

Regards,
Quenten Grasso

-----Original Message-----
From: Martins Lazdans [mailto:mar...@vp...] 
Sent: Saturday, 21 April 2012 6:55 PM
To: moo...@li...
Subject: Re: [Moosefs-users] Hardware choice

Nowadays ECC registered memory is cheaper than non-ECC desktop. You can
get 2GB ECC REG modules for about $5-$7 ea on eBay depending on amount
purchased.

On 2012.04.21. 7:18, Quenten Grasso wrote:
> Hi All,
> 
> I've been thinking about this myself quit a bit in the last few months.
> 
> MooseFS is very simular to Apache HDFS, which uses 128mb chunks instead of 64mb and a single metadata server with metaloggers etc.
> 
> I mention this is because I've been investigating how the likes of Google, Yahoo etc have been setting up storage and compute clusters.
> 
> What I've found is very interesting, For example yahoo use 4 x hard disks and 2 x Quad Core CPU's and 2 x 1GbE per node. 
> They have up to 3500 nodes per cluster. Which I think is very interesting way of truly distributing there workload.
> 
> Why 2x Quad CPU? Well they also use MapReduce (which is basically a distributed compute platform, think of "seti" or "folding at home" projects) 
> 
> So what I've basically found is certainly is "less is more", and as MFS/HDFS is always "limited" to the write speed of a single disk per process which may sound slow to some, however at scale is pretty impressive distributed platform if you think about it, so you're limited to around 50-60mb/s write per disk and reads should be the speed of your replica levels (give or take a bit).
> 
> So I've set on an idea of well why not commoditise the storage nodes further and basically build them "cheap as possible" without sacrificing to much in the way of reliability e.g.: still use ECC memory or maybe we can build enough safe guards in MFS not even "require" ECC Memory in the storage nodes???
> 
> I think separating storage from compute has some significant benefits as well as combining the two so this one is always left up to the individual. But for the sake of what I'm trying to do here is separate the storage from compute in this example.
> 
> Using the new Rack/Zone method you could build cheaper storage nodes with single power supply's and by using 2 x 1GBE instead of 10GBE or Infiniband you can save yourself some money without sacrificing reliability or performance so my idea was to use yahoo's example, and build 30 nodes with single Power Supply's around 4 or 8GB of RAM and 4 Hard Disks Per Node.
> 
> For example if you have 20 nodes and 3 x replication and A & B power in your site you would only need to put 10 in Zone 1 and 10 in Zone 2 and set replica level of 3 and you'll always have access to your data. As long as your metadata servers have dual power supply's & ECC memory you should be perfect.
> 
> Using this method we maybe able to use something like a low power Mini ITX with ECC memory and Integrated CPU ideally with a built in Software KVM/Monitoring Access simular to the Supermicro's motherboards.
> 
> So what do you all think of this?? I always welcome any input =)
> 
> Regards,
> Quenten Grasso
> 
> 
> -----Original Message-----
> From: Atom Powers [mailto:ap...@di...] 
> Sent: Saturday, 21 April 2012 1:58 AM
> To: moo...@li...
> Subject: Re: [Moosefs-users] Hardware choice
> 
> On 04/20/2012 04:09 AM, Chris Picton wrote:
>> I was looking at supermicro chassis and found the following chassis
>> types which seem to offer highest density:
>>
>> 2u: 12x 3.5" http://www.supermicro.com/products/chassis/2U/?chs=827
>>
>> Does anyone have feedback on supermicro/these chassis?
> 
> I use a lot of SuperMicro kit here. It performs well, is very reliable,
> and at the right price. (I buy from http://www.siliconmechanics.com/)
> 
> I have three of the above chassis and a couple older 8-bay systems in my 
> cluster. Because Moose is so good at dealing with system failure but 
> slow to re-balance chunks I would recommend several "smaller" capacity 
> servers over a few very large ones. Even at 10TB per server it takes a 
> very long time to re-balance when I add or remove a system from the 
> cluster; I would avoid going over about 10TB per server. Less is more in 
> this case.
> 

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] IPv6 support

[Fabien G. <fab...@gm...>]

Hi Scott, hi list !

2012/4/20 Scott Duckworth <sd...@cl...>:
> IPv6 is coming whether we want it or not.  Here's some evidence:

I agree with your arguments, and that IPv6 is a need in MooseFS. But
just one question :

> Many parts of the world have exhausted their IPv4 address blocks, meaning
> you can only get new IPv4 addresses if they are already allocated to you or
> from IPv4 scavengers [http://en.wikipedia.org/wiki/IPv4_address_exhaustion]

Why would someone use *public* IPv4 addresses on chunkservers ??? In
our case, we use private IP on a dedicated VLAN for all the cluster,
and several clients have a public IP (to "export" the data), and a
private IP in the MooseFS VLAN.

Fabien

Re: [Moosefs-users] Hardware choice

[Martins L. <mar...@vp...>]

Nowadays ECC registered memory is cheaper than non-ECC desktop. You can
get 2GB ECC REG modules for about $5-$7 ea on eBay depending on amount
purchased.

On 2012.04.21. 7:18, Quenten Grasso wrote:
> Hi All,
> 
> I've been thinking about this myself quit a bit in the last few months.
> 
> MooseFS is very simular to Apache HDFS, which uses 128mb chunks instead of 64mb and a single metadata server with metaloggers etc.
> 
> I mention this is because I've been investigating how the likes of Google, Yahoo etc have been setting up storage and compute clusters.
> 
> What I've found is very interesting, For example yahoo use 4 x hard disks and 2 x Quad Core CPU's and 2 x 1GbE per node. 
> They have up to 3500 nodes per cluster. Which I think is very interesting way of truly distributing there workload.
> 
> Why 2x Quad CPU? Well they also use MapReduce (which is basically a distributed compute platform, think of "seti" or "folding at home" projects) 
> 
> So what I've basically found is certainly is "less is more", and as MFS/HDFS is always "limited" to the write speed of a single disk per process which may sound slow to some, however at scale is pretty impressive distributed platform if you think about it, so you're limited to around 50-60mb/s write per disk and reads should be the speed of your replica levels (give or take a bit).
> 
> So I've set on an idea of well why not commoditise the storage nodes further and basically build them "cheap as possible" without sacrificing to much in the way of reliability e.g.: still use ECC memory or maybe we can build enough safe guards in MFS not even "require" ECC Memory in the storage nodes???
> 
> I think separating storage from compute has some significant benefits as well as combining the two so this one is always left up to the individual. But for the sake of what I'm trying to do here is separate the storage from compute in this example.
> 
> Using the new Rack/Zone method you could build cheaper storage nodes with single power supply's and by using 2 x 1GBE instead of 10GBE or Infiniband you can save yourself some money without sacrificing reliability or performance so my idea was to use yahoo's example, and build 30 nodes with single Power Supply's around 4 or 8GB of RAM and 4 Hard Disks Per Node.
> 
> For example if you have 20 nodes and 3 x replication and A & B power in your site you would only need to put 10 in Zone 1 and 10 in Zone 2 and set replica level of 3 and you'll always have access to your data. As long as your metadata servers have dual power supply's & ECC memory you should be perfect.
> 
> Using this method we maybe able to use something like a low power Mini ITX with ECC memory and Integrated CPU ideally with a built in Software KVM/Monitoring Access simular to the Supermicro's motherboards.
> 
> So what do you all think of this?? I always welcome any input =)
> 
> Regards,
> Quenten Grasso
> 
> 
> -----Original Message-----
> From: Atom Powers [mailto:ap...@di...] 
> Sent: Saturday, 21 April 2012 1:58 AM
> To: moo...@li...
> Subject: Re: [Moosefs-users] Hardware choice
> 
> On 04/20/2012 04:09 AM, Chris Picton wrote:
>> I was looking at supermicro chassis and found the following chassis
>> types which seem to offer highest density:
>>
>> 2u: 12x 3.5" http://www.supermicro.com/products/chassis/2U/?chs=827
>>
>> Does anyone have feedback on supermicro/these chassis?
> 
> I use a lot of SuperMicro kit here. It performs well, is very reliable,
> and at the right price. (I buy from http://www.siliconmechanics.com/)
> 
> I have three of the above chassis and a couple older 8-bay systems in my 
> cluster. Because Moose is so good at dealing with system failure but 
> slow to re-balance chunks I would recommend several "smaller" capacity 
> servers over a few very large ones. Even at 10TB per server it takes a 
> very long time to re-balance when I add or remove a system from the 
> cluster; I would avoid going over about 10TB per server. Less is more in 
> this case.
> 

Re: [Moosefs-users] Hardware choice

[Quenten G. <QG...@on...>]

Hi All,

I've been thinking about this myself quit a bit in the last few months.

MooseFS is very simular to Apache HDFS, which uses 128mb chunks instead of 64mb and a single metadata server with metaloggers etc.

I mention this is because I've been investigating how the likes of Google, Yahoo etc have been setting up storage and compute clusters.

What I've found is very interesting, For example yahoo use 4 x hard disks and 2 x Quad Core CPU's and 2 x 1GbE per node. 
They have up to 3500 nodes per cluster. Which I think is very interesting way of truly distributing there workload.

Why 2x Quad CPU? Well they also use MapReduce (which is basically a distributed compute platform, think of "seti" or "folding at home" projects) 

So what I've basically found is certainly is "less is more", and as MFS/HDFS is always "limited" to the write speed of a single disk per process which may sound slow to some, however at scale is pretty impressive distributed platform if you think about it, so you're limited to around 50-60mb/s write per disk and reads should be the speed of your replica levels (give or take a bit).

So I've set on an idea of well why not commoditise the storage nodes further and basically build them "cheap as possible" without sacrificing to much in the way of reliability e.g.: still use ECC memory or maybe we can build enough safe guards in MFS not even "require" ECC Memory in the storage nodes???

I think separating storage from compute has some significant benefits as well as combining the two so this one is always left up to the individual. But for the sake of what I'm trying to do here is separate the storage from compute in this example.

Using the new Rack/Zone method you could build cheaper storage nodes with single power supply's and by using 2 x 1GBE instead of 10GBE or Infiniband you can save yourself some money without sacrificing reliability or performance so my idea was to use yahoo's example, and build 30 nodes with single Power Supply's around 4 or 8GB of RAM and 4 Hard Disks Per Node.

For example if you have 20 nodes and 3 x replication and A & B power in your site you would only need to put 10 in Zone 1 and 10 in Zone 2 and set replica level of 3 and you'll always have access to your data. As long as your metadata servers have dual power supply's & ECC memory you should be perfect.

Using this method we maybe able to use something like a low power Mini ITX with ECC memory and Integrated CPU ideally with a built in Software KVM/Monitoring Access simular to the Supermicro's motherboards.

So what do you all think of this?? I always welcome any input =)

Regards,
Quenten Grasso


-----Original Message-----
From: Atom Powers [mailto:ap...@di...] 
Sent: Saturday, 21 April 2012 1:58 AM
To: moo...@li...
Subject: Re: [Moosefs-users] Hardware choice

On 04/20/2012 04:09 AM, Chris Picton wrote:
> I was looking at supermicro chassis and found the following chassis
> types which seem to offer highest density:
>
> 2u: 12x 3.5" http://www.supermicro.com/products/chassis/2U/?chs=827
>
> Does anyone have feedback on supermicro/these chassis?

I use a lot of SuperMicro kit here. It performs well, is very reliable,
and at the right price. (I buy from http://www.siliconmechanics.com/)

I have three of the above chassis and a couple older 8-bay systems in my 
cluster. Because Moose is so good at dealing with system failure but 
slow to re-balance chunks I would recommend several "smaller" capacity 
servers over a few very large ones. Even at 10TB per server it takes a 
very long time to re-balance when I add or remove a system from the 
cluster; I would avoid going over about 10TB per server. Less is more in 
this case.

-- 
--
Perfection is just a word I use occasionally with mustard.
--Atom Powers--
Director of IT
DigiPen Institute of Technology
+1 (425) 895-4443

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Backup strategies

[Michał B. <mic...@co...>]

Hi Group!

As already mentioned we'd just like to stress how important it is to make
regular metadata backups. The future 1.6.26 version will keep several
metadata files back on the disk (as already done by Ben) and still it would
be wise to copy them somewhere else.

The full rack awareness which you'd expect would probably be introduced in a
way suggested by Ken with levelgoals. But still remember that running such a
MooseFS installation would require a really quick connection between the two
sites which may be difficult to accomplish. Synchronizing two MooseFS
installations might be really better done with simple rsync.

PS. @Ken, mind that your solution (at least when we looked at it) didn't
have support for levelgoals in mfsmetarestore (did you run a recovery test?)
and in mfsmetadump.


Kind regards
Michał Borychowski 
MooseFS Support Manager


-----Original Message-----
From: Steve Wilson [mailto:st...@pu...] 
Sent: Wednesday, April 04, 2012 7:30 PM
To: moo...@li...
Subject: Re: [Moosefs-users] Backup strategies

On 04/03/2012 03:56 PM, Steve Thompson wrote:
> OK, so now you have a nice and shiny and absolutely massive MooseFS 
> file system. How do you back it up?
>
> I am using Bacula and divide the MFS file system into separate areas 
> (eg directories beginning with a, those beginning with b, and so on) 
> and use several different chunkservers to run the backup jobs, on the 
> theory that at least some of the data is local to the backup process. 
> But this still leaves the vast majority of data to travel the network 
> twice (a planned dedicated storage network has not yet been 
> implemented). This results in pretty bad backup performance and high
network load. Any clever ideas?
>
> Steve

We have four 22TB and one 14TB MooseFS volumes that we backup onto
disk-based backup servers.  We used to use rsnapshot but now we use rsync in
combination with ZFS snapshots.  Each evening before our backup run, we take
a snapshot on the backup filesystem and label it with the date.  Then we run
rsync on the volumes being backed up and only what has been modified since
the previous backup is transfered over the network.  The result is the
equivalent of taking a full backup each night and it's very easy to recover
data.  I also use ZFS compression and dedup to help conserve space on our
backup servers.  The dedup option is especially helpful when a user decides
to rename a large directory; rsync may have to bring it across the network
and write it to the filesystem but ZFS will recognize the data as duplicates
of already stored data.

Steve


----------------------------------------------------------------------------
--
Better than sec? Nothing is better than sec when it comes to monitoring Big
Data applications. Try Boundary one-second resolution app monitoring today.
Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] IPv6 support

[Scott D. <sd...@cl...>]

IPv6 is coming whether we want it or not.  Here's some evidence:

   - Many parts of the world have exhausted their IPv4 address blocks,
   meaning you can only get new IPv4 addresses if they are already allocated
   to you or from IPv4 scavengers [
   http://en.wikipedia.org/wiki/IPv4_address_exhaustion]
   - The World IPv6 Launch Day is June 6, 2012: on this day many major
   ISPs, networking equipment manufacturers, and web companies around the
   world are permanently enabling IPv6 [http://www.worldipv6launch.org/]
   - It's already in wide use in China [
   http://en.wikipedia.org/wiki/China_Next_Generation_Internet]
   - All US government agencies will be required to use IPv6 for all public
   facing services by September 30, 2012 and all internal services in 2014 [
   http://www.networkworld.com/news/2010/092810-white-house-ipv6-directive.html
   ]
   - At least one public university will soon be requiring that all new
   application deployments support IPv6 to the same level that they support
   IPv4 [http://www.clemson.edu/ccit/get_connected/ipv6.html]


Dual-stack IPv4 & IPv6 environments are meant to be a transitional stage.
 Existing services on IPv4 have no reason to drop their IPv4 addresses but
it will become harder and harder to acquire IPv4 addresses for new
services.  With major corporations and governments already pushing hard on
IPv6, it will quickly pick up momentum.

Services that do not support IPv6 will be superseded by those that do.  Why
not take the chance to implement IPv6 support while it's still mostly in
the transitional phase rather than waiting until it's in the required stage?

Scott Duckworth, Systems Programmer II
Clemson University School of Computing
AIM/Skype: ClemsonUnixDuck



2012/4/20 Michał Borychowski <mic...@co...>

> You can have a look here at our list archive:****
>
>
> http://sourceforge.net/mailarchive/forum.php?thread_name=4DDB66C6.8060906%40coppint.com&forum_name=moosefs-users
> ****
>
> ** **
>
> For the moment we do not plan to introduce it - somebody needs to convince
> us MooseFS really needs this :)****
>
> ** **
>
> ** **
>
> Kind regards****
>
> Michał Borychowski ****
>
> MooseFS Support Manager****
>
> ** **
>
> *From:* Scott Duckworth [mailto:sd...@cl...]
> *Sent:* Friday, April 20, 2012 7:44 PM
> *To:* moo...@li...
> *Subject:* [Moosefs-users] IPv6 support****
>
> ** **
>
> Currently, neither mfschunkserver nor mfsmount work when using IPv6
> addresses or host names which resolve only to IPv6 addresses.  Are there
> any plans to implement IPv6 support in MooseFS?****
>
> ** **
>
> Our university will soon be requiring new software purchases to support
> IPv6, and while MooseFS is free and likely won't be restricted by this
> rule, IPv6 support is still probably a good thing to have.****
>
>
> Scott Duckworth, Systems Programmer II
> Clemson University School of Computing
> AIM/Skype: ClemsonUnixDuck****
>

Re: [Moosefs-users] write to chunkservers timing out - too much load?

[Michał B. <mic...@co...>]

Do you have this problem with only one chunk server or with each of them? If
with one you should check its performance, charts, hard disk operation
times, etc. in the CGI monitor.

You may experiment with CHUNKS_WRITE_REP_LIMIT and CHUNKS_READ_REP_LIMIT
options.


Kind regards
Michał Borychowski 
MooseFS Support Manager

-----Original Message-----
From: Elliot Finley [mailto:efi...@gm...] 
Sent: Thursday, April 19, 2012 12:57 AM
To: moo...@li...
Subject: [Moosefs-users] write to chunkservers timing out - too much load?

I get a steady stream of:

'file: NNN, index: NNN, chunk: NNN, version: NNN - writeworker:
connection with (XXXXXXXX:PPPP) was timed out (unfinished writes: Y; try
counter: Z)'

Where Y is usually 1 or 2 and Z is almost always 1.  From the FAQ, I know
this is informational and not an error.

My question is:  If I'm getting a steady stream of these, is it because I've
got too much load on my chunkservers (i.e. too many network connections, too
many pending writes?)  Would adding more chunkservers alleviate this
situation?

MFS seems to only use 10 write threads on a chunkserver.  Would it make
sense to increase the number of write threads to equal the number of disks
on the chunkserver?  Most of my chunkservers have 16 disks.
One of them has 36 disks.

TIA,
Elliot

----------------------------------------------------------------------------
--
Better than sec? Nothing is better than sec when it comes to monitoring Big
Data applications. Try Boundary one-second resolution app monitoring today.
Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] chunkserver crash/shutdown

[Michał B. <mic...@co...>]

Hi!

 

This behaviour was already talked about on our group:
http://sourceforge.net/mailarchive/forum.php?thread_name=AANLkTi%3DZOuMkbhhE
dN0F672Xev2gaJ0WPdZe-t%3DScSbX%40mail.gmail.com
<http://sourceforge.net/mailarchive/forum.php?thread_name=AANLkTi%3DZOuMkbhh
EdN0F672Xev2gaJ0WPdZe-t%3DScSbX%40mail.gmail.com&forum_name=moosefs-users>
&forum_name=moosefs-users

 

So these errors were caused by a "disconnected" hdd. If you looked in the
cgi monitor you would see a disk with "damaged" status (it happens when
MooseFS encounters 3 errors in one mintue; it doesn't mean that the hdd is
really broken, but for the sake of the whole system, such disk gets
disconnected by the system). 

 

On the other hand we strongly recommend upgrading MooseFS to the latest
version. There are lots of new things and bugfixes. You can wait for 1.6.25
which will be released next week.

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

 

From: Hanser [mailto:se...@gm...] 
Sent: Friday, April 20, 2012 10:03 AM
To: 'moosefs-users'
Subject: [Moosefs-users] chunkserver crash/shutdown

 

Today, one of our chunkserver,  running 1.6.20-2 crashed giving out the
following errors. Going thru the docs and couldn't find anything relevant to
it.

I also noticed 3-errors-in-60 seconds ruleset, is there anyway to increase
the threshold in order to avoid further disruptions ? 

Thanks,

 

 

Apr 20 08:18:08 cluster-4 mfschunkserver[7892]: chunk_readcrc:
file:/mnt/mncloud/75/chunk_00000000005F0475_00000001.mfs - wrong id/version
in header (00000000005F0704_00000001)

Apr 20 08:18:08 cluster-4 mfschunkserver[7892]: hdd_io_begin:
file:/mnt/mncloud/75/chunk_00000000005F0475_00000001.mfs - read error:
Unknown error

Apr 20 08:18:10 cluster-4 mfschunkserver[7892]: testing chunk:
/mnt/mncloud/06/chunk_00000000005F0706_00000001.mfs

Apr 20 08:18:12 cluster-4 mfschunkserver[7892]: testing chunk:
/mnt/mncloud/77/chunk_00000000005F0477_00000001.mfs

Apr 20 08:18:12 cluster-4 mfschunkserver[7892]: chunk_readcrc:
file:/mnt/mncloud/77/chunk_00000000005F0477_00000001.mfs - wrong id/version
in header (00000000005F0706_00000001)

Apr 20 08:18:12 cluster-4 mfschunkserver[7892]: hdd_io_begin:
file:/mnt/mncloud/77/chunk_00000000005F0477_00000001.mfs - read error:
Unknown error

Apr 20 08:18:14 cluster-4 mfschunkserver[7892]: testing chunk:
/mnt/mncloud/08/chunk_00000000005F0708_00000001.mfs

Apr 20 08:18:16 cluster-4 mfschunkserver[7892]: testing chunk:
/mnt/mncloud/79/chunk_00000000005F0479_00000001.mfs

Apr 20 08:18:16 cluster-4 mfschunkserver[7892]: chunk_readcrc:
file:/mnt/mncloud/79/chunk_00000000005F0479_00000001.mfs - wrong id/version
in header (00000000005F0708_00000001)

Apr 20 08:18:16 cluster-4 mfschunkserver[7892]: hdd_io_begin:
file:/mnt/mncloud/79/chunk_00000000005F0479_00000001.mfs - read error:
Unknown error

Apr 20 08:18:17 cluster-4 mfschunkserver[7892]: 3 errors occurred in 60
seconds on folder: /mnt/mncloud/

Re: [Moosefs-users] IPv6 support

[Michał B. <mic...@co...>]

You can have a look here at our list archive:

http://sourceforge.net/mailarchive/forum.php?thread_name=4DDB66C6.8060906%40
coppint.com
<http://sourceforge.net/mailarchive/forum.php?thread_name=4DDB66C6.8060906%4
0coppint.com&forum_name=moosefs-users> &forum_name=moosefs-users

 

For the moment we do not plan to introduce it - somebody needs to convince
us MooseFS really needs this :)

 

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

 

From: Scott Duckworth [mailto:sd...@cl...] 
Sent: Friday, April 20, 2012 7:44 PM
To: moo...@li...
Subject: [Moosefs-users] IPv6 support

 

Currently, neither mfschunkserver nor mfsmount work when using IPv6
addresses or host names which resolve only to IPv6 addresses.  Are there any
plans to implement IPv6 support in MooseFS?

 

Our university will soon be requiring new software purchases to support
IPv6, and while MooseFS is free and likely won't be restricted by this rule,
IPv6 support is still probably a good thing to have.


Scott Duckworth, Systems Programmer II
Clemson University School of Computing
AIM/Skype: ClemsonUnixDuck

[Moosefs-users] IPv6 support

[Scott D. <sd...@cl...>]

Currently, neither mfschunkserver nor mfsmount work when using IPv6
addresses or host names which resolve only to IPv6 addresses.  Are there
any plans to implement IPv6 support in MooseFS?

Our university will soon be requiring new software purchases to support
IPv6, and while MooseFS is free and likely won't be restricted by this
rule, IPv6 support is still probably a good thing to have.

Scott Duckworth, Systems Programmer II
Clemson University School of Computing
AIM/Skype: ClemsonUnixDuck

Re: [Moosefs-users] Hardware choice

[Atom P. <ap...@di...>]

On 04/20/2012 04:09 AM, Chris Picton wrote:
> I was looking at supermicro chassis and found the following chassis
> types which seem to offer highest density:
>
> 2u: 12x 3.5" http://www.supermicro.com/products/chassis/2U/?chs=827
>
> Does anyone have feedback on supermicro/these chassis?

I use a lot of SuperMicro kit here. It performs well, is very reliable,
and at the right price. (I buy from http://www.siliconmechanics.com/)

I have three of the above chassis and a couple older 8-bay systems in my 
cluster. Because Moose is so good at dealing with system failure but 
slow to re-balance chunks I would recommend several "smaller" capacity 
servers over a few very large ones. Even at 10TB per server it takes a 
very long time to re-balance when I add or remove a system from the 
cluster; I would avoid going over about 10TB per server. Less is more in 
this case.

-- 
--
Perfection is just a word I use occasionally with mustard.
--Atom Powers--
Director of IT
DigiPen Institute of Technology
+1 (425) 895-4443

Re: [Moosefs-users] Hardware choice

[Chris P. <ch...@na...>]

I am in the same situation as OP - I have a cluster of 6 machines (5 hdd
each) running in test, and am thinking about what I would need in
production.

I was looking at supermicro chassis and found the following chassis
types which seem to offer highest density:

1u: 10x 2.5"
http://www.supermicro.com/products/chassis/1U/?chs=116 

2u: 24x 2.5"
http://www.supermicro.com/products/chassis/2U/?chs=216

2u: 12x 3.5"
http://www.supermicro.com/products/chassis/2U/?chs=827

4u: 88x 2.5"
http://www.supermicro.com/products/chassis/4U/417/SC417E26-R1400U.cfm

Does anyone have feedback on supermicro/these chassis?

Chris


On Fri, 2012-04-20 at 18:45 +0800, Wang Jian wrote:
> 于 2012/4/19 22:23, Schmurfy 写道:
> > Hi,
> >
> > I did some tests with moosefs on vms and "standard" machines (and I really love his project !) but now I need to decide on some rackable hardware to install in a datacenter and that's where things 
> > become
> > annoying since I have no idea what to choose :(
> >
> > I was thinking of starting with two nodes one of them being the master as well as a chunk server and the other will be a backup master and a chunk server too so I suppose both
> > servers will need a decent amount of memory and a not too slow cpu but they don't require some high end muti-core processor and of course they will need some disks.
> >
> > I think we will start with less than 10Tb on the cluster and replication goals set to 2.
> >
> > We are currently using some ProLiant DL160 G6 which has 4 cores running at 2.0GHz but judging by the theoretical needs of our moosefs machine I think these servers
> > are way too powerful for this usage.
> >
> >
> > Does anyone can give me some advices on what would be a good start ?
> >
> >
> 
> Dell C5220 in C5000 chassis?
> 
> Downside is you can only use 2.5' hdd/ssd on this setup. But with 4 hdd/ssd slot per node by 8 nodes, it can be 32TB in capacity already at good performance to price ratio, and you can save a lot in 
> hosting.
> 
> ------------------------------------------------------------------------------
> For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You.
> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

-- 
CHRIS PICTON
Executive Manager: Systems



Cell: +27 (0)79 721 8521  Email: ch...@na...
Tel: +27 (0)10 590 0031   Fax: +27 (0)87 941 0813  
Rosebank Terrace, 23 Sturdee Avenue, Rosebank
www.nashua-ecn.com

"Lowering the cost of doing business"

Re: [Moosefs-users] Hardware choice

[Wang J. <jia...@re...>]

于 2012/4/19 22:23, Schmurfy 写道:
> Hi,
>
> I did some tests with moosefs on vms and "standard" machines (and I really love his project !) but now I need to decide on some rackable hardware to install in a datacenter and that's where things 
> become
> annoying since I have no idea what to choose :(
>
> I was thinking of starting with two nodes one of them being the master as well as a chunk server and the other will be a backup master and a chunk server too so I suppose both
> servers will need a decent amount of memory and a not too slow cpu but they don't require some high end muti-core processor and of course they will need some disks.
>
> I think we will start with less than 10Tb on the cluster and replication goals set to 2.
>
> We are currently using some ProLiant DL160 G6 which has 4 cores running at 2.0GHz but judging by the theoretical needs of our moosefs machine I think these servers
> are way too powerful for this usage.
>
>
> Does anyone can give me some advices on what would be a good start ?
>
>

Dell C5220 in C5000 chassis?

Downside is you can only use 2.5' hdd/ssd on this setup. But with 4 hdd/ssd slot per node by 8 nodes, it can be 32TB in capacity already at good performance to price ratio, and you can save a lot in 
hosting.

Re: [Moosefs-users] Hardware choice

[Andreas H. <ah...@it...>]

Schmurfy <sch...@gm...> writes:

> We are currently using some ProLiant DL160 G6 which has 4 cores running at
> 2.0GHz but judging by the theoretical needs of our moosefs machine I think
> these servers
> are way too powerful for this usage.

We use the same machines at 3 GHz (Intel Xeon X3470 @ 2.93GHz, 16 GByte
RAM). You are right, CPU usage is barely noticeable. To make better use
of resources we put some virtual machines (Webserver, Kerberos, AFS-DB)
also on those nodes.

Best regards
Andreas
-- 
Andreas Hirczy <ah...@it...>                   http://itp.tugraz.at/~ahi/
Graz University of Technology                        phone: +43/316/873-   8190
Institute of Theoretical and Computational Physics     fax: +43/316/873-10 8190
Petersgasse 16, A-8010 Graz                         mobile: +43/664/859 23 57

[Moosefs-users] chunkserver crash/shutdown

[Hanser <se...@gm...>]

Today, one of our chunkserver,  running 1.6.20-2 crashed giving out the
following errors. Going thru the docs and couldn't find anything relevant to
it.

I also noticed 3-errors-in-60 seconds ruleset, is there anyway to increase
the threshold in order to avoid further disruptions ? 

Thanks,

 

 

Apr 20 08:18:08 cluster-4 mfschunkserver[7892]: chunk_readcrc:
file:/mnt/mncloud/75/chunk_00000000005F0475_00000001.mfs - wrong id/version
in header (00000000005F0704_00000001)

Apr 20 08:18:08 cluster-4 mfschunkserver[7892]: hdd_io_begin:
file:/mnt/mncloud/75/chunk_00000000005F0475_00000001.mfs - read error:
Unknown error

Apr 20 08:18:10 cluster-4 mfschunkserver[7892]: testing chunk:
/mnt/mncloud/06/chunk_00000000005F0706_00000001.mfs

Apr 20 08:18:12 cluster-4 mfschunkserver[7892]: testing chunk:
/mnt/mncloud/77/chunk_00000000005F0477_00000001.mfs

Apr 20 08:18:12 cluster-4 mfschunkserver[7892]: chunk_readcrc:
file:/mnt/mncloud/77/chunk_00000000005F0477_00000001.mfs - wrong id/version
in header (00000000005F0706_00000001)

Apr 20 08:18:12 cluster-4 mfschunkserver[7892]: hdd_io_begin:
file:/mnt/mncloud/77/chunk_00000000005F0477_00000001.mfs - read error:
Unknown error

Apr 20 08:18:14 cluster-4 mfschunkserver[7892]: testing chunk:
/mnt/mncloud/08/chunk_00000000005F0708_00000001.mfs

Apr 20 08:18:16 cluster-4 mfschunkserver[7892]: testing chunk:
/mnt/mncloud/79/chunk_00000000005F0479_00000001.mfs

Apr 20 08:18:16 cluster-4 mfschunkserver[7892]: chunk_readcrc:
file:/mnt/mncloud/79/chunk_00000000005F0479_00000001.mfs - wrong id/version
in header (00000000005F0708_00000001)

Apr 20 08:18:16 cluster-4 mfschunkserver[7892]: hdd_io_begin:
file:/mnt/mncloud/79/chunk_00000000005F0479_00000001.mfs - read error:
Unknown error

Apr 20 08:18:17 cluster-4 mfschunkserver[7892]: 3 errors occurred in 60
seconds on folder: /mnt/mncloud/

Re: [Moosefs-users] mfschunkserver graceful shutdown

[Ken <ken...@gm...>]

Nobody interesting this?

There is one-thousandth of the possibility which cause file damage.

Maybe log like:
mfsmaster[7192]: chunk 00000000000EC3A8 has only invalid copies (2) -
please repair it manually
mfsmaster[7192]: chunk 00000000000EC3A8_00000002 - invalid copy on
(10.1.1.3 - ver:00000001)


-Ken



On Thu, Apr 19, 2012 at 9:01 AM, Ken <ken...@gm...> wrote:

> hi, list
>
> We found some crashes in mfschunkserver(1.6.24) in stopping. The test
> script maybe weired:
>
> while true:
>   select a ChunkServer
>   stop_it
>   start_it
>   sleep 1 second
>
> Almost 20MiB/s are writing to the system when the script running. It's a
> little crazy? er.
>
>
> The crash stack:
> #0  0x00000000004139e7 in masterconn_replicationfinished (status=0 '\0',
> packet=0x269b170) at masterconn.c:351
> 351        if (eptr->mode==DATA || eptr->mode==HEADER) {
>
> #0  0x00000000004139e7 in masterconn_replicationfinished (status=0 '\0',
> packet=0x269b170) at masterconn.c:351
> #1  0x0000000000403b6e in job_pool_check_jobs (jpool=0x7f39b43ddea0) at
> bgjobs.c:338
> #2  0x0000000000403f17 in job_pool_delete (jpool=0x7f39b43ddea0) at
> bgjobs.c:365
> #3  0x0000000000414b31 in masterconn_term () at masterconn.c:864
> #4  0x0000000000419173 in destruct () at ../mfscommon/main.c:312
> #5  0x000000000041b60f in main (argc=1, argv=0x7fffc810dda0) at
> ../mfscommon/main.c:1162
>
> # mfschunkserver -v
> version: 1.6.24
>
> I think masterconn_termm cause crash:
>
> void masterconn_term(void) {
>     packetstruct *pptr,*paptr;//    syslog(LOG_INFO,"closing %s:%s",MasterHost,MasterPort);
>     masterconn *eptr = masterconnsingleton;
>
>     if (eptr->mode!=FREE && eptr->mode!=CONNECTING) {
>         tcpclose(eptr->sock);
>
>         if (eptr->inputpacket.packet) {
>             free(eptr->inputpacket.packet);
>         }
>         pptr = eptr->outputhead;
>         while (pptr) {
>             if (pptr->packet) {
>                 free(pptr->packet);
>             }
>             paptr = pptr;
>             pptr = pptr->next;
>             free(paptr);
>         }
>     }
>
>     free(eptr);
>     masterconnsingleton = NULL;*    job_pool_delete(jpool);  // this is too later*
>     free(MasterHost);
>     free(MasterPort);
>     free(BindHost);}
>
> So we move the line to start. And patch below
>
> --- a/mfschunkserver/masterconn.c
> +++ b/mfschunkserver/masterconn.c
> @@ -842,6 +842,8 @@ void masterconn_term(void) {
>  //     syslog(LOG_INFO,"closing %s:%s",MasterHost,MasterPort);
>         masterconn *eptr = masterconnsingleton;
>
> +  job_pool_delete(jpool);
> +
>         if (eptr->mode!=FREE && eptr->mode!=CONNECTING) {
>                 tcpclose(eptr->sock);
>
> @@ -861,7 +863,7 @@ void masterconn_term(void) {
>
>         free(eptr);
>         masterconnsingleton = NULL;
> -       job_pool_delete(jpool);
> +
>         free(MasterHost);
>         free(MasterPort);
>         free(BindHost);
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ patch end
>
> Crash did not happened again with the patch, and the test almost run 12
> hours.
>
>
> HTH
>
> -Ken
>
>