moosefs-users — Mailing list for MooseFS Users

Re: [Moosefs-users] Small file sizes revisited - 12x space used

[Ken <ken...@gm...>]

On Fri, Sep 30, 2011 at 3:44 PM, Patrick Feliciano <fus...@gm...> wrote:
> On 09/28/2011 07:26 PM, Kristofer Pettijohn wrote:
>> GFS2 in Google was redesigned for smaller files.  Multi-master design is needed, but that is a huge overhaul and a lot of work to complete.
>>
>> Ask and beg for it; you might see it some day.
>>
> Those are interesting points, that MooseFS has an architecture like
> GoogleFS and now Google has the GFS2 aka Colossus.  Colossus is designed
> for smaller files and has a distributed master design.  Maybe that is
> what MooseFS 2 will work to emulate as well.
>> On Sep 28, 2011, at 8:55 PM, Ken wrote:
>>
>>> Distribute filesystem always design for huge space. Waste often exist. eg:
>>> Haystack in facebook, GFS in google never recycling space of delete
>>> files, they mark flag for deleted status.
>>>
> It isn't true that all distributed file systems are designed for huge
> files.  Lustre for instance uses the block size of the underlying file
> system.  I disagree that the concept of distributed file systems is
> synonymous with large files.  That doesn't strike me as a valid reason
> to dismiss the idea of variable block sizes at compile time.
It is true what you said.
We have plan to use moosefs for a photo storage which growth is 1
terabyte per week.
Sincerely hope moosefs support small files. You know photos are small files.


>>> Much small size files put into moose filesystem cause master server
>>> memory bottleneck.
>>> IMHO, space saving will never be main target in these systems.
>>>
> My servers can support 148GB of RAM which is enough for hundreds of
> millions of files.  That would give our site years of growth, I'm not as
> worried about that as I am about the fact that we only have 10TB of
> space unused on the web farm that I want to use with MooseFS. With 64KB
> blocks we will run out of that space well before we reach a hundred
> million files.  With 3 copies of the data we'd be out already with just
> the 50 million files  we currently have.
Let's count a few.
In master server failover, mfsmetarestore should read meta log for
building filesystem.
Generally read speed can reach 100MB per second, 148G RAM meta data
should recover in 148*1024/100=1515 seconds
That mean a failure restore more than 25 minute. It's not easy to
resolve these problem.

In moosefs source core, filesystem.c and chunk.c have been too
difficult to understand now. These feature may make it worse.


>>> If we must handle much small files, just like photo files, should
>>> bundle them into a big file(s). And use URL locate content, like
>>> '/prefix/bundle_filename/offset/length/check_sum.jpg'.
> That is an interesting idea and I'm not against it if you can tell me
> what tools will do that and allow me to present it as a standard POSIX
> filesystem path.   Seems to me though that a smaller block size for this
> awesome filesystem is still the better fix.
We have plan to open source these tools.

>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2dcopy2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

Re: [Moosefs-users] Small file sizes revisited - 12x space used

[Michał B. <mic...@ge...>]

We had some tests with underlying systems of ext3 and FAT32. But didn't try
to mount it by multiple clients - I'm not sure if Truecrypt is ready for
this. But probably it could be mounted once (by the "main" client) and later
other machines could also write to this mount. 

Probably for data which don't need to be encrypted, Truecrypt won't be the
best option (though extra layer of encryption shouldn't hurt). Do you know
any other cross-platfrom "volume" filesystem without encryption, but
probably with compression?


Regards
Michal

-----Original Message-----
From: Allen, Benjamin S [mailto:bs...@la...] 
Sent: Friday, September 30, 2011 5:06 PM
To: Patrick Feliciano
Cc: moo...@li...
Subject: Re: [Moosefs-users] Small file sizes revisited - 12x space used

Curious if your underlying filesystem was ZFS (or similar), you could enable
compression. I'd guess that chunks that were padded to be 64k, i.e. only 4k
of data would be well compressed to near 4k. I haven't tested this, but it
would be an interesting work around. Of course you're adding CPU load to
your chunk servers by doing this. I'll test this theory at some point since
I plan on using compression behind MooseFS anyways.

Ben

On Sep 30, 2011, at 1:44 AM, Patrick Feliciano wrote:

> On 09/28/2011 07:26 PM, Kristofer Pettijohn wrote:
>> GFS2 in Google was redesigned for smaller files.  Multi-master design is
needed, but that is a huge overhaul and a lot of work to complete.
>> 
>> Ask and beg for it; you might see it some day.
>> 
> Those are interesting points, that MooseFS has an architecture like 
> GoogleFS and now Google has the GFS2 aka Colossus.  Colossus is designed 
> for smaller files and has a distributed master design.  Maybe that is 
> what MooseFS 2 will work to emulate as well.
>> On Sep 28, 2011, at 8:55 PM, Ken wrote:
>> 
>>> Distribute filesystem always design for huge space. Waste often exist.
eg:
>>> Haystack in facebook, GFS in google never recycling space of delete
>>> files, they mark flag for deleted status.
>>> 
> It isn't true that all distributed file systems are designed for huge 
> files.  Lustre for instance uses the block size of the underlying file 
> system.  I disagree that the concept of distributed file systems is 
> synonymous with large files.  That doesn't strike me as a valid reason 
> to dismiss the idea of variable block sizes at compile time.
>>> Much small size files put into moose filesystem cause master server
>>> memory bottleneck.
>>> IMHO, space saving will never be main target in these systems.
>>> 
> My servers can support 148GB of RAM which is enough for hundreds of 
> millions of files.  That would give our site years of growth, I'm not as 
> worried about that as I am about the fact that we only have 10TB of 
> space unused on the web farm that I want to use with MooseFS. With 64KB 
> blocks we will run out of that space well before we reach a hundred 
> million files.  With 3 copies of the data we'd be out already with just 
> the 50 million files  we currently have.
>>> If we must handle much small files, just like photo files, should
>>> bundle them into a big file(s). And use URL locate content, like
>>> '/prefix/bundle_filename/offset/length/check_sum.jpg'.
> That is an interesting idea and I'm not against it if you can tell me 
> what tools will do that and allow me to present it as a standard POSIX 
> filesystem path.   Seems to me though that a smaller block size for this 
> awesome filesystem is still the better fix.
> 
> 
>
----------------------------------------------------------------------------
--
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2dcopy2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users


----------------------------------------------------------------------------
--
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Small file sizes revisited - 12x space used

[Allen, B. S <bs...@la...>]

Curious if your underlying filesystem was ZFS (or similar), you could enable compression. I'd guess that chunks that were padded to be 64k, i.e. only 4k of data would be well compressed to near 4k. I haven't tested this, but it would be an interesting work around. Of course you're adding CPU load to your chunk servers by doing this. I'll test this theory at some point since I plan on using compression behind MooseFS anyways.

Ben

On Sep 30, 2011, at 1:44 AM, Patrick Feliciano wrote:

> On 09/28/2011 07:26 PM, Kristofer Pettijohn wrote:
>> GFS2 in Google was redesigned for smaller files.  Multi-master design is needed, but that is a huge overhaul and a lot of work to complete.
>> 
>> Ask and beg for it; you might see it some day.
>> 
> Those are interesting points, that MooseFS has an architecture like 
> GoogleFS and now Google has the GFS2 aka Colossus.  Colossus is designed 
> for smaller files and has a distributed master design.  Maybe that is 
> what MooseFS 2 will work to emulate as well.
>> On Sep 28, 2011, at 8:55 PM, Ken wrote:
>> 
>>> Distribute filesystem always design for huge space. Waste often exist. eg:
>>> Haystack in facebook, GFS in google never recycling space of delete
>>> files, they mark flag for deleted status.
>>> 
> It isn't true that all distributed file systems are designed for huge 
> files.  Lustre for instance uses the block size of the underlying file 
> system.  I disagree that the concept of distributed file systems is 
> synonymous with large files.  That doesn't strike me as a valid reason 
> to dismiss the idea of variable block sizes at compile time.
>>> Much small size files put into moose filesystem cause master server
>>> memory bottleneck.
>>> IMHO, space saving will never be main target in these systems.
>>> 
> My servers can support 148GB of RAM which is enough for hundreds of 
> millions of files.  That would give our site years of growth, I'm not as 
> worried about that as I am about the fact that we only have 10TB of 
> space unused on the web farm that I want to use with MooseFS. With 64KB 
> blocks we will run out of that space well before we reach a hundred 
> million files.  With 3 copies of the data we'd be out already with just 
> the 50 million files  we currently have.
>>> If we must handle much small files, just like photo files, should
>>> bundle them into a big file(s). And use URL locate content, like
>>> '/prefix/bundle_filename/offset/length/check_sum.jpg'.
> That is an interesting idea and I'm not against it if you can tell me 
> what tools will do that and allow me to present it as a standard POSIX 
> filesystem path.   Seems to me though that a smaller block size for this 
> awesome filesystem is still the better fix.
> 
> 
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2dcopy2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Small file sizes revisited - 12x space used

[Michał B. <mic...@ge...>]

Hi

We had some tests with creating big (hundreds of gigabytes in size)
truecrypt (http://www.truecrypt.org/) volumes stored in the MooseFS which is
seen as one file and underneath you can have as much small files as you
want. Truecrypt volumes are easily "rsyncable", so that one minor change
causes also a small change only in one part of the file
(http://www.rsync.net/resources/howto/windows_truecrypt.html). Though in
MooseFS it causes replacement of the whole chunk, but the replaced chunk
gets deleted in the background. This way you do not lose space having lots
of small files. It is very good solution for read only files, but would need
some further performance tests if small files get modified very often.


Kind regards
Michal Borychowski 



-----Original Message-----
From: Patrick Feliciano [mailto:fus...@gm...] 
Sent: Friday, September 30, 2011 9:45 AM
To: moo...@li...
Subject: Re: [Moosefs-users] Small file sizes revisited - 12x space used

On 09/28/2011 07:26 PM, Kristofer Pettijohn wrote:
> GFS2 in Google was redesigned for smaller files.  Multi-master design is
needed, but that is a huge overhaul and a lot of work to complete.
>
> Ask and beg for it; you might see it some day.
>
Those are interesting points, that MooseFS has an architecture like 
GoogleFS and now Google has the GFS2 aka Colossus.  Colossus is designed 
for smaller files and has a distributed master design.  Maybe that is 
what MooseFS 2 will work to emulate as well.
> On Sep 28, 2011, at 8:55 PM, Ken wrote:
>
>> Distribute filesystem always design for huge space. Waste often exist.
eg:
>> Haystack in facebook, GFS in google never recycling space of delete
>> files, they mark flag for deleted status.
>>
It isn't true that all distributed file systems are designed for huge 
files.  Lustre for instance uses the block size of the underlying file 
system.  I disagree that the concept of distributed file systems is 
synonymous with large files.  That doesn't strike me as a valid reason 
to dismiss the idea of variable block sizes at compile time.
>> Much small size files put into moose filesystem cause master server
>> memory bottleneck.
>> IMHO, space saving will never be main target in these systems.
>>
My servers can support 148GB of RAM which is enough for hundreds of 
millions of files.  That would give our site years of growth, I'm not as 
worried about that as I am about the fact that we only have 10TB of 
space unused on the web farm that I want to use with MooseFS. With 64KB 
blocks we will run out of that space well before we reach a hundred 
million files.  With 3 copies of the data we'd be out already with just 
the 50 million files  we currently have.
>> If we must handle much small files, just like photo files, should
>> bundle them into a big file(s). And use URL locate content, like
>> '/prefix/bundle_filename/offset/length/check_sum.jpg'.
That is an interesting idea and I'm not against it if you can tell me 
what tools will do that and allow me to present it as a standard POSIX 
filesystem path.   Seems to me though that a smaller block size for this 
awesome filesystem is still the better fix.


----------------------------------------------------------------------------
--
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Small file sizes revisited - 12x space used

[Patrick F. <fus...@gm...>]

On 09/28/2011 07:26 PM, Kristofer Pettijohn wrote:
> GFS2 in Google was redesigned for smaller files.  Multi-master design is needed, but that is a huge overhaul and a lot of work to complete.
>
> Ask and beg for it; you might see it some day.
>
Those are interesting points, that MooseFS has an architecture like 
GoogleFS and now Google has the GFS2 aka Colossus.  Colossus is designed 
for smaller files and has a distributed master design.  Maybe that is 
what MooseFS 2 will work to emulate as well.
> On Sep 28, 2011, at 8:55 PM, Ken wrote:
>
>> Distribute filesystem always design for huge space. Waste often exist. eg:
>> Haystack in facebook, GFS in google never recycling space of delete
>> files, they mark flag for deleted status.
>>
It isn't true that all distributed file systems are designed for huge 
files.  Lustre for instance uses the block size of the underlying file 
system.  I disagree that the concept of distributed file systems is 
synonymous with large files.  That doesn't strike me as a valid reason 
to dismiss the idea of variable block sizes at compile time.
>> Much small size files put into moose filesystem cause master server
>> memory bottleneck.
>> IMHO, space saving will never be main target in these systems.
>>
My servers can support 148GB of RAM which is enough for hundreds of 
millions of files.  That would give our site years of growth, I'm not as 
worried about that as I am about the fact that we only have 10TB of 
space unused on the web farm that I want to use with MooseFS. With 64KB 
blocks we will run out of that space well before we reach a hundred 
million files.  With 3 copies of the data we'd be out already with just 
the 50 million files  we currently have.
>> If we must handle much small files, just like photo files, should
>> bundle them into a big file(s). And use URL locate content, like
>> '/prefix/bundle_filename/offset/length/check_sum.jpg'.
That is an interesting idea and I'm not against it if you can tell me 
what tools will do that and allow me to present it as a standard POSIX 
filesystem path.   Seems to me though that a smaller block size for this 
awesome filesystem is still the better fix.

Re: [Moosefs-users] Small file sizes revisited - 12x space used

[Kristofer P. <kri...@cy...>]

GFS2 in Google was redesigned for smaller files.  Multi-master design is needed, but that is a huge overhaul and a lot of work to complete.

Ask and beg for it; you might see it some day.

On Sep 28, 2011, at 8:55 PM, Ken wrote:

> Distribute filesystem always design for huge space. Waste often exist. eg:
> Haystack in facebook, GFS in google never recycling space of delete
> files, they mark flag for deleted status.
> 
> Much small size files put into moose filesystem cause master server
> memory bottleneck.
> IMHO, space saving will never be main target in these systems.
> 
> If we must handle much small files, just like photo files, should
> bundle them into a big file(s). And use URL locate content, like
> '/prefix/bundle_filename/offset/length/check_sum.jpg'.
> 
> 
> Best Regards
> -Ken
> 
> 
> On Thu, Sep 29, 2011 at 4:55 AM, Patrick Feliciano <fus...@gm...> wrote:
>> 
>> I'd like to start with how very impressed I am with the MooseFS features
>> and architecture.   I even prepared a presentation to sell the benefits
>> of MooseFS for our web services to management.  It is the only thing
>> I've found that is easy to manage, easily extendible, with good
>> documentation, has automated replication, fault tolerance, self healing,
>> and POSIX ( a requirement of our design ).  Only one problem, many of
>> our files are approx. 4KB.  So average space used on MooseFS for that
>> class of files is in excess of 12 times the expected.
>> 
>> Now before you reply with the same response I've read in the FAQ and
>> seen in the mailing list archives; I understand that MooseFS was written
>> for large files and that is what it is used for by Gemius.  And I've
>> seen that others point to other systems that can handle small files.
>> 
>> However none of those systems pointed to have the same feature set as
>> MooseFS.  Even if they have extendibility and fault tolerance, none I've
>> seen also present a POSIX file system like we need.
>> 
>> Also I agree that the block size should not be a configurable of the
>> compiled FS.  There are too many pieces to manage to be worried that you
>> set the right block size configurable on each chunk server and add extra
>> code to deal with variable block sizes in the master etc.  Ugh.  Mess, I
>> totally agree.
>> 
>> But how about at compile time as a option to ./configure ?  How about I
>> pick block size then and compile a complete set of master, metalogger,
>> chunk, and client apps and/or RPMs that all have the hardcoded block
>> size I pick then.   I would think this change would be much easier to
>> implement.  I imagine that a constant would need to be changed somewhere.
>> 
>> This would be very good for the spread and reputation of MooseFS,
>> enabling its wider use and adoption as a general purpose DFS, adaptable
>> to suit individual application needs. Also we'd be able to add our
>> website with millions of users to the "Using MooseFS" list. :)
>> 
>> So unless someone can point me to something else that REALLY has all of
>> MooseFS's features, including POSIX... Well then, I think it is simply
>> cruel to limit such an amazing tool and exclude those of us who could
>> make such wonderful use of it.
>> 
>> Of course, I have the source code and I can try to figure it out myself,
>> but it would be much easier going with your cooperation and guidance.  I
>> would be willing to do the implementation myself and contribute it back.
>> 
>> Please truly consider this, and if not, please consider at least
>> pointing me to the right places in the source code I should look to
>> implement the changes myself.
>> 
>> Thank you very much,
>> 
>> Patrick Feliciano
>> Systems Administrator
>> Livemocha, Inc.
>> 
>> 
>> ------------------------------------------------------------------------------
>> All the data continuously generated in your IT infrastructure contains a
>> definitive record of customers, application performance, security
>> threats, fraudulent activity and more. Splunk takes this data and makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2dcopy1
>> _______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> 
> 
> -Ken
> 
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2dcopy1
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Small file sizes revisited - 12x space used

[Ken <ken...@gm...>]

Distribute filesystem always design for huge space. Waste often exist. eg:
Haystack in facebook, GFS in google never recycling space of delete
files, they mark flag for deleted status.

Much small size files put into moose filesystem cause master server
memory bottleneck.
IMHO, space saving will never be main target in these systems.

If we must handle much small files, just like photo files, should
bundle them into a big file(s). And use URL locate content, like
'/prefix/bundle_filename/offset/length/check_sum.jpg'.


Best Regards
-Ken


On Thu, Sep 29, 2011 at 4:55 AM, Patrick Feliciano <fus...@gm...> wrote:
>
> I'd like to start with how very impressed I am with the MooseFS features
> and architecture.   I even prepared a presentation to sell the benefits
> of MooseFS for our web services to management.  It is the only thing
> I've found that is easy to manage, easily extendible, with good
> documentation, has automated replication, fault tolerance, self healing,
> and POSIX ( a requirement of our design ).  Only one problem, many of
> our files are approx. 4KB.  So average space used on MooseFS for that
> class of files is in excess of 12 times the expected.
>
> Now before you reply with the same response I've read in the FAQ and
> seen in the mailing list archives; I understand that MooseFS was written
> for large files and that is what it is used for by Gemius.  And I've
> seen that others point to other systems that can handle small files.
>
> However none of those systems pointed to have the same feature set as
> MooseFS.  Even if they have extendibility and fault tolerance, none I've
> seen also present a POSIX file system like we need.
>
> Also I agree that the block size should not be a configurable of the
> compiled FS.  There are too many pieces to manage to be worried that you
> set the right block size configurable on each chunk server and add extra
> code to deal with variable block sizes in the master etc.  Ugh.  Mess, I
> totally agree.
>
> But how about at compile time as a option to ./configure ?  How about I
> pick block size then and compile a complete set of master, metalogger,
> chunk, and client apps and/or RPMs that all have the hardcoded block
> size I pick then.   I would think this change would be much easier to
> implement.  I imagine that a constant would need to be changed somewhere.
>
> This would be very good for the spread and reputation of MooseFS,
> enabling its wider use and adoption as a general purpose DFS, adaptable
> to suit individual application needs. Also we'd be able to add our
> website with millions of users to the "Using MooseFS" list. :)
>
> So unless someone can point me to something else that REALLY has all of
> MooseFS's features, including POSIX... Well then, I think it is simply
> cruel to limit such an amazing tool and exclude those of us who could
> make such wonderful use of it.
>
> Of course, I have the source code and I can try to figure it out myself,
> but it would be much easier going with your cooperation and guidance.  I
> would be willing to do the implementation myself and contribute it back.
>
> Please truly consider this, and if not, please consider at least
> pointing me to the right places in the source code I should look to
> implement the changes myself.
>
> Thank you very much,
>
> Patrick Feliciano
> Systems Administrator
> Livemocha, Inc.
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2dcopy1
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users


-Ken

[Moosefs-users] lots of read block errors and can't connect to poper chunkserver after replaced three new hard drive.

[yezhou(Joe) <fan...@gm...>]

I have a chunkserver has three hard drive died last week, and I replace them
with new drives.

Since I have the goal of two ,there are no missing chunks.

but it still reports lots of read block errors and can't connect to poper
chunkserver, and I found out the reason is there are lots of write of the
three new drives. so the read comes to low and the chunkserver stuck at that
monent.

who knows whats wrong with it?

[Moosefs-users] Small file sizes revisited - 12x space used

[Patrick F. <fus...@gm...>]

I'd like to start with how very impressed I am with the MooseFS features 
and architecture.   I even prepared a presentation to sell the benefits 
of MooseFS for our web services to management.  It is the only thing 
I've found that is easy to manage, easily extendible, with good 
documentation, has automated replication, fault tolerance, self healing, 
and POSIX ( a requirement of our design ).  Only one problem, many of 
our files are approx. 4KB.  So average space used on MooseFS for that 
class of files is in excess of 12 times the expected.

Now before you reply with the same response I've read in the FAQ and 
seen in the mailing list archives; I understand that MooseFS was written 
for large files and that is what it is used for by Gemius.  And I've 
seen that others point to other systems that can handle small files.

However none of those systems pointed to have the same feature set as 
MooseFS.  Even if they have extendibility and fault tolerance, none I've 
seen also present a POSIX file system like we need.

Also I agree that the block size should not be a configurable of the 
compiled FS.  There are too many pieces to manage to be worried that you 
set the right block size configurable on each chunk server and add extra 
code to deal with variable block sizes in the master etc.  Ugh.  Mess, I 
totally agree.

But how about at compile time as a option to ./configure ?  How about I 
pick block size then and compile a complete set of master, metalogger, 
chunk, and client apps and/or RPMs that all have the hardcoded block 
size I pick then.   I would think this change would be much easier to 
implement.  I imagine that a constant would need to be changed somewhere.

This would be very good for the spread and reputation of MooseFS, 
enabling its wider use and adoption as a general purpose DFS, adaptable 
to suit individual application needs. Also we'd be able to add our 
website with millions of users to the "Using MooseFS" list. :)

So unless someone can point me to something else that REALLY has all of 
MooseFS's features, including POSIX... Well then, I think it is simply 
cruel to limit such an amazing tool and exclude those of us who could 
make such wonderful use of it.

Of course, I have the source code and I can try to figure it out myself, 
but it would be much easier going with your cooperation and guidance.  I 
would be willing to do the implementation myself and contribute it back.

Please truly consider this, and if not, please consider at least 
pointing me to the right places in the source code I should look to 
implement the changes myself.

Thank you very much,

Patrick Feliciano
Systems Administrator
Livemocha, Inc.

Re: [Moosefs-users] mfsmetalogger segfault

[Laurent W. <lw...@hy...>]

On Mon, 26 Sep 2011 17:45:00 -0400
Robert Sandilands <rsa...@ne...> wrote:

> I saw similar crashes when mfsmaster was unresponsive for long times. 
> Specifically around the hour when it dumps the meta-data to disk and the 
> master becomes very busy.
> 
> If this sounds familiar ensure that you have 2 times the memory usage of 
> mfsmaster available on the master and it should be better. This fixed it 
> for me.
the machine has 32 gb ram, while mfsmaster eats only 3gb, so I don't
think this is the reason. It may have happened while mfs was especially
loaded (master also runs as CS on top of a 12 disks raid 10 volume).
About Ricardo's errors, these sound not to be the same. Here, some
double free (ou whatever else) makes glibc grumpy. Unfortunately, I
can't play with cables as this is a prod system…
Thanks,
-- 
Laurent Wandrebeck
HYGEOS, Earth Observation Department / Observation de la Terre
Euratechnologies
165 Avenue de Bretagne
59000 Lille, France
tel: +33 3 20 08 24 98
http://www.hygeos.com
GPG fingerprint/Empreinte GPG: F5CA 37A4 6D03 A90C 7A1D  2A62 54E6 EF2C
D17C F64C

[Moosefs-users] read request to mfsmaster

[Jianwei L. <lia...@gm...>]

Dear all,
I checked the statistics on different operations to master node. I was
wondering that in which kind of situation, the clients send the "read"
request to master node.
you know, It seems that the client lookups the file location from the master
node, then contacts to the chunkserver to read the file data, so, why does
the client send the "read" requests to the master nod.

Re: [Moosefs-users] mfsmetalogger segfault

[Robert S. <rsa...@ne...>]

I saw similar crashes when mfsmaster was unresponsive for long times. 
Specifically around the hour when it dumps the meta-data to disk and the 
master becomes very busy.

If this sounds familiar ensure that you have 2 times the memory usage of 
mfsmaster available on the master and it should be better. This fixed it 
for me.

Robert

On 9/26/11 1:57 PM, Ricardo J. Barberis wrote:
> El Lunes 26 Septiembre 2011, Laurent Wandrebeck escribió:
>> Hi,
>>
>> CentOS 6 x86_64, mfs 1.6.20 package from rpmforge.
>> mfsmetalogger[2043]: segfault at 7f261ddf6380 ip 00007f261dae42e5 sp
>> 00007fff6dee7d50 error 7 in libc-2.12.so[7f261da7e000+175000]
>>
>> Has anyone seen that ?
> Under CentOS 5 x86_64, mfs-1.6.20 from rpmforge, I got this on September 18:
>
> kernel: mfsmetalogger[1967]: segfault at 0000000000000060 rip 0000003fb4a612ed
> rsp 00007fff709d0860 error 4
>
> On another metalogger server, from September 8:
>
> kernel: mfsmetalogger[2150]: segfault at 0000000000000008 rip 00002b2f88718350
> rsp 00007fff9c3044e0 error 4
>
> kernel: mfsmetalogger[2149]: segfault at 0000000000000008 rip 00002b16ea2f9350
> rsp 00007fff58b948d0 error 4
>
>
> I think I can reproduce it (thought I haven't really tested it) by unpluggin
> the network cable, or at least I've seen it happen a couple of times while
> moving some servers to a new switch.
>
> It's an easy test so I guess I could try it and see if it crashes.
>
> Cheers,

Re: [Moosefs-users] mfsmetalogger segfault

[Ricardo J. B. <ric...@da...>]

El Lunes 26 Septiembre 2011, Laurent Wandrebeck escribió:
> Hi,
>
> CentOS 6 x86_64, mfs 1.6.20 package from rpmforge.
> mfsmetalogger[2043]: segfault at 7f261ddf6380 ip 00007f261dae42e5 sp
> 00007fff6dee7d50 error 7 in libc-2.12.so[7f261da7e000+175000]
>
> Has anyone seen that ?

Under CentOS 5 x86_64, mfs-1.6.20 from rpmforge, I got this on September 18:

kernel: mfsmetalogger[1967]: segfault at 0000000000000060 rip 0000003fb4a612ed 
rsp 00007fff709d0860 error 4

On another metalogger server, from September 8:

kernel: mfsmetalogger[2150]: segfault at 0000000000000008 rip 00002b2f88718350 
rsp 00007fff9c3044e0 error 4

kernel: mfsmetalogger[2149]: segfault at 0000000000000008 rip 00002b16ea2f9350 
rsp 00007fff58b948d0 error 4


I think I can reproduce it (thought I haven't really tested it) by unpluggin 
the network cable, or at least I've seen it happen a couple of times while 
moving some servers to a new switch.

It's an easy test so I guess I could try it and see if it crashes.

Cheers,
-- 
Ricardo J. Barberis
Senior SysAdmin / ITI
Dattatec.com :: Soluciones de Web Hosting
Tu Hosting hecho Simple!

[Moosefs-users] mfsmetalogger segfault

[Laurent W. <lw...@hy...>]

Hi,

CentOS 6 x86_64, mfs 1.6.20 package from rpmforge.
mfsmetalogger[2043]: segfault at 7f261ddf6380 ip 00007f261dae42e5 sp
00007fff6dee7d50 error 7 in libc-2.12.so[7f261da7e000+175000]

Has anyone seen that ?
-- 
Laurent Wandrebeck
HYGEOS, Earth Observation Department / Observation de la Terre
Euratechnologies
165 Avenue de Bretagne
59000 Lille, France
tel: +33 3 20 08 24 98
http://www.hygeos.com
GPG fingerprint/Empreinte GPG: F5CA 37A4 6D03 A90C 7A1D  2A62 54E6 EF2C
D17C F64C

Re: [Moosefs-users] Data deduplication

[Michał B. <mic...@ge...>]

Unfortunately quotas won't be in the next release

 

Regards

Michal

 

 

From: Florent Bautista [mailto:fl...@co...] 
Sent: Monday, September 26, 2011 10:05 AM
To: Michal Borychowski
Cc: moo...@li...
Subject: Re: [Moosefs-users] Data deduplication

 

Hi,

Thank you for you answer, I understand!

By the way, what about quotas ? Will it be implemented in next version or later ?

Thank you.

Le 24/09/2011 15:04, Michał Borychowski a écrit : 

Hi!

 

We talk among us about this feature from time to time and our decision is not to implement it for the moment. There are much more important things to do about MooseFS (like quota, acls, RAM optimizations, optimization of deleting files process, making snapshots even better, etc.) that file deduplication is not that important. We know it is loss of space but nowadays it's not that expensive. So maybe one time we come back to this, but unfortunately not in the near future.

 

 

Kind regards

Michał Borychowski

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

 

From: Florent Bautista [mailto:fl...@co...] 
Sent: Wednesday, August 31, 2011 4:54 PM
To: moo...@li...
Subject: [Moosefs-users] Data deduplication

 

Hi all,

I would like to know if MooseFS will (in which future ?) take care about data deduplication.

I know that MooseFS makes a checksum of every chunk, so would it be possible to have data deduplication at that level ?
If two (or more) files with goal=3 each, have chunk(s) in common, only store 3 copies of that (those) chunk(s) and not 6 (or more) like today...

For files having different goals, use the more important goal.

I think it does not change the architecture of MooseFS... maybe the problem is that MFS Master do not know about checksums, it is made on CS... but we could find a way to go through

Is it difficult to add that feature ? What do you think about it ?

Thank you guys!

-- 


Florent Bautista

  _____  

Ce message et ses éventuelles pièces jointes sont personnels, confidentiels et à l'usage exclusif de leur destinataire.
Si vous n'êtes pas la personne à laquelle ce message est destiné, veuillez noter que vous avez reçu ce courriel par erreur et qu'il vous est strictement interdit d'utiliser, de diffuser, de transférer, d'imprimer ou de copier ce message.

This e-mail and any attachments hereto are strictly personal, confidential and intended solely for the addressee.
If you are not the intended recipient, be advised that you have received this email in error and that any use, dissemination, forwarding, printing, or copying of this message is strictly prohibited.

  _____  

 

-- 


Florent Bautista

  _____  

Ce message et ses éventuelles pièces jointes sont personnels, confidentiels et à l'usage exclusif de leur destinataire.
Si vous n'êtes pas la personne à laquelle ce message est destiné, veuillez noter que vous avez reçu ce courriel par erreur et qu'il vous est strictement interdit d'utiliser, de diffuser, de transférer, d'imprimer ou de copier ce message.

This e-mail and any attachments hereto are strictly personal, confidential and intended solely for the addressee.
If you are not the intended recipient, be advised that you have received this email in error and that any use, dissemination, forwarding, printing, or copying of this message is strictly prohibited.

  _____  

Re: [Moosefs-users] Data deduplication

[Florent B. <fl...@co...>]

Hi,

Thank you for you answer, I understand!

By the way, what about quotas ? Will it be implemented in next version
or later ?

Thank you.

Le 24/09/2011 15:04, Michał Borychowski a écrit :
>
> Hi!
>
>  
>
> We talk among us about this feature from time to time and our decision
> is not to implement it for the moment. There are much more important
> things to do about MooseFS (like quota, acls, RAM optimizations,
> optimization of deleting files process, making snapshots even better,
> etc.) that file deduplication is not that important. We know it is
> loss of space but nowadays it's not that expensive. So maybe one time
> we come back to this, but unfortunately not in the near future.
>
>  
>
>  
>
> Kind regards
>
> Michał Borychowski
>
> MooseFS Support Manager
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Gemius S.A.
>
> ul. Wołoska 7, 02-672 Warszawa
>
> Budynek MARS, klatka D
>
> Tel.: +4822 874-41-00
>
> Fax : +4822 874-41-01
>
>  
>
>  
>
> *From:*Florent Bautista [mailto:fl...@co...]
> *Sent:* Wednesday, August 31, 2011 4:54 PM
> *To:* moo...@li...
> *Subject:* [Moosefs-users] Data deduplication
>
>  
>
> Hi all,
>
> I would like to know if MooseFS will (in which future ?) take care
> about data deduplication.
>
> I know that MooseFS makes a checksum of every chunk, so would it be
> possible to have data deduplication at that level ?
> If two (or more) files with goal=3 each, have chunk(s) in common, only
> store 3 copies of that (those) chunk(s) and not 6 (or more) like today...
>
> For files having different goals, use the more important goal.
>
> I think it does not change the architecture of MooseFS... maybe the
> problem is that MFS Master do not know about checksums, it is made on
> CS... but we could find a way to go through
>
> Is it difficult to add that feature ? What do you think about it ?
>
> Thank you guys!
>
> -- 
>
>
> Florent Bautista
>
> ------------------------------------------------------------------------
>
> Ce message et ses éventuelles pièces jointes sont personnels,
> confidentiels et à l'usage exclusif de leur destinataire.
> Si vous n'êtes pas la personne à laquelle ce message est destiné,
> veuillez noter que vous avez reçu ce courriel par erreur et qu'il vous
> est strictement interdit d'utiliser, de diffuser, de transférer,
> d'imprimer ou de copier ce message.
>
> This e-mail and any attachments hereto are strictly personal,
> confidential and intended solely for the addressee.
> If you are not the intended recipient, be advised that you have
> received this email in error and that any use, dissemination,
> forwarding, printing, or copying of this message is strictly prohibited.
>
> ------------------------------------------------------------------------


-- 


Florent Bautista

------------------------------------------------------------------------

Ce message et ses éventuelles pièces jointes sont personnels,
confidentiels et à l'usage exclusif de leur destinataire.
Si vous n'êtes pas la personne à laquelle ce message est destiné,
veuillez noter que vous avez reçu ce courriel par erreur et qu'il vous
est strictement interdit d'utiliser, de diffuser, de transférer,
d'imprimer ou de copier ce message.

This e-mail and any attachments hereto are strictly personal,
confidential and intended solely for the addressee.
If you are not the intended recipient, be advised that you have received
this email in error and that any use, dissemination, forwarding,
printing, or copying of this message is strictly prohibited.

------------------------------------------------------------------------

Re: [Moosefs-users] Auto MetaRestore

[Michał B. <mic...@ge...>]

It depends on the type of failure. Some changelogs may be missing or
corrupted. If you run out of your disk space, the metadata dump maybe be
corrupted. You should also use changelogs from metaloggers for recovery,
etc. You may want to check the history of our group
(http://sourceforge.net/search/?group_id=228631&type_of_search=mlists) and
find some detailed explanations made by Thomas S Hatch, eg.:
http://sourceforge.net/mailarchive/message.php?msg_id=27235785


Kind regards
Michal 

-----Original Message-----
From: Alexander Akhobadze [mailto:akh...@ri...] 
Sent: Monday, September 26, 2011 8:10 AM
To: Michał Borychowski
Cc: moo...@li...
Subject: Re: [Moosefs-users] Auto MetaRestore


Hm...
So as far as I understand something may go wrong during restore
and/or something needs my attension.

Could you describe more exactly what scenarios may I face to ?

wbr
Alexander

======================================================

Hi!

Of course you may do it but we recommend recovering the master server
manually - it is wise to observe the mfsmetarestore process. The automatic
recovery process is quite complicated and until some better scripts are
prepared, you'd rather recover it manually.


Kind regards
Michal


-----Original Message-----
From: Alexander Akhobadze [mailto:akh...@ri...] 
Sent: Friday, August 26, 2011 12:51 PM
To: moo...@li...
Subject: [Moosefs-users] Auto MetaRestore

Hi All!

I wonder if it is a good idea to put in init/rc script
which starts mfsmaster something like

mfs_data_path=`grep DATA_PATH /etc/mfsmaster.cfg | grep -v ^# | awk '{print
$NF}'`
if [ ! -f $mfs_data_path/metadata.mfs ]
then
    mfsmetarestore -a -d $mfs_data_path
fi

???????
In my opinion it is bare necessity to automatically make MooseFS healthy
in case of power failure/restore.

wbr
Alexander


----------------------------------------------------------------------------
--
EMC VNX: the world's simplest storage, starting under $10K
The only unified storage solution that offers unified management 
Up to 160% more powerful than alternatives and 25% more efficient. 
Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users


----------------------------------------------------------------------------
--
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2dcopy1
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Auto MetaRestore

[Alexander A. <akh...@ri...>]

Hm...
So as far as I understand something may go wrong during restore
and/or something needs my attension.

Could you describe more exactly what scenarios may I face to ?

wbr
Alexander

======================================================

Hi!

Of course you may do it but we recommend recovering the master server
manually - it is wise to observe the mfsmetarestore process. The automatic
recovery process is quite complicated and until some better scripts are
prepared, you'd rather recover it manually.


Kind regards
Michal


-----Original Message-----
From: Alexander Akhobadze [mailto:akh...@ri...] 
Sent: Friday, August 26, 2011 12:51 PM
To: moo...@li...
Subject: [Moosefs-users] Auto MetaRestore

Hi All!

I wonder if it is a good idea to put in init/rc script
which starts mfsmaster something like

mfs_data_path=`grep DATA_PATH /etc/mfsmaster.cfg | grep -v ^# | awk '{print
$NF}'`
if [ ! -f $mfs_data_path/metadata.mfs ]
then
    mfsmetarestore -a -d $mfs_data_path
fi

???????
In my opinion it is bare necessity to automatically make MooseFS healthy
in case of power failure/restore.

wbr
Alexander


----------------------------------------------------------------------------
--
EMC VNX: the world's simplest storage, starting under $10K
The only unified storage solution that offers unified management 
Up to 160% more powerful than alternatives and 25% more efficient. 
Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Why is it increasing my DEL_LIMIT when I don't want it to!

[Davies L. <dav...@gm...>]

Maybe we could try int another way: limit the total operations in every
event loop,
then distribute the operation limit by priority for every kind operation,
such as delete, replicated.
Then the mfsmaster ban balance throughput performance and operation latency.

Davies

2011/9/25 wk...@bn... <wk...@bn...>

> On 9/24/11 6:04 AM, Michał Borychowski wrote:
> > Hi!
> >
> > We discussed similiar problem with Ólafur in July as far as I remember.
> Yes,
> > we know it is not very optimal - if there are many files to be deleted
> > system tries to increase the limit so that they get deleted, but
> > unfortunately with realy huge number of files to be deleted, system
> stucks
> > up... We have some ideas for improvement, eg. first doing truncate
> (setting
> > to 0 bytes) and later do the real deleting. Also setting of this limit
> would
> > be possible on the fly with mastertools.
> >
> >
> >
>
> What creates the Deletion Problem is this code in chunks.c
>
> if (delnotdone > deldone && delnotdone > prevdelnotdone) {
>     TmpMaxDelFrac *= 1.3;
>     TmpMaxDel = TmpMaxDelFrac;
>      syslog(LOG_NOTICE,"DEL_LIMIT temporary increased to: %u/s",TmpMaxDel);
> }
>
> This allows the deletion rate to be increased at very fast rate (30%
> every 5 minutes) and there is no hard LIMIT, so the deletion rate keeps
> on going up until the server is overwhelmed and the deletions are
> consuming all the resources.
>
> Unless you are running out of space on the server, deletion from the
> Trash is a very, very low priority process and should only be happening
> AFTER normal reads/writes and replications. So it not necessary to ramp
> up the deletion rate in most cases.
>
> As I previously indicated, we have tested (and now are running in
> production on 3 clusters) the following replacement code:
>
> // Local version 09-23-2011
> if (delnotdone > deldone && delnotdone > prevdelnotdone) {
>     if (TmpMaxDelFrac < (MaxDel*2)) {
>     TmpMaxDelFrac *= 1.1;
>     TmpMaxDel = TmpMaxDelFrac;
>      syslog(LOG_NOTICE,"DEL_LIMIT temporary increased to: %u/s",TmpMaxDel);
>     }
> }
>
> This minor change limits the Deletion rate to a HARD LIMIT of 2x the
> CHUNK_DEL_LIMIT and only increases it by 10% each 5 minutes when its in
> ramp up phase.
>
> This is working very well for us. We are no longer terrified about
> deleting large folders and we don't care it it takes 6-8 hours to clear
> the post-trashtime deletion queue instead of the cluster being unusable
> for 1-2 hours.
>
> If we were to find that the number of post-trashtime files were growing
> to an unreasonably large level, then we would raise the rate for a
> limited time (probably in the evening when nothing else is going on) and
> take the performance hit (or add chunkservers/better equipment).
>
> So we would like to see a HARD_DEL_LIMIT in mfsmaster.cfg (instead of
> just assuming 2x DEL_LIMIT as in our example) and the ability to change
> those settings on the fly as you mentioned. (ideally via a cron job, so
> we could automatically speed things up a bit in the evenings).
>
> Further down our todo list would be some logic that makes the deletion
> rate subject to the other activity, so if the cluster is otherwise not
> busy doing read/writes and replications (as in our at night scenario)
> then it could go ahead and speed things up and conversely if the server
> is really busy, then postpone the post-trashtime deletions completely.
>
> The truncate to 0 idea sounds interesting if there is an actual
> performance gain and it doesn't introduce complications, but if
> mfsmaster was more intelligent about 'when' and 'how quickly' it deletes
> files from the trash queue, its not really high on our wishlist.
>
> Finally, I'd like to thank the maintainers for MFS.
>
> Now that we have the deletion issue solved and we learned not to let the
> mfsmaster process exceed 50% of RAM, MFS is a huge improvement over our
> NFS/DRBD setups in regards to administration and even the ability to use
> somewhat older servers in the cluster, allowing us to save the state of
> the art kit for databases and VM's.
>
> We've even had a few incidents where equipment failed or we did
> something stupid and we were able to recover cleanly. The process was
> well documented, easy to follow and 'just worked'.
>
> -bill
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2dcopy2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>



-- 
 - Davies

Re: [Moosefs-users] Why is it increasing my DEL_LIMIT when I don't want it to!

[<wk...@bn...>]

On 9/24/11 6:04 AM, Michał Borychowski wrote:
> Hi!
>
> We discussed similiar problem with Ólafur in July as far as I remember. Yes,
> we know it is not very optimal - if there are many files to be deleted
> system tries to increase the limit so that they get deleted, but
> unfortunately with realy huge number of files to be deleted, system stucks
> up... We have some ideas for improvement, eg. first doing truncate (setting
> to 0 bytes) and later do the real deleting. Also setting of this limit would
> be possible on the fly with mastertools.
>
>
>

What creates the Deletion Problem is this code in chunks.c

if (delnotdone > deldone && delnotdone > prevdelnotdone) {
     TmpMaxDelFrac *= 1.3;
     TmpMaxDel = TmpMaxDelFrac;
     syslog(LOG_NOTICE,"DEL_LIMIT temporary increased to: %u/s",TmpMaxDel);
}

This allows the deletion rate to be increased at very fast rate (30% 
every 5 minutes) and there is no hard LIMIT, so the deletion rate keeps 
on going up until the server is overwhelmed and the deletions are 
consuming all the resources.

Unless you are running out of space on the server, deletion from the 
Trash is a very, very low priority process and should only be happening 
AFTER normal reads/writes and replications. So it not necessary to ramp 
up the deletion rate in most cases.

As I previously indicated, we have tested (and now are running in 
production on 3 clusters) the following replacement code:

// Local version 09-23-2011
if (delnotdone > deldone && delnotdone > prevdelnotdone) {
     if (TmpMaxDelFrac < (MaxDel*2)) {
     TmpMaxDelFrac *= 1.1;
     TmpMaxDel = TmpMaxDelFrac;
     syslog(LOG_NOTICE,"DEL_LIMIT temporary increased to: %u/s",TmpMaxDel);
     }
}

This minor change limits the Deletion rate to a HARD LIMIT of 2x the 
CHUNK_DEL_LIMIT and only increases it by 10% each 5 minutes when its in 
ramp up phase.

This is working very well for us. We are no longer terrified about 
deleting large folders and we don't care it it takes 6-8 hours to clear 
the post-trashtime deletion queue instead of the cluster being unusable 
for 1-2 hours.

If we were to find that the number of post-trashtime files were growing 
to an unreasonably large level, then we would raise the rate for a 
limited time (probably in the evening when nothing else is going on) and 
take the performance hit (or add chunkservers/better equipment).

So we would like to see a HARD_DEL_LIMIT in mfsmaster.cfg (instead of 
just assuming 2x DEL_LIMIT as in our example) and the ability to change 
those settings on the fly as you mentioned. (ideally via a cron job, so 
we could automatically speed things up a bit in the evenings).

Further down our todo list would be some logic that makes the deletion 
rate subject to the other activity, so if the cluster is otherwise not 
busy doing read/writes and replications (as in our at night scenario) 
then it could go ahead and speed things up and conversely if the server 
is really busy, then postpone the post-trashtime deletions completely.

The truncate to 0 idea sounds interesting if there is an actual 
performance gain and it doesn't introduce complications, but if 
mfsmaster was more intelligent about 'when' and 'how quickly' it deletes 
files from the trash queue, its not really high on our wishlist.

Finally, I'd like to thank the maintainers for MFS.

Now that we have the deletion issue solved and we learned not to let the 
mfsmaster process exceed 50% of RAM, MFS is a huge improvement over our 
NFS/DRBD setups in regards to administration and even the ability to use 
somewhat older servers in the cluster, allowing us to save the state of 
the art kit for databases and VM's.

We've even had a few incidents where equipment failed or we did 
something stupid and we were able to recover cleanly. The process was 
well documented, easy to follow and 'just worked'.

-bill

Re: [Moosefs-users] Why is it increasing my DEL_LIMIT when I don't want it to!

[Michał B. <mic...@ge...>]

Hi!

We discussed similiar problem with Ólafur in July as far as I remember. Yes,
we know it is not very optimal - if there are many files to be deleted
system tries to increase the limit so that they get deleted, but
unfortunately with realy huge number of files to be deleted, system stucks
up... We have some ideas for improvement, eg. first doing truncate (setting
to 0 bytes) and later do the real deleting. Also setting of this limit would
be possible on the fly with mastertools.



Kind regards
Michał Borychowski 
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01


-----Original Message-----
From: WK [mailto:wk...@bn...] 
Sent: Friday, September 23, 2011 3:07 AM
To: moo...@li...
Subject: Re: [Moosefs-users] Why is it increasing my DEL_LIMIT when I don't
want it to!

Well I suppose it would be easy enough to grep the source code for 
'DEL_LIMIT temporary increase' and start commenting some things out for 
a quick fix.

However, I'd prefer if the maintainers addressed the issue with 
something more comprehensive and/or a flag for strict enforcement of the 
DEL_LIMIT setting or the current setup which is obviously some sort of 
'oh no, we have a LOT of files we need to work through and need more 
resources' logic.

There may also be some sort of reason they do this, such as some 
resource issue.

We will see what they have to say. Maybe its already addressed in the 
next version.

-bill


On 9/22/2011 2:42 AM, Ólafur Ósvaldsson wrote:
> We have the exact same problem, chunk deletions have caused problems in
the past and we have DEL_LIMIT set at 5, but mfsmaster increases it to 40-50
right away and sometimes goes to 70/s
>
> This is also the case for chunk replications, it does not seem to honor
the mfsmaster.cfg settings, although that does not get logged.
>
> /Oli
>
> On 22.9.2011, at 05:03, wk...@bn... wrote:
>
>> Ok, we deleted a couple hundred thousand files from a large Maildir
>> folder set.
>>
>> We have had problems with deletions overwhelming the cluster in the
>> past, so we have a DEL_LIMIT set to 20 (which we will probably lower)
>>
>> But when the expiretime hit, the server became lethargic. in checking
>> the logs I see this
>>
>> Sep 21 21:10:31 mfs1master mfsmaster[2373]: DEL_LIMIT temporary
>> increased to: 26/s
>> Sep 21 21:15:30 mfs1master mfsmaster[2373]: DEL_LIMIT temporary
>> increased to: 33/s
>> Sep 21 21:55:24 mfs1master mfsmaster[2373]: DEL_LIMIT decreased back to:
>> 26/s
>>
>> OK, WHY IS DOING THIS! I told it no more than 20
>>
>> I do NOT want this, it kills my server and we've learned the hard way it
>> can really screw up VM images (they go read-only) if the deletions
>> overwhelm the cluster.
>>
>>
>> -bill
>>
>>
>>
>>
----------------------------------------------------------------------------
--
>> All the data continuously generated in your IT infrastructure contains a
>> definitive record of customers, application performance, security
>> threats, fraudulent activity and more. Splunk takes this data and makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2dcopy1
>> _______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> --
> Ólafur Osvaldsson
> System Administrator
> Nethonnun ehf.
> e-mail: osv...@ne...
> phone: +354 517 3400
>
>
>
----------------------------------------------------------------------------
--
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2dcopy1
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users


----------------------------------------------------------------------------
--
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Release

[Michał B. <mic...@ge...>]

Hi!

 

Yes, next release is expected to be Q4-2011 and of course MooseFS is in development :) We have some initialization time improvements, rewritten init scripts, improved tolerance for damaged changelog files and some bug fixes. So please be patient just a little bit more :)

 

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

From: Kristofer Pettijohn [mailto:kri...@cy...] 
Sent: Friday, September 23, 2011 8:15 PM
To: moo...@li...
Subject: [Moosefs-users] Release

 

I know that you guys can't given an exact date, but when is the next release expected?  Q4-2011?  It's been a long time, starting to wonder if MFS is still in development :)

Re: [Moosefs-users] max filesystem size?

[Michał B. <mic...@ge...>]

Hi!

 

Generally speaking there is no limit - if you have enough RAM in the master
server to handle the metadata. There are some installations of one petabyte
in size (see http://www.moosefs.org/who-is-using-moosefs.html). Of course,
if you have many files (hundreds of millions) and lots of file operations
per second the system will hung up (take into consideration also disk speeds
and network performance).

 

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

From: Brian Pontz [mailto:axe...@ya...] 
Sent: Friday, September 23, 2011 8:53 PM
To: moo...@li...
Subject: [Moosefs-users] max filesystem size?

 

Hi, 

I've looked around some and I see moosefs has a max filesize of about 2TB.
But, I am unable to find what the max file system size is. Can someone
please tell me?

thanks, 
Brian 

Re: [Moosefs-users] Auto MetaRestore

[Michał B. <mic...@ge...>]

Hi!

Of course you may do it but we recommend recovering the master server
manually - it is wise to observe the mfsmetarestore process. The automatic
recovery process is quite complicated and until some better scripts are
prepared, you'd rather recover it manually.


Kind regards
Michal


-----Original Message-----
From: Alexander Akhobadze [mailto:akh...@ri...] 
Sent: Friday, August 26, 2011 12:51 PM
To: moo...@li...
Subject: [Moosefs-users] Auto MetaRestore

Hi All!

I wonder if it is a good idea to put in init/rc script
which starts mfsmaster something like

mfs_data_path=`grep DATA_PATH /etc/mfsmaster.cfg | grep -v ^# | awk '{print
$NF}'`
if [ ! -f $mfs_data_path/metadata.mfs ]
then
    mfsmetarestore -a -d $mfs_data_path
fi

???????
In my opinion it is bare necessity to automatically make MooseFS healthy
in case of power failure/restore.

wbr
Alexander


----------------------------------------------------------------------------
--
EMC VNX: the world's simplest storage, starting under $10K
The only unified storage solution that offers unified management 
Up to 160% more powerful than alternatives and 25% more efficient. 
Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Data deduplication

[Michał B. <mic...@ge...>]

Hi!

 

We talk among us about this feature from time to time and our decision is not to implement it for the moment. There are much more important things to do about MooseFS (like quota, acls, RAM optimizations, optimization of deleting files process, making snapshots even better, etc.) that file deduplication is not that important. We know it is loss of space but nowadays it's not that expensive. So maybe one time we come back to this, but unfortunately not in the near future.

 

 

Kind regards

Michał Borychowski

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

 

From: Florent Bautista [mailto:fl...@co...] 
Sent: Wednesday, August 31, 2011 4:54 PM
To: moo...@li...
Subject: [Moosefs-users] Data deduplication

 

Hi all,

I would like to know if MooseFS will (in which future ?) take care about data deduplication.

I know that MooseFS makes a checksum of every chunk, so would it be possible to have data deduplication at that level ?
If two (or more) files with goal=3 each, have chunk(s) in common, only store 3 copies of that (those) chunk(s) and not 6 (or more) like today...

For files having different goals, use the more important goal.

I think it does not change the architecture of MooseFS... maybe the problem is that MFS Master do not know about checksums, it is made on CS... but we could find a way to go through

Is it difficult to add that feature ? What do you think about it ?

Thank you guys!

-- 


Florent Bautista

  _____  

Ce message et ses éventuelles pièces jointes sont personnels, confidentiels et à l'usage exclusif de leur destinataire.
Si vous n'êtes pas la personne à laquelle ce message est destiné, veuillez noter que vous avez reçu ce courriel par erreur et qu'il vous est strictement interdit d'utiliser, de diffuser, de transférer, d'imprimer ou de copier ce message.

This e-mail and any attachments hereto are strictly personal, confidential and intended solely for the addressee.
If you are not the intended recipient, be advised that you have received this email in error and that any use, dissemination, forwarding, printing, or copying of this message is strictly prohibited.

  _____