Re: [MooseFS-Users] Min goal to prevent data loss

[Piotr R. K. <pio...@mo...>]

One chunk minimum size is 64 kB + 5kB MFS chunk header (so 69kB)
maximum chunk size is 64MB + 5kB header

If file size exceeds 64MB (and does not exceed 128MB), the file has 2 chunks, if >128MB and <192MB: 3 chunks and so on.


BR
Peter

-- 
Piotr Robert Konopelko
MooseFS Client Support Team | moosefs.com <https://moosefs.com/>
> On 12 May 2017, at 5:12 PM, Alex Crow <ac...@in...> wrote:
> 
> Hi,
> 
> If you have a lot of very small files there will be some overhead as there is a minimum chunk size. Used space also will vary depending on your underlying FS. I use ZFS with compression so I see relatively less overhead.
> Alex
> 
> On 12/05/17 16:02, David Myer wrote:
>> Thanks Piotr and Wolfgang.
>> 
>> I have interpreted the term "copy" incorrectly. I thought a copy was an additional instance of the file, so a goal of 2 would result in the file plus 2 copies of it, thus 3 instances of the file.
>> 
>> I also thought this because it explained the size of my mfs mount - I have rsync'd ~500GB of data into my mfs mount, and the CGI stats show ~1.5TB of disk used at a goal of 2. Based off this I assumed each file had 3 instances. What would account for 1.5TB of storage usage if 500GB of files only exist twice at a goal of 2?
>> 
>> Thanks,
>> Dave
>> 
>> 
>>> -------- Original Message --------
>>> Subject: Re: [MooseFS-Users] Min goal to prevent data loss
>>> Local Time: May 12, 2017 10:31 AM
>>> UTC Time: May 12, 2017 2:31 PM
>>> From: pio...@mo... <mailto:pio...@mo...>
>>> To: David Myer <dav...@pr...> <mailto:dav...@pr...>
>>> moo...@li... <mailto:moo...@li...> <moo...@li...> <mailto:moo...@li...>
>>> 
>>> Hi David,
>>> 
>>> > Wouldn't a file exist in 2 places if the goal was 1?
>>> 
>>> No. If file had goal = 1, it had only one copy.
>>> 
>>> Goal means number of copies, so the best practice is to set the goal to 2 minimum for whole filesystem, e.g.:
>>> mfssetgoal -r 2 /mnt/mfs
>>> 
>>> MooseFS will then automatically take care of replication of files to other servers to fulfil the goal.
>>> 
>>> 
>>> Best regards,
>>> Peter
>>> 
>>> -- 
>>> Piotr Robert Konopelko
>>> MooseFS Client Support Team | moosefs.com
>>> 
>>> > On 12 May 2017, at 3:54 PM, David Myer <dav...@pr...> <mailto:dav...@pr...> wrote:
>>> > 
>>> > Hi,
>>> > 
>>> > The MooseFS best practices recommend a min goal of 2 to prevent eventual data loss. Wouldn't a file exist in 2 places if the goal was 1?
>>> > 
>>> > Thanks,
>>> > Dave
>>> > 
>>> > ------------------------------------------------------------------------------
>>> > Check out the vibrant tech community on one of the world's most
>>> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot_________________________________________ <http://sdm.link/slashdot_________________________________________>
>>> > moosefs-users mailing list
>>> > moo...@li... <mailto:moo...@li...>
>>> > https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>
>>> 
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot <http://sdm.link/slashdot>
>> 
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li... <mailto:moo...@li...>
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>
> 
> 
> --
> This message is intended only for the addressee and may contain
> confidential information. Unless you are that person, you may not
> disclose its contents or use it in any way and are requested to delete
> the message along with any attachments and notify us immediately.
> This email is not intended to, nor should it be taken to, constitute advice.
> The information provided is correct to our knowledge & belief and must not
> be used as a substitute for obtaining tax, regulatory, investment, legal or
> any other appropriate advice.
> 
> "Transact" is operated by Integrated Financial Arrangements Ltd.
> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
> (Registered office: as above; Registered in England and Wales under
> number: 3727592). Authorised and regulated by the Financial Conduct
> Authority (entered on the Financial Services Register; no. 190856).
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot_________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users