Re: [Moosefs-users] chunkserver can't connect to masterserver

[l. <lwx...@16...>]

thanks for you help!

I think,Why not make the "block size of 64KiB" be a Parameter of the MFS system;  there will result of less waste of space for " small file ";
Many "hard-coded " values In MFS Can Changed to be a parameter of config file, So the system will be more flexible;

Regards
lwxian_aha 

2010-05-14 



lwxian_aha 



发件人： Michał_Borychowski 
发送时间： 2010-05-14  13:34:12 
收件人： 'lwxian_aha' 
抄送： 'moosefs-users' 
主题： RE: [Moosefs-users] chunkserver can't connect to masterserver 
 
 
From: lwxian_aha [mailto:lwx...@16...] 
Sent: Thursday, May 13, 2010 8:22 AM
To: Michał_Borychowski
Cc: moosefs-users
Subject: Re: [Moosefs-users] chunkserver can't connect to masterserver
 
with you suggestions, I have fixes the problem.thanks!!!
[MB] We are very happy that the patch has helped you!
   
On my product system ,there is a lots of small files (less then 10K), so there are about 5,000,000 chunks with only 1.4T DATA;
I make suggestions :
1, the first way   :you can separate the "registration packet " for serveral times ,every time limit to 500,000; so we will have no limit with the amount of chunks every chunkserver;
[MB] That’s what we thought of. Thank you for the suggestion.
 
2, the second way :  maybe you can store several files in one chunks(every chunk limits to 64M)? so you can reduce the amount of chunks?
[MB] It looks interesting but in case of a modification of a file in “the middle” of the chunk it would bring a big problem
 
 
Here ,I have another question:
on my chunkserver disk,I have see the smallest file of chunks is 70656, and my smallest source file is only about 4K; why the chunk file size is substantially exceeds the soursefile;
[MB] The system was initially designed for keeping large amounts (like several thousands) of very big files (of tens of gigabytes) and has a hard-coded chunk size of 64MiB and block size of 64KiB. That’s why even a small file will occupy 64KiB plus additionally 4KiB of checksums and 1KiB for the header. The whole transfer which takes place in the system is done in blocks of 64KiB. However it doesn’t have any impact on the performance (in a normal filesystem it is also usual to read ahead some superfluous data).
 
The issue regarding the occupied space is really more significant, but in our opinion it is still negligible. Let’s take 25 million files with a goal set to 2, so we have about 50 million of “last” chunks. If in every chunk we lose even 64KiB we will have an overall waste of 3.2TB which nowadays should not be a very big concern.
 
In your case 10,000 files with goal=2 would produce maximally 1.2 GB of wasted space. It is still perfect to keep source files on the MooseFS system if they are going to be served somewhere or just stored and developed there. 
 
If you need any further assistance please let us know.
 
Regards
Michał Borychowski 
 
 
thanks !!
 
2010-05-13 



lwxian_aha