Re: [MooseFS-Users] Real experiences from real users

[Marco M. <mar...@gm...>]

On 05/22/2018 08:23 AM, Wilson, Steven M wrote:
> 
> ________________________________________
> From: Jakub Kruszona-Zawadzki <jak...@ge...>
> Sent: Tuesday, May 22, 2018 1:55 AM
> To: Gandalf Corvotempesta
> Cc: Wilson, Steven M; moo...@li...
> Subject: Re: [MooseFS-Users] Real experiences from real users
> 
>> On 21 May, 2018, at 17:57, Gandalf Corvotempesta <gan...@gm...> wrote:
>>
>> Il giorno lun 21 mag 2018 alle ore 15:16 Wilson, Steven M
>> <st...@pu...>
>> ha scritto:
>>> This might be a good use case for implementing ZFS with compression
>> enabled.  For one of my MooseFS installations where we have millions of
>> typically very small text files, I started using ZFS with compression as
>> the underlying file system and I'm seeing a 4X or 5X compression ratio.  I
>> basically use ZFS only for its compression feature since I am only using
>> one ZFS file system per disk (i.e., no raid-z*, etc.).
>>
>> Do you use any ZIL or L2ARC on SSDs ?
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> 
> 
> Are you sure that using compression in underlying system gives you such ratio?
> 
> Once I was considering adding compression on the chunkserver layer (just to compress chunks not modified in X days by chunkserver). This is quite simple to implement, but we made tests in our cluster and have found that only 1/10th of chunks are "compressible" and the ratio was not so good (about 2x).
> 
> --
> Regards,
> Jakub Kruszona-Zawadzki
> - - - - - - - - - - - - - - - -
> Segmentation fault (core dumped)
> Phone: +48 602 212 039
> 
> ________________________________________
> 
> Yes, this is the compression ratio reported by ZFS.  Of course it all depends on the type of data being stored.  In the case of this particular storage system, the majority of the files are text files making it a great candidate for compression.  My other installations, though, are similar to your experience and they only show about 1.25X compression ratio on average.  But that is still enough to be worth considering the use of compression in my opinion.
> 
> Steve
> 

What is wrong with saying that if you want compression use ZFS and turn on compression?
ZFS is available on both Linux and FreeBSD.
Why does compression need to be on the MooseFS level ?

-- Marco

> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>