moosefs-users — Mailing list for MooseFS Users

Re: [MooseFS-Users] Real experiences from real users

[Gandalf C. <gan...@gm...>]

Il giorno mar 22 mag 2018 alle ore 21:00 Marco Milano <mar...@gm...>
ha scritto:
> Assuming that you have minimum of 2 copies in MooseFS, it will read,
detect
> and read from second copy and will heal the first copy.
> So, I don't know what you mean exactly by "does the same" but
> it is not the *same*

You are right, in this case, MooseFS is better because is able to
recontruct the chunk, ZFS don't.

> MooseFS will be *aware* of the corruption during the read and will self
heal
> as I explained above. (Or during the checksum checking (native scrub)
loop,
> whichever comes first.)

Yes, excactly, that's the problem.
Only during a read or a (very, very, very) slow scrub.
ZFS scrub is much faster, but is useless without a RAID.

> You seem to be making these constant claims about "native scrub taking
months",
> but I believe it was explained in earlier emails that this will depend on
your
> hardware configuration. I believe there was another email which basically
said
> this "native scrub speed" was much improved in version 4.
> So I think it is fair to say that you should stop repeating this "native
scrub takes months" claim,
> or if you are not going to stop repeating it, at least put some
qualifiers around it.

Probably is unclear to me how to calculate the chunk loop speed. It's
unclear even with LizardFS (is almost the same).
Let's see:

15.000.000 chunks.

HDD_TEST_FREQ is the frequency for 1 chunk check. Default to 1 seconds.
Lowering it to 1 (will destroy cluster performance, already tried) means 1
full scrub every
15.000.000 seconds.

15.000.000 seconds = 174 days = 6 months.

> Or download v4, and see if the speed improved...

Testing right now....
There is a HDD_TEST_SPEED in MB/s, this could be useful

Re: [MooseFS-Users] Real experiences from real users

[Joe L. <jo...@ge...>]

On May 22, 2018, at 1:59 PM, Marco Milano <mar...@gm...> wrote:
> 
> 
> On 05/22/2018 02:36 PM, Gandalf Corvotempesta wrote:
>> Il giorno mar 22 mag 2018 alle ore 19:28 Marin Bernard <li...@ol...>
>> ha scritto:
>> Anyway, even if you scrub the whole ZFS pool, you won't get any advantage,
>> ZFS is unable
>> to recover by itself (without raid) and MooseFS is still unaware of
>> corruption.
> 
> MooseFS will be *aware* of the corruption during the read and will self heal
> as I explained above. (Or during the checksum checking (native scrub) loop,
> whichever comes first.)
> 
>> Ok, chunk1 is corrupted, ZFS detected it during a scrub. And now ?
>> ZFS doesn't have any replica to rebuild from.
>> MooseFS is unaware of this because their native scrub takes months and
>> no one is reading that file from a client (forcing the checksum
>> verification).
> 
> You seem to be making these constant claims about "native scrub taking months",
> but I believe it was explained in earlier emails that this will depend on your
> hardware configuration. I believe there was another email which basically said
> this "native scrub speed" was much improved in version 4.
> So I think it is fair to say that you should stop repeating this "native scrub takes months" claim,
> or if you are not going to stop repeating it, at least put some qualifiers around it.
> Or download v4, and see if the speed improved…

It sounds like MooseFS does data consistency checking similarly to ZFS.
They both seem to do periodic scrubs, and they both check the block they read/write.

It seems like the complaint is that ZFS can’t tell MooseFS that it found a bad block.

-Joe

Re: [MooseFS-Users] Real experiences from real users

[Marco M. <mar...@gm...>]

On 05/22/2018 02:36 PM, Gandalf Corvotempesta wrote:
> Il giorno mar 22 mag 2018 alle ore 19:28 Marin Bernard <li...@ol...>
> ha scritto:
>> So does Proxmox VE.
> 
> Not all server are using proxmox.
> Proxmox repackage ZFS on every release, because they support it.
> If you have to mantain multiple different system, using DKMS is more prone
> to error
> than without. A small kernel upgrade could break everything.
> 
>> That's a myth. ZFS never required ECC RAM, and I run it on boxes with
>> as little as 1GB RAM. Every bit of it can be tuned, including the size
>> of the ARC.
> 
> Is not a myth. Is the truth. ECC RAM is not required to run ZFS, but you
> won't be sure
> that what are you writing to disks (and checksumming) is exactly the same
> you received.
> 
> In other words, without ECC RAM you could experience in memory data
> corruption and then
> you will write corrupted data (with a proper checksum), so that ZFS will
> reply with corrupted data.
> 
> ECC is not mandatory, but highly suggested.
> Without ECC you'll fix the bit-rot, but you are still subject to in-memory
> corruption,
> so, the original issue (data corruption) is still unfixed and ZFS can't do
> nothing if data is
> corrupted before ZFS.
> 
>> Checksumming and duplication (ditto blocks) of pool metadata are NOT
>> provided by the master. This is a much appreciated feature when you
>> come from an XFS background where a single urecoverable read can crash
>> an entire filesystem. I've been there before; never ever!
> 
> At which pool metadata are you referring to ?
> Anyway, I hate XFS :-) I had multiple failures......
> 
>> MooseFS background verification may take months to check the whole
>> dataset
> 
> True.
> 
>> ZFS does scrub a whole chunkserver within a few hours, with
>> adaptive, tunable throughput to minimize the impact on the cluster.
> 
> Is not the same.
> When ZFS detect a corruption, it does nothing without a RAID. it simply
> discard data
> during a read. But if you are reading a file, MooseFS will check the
> checksum automatically
> and does the same.

Assuming that you have minimum of 2 copies in MooseFS, it will read, detect
and read from second copy and will heal the first copy.
So, I don't know what you mean exactly by "does the same" but
it is not the *same*


> 
> Anyway, even if you scrub the whole ZFS pool, you won't get any advantage,
> ZFS is unable
> to recover by itself (without raid) and MooseFS is still unaware of
> corruption.

MooseFS will be *aware* of the corruption during the read and will self heal
as I explained above. (Or during the checksum checking (native scrub) loop,
whichever comes first.)

> 
> Ok, chunk1 is corrupted, ZFS detected it during a scrub. And now ?
> ZFS doesn't have any replica to rebuild from.
> MooseFS is unaware of this because their native scrub takes months and
> no one is reading that file from a client (forcing the checksum
> verification).

You seem to be making these constant claims about "native scrub taking months",
but I believe it was explained in earlier emails that this will depend on your
hardware configuration. I believe there was another email which basically said
this "native scrub speed" was much improved in version 4.
So I think it is fair to say that you should stop repeating this "native scrub takes months" claim,
or if you are not going to stop repeating it, at least put some qualifiers around it.
Or download v4, and see if the speed improved...

> So? Which solution do you have ?
> 

Re: [MooseFS-Users] Real experiences from real users

[Gandalf C. <gan...@gm...>]

Il giorno mar 22 mag 2018 alle ore 19:28 Marin Bernard <li...@ol...>
ha scritto:
> So does Proxmox VE.

Not all server are using proxmox.
Proxmox repackage ZFS on every release, because they support it.
If you have to mantain multiple different system, using DKMS is more prone
to error
than without. A small kernel upgrade could break everything.

> That's a myth. ZFS never required ECC RAM, and I run it on boxes with
> as little as 1GB RAM. Every bit of it can be tuned, including the size
> of the ARC.

Is not a myth. Is the truth. ECC RAM is not required to run ZFS, but you
won't be sure
that what are you writing to disks (and checksumming) is exactly the same
you received.

In other words, without ECC RAM you could experience in memory data
corruption and then
you will write corrupted data (with a proper checksum), so that ZFS will
reply with corrupted data.

ECC is not mandatory, but highly suggested.
Without ECC you'll fix the bit-rot, but you are still subject to in-memory
corruption,
so, the original issue (data corruption) is still unfixed and ZFS can't do
nothing if data is
corrupted before ZFS.

> Checksumming and duplication (ditto blocks) of pool metadata are NOT
> provided by the master. This is a much appreciated feature when you
> come from an XFS background where a single urecoverable read can crash
> an entire filesystem. I've been there before; never ever!

At which pool metadata are you referring to ?
Anyway, I hate XFS :-) I had multiple failures......

> MooseFS background verification may take months to check the whole
> dataset

True.

> ZFS does scrub a whole chunkserver within a few hours, with
> adaptive, tunable throughput to minimize the impact on the cluster.

Is not the same.
When ZFS detect a corruption, it does nothing without a RAID. it simply
discard data
during a read. But if you are reading a file, MooseFS will check the
checksum automatically
and does the same.

Anyway, even if you scrub the whole ZFS pool, you won't get any advantage,
ZFS is unable
to recover by itself (without raid) and MooseFS is still unaware of
corruption.

Ok, chunk1 is corrupted, ZFS detected it during a scrub. And now ?
ZFS doesn't have any replica to rebuild from.
MooseFS is unaware of this because their native scrub takes months and
no one is reading that file from a client (forcing the checksum
verification).
So? Which solution do you have ?

Re: [MooseFS-Users] MooseFS 4.0 and new features

[Devin A. <lin...@gm...>]

I think it would be really NEAT if you actually did create a MooseFS driver to use with QEMU. I think the more options available the better. :)


--
Devin Acosta RHCA|LFCE
Red Hat Certified Architect
e: de...@li...oud
p: 602-354-1220
On May 21, 2018, 11:06 PM -0700, Jakub Kruszona-Zawadzki <jak...@ge...>, wrote:
>
> > On 20 May, 2018, at 16:00, Gandalf Corvotempesta <gan...@gm...> wrote:
> >
> > > Il dom 20 mag 2018, 15:56 Zlatko Čalušić <zca...@bi...> ha scritto:
> > > > Even if LizardFS currently has a feature or two missing in current
> > > > MooseFS 3.0, it seems that MooseFS 4.0 will be a clear winner, and
> > > > definitely worth a wait!
> >
> > Totally agree
> > Probably the real missing thing, IMHO, is the qemu driver
>
> I've checked qemu today - there is no such thing as "qemu driver". They have support for glusterfs - but this is part of qemu code.
>
> I probably could write similar code for qemu to integrate qemu with MFS, but I'm not sure if it would be accepted by qemu programmers.
>
> --
> Regards,
> Jakub Kruszona-Zawadzki
> - - - - - - - - - - - - - - - -
> Segmentation fault (core dumped)
> Phone: +48 602 212 039
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [MooseFS-Users] Real experiences from real users

[Marin B. <li...@ol...>]

> Il giorno mar 22 mag 2018 alle ore 14:39 Marco Milano <marco.milano@g
> mx.com>
> ha scritto:
> > What is wrong with saying that if you want compression use ZFS and
> > turn
> 
> on compression?
> > ZFS is available on both Linux and FreeBSD.
> > Why does compression need to be on the MooseFS level ?
> 
> ZFS on Linux is not "native", it's an external project. It require a
> DKMS
> or Ubuntu, that provide an already built module as default.

So does Proxmox VE.

Nothing is "native" on Linux. Any distro is a sum of external projects.
Btrfs used to be such an external project. The only reason why ZFS is
not included in most distros is because it is released under the CDDL
license, which is not compatible with the GPL.

> If you only need compression, forcing users to use ZFS is like
> shooting
> fish in barrel.
> ZFS requires tons of (ECC) RAM that could be used for something else 

That's a myth. ZFS never required ECC RAM, and I run it on boxes with
as little as 1GB RAM. Every bit of it can be tuned, including the size
of the ARC.

> (ie,
> putting mfsmaster on chunkservers),
> and, on average, is slower than other FS due to checksumming (already
> done
> by MooseFS, thus it's a
> redundant check)
> 

Checksumming and duplication (ditto blocks) of pool metadata are NOT
provided by the master. This is a much appreciated feature when you
come from an XFS background where a single urecoverable read can crash
an entire filesystem. I've been there before; never ever!

MooseFS background verification may take months to check the whole
dataset. ZFS does scrub a whole chunkserver within a few hours, with
adaptive, tunable throughput to minimize the impact on the cluster.

Speed is a very complex topic. ZFS is actually faster than most non-COW 
file systems in sequential I/O, but less efficient in other scenarios.
The same story goes on with btrfs. This has nothing to do with
checksumming, especially with recent AES-NI-capable chips with hardware
checksum offloading.

Low latency is what the ARC was designed for, and where is shines. It
can also be tuned to prioritize speed if this is wished. A chunkserver
with enough RAM is able to serve recently requested chunks at the speed
of light. And as a bonus, the ARC tries to make the best use of memory
by caching compressed data.

> I'm not against ZFS (probably i'll use ZFS in my cluster), but if the
> only
> used feature is compression, having it in MooseFS is not a bad idea.
> 

There are a lot of benefits in using ZFS. People wrote entire books
about it. Of course, it may not always be the best answer: there are
always those few edge cases where it's not worth it, but there's not a
lot of them.

Personally, I no longer wonder why I should use it; I rather try to
find reasons to go without it. If there are none, why not use it ?

> In this case, MooseFS will be really hardware and software
> indipendent.
> 
> Obviously, i'm talking after reading that adding compression "is
> quite
> simple to implement".
> 
> -------------------------------------------------------------------
> -----------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [MooseFS-Users] Real experiences from real users

[Gandalf C. <gan...@gm...>]

Il giorno mar 22 mag 2018 alle ore 14:39 Marco Milano <mar...@gm...>
ha scritto:
> What is wrong with saying that if you want compression use ZFS and turn
on compression?
> ZFS is available on both Linux and FreeBSD.
> Why does compression need to be on the MooseFS level ?

ZFS on Linux is not "native", it's an external project. It require a DKMS
or Ubuntu, that provide an already built module as default.
If you only need compression, forcing users to use ZFS is like shooting
fish in barrel.
ZFS requires tons of (ECC) RAM that could be used for something else (ie,
putting mfsmaster on chunkservers),
and, on average, is slower than other FS due to checksumming (already done
by MooseFS, thus it's a
redundant check)

I'm not against ZFS (probably i'll use ZFS in my cluster), but if the only
used feature is compression, having it in MooseFS is not a bad idea.

In this case, MooseFS will be really hardware and software indipendent.

Obviously, i'm talking after reading that adding compression "is quite
simple to implement".

Re: [MooseFS-Users] Real experiences from real users

[Marco M. <mar...@gm...>]

On 05/22/2018 08:23 AM, Wilson, Steven M wrote:
> 
> ________________________________________
> From: Jakub Kruszona-Zawadzki <jak...@ge...>
> Sent: Tuesday, May 22, 2018 1:55 AM
> To: Gandalf Corvotempesta
> Cc: Wilson, Steven M; moo...@li...
> Subject: Re: [MooseFS-Users] Real experiences from real users
> 
>> On 21 May, 2018, at 17:57, Gandalf Corvotempesta <gan...@gm...> wrote:
>>
>> Il giorno lun 21 mag 2018 alle ore 15:16 Wilson, Steven M
>> <st...@pu...>
>> ha scritto:
>>> This might be a good use case for implementing ZFS with compression
>> enabled.  For one of my MooseFS installations where we have millions of
>> typically very small text files, I started using ZFS with compression as
>> the underlying file system and I'm seeing a 4X or 5X compression ratio.  I
>> basically use ZFS only for its compression feature since I am only using
>> one ZFS file system per disk (i.e., no raid-z*, etc.).
>>
>> Do you use any ZIL or L2ARC on SSDs ?
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> 
> 
> Are you sure that using compression in underlying system gives you such ratio?
> 
> Once I was considering adding compression on the chunkserver layer (just to compress chunks not modified in X days by chunkserver). This is quite simple to implement, but we made tests in our cluster and have found that only 1/10th of chunks are "compressible" and the ratio was not so good (about 2x).
> 
> --
> Regards,
> Jakub Kruszona-Zawadzki
> - - - - - - - - - - - - - - - -
> Segmentation fault (core dumped)
> Phone: +48 602 212 039
> 
> ________________________________________
> 
> Yes, this is the compression ratio reported by ZFS.  Of course it all depends on the type of data being stored.  In the case of this particular storage system, the majority of the files are text files making it a great candidate for compression.  My other installations, though, are similar to your experience and they only show about 1.25X compression ratio on average.  But that is still enough to be worth considering the use of compression in my opinion.
> 
> Steve
> 

What is wrong with saying that if you want compression use ZFS and turn on compression?
ZFS is available on both Linux and FreeBSD.
Why does compression need to be on the MooseFS level ?

-- Marco

> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> 

Re: [MooseFS-Users] Real experiences from real users

[Marco M. <617...@gm...>]

On 05/22/2018 08:23 AM, Wilson, Steven M wrote:
> 
> ________________________________________
> From: Jakub Kruszona-Zawadzki <jak...@ge...>
> Sent: Tuesday, May 22, 2018 1:55 AM
> To: Gandalf Corvotempesta
> Cc: Wilson, Steven M; moo...@li...
> Subject: Re: [MooseFS-Users] Real experiences from real users
> 
>> On 21 May, 2018, at 17:57, Gandalf Corvotempesta <gan...@gm...> wrote:
>>
>> Il giorno lun 21 mag 2018 alle ore 15:16 Wilson, Steven M
>> <st...@pu...>
>> ha scritto:
>>> This might be a good use case for implementing ZFS with compression
>> enabled.  For one of my MooseFS installations where we have millions of
>> typically very small text files, I started using ZFS with compression as
>> the underlying file system and I'm seeing a 4X or 5X compression ratio.  I
>> basically use ZFS only for its compression feature since I am only using
>> one ZFS file system per disk (i.e., no raid-z*, etc.).
>>
>> Do you use any ZIL or L2ARC on SSDs ?
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> 
> 
> Are you sure that using compression in underlying system gives you such ratio?
> 
> Once I was considering adding compression on the chunkserver layer (just to compress chunks not modified in X days by chunkserver). This is quite simple to implement, but we made tests in our cluster and have found that only 1/10th of chunks are "compressible" and the ratio was not so good (about 2x).
> 
> --
> Regards,
> Jakub Kruszona-Zawadzki
> - - - - - - - - - - - - - - - -
> Segmentation fault (core dumped)
> Phone: +48 602 212 039
> 
> ________________________________________
> 
> Yes, this is the compression ratio reported by ZFS.  Of course it all depends on the type of data being stored.  In the case of this particular storage system, the majority of the files are text files making it a great candidate for compression.  My other installations, though, are similar to your experience and they only show about 1.25X compression ratio on average.  But that is still enough to be worth considering the use of compression in my opinion.
> 
> Steve

What is wrong with saying that if you want compression use ZFS and turn on compression?
ZFS is available on both Linux and FreeBSD.
Why does compression need to be on the MooseFS level ?

-- Marco

> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> 

Re: [MooseFS-Users] Real experiences from real users

[Wilson, S. M <st...@pu...>]

________________________________________
From: Jakub Kruszona-Zawadzki <jak...@ge...>
Sent: Tuesday, May 22, 2018 1:55 AM
To: Gandalf Corvotempesta
Cc: Wilson, Steven M; moo...@li...
Subject: Re: [MooseFS-Users] Real experiences from real users

> On 21 May, 2018, at 17:57, Gandalf Corvotempesta <gan...@gm...> wrote:
>
> Il giorno lun 21 mag 2018 alle ore 15:16 Wilson, Steven M
> <st...@pu...>
> ha scritto:
>> This might be a good use case for implementing ZFS with compression
> enabled.  For one of my MooseFS installations where we have millions of
> typically very small text files, I started using ZFS with compression as
> the underlying file system and I'm seeing a 4X or 5X compression ratio.  I
> basically use ZFS only for its compression feature since I am only using
> one ZFS file system per disk (i.e., no raid-z*, etc.).
>
> Do you use any ZIL or L2ARC on SSDs ?
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users


Are you sure that using compression in underlying system gives you such ratio?

Once I was considering adding compression on the chunkserver layer (just to compress chunks not modified in X days by chunkserver). This is quite simple to implement, but we made tests in our cluster and have found that only 1/10th of chunks are "compressible" and the ratio was not so good (about 2x).

--
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

________________________________________

Yes, this is the compression ratio reported by ZFS.  Of course it all depends on the type of data being stored.  In the case of this particular storage system, the majority of the files are text files making it a great candidate for compression.  My other installations, though, are similar to your experience and they only show about 1.25X compression ratio on average.  But that is still enough to be worth considering the use of compression in my opinion.

Steve

Re: [MooseFS-Users] Real experiences from real users

[Gandalf C. <gan...@gm...>]

Il giorno mar 22 mag 2018 alle ore 07:46 Jakub Kruszona-Zawadzki <
jak...@ge...> ha scritto:
> Are we still talking about MFS or LizardFS?

MooseFS.

> In MooseFS csid has been implemented in version 2.x. Master recognizes
chunkservers using their ID not IP.

> Chunkserver also remembers unique ID of MFS instance (master) and will
not talk to other masters.

So, what happens when spinning up an old chunkserver that was already
connected to that master, with an already used IP ?

If I understood correctly, the issue is not chunkserver talking to a wrong
master, but master accepting a chunkserver with outdated data AND with a
conflicting IP.

Re: [MooseFS-Users] MooseFS 4.0 and new features

[Gandalf C. <gan...@gm...>]

Il giorno mar 22 mag 2018 alle ore 08:05 Jakub Kruszona-Zawadzki <
jak...@ge...> ha scritto:
> I've checked qemu today - there is no such thing as "qemu driver". They
have support for glusterfs - but this is part of qemu code.

> I probably could write similar code for qemu to integrate qemu with MFS,
but I'm not sure if it would be accepted by qemu programmers.

I don't see any reasons why they won't accept your qemu block driver.
They have tons of block drivers, even for software with much less features
than MooseFS, like sheepdog.

Yes, qemu doesn't have "loadable modules", every block driver is compiled
directly, but I'm sure that if you make a pull-request on latest master,
they will accept.

Re: [MooseFS-Users] MooseFS 4.0 and new features

[Jakub Kruszona-Z. <jak...@ge...>]

> On 20 May, 2018, at 16:00, Gandalf Corvotempesta <gan...@gm...> wrote:
> 
> Il dom 20 mag 2018, 15:56 Zlatko Čalušić <zca...@bi... <mailto:zca...@bi...>> ha scritto:
> Even if LizardFS currently has a feature or two missing in current 
> MooseFS 3.0, it seems that MooseFS 4.0 will be a clear winner, and 
> definitely worth a wait!
> 
> Totally agree
> Probably the real missing thing, IMHO, is the qemu driver

I've checked qemu today - there is no such thing as "qemu driver". They have support for glusterfs - but this is part of qemu code.

I probably could write similar code for qemu to integrate qemu with MFS, but I'm not sure if it would be accepted by qemu programmers.

-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

Re: [MooseFS-Users] Real experiences from real users

[Jakub Kruszona-Z. <jak...@ge...>]

> On 21 May, 2018, at 17:57, Gandalf Corvotempesta <gan...@gm...> wrote:
> 
> Il giorno lun 21 mag 2018 alle ore 15:16 Wilson, Steven M
> <st...@pu...>
> ha scritto:
>> This might be a good use case for implementing ZFS with compression
> enabled.  For one of my MooseFS installations where we have millions of
> typically very small text files, I started using ZFS with compression as
> the underlying file system and I'm seeing a 4X or 5X compression ratio.  I
> basically use ZFS only for its compression feature since I am only using
> one ZFS file system per disk (i.e., no raid-z*, etc.).
> 
> Do you use any ZIL or L2ARC on SSDs ?
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users


Are you sure that using compression in underlying system gives you such ratio?

Once I was considering adding compression on the chunkserver layer (just to compress chunks not modified in X days by chunkserver). This is quite simple to implement, but we made tests in our cluster and have found that only 1/10th of chunks are "compressible" and the ratio was not so good (about 2x).

-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

Re: [MooseFS-Users] Real experiences from real users

[Jakub Kruszona-Z. <jak...@ge...>]

> On 21 May, 2018, at 21:09, Gandalf Corvotempesta <gan...@gm...> wrote:
> 
> Il giorno lun 21 mag 2018 alle ore 20:51 WK <wk...@bn...> ha scritto:
>> Maybe MFS could also detect if there is something bad happening and go RO.
> 
> How to detect this ?
> Another solution would be to generate an unique id for each chunkserver on
> first
> connection (an UUID would be great) and save the same value in both master
> and chunkserver.
> Then if chunkserver at 1.2.3.4 is talking with a different UUID than the
> one stored on
> master server, something bad happened.
> 

Are we still talking about MFS or LizardFS?

In MooseFS csid has been implemented in version 2.x. Master recognizes chunkservers using their ID not IP.

Chunkserver also remembers unique ID of MFS instance (master) and will not talk to other masters.

As I remember something wrong had happened when somebody had used MFS 1.6 - not MFS 3.x. If anybody knows any scenario (other than intentionally deleting data) leading to data corruption in MFS (3.x or 4.x - not 1.x) then let me know.

-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

Re: [MooseFS-Users] storage classes and archive

[Jakub Kruszona-Z. <jak...@ge...>]

> On 21 May, 2018, at 21:52, Gandalf Corvotempesta <gan...@gm...> wrote:
> 
> Hi to all.
> If I understood properly, with storage classes I can move chunks to a cold
> storage after a defined amount of days from the last mtime.
> 
> Ie: "-A hdd -d 14"
> 
> will move all chunks, not modified for at least 14 days, to chunkservers
> labelled "hdd"
> 
> But what happens if a chunk is modified after the archival ? Will it
> promoted back and removed from the archive ?

In 3.x automatically it can go only one way (then you can always manually clear archive flag).

In 4.x you can define if "ARCHIVE" is reversible or not. Also you can specify which times (atime/mtime/ctime) should be used.

-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

[MooseFS-Users] storage classes and archive

[Gandalf C. <gan...@gm...>]

Hi to all.
If I understood properly, with storage classes I can move chunks to a cold
storage after a defined amount of days from the last mtime.

Ie: "-A hdd -d 14"

will move all chunks, not modified for at least 14 days, to chunkservers
labelled "hdd"

But what happens if a chunk is modified after the archival ? Will it
promoted back and removed from the archive ?

Re: [MooseFS-Users] Real experiences from real users

[Gandalf C. <gan...@gm...>]

Il giorno lun 21 mag 2018 alle ore 20:51 WK <wk...@bn...> ha scritto:
> Maybe MFS could also detect if there is something bad happening and go RO.

How to detect this ?
Another solution would be to generate an unique id for each chunkserver on
first
connection (an UUID would be great) and save the same value in both master
and chunkserver.
Then if chunkserver at 1.2.3.4 is talking with a different UUID than the
one stored on
master server, something bad happened.

If you want to add a new chunkserver, just add it with a brand new IP.
if you want to replace an existing chunkserver, by keeping the old IP,
something like the following
should be executed:

mfschunkserver --uuid-clear

to clear the UUID before allowing connections to master.

Re: [MooseFS-Users] Real experiences from real users

[WK <wk...@bn...>]

On 5/21/2018 11:07 AM, Gandalf Corvotempesta wrote:
> Il giorno lun 21 mag 2018 alle ore 19:05 WK <wk...@bn...> ha scritto:
>> We switched to Gluster for the VMs and have been happy with that.
> Brrrrrrr
> There was a well-known corruption bug open for about 2 years!


yeah, but only if you are using shards, distributed and add capacity 
requiring a rebalance.

We use simple replication and do rip and replace upgrades if we need 
more capacity because its really easy to live migrate to the newer 
storage from the older.

Yes, it was annoying that they (RedHat) seemed to ignore the issue and 
their fix was a "dont do that" added to the notes, when people complained.

I'm not even sure if the issue is fixed yet. Last I read, they think it 
is fixed but haven't fully tested it yet.



>
>> I don't know if the newer versions fixed that, but it was ugly.
>> and again it was OUR fault.
> How would be possible for MFS to fix this kind of issue ? I'm just
> thinking...
> It's more a network issue that should be fixed on network layer rather than
> on Moose.
>
> Maybe by adding an IP/MAC address mapping in MooseFS could be a workaround

I don't know. Yes it is a network issue not a Moose issue. Your idea has 
merit.

Maybe MFS could also detect if there is something bad happening and go RO.

But its a bad thing to do with or without MFS.  Its just that in MFS, 
there are more serious consequences then a webserver simply not being 
available to the outside.

Again, in our case the tech fumbled fingered the IP and should have 
tested/reviewed things better before turning on the chunkserver daemon. 
We now have that box on the chunkserver checklist.

Re: [MooseFS-Users] Real experiences from real users

[Gandalf C. <gan...@gm...>]

Il giorno lun 21 mag 2018 alle ore 19:05 WK <wk...@bn...> ha scritto:
> We switched to Gluster for the VMs and have been happy with that.

Brrrrrrr
There was a well-known corruption bug open for about 2 years!

> I don't know if the newer versions fixed that, but it was ugly.
> and again it was OUR fault.

How would be possible for MFS to fix this kind of issue ? I'm just
thinking...
It's more a network issue that should be fixed on network layer rather than
on Moose.

Maybe by adding an IP/MAC address mapping in MooseFS could be a workaround

Re: [MooseFS-Users] Real experiences from real users

[WK <wk...@bn...>]

On 5/21/2018 12:45 AM, Gandalf Corvotempesta wrote:
>> Early on in our MFS history, we *did* have issues with VMs when they
>> were under heavy i/o load AND the MFS cluster was busy doing
>> rebalancing. They would go read-only and/or lose a chunk, requiring an
>> fsck to recover.
> Did you have time to figure this out? Why this happened?
> It was due to a bug in the older version of MooseFS or something else?

No, We assumed that the MFS cluster was so busy that all of the chunks 
weren't in place yet and the VM got unhappy.

We were unaware of the fsync option (actually learned about this week on 
the other thread about 4.x)

We switched to Gluster for the VMs and have been happy with that.

>
>> At the time we stopped using MFS for VM images and purposed MFS solely
>> for Email, NAS type File Server loads and Archive backups.
> Email will be one of our primary use case.

Works great for Maildirs, just buy lots of disks as 3 copies plus the 
small file penalty really shows up!.
Maybe the EC will cut that down.
If your storage is a db then thas less of an issue.  You could also 
create a vm disk and put the maildirs in there, to avoid the small file 
issue.
We do that for email archive storage, but you take another performance hit.

>
>> Since 3.x we have begun to resume using MFS for "some" VM images with no
>> failures, but we are still a little skittish and reserve that for
>> 'Cloud-Native' installs where there are other VM copies on other
>> hosts/storage, just in case something bad happens.
> Why ? Did you have any more issues with 3.x ?

No, the VMs on 3.x are actually fine. I recall seeing a note from the 
devs that they fixed something about VMs in that tree.
We are just paranoid, having burned our fingers on 1.6.x


>> We have experienced all sorts of disasters, crashes, bad drives, etc and
>> were always able to recover using a metalogger or other backups with no
>> data loss (expect on the fly data).
> I'm really interested in this.
> How MFS react to disk failures (or disk still working but with some URE) ?
> Is it safe to use MFS without any RAID, as suggested in the official site ?

We have goal=3, we run on plain XFS drives (no raid) on the 
chunkservers, which are actually really old kit.

One a drive or chunkserver dies, the clients never see the issue. We 
replace the broken component and it heals.

If you are really observant, you may notice a slight speed decrease 
during a massive rebalance at the client side, but only people using 
things like Mutt IMAP (which doesn't cache), really notice it.

>
>> a) a Tech brought up a chunkserver with the same IP as another
>> chunkserver. Not a good result, as it swiss cheesed the chunkserver data
>> on any file that was active during the period.
> What happens in this case ? MFS will start to send bad data coming from the
> chunkserver with the bad IP ?
and write to it.
I don't know if the newer versions fixed that, but it was ugly.
and again it was OUR fault.


>
>> Note: The imap email storage is a funny use case. It works really well,
>> but it really balloons storage space because of the small files. Plan
>> for as much a 5x-7x needed capacity.
> Why? 64MB chunks should be useless in email hosting.
> If a file is smaller than 64MB, chunk will get the real file size.
> Why we should plan for 5x-75x needed capacity ?
well first you have goals, so that is 3x. Then you have a minimum of 64K 
at the moose level.
so that 2k file is 64K on MFS.

-wk

Re: [MooseFS-Users] Real experiences from real users

[Wilson, S. M <st...@pu...>]

________________________________________
From: Gandalf Corvotempesta <gan...@gm...>
Sent: Monday, May 21, 2018 11:57 AM
To: Wilson, Steven M
Cc: wkmail; moo...@li...
Subject: Re: [MooseFS-Users] Real experiences from real users

Il giorno lun 21 mag 2018 alle ore 15:16 Wilson, Steven M
<st...@pu...>
ha scritto:
> This might be a good use case for implementing ZFS with compression
enabled.  For one of my MooseFS installations where we have millions of
typically very small text files, I started using ZFS with compression as
the underlying file system and I'm seeing a 4X or 5X compression ratio.  I
basically use ZFS only for its compression feature since I am only using
one ZFS file system per disk (i.e., no raid-z*, etc.).

Do you use any ZIL or L2ARC on SSDs ?

________________________________________

No, I don't.  About the only benefit I'm getting from the many features of ZFS is its compression.

Re: [MooseFS-Users] Real experiences from real users

[Gandalf C. <gan...@gm...>]

Il giorno lun 21 mag 2018 alle ore 15:16 Wilson, Steven M
<st...@pu...>
ha scritto:
> This might be a good use case for implementing ZFS with compression
enabled.  For one of my MooseFS installations where we have millions of
typically very small text files, I started using ZFS with compression as
the underlying file system and I'm seeing a 4X or 5X compression ratio.  I
basically use ZFS only for its compression feature since I am only using
one ZFS file system per disk (i.e., no raid-z*, etc.).

Do you use any ZIL or L2ARC on SSDs ?

Re: [MooseFS-Users] Real experiences from real users

[Marco M. <mar...@gm...>]

On 05/21/2018 03:45 AM, Gandalf Corvotempesta wrote:

> 
> I'm really interested in this.
> How MFS react to disk failures (or disk still working but with some URE) ?
> Is it safe to use MFS without any RAID, as suggested in the official site ?

Not only it is safe, but it is the recommended preferred way to run it.

-- Marco

Re: [MooseFS-Users] Real experiences from real users

[Wilson, S. M <st...@pu...>]

________________________________________
> From: WK <wk...@bn...>
> Sent: Sunday, May 20, 2018 5:39 PM
> To: moo...@li...
> Subject: Re: [MooseFS-Users] Real experiences from real users
> 
> Note: The imap email storage is a funny use case. It works really well,
> but it really balloons storage space because of the small files. Plan
> for as much a 5x-7x needed capacity.

This might be a good use case for implementing ZFS with compression enabled.  For one of my MooseFS installations where we have millions of typically very small text files, I started using ZFS with compression as the underlying file system and I'm seeing a 4X or 5X compression ratio.  I basically use ZFS only for its compression feature since I am only using one ZFS file system per disk (i.e., no raid-z*, etc.).

Steve