moosefs-users — Mailing list for MooseFS Users

Re: [MooseFS-Users] Multiple CRC failures

[Jakub Kruszona-Z. <jak...@ge...>]

> On 14 Jun, 2018, at 13:33, Gandalf Corvotempesta <gan...@gm...> wrote:
> 
> Il giorno gio 14 giu 2018 alle ore 13:19 Jakub Kruszona-Zawadzki
> <jak...@ge...> ha scritto:
>> I see that you have goal equal to number of servers. This is rare situation - MFS is usually used in big installation.
>> 
>> For such case I usually need special conditions in the code.
>> 
>> In general such problems are resolved in a way that first I create another copy and only then delete "broken" one. When goal is equal to number of servers than there is no place for creating such valid copy, so first I need to delete "broken" copy and then create valid one on the same server. Maybe this is why system didn't fix it. I'll check it.
> 
> Can't you directly overwrite the broken chunk insted of deleting it?
> Yes, I'm using goal equal to the number of servers, i'll plan to add
> another server (so that i'll have goal=3, 4 masters, 4 chunkservers)
> but keep in mind that allowing goal equal to the number or servers is
> an advantage,
> as this will lower the MooseFS TCO. In example, with gluster you can't
> add a single server, you have to add a number of server multiple of
> the redundancy level, in other words, if you start with 3 servers and
> redundancy set to 3, you have to add 3 servers more at once. This has
> a very high operating cost on small installation.
> 
> With Moose there is no need to respect the goal level when adding
> server, thus running with goal=2 and 2 servers or goal=3 and 3 servers
> and adding a single one when needed is an huge advantage
> (that explains why on GH i've asked if HDD removal could move chunks
> to another disk on the same server... you don't need a "spare" server
> for these movements lowering TCO aven more)


ok. I've just added this scenario to my testing scripts and I can confirm that this is a bug and only in case of goal equal to number of servers.

I'm off work as of now, so I'll continue tomorrow morning.

-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

Re: [MooseFS-Users] Multiple CRC failures

[Jakub Kruszona-Z. <jak...@ge...>]

> On 14 Jun, 2018, at 13:33, Gandalf Corvotempesta <gan...@gm...> wrote:
> 
> Il giorno gio 14 giu 2018 alle ore 13:19 Jakub Kruszona-Zawadzki
> <jak...@ge...> ha scritto:
>> I see that you have goal equal to number of servers. This is rare situation - MFS is usually used in big installation.
>> 
>> For such case I usually need special conditions in the code.
>> 
>> In general such problems are resolved in a way that first I create another copy and only then delete "broken" one. When goal is equal to number of servers than there is no place for creating such valid copy, so first I need to delete "broken" copy and then create valid one on the same server. Maybe this is why system didn't fix it. I'll check it.
> 
> Can't you directly overwrite the broken chunk insted of deleting it?

In general not very good idea. Usually CRC errors means that there is something wrong with this machine.

> Yes, I'm using goal equal to the number of servers, i'll plan to add
> another server (so that i'll have goal=3, 4 masters, 4 chunkservers)
> but keep in mind that allowing goal equal to the number or servers is
> an advantage,

Yes. I only said that it needs to be treated in special way.

> as this will lower the MooseFS TCO. In example, with gluster you can't
> add a single server, you have to add a number of server multiple of
> the redundancy level, in other words, if you start with 3 servers and
> redundancy set to 3, you have to add 3 servers more at once. This has
> a very high operating cost on small installation.
> 
> With Moose there is no need to respect the goal level when adding
> server, thus running with goal=2 and 2 servers or goal=3 and 3 servers
> and adding a single one when needed is an huge advantage
> (that explains why on GH i've asked if HDD removal could move chunks
> to another disk on the same server... you don't need a "spare" server
> for these movements lowering TCO aven more)

-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

Re: [MooseFS-Users] Multiple CRC failures

[Gandalf C. <gan...@gm...>]

Il giorno gio 14 giu 2018 alle ore 13:19 Jakub Kruszona-Zawadzki
<jak...@ge...> ha scritto:
> I see that you have goal equal to number of servers. This is rare situation - MFS is usually used in big installation.
>
> For such case I usually need special conditions in the code.
>
> In general such problems are resolved in a way that first I create another copy and only then delete "broken" one. When goal is equal to number of servers than there is no place for creating such valid copy, so first I need to delete "broken" copy and then create valid one on the same server. Maybe this is why system didn't fix it. I'll check it.

Can't you directly overwrite the broken chunk insted of deleting it?
Yes, I'm using goal equal to the number of servers, i'll plan to add
another server (so that i'll have goal=3, 4 masters, 4 chunkservers)
but keep in mind that allowing goal equal to the number or servers is
an advantage,
as this will lower the MooseFS TCO. In example, with gluster you can't
add a single server, you have to add a number of server multiple of
the redundancy level, in other words, if you start with 3 servers and
redundancy set to 3, you have to add 3 servers more at once. This has
a very high operating cost on small installation.

With Moose there is no need to respect the goal level when adding
server, thus running with goal=2 and 2 servers or goal=3 and 3 servers
and adding a single one when needed is an huge advantage
(that explains why on GH i've asked if HDD removal could move chunks
to another disk on the same server... you don't need a "spare" server
for these movements lowering TCO aven more)

Re: [MooseFS-Users] Multiple CRC failures

[Jakub Kruszona-Z. <jak...@ge...>]

> 
>> If there is lot of other things to do (replication of other undergoal chunks for example) then yes. Such chunk should be treated in the same way as any other undergoal chunk. Did you check the GOAL/VC table on the Info tab in mfs.cgi? Did you have "yellow" chunks? If not then maybe system created valid copy, but didn't remove damaged chunk.
> 
> Honestly, i've not looked at CGI page because I was seeing in the log
> multipe "chunk_readcrc", thus, CRC mismatch was still detected.
> If CRC is detected, I think that MooseFS still haven't replicated
> (fixing the CRC) the chunk.

I see that you have goal equal to number of servers. This is rare situation - MFS is usually used in big installation.

For such case I usually need special conditions in the code.

In general such problems are resolved in a way that first I create another copy and only then delete "broken" one. When goal is equal to number of servers than there is no place for creating such valid copy, so first I need to delete "broken" copy and then create valid one on the same server. Maybe this is why system didn't fix it. I'll check it.

> 
> This is a test cluster, no other operations where running.

Great !!!.

BTW. I've noticed another small issue. file '.metaid' on disks is not created when metaid is not known during initialization of chunkserver (case of first run). When metaid is received from the master it is stored only in 'chunkserverid.mfs', but disks are left unchanged. This is not serious, but still needs to be fixed.

> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

Re: [MooseFS-Users] Multiple CRC failures

[Gandalf C. <gan...@gm...>]

Il giorno gio 14 giu 2018 alle ore 12:16 Jakub Kruszona-Zawadzki
<jak...@ge...> ha scritto:
> Number is the same, but version can be higher.
> File name pattern is "chunk_UNIQUENUMBEROFCHUNK_VERSION.mfs"
> UNIQUENUMBEROFCHUNK should be the same, but VERSION can be different.

No, there isn't any "new version".
This is the error message I had:

chunk_readcrc: file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)

but /mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
doesn't exist in any version. Probably it was replicated with a new
chunk name

# find /mnt/chunks/ -name 'chunk_00000000006639E9_*'
#


> If there is lot of other things to do (replication of other undergoal chunks for example) then yes. Such chunk should be treated in the same way as any other undergoal chunk. Did you check the GOAL/VC table on the Info tab in mfs.cgi? Did you have "yellow" chunks? If not then maybe system created valid copy, but didn't remove damaged chunk.

Honestly, i've not looked at CGI page because I was seeing in the log
multipe "chunk_readcrc", thus, CRC mismatch was still detected.
If CRC is detected, I think that MooseFS still haven't replicated
(fixing the CRC) the chunk.

This is a test cluster, no other operations where running.

Re: [MooseFS-Users] Multiple CRC failures

[Jakub Kruszona-Z. <jak...@ge...>]

> On 9 Jun, 2018, at 12:20, Gandalf Corvotempesta <gan...@gm...> wrote:
> 
> Last crc failure i've seen for the "rotted" file is here:
> 
> Jun  8 22:03:36 cs02 mfschunkserver[22075]: chunk_readcrc:
> file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
> - read error: Success (errno=0)
> Jun  8 22:03:36 cs02 mfschunkserver[22075]: hdd_io_begin:
> file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
> - read error: Success (errno=0)
> 
> But file /mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
> doesn't exist anymore.
> In case of crc mismatch, MooseFS will create a new chunk with a
> different file name ?

Number is the same, but version can be higher.
File name pattern is "chunk_UNIQUENUMBEROFCHUNK_VERSION.mfs"
UNIQUENUMBEROFCHUNK should be the same, but VERSION can be different.

> Is normal that with standard configuration, the crc mismatch is
> resolved after about 10 hours from discovery ?

If there is lot of other things to do (replication of other undergoal chunks for example) then yes. Such chunk should be treated in the same way as any other undergoal chunk. Did you check the GOAL/VC table on the Info tab in mfs.cgi? Did you have "yellow" chunks? If not then maybe system created valid copy, but didn't remove damaged chunk.



-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

Re: [MooseFS-Users] Real experiences from real users

[Gandalf C. <gan...@gm...>]

Il giorno mar 22 mag 2018 alle ore 07:55 Jakub Kruszona-Zawadzki
<jak...@ge...> ha scritto:
> Once I was considering adding compression on the chunkserver layer (just to compress chunks not modified in X days by chunkserver). This is quite simple to implement, but we made tests in our cluster and have found that only 1/10th of chunks are "compressible" and the ratio was not so good (about 2x).

it seems that using ZFS without any kind of redundancy (so, using one
pool per disks, even with failmode=continue) is not good (this is what
ZFS mailing list told me.
So, as ZFS mailing list doesn't suggest to use ZFS in one pool per
disk, adding compression directly to MooseFS (with the ability to
enable or disable the feature) could be an interesting idea, but why
compressing only unmodified chunks and not all chunks ?

Re: [MooseFS-Users] v4: lot of disconnections

[Gandalf C. <gan...@gm...>]

Any update ?
Il giorno gio 7 giu 2018 alle ore 14:37 Gandalf Corvotempesta
<gan...@gm...> ha scritto:
>
> Are the following lines normal ? I have the same output every 5
> seconds, but only on a single server.
>
> Jun  7 14:35:25 cs03 mfsmaster[19504]: csdb: found cs using ip:port
> and csid (10.200.1.11:9422,1)
> Jun  7 14:35:25 cs03 mfsmaster[19504]: chunkserver register begin
> (packet version: 6) - ip: 10.200.1.11 / port: 9422, usedspace:
> 698556416 (0.65 GiB), totalspace: 3735778050048 (3479.21 GiB)
> Jun  7 14:35:25 cs03 mfsmaster[19504]: connection with CS(10.200.1.11)
> has been closed by peer
> Jun  7 14:35:25 cs03 mfsmaster[19504]: chunkserver disconnected - ip:
> 10.200.1.11 / port: 9422, usedspace: 698556416 (0.65 GiB), totalspace:
> 3735778050048 (3479.21 GiB)
> Jun  7 14:35:25 cs03 mfsmaster[19504]: connection with
> client(ip:10.200.1.101) has been closed by peer
> Jun  7 14:35:25 cs03 mfsmaster[19504]: server ip: 10.200.1.11 / port:
> 9422 has been fully removed from data structures
> Jun  7 14:35:30 cs03 mfsmaster[19504]: csdb: found cs using ip:port
> and csid (10.200.1.11:9422,1)
> Jun  7 14:35:30 cs03 mfsmaster[19504]: chunkserver register begin
> (packet version: 6) - ip: 10.200.1.11 / port: 9422, usedspace:
> 698556416 (0.65 GiB), totalspace: 3735778050048 (3479.21 GiB)
> Jun  7 14:35:30 cs03 mfsmaster[19504]: connection with CS(10.200.1.11)
> has been closed by peer
> Jun  7 14:35:30 cs03 mfsmaster[19504]: chunkserver disconnected - ip:
> 10.200.1.11 / port: 9422, usedspace: 698556416 (0.65 GiB), totalspace:
> 3735778050048 (3479.21 GiB)
> Jun  7 14:35:30 cs03 mfsmaster[19504]: connection with
> client(ip:10.200.1.101) has been closed by peer
> Jun  7 14:35:30 cs03 mfsmaster[19504]: server ip: 10.200.1.11 / port:
> 9422 has been fully removed from data structures
>
>
> On a folllower:
>
> Jun  7 14:37:00 cs01 mfschunkserver[25307]: connected to Master
> Jun  7 14:37:00 cs01 mfschunkserver[25307]: MATOCS_MASTER_ACK - wrong
> meta data id. Can't connect to master

Re: [MooseFS-Users] Multiple CRC failures

[Gandalf C. <gan...@gm...>]

Last crc failure i've seen for the "rotted" file is here:

Jun  8 22:03:36 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 22:03:36 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)

But file /mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
doesn't exist anymore.
In case of crc mismatch, MooseFS will create a new chunk with a
different file name ?
Is normal that with standard configuration, the crc mismatch is
resolved after about 10 hours from discovery ?
Il giorno ven 8 giu 2018 alle ore 15:45 Gandalf Corvotempesta
<gan...@gm...> ha scritto:
>
> Il giorno ven 8 giu 2018 alle ore 15:44 Gandalf Corvotempesta
> <gan...@gm...> ha scritto:
> > is this OK ? The file was manually "rotted" by me, is normal that
> > MooseFS detect corruption at 12:41:16 and did nothing until 15:34:25 ?
> >  15:34 is also the same time I see in CGI, in last error field
>
> Correction: rotted chunk is still unrepaired.

Re: [MooseFS-Users] Multiple CRC failures

[Gandalf C. <gan...@gm...>]

Il giorno ven 8 giu 2018 alle ore 15:44 Gandalf Corvotempesta
<gan...@gm...> ha scritto:
> is this OK ? The file was manually "rotted" by me, is normal that
> MooseFS detect corruption at 12:41:16 and did nothing until 15:34:25 ?
>  15:34 is also the same time I see in CGI, in last error field

Correction: rotted chunk is still unrepaired.

[MooseFS-Users] Multiple CRC failures

[Gandalf C. <gan...@gm...>]

 I'm getting this:

Jun  8 12:41:16 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 12:41:16 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 12:51:02 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 12:51:02 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:00:00 cs02 mfsmaster[21945]: child finished
Jun  8 13:00:00 cs02 mfsmaster[21945]: store process has finished -
store time: 0.213
Jun  8 13:00:36 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:00:36 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:10:01 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:10:01 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:17:01 cs02 CRON[12342]: (root) CMD (   cd / && run-parts
--report /etc/cron.hourly)
Jun  8 13:19:31 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:19:31 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:23:01 cs02 systemd[1]: Starting Daily apt download activities...
Jun  8 13:23:01 cs02 systemd[1]: Started Daily apt download activities.
Jun  8 13:23:01 cs02 systemd[1]: apt-daily.timer: Adding 9h 54min
51.316685s random time.
Jun  8 13:23:01 cs02 systemd[1]: apt-daily.timer: Adding 5h 38min
50.643138s random time.
Jun  8 13:29:06 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:29:06 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:38:39 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:38:39 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:48:10 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:48:10 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:58:16 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 13:58:16 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:00:00 cs02 mfsmaster[21945]: child finished
Jun  8 14:00:00 cs02 mfsmaster[21945]: store process has finished -
store time: 0.208
Jun  8 14:08:02 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:08:02 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:17:01 cs02 CRON[12423]: (root) CMD (   cd / && run-parts
--report /etc/cron.hourly)
Jun  8 14:17:23 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:17:23 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:26:41 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:26:41 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:36:15 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:36:15 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:46:01 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:46:01 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:55:32 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 14:55:32 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 15:00:00 cs02 mfsmaster[21945]: child finished
Jun  8 15:00:00 cs02 mfsmaster[21945]: store process has finished -
store time: 0.210
Jun  8 15:05:45 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 15:05:45 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 15:15:20 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 15:15:20 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 15:17:01 cs02 CRON[12539]: (root) CMD (   cd / && run-parts
--report /etc/cron.hourly)
Jun  8 15:24:41 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 15:24:41 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 15:34:25 cs02 mfschunkserver[22075]: chunk_readcrc:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)
Jun  8 15:34:25 cs02 mfschunkserver[22075]: hdd_io_begin:
file:/mnt/chunks/chunk0/chunk//01/chunk_00000000006639E9_00000001.mfs
- read error: Success (errno=0)


is this OK ? The file was manually "rotted" by me, is normal that
MooseFS detect corruption at 12:41:16 and did nothing until 15:34:25 ?
 15:34 is also the same time I see in CGI, in last error field

Re: [MooseFS-Users] changelogs: Writes per day

[Gandalf C. <gan...@gm...>]

Il giorno gio 7 giu 2018 alle ore 18:08 Zlatko Čalušić
<zca...@bi...> ha scritto:
>
> Hello Gandalf,

Hi

> That is just impossible to answer without knowing your workload + your
> parameters (noatime mounts or not?). You're the only one that can see
> that for yourself and decide appropriately. You can never go wrong with
>  >1 DPWD datacenter grade SSD, becuase you always get what you paid for.

I know that would be impossible, but it's also impossible to forecast
metadata operations
in a brand new cluster.

That's why I've asked. If, on heavy cluster, maybe with hundreds of
VMs or a lot of maildirs
(that should do metadata operation more than VMs), an average is more
or less 100GB per day,
any enterprise-grade SSD (even the read-intensive) would be ok.

On the other hand, if that cluster is making 1 TB per day, I'll be
better to buy a write-intensive SSD
or a much bigger read-intesive SSD

Anyway, the disk space requirements shown in the following formula:
SPACE = RAM * (BACK META KEEP PREVIOUS + 2) + 1 * (BACK LOGS + 1)

talks about "RAM". Is "RAM" the current metadata usage or the server RAM ?

In example, If I have a master server with 64GB ram but i'm only using
2GB for metadata, the space requirement
is evaluated by using "2GB" or "64GB" ? There is a lot of difference
in needed space and in costs (when using datacenter grade SSDs)

Re: [MooseFS-Users] changelogs: Writes per day

[Zlatko Č. <zca...@bi...>]

Hello Gandalf,

That is just impossible to answer without knowing your workload + your 
parameters (noatime mounts or not?). You're the only one that can see 
that for yourself and decide appropriately. You can never go wrong with 
 >1 DPWD datacenter grade SSD, becuase you always get what you paid for.

Now, having said that, unless you really want and get sync operations on 
changelog, in my experience any SSD would do. But, if you ever get 
support for sync changelog writing and turn it on, well... then go for 
the most expensive SSD you can buy.

But again, depends... how many metadata operations you expect? Nobody 
can know that except you.

Regards,

On 07.06.2018 17:57, Gandalf Corvotempesta wrote:
> I'm planning which kind of SSD to use for storing /var/lib/mfs
> In that directory, moose is writing all changelog.
>
> On a huge cluster, how many GB/TBs written should I expect, per day?
> Is a write-intensive SSD needed or can I also use a mixed workload SSD
> like the ones with 0.8 DWPD (Intel S4500 480GB)
>
> Any suggestion ?
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

-- 
Zlatko

[MooseFS-Users] changelogs: Writes per day

[Gandalf C. <gan...@gm...>]

I'm planning which kind of SSD to use for storing /var/lib/mfs
In that directory, moose is writing all changelog.

On a huge cluster, how many GB/TBs written should I expect, per day?
Is a write-intensive SSD needed or can I also use a mixed workload SSD
like the ones with 0.8 DWPD (Intel S4500 480GB)

Any suggestion ?

[MooseFS-Users] v4: lot of disconnections

[Gandalf C. <gan...@gm...>]

Are the following lines normal ? I have the same output every 5
seconds, but only on a single server.

Jun  7 14:35:25 cs03 mfsmaster[19504]: csdb: found cs using ip:port
and csid (10.200.1.11:9422,1)
Jun  7 14:35:25 cs03 mfsmaster[19504]: chunkserver register begin
(packet version: 6) - ip: 10.200.1.11 / port: 9422, usedspace:
698556416 (0.65 GiB), totalspace: 3735778050048 (3479.21 GiB)
Jun  7 14:35:25 cs03 mfsmaster[19504]: connection with CS(10.200.1.11)
has been closed by peer
Jun  7 14:35:25 cs03 mfsmaster[19504]: chunkserver disconnected - ip:
10.200.1.11 / port: 9422, usedspace: 698556416 (0.65 GiB), totalspace:
3735778050048 (3479.21 GiB)
Jun  7 14:35:25 cs03 mfsmaster[19504]: connection with
client(ip:10.200.1.101) has been closed by peer
Jun  7 14:35:25 cs03 mfsmaster[19504]: server ip: 10.200.1.11 / port:
9422 has been fully removed from data structures
Jun  7 14:35:30 cs03 mfsmaster[19504]: csdb: found cs using ip:port
and csid (10.200.1.11:9422,1)
Jun  7 14:35:30 cs03 mfsmaster[19504]: chunkserver register begin
(packet version: 6) - ip: 10.200.1.11 / port: 9422, usedspace:
698556416 (0.65 GiB), totalspace: 3735778050048 (3479.21 GiB)
Jun  7 14:35:30 cs03 mfsmaster[19504]: connection with CS(10.200.1.11)
has been closed by peer
Jun  7 14:35:30 cs03 mfsmaster[19504]: chunkserver disconnected - ip:
10.200.1.11 / port: 9422, usedspace: 698556416 (0.65 GiB), totalspace:
3735778050048 (3479.21 GiB)
Jun  7 14:35:30 cs03 mfsmaster[19504]: connection with
client(ip:10.200.1.101) has been closed by peer
Jun  7 14:35:30 cs03 mfsmaster[19504]: server ip: 10.200.1.11 / port:
9422 has been fully removed from data structures


On a folllower:

Jun  7 14:37:00 cs01 mfschunkserver[25307]: connected to Master
Jun  7 14:37:00 cs01 mfschunkserver[25307]: MATOCS_MASTER_ACK - wrong
meta data id. Can't connect to master

Re: [MooseFS-Users] Separate network for chunk servers

[Piotr R. K. <pio...@mo...>]

Thanks for sharing!

Peter

-- 
Piotr Robert Konopelko | mobile: +48 601 476 440
MooseFS Client Support Team | moosefs.com <http://moosefs.com/>

GitHub <https://github.com/moosefs/moosefs> | Twitter <https://twitter.com/moosefs> | Facebook <https://www.facebook.com/moosefs> | LinkedIn <https://www.linkedin.com/company/moosefs>

> On 6 Jun 2018, at 8:26 PM, Wilson, Steven M <st...@pu... <mailto:st...@pu...>> wrote:
> 
> 
> From: Wilson, Steven M <st...@pu... <mailto:st...@pu...>>
> Sent: Monday, May 7, 2018 10:36 AM
> To: MooseFS-Users
> Subject: [MooseFS-Users] Separate network for chunk servers
>  
> Hi,
> 
> I'm considering implementing a dedicated network for our chunk servers to use soley for replication among themselves.  By doing this, I hope to separate the chunk traffic from the clients from the replication traffic that takes place among the chunk servers.  If my understanding is correct, this is not achieved by using the REMAP_* options in mfsmaster.cfg which only separates out the traffic to/from the master.
> 
> If anyone else has done this, I'd be grateful to hear about your experience, especially in these two areas:
>     1)  what level of performance improvement was seen
>     2)  what needed to be done in the MooseFS configuration and OS networking to implement it
> 
> ​Thanks!
> Steve
> 
> =====================================================================================
> 
> In case it's helpful to anyone else, I did find an easy way to shift traffic between chunk servers onto a separate network.  On each chunk server I created a few iptables rules using NAT like the following (these would need to be repeated for each additional chunk server):
> ​iptables -t nat -A OUTPUT -s $MY_IP/32 -d $OTHER_CS_IP/32 -j DNAT --to-destination $OTHER_CS_PRIV_IP -p tcp -m tcp --dport $CS_PORT
> iptables -t nat -A POSTROUTING -s $MY_IP/32 -d $OTHER_CS_PRIV_IP/32 -j SNAT --to-source $MY_PRIV_IP
> 
> I did some performance testing and found that, for fairly obvious reasons, this private chunk server network only helps when the network is near saturation and most of the I/O is writing to the file system (which generates inter-chunkserver traffic for producing redundant copies of the data).  When this type of environment is simulated, I saw between 10 to 20% improvement in speed for writing files to the file system.
> 
> Steve
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org <http://slashdot.org/>! http://sdm.link/slashdot_________________________________________ <http://sdm.link/slashdot_________________________________________>
> moosefs-users mailing list
> moo...@li... <mailto:moo...@li...>
> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>



Best regards,
Peter

-- 
Piotr Robert Konopelko | mobile: +48 601 476 440
MooseFS Client Support Team | moosefs.com <http://moosefs.com/>

GitHub <https://github.com/moosefs/moosefs> | Twitter <https://twitter.com/moosefs> | Facebook <https://www.facebook.com/moosefs> | LinkedIn <https://www.linkedin.com/company/moosefs>

Re: [MooseFS-Users] Separate network for chunk servers

[Wilson, S. M <st...@pu...>]

________________________________
From: Wilson, Steven M <st...@pu...>
Sent: Monday, May 7, 2018 10:36 AM
To: MooseFS-Users
Subject: [MooseFS-Users] Separate network for chunk servers


Hi,


I'm considering implementing a dedicated network for our chunk servers to use soley for replication among themselves.  By doing this, I hope to separate the chunk traffic from the clients from the replication traffic that takes place among the chunk servers.  If my understanding is correct, this is not achieved by using the REMAP_* options in mfsmaster.cfg which only separates out the traffic to/from the master.


If anyone else has done this, I'd be grateful to hear about your experience, especially in these two areas:

    1)  what level of performance improvement was seen

    2)  what needed to be done in the MooseFS configuration and OS networking to implement it


?Thanks!

Steve


=====================================================================================


In case it's helpful to anyone else, I did find an easy way to shift traffic between chunk servers onto a separate network.  On each chunk server I created a few iptables rules using NAT like the following (these would need to be repeated for each additional chunk server):

?iptables -t nat -A OUTPUT -s $MY_IP/32 -d $OTHER_CS_IP/32 -j DNAT --to-destination $OTHER_CS_PRIV_IP -p tcp -m tcp --dport $CS_PORT

iptables -t nat -A POSTROUTING -s $MY_IP/32 -d $OTHER_CS_PRIV_IP/32 -j SNAT --to-source $MY_PRIV_IP

I did some performance testing and found that, for fairly obvious reasons, this private chunk server network only helps when the network is near saturation and most of the I/O is writing to the file system (which generates inter-chunkserver traffic for producing redundant copies of the data).  When this type of environment is simulated, I saw between 10 to 20% improvement in speed for writing files to the file system.

Steve

Re: [MooseFS-Users] Master IP in v4

[Gandalf C. <gan...@gm...>]

Il giorno mar 5 giu 2018 alle ore 11:32 Jakub Kruszona-Zawadzki
<jak...@ge...> ha scritto:
> Are you sure that all computers have the same DNS and on each of them your 'mfsmaster' name is resolved properly to all masters?

Yes, sure.

> Great. Piotr will contact you directly.

Ok, waiting for him

Re: [MooseFS-Users] Master IP in v4

[Jakub Kruszona-Z. <jak...@ge...>]

> On 5 Jun, 2018, at 9:42, Gandalf Corvotempesta <gan...@gm...> wrote:
> 
> Il giorno mar 5 giu 2018 alle ore 07:14 Jakub Kruszona-Zawadzki
> <jak...@ge...> ha scritto:
>> There must be something wrong with your configuration
> 
> Maybe, but it's a very simple config, almost everything kept as default.

Are you sure that all computers have the same DNS and on each of them your 'mfsmaster' name is resolved properly to all masters? 

> 
>> but still MFS shouldn't work like that. We have to fix it.
>> I'll prepare for you packages with more logs and we will see. This is very interesting case.
> 
> Thank you. If you need, I can provide SSH access to these
> master/chunkserver servers (not the client where MooseFS is mounted)

Great. Piotr will contact you directly.

-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

Re: [MooseFS-Users] Master IP in v4

[Jakub Kruszona-Z. <jak...@ge...>]

> On 5 Jun, 2018, at 9:47, Gandalf Corvotempesta <gan...@gm...> wrote:
> 
> Some log from the client trying to mount the storage:
> 
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying using different IP
> mfsmaster 10.200.1.11 - found leader: 10.200.1.12
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying using different IP
> mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
> a moment and retrying
> 
> and so on...
> 
 
it is ok.

> (please add a date and time in front of each line)

We mainly use system syslog, so OS should add dates to each line.

If you run MFS in foreground then of course we do not print dates and times - we can add them, but you can always read those lines from your syslog with date/time.

-- 
Pozdrawiam,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Tel. +48 602 212 039

Re: [MooseFS-Users] Master IP in v4

[Gandalf C. <gan...@gm...>]

Some log from the client trying to mount the storage:

mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying using different IP
mfsmaster 10.200.1.11 - found leader: 10.200.1.12
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying using different IP
mfsmaster 10.200.1.12 - this is ELECT waitng for being LEADER, waiting
a moment and retrying

and so on...

(please add a date and time in front of each line)

Re: [MooseFS-Users] Master IP in v4

[Gandalf C. <gan...@gm...>]

Il giorno mar 5 giu 2018 alle ore 07:14 Jakub Kruszona-Zawadzki
<jak...@ge...> ha scritto:
>There must be something wrong with your configuration

Maybe, but it's a very simple config, almost everything kept as default.

> but still MFS shouldn't work like that. We have to fix it.
> I'll prepare for you packages with more logs and we will see. This is very interesting case.

Thank you. If you need, I can provide SSH access to these
master/chunkserver servers (not the client where MooseFS is mounted)

Re: [MooseFS-Users] Master IP in v4

[Jakub Kruszona-Z. <jak...@ge...>]

> On 4 Jun, 2018, at 22:21, Gandalf Corvotempesta <gan...@gm...> wrote:
> 
> Il giorno lun 4 giu 2018 alle ore 14:18 Gandalf Corvotempesta
> <gan...@gm...> ha scritto:
>> I've tried right now and transation is happening properly.
>> Nothing changed from last time i've tested and today. Absolutely nothing.
> 
> Ok, now transition is broken.
> Just a simple "kill -9 master_pid".
> Is stuck in this state from about 20 minutes.
> <screen.jpg>


I've never seen such behaviour. There must be something wrong with your configuration, but still MFS shouldn't work like that. We have to fix it.

I'll prepare for you packages with more logs and we will see. This is very interesting case.


-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

Re: [MooseFS-Users] Master IP in v4

[Gandalf C. <gan...@gm...>]

Il giorno lun 4 giu 2018 alle ore 14:18 Gandalf Corvotempesta
<gan...@gm...> ha scritto:
> I've tried right now and transation is happening properly.
> Nothing changed from last time i've tested and today. Absolutely nothing.

Ok, now transition is broken.
Just a simple "kill -9 master_pid".
Is stuck in this state from about 20 minutes.

Re: [MooseFS-Users] Master IP in v4

[Diego R. <dij...@ae...>]

See below...

>>
>> In these 3 servers i'm running master and chunkservers, thus, i have 
>> 3 chunkservers
>
> ok.
>
>>
>> I've not checked for any disconnected chunkserver but it should block 
>> after 2 disconnected chunkservers, right?
>
> yes (if the total number of known chunkserver is 3 or 4).
>
>> In a 3 nodes cluster, quorum is met at 2, so it should survive at 1 
>> chunkserver failure and i'm pretty sure that i don't have 2 
>> chunkservers down during the master switch
>
> Yes. I've seen in your log three chunkservers connected to ELECT - 
> this is really strange. Could you please send us some screenshots from 
> your CGI?
>
> As I understand you have three masters and three chunkservers on the 
> same machines and ELECT is not becoming LEADER for a long time 
> (minutes) after killing LEADER, but when you stop everything and start 
> again then you have LEADER?
>

Here are my observations, I have 3 servers running 4.6.0, two (ae-fs02 
and ae-fs03) of them have both roles, master and chunkserver, the other 
(ae-fs01), has meta logger and chunkserver.

When ae-fs03 is leader, I stop the chunkserver process on ae-fs01, then 
stop master on ae-fs03, then ae-fs02 which was follower, becomes elect 
and within 2-3 seconds, it becomes leader while ae-fs03 shows up as dead.

At this point, ae-fs02 is the leader and ae-fs03 becomes follower when I 
start the service moosefs-master. If I reboot ae-fs02, then ae-fs03 
continues to show up as FOLLOWER, but it takes a long time, over a 
minute, for it to become elect and then leader. The GUI shows:



Until it finally changes to:



After ae-fs02 has rebooted, and the moosefs services started, the 
cluster is back to normal.



HTH,

Diego