moosefs-users — Mailing list for MooseFS Users

Re: [MooseFS-Users] Network overhead and best method of sharing

[Michael T. <mic...@ho...>]

Why not NFS?  Are there any incompatibilities between MFS client and NFS?

> 
> Message: 2
> Date: Wed, 19 Aug 2015 08:17:26 +0200
> From: Aleksander Wieliczko <ale...@mo...>
> Subject: Re: [MooseFS-Users] Network overhead and best method of
> 	sharing
> To: "R. C." <mil...@gm...>
> Cc: moo...@li...
> Message-ID: <47E...@mo...>
> Content-Type: text/plain; charset="us-ascii"
> 
> Hi Raffaello
> 
> If you use OS which supports MooseFS-client try to not use any re-share method. You will always add another layer in communication during re-share. Direct use of mfsmount will always be better.
> 
> But if you like to use MooseFS on Windows or storage for Virtualisation try to use SAMBA/iSCSI Target.
> We are not advising to use NFS as re-share method.
> 
> Best Regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <http://moosefs.com/>

 		 	   		  

Re: [MooseFS-Users] Mounted disk size varies, huge disk size drops, high iowait

[Edik M. <edi...@tu...>]

Sorry, I think this one is also may be useful

​

On Thu, Aug 20, 2015 at 3:28 PM, Edik Mkoyan <edi...@tu...> wrote:

> Hi Piotr,
>
> I use 2.0.73-1, I know that 2.0.72 is the stable version, but the apt-get
> install that version on mu chunk server, and mrs clients.
> On MoosFS I use 2.0.50-1(yet), going to update to 2.0.7X, or maybe
> 3.0.X(if there is a good document on that).
> Reexporting is not my idea, I have what I have, do not know much about
> that, the fact is that we had not problems until two or so months ago, then
> I have installed zabbix agent and it instantly alarmed a high iowait, see
> the graphs please.
>
>
> ​
>
> I think this is not normal, I know that two most probabel reasons are the
> network issues and the faulty storage. Here is how the network load
> correlates with the iowait on all four chunk servers
>
> ​  And just yesterday friend of mine noticed this
>
> ​
> And when I started to monitor the storage size of mrs mount on mfsclient,
> I have found this
>
> ​the devising started somewhere around 6.00 am, at night it was a
> horizontal line, yesterday it increased to the maximal number for few
> hours, our stuff starts to work at 10:00 am.
>
>
> On Thu, Aug 20, 2015 at 2:25 PM, Piotr Robert Konopelko <
> pio...@mo...> wrote:
>
>> Hello Edik,
>>
>> Which version of MooseFS do you use?
>> Is it 1.6, 2.0 or 3.0?
>>
>> Why do you re-share MFS via NFS? Do you *really* need it?
>> We know, that performance is poor when such operation (re-share via NFS)
>> is done.
>>
>> Also, your charts are very compressed, it is hard to see anything in
>> legend.
>> Could you post a bigger image?
>>
>>
>> Best regards,
>>
>> --
>> Piotr Robert Konopelko
>> *MooseFS Technical Support Engineer* | moosefs.com
>>
>> On 20 Aug 2015, at 11:37 am, Edik Mkoyan <edi...@tu...> wrote:
>>
>> FYI
>>
>> http://superuser.com/questions/960027/disk-size-varies-moosefs
>>
>> Thanks
>>
>> --
>> --------
>> Best,
>> Edik.
>>
>> This message contains confidential information and is intended only for
>> the individual named. If you are not the named addressee you should not
>> disseminate, distribute or copy this e-mail. Please notify the sender
>> immediately by e-mail if you have received this e-mail by mistake and
>> delete this e-mail from your system. E-mail transmission cannot be
>> guaranteed to be secure or error-free as information could be intercepted,
>> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
>> The sender therefore does not accept liability for any errors or omissions
>> in the contents of this message, which arise as a result of e-mail
>> transmission. If verification is required please request a hard-copy
>> version. Simonian Educational Foundation, Halabyan 16, Yerevan, Rep. of
>> Armenia. www.tumo.org
>>
>> ------------------------------------------------------------------------------
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>>
>>
>
>
> --
> --------
> Best,
> Edik.
>



-- 
--------
Best,
Edik.

-- 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
The sender therefore does not accept liability for any errors or omissions 
in the contents of this message, which arise as a result of e-mail 
transmission. If verification is required please request a hard-copy 
version. Simonian Educational Foundation, Halabyan 16, Yerevan, Rep. of 
Armenia. www.tumo.org

Re: [MooseFS-Users] Mounted disk size varies, huge disk size drops, high iowait

[Edik M. <edi...@tu...>]

Hi Piotr,

I use 2.0.73-1, I know that 2.0.72 is the stable version, but the apt-get
install that version on mu chunk server, and mrs clients.
On MoosFS I use 2.0.50-1(yet), going to update to 2.0.7X, or maybe 3.0.X(if
there is a good document on that).
Reexporting is not my idea, I have what I have, do not know much about
that, the fact is that we had not problems until two or so months ago, then
I have installed zabbix agent and it instantly alarmed a high iowait, see
the graphs please.


​

I think this is not normal, I know that two most probabel reasons are the
network issues and the faulty storage. Here is how the network load
correlates with the iowait on all four chunk servers

​  And just yesterday friend of mine noticed this

​
And when I started to monitor the storage size of mrs mount on mfsclient, I
have found this

​the devising started somewhere around 6.00 am, at night it was a
horizontal line, yesterday it increased to the maximal number for few
hours, our stuff starts to work at 10:00 am.


On Thu, Aug 20, 2015 at 2:25 PM, Piotr Robert Konopelko <
pio...@mo...> wrote:

> Hello Edik,
>
> Which version of MooseFS do you use?
> Is it 1.6, 2.0 or 3.0?
>
> Why do you re-share MFS via NFS? Do you *really* need it?
> We know, that performance is poor when such operation (re-share via NFS)
> is done.
>
> Also, your charts are very compressed, it is hard to see anything in
> legend.
> Could you post a bigger image?
>
>
> Best regards,
>
> --
> Piotr Robert Konopelko
> *MooseFS Technical Support Engineer* | moosefs.com
>
> On 20 Aug 2015, at 11:37 am, Edik Mkoyan <edi...@tu...> wrote:
>
> FYI
>
> http://superuser.com/questions/960027/disk-size-varies-moosefs
>
> Thanks
>
> --
> --------
> Best,
> Edik.
>
> This message contains confidential information and is intended only for
> the individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. E-mail transmission cannot be
> guaranteed to be secure or error-free as information could be intercepted,
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
> The sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> transmission. If verification is required please request a hard-copy
> version. Simonian Educational Foundation, Halabyan 16, Yerevan, Rep. of
> Armenia. www.tumo.org
>
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
>


-- 
--------
Best,
Edik.

-- 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
The sender therefore does not accept liability for any errors or omissions 
in the contents of this message, which arise as a result of e-mail 
transmission. If verification is required please request a hard-copy 
version. Simonian Educational Foundation, Halabyan 16, Yerevan, Rep. of 
Armenia. www.tumo.org

Re: [MooseFS-Users] Mounted disk size varies, huge disk size drops, high iowait

[Piotr R. K. <pio...@mo...>]

Hello Edik,

Which version of MooseFS do you use?
Is it 1.6, 2.0 or 3.0?

Why do you re-share MFS via NFS? Do you really need it?
We know, that performance is poor when such operation (re-share via NFS) is done.

Also, your charts are very compressed, it is hard to see anything in legend.
Could you post a bigger image?


Best regards,

-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | moosefs.com <https://moosefs.com/>
> On 20 Aug 2015, at 11:37 am, Edik Mkoyan <edi...@tu...> wrote:
> 
> FYI
> 
> http://superuser.com/questions/960027/disk-size-varies-moosefs <http://superuser.com/questions/960027/disk-size-varies-moosefs>
> 
> Thanks
> 
> -- 
> --------
> Best,
> Edik.
> 
> This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Simonian Educational Foundation, Halabyan 16, Yerevan, Rep. of Armenia. www.tumo.org <http://www.tumo.org/>
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

[MooseFS-Users] Mounted disk size varies, huge disk size drops, high iowait

[Edik M. <edi...@tu...>]

FYI

http://superuser.com/questions/960027/disk-size-varies-moosefs

Thanks

-- 
--------
Best,
Edik.

-- 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
The sender therefore does not accept liability for any errors or omissions 
in the contents of this message, which arise as a result of e-mail 
transmission. If verification is required please request a hard-copy 
version. Simonian Educational Foundation, Halabyan 16, Yerevan, Rep. of 
Armenia. www.tumo.org

Re: [MooseFS-Users] 3.0.39 - linux client, freebsd servers, seems to fail to write data

[Aleksander W. <ale...@mo...>]

Hi Joseph
Good to hear that all is working ok.

We are interested in your performance test.
Can you send as some feedback after you get some results / observations?

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

On 18.08.2015 15:22, Joseph Love wrote:
> *Solution:*
> Put the linux client on the right network, and it works now.
>
>
> Rest of my email that I was writing up before I realized what I’d done:
>
> Sorry for the confusion, the performance issue was on a FreeBSD client
> with the FreeBSD servers, which became my drive to test some other
> clients (linux, mac).  Both the linux and mac clients do the same
> thing: connect with mfsmount, let me list and cd to different folders,
> then fail to do any file I/O.
>
> Additionally, I have realized why my linux client wasn’t working - I
> had screwed up its network settings.. it was on the wrong network.
>
> About the network setup:
> All 4 systems have Intel X520-DA2 10gbit network cards.  There are no
> firewalls turn on, and no VLANs configured.  
> They’re connected to two switches: 
> - Brocade 10gbit network switch, with 172.31.31.0/24 network, no other
> traffic here.
> - Juniper 10gbit network switch, for managing them, this is where my
> other traffic is, and where I mistakenly put the linux client.
>
>
>
> Sorry about that, it actually wasn’t until I’d started writing this
> email regarding the network setup where it finally clicked that the
> chunk servers are all expecting to only talk on the 172.31.31.0
> network, which is *not* where I put that linux client initially.
>  Moving it there solved that issue.  I’ll follow this up with some
> performance testing, and get back to my previous issue, where I feel
> like the freebsd client performance is poor.  Quick dd test on the
> linux client gives me about a 2.2x performance of what I was seeing on
> FreeBSD.
>
> So, thanks for making me think through my issue again, back to
> performance testing.
>
> -Joe
>
>> On Aug 18, 2015, at 1:22 AM, Aleksander Wieliczko
>> <ale...@mo...
>> <mailto:ale...@mo...>> wrote:
>>
>> Hi Joe!
>>
>> First of all I wouldn't call that performance issue because it's look
>> like your client have problem with chunkservers communication.
>>
>> Please check your firewall configuration.
>> Mfsclient use 9422 port to communicate with chunkservers and 9420 to
>> communicate with master.
>> Does your all moosefs components are in the same network, the same VALN?
>>
>> All ports that are used in MooseFS cluster communication are from
>> range 9419-9422
>>
>> I would like to add that default block size for dd is 512 bytes, so
>> you will not achieve good performance not only in MooseFS but in all
>> distributed file systems.
>> Try to increase block size to something like 4k, 8k, 32k, 64k 1M and
>> see the results.
>>
>> Can you tell as something more about your network configurations?
>>
>> Best regards
>> Aleksander Wieliczko
>> Technical Support Engineer
>> MooseFS.com <x-msg://30/moosefs.com>
>>
>> On 14.08.2015 21:13, Joseph Love wrote:
>>> Hi,
>>>
>>> I was testing out moosefs (again) on FreeBSD, when I ran into what appeared to be a performance issue, which I assumed was related to the client.  So, I decided to try a linux client with my FreeBSD master & chunk servers.
>>> All systems are running 3.0.39, the master & chunk servers are on FreeBSD, and the client is Debian (jessie).
>>>
>>> I can mount the moosefs share on the client, it shows me existing content.  I can traverse folders, create new folders, ‘touch’ new files, but the second I try and put data into a file, (“echo blah > test.txt” or “dd if=/dev/zero of=test.dd count=1”) the client seems to just lock-up.
>>>
>>> It did work with a FreeBSD client, but again, I think there may be a performance issue, as I couldn’t seem to exceed about 140MB/s (SSDs, 10gbit ethernet, many cores, lots of memory) hence why I wanted to try a linux client.  Ultimately I don’t care to run linux clients, but I would have liked to see a comparison in performance between the two.
>>>
>>> Any ideas why L linux client doesn’t seem to like my FreeBSD master/chunk servers?
>>>
>>> -Joe
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _________________________________________
>>> moosefs-users mailing list
>>> moo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>

Re: [MooseFS-Users] Network overhead and best method of sharing

[Aleksander W. <ale...@mo...>]

Hi Raffaello

If you use OS which supports MooseFS-client try to not use any re-share method. You will always add another layer in communication during re-share. Direct use of mfsmount will always be better.

But if you like to use MooseFS on Windows or storage for Virtualisation try to use SAMBA/iSCSI Target.
We are not advising to use NFS as re-share method.

Best Regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <http://moosefs.com/>

> On 18 Aug 2015, at 16:46, R. C. <mil...@gm...> wrote:
> 
> Hi All,
> 
> when accessing MFS from a LAN machine other than the MFS member machines, is it preferred to mount MFS on a "as near as possible" or, better, "MFS local" machine (e.g. Master or Chunkserver) and then share with NFS or directly mount MFS on the accessing client?
> What is the method that delivers better performance/less protocol overhead on the LAN?
> 
> Thank you
> 
> Raffaello
> 
> 
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

[MooseFS-Users] Network overhead and best method of sharing

[R. C. <mil...@gm...>]

Hi All,

when accessing MFS from a LAN machine other than the MFS member machines,
is it preferred to mount MFS on a "as near as possible" or, better, "MFS
local" machine (e.g. Master or Chunkserver) and then share with NFS or
directly mount MFS on the accessing client?
What is the method that delivers better performance/less protocol overhead
on the LAN?

Thank you

Raffaello

Re: [MooseFS-Users] Multi-Master setup - Failover not working

[Piotr R. K. <pio...@mo...>]

Hi Bernd,

That’s good :)


Best regards,

-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | moosefs.com <https://moosefs.com/>
> On 18 Aug 2015, at 3:44 pm, Bernd Burg <b....@ho...> wrote:
> 
> Hello Piotr Robert,
> 
> thanks for your answer. I was already told this by you in an other email.
> 
> i got 5 chunkservers and 2 Master working as separate physical maschines in my setup now.
> The tests so far were fine an i am 1 test away from purchasing
> 
> I will do another Stress Test to see a complete powerup off my cluster.
> I hope the moosfs system behaves ok.
> 
> 
> Mit freundlichen Grüßen
> 
> Bernd Burg
> 
> ------------------------------------------------------------
> HOLDE AG
> Zum Roppertsborn 14, 66646 Marpingen
> 
> Telefon   +49-6827-267988-0
> Telefax   +49-6827-267988-9
> 
> Email       b....@ho... <mailto:kr...@ac...>
> 
> Sitz der Gesellschaft: Marpingen
> AG Saarbrücken, HRB 101630
> Ust-Id-Nr.: DE29460253
> 
> Vorstand: 
> Dipl.-Ing. Bernd Burg
> 
> Aufsichtsrat: 
> Dipl.-Ing. Axel Gaus (Vorsitz)
> Dipl.-Ing. Andreas Krolzig
> Dipl.-Ing. Gabor Richter 
> ------------------------------------------------------------
> 
> Am 17.08.2015 um 19:04 schrieb Piotr Robert Konopelko:
>> Hi Bernd,
>> 
>>> I have the following questions to that
>>> 
>>> 1. Do i have to have the master software on separate hardware to the chunks ? so that master and chunk servers are physically spit systems ?
>> 
>> No, you don’t have to, but it is recommended to have machines dedicated for Master Servers.
>> 
>>> 2. If that is the case how can i deal in a 2 storage Room situation with 2 UPS Systems and Master and Chunk Server on the same ups
>>>     Do i experience the same behavior ihe if i loose one UPS System amd with that 1 server with master service and one server vith chunk service?
>> 
>> Problem you encounter is caused, because to run MooseFS with HA feature, you need to have minimum 3 chunkservers.
>> In case of one master failure (e.g. as you said - powerloss), MooseFS uses a quorum mechanism.
>> So you need to have quorum of chunkservers, i.e. a half + 1 to choose a Leader Master.
>> 
>> When you poweroff one of your two chunkservers you have, you loss quorum (2/2+1 = 2, so quorum of 2 chunkservers is 2).
>> 
>> 
>> PS: I’m sorry for so late answer, but our mailing list sometimes holds messages for some time.
>> 
>> 
>> Best regards,
>> 
>> -- 
>> Piotr Robert Konopelko
>> MooseFS Technical Support Engineer | moosefs.com <https://moosefs.com/>
>>> On 23 Jul 2015, at 6:31 pm, Bernd Burg < <mailto:b....@ho...>b....@ho... <mailto:b....@ho...>> wrote:
>>> 
>>> Hallo togehter
>>> 
>>> First of all, Thanks to sales i am able to have a test License for the Pro versions 3.0.38
>>> 
>>> i did set up 2 Servers as shared Master and Chunkserver with 5 Disks each from scratch (no Metalogger)
>>> I have 3 Proxmox Servers which i set up with moosfs-pro-clients
>>> I changed my dns-server in the way, that it displays  
>>> 
>>> -------------------------------------------------------------------
>>> root@proxmox-4:~# host mfsmaster
>>> mfsmaster.holde.lan has address 10.120.3.162
>>> mfsmaster.holde.lan has address 10.120.3.161
>>> -------------------------------------------------------------------
>>> 
>>> In this setup the system works as expected with a very good performance
>>> 
>>> If i stop the leader with mfsmaster stop the follower ist promoted to master
>>> If i start the old leader again with mfsmaster start that machine is denoted as follower
>>> 
>>> so far so good and exected.
>>> 
>>> What i don't unterstand is the following behaviour
>>> 
>>> 
>>> Status before test:
>>> ------------------------------------------------------------------------------
>>> root@mfsmaster1:~# mfssupervisor
>>> 1 (10.120.3.161:9419) : FOLLOWER (leader ip: 10.120.3.162 ; meta version: 126865 ; synchronized ; duration: 1511)
>>> 2 (10.120.3.162:9419) : LEADER (meta version: 126865 ; duration: 1504)
>>> ------------------------------------------------------------------------
>>> 
>>> mfsmaster1 is ip:xxx.161
>>> mfsmaster2 is ip:xxx.162
>>> 
>>> 
>>> If i poweroff the leader  (machine 162) Then i have this result
>>> 
>>> -------------------------------------------------------------------------------
>>> root@mfsmaster1:~# mfssupervisor
>>> 10.120.3.162: connection timed out
>>> master states:
>>> 1 (10.120.3.161:9419) : ELECT (meta version: 127149 ; synchronized ; duration: 96)
>>> 2 (10.120.3.162:9419) : DEAD
>>> -------------------------------------------------------------------------------------
>>> The cgi shows additionally the message:
>>> 
>>> ------------------------------------------------------------------
>>> Leader master server not found, but there is an elect, so make sure that all chnunkservers are running - elect should became a leader soon
>>> ------------------------------------------------------------------
>>> 
>>> This stays like this until the server is powered on again. There was no change for over 15 min then i powereup the machine (162) again 
>>> 
>>> --------------------------------------------------------------------------------------------
>>> 1 (10.120.3.161:9419) : LEADER (meta version: 127253 ; duration: 63)
>>> 2 (10.120.3.162:9419) : FOLLOWER (leader ip: 10.120.3.161 ; meta version: 127253 ; synchronized ; duration: 69)
>>> --------------------------------------------------------------------------------------------------------------
>>> 
>>> 
>>> This is in both ways
>>> 
>>> I have the following questions to that
>>> 
>>> 1. Do i have to have the master software on separate hardware to the chunks ? so that master and chunk servers are physically spit systems ?
>>> 
>>> 2. If that is the case how can i deal in a 2 storage Room situation with 2 UPS Systems and Master and Chunk Server on the same ups
>>>     Do i experience the same behavior ihe if i loose one UPS System amd with that 1 server with master service and one server vith chunk service?
>>> 
>>> 
>>> I would be very thankful if somebody could give me a hint
>>> 
>>> Thanks in  advance
>>> 
>>> -- 
>>> 
>>> Mit freundlichen Grüßen
>>> 
>>> Bernd Burg
>>> 
>>> ------------------------------------------------------------
>>> HOLDE AG
>>> Zum Roppertsborn 14, 66646 Marpingen
>>> 
>>> Telefon   +49-6827-267988-0
>>> Telefax   +49-6827-267988-9
>>> 
>>> Email       b....@ho... <mailto:kr...@ac...>
>>> 
>>> Sitz der Gesellschaft: Marpingen
>>> AG Saarbrücken, HRB 101630
>>> Ust-Id-Nr.: DE29460253
>>> 
>>> Vorstand: 
>>> Dipl.-Ing. Bernd Burg
>>> 
>>> Aufsichtsrat: 
>>> Dipl.-Ing. Axel Gaus (Vorsitz)
>>> Dipl.-Ing. Andreas Krolzig
>>> Dipl.-Ing. Gabor Richter 
>>> ------------------------------------------------------------
>>> 
>>> ------------------------------------------------------------------------------
>>> _________________________________________
>>> moosefs-users mailing list
>>> moo...@li... <mailto:moo...@li...>
>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>
>> 
> 
> 

Re: [MooseFS-Users] Multi-Master setup - Failover not working

[Bernd B. <b....@ho...>]

Hello Piotr Robert,

thanks for your answer. I was already told this by you in an other email.

i got 5 chunkservers and 2 Master working as separate physical maschines
in my setup now.
The tests so far were fine an i am 1 test away from purchasing

I will do another Stress Test to see a complete powerup off my cluster.
I hope the moosfs system behaves ok.


Mit freundlichen Grüßen


Bernd Burg

------------------------------------------------------------
HOLDE AG
Zum Roppertsborn 14, 66646 Marpingen

Telefon   +49-6827-267988-0
Telefax   +49-6827-267988-9

Email       b....@ho... <mailto:kr...@ac...>

Sitz der Gesellschaft: Marpingen
AG Saarbrücken, HRB 101630

Ust-Id-Nr.: DE29460253

Vorstand:
Dipl.-Ing. Bernd Burg

Aufsichtsrat:
Dipl.-Ing. Axel Gaus (Vorsitz)
Dipl.-Ing. Andreas Krolzig

Dipl.-Ing. Gabor Richter
------------------------------------------------------------


Am 17.08.2015 um 19:04 schrieb Piotr Robert Konopelko:
> Hi Bernd,
>
>> I have the following questions to that
>>
>> 1. Do i have to have the master software on separate hardware to the
>> chunks ? so that master and chunk servers are physically spit systems ?
>
> No, you don’t have to, but it is recommended to have machines
> dedicated for Master Servers.
>
>> 2. If that is the case how can i deal in a 2 storage Room situation
>> with 2 UPS Systems and Master and Chunk Server on the same ups
>>     Do i experience the same behavior ihe if i loose one UPS System
>> amd with that 1 server with master service and one server vith chunk
>> service?
>
> Problem you encounter is caused, because to run MooseFS with HA
> feature, you need to have minimum 3 chunkservers.
> In case of one master failure (e.g. as you said - powerloss), MooseFS
> uses a quorum mechanism.
> So you need to have quorum of chunkservers, i.e. a half + 1 to choose
> a Leader Master.
>
> When you poweroff one of your two chunkservers you have, you loss
> quorum (2/2+1 = 2, so quorum of 2 chunkservers is 2).
>
>
> PS: I’m sorry for so late answer, but our mailing list sometimes holds
> messages for some time.
>
>
> Best regards,
>
> -- 
> Piotr Robert Konopelko
> *MooseFS Technical Support Engineer* | moosefs.com <https://moosefs.com>
>
>> On 23 Jul 2015, at 6:31 pm, Bernd Burg <b....@ho...
>> <mailto:b....@ho...>> wrote:
>>
>> Hallo togehter
>>
>> First of all, Thanks to sales i am able to have a test License for
>> the Pro versions 3.0.38
>>
>> i did set up 2 Servers as shared Master and Chunkserver with 5 Disks
>> each from scratch (no Metalogger)
>> I have 3 Proxmox Servers which i set up with moosfs-pro-clients
>> I changed my dns-server in the way, that it displays  
>>
>> -------------------------------------------------------------------
>> root@proxmox-4:~# host mfsmaster
>> mfsmaster.holde.lan has address 10.120.3.162
>> mfsmaster.holde.lan has address 10.120.3.161
>> -------------------------------------------------------------------
>>
>> In this setup the system works as expected with a very good performance
>>
>> If i stop the leader with mfsmaster stop the follower ist promoted to
>> master
>> If i start the old leader again with mfsmaster start that machine is
>> denoted as follower
>>
>> so far so good and exected.
>>
>> What i don't unterstand is the following behaviour
>>
>>
>> Status before test:
>> ------------------------------------------------------------------------------
>> root@mfsmaster1:~# mfssupervisor
>> 1 (10.120.3.161:9419) : FOLLOWER (leader ip: 10.120.3.162 ; meta
>> version: 126865 ; synchronized ; duration: 1511)
>> 2 (10.120.3.162:9419) : LEADER (meta version: 126865 ; duration: 1504)
>> ------------------------------------------------------------------------
>>
>> mfsmaster1 is ip:xxx.161
>> mfsmaster2 is ip:xxx.162
>>
>>
>> If i poweroff the leader  (machine 162) Then i have this result
>>
>> -------------------------------------------------------------------------------
>> root@mfsmaster1:~# mfssupervisor
>> 10.120.3.162: connection timed out
>> master states:
>> 1 (10.120.3.161:9419) : ELECT (meta version: 127149 ; synchronized ;
>> duration: 96)
>> 2 (10.120.3.162:9419) : DEAD
>> -------------------------------------------------------------------------------------
>> The cgi shows additionally the message:
>>
>> ------------------------------------------------------------------
>> Leader master server not found, but there is an elect, so make sure
>> that all chnunkservers are running - elect should became a leader soon
>> ------------------------------------------------------------------
>>
>> This stays like this until the server is powered on again. There was
>> no change for over 15 min then i powereup the machine (162) again 
>>
>> --------------------------------------------------------------------------------------------
>> 1 (10.120.3.161:9419) : LEADER (meta version: 127253 ; duration: 63)
>> 2 (10.120.3.162:9419) : FOLLOWER (leader ip: 10.120.3.161 ; meta
>> version: 127253 ; synchronized ; duration: 69)
>> --------------------------------------------------------------------------------------------------------------
>>
>>
>> This is in both ways
>>
>> I have the following questions to that
>>
>> 1. Do i have to have the master software on separate hardware to the
>> chunks ? so that master and chunk servers are physically spit systems ?
>>
>> 2. If that is the case how can i deal in a 2 storage Room situation
>> with 2 UPS Systems and Master and Chunk Server on the same ups
>>     Do i experience the same behavior ihe if i loose one UPS System
>> amd with that 1 server with master service and one server vith chunk
>> service?
>>
>>
>> I would be very thankful if somebody could give me a hint
>>
>> Thanks in  advance
>>
>> -- 
>>
>>
>> Mit freundlichen Grüßen
>>
>>
>> Bernd Burg
>>
>> ------------------------------------------------------------
>> HOLDE AG
>> Zum Roppertsborn 14, 66646 Marpingen
>>
>> Telefon   +49-6827-267988-0
>> Telefax   +49-6827-267988-9
>>
>> Email       b....@ho... <mailto:kr...@ac...>
>>
>> Sitz der Gesellschaft: Marpingen
>> AG Saarbrücken, HRB 101630
>>
>> Ust-Id-Nr.: DE29460253
>>
>> Vorstand: 
>> Dipl.-Ing. Bernd Burg
>>
>> Aufsichtsrat: 
>> Dipl.-Ing. Axel Gaus (Vorsitz)
>> Dipl.-Ing. Andreas Krolzig
>>
>> Dipl.-Ing. Gabor Richter 
>> ------------------------------------------------------------
>>
>>
>> ------------------------------------------------------------------------------
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> <mailto:moo...@li...>
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

Re: [MooseFS-Users] 3.0.39 - linux client, freebsd servers, seems to fail to write data

[Joseph L. <jo...@ge...>]

Solution:
Put the linux client on the right network, and it works now.


Rest of my email that I was writing up before I realized what I’d done:

Sorry for the confusion, the performance issue was on a FreeBSD client with the FreeBSD servers, which became my drive to test some other clients (linux, mac).  Both the linux and mac clients do the same thing: connect with mfsmount, let me list and cd to different folders, then fail to do any file I/O.

Additionally, I have realized why my linux client wasn’t working - I had screwed up its network settings.. it was on the wrong network.

About the network setup:
All 4 systems have Intel X520-DA2 10gbit network cards.  There are no firewalls turn on, and no VLANs configured.  
They’re connected to two switches: 
- Brocade 10gbit network switch, with 172.31.31.0/24 network, no other traffic here.
- Juniper 10gbit network switch, for managing them, this is where my other traffic is, and where I mistakenly put the linux client.



Sorry about that, it actually wasn’t until I’d started writing this email regarding the network setup where it finally clicked that the chunk servers are all expecting to only talk on the 172.31.31.0 network, which is *not* where I put that linux client initially.  Moving it there solved that issue.  I’ll follow this up with some performance testing, and get back to my previous issue, where I feel like the freebsd client performance is poor.  Quick dd test on the linux client gives me about a 2.2x performance of what I was seeing on FreeBSD.

So, thanks for making me think through my issue again, back to performance testing.

-Joe

> On Aug 18, 2015, at 1:22 AM, Aleksander Wieliczko <ale...@mo...> wrote:
> 
> Hi Joe!
> 
> First of all I wouldn't call that performance issue because it's look like your client have problem with chunkservers communication.
> 
> Please check your firewall configuration.
> Mfsclient use 9422 port to communicate with chunkservers and 9420 to communicate with master.
> Does your all moosefs components are in the same network, the same VALN?
> 
> All ports that are used in MooseFS cluster communication are from range 9419-9422
> 
> I would like to add that default block size for dd is 512 bytes, so you will not achieve good performance not only in MooseFS but in all distributed file systems.
> Try to increase block size to something like 4k, 8k, 32k, 64k 1M and see the results.
> 
> Can you tell as something more about your network configurations?
> 
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <x-msg://30/moosefs.com>
> 
> On 14.08.2015 21:13, Joseph Love wrote:
>> Hi,
>> 
>> I was testing out moosefs (again) on FreeBSD, when I ran into what appeared to be a performance issue, which I assumed was related to the client.  So, I decided to try a linux client with my FreeBSD master & chunk servers.
>> All systems are running 3.0.39, the master & chunk servers are on FreeBSD, and the client is Debian (jessie).
>> 
>> I can mount the moosefs share on the client, it shows me existing content.  I can traverse folders, create new folders, ‘touch’ new files, but the second I try and put data into a file, (“echo blah > test.txt” or “dd if=/dev/zero of=test.dd count=1”) the client seems to just lock-up.
>> 
>> It did work with a FreeBSD client, but again, I think there may be a performance issue, as I couldn’t seem to exceed about 140MB/s (SSDs, 10gbit ethernet, many cores, lots of memory) hence why I wanted to try a linux client.  Ultimately I don’t care to run linux clients, but I would have liked to see a comparison in performance between the two.
>> 
>> Any ideas why L linux client doesn’t seem to like my FreeBSD master/chunk servers?
>> 
>> -Joe
>> 
>> 
>> ------------------------------------------------------------------------------
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li... <mailto:moo...@li...>
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>
> 

Re: [MooseFS-Users] 3.0.39 - linux client, freebsd servers, seems to fail to write data

[Aleksander W. <ale...@mo...>]

Hi Joe!

First of all I wouldn't call that performance issue because it's look
like your client have problem with chunkservers communication.

Please check your firewall configuration.
Mfsclient use 9422 port to communicate with chunkservers and 9420 to
communicate with master.
Does your all moosefs components are in the same network, the same VALN?

All ports that are used in MooseFS cluster communication are from range
9419-9422

I would like to add that default block size for dd is 512 bytes, so you
will not achieve good performance not only in MooseFS but in all
distributed file systems.
Try to increase block size to something like 4k, 8k, 32k, 64k 1M and see
the results.

Can you tell as something more about your network configurations?

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

On 14.08.2015 21:13, Joseph Love wrote:
> Hi,
>
> I was testing out moosefs (again) on FreeBSD, when I ran into what appeared to be a performance issue, which I assumed was related to the client.  So, I decided to try a linux client with my FreeBSD master & chunk servers.
> All systems are running 3.0.39, the master & chunk servers are on FreeBSD, and the client is Debian (jessie).
>
> I can mount the moosefs share on the client, it shows me existing content.  I can traverse folders, create new folders, ‘touch’ new files, but the second I try and put data into a file, (“echo blah > test.txt” or “dd if=/dev/zero of=test.dd count=1”) the client seems to just lock-up.
>
> It did work with a FreeBSD client, but again, I think there may be a performance issue, as I couldn’t seem to exceed about 140MB/s (SSDs, 10gbit ethernet, many cores, lots of memory) hence why I wanted to try a linux client.  Ultimately I don’t care to run linux clients, but I would have liked to see a comparison in performance between the two.
>
> Any ideas why L linux client doesn’t seem to like my FreeBSD master/chunk servers?
>
> -Joe
>
>
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [MooseFS-Users] Multi-Master setup - Failover not working

[Piotr R. K. <pio...@mo...>]

Hi Bernd,

> I have the following questions to that
> 
> 1. Do i have to have the master software on separate hardware to the chunks ? so that master and chunk servers are physically spit systems ?

No, you don’t have to, but it is recommended to have machines dedicated for Master Servers.

> 2. If that is the case how can i deal in a 2 storage Room situation with 2 UPS Systems and Master and Chunk Server on the same ups
>     Do i experience the same behavior ihe if i loose one UPS System amd with that 1 server with master service and one server vith chunk service?

Problem you encounter is caused, because to run MooseFS with HA feature, you need to have minimum 3 chunkservers.
In case of one master failure (e.g. as you said - powerloss), MooseFS uses a quorum mechanism.
So you need to have quorum of chunkservers, i.e. a half + 1 to choose a Leader Master.

When you poweroff one of your two chunkservers you have, you loss quorum (2/2+1 = 2, so quorum of 2 chunkservers is 2).


PS: I’m sorry for so late answer, but our mailing list sometimes holds messages for some time.


Best regards,

-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | moosefs.com <https://moosefs.com/>
> On 23 Jul 2015, at 6:31 pm, Bernd Burg <b....@ho...> wrote:
> 
> Hallo togehter
> 
> First of all, Thanks to sales i am able to have a test License for the Pro versions 3.0.38
> 
> i did set up 2 Servers as shared Master and Chunkserver with 5 Disks each from scratch (no Metalogger)
> I have 3 Proxmox Servers which i set up with moosfs-pro-clients
> I changed my dns-server in the way, that it displays  
> 
> -------------------------------------------------------------------
> root@proxmox-4:~# host mfsmaster
> mfsmaster.holde.lan has address 10.120.3.162
> mfsmaster.holde.lan has address 10.120.3.161
> -------------------------------------------------------------------
> 
> In this setup the system works as expected with a very good performance
> 
> If i stop the leader with mfsmaster stop the follower ist promoted to master
> If i start the old leader again with mfsmaster start that machine is denoted as follower
> 
> so far so good and exected.
> 
> What i don't unterstand is the following behaviour
> 
> 
> Status before test:
> ------------------------------------------------------------------------------
> root@mfsmaster1:~# mfssupervisor
> 1 (10.120.3.161:9419) : FOLLOWER (leader ip: 10.120.3.162 ; meta version: 126865 ; synchronized ; duration: 1511)
> 2 (10.120.3.162:9419) : LEADER (meta version: 126865 ; duration: 1504)
> ------------------------------------------------------------------------
> 
> mfsmaster1 is ip:xxx.161
> mfsmaster2 is ip:xxx.162
> 
> 
> If i poweroff the leader  (machine 162) Then i have this result
> 
> -------------------------------------------------------------------------------
> root@mfsmaster1:~# mfssupervisor
> 10.120.3.162: connection timed out
> master states:
> 1 (10.120.3.161:9419) : ELECT (meta version: 127149 ; synchronized ; duration: 96)
> 2 (10.120.3.162:9419) : DEAD
> -------------------------------------------------------------------------------------
> The cgi shows additionally the message:
> 
> ------------------------------------------------------------------
> Leader master server not found, but there is an elect, so make sure that all chnunkservers are running - elect should became a leader soon
> ------------------------------------------------------------------
> 
> This stays like this until the server is powered on again. There was no change for over 15 min then i powereup the machine (162) again 
> 
> --------------------------------------------------------------------------------------------
> 1 (10.120.3.161:9419) : LEADER (meta version: 127253 ; duration: 63)
> 2 (10.120.3.162:9419) : FOLLOWER (leader ip: 10.120.3.161 ; meta version: 127253 ; synchronized ; duration: 69)
> --------------------------------------------------------------------------------------------------------------
> 
> 
> This is in both ways
> 
> I have the following questions to that
> 
> 1. Do i have to have the master software on separate hardware to the chunks ? so that master and chunk servers are physically spit systems ?
> 
> 2. If that is the case how can i deal in a 2 storage Room situation with 2 UPS Systems and Master and Chunk Server on the same ups
>     Do i experience the same behavior ihe if i loose one UPS System amd with that 1 server with master service and one server vith chunk service?
> 
> 
> I would be very thankful if somebody could give me a hint
> 
> Thanks in  advance
> 
> -- 
> 
> Mit freundlichen Grüßen
> 
> Bernd Burg
> 
> ------------------------------------------------------------
> HOLDE AG
> Zum Roppertsborn 14, 66646 Marpingen
> 
> Telefon   +49-6827-267988-0
> Telefax   +49-6827-267988-9
> 
> Email       b....@ho... <mailto:kr...@ac...>
> 
> Sitz der Gesellschaft: Marpingen
> AG Saarbrücken, HRB 101630
> Ust-Id-Nr.: DE29460253
> 
> Vorstand: 
> Dipl.-Ing. Bernd Burg
> 
> Aufsichtsrat: 
> Dipl.-Ing. Axel Gaus (Vorsitz)
> Dipl.-Ing. Andreas Krolzig
> Dipl.-Ing. Gabor Richter 
> ------------------------------------------------------------
> 
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li... <mailto:moo...@li...>
> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>

[MooseFS-Users] 3.0.39 - linux client, freebsd servers, seems to fail to write data

[Joseph L. <jo...@ge...>]

Hi,

I was testing out moosefs (again) on FreeBSD, when I ran into what appeared to be a performance issue, which I assumed was related to the client.  So, I decided to try a linux client with my FreeBSD master & chunk servers.
All systems are running 3.0.39, the master & chunk servers are on FreeBSD, and the client is Debian (jessie).

I can mount the moosefs share on the client, it shows me existing content.  I can traverse folders, create new folders, ‘touch’ new files, but the second I try and put data into a file, (“echo blah > test.txt” or “dd if=/dev/zero of=test.dd count=1”) the client seems to just lock-up.

It did work with a FreeBSD client, but again, I think there may be a performance issue, as I couldn’t seem to exceed about 140MB/s (SSDs, 10gbit ethernet, many cores, lots of memory) hence why I wanted to try a linux client.  Ultimately I don’t care to run linux clients, but I would have liked to see a comparison in performance between the two.

Any ideas why L linux client doesn’t seem to like my FreeBSD master/chunk servers?

-Joe

Re: [MooseFS-Users] MooseFS upgrade plan 1.6 -> 3.0

[Piotr R. K. <pio...@mo...>]

Hello Casper,

there are some aspects I would like to pay your attention to:

We do not recommend and support ucarp failover solution - neither officially, nor unofficially.
It may lead to split-brain and/or rollback metadata to a version which is not the latest one.
And in consequence - to data loss.

We had such scenario in one of our client’s environment.


Please be aware, that version 3.0.x you want to upgrade to is not a stable version yet.
The 3.0 version is not recommended for production environments yet.


We support upgrade only from version 1.6.27-5, not 1.6.25, as mentioned in our document about the upgrade
("MooseFS Upgrade Guide”, you can find it here: https://moosefs.com/documentation/moosefs-2-0.html <https://moosefs.com/documentation/moosefs-2-0.html>).


I’d like to talk/chat with you e.g. via Skype about whole upgrade scenario, because here I mentioned only the most important aspects.
Would it be possible? Could you please send me in priv. message your Skype ID?



Best regards,

-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | moosefs.com <https://moosefs.com/>
> On 12 Aug 2015, at 10:28 pm, Casper Langemeijer <ca...@la...> wrote:
> 
> Hi All,
> 
> I'm planning for the upgrade of our MooseFS system. I've waited much too 
> long as we're still at 1.6.25.
> 
> One of the main reasons that we've waited so long is because we do not 
> like downtime. To minimize downtime I've come up with the 9 step plan 
> below, and if you have the time I would really appreciate if you can 
> have a look at it. I want to know it this makes any sense. Some of you 
> might even learn from some of the clever tricks I pull. Instead of an 
> in-place upgrade of the servers I plan to create a new master, copy the 
> master data to a new master and do a ucarp failover.
> 
> The other reason I like this plan is that right after the failover I can 
> test the system and if I see something I do not like I can rollback to 
> the previous master. Only step 6 is the point of no return.
> 
> 
> I have two assumptions that I'm unsure about.
> 
> 1. mfsmaster version 3.0.39 can read the metadata.mfs written by 1.6.25 
> mfsmetarestore -a
> 2. 1.6.25 metaloggers, chunkservers and clients automatically reconnect 
> correctly to a mfsmaster 'failover' with new master version 3.0.39 
> without problems.
> 
> If these steps all work, the filesystem is almost fully available for 
> reading during the upgrade.
> The exceptions to these are of a very small window (subsecond) to 
> remount in step 2 and 6, and a bigger window where all the clients and 
> chunkservers need to detect master failure and reconnect to the new 
> upgraded master. I expect this to take a few seconds.
> 
> The filesystem is closed for writes from step 2 to 6. On my system I 
> roughly estimate this (step 3 and 4) to take about 5-10 minutes. RAM 
> used by my 1.6.25 mfsmaster is 11GiB, I run on commodity server hardware.
> 
> 
> What do you think of this plan?
> 
> Step 1:
> Install a new MooseFS master server, latest version (as time of writing 
> 3.0.39)
> with cgi server too! ucarp config same as current primary master. (ucarp 
> failover will make this new server the new mfsmaster when current master 
> is shut down)
> 
> Step 2:
> On all clients, change mfsclient into readonly mode.
> 
> This is done by doing a lazy umount, immediately followed by a readonly 
> mount. This means that any open files will keep using the read-write 
> mount, all new files are opened throught the read-only mount. When all 
> opened files on the old read-write mount have been closed, the 
> read-write mfsmount process terminates. We monitor it with 'ps x | grep 
> rw,mfsmaste[r]' and we wait until it has terminated.
> 
> /usr/bin/lsof /mnt/mfs; /bin/umount -l /mnt/mfs; /usr/bin/mfsmount 
> /mnt/mfs -o ro,mfsmaster=10.1.7.1,mfsport=9421,dev,suid
> echo -n "Waiting for rw mfsmount to stop: "; while ps auxf | grep 
> rw,mfsmaste[r] > /dev/null; do echo -n "."; sleep 1; done; echo " done";
> 
> If this takes too long we could kill the processes that had open files 
> (that is why I did a lsof /mnt/mfs before we umounted)
> 
> Step 3:
> On a metalogger server metarestore the backups to a new metadata.mfs 
> file, and copy that file to the new MooseFS master server. 
> mfsmetarestore is part of the mfsmaster package, so it's best to do this 
> on the secondary (or standby) master server.
> 
> /usr/sbin/mfsmetalogger stop
> /usr/sbin/mfsmetarestore -a
> /usr/bin/scp /var/lib/mfs/metadata.mfs root@10.1.8.100:/var/lib/mfs/
> 
> Step 4:
> On the new MooseFS master server start mfsmaster with the metadata.mfs 
> created in step 2.
> 
> /usr/sbin/mfsmaster start
> 
> Step 5:
> Do an ip failover by shutting down the network connections to the first 
> MooseFS server (ucarp should handle the rest)
> 
> We now should now have a functioning MooseFS cluster again, with a 
> 3.0.39 master and 1.6.25 metaloggers, chunkservers and clients.
> 
> Step 6:
> On all clients, change mfsclient from readonly mode to read-write. Lazy 
> again, which makes it faster.
> 
> /bin/umount -l /mnt/mfs; /bin/mount /mnt/mfs
> 
> Step 7:
> Upgrade metaloggers to 3.0.39
> 
> Step 8:
> Upgrade chunkservers to 3.0.39
> 
> Step 9:
> Upgrade clients to 3.0.39
> 
> 
> Thank you for taking the time to read this.
> 
> Greetings, Casper
> 
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

[MooseFS-Users] MooseFS upgrade plan 1.6 -> 3.0

[Casper L. <ca...@la...>]

Hi All,

I'm planning for the upgrade of our MooseFS system. I've waited much too 
long as we're still at 1.6.25.

One of the main reasons that we've waited so long is because we do not 
like downtime. To minimize downtime I've come up with the 9 step plan 
below, and if you have the time I would really appreciate if you can 
have a look at it. I want to know it this makes any sense. Some of you 
might even learn from some of the clever tricks I pull. Instead of an 
in-place upgrade of the servers I plan to create a new master, copy the 
master data to a new master and do a ucarp failover.

The other reason I like this plan is that right after the failover I can 
test the system and if I see something I do not like I can rollback to 
the previous master. Only step 6 is the point of no return.


I have two assumptions that I'm unsure about.

1. mfsmaster version 3.0.39 can read the metadata.mfs written by 1.6.25 
mfsmetarestore -a
2. 1.6.25 metaloggers, chunkservers and clients automatically reconnect 
correctly to a mfsmaster 'failover' with new master version 3.0.39 
without problems.

If these steps all work, the filesystem is almost fully available for 
reading during the upgrade.
The exceptions to these are of a very small window (subsecond) to 
remount in step 2 and 6, and a bigger window where all the clients and 
chunkservers need to detect master failure and reconnect to the new 
upgraded master. I expect this to take a few seconds.

The filesystem is closed for writes from step 2 to 6. On my system I 
roughly estimate this (step 3 and 4) to take about 5-10 minutes. RAM 
used by my 1.6.25 mfsmaster is 11GiB, I run on commodity server hardware.


What do you think of this plan?

Step 1:
Install a new MooseFS master server, latest version (as time of writing 
3.0.39)
with cgi server too! ucarp config same as current primary master. (ucarp 
failover will make this new server the new mfsmaster when current master 
is shut down)

Step 2:
On all clients, change mfsclient into readonly mode.

This is done by doing a lazy umount, immediately followed by a readonly 
mount. This means that any open files will keep using the read-write 
mount, all new files are opened throught the read-only mount. When all 
opened files on the old read-write mount have been closed, the 
read-write mfsmount process terminates. We monitor it with 'ps x | grep 
rw,mfsmaste[r]' and we wait until it has terminated.

/usr/bin/lsof /mnt/mfs; /bin/umount -l /mnt/mfs; /usr/bin/mfsmount 
/mnt/mfs -o ro,mfsmaster=10.1.7.1,mfsport=9421,dev,suid
echo -n "Waiting for rw mfsmount to stop: "; while ps auxf | grep 
rw,mfsmaste[r] > /dev/null; do echo -n "."; sleep 1; done; echo " done";

If this takes too long we could kill the processes that had open files 
(that is why I did a lsof /mnt/mfs before we umounted)

Step 3:
On a metalogger server metarestore the backups to a new metadata.mfs 
file, and copy that file to the new MooseFS master server. 
mfsmetarestore is part of the mfsmaster package, so it's best to do this 
on the secondary (or standby) master server.

/usr/sbin/mfsmetalogger stop
/usr/sbin/mfsmetarestore -a
/usr/bin/scp /var/lib/mfs/metadata.mfs root@10.1.8.100:/var/lib/mfs/

Step 4:
On the new MooseFS master server start mfsmaster with the metadata.mfs 
created in step 2.

/usr/sbin/mfsmaster start

Step 5:
Do an ip failover by shutting down the network connections to the first 
MooseFS server (ucarp should handle the rest)

We now should now have a functioning MooseFS cluster again, with a 
3.0.39 master and 1.6.25 metaloggers, chunkservers and clients.

Step 6:
On all clients, change mfsclient from readonly mode to read-write. Lazy 
again, which makes it faster.

/bin/umount -l /mnt/mfs; /bin/mount /mnt/mfs

Step 7:
Upgrade metaloggers to 3.0.39

Step 8:
Upgrade chunkservers to 3.0.39

Step 9:
Upgrade clients to 3.0.39


Thank you for taking the time to read this.

Greetings, Casper

Re: [MooseFS-Users] a problem replace now use metadata log server machine to new one

[Krzysztof K. <krz...@mo...>]

Hi,

It’s perfectly supported to connect more than one mfsmetalogger to mfsmaster, so you can connect new metalogger, wait for metadata synchronisation and disconnect the old one.


> On 05 Aug 2015, at 05:55, 杨志敏 <zhi...@cn...> wrote:
> 
> 
>   Sir:
>          I now use a -2.0.60 moosefs system, and now there is a problem, I would like to
>  now Metadatalog server for a new server,what  I should do, whether i can stop the
>  service on the original server, and then  the new metadatalog server the
>  metadatalog service to start adding to the original moosefs system.
>   
> Version:2.0.640
> OS:centos6.5
> File System:ext4
> TKS!!
> 
> ____________________
> 
> yang hugo
> ===============================================
> 本邮件及其附件含有元征公司的保密信息，仅限于发送给上面地址中
> 列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于
> 全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收
> 了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
> 
> This e-mail and its attachments contain confidential information
> from LAUNCH,which is intended only for the person or entity whose
> address is listed above.
> Any use of the information contained herein in any way (including,
> but notlimited to, total or partial disclosure, reproduction, or
> dissemination) by persons other than the intended recipient(s)
> is prohibited.
> If you receive this e-mail in error, please notify the sender 
> by phone or email immediately and delete it! 
> 
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li... <mailto:moo...@li...>
> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>
Best Regards,
Krzysztof Kielak
Director of Operations and Customer Support
Mobile: +48 601 476 440

[MooseFS-Users] a problem replace now use metadata log server machine to new one

[杨志敏 <zhi...@cn...>]

  Sir:
         I now use a -2.0.60 moosefs system, and now there is a problem, I would like to
 now Metadatalog server for a new server,what  I should do, whether i can stop the
 service on the original server, and then  the new metadatalog server the
 metadatalog service to start adding to the original moosefs system.

Version:2.0.640
OS:centos6.5
File System:ext4
TKS!!

____________________

yang hugo
===============================================

本邮件及其附件含有元征公司的保密信息，仅限于发送给上面地址中
列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于
全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收
了本邮件，请您立即电话或邮件通知发件人并删除本邮件！

This e-mail and its attachments contain confidential information
from LAUNCH,which is intended only for the person or entity whose
address is listed above.
Any use of the information contained herein in any way (including,
but notlimited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient(s)
is prohibited.
If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!

[MooseFS-Users] About the MFS du - sh do not match the size of the actual directory;

[F. <748...@qq...>]

Hello, we use the MFS to store small files that are static, but found a problem, we are a directory of the file size may not be enough 100 k, but in the upper du - sh found the directory size is 1.1 m, could you tell me the size of the directory in the MFS is determined by what or cause, will store the information, how directory the assembly. We applied many of these directories, excuse me, how can we solve or what solution is better.
There is a problem, can you tell me our chunkserver is based on the Lvm, want to ask next chunker how to obtain the size of the size
The results of our experiment is as follows:



[root@ftp-1-1 testdir]# stat ../testdir
  File: "../testdir"
  Size: 0               Blocks: 0          IO Block: 65536  目录
Device: 13h/19d Inode: 471775      Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1001/   mypep)   Gid: ( 1001/   mypep)
Access: 2015-07-30 14:09:57.000000000 +0800
Modify: 2015-07-30 14:09:57.000000000 +0800
Change: 2015-07-30 14:09:57.000000000 +0800
[root@ftp-1-1 testdir]# echo "a" >test			#我只echo一个字节
[root@ftp-1-1 testdir]# stat ../testdir
  File: "../testdir"							#目录的大小，却增加了200
  Size: 200             Blocks: 1          IO Block: 65536  目录
Device: 13h/19d Inode: 471775      Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1001/   mypep)   Gid: ( 1001/   mypep)
Access: 2015-07-30 14:09:57.000000000 +0800
Modify: 2015-07-30 14:10:18.000000000 +0800
Change: 2015-07-30 14:10:18.000000000 +0800
[root@ftp-1-1 testdir]# echo "ab" >test 			#如果echo 两个字节
[root@ftp-1-1 testdir]# stat ../testdir
  File: "../testdir"
  Size: 300             Blocks: 1          IO Block: 65536  目录大小就增加到了300个b	这个地方有待考究
Device: 13h/19d Inode: 471775      Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1001/   mypep)   Gid: ( 1001/   mypep)
Access: 2015-07-30 14:09:57.000000000 +0800
Modify: 2015-07-30 14:10:18.000000000 +0800
Change: 2015-07-30 14:10:18.000000000 +0800

Re: [MooseFS-Users] all chunk servers disconnected after master server restart

[Krzysztof K. <krz...@mo...>]

Hello Dimitry,

Can you send some logs from mfschunkservers and mfsmaster?

> On 25 Jul 2015, at 18:56, Дмитрий Коржевин <dko...@gm...> wrote:
> 
> Hi, 
> 
> I use 5 servers:
> 
> 1 master
> 1 meta
> 3 chunk servers
> 
> All servers with CentOS x86_64 and moosefs 2.0.72 latest stable
> 
> I restarted master server, and after restart I see, that all chunk servers disconnected. I can't reconnect them:
> 
> http://dpaste.com/3D526MW <http://dpaste.com/3D526MW>
> 
> [root@master1 mfs]# mfscli -S -SCS
> +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> |                                                                              Chunk Servers                                                                              |
> +--------------+------+------+-------------+----------+-----------------+------------------------------------------------+------------------------------------------------+
> |              |      |      |             |          |                 |              'regular' hdd space               |         'marked for removal' hdd space         |
> |   ip/host    | port |  id  |   version   |   load   |   maintenance   +------------+----------+-----------+------------+------------+----------+-----------+------------+
> |              |      |      |             |          |                 |   chunks   |   used   |   total   |   % used   |   chunks   |   used   |   total   |   % used   |
> +--------------+------+------+-------------+----------+-----------------+------------+----------+-----------+------------+------------+----------+-----------+------------+
> +--------------+------+------+-------------+----------+-----------------+------------+----------+-----------+------------+------------+----------+-----------+------------+
> |                                                                           disconnected servers                                                                          |
> +--------------+------+------+-------------+----------------------------------------------------------------+-------------------------------------------------------------+
> |   ip/host    | port | csid | maintenance |                   change maintenance command                   |                        remove command                       |
> +--------------+------+------+-------------+----------------------------------------------------------------+-------------------------------------------------------------+
> | 10.100.80.65 | 9422 |    1 |         off | /usr/bin/mfscli -H mfsmaster -P 9421 -CM1/10.100.80.65/9422 <http://10.100.80.65/9422>    | /usr/bin/mfscli -H mfsmaster -P 9421 -CRC/10.100.80.65/9422 <http://10.100.80.65/9422> |
> | 10.100.80.66 | 9422 |    2 |         off | /usr/bin/mfscli -H mfsmaster -P 9421 -CM1/10.100.80.66/9422 <http://10.100.80.66/9422>    | /usr/bin/mfscli -H mfsmaster -P 9421 -CRC/10.100.80.66/9422 <http://10.100.80.66/9422> |
> | 10.100.80.67 | 9422 |    3 |         off | /usr/bin/mfscli -H mfsmaster -P 9421 -CM1/10.100.80.67/9422 <http://10.100.80.67/9422>    | /usr/bin/mfscli -H mfsmaster -P 9421 -CRC/10.100.80.67/9422 <http://10.100.80.67/9422> |
> +--------------+------+------+-------------+----------------------------------------------------------------+-------------------------------------------------------------+
> [root@master1 mfs]#
> 
> 
> Please, advice - how to reconnect them
> 
> Thank you,
> 
> Dmitry
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Best Regards,
Krzysztof Kielak
Director of Operations and Customer Support
Mobile: +48 601 476 440

Re: [MooseFS-Users] MooseFS on OpenBSD

[Tom I. H. <ti...@ha...>]

"Neddy, NH. Nam" <na...@nd...> writes:

> Hi, I accidentally lately read about fuse(4) is supported in OpenBSD
> (since 5.4), and think to try moosefs on OpenBSD-current in
> later. What's your thought about it? Besides for fun part, can OpenBSD
> be a choice of moosefs production OS as others?

I suspect you won't get Gemius / Core Technology to encourage using
MooseFS on OpenBSD in production, since it's not one of their officially
supported platforms.  On the other hand, FreeBSD is supported, and I can
confirm that MooseFS compiles, and works perfectly, without any changes
at all, on NetBSD.  Therefore, I would be quite surprised if it didn't
also turn out to do the same running on OpenBSD.  How much testing you
should do before deciding to trust it with production data is up to you.

-tih
-- 
Popularity is the hallmark of mediocrity.  --Niles Crane, "Frasier"

Re: [MooseFS-Users] all chunk servers disconnected after master server restart

[Tru H. <tr...@pa...>]

On Sat, Jul 25, 2015 at 07:56:33PM +0300, Дмитрий Коржевин wrote:
> Hi,
> 
> I use 5 servers:
> 
> 1 master
> 1 meta
> 3 chunk servers
> 
> All servers with CentOS x86_64 and moosefs 2.0.72 latest stable
"same" setup here
1 master/2 metaloggers/13 chunkservers
> 
> I restarted master server, and after restart I see, that all chunk servers
> disconnected. I can't reconnect them:
> 
> http://dpaste.com/3D526MW
...
what does "/sbin/service moosefs-chunkserver {status|restart}" commands
achieve? Anything on the /var/log/messages of the chunkservers? master?
iptables on the way? can you reach mfsmaster from the chunkservers?

Cheers

Tru

-- 
Dr Tru Huynh          | http://www.pasteur.fr/research/bis           
mailto:tr...@pa... | tel/fax +33 1 45 68 87 37/19
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France  

[MooseFS-Users] MooseFS on OpenBSD

[Neddy, N. N. <na...@nd...>]

Hi, I accidentally lately read about fuse(4) is supported in OpenBSD (since
5.4), and think to try moosefs on OpenBSD-current in later. What's your
thought about it? Besides for fun part, can OpenBSD be a choice of moosefs
production OS as others?

[MooseFS-Users] all chunk servers disconnected after master server restart

[Дмитрий К. <dko...@gm...>]

Hi,

I use 5 servers:

1 master
1 meta
3 chunk servers

All servers with CentOS x86_64 and moosefs 2.0.72 latest stable

I restarted master server, and after restart I see, that all chunk servers
disconnected. I can't reconnect them:

http://dpaste.com/3D526MW

[root@master1 mfs]# mfscli -S -SCS
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
Chunk
Servers
|
+--------------+------+------+-------------+----------+-----------------+------------------------------------------------+------------------------------------------------+
|              |      |      |             |          |
|              'regular' hdd space               |         'marked for
removal' hdd space         |
|   ip/host    | port |  id  |   version   |   load   |   maintenance
+------------+----------+-----------+------------+------------+----------+-----------+------------+
|              |      |      |             |          |                 |
chunks   |   used   |   total   |   % used   |   chunks   |   used   |
total   |   % used   |
+--------------+------+------+-------------+----------+-----------------+------------+----------+-----------+------------+------------+----------+-----------+------------+
+--------------+------+------+-------------+----------+-----------------+------------+----------+-----------+------------+------------+----------+-----------+------------+
|
disconnected
servers
|
+--------------+------+------+-------------+----------------------------------------------------------------+-------------------------------------------------------------+
|   ip/host    | port | csid | maintenance |                   change
maintenance command                   |                        remove
command                       |
+--------------+------+------+-------------+----------------------------------------------------------------+-------------------------------------------------------------+
| 10.100.80.65 | 9422 |    1 |         off | /usr/bin/mfscli -H mfsmaster
-P 9421 -CM1/10.100.80.65/9422    | /usr/bin/mfscli -H mfsmaster -P 9421
-CRC/10.100.80.65/9422 |
| 10.100.80.66 | 9422 |    2 |         off | /usr/bin/mfscli -H mfsmaster
-P 9421 -CM1/10.100.80.66/9422    | /usr/bin/mfscli -H mfsmaster -P 9421
-CRC/10.100.80.66/9422 |
| 10.100.80.67 | 9422 |    3 |         off | /usr/bin/mfscli -H mfsmaster
-P 9421 -CM1/10.100.80.67/9422    | /usr/bin/mfscli -H mfsmaster -P 9421
-CRC/10.100.80.67/9422 |
+--------------+------+------+-------------+----------------------------------------------------------------+-------------------------------------------------------------+
[root@master1 mfs]#


Please, advice - how to reconnect them

Thank you,

Dmitry

Re: [MooseFS-Users] 回复: Re: mfs_master正点失去响应

[Aleksander W. <ale...@mo...>]

Hi.
This is not a problem. It's rather info than an error. This message
means, that during data sending through the socket the connection has
been closed by other side. It usually means that there were connection
timeout. This kind of messages can appear in highly loaded network, but
this will not cause any data missing. When system reconnect then such
packet will be send again.


Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>
On 24.07.2015 08:20, 刘亚磊 wrote:
> 你好：
>         根据提示，修改系统内核后，这个问题解决了。但是现在有个新问题，
> 正点的时候，master会报错
> Jul 23 20:01:16 mfsmaster1 mfsmaster[22443]: main master server
> module: (ip:192.168.1.46) write error: EPIPE (Broken pipe)
>
> ------------------------------------------------------------------------
> 刘亚磊 |  买卖宝信息技术有限公司
> 北京市朝阳区红军营南 路傲城融富中心C座三层（100012） 
> 直线: (86) 10 56716100-8995
> 电子邮件: liu...@eb... | 移动电话: (86) 18801039545
>
>      
>     *发件人：* Jakub Kruszona-Zawadzki <mailto:jak...@ge...>
>     *发送时间：* 2015-07-24 13:29
>     *收件人：* 刘亚磊 <mailto:liu...@eb...>
>     *抄送：* moosefs-users <mailto:moo...@li...>
>     *主题：* Re: [MooseFS-Users] mfs_master正点失去响应
>     This is caused by check for available memory in Linux. Linux
>     before "fork" checks if it's enough memory for "two" copies of
>     forking process (which is rather stupid because memory is
>     duplicated in COW mode, so usually both processes shares most of
>     their memory). To "fix" this you can change this behaviour to
>     "classic" using this command (as root):
>
>     echo "1" > /proc/sys/vm/overcommit_memory
>
>     On 22 Jul, 2015, at 3:27, 刘亚磊 <liu...@eb...
>     <mailto:liu...@eb...>> wrote:
>
>>     mfs版本：2.0.72社区版
>>     master、chunkserver、dataserver、client操作系统：centso 5.9 x64
>>
>>     问题描述：
>>     文件数量千万级，发现mfs集群master每到正点会失去响应1-2分钟。
>>     master内存、cpu、硬盘、网络监控正常。最开始使用的是1.6.25版本，怀
>>     疑软件自身存 在bug，后来升级到2.0.72 社区版，问题依然存在。以下是
>>     正点的错误日志：
>>
>>
>>     Jul 22 07:00:00 mfsmaster1 mfsmaster[22443]: fork error (store
>>     data in foreground - it will block master for a while): ENOMEM
>>     (Cannot allocate memory) 
>>     Jul 22 07:01:47 mfsmaster1 mfsmaster[22443]: csdb: found cs using
>>     ip:port and csid (192.168.1.82:9422,5), but server is still
>>     connected 
>>     Jul 22 07:01:47 mfsmaster1 mfsmaster[22443]: can't accept
>>     chunkserver (ip: 192.168.1.82 / port: 9422)
>>
>>     ------------------------------------------------------------------------
>>     刘亚磊 |  买卖宝信息技术有限公司
>>     北京市朝阳区红军营南路傲城融富中心C座三层（100012） 
>>     直线: (86) 10 56716100-8995
>>     电子邮件: liu...@eb... <mailto:liu...@eb...> | 移动电
>>     话: (86) 18801039545
>>     ------------------------------------------------------------------------------
>>     _________________________________________
>>     moosefs-users mailing list
>>     moo...@li...
>>     <mailto:moo...@li...>
>>     https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>     -- 
>     Regards,
>     Jakub Kruszona-Zawadzki
>     - - - - - - - - - - - - - - - -
>     Segmentation fault (core dumped)
>     Phone: +48 602 212 039
>
>
>
> ------------------------------------------------------------------------------
>
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users