Re: [MooseFS-Users] high packet travel time

[Aleksander W. <ale...@mo...>]

Hi,

OK. Now all is clear for me.

So I will talk to our developers and return to this topic tomorrow.


Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

On 08/24/2016 03:02 PM, Wilson, Steven M wrote:
>
> Hi Aleksander,
>
>
> I wasn't very clear in my message.  We aren't using a "pro"
> installation with a leader master and follower masters.  I meant that
> we have multiple MooseFS data clusters, each with their own
> chunkservers, metaloggers, and master server.  A typical client in our
> environment will have MooseFS file systems mounted from three or four
> different MooseFS master servers at the same time.
>
>
> Best regards,
> Steve
>
>
> ------------------------------------------------------------------------
> *From:* awi...@ge... <awi...@ge...> on behalf of
> Aleksander Wieliczko <ale...@mo...>
> *Sent:* Wednesday, August 24, 2016 8:55 AM
> *To:* Wilson, Steven M
> *Cc:* MooseFS-Users
> *Subject:* Re: [MooseFS-Users] high packet travel time
>  
>
> Hi,
>
> Thank you for all this information.
>
>
> I would like to add that all clients and chunkservers always
> communicate only with LEADER MASTER.
>
>
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <moosefs.com>
>
> On 08/24/2016 02:48 PM, Wilson, Steven M wrote:
>>
>> ​Hi,
>>
>>
>> Thanks for your response.  All our MooseFS masters are running on
>> physical hardware (not a VM).  The "high packet travel" message is
>> primarily seen on systems located in other buildings and on different
>> networks (network traffic must go through two routers besides
>> switches).  The master servers don't approach 100% CPU utilization so
>> I think the problem must be a network-related issue.  And I've not
>> seen any network interface errors.  I will take a look to see if
>> there's anything we can do to improve our network performance.
>>
>>
>> Regarding the "time desync" message, I do have NTP synchronization in
>> use on all our systems using local NTP servers here on campus.  It
>> looks like most of the systems reporting this message are also
>> located in other buildings.
>>
>>
>> I do have one suggestion.  It would be helpful in our environment
>> (many different MooseFS master servers) to include in the log message
>> which master server is having an issue.  For example, an IP address
>> could be added in parentheses after "master" in the log message:
>>
>>     mfsmount[9898]: time desync between client and master
>> *(ip:10.163.216.36)* is higher than a second - it might lead to
>> strange atime/mtime behaviour - consider time synchronization in your
>> moosefs cluster
>>
>>
>> Thanks!
>>
>>
>> Steve
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Aleksander Wieliczko <ale...@mo...>
>> *Sent:* Tuesday, August 23, 2016 3:45 AM
>> *To:* Wilson, Steven M; MooseFS-Users
>> *Subject:* Re: [MooseFS-Users] high packet travel time
>>  
>> Hi,
>>
>> According to your system log, we can notice two different problems:
>>
>> 1.) First problem is "high packet travel".
>>
>> I would like to inform you that since version 3.0.x MooseFS has new
>> time calculation method.
>> Master is able to calculate "real" packet time travel between MooseFS
>> Master and clients.
>> In a glance. Echo packet is send  through TCP and goes to master
>> queue. So time of operation is a sum of time, that Master spends on
>> ECHO reply and packet travel time.
>> Time difference larger than 10ms can influence on all operations
>> performance.
>>
>> Information about "high packet travel time" can mean, that:
>> - Network traffic is huge,
>> - Router is not able to handle large number of packets traffic,
>> - Master is busy - CPU utilisation is close to 100%,
>> - Some interfaces have TX/RX errors.
>>
>> If you notice this entry in your system log rarely you can ignore it,
>> but if you see it often, consider to do some investigation.
>> Also please check if you network configuration is error-free.
>> Always good idea is do perform some simple ping test to see if your
>> latency is really small.
>>
>> By the way. Your MooseFS master is hardware or virtual machine?
>>
>>
>> 2.) Second problem i "time desync".
>>
>> Information about time de-synchronization between client and master
>> means that local data is different on master and client.
>> So please check if you have enabled some ntp time synchronization on
>> all MooseFS components.
>>
>> I'm looking forward to hearing from you.
>>
>> Best regards,
>> Aleksander Wieliczko
>> Technical Support Engineer
>> MooseFS.com <moosefs.com>
>