Re: [MooseFS-Users] Master segfault

[Alex C. <ac...@in...>]

Hi Piotr,

I am using latest Centos (equivalent to RHEL 7.2) and chunkservers were
on latest repo from zfsonlinux.org. For now I'm running a git pull of
ZoL due to a fix in L2ARC eviction code that caused cache eviction to
use 100% CPU. Kernel is what comes with the OS, no custom stuff.

I've tuned the kernel on masters and chunkserver like this:

vm.vfs_cache_pressure = 1000
vm.swappiness = 25

to reduce thrashing to swap.

We have no plans to ever move away from CentOS, RHEL, or any other
tracking derivative. We're a small. low-risk financial intermediary with
good profits so we see no reason to risk anything else. MooseFS gives up
a layer of protection over the ZoL code, which gives us the benefit of
compression, checksums and flexible usage of raw disks without the worry
of RAID controllers going out of date.

Target usage is Windows clients accessing CIFS via Samba+CTDB and backup
from this cluster to another, both clusters split over sites about 2-3km
apart, so network latency is will below disk.

I hope this is enough info for you!

Many thanks

Alex

On 28/01/16 19:32, Piotr Robert Konopelko wrote:
> Hi Alex,
>
>> I saw that just now. I seriously love you guys, your responsiveness
>> is amazing.
>
> It's really nice to read this :) Thanks!
>
> Could you let us know, which OS you use on Master Server
> so we can try to reproduce it in a similar environment?
>
>
> Best regards,
>
> -- 
> Piotr Robert Konopelko
> *MooseFS Technical Support Engineer* | moosefs.com <https://moosefs.com>
>
>> On 28 Jan 2016, at 8:20 PM, Alex Crow <ac...@in...
>> <mailto:ac...@in...>> wrote:
>>
>> I saw that just now. I seriously love you guys, your responsiveness
>> is amazing.
>>
>> I've upgraded both machines and just waiting for a scan to finish.
>>
>> Brilliant!
>>
>> Alex
>>
>> On 28/01/16 17:18, Piotr Robert Konopelko wrote:
>>> Dear Alex,
>>>
>>> It may be connected with segfault fixed recently:
>>>
>>> https://moosefs.com/documentation/changes-in-moosefs-3-0.html
>>>
>>> *MooseFS 3.0.71-1 (2016-01-21)*
>>>
>>>   * (master) fixed emptying trash issue (intr. in 3.0.64)
>>>   * *(master) fixed possible segfault in chunkservers database
>>>     (intr. in 3.0.67)*
>>>   * (master) changed trash part choice from nondeterministic to
>>>     deterministic
>>>
>>>
>>> We can check it tomorrow; for now please upgrade to 3.0.71-1 and let
>>> us know if problem persists.
>>> We recommend to use always the newest version (especially if you use
>>> 3.0/unstable/testing branch).
>>>
>>>
>>> Best regards,
>>>
>>> -- 
>>> Piotr Robert Konopelko
>>> *MooseFS Technical Support Engineer* | moosefs.com
>>> <https://moosefs.com/>
>>>
>>>> On 28 Jan 2016, at 4:23 PM, Alex Crow <ac...@in...> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have a pair of servers running 3.0.69 and under high load (many
>>>> rsync
>>>> processes) the master (running on one of the servers) is segfaulting:
>>>>
>>>> [1044721.599157] mfsmaster[18388]: segfault at 15 ip
>>>> 00000000004415ac sp
>>>> 00007ffdf4df0f70 error 4 in mfsmaster[400000+9a000]
>>>> [1107293.383944] mfsmaster[18907]: segfault at 15 ip
>>>> 00000000004415ac sp
>>>> 00007ffed9db6780 error 4 in mfsmaster[400000+9a000]
>>>>
>>>> Any ideas of how to fix this?
>>>>
>>>> Many thanks
>>>>
>>>> Alex
>>>>
>>>> --
>>>> This message is intended only for the addressee and may contain
>>>> confidential information. Unless you are that person, you may not
>>>> disclose its contents or use it in any way and are requested to delete
>>>> the message along with any attachments and notify us immediately.
>>>> This email is not intended to, nor should it be taken to,
>>>> constitute advice.
>>>> The information provided is correct to our knowledge & belief and
>>>> must not
>>>> be used as a substitute for obtaining tax, regulatory, investment,
>>>> legal or
>>>> any other appropriate advice.
>>>>
>>>> "Transact" is operated by Integrated Financial Arrangements Ltd.
>>>> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020)
>>>> 7608 5300.
>>>> (Registered office: as above; Registered in England and Wales under
>>>> number: 3727592). Authorised and regulated by the Financial Conduct
>>>> Authority (entered on the Financial Services Register; no. 190856).
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>>> Monitor end-to-end web transactions and take corrective actions now
>>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>>>> _________________________________________
>>>> moosefs-users mailing list
>>>> moo...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>
>>
>>
>> --
>> This message is intended only for the addressee and may contain
>> confidential information. Unless you are that person, you may not
>> disclose its contents or use it in any way and are requested to delete
>> the message along with any attachments and notify us immediately.
>> This email is not intended to, nor should it be taken to, constitute advice.
>> The information provided is correct to our knowledge & belief and must not
>> be used as a substitute for obtaining tax, regulatory, investment, legal or
>> any other appropriate advice.
>>
>> "Transact" is operated by Integrated Financial Arrangements Ltd.
>> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
>> (Registered office: as above; Registered in England and Wales under
>> number: 3727592). Authorised and regulated by the Financial Conduct
>> Authority (entered on the Financial Services Register; no. 190856).
>>
>> ------------------------------------------------------------------------------
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140_________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>


--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).