moosefs-users — Mailing list for MooseFS Users

Re: [MooseFS-Users] Performance/Througput high Wait On Disc

[Wolfgang <moo...@wo...>]

Additional Info: Operating System on all machines is Ubuntu 14.04 LTS


On 2016-09-23 09:47, Wolfgang wrote:
> Hi Group!
>
> I'm using current moosefs 3.0.81-1 with 7 machines.
>
> All kind of hp proliant g5 machines (dl-160, dl-380, dl-385, ...) with
> SATA discs creating about 25TB Storage
> Master has 32GB of RAM
> Load on all machines is low (<0.20)
> all machines have 1gbit, master 2Gbit via LACP all connected by a
> TP-Link TL-SG3424 managed switch.
>
> Now I want to copy 1,4TB files (photos, documents, ...) from mfs to a
> 2TB SATA disc plugged into another proliant server as a backup of mfs.
> For this I mounted the sata disc on /mnt and the mfswith
> mfsmount /data/moos
> and started
> rsync -avz /data/moos/snapshot/stuff_2016-09-21 /mnt/backup-on-hdd/
>
> But speed is very slow (about 2MB/s)
> I think speed at the beginning was faster but now nearly stucked.
>
> For true speed mesurements I normally do a:
> watch -d -n 10 du -sh /mnt/backup-on-hdd/
> so I see how much was transfered within 10 seconds and I divide this by
> 10 so I get the data per second.
> This is currently below 1 MB/s
>
> top shows a Load of 1.17 and a Wait on Disc of 25 at the moment.
>
> So speed of the data transfer is very very slow.
>
> So to test if the source or the target are slow I tried (while the rsync
> is running) a:
> sudo dd if=/dev/zero of=/mnt/test.bin bs=1M count=1024
> gives me 90MB/s
> and
> sudo dd if=/dev/zero of=/data/moos/test.bin bs=1M count=1024
> gives me 117MB/s so the source is not saturated nor the network nor the
> target.
>
>
> Can you help me getting better performance throughput ?
>
>
> Thank you & greetings from austria
>
> -Wolfgang
>
>
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

[MooseFS-Users] Performance/Througput high Wait On Disc

[Wolfgang <moo...@wo...>]

Hi Group!

I'm using current moosefs 3.0.81-1 with 7 machines.

All kind of hp proliant g5 machines (dl-160, dl-380, dl-385, ...) with 
SATA discs creating about 25TB Storage
Master has 32GB of RAM
Load on all machines is low (<0.20)
all machines have 1gbit, master 2Gbit via LACP all connected by a 
TP-Link TL-SG3424 managed switch.

Now I want to copy 1,4TB files (photos, documents, ...) from mfs to a 
2TB SATA disc plugged into another proliant server as a backup of mfs.
For this I mounted the sata disc on /mnt and the mfswith
mfsmount /data/moos
and started
rsync -avz /data/moos/snapshot/stuff_2016-09-21 /mnt/backup-on-hdd/

But speed is very slow (about 2MB/s)
I think speed at the beginning was faster but now nearly stucked.

For true speed mesurements I normally do a:
watch -d -n 10 du -sh /mnt/backup-on-hdd/
so I see how much was transfered within 10 seconds and I divide this by 
10 so I get the data per second.
This is currently below 1 MB/s

top shows a Load of 1.17 and a Wait on Disc of 25 at the moment.

So speed of the data transfer is very very slow.

So to test if the source or the target are slow I tried (while the rsync 
is running) a:
sudo dd if=/dev/zero of=/mnt/test.bin bs=1M count=1024
gives me 90MB/s
and
sudo dd if=/dev/zero of=/data/moos/test.bin bs=1M count=1024
gives me 117MB/s so the source is not saturated nor the network nor the 
target.


Can you help me getting better performance throughput ?


Thank you & greetings from austria

-Wolfgang

Re: [MooseFS-Users] Strange problems compiling on a moosefs share.

[Matt W. <mat...@gm...>]

Using DIRECT appears to have fixed the problem. I was able to compile and
run all the tests successfully.

Thanks.

On Wed, Sep 21, 2016 at 4:59 AM, Aleksander Wieliczko <
ale...@mo...> wrote:

> Hi,
>
> I would like to commit update.
>
> We suspect that problem is connected with MooseFS client cache.
> Right now we are in progress of debugging, but would you be so kind and
> perform simple test in your environment too.
>
> Please try to mount MooseFS client with mfscachemode=DIRECT option:
>
> mfsmount -o mfscachemode=DIRECT -H mfs.master.host /mfs/mount/point
>
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <http://moosefs.com>
>
> ------------------------------------------------------------
> ------------------
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>

Re: [MooseFS-Users] MooseFS Support Request - YuMing Zhao [Support #2342]

[赵宇明 <yum...@le...>]

Hi Peter, Alex :
   Much more detailed information of this issue is described below:

   Environment:
   Machines: 1 Master + 1 Logger + 3 Chunks
      Data Size： 25T
      File Number：260,000,000
  
      Problem Description:
      1、Master Node suddenly cannot  provide the services  at run time.  Meanwhile Mfscgi cannot be connected.  After stopping and restarting Master Server,  the problem is still there.
      2、When we tried to stop a Chunk server,  Master Server could be connected.  But Mfscgi displayed that the other two chunk servers failed to work properly.
      3、After repeated attempts, we did "restore".  Prompt is "hole in change files (entries from :5125524439 to :5125524550 are missing) - add more files".
      When we deleted some changelogs, Restore could be done correctly.
   
     ---------------------------------
     -- Client cannot be mounted
     root@mfsmaster-75:~# /opt/mfs/bin/mfsmount  /allmfs/   -S /
     error receiving data from mfsmaster: ETIMEDOUT (Operation timed out)
     ---------------------------------

   Screenshot of Mfscgi:
      
  
   Finally, attachment is the execution result of the commands, FYI.
   Thanks.
   
Best regards
YuMing Zhao 


 
From: 赵宇明
Date: 2016-09-21 20:20
To: MooseFS Support
CC: 张伟军; 李学宝; 马军伟; 常战青
Subject: Re: Re: MooseFS Support Request - YuMing Zhao [Support #2342]
Hi Peter,
   Thank you for your kind reply.
   We will execute the commands that you suggested and report much more detailed information to you ASAP.
   Thanks.

Best regards
YuMing Zhao
 
From: MooseFS Support
Date: 2016-09-21 19:03
To: yuming.zhao
Subject: Re: MooseFS Support Request - YuMing Zhao [Support #2342]
---- Please reply above this line ----
Ticket ID: #2342
Dear 赵宇明,
We say your message on our Community Mailing List "[MooseFS-Users] Asking for Help, MooseFS doesn't work". Alex Wieliczko has just replied you.
Can you let us know some details, what happened? Also ls -la /var/lib/mfs or ls -la /var/mfs and df -h both from Master and Metalogger would be appreciated.
Best regards,
Peter
-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | https://moosefs.com

Re: [MooseFS-Users] Asking for Help, MooseFS doesn't work.

[赵宇明 <yum...@le...>]

Hi Aleksander,
   Thanks for your quick response.
   You are right, under heavy system loads, the master server crushed.
   Please excuse us for not able to provide much more detailed information, we are going to follow this issue and report back to you.
   We also found some commands, such as "mfsmetarestore -a" 、 "mfsmetarestore -m metadata.mfs.back -o metadata.mfs changelog_ml.*.mfs" and "mfsfilerapair", we will try the steps that you suggested and these commands. At the same time, we also look for your help in this.
   Thanks a lot.

Best regards
YuMing Zhao 
   
 
From: Aleksander Wieliczko
Date: 2016-09-21 18:55
To: 赵宇明; moosefs-users
CC: 张伟军; 李学宝; 马军伟; 常战青
Subject: Re: [MooseFS-Users] Asking for Help, MooseFS doesn't work.
Hi.
Great to hear that You are using our product.

In fact MooseFS 1.6.25 is very, very old software.
MooseFS 1.6 has a lot of bugs and it's not supported any more.

Please consider to update to 3.0.81 version.

It looks like your master server crushed and you lost your metadata.mfs file(What had happened?)
Than you copied metadata from metalogger.

First of all please add some changelogs from metalogger to master server and try to start master.

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com

Re: [MooseFS-Users] Strange problems compiling on a moosefs share.

[Aleksander W. <ale...@mo...>]

Hi,

I would like to commit update.

We suspect that problem is connected with MooseFS client cache.
Right now we are in progress of debugging, but would you be so kind and
perform simple test in your environment too.

Please try to mount MooseFS client with mfscachemode=DIRECT option:

mfsmount -o mfscachemode=DIRECT -H mfs.master.host /mfs/mount/point

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

Re: [MooseFS-Users] Asking for Help, MooseFS doesn't work.

[Aleksander W. <ale...@mo...>]

Hi.
Great to hear that You are using our product.

In fact MooseFS 1.6.25 is very, very old software.
MooseFS 1.6 has a lot of bugs and it's not supported any more.

Please consider to update to 3.0.81 version.

It looks like your master server crushedand you lost your metadata.mfs
file(What had happened?)
Than you copied metadata from metalogger.

First of all please add some changelogs from metalogger to master server
and try to start master.

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

Re: [MooseFS-Users] Asking for Help, MooseFS doesn't work.

[Alex C. <ac...@in...>]

On 21/09/16 09:58, 赵宇明 wrote:

> Dear Support Professional,
> Our MooseFS server is not available, let me introduce the 
> Deployment Environment:
> *Network Topology:*
>
>
> *The parameters:*
> Version of MooseFS: 1.6.25

That's positively antique. Current versions are 2.0.89-1 and 3.0.81.1. 
I'd suggest trying one of those...

Alex

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

[MooseFS-Users] Asking for Help, MooseFS doesn't work.

[赵宇明 <yum...@le...>]

Dear Support Professional,
     Our MooseFS server is not available, let me introduce the Deployment Environment:
     Network Topology:

     

     The parameters:
     Version of MooseFS: 1.6.25
     Operating system of master server(IP:71): debian7.5
     Operating system of Logger server(IP:73): debian7.5

     Operating system of chunk server1(IP:71): debian7.5
     Operating system of chunk server2(IP:73): debian6.0.9
     Operating system of chunk server3(IP:74): debian6.0.9

     Operating system of clients: debian6.0.10
     -----------------------
     Files system of master server(IP:71):  ext4
     Files system of chunk server1: ext4
     Files system of chunk servers: ext4
     Files system of chunk servers: ext4
     Files system of clients: ext3

     Problem Description:
     1、Force start MooseFS.
     2、When we mount the node of Chunk1(IP:71), Master node can not provide the services.
     3、If we don't mount the node of Chunk1, Master node is available. But Chunk2 and Chunk3 are invalid, prompt is "File is missing".
     4、When we do "restore"，prompt is "hole in change files (entries from :5125524439 to :5125524550 are missing) - add more files".
        Complete information is as follows:
        ----------------------------------------------
        root@mfs2:/opt/mfs/etc# /opt/mfs/sbin/mfsmetarestore -a
        file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' instead
        loading objects (files,directories,etc.) … ok
        loading names … ok
        loading deletion timestamps … ok
        loading chunks data … ok
        checking filesystem consistency … ok
        connecting files and chunks … ok
        hole in change files (entries from 0￡C-:5125524439 to 0￡C-:5125524550 are missing) - add more files
        ----------------------------------------------
 
     We suspect:
     1、The matedata is corrupted.
     2、The problems of Operating System.
     3、Software version is too old.

     We plan to take the following action:
     1、In order to find out the problems，we want to re-install the running environment of MooseFS.
     2、To upgrade to the latest version of MooseFS.
 
     Please tell us if these methods are powerful to process the problem. 
     Looking forward to receiving your immediate reply, thanks a lot.

     
Best Regards.

Re: [MooseFS-Users] Strange problems compiling on a moosefs share.

[Jakub Kruszona-Z. <jak...@ge...>]

On 20 Sep, 2016, at 18:35, Matt Welland <mat...@gm...> wrote:

> I think I used nanomsg from apt for that build of chicken/iup.
> 
> I did some testing on other fuse based filesystems to see if I could discover anything useful. On sshfs everything worked fine. Using the fuse egg for chicken with the example hashfs.scm I had problems with compiling common.scm. I posted a question on this to the chicken-users list and a possible root cause is the fact that the hashfs.scm example does not have flock. Perhaps this is a factor in the moosefs issue? I seem to remember reading that there were changes recently in the flock support.
> 
> I will try to generate a smaller, focused testcase for reproducing this problem but it will take some time. 
> 

It would be helpful.

In the meantime we tried to compile moosefs on moosefs - just to check if gcc works ok on moosefs (chicken-scheme translates scheme sources to C sources and then uses gcc to compile - this is why we checked classic gcc on moosefs). We noticed strange behaviour. We compiled the same sources in the loop (each time using make -j) and after each loop checked md5 of compiled files. Usually md5 was constant, but from time to time md5 sums were different (two other values). It doesn't mean that those binaries were corrupted - it might be just related to differences in compiling order. We will check it again - this time we will test all binaries. Maybe it will lead us to the same bug that triggers your problems.

-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

Re: [MooseFS-Users] moosefs-chunkserver EADDRNOTAVAIL during boot process

[Aleksander W. <ale...@mo...>]

Hi,
Please check you /etc/mfs/mfschunkserver.cfg file and BIND_HOST parameter (if you have such one)
It looks like your chunkserver have different IP addres in cfg file than your NIC IP.

MooseFS ports are in range 9419-9425

Best Regards
Aleksander

> On 20 wrz 2016, at 18:54, Markus Koeberl <mar...@tu...> wrote:
> 
> Hi!
> 
> Today I recognized that one of my chunkservers (Debian jessie version 3.0.81-1) failed to start during boot process:
> 
> $ systemctl status moosefs-chunkserver.service -l
> ● moosefs-chunkserver.service - LSB: Start moosefs-chunkserver at boot time
>   Loaded: loaded (/etc/init.d/moosefs-chunkserver)
>   Active: failed (Result: exit-code) since Tue 2016-09-20 18:24:55 CEST; 22s ago
>  Process: 835 ExecStart=/etc/init.d/moosefs-chunkserver start (code=exited, status=1/FAILURE)
> 
> Sep 20 18:24:35 apollo mfschunkserver[910]: exititng ...
> Sep 20 18:24:35 apollo moosefs-chunkserver[835]: hdd space manager: path to scan: /srv/MooseFS2/
> Sep 20 18:24:35 apollo moosefs-chunkserver[835]: hdd space manager: path to scan: /srv/MooseFS1/
> Sep 20 18:24:35 apollo moosefs-chunkserver[835]: hdd space manager: start background hdd scanning (searching for available chunks)
> Sep 20 18:24:35 apollo moosefs-chunkserver[835]: main server module: can't listen on socket: EADDRNOTAVAIL (Cannot assign requested address)
> Sep 20 18:24:35 apollo moosefs-chunkserver[835]: init: main server acceptor failed !!!
> Sep 20 18:24:35 apollo moosefs-chunkserver[835]: error occurred during initialization - exiting
> Sep 20 18:24:55 apollo systemd[1]: moosefs-chunkserver.service: control process exited, code=exited status=1
> Sep 20 18:24:55 apollo systemd[1]: Failed to start LSB: Start moosefs-chunkserver at boot time.
> Sep 20 18:24:55 apollo systemd[1]: Unit moosefs-chunkserver.service entered failed state.
> 
> $ cat /proc/sys/net/ipv4/ip_local_port_range
> 32768   61000
> 
> 
> Is there anything I can do about this? It seams that this does not happen on my other Debian jessie chunkserver.
> 
> 
> 
> regards
> Markus Köberl
> -- 
> Markus Koeberl
> Graz University of Technology
> Signal Processing and Speech Communication Laboratory
> E-mail: mar...@tu...
> 
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

[MooseFS-Users] moosefs-chunkserver EADDRNOTAVAIL during boot process

[Markus K. <mar...@tu...>]

Hi!

Today I recognized that one of my chunkservers (Debian jessie version 3.0.81-1) failed to start during boot process:

$ systemctl status moosefs-chunkserver.service -l
● moosefs-chunkserver.service - LSB: Start moosefs-chunkserver at boot time
   Loaded: loaded (/etc/init.d/moosefs-chunkserver)
   Active: failed (Result: exit-code) since Tue 2016-09-20 18:24:55 CEST; 22s ago
  Process: 835 ExecStart=/etc/init.d/moosefs-chunkserver start (code=exited, status=1/FAILURE)

Sep 20 18:24:35 apollo mfschunkserver[910]: exititng ...
Sep 20 18:24:35 apollo moosefs-chunkserver[835]: hdd space manager: path to scan: /srv/MooseFS2/
Sep 20 18:24:35 apollo moosefs-chunkserver[835]: hdd space manager: path to scan: /srv/MooseFS1/
Sep 20 18:24:35 apollo moosefs-chunkserver[835]: hdd space manager: start background hdd scanning (searching for available chunks)
Sep 20 18:24:35 apollo moosefs-chunkserver[835]: main server module: can't listen on socket: EADDRNOTAVAIL (Cannot assign requested address)
Sep 20 18:24:35 apollo moosefs-chunkserver[835]: init: main server acceptor failed !!!
Sep 20 18:24:35 apollo moosefs-chunkserver[835]: error occurred during initialization - exiting
Sep 20 18:24:55 apollo systemd[1]: moosefs-chunkserver.service: control process exited, code=exited status=1
Sep 20 18:24:55 apollo systemd[1]: Failed to start LSB: Start moosefs-chunkserver at boot time.
Sep 20 18:24:55 apollo systemd[1]: Unit moosefs-chunkserver.service entered failed state.

$ cat /proc/sys/net/ipv4/ip_local_port_range
32768   61000


Is there anything I can do about this? It seams that this does not happen on my other Debian jessie chunkserver.



regards
Markus Köberl
-- 
Markus Koeberl
Graz University of Technology
Signal Processing and Speech Communication Laboratory
E-mail: mar...@tu...

Re: [MooseFS-Users] Strange problems compiling on a moosefs share.

[Matt W. <mat...@gm...>]

I think I used nanomsg from apt for that build of chicken/iup.

I did some testing on other fuse based filesystems to see if I could
discover anything useful. On sshfs everything worked fine. Using the fuse
egg for chicken with the example hashfs.scm I had problems with compiling
common.scm. I posted a question on this to the chicken-users list and a
possible root cause is the fact that the hashfs.scm example does not have
flock. Perhaps this is a factor in the moosefs issue? I seem to remember
reading that there were changes recently in the flock support.

I will try to generate a smaller, focused testcase for reproducing this
problem but it will take some time.



On Tue, Sep 20, 2016 at 7:20 AM, Aleksander Wieliczko <
ale...@mo...> wrote:

> Hi,
> We would love to resolve you problem but we are forcing some problems with
> standard build on hdd.
> I would like to add that we are not familial with scheme programming
> language, so it can take some time to learn little bit more about it.
>
> Right now we are not able to build dashboard because file libnanomsg.so.4
> is missing:
>
> *Error: (load) unable to load compiled module - libnanomsg.so.4: cannot
> open shared object file: No such file or directory:
> "/opt/chicken/4.11rc2_x86_64/lib/chicken/8/nanomsg.so"*
>
> Can we get this module or use standard one from debian repositories ?
> Or there is something that we can do about it?
>
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <http://moosefs.com>
> On 09/20/2016 05:44 AM, Matt Welland wrote:
>
> Thank you for looking into this. It might be good to first build in /tmp
> for a reference baseline. If I run Megatest from a location where it was
> compiled before the upgrade to Moosefs 3.0.81 it runs fine. The problem
> seems to happen during the compile.
>
> How to reproduce:
>
> 1. Install chicken + IUP gui (alternative: use Makefile.installall from
> utils dir in Megatest)
>
> wget http://www.kiatoa.com/matt/chicken_4.11rc2_x86.tar.gz
> /tmp/chicken.tar.gz
> cd /opt;tar xf /tmp/chicken.tar.gz
> source /opt/chicken/4.11rc2_x86_64/setup-chicken4x.sh
>
> 2. Get Megatest source for v1.61 (project is at https://www.kiatoa.com/
> fossils/megatest)
>
>  wget http://www.kiatoa.com/matt/megatest_v1.61_src.tar.gz
> /tmp/megatest.tar.gz
> tar xf /tmp/megatest.tar.gz
> cd megatest
> make clean;make -j && make install
> cd tests
> source fixpath.sh
> make dashboard
> ========== results in ==========
> cd fullrun && /mfs/tmp/megatest/bin/dashboard -skip-version-check -rows
> 20 &
> matt@xena:/mfs/tmp/megatest/tests$
> Error: segmentation violation
>
>     Call history:
>
>     dcommon.scm:68: hash-table-set!
>     tree.scm:12: ##sys#require
>     tree.scm:13: ##sys#require
>     tree.scm:15: ##sys#require
>     tree.scm:17: ##sys#require
>     tree.scm:17: ##sys#require
>     tree.scm:17: ##sys#require
>     altdb.scm:2: make-hash-table
>     altdb.scm:3: ##sys#require
>     altdb.scm:3: hash-table-set!
>     vg.scm:13: ##sys#require
>     vg.scm:16: ##sys#require
>     vg.scm:16: ##sys#require
>     vg_records.scm:4: ##sys#require
>     vg_records.scm:5: make-exception
>     vg_records.scm:17: ##sys#require          <--
>
> If I compile Megatest v1.62 it also fails but I get a different signature:
>
> matt@xena:/mfs/tmp/megatest/tests$ [panic] Detected corrupted data in
> stack - execution terminated
>
> ...more...
> altdb.scm:3: hash-table-set!
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> altdb.scm:2: make-hash-table
> altdb.scm:3: ##sys#require
> altdb.scm:3: hash-table-set!
> db.scm:35: make-mutex     <--
>
>
> On Mon, Sep 19, 2016 at 2:31 AM, Jakub Kruszona-Zawadzki <
> jak...@ge...> wrote:
>
>>
>> On 18 Sep, 2016, at 2:34, Matt Welland <mat...@gm...> wrote:
>>
>> Previous (rock solid) setup:
>>
>> Moosefs 2.0 installed via apt all on 64bit x86.
>>
>> Current (problematic) setup:
>>
>> Moosefs 3.0.81 installed from source using "dpkg-buildpackage -b"
>>
>> Master is on a cubietruck running Debian Jessie. chunkservers are on x86.
>>
>> The problem:
>>
>> Compiling a Chicken Scheme program results in the program crashing.
>> Compiling the program on local disk works fine.
>>
>>
>> We need to reproduce this behaviour on our instance. Could you write
>> more? I tried simple 'hello world' and it works:
>>
>> $ cat hello.scm
>> (print "Hello, world!")
>> $ csi -s hello.scm
>> Hello, world!
>> $ csc hello.scm
>> $ ./hello
>> Hello, world!
>>
>> I've also downloaded some benchmarks from:
>>
>> https://github.com/ashinn/chibi-scheme/tree/master/benchmarks/gabriel
>>
>> I've tested 'kanren.sch' using MooseFS 3.0.81, Ubuntu 16.04 and chicken
>> scheme installed from packages (version 4.9 as I remember)
>>
>> core@worker6:/mnt/mfstest/scheme/chibi-scheme/benchmarks/gabriel$ csc
>> kanren.sch
>> core@worker6:/mnt/mfstest/scheme/chibi-scheme/benchmarks/gabriel$
>> ./kanren
>> Testing eigen
>> eigens: (!x_!$gen$!x3 !y_!$gen$!x4)
>> Testing test-unify/pairs-oleg1
>> Testing test-unify/pairs-oleg2
>> (...) <--- long output
>> Testing logo-3-3
>> Testing powers-of-3
>> Testing powers-of-exp-3
>> 2.7s CPU time, 0.432s GC time (major), 4601 mutations, 202/5439 GCs
>> (major/minor)
>>
>> It looks correct.
>>
>> I've also checked it on Debian Jessie (64bit) and it also works fine.
>>
>>
>> Maybe we need something more sophisticated. Could you send us something
>> that triggers this problem?
>>
>>
>>
>>
>> I also see these messages in syslog:
>>
>> [11622.232964] mfsmount (13759): /proc/13759/oom_adj is deprecated,
>> please use /proc/13759/oom_score_adj instead.
>>
>>
>> It doesn't matter. I use both 'oom_adj' and 'oom_score_adj', so it works
>> fine. This is only disabling 'out of memory killer', so this is not related
>> to any file operations.
>>
>>
>> Should I revert to my old setup?
>> ------------------------------------------------------------
>> ------------------
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>>
>> --
>> Regards,
>> Jakub Kruszona-Zawadzki
>> - - - - - - - - - - - - - - - -
>> Segmentation fault (core dumped)
>> Phone: +48 602 212 039 <%2B48%20602%20212%20039>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>>
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>>
>
>
> ------------------------------------------------------------------------------
>
>
>
> _________________________________________
> moosefs-users mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
>
> ------------------------------------------------------------
> ------------------
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>

Re: [MooseFS-Users] Strange problems compiling on a moosefs share.

[Aleksander W. <ale...@mo...>]

Hi,
We would love to resolve you problem but we are forcing some problems
with standard build on hdd.
I would like to add that we are not familial with scheme programming
language, so it can take some time to learn little bit more about it.

Right now we are not able to build dashboard because file
libnanomsg.so.4 is missing:

*Error: (load) unable to load compiled module - libnanomsg.so.4: cannot
open shared object file: No such file or directory:
"/opt/chicken/4.11rc2_x86_64/lib/chicken/8/nanomsg.so"*

Can we get this module or use standard one from debian repositories ?
Or there is something that we can do about it?

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>
On 09/20/2016 05:44 AM, Matt Welland wrote:
> Thank you for looking into this. It might be good to first build in
> /tmp for a reference baseline. If I run Megatest from a location where
> it was compiled before the upgrade to Moosefs 3.0.81 it runs fine. The
> problem seems to happen during the compile.
>
> How to reproduce:
>
> 1. Install chicken + IUP gui (alternative: use Makefile.installall
> from utils dir in Megatest)
>
> wget http://www.kiatoa.com/matt/chicken_4.11rc2_x86.tar.gz
> /tmp/chicken.tar.gz
> cd /opt;tar xf /tmp/chicken.tar.gz
> source /opt/chicken/4.11rc2_x86_64/setup-chicken4x.sh
>
> 2. Get Megatest source for v1.61 (project is at
> https://www.kiatoa.com/fossils/megatest)
>
>  wget http://www.kiatoa.com/matt/megatest_v1.61_src.tar.gz
> /tmp/megatest.tar.gz
> tar xf /tmp/megatest.tar.gz
> cd megatest
> make clean;make -j && make install
> cd tests
> source fixpath.sh
> make dashboard
> ========== results in ==========
> cd fullrun && /mfs/tmp/megatest/bin/dashboard -skip-version-check
> -rows 20 &
> matt@xena:/mfs/tmp/megatest/tests$
> Error: segmentation violation
>
>     Call history:
>
>     dcommon.scm:68: hash-table-set!     
>     tree.scm:12: ##sys#require     
>     tree.scm:13: ##sys#require     
>     tree.scm:15: ##sys#require     
>     tree.scm:17: ##sys#require     
>     tree.scm:17: ##sys#require     
>     tree.scm:17: ##sys#require     
>     altdb.scm:2: make-hash-table     
>     altdb.scm:3: ##sys#require     
>     altdb.scm:3: hash-table-set!     
>     vg.scm:13: ##sys#require     
>     vg.scm:16: ##sys#require     
>     vg.scm:16: ##sys#require     
>     vg_records.scm:4: ##sys#require     
>     vg_records.scm:5: make-exception     
>     vg_records.scm:17: ##sys#require          <--
>
> If I compile Megatest v1.62 it also fails but I get a different signature:
>
> matt@xena:/mfs/tmp/megatest/tests$ [panic] Detected corrupted data in
> stack - execution terminated
>
> ...more...
> altdb.scm:3: hash-table-set!
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> db.scm:19: ##sys#require
> altdb.scm:2: make-hash-table
> altdb.scm:3: ##sys#require
> altdb.scm:3: hash-table-set!
> db.scm:35: make-mutex     <--
>
>
> On Mon, Sep 19, 2016 at 2:31 AM, Jakub Kruszona-Zawadzki
> <jak...@ge... <mailto:jak...@ge...>> wrote:
>
>
>     On 18 Sep, 2016, at 2:34, Matt Welland <mat...@gm...
>     <mailto:mat...@gm...>> wrote:
>
>>     Previous (rock solid) setup:
>>
>>     Moosefs 2.0 installed via apt all on 64bit x86.
>>
>>     Current (problematic) setup:
>>
>>     Moosefs 3.0.81 installed from source using "dpkg-buildpackage -b"
>>
>>     Master is on a cubietruck running Debian Jessie. chunkservers are
>>     on x86.
>>
>>     The problem:
>>
>>     Compiling a Chicken Scheme program results in the program
>>     crashing. Compiling the program on local disk works fine.
>
>     We need to reproduce this behaviour on our instance. Could you
>     write more? I tried simple 'hello world' and it works:
>
>     $ cat hello.scm
>     (print "Hello, world!")
>     $ csi -s hello.scm
>     Hello, world!
>     $ csc hello.scm
>     $ ./hello
>     Hello, world!
>
>     I've also downloaded some benchmarks from:
>
>     https://github.com/ashinn/chibi-scheme/tree/master/benchmarks/gabriel
>     <https://github.com/ashinn/chibi-scheme/tree/master/benchmarks/gabriel>
>
>     I've tested 'kanren.sch' using MooseFS 3.0.81, Ubuntu 16.04 and
>     chicken scheme installed from packages (version 4.9 as I remember)
>
>     core@worker6:/mnt/mfstest/scheme/chibi-scheme/benchmarks/gabriel$
>     csc kanren.sch
>     core@worker6:/mnt/mfstest/scheme/chibi-scheme/benchmarks/gabriel$
>     ./kanren
>     Testing eigen
>     eigens: (!x_!$gen$!x3 !y_!$gen$!x4)
>     Testing test-unify/pairs-oleg1
>     Testing test-unify/pairs-oleg2
>     (...) <--- long output
>     Testing logo-3-3
>     Testing powers-of-3
>     Testing powers-of-exp-3
>     2.7s CPU time, 0.432s GC time (major), 4601 mutations, 202/5439
>     GCs (major/minor)
>
>     It looks correct.
>
>     I've also checked it on Debian Jessie (64bit) and it also works fine.
>
>
>     Maybe we need something more sophisticated. Could you send us
>     something that triggers this problem?
>
>
>
>>
>>     I also see these messages in syslog:
>>
>>     [11622.232964] mfsmount (13759): /proc/13759/oom_adj is
>>     deprecated, please use /proc/13759/oom_score_adj instead.
>
>     It doesn't matter. I use both 'oom_adj' and 'oom_score_adj', so it
>     works fine. This is only disabling 'out of memory killer', so this
>     is not related to any file operations.
>
>>
>>     Should I revert to my old setup?
>>     ------------------------------------------------------------------------------
>>     _________________________________________
>>     moosefs-users mailing list
>>     moo...@li...
>>     <mailto:moo...@li...>
>>     https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>     <https://lists.sourceforge.net/lists/listinfo/moosefs-users>
>
>     -- 
>     Regards,
>     Jakub Kruszona-Zawadzki
>     - - - - - - - - - - - - - - - -
>     Segmentation fault (core dumped)
>     Phone: +48 602 212 039 <tel:%2B48%20602%20212%20039>
>
>
>     ------------------------------------------------------------------------------
>
>     _________________________________________
>     moosefs-users mailing list
>     moo...@li...
>     <mailto:moo...@li...>
>     https://lists.sourceforge.net/lists/listinfo/moosefs-users
>     <https://lists.sourceforge.net/lists/listinfo/moosefs-users>
>
>
>
>
> ------------------------------------------------------------------------------
>
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [MooseFS-Users] Strange problems compiling on a moosefs share.

[Matt W. <mat...@gm...>]

Thank you for looking into this. It might be good to first build in /tmp
for a reference baseline. If I run Megatest from a location where it was
compiled before the upgrade to Moosefs 3.0.81 it runs fine. The problem
seems to happen during the compile.

How to reproduce:

1. Install chicken + IUP gui (alternative: use Makefile.installall from
utils dir in Megatest)

wget http://www.kiatoa.com/matt/chicken_4.11rc2_x86.tar.gz
/tmp/chicken.tar.gz
cd /opt;tar xf /tmp/chicken.tar.gz
source /opt/chicken/4.11rc2_x86_64/setup-chicken4x.sh

2. Get Megatest source for v1.61 (project is at
https://www.kiatoa.com/fossils/megatest)

 wget http://www.kiatoa.com/matt/megatest_v1.61_src.tar.gz
/tmp/megatest.tar.gz
tar xf /tmp/megatest.tar.gz
cd megatest
make clean;make -j && make install
cd tests
source fixpath.sh
make dashboard
========== results in ==========
cd fullrun && /mfs/tmp/megatest/bin/dashboard -skip-version-check -rows 20 &
matt@xena:/mfs/tmp/megatest/tests$
Error: segmentation violation

    Call history:

    dcommon.scm:68: hash-table-set!
    tree.scm:12: ##sys#require
    tree.scm:13: ##sys#require
    tree.scm:15: ##sys#require
    tree.scm:17: ##sys#require
    tree.scm:17: ##sys#require
    tree.scm:17: ##sys#require
    altdb.scm:2: make-hash-table
    altdb.scm:3: ##sys#require
    altdb.scm:3: hash-table-set!
    vg.scm:13: ##sys#require
    vg.scm:16: ##sys#require
    vg.scm:16: ##sys#require
    vg_records.scm:4: ##sys#require
    vg_records.scm:5: make-exception
    vg_records.scm:17: ##sys#require          <--

If I compile Megatest v1.62 it also fails but I get a different signature:

matt@xena:/mfs/tmp/megatest/tests$ [panic] Detected corrupted data in stack
- execution terminated

...more...
altdb.scm:3: hash-table-set!
db.scm:19: ##sys#require
db.scm:19: ##sys#require
db.scm:19: ##sys#require
db.scm:19: ##sys#require
db.scm:19: ##sys#require
db.scm:19: ##sys#require
db.scm:19: ##sys#require
db.scm:19: ##sys#require
db.scm:19: ##sys#require
db.scm:19: ##sys#require
db.scm:19: ##sys#require
altdb.scm:2: make-hash-table
altdb.scm:3: ##sys#require
altdb.scm:3: hash-table-set!
db.scm:35: make-mutex     <--


On Mon, Sep 19, 2016 at 2:31 AM, Jakub Kruszona-Zawadzki <
jak...@ge...> wrote:

>
> On 18 Sep, 2016, at 2:34, Matt Welland <mat...@gm...> wrote:
>
> Previous (rock solid) setup:
>
> Moosefs 2.0 installed via apt all on 64bit x86.
>
> Current (problematic) setup:
>
> Moosefs 3.0.81 installed from source using "dpkg-buildpackage -b"
>
> Master is on a cubietruck running Debian Jessie. chunkservers are on x86.
>
> The problem:
>
> Compiling a Chicken Scheme program results in the program crashing.
> Compiling the program on local disk works fine.
>
>
> We need to reproduce this behaviour on our instance. Could you write more?
> I tried simple 'hello world' and it works:
>
> $ cat hello.scm
> (print "Hello, world!")
> $ csi -s hello.scm
> Hello, world!
> $ csc hello.scm
> $ ./hello
> Hello, world!
>
> I've also downloaded some benchmarks from:
>
> https://github.com/ashinn/chibi-scheme/tree/master/benchmarks/gabriel
>
> I've tested 'kanren.sch' using MooseFS 3.0.81, Ubuntu 16.04 and chicken
> scheme installed from packages (version 4.9 as I remember)
>
> core@worker6:/mnt/mfstest/scheme/chibi-scheme/benchmarks/gabriel$ csc
> kanren.sch
> core@worker6:/mnt/mfstest/scheme/chibi-scheme/benchmarks/gabriel$ ./kanren
> Testing eigen
> eigens: (!x_!$gen$!x3 !y_!$gen$!x4)
> Testing test-unify/pairs-oleg1
> Testing test-unify/pairs-oleg2
> (...) <--- long output
> Testing logo-3-3
> Testing powers-of-3
> Testing powers-of-exp-3
> 2.7s CPU time, 0.432s GC time (major), 4601 mutations, 202/5439 GCs
> (major/minor)
>
> It looks correct.
>
> I've also checked it on Debian Jessie (64bit) and it also works fine.
>
>
> Maybe we need something more sophisticated. Could you send us something
> that triggers this problem?
>
>
>
>
> I also see these messages in syslog:
>
> [11622.232964] mfsmount (13759): /proc/13759/oom_adj is deprecated, please
> use /proc/13759/oom_score_adj instead.
>
>
> It doesn't matter. I use both 'oom_adj' and 'oom_score_adj', so it works
> fine. This is only disabling 'out of memory killer', so this is not related
> to any file operations.
>
>
> Should I revert to my old setup?
> ------------------------------------------------------------
> ------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
> --
> Regards,
> Jakub Kruszona-Zawadzki
> - - - - - - - - - - - - - - - -
> Segmentation fault (core dumped)
> Phone: +48 602 212 039
>
>
> ------------------------------------------------------------
> ------------------
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>

[MooseFS-Users] Problems with MooseFS + git + Dropbox

[Wilson, S. M <st...@pu...>]

Hi,


I'd like to make you aware of another problem that has appeared in our MooseFS environment (which, by the way, we think is really great overall!)


I have a user who has a cloned git repository in his MooseFS home directory that is part of a Dropbox directory.  He found that his Dropbox service always crashes when trying to sync with this git directory tree in his Dropbox directory.  I had him copy his Dropbox directory to a local disk and he was able to synchronize the directory without any problems.


I attempted the same thing using a git clone of the MooseFS source in my own MooseFS home directory and saw the same behavior.


?We use Ubuntu 16.04 and MooseFS 3.0.79 through 3.0.81.


?Thanks,


Steve

Re: [MooseFS-Users] Strange problems compiling on a moosefs share.

[Jakub Kruszona-Z. <jak...@ge...>]

On 18 Sep, 2016, at 2:34, Matt Welland <mat...@gm...> wrote:

> Previous (rock solid) setup:
> 
> Moosefs 2.0 installed via apt all on 64bit x86.
> 
> Current (problematic) setup:
> 
> Moosefs 3.0.81 installed from source using "dpkg-buildpackage -b"
> 
> Master is on a cubietruck running Debian Jessie. chunkservers are on x86.
> 
> The problem:
> 
> Compiling a Chicken Scheme program results in the program crashing. Compiling the program on local disk works fine.

We need to reproduce this behaviour on our instance. Could you write more? I tried simple 'hello world' and it works:

$ cat hello.scm
(print "Hello, world!")
$ csi -s hello.scm
Hello, world!
$ csc hello.scm
$ ./hello
Hello, world!

I've also downloaded some benchmarks from:

https://github.com/ashinn/chibi-scheme/tree/master/benchmarks/gabriel

I've tested 'kanren.sch' using MooseFS 3.0.81, Ubuntu 16.04 and chicken scheme installed from packages (version 4.9 as I remember)

core@worker6:/mnt/mfstest/scheme/chibi-scheme/benchmarks/gabriel$ csc kanren.sch
core@worker6:/mnt/mfstest/scheme/chibi-scheme/benchmarks/gabriel$ ./kanren
Testing eigen
eigens: (!x_!$gen$!x3 !y_!$gen$!x4)
Testing test-unify/pairs-oleg1
Testing test-unify/pairs-oleg2
(...) <--- long output
Testing logo-3-3
Testing powers-of-3
Testing powers-of-exp-3
2.7s CPU time, 0.432s GC time (major), 4601 mutations, 202/5439 GCs (major/minor)

It looks correct.

I've also checked it on Debian Jessie (64bit) and it also works fine.


Maybe we need something more sophisticated. Could you send us something that triggers this problem?



> 
> I also see these messages in syslog:
> 
> [11622.232964] mfsmount (13759): /proc/13759/oom_adj is deprecated, please use /proc/13759/oom_score_adj instead.

It doesn't matter. I use both 'oom_adj' and 'oom_score_adj', so it works fine. This is only disabling 'out of memory killer', so this is not related to any file operations.

> 
> Should I revert to my old setup?
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users


-- 
Regards,
Jakub Kruszona-Zawadzki
- - - - - - - - - - - - - - - -
Segmentation fault (core dumped)
Phone: +48 602 212 039

[MooseFS-Users] Strange problems compiling on a moosefs share.

[Matt W. <mat...@gm...>]

Previous (rock solid) setup:

Moosefs 2.0 installed via apt all on 64bit x86.

Current (problematic) setup:

Moosefs 3.0.81 installed from source using "dpkg-buildpackage -b"

Master is on a cubietruck running Debian Jessie. chunkservers are on x86.

The problem:

Compiling a Chicken Scheme program results in the program crashing.
Compiling the program on local disk works fine.

I also see these messages in syslog:

[11622.232964] mfsmount (13759): /proc/13759/oom_adj is deprecated, please
use /proc/13759/oom_score_adj instead.

Should I revert to my old setup?

Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

[Wilson, S. M <st...@pu...>]

??Hi Aleksander,


Yes, I agree that this seems really odd!  I'll send the system log to you directly in a separate email.


Thanks,

Steve


________________________________
From: Aleksander Wieliczko <ale...@mo...>
Sent: Thursday, September 15, 2016 1:34 AM
To: Wilson, Steven M
Cc: MooseFS-Users; Krzysztof Kielak
Subject: Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

Hi,
This sounds really odd.
Another problem is that we are not able to repeat this scenario.
Can you provide us system log from MooseFS master server.
Maybe this will tell us something more.

I'm looking forward to hearing from you.
Aleksander Wieliczko

On 14.09.2016 22:29, Wilson, Steven M wrote:

Hi Aleksander,


I needed to add another line to mfsexports.cfg today.  I didn't change any existing lines; I just added the following line:

    10.163.216.88   /       rw,alldirs,maproot=0


After adding this line, I took a deep breath and then sent a HUP signal to the mfsmaster process.  ?Rats...  all the clients for that MooseFS server immediately had empty mount points.


This occurred on a different MooseFS master server than the one mentioned in my original email.  The version of this mfsmaster is 3.0.78.  This time I did not use Ansible but instead manually edited the mfsexports.cfg file and manually sent a HUP signal to mfsmaster process.


Unfortunately, this looks like a reproducible problem in my environment and not just a one-time fluke.


Is there any additional information that I can send to be of help to you?  The clients and server are running on Ubuntu 16.04.


Best regards,

Steve


________________________________
From: Wilson, Steven M <st...@pu...><mailto:st...@pu...>
Sent: Monday, September 12, 2016 9:28 AM
To: Aleksander Wieliczko; Krzysztof Kielak
Cc: MooseFS-Users
Subject: Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?


?Hi Aleksander,


Thanks for attempting to reproduce the problem in your environment.  I'll send the mfsexports.cfg files to you (prior to the change and after the change).  I use Ansible for configuration management and I normally make all changes on my Ansible server in a file of my own format that is then used to build mfsexports.cfg files on each MooseFS master server.  So I'll also send the script that Ansible calls to build the mfsexports.cfg file and issue a HUP signal to mfsmaster.


Thanks,

Steve


________________________________
From: Aleksander Wieliczko <ale...@mo...><mailto:ale...@mo...>
Sent: Monday, September 12, 2016 6:25 AM
To: Wilson, Steven M; Krzysztof Kielak
Cc: MooseFS-Users
Subject: Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

Hi,

I would like to inform you that we are not able to reproduce this problem in our development environment.
Also I would like to add that in MooseFS is scenario which can cause mfsmount session drop.
When you change permissions in mfsexports.cfg file for connected mounts and you send HUP signal, MooseFS master will check permissions again, and if permissions have been changed access to cluster will be denied.

We have another question for you.
Can you send us old and new "mfsexports.cfg" files, so we will be able to understand what had happened?

You can send it to dwt[at]moosefs.com

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com<moosefs.com>

Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

[Wilson, S. M <st...@pu...>]

Hi,


This is the command line that I use when mounting MooseFS file systems:

?/usr/bin/mfsmount {mount_dir} -o mfsbind={client_ip} -H{server} -S{export_dir}


It is executed by a cron script that checks to make sure all necessary MooseFS file systems are mounted (I don't use automount or fstab).


Regards,

?Steve


________________________________
From: Aleksander Wieliczko <ale...@mo...>
Sent: Thursday, September 15, 2016 9:04 AM
To: Wilson, Steven M
Cc: MooseFS-Users; Krzysztof Kielak
Subject: Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

Hi,
I have another question for you.
Do you use some extra mount options or automount entries for systemd in fstab?

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com

On 14.09.2016 22:29, Wilson, Steven M wrote:

Hi Aleksander,


I needed to add another line to mfsexports.cfg today.  I didn't change any existing lines; I just added the following line:

    10.163.216.88   /       rw,alldirs,maproot=0


After adding this line, I took a deep breath and then sent a HUP signal to the mfsmaster process.  ?Rats...  all the clients for that MooseFS server immediately had empty mount points.


This occurred on a different MooseFS master server than the one mentioned in my original email.  The version of this mfsmaster is 3.0.78.  This time I did not use Ansible but instead manually edited the mfsexports.cfg file and manually sent a HUP signal to mfsmaster process.


Unfortunately, this looks like a reproducible problem in my environment and not just a one-time fluke.


Is there any additional information that I can send to be of help to you?  The clients and server are running on Ubuntu 16.04.


Best regards,

Steve


________________________________
From: Wilson, Steven M <st...@pu...><mailto:st...@pu...>
Sent: Monday, September 12, 2016 9:28 AM
To: Aleksander Wieliczko; Krzysztof Kielak
Cc: MooseFS-Users
Subject: Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?


?Hi Aleksander,


Thanks for attempting to reproduce the problem in your environment.  I'll send the mfsexports.cfg files to you (prior to the change and after the change).  I use Ansible for configuration management and I normally make all changes on my Ansible server in a file of my own format that is then used to build mfsexports.cfg files on each MooseFS master server.  So I'll also send the script that Ansible calls to build the mfsexports.cfg file and issue a HUP signal to mfsmaster.


Thanks,

Steve


________________________________
From: Aleksander Wieliczko <ale...@mo...><mailto:ale...@mo...>
Sent: Monday, September 12, 2016 6:25 AM
To: Wilson, Steven M; Krzysztof Kielak
Cc: MooseFS-Users
Subject: Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

Hi,

I would like to inform you that we are not able to reproduce this problem in our development environment.
Also I would like to add that in MooseFS is scenario which can cause mfsmount session drop.
When you change permissions in mfsexports.cfg file for connected mounts and you send HUP signal, MooseFS master will check permissions again, and if permissions have been changed access to cluster will be denied.

We have another question for you.
Can you send us old and new "mfsexports.cfg" files, so we will be able to understand what had happened?

You can send it to dwt[at]moosefs.com

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com<moosefs.com>

Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

[Aleksander W. <ale...@mo...>]

Hi,
I have another question for you.
Do you use some extra mount options or automount entries for systemd in 
fstab?

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com

On 14.09.2016 22:29, Wilson, Steven M wrote:
>
> Hi Aleksander,
>
>
> I needed to add another line to mfsexports.cfg today.  I didn't change 
> any existing lines; I just added the following line:
>
>     10.163.216.88   /       rw,alldirs,maproot=0
>
>
> After adding this line, I took a deep breath and then sent a HUP 
> signal to the mfsmaster process.  ​Rats...  all the clients for that 
> MooseFS server immediately had empty mount points.
>
>
> This occurred on a different MooseFS master server than the one 
> mentioned in my original email.  The version of this mfsmaster is 
> 3.0.78.  This time I did not use Ansible but instead manually edited 
> the mfsexports.cfg file and manually sent a HUP signal to mfsmaster 
> process.
>
>
> Unfortunately, this looks like a reproducible problem in my 
> environment and not just a one-time fluke.
>
>
> Is there any additional information that I can send to be of help to 
> you? The clients and server are running on Ubuntu 16.04.
>
>
> Best regards,
>
> Steve
>
>
> ------------------------------------------------------------------------
> *From:* Wilson, Steven M <st...@pu...>
> *Sent:* Monday, September 12, 2016 9:28 AM
> *To:* Aleksander Wieliczko; Krzysztof Kielak
> *Cc:* MooseFS-Users
> *Subject:* Re: [MooseFS-Users] mfsmaster config reload breaks client 
> mounts?
>
> ​Hi Aleksander,
>
>
> Thanks for attempting to reproduce the problem in your environment. 
>  I'll send the mfsexports.cfg files to you (prior to the change and 
> after the change).  I use Ansible for configuration management and I 
> normally make all changes on my Ansible server in a file of my own 
> format that is then used to build mfsexports.cfg files on each MooseFS 
> master server.  So I'll also send the script that Ansible calls to 
> build the mfsexports.cfg file and issue a HUP signal to mfsmaster.
>
>
> Thanks,
>
>
> Steve
>
>
> ------------------------------------------------------------------------
> *From:* Aleksander Wieliczko <ale...@mo...>
> *Sent:* Monday, September 12, 2016 6:25 AM
> *To:* Wilson, Steven M; Krzysztof Kielak
> *Cc:* MooseFS-Users
> *Subject:* Re: [MooseFS-Users] mfsmaster config reload breaks client 
> mounts?
> Hi,
>
> I would like to inform you that we are not able to reproduce this 
> problem in our development environment.
> Also I would like to add that in MooseFS is scenario which can cause 
> mfsmount session drop.
> When you change permissions in mfsexports.cfg file for connected 
> mounts and you send HUP signal, MooseFS master will check permissions 
> again, and if permissions have been changed access to cluster will be 
> denied.
>
> We have another question for you.
> Can you send us old and new "mfsexports.cfg" files, so we will be able 
> to understand what had happened?
>
> You can send itto dwt[at]moosefs.com
>
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <moosefs.com>

Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

[Aleksander W. <ale...@mo...>]

Hi,
This sounds really odd.
Another problem is that we are not able to repeat this scenario.
Can you provide us system log from MooseFS master server.
Maybe this will tell us something more.

I'm looking forward to hearing from you.
Aleksander Wieliczko

On 14.09.2016 22:29, Wilson, Steven M wrote:
>
> Hi Aleksander,
>
>
> I needed to add another line to mfsexports.cfg today.  I didn't change 
> any existing lines; I just added the following line:
>
>     10.163.216.88   /       rw,alldirs,maproot=0
>
>
> After adding this line, I took a deep breath and then sent a HUP 
> signal to the mfsmaster process.  ​Rats...  all the clients for that 
> MooseFS server immediately had empty mount points.
>
>
> This occurred on a different MooseFS master server than the one 
> mentioned in my original email.  The version of this mfsmaster is 
> 3.0.78.  This time I did not use Ansible but instead manually edited 
> the mfsexports.cfg file and manually sent a HUP signal to mfsmaster 
> process.
>
>
> Unfortunately, this looks like a reproducible problem in my 
> environment and not just a one-time fluke.
>
>
> Is there any additional information that I can send to be of help to 
> you? The clients and server are running on Ubuntu 16.04.
>
>
> Best regards,
>
> Steve
>
>
> ------------------------------------------------------------------------
> *From:* Wilson, Steven M <st...@pu...>
> *Sent:* Monday, September 12, 2016 9:28 AM
> *To:* Aleksander Wieliczko; Krzysztof Kielak
> *Cc:* MooseFS-Users
> *Subject:* Re: [MooseFS-Users] mfsmaster config reload breaks client 
> mounts?
>
> ​Hi Aleksander,
>
>
> Thanks for attempting to reproduce the problem in your environment. 
>  I'll send the mfsexports.cfg files to you (prior to the change and 
> after the change).  I use Ansible for configuration management and I 
> normally make all changes on my Ansible server in a file of my own 
> format that is then used to build mfsexports.cfg files on each MooseFS 
> master server.  So I'll also send the script that Ansible calls to 
> build the mfsexports.cfg file and issue a HUP signal to mfsmaster.
>
>
> Thanks,
>
>
> Steve
>
>
> ------------------------------------------------------------------------
> *From:* Aleksander Wieliczko <ale...@mo...>
> *Sent:* Monday, September 12, 2016 6:25 AM
> *To:* Wilson, Steven M; Krzysztof Kielak
> *Cc:* MooseFS-Users
> *Subject:* Re: [MooseFS-Users] mfsmaster config reload breaks client 
> mounts?
> Hi,
>
> I would like to inform you that we are not able to reproduce this 
> problem in our development environment.
> Also I would like to add that in MooseFS is scenario which can cause 
> mfsmount session drop.
> When you change permissions in mfsexports.cfg file for connected 
> mounts and you send HUP signal, MooseFS master will check permissions 
> again, and if permissions have been changed access to cluster will be 
> denied.
>
> We have another question for you.
> Can you send us old and new "mfsexports.cfg" files, so we will be able 
> to understand what had happened?
>
> You can send itto dwt[at]moosefs.com
>
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <moosefs.com>

Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

[Wilson, S. M <st...@pu...>]

Hi Aleksander,


I needed to add another line to mfsexports.cfg today.  I didn't change any existing lines; I just added the following line:

    10.163.216.88   /       rw,alldirs,maproot=0


After adding this line, I took a deep breath and then sent a HUP signal to the mfsmaster process.  ?Rats...  all the clients for that MooseFS server immediately had empty mount points.


This occurred on a different MooseFS master server than the one mentioned in my original email.  The version of this mfsmaster is 3.0.78.  This time I did not use Ansible but instead manually edited the mfsexports.cfg file and manually sent a HUP signal to mfsmaster process.


Unfortunately, this looks like a reproducible problem in my environment and not just a one-time fluke.


Is there any additional information that I can send to be of help to you?  The clients and server are running on Ubuntu 16.04.


Best regards,

Steve


________________________________
From: Wilson, Steven M <st...@pu...>
Sent: Monday, September 12, 2016 9:28 AM
To: Aleksander Wieliczko; Krzysztof Kielak
Cc: MooseFS-Users
Subject: Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?


?Hi Aleksander,


Thanks for attempting to reproduce the problem in your environment.  I'll send the mfsexports.cfg files to you (prior to the change and after the change).  I use Ansible for configuration management and I normally make all changes on my Ansible server in a file of my own format that is then used to build mfsexports.cfg files on each MooseFS master server.  So I'll also send the script that Ansible calls to build the mfsexports.cfg file and issue a HUP signal to mfsmaster.


Thanks,

Steve


________________________________
From: Aleksander Wieliczko <ale...@mo...>
Sent: Monday, September 12, 2016 6:25 AM
To: Wilson, Steven M; Krzysztof Kielak
Cc: MooseFS-Users
Subject: Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

Hi,

I would like to inform you that we are not able to reproduce this problem in our development environment.
Also I would like to add that in MooseFS is scenario which can cause mfsmount session drop.
When you change permissions in mfsexports.cfg file for connected mounts and you send HUP signal, MooseFS master will check permissions again, and if permissions have been changed access to cluster will be denied.

We have another question for you.
Can you send us old and new "mfsexports.cfg" files, so we will be able to understand what had happened?

You can send it to dwt[at]moosefs.com

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com<moosefs.com>

Re: [MooseFS-Users] IMPORTANT Changes in MooseFS Repository structure

[Piotr R. K. <pio...@mo...>]

Dear MooseFS Users,

we'd like to inform you, that we decided to remove current and stable repository symlinks. Starting from *** Monday, September 19, 2016 *** the only way to install / update MooseFS will be to use a new repository structure announced previously and described on https://moosefs.com/download.html <https://moosefs.com/download.html> and subpages:

(examples below are based on Ubuntu 14.04 LTS)


deb http://ppa.moosefs.com/moosefs-3/apt/ubuntu/trusty trusty main
--> this entry points to the newest stable, production-recommended MooseFS 3.0.x version (3.0.81 at the time of writing this e-mail)

deb http://ppa.moosefs.com/moosefs-2/apt/ubuntu/trusty trusty main
--> this entry points to the newest stable MooseFS 2.0.x version (2.0.89 at the time of writing this e-mail)



Of course it is still possible to control MooseFS version in the repository URL, i.e. use version number instead of moosefs-3 or moosefs-2:

deb http://ppa.moosefs.com/3.0.81/apt/ubuntu/trusty trusty main
--> the entry above will still work and at the moment we don't plan to change it.



We decided to remove stable instead of changing it to moosefs-3, because we are not able to inform all MooseFS Users about this change (and this change may have influence on Users' MooseFS instances, as we noticed previously). We hope, that error returned by package manager while trying to update information about available packages will be the best information to check valid repository entries on https://moosefs.com <https://moosefs.com/>.


In case of any questions regarding this change or MooseFS, please don't hesitate to contact me!


Best regards,
Peter / MooseFS Team

-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | moosefs.com <http://moosefs.com/>

> On 05 Aug 2016, at 12:02 PM, Piotr Robert Konopelko <pio...@mo...> wrote:
> 
> Hi Wolfgang,
> 
> it was not changed yet, because we still notice a lot of users using "stable" in their repositories - there's a lot of entries with "stable" in our http server logs.
> We decided to give users some extra time - it will be changed soon.
>> Am I save if I use the
>> http://ppa.moosefs.com/moosefs-3/apt/ubuntu/trusty
>> URL in my sources.list file
>> 
> Of course! This is the best way - i.e. to decide consciously which version you want to use at the moment and set it explicite in sources.list and stop using "stable".
> 
> In case of big installations, some users still may want to stay with MooseFS 2.0 for a while, we understand this - because of this reason we created "moosefs-2" - to let them stay and prepare to the conscious and planned upgrade.
> 
> Best regards, 
> 
> --
> 
> <Mail Attachment.png>
> 
> Piotr Robert Konopelko
> MooseFS Technical Support Engineer
> e-mail : pio...@mo...
> www : https://moosefs.com
> 
> <Mail Attachment.png> <Mail Attachment.png> <Mail Attachment.png> <Mail Attachment.png>
> 
> This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received it by mistake, please let us know by e-mail reply and delete it from your system; you may not copy this message or disclose its contents to anyone. Finally, the recipient should check this email and any attachments for the presence of viruses. Core Technology accepts no liability for any damage caused by any virus transmitted by this email.
> 
> 
>> On 05 Aug 2016, at 9:33 AM, Wolfgang <moo...@wo...> wrote:
>> 
>> Hi Peter!
>> 
>> When checking the URL:
>> http://ppa.moosefs.com/stable/apt/ubuntu/trusty/pool/main/m/moosefs/
>> it still shows me the 2.0.89-1 version files - was the change at the repository structure not executed as you pointed out?
>> 
>> Am I save if I use the
>> http://ppa.moosefs.com/moosefs-3/apt/ubuntu/trusty
>> URL in my sources.list file?
>> 
>> Thank you for this great product!
>> Wolfgang, Austria
>> 
>> 
>> 
>> On 2016-07-16 03:51, Piotr Robert Konopelko wrote:
>>> Dear MooseFS Users,
>>> 
>>> please remember about those important changes in MooseFS repository structure which will be introduced in about 2 weeks.
>>> 
>>> 
>>> Best regards,
>>> Peter from MooseFS Team
>>> 
>>> 
>>> -- 
>>> Piotr Robert Konopelko
>>> MooseFS Technical Support Engineer | 
>>> moosefs.com
>>> 
>>> 
>>> 
>>>> > On 10 Jun 2016, at 7:20 PM, Piotr Robert Konopelko <pio...@mo...>
>>>>  wrote:
>>>> 
>>>> > 
>>>> > 
>>>> Dear MooseFS Users,
>>>> 
>>>> > 
>>>> > 
>>>> we'd like to inform you, that we decided to change our repository structure.
>>>> 
>>>> > 
>>>> (examples based on Ubuntu 14.04)
>>>> 
>>>> > 
>>>> > 
>>>> > 
>>>> > deb http://ppa.moosefs.com/stable/apt/ubuntu/trusty
>>>>  trusty main
>>>> 
>>>> > 
>>>> > 
>>>> "stable" in the entry above now points to the latest stable MooseFS from 2.0 branch (at this moment - 2.0.89).
>>>> 
>>>> > 
>>>> *This behaviour will be kept until end of July 2016*
>>>> 
>>>> > 
>>>> > 
>>>> *Starting from August 1, 2016, "stable" in the entry above will point to the latest stable MooseFS from 3.0 branch (at this moment - 3.0.77)*
>>>> 
>>>> > 
>>>> > 
>>>> "stable" symlink is going to be removed at December 31, 2016 - this method will **NOT** work after this date.
>>>> 
>>>> > 
>>>> > 
>>>> > 
>>>> > 
>>>> Starting from today, it is also possible to use the following entries, so you can switch your repositories consciously either to MooseFS 2 or MooseFS 3 (***we recommend reconfigure your repos before mentioned end of July 2016***):
>>>> 
>>>> > 
>>>> > deb http://ppa.moosefs.com/moosefs-3/apt/ubuntu/trusty
>>>>  trusty main
>>>> 
>>>> > 
>>>> --> this entry points to the newest stable MooseFS 3.0.x version (3.0.77 at the time of writing this e-mail)
>>>> 
>>>> > 
>>>> > deb http://ppa.moosefs.com/moosefs-2/apt/ubuntu/trusty
>>>>  trusty main
>>>> 
>>>> > 
>>>> --> this entry points to the newest stable MooseFS 2.0.x version (2.0.89 at the time of writing this e-mail)
>>>> 
>>>> > 
>>>> > 
>>>> > 
>>>> > 
>>>> Of course it is still possible to use "version control" in the repository URL, i.e. version number instead of "stable", "moosefs-3" or "moosefs-2":
>>>> 
>>>> > 
>>>> > deb http://ppa.moosefs.com/3.0.77/apt/ubuntu/trusty
>>>>  trusty main
>>>> 
>>>> > 
>>>> > 
>>>> --> the entry above will still work and at the moment we don't plan to change it.
>>>> 
>>>> > 
>>>> > 
>>>> > 
>>>> > 
>>>> This information will be published on our website soon.
>>>> 
>>>> > 
>>>> > In case of any questions, please don't hesitate to ask! :)
>>>> > 
>>>> > 
>>>> > 
>>>> Best regards,
>>>> 
>>>> > 
>>>> Peter from MooseFS Team
>>>> 
>>>> > 
>>>> > 
>>>> -- 
>>>> 
>>>> > 
>>>> Piotr Robert Konopelko
>>>> 
>>>> > MooseFS Technical Support Engineer | moosefs.com
>>> ------------------------------------------------------------------------------
>>> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
>>> patterns at an interface-level. Reveals which users, apps, and protocols are 
>>> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
>>> J-Flow, sFlow and other flows. Make informed decisions using capacity planning
>>> reports.
>>> http://sdm.link/zohodev2dev
>>> 
>>> _________________________________________
>>> moosefs-users mailing list
>>> 
>>> moo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>> 
>> ------------------------------------------------------------------------------
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> 
> ------------------------------------------------------------------------------
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [MooseFS-Users] mfsmaster config reload breaks client mounts?

[Aleksander W. <ale...@mo...>]

Hi,
Thanks.
We are waiting for your files.

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

On 09/12/2016 03:28 PM, Wilson, Steven M wrote:
>
> ​Hi Aleksander,
>
>
> Thanks for attempting to reproduce the problem in your environment.
>  I'll send the mfsexports.cfg files to you (prior to the change and
> after the change).  I use Ansible for configuration management and I
> normally make all changes on my Ansible server in a file of my own
> format that is then used to build mfsexports.cfg files on each MooseFS
> master server.  So I'll also send the script that Ansible calls to
> build the mfsexports.cfg file and issue a HUP signal to mfsmaster.
>
>
> Thanks,
>
>
> Steve
>
>
> ------------------------------------------------------------------------
> *From:* Aleksander Wieliczko <ale...@mo...>
> *Sent:* Monday, September 12, 2016 6:25 AM
> *To:* Wilson, Steven M; Krzysztof Kielak
> *Cc:* MooseFS-Users
> *Subject:* Re: [MooseFS-Users] mfsmaster config reload breaks client
> mounts?
>  
> Hi,
>
> I would like to inform you that we are not able to reproduce this
> problem in our development environment.
> Also I would like to add that in MooseFS is scenario which can cause
> mfsmount session drop.
> When you change permissions in mfsexports.cfg file for connected
> mounts and you send HUP signal, MooseFS master will check permissions
> again, and if permissions have been changed access to cluster will be
> denied.
>
> We have another question for you.
> Can you send us old and new "mfsexports.cfg" files, so we will be able
> to understand what had happened?
>
> You can send itto dwt[at]moosefs.com
>
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <moosefs.com>