moosefs-users — Mailing list for MooseFS Users

Re: [MooseFS-Users] SIGBUS when mmap

[Piotr R. K. <pio...@mo...>]

Hey Zhongbo,

We've followed you suggestion and added this change to MooseFS sources. Thanks!

Please find an URL to appropriate commit below:
https://github.com/moosefs/moosefs/commit/709eba4e72c997888f7e319740dcf45cbde5acbd <https://github.com/moosefs/moosefs/commit/709eba4e72c997888f7e319740dcf45cbde5acbd>


Best regards,
Peter

-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | moosefs.com <https://moosefs.com/>
> On 30 Mar 2017, at 5:02 AM, 田忠博(Zhongbo Tian) <win...@gm...> wrote:
> 
> Hi Aleksander,
> 
>     Thanks for the quick reply. 3.0.90 DO fix the issue! Thank you.
> 
>     And on our production cluster, we found a lot of unconsumed messages in client's TCP receive queue. This led to periodically high load. After some investigation, we guess the client's `conncache` is too slow to digest KEEPALIVE messages. So we modified the source code to decrease the sleep time, and it seemed working for us. Here is our patch:
> 
> """
> diff --git a/mfscommon/conncache.c b/mfscommon/conncache.c
> index 4d33c19..b7a99bf 100644
> --- a/mfscommon/conncache.c
> +++ b/mfscommon/conncache.c
> @@ -161,7 +161,7 @@ void* conncache_keepalive_thread(void* arg) {
> }
> ka = keep_alive;
> zassert(pthread_mutex_unlock(&glock));
> -	portable_usleep(10000);
> +	portable_usleep(5000);
> }
> return arg;
> }
> """
> 
>   Finally, I am curious on the progress of MooseFS 4.0. we are looking forward for the erase-coding implementation for a quite long time. And we also want to know how the MooseFS guys's option on Container Storage Interface (CSI), here you can find more details on it: https://github.com/docker/docker/issues/31923 <https://github.com/docker/docker/issues/31923>
> 
> 
>   And at the end, thank you for this excellent project.
> 
> On Wed, Mar 29, 2017 at 6:04 PM Aleksander Wieliczko <ale...@mo... <mailto:ale...@mo...>> wrote:
> Hi.
> Did you tried the last stable MooseFS version 3.0.90?
> 
> MooseFS 3.0.86 client has a few bugs, but they were fixed.
> 
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <http://moosefs.com/>
> On 29.03.2017 11:46, 田忠博(Zhongbo Tian) wrote:
>> Hi all,
>> 
>>     We had encountered a weird issue after upgrading to moosefs 3.0.86. When we try to run ' TMPDIR=/some/moosefs/path python -c "import ctypes" ', we end up with a SIGBUS.
>>     After some investigations, we found it seems related with mmap, and we can reproduce this bug using following C code:
>> """
>> 
>> #include <stdio.h>
>> #include <fcntl.h>
>> #include <unistd.h>
>> #include <sys/mman.h>
>> 
>> int main(int argc, char** argv) {
>>     int fd;
>>     char* filename;
>>     char *c2;
>>     if (argc != 2) {
>>         fprintf(stderr, "usage: %s <file>\n", argv[0]);
>>         return 1;
>>     }
>>     filename = argv[1];
>>     unlink(filename);
>>     fd = open(filename, O_RDWR|O_CREAT, 0600);
>>     ftruncate(fd, 4096);
>>     c2 = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
>>     *c2 = '\0'; // SIGBUS
>>     return 0;
>> }
>> 
>> """
>>    Here is the strace for when we run this on a moosefs path:
>> 
>> """
>> 
>> $ strace ./test /mfs/user/tianzhongbo/temp/test
>> execve("./test", ["./test", "/mfs/user/tianzhongbo/temp/test"], [/* 52 vars */]) = 0
>> brk(0)                                  = 0x949000
>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f825d85a000
>> access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
>> open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
>> fstat(3, {st_mode=S_IFREG|0644, st_size=114873, ...}) = 0
>> mmap(NULL, 114873, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f825d83d000
>> close(3)                                = 0
>> open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
>> read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\t\2\0\0\0\0\0"..., 832) = 832
>> fstat(3, {st_mode=S_IFREG|0755, st_size=1697568, ...}) = 0
>> mmap(NULL, 3804928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f825d299000
>> mprotect(0x7f825d430000, 2097152, PROT_NONE) = 0
>> mmap(0x7f825d630000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x197000) = 0x7f825d630000
>> mmap(0x7f825d636000, 16128, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f825d636000
>> close(3)                                = 0
>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f825d83c000
>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f825d83b000
>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f825d83a000
>> arch_prctl(ARCH_SET_FS, 0x7f825d83b700) = 0
>> mprotect(0x7f825d630000, 16384, PROT_READ) = 0
>> mprotect(0x600000, 4096, PROT_READ)     = 0
>> mprotect(0x7f825d85b000, 4096, PROT_READ) = 0
>> munmap(0x7f825d83d000, 114873)          = 0
>> unlink("/mfs/user/tianzhongbo/temp/test") = 0
>> open("/mfs/user/tianzhongbo/temp/test", O_RDWR|O_CREAT, 0600) = 3
>> ftruncate(3, 4096)                      = 0
>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7f825d859000
>> --- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7f825d859000} ---
>> +++ killed by SIGBUS +++
>> Bus error
>> 
>> """
>> 
>>    Can anyone help to resolve this?
>> 
>> 
> 
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot <http://sdm.link/slashdot>
>> 
>> _________________________________________
>> moosefs-users mailing list
>> moo...@li... <mailto:moo...@li...>
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot_________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [MooseFS-Users] Fw: Number of files > number of chunks?

[Wilson, S. M <st...@pu...>]

No, the goal is set at 2 for this file system.


Steve


________________________________
From: Casper Langemeijer <cas...@pr...>
Sent: Friday, April 7, 2017 4:37 AM
To: Wilson, Steven M; MooseFS-Users
Subject: Re: [MooseFS-Users] Fw: Number of files > number of chunks?

A file of zero length does not have any chunks. But still: a number of chunks very similar to number of files. Are you running a system with a goal of 1?

Op do 6 apr. 2017 om 14:31 schreef Wilson, Steven M <st...@pu...<mailto:st...@pu...>>:
?Ah, yes, that's probably what happening here.  Thanks!

Steve


On Apr 5, 2017 5:50 PM, "Ricardo J. Barberis" <ric...@do...<mailto:ric...@do...>> wrote:
In my case, whenever I see more files than chunks it's almost always empty
files, like temporary locks.

El Miércoles 05/04/2017 a las 17:26, Wilson, Steven M escribió:
> Hi,
>
> I was looking at one of our MooseFS file systems today and noticed that it
> has more files than it has chunks.  Can anyone explain how that can happen?
>  Here's the output of mfsdirinfo at the root of the file system:
> ?
>
>  inodes:                  224665855
>   directories:              6659685
>   files:                  217339466
>  chunks:                  215970818
>  length:             44007258442765
>  size:               57959637532672
>  realsize:          115919275065344
>
> Thanks,
> Steve

Cheers,
--
Ricardo J. Barberis
Senior SysAdmin / IT Architect
DonWeb
La Actitud Es Todo
www.DonWeb.com<http://www.DonWeb.com>
  _____

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot_________________________________________
moosefs-users mailing list
moo...@li...<mailto:moo...@li...>
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [MooseFS-Users] Fw: Number of files > number of chunks?

[Casper L. <cas...@pr...>]

A file of zero length does not have any chunks. But still: a number of
chunks very similar to number of files. Are you running a system with a
goal of 1?

Op do 6 apr. 2017 om 14:31 schreef Wilson, Steven M <st...@pu...>:

> ​Ah, yes, that's probably what happening here.  Thanks!
>
> Steve
>
>
> On Apr 5, 2017 5:50 PM, "Ricardo J. Barberis" <ric...@do...>
> wrote:
>
> In my case, whenever I see more files than chunks it's almost always empty
> files, like temporary locks.
>
> El Miércoles 05/04/2017 a las 17:26, Wilson, Steven M escribió:
> > Hi,
> >
> > I was looking at one of our MooseFS file systems today and noticed that
> it
> > has more files than it has chunks.  Can anyone explain how that can
> happen?
> >  Here's the output of mfsdirinfo at the root of the file system:
> > ?
> >
> >  inodes:                  224665855
> >   directories:              6659685
> >   files:                  217339466
> >  chunks:                  215970818
> >  length:             44007258442765
> >  size:               57959637532672
> >  realsize:          115919275065344
> >
> > Thanks,
> > Steve
>
> Cheers,
> --
> Ricardo J. Barberis
> Senior SysAdmin / IT Architect
> DonWeb
> La Actitud Es Todo
> www.DonWeb.com
>   _____
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

[MooseFS-Users] Fw: Number of files > number of chunks?

[Wilson, S. M <st...@pu...>]

?Ah, yes, that's probably what happening here.  Thanks!

Steve

On Apr 5, 2017 5:50 PM, "Ricardo J. Barberis" <ric...@do...> wrote:
In my case, whenever I see more files than chunks it's almost always empty
files, like temporary locks.

El Miércoles 05/04/2017 a las 17:26, Wilson, Steven M escribió:
> Hi,
>
> I was looking at one of our MooseFS file systems today and noticed that it
> has more files than it has chunks.  Can anyone explain how that can happen?
>  Here's the output of mfsdirinfo at the root of the file system:
> ?
>
>  inodes:                  224665855
>   directories:              6659685
>   files:                  217339466
>  chunks:                  215970818
>  length:             44007258442765
>  size:               57959637532672
>  realsize:          115919275065344
>
> Thanks,
> Steve

Cheers,
--
Ricardo J. Barberis
Senior SysAdmin / IT Architect
DonWeb
La Actitud Es Todo
www.DonWeb.com<http://www.DonWeb.com>
  _____

Re: [MooseFS-Users] Number of files > number of chunks?

[Ben H. <bj...@ba...>]

I was of the understanding that files smaller than 64mb were concatenated together into a single chunk, unless I've been getting that wrong for like... Four years?

Re: [MooseFS-Users] Number of files > number of chunks?

[Ricardo J. B. <ric...@do...>]

In my case, whenever I see more files than chunks it's almost always empty 
files, like temporary locks.

El Miércoles 05/04/2017 a las 17:26, Wilson, Steven M escribió:
> Hi,
>
> I was looking at one of our MooseFS file systems today and noticed that it
> has more files than it has chunks.  Can anyone explain how that can happen?
>  Here's the output of mfsdirinfo at the root of the file system:
> ?
>
>  inodes:                  224665855
>   directories:              6659685
>   files:                  217339466
>  chunks:                  215970818
>  length:             44007258442765
>  size:               57959637532672
>  realsize:          115919275065344
>
> Thanks,
> Steve

Cheers,
-- 
Ricardo J. Barberis
Senior SysAdmin / IT Architect
DonWeb
La Actitud Es Todo
www.DonWeb.com
  _____

[MooseFS-Users] Number of files > number of chunks?

[Wilson, S. M <st...@pu...>]

Hi,


I was looking at one of our MooseFS file systems today and noticed that it has more files than it has chunks.  Can anyone explain how that can happen?  Here's the output of mfsdirinfo at the root of the file system:

?

 inodes:                  224665855
  directories:              6659685
  files:                  217339466
 chunks:                  215970818
 length:             44007258442765
 size:               57959637532672
 realsize:          115919275065344



Thanks,

Steve

Re: [MooseFS-Users] SIGBUS when mmap

[田忠博(Zhongbo T. <win...@gm...>]

Hi Aleksander,

    Thanks for the quick reply. 3.0.90 DO fix the issue! Thank you.

    And on our production cluster, we found a lot of unconsumed messages in
client's TCP receive queue. This led to periodically high load. After some
investigation, we guess the client's `conncache` is too slow to digest
KEEPALIVE messages. So we modified the source code to decrease the sleep
time, and it seemed working for us. Here is our patch:

"""
diff --git a/mfscommon/conncache.c b/mfscommon/conncache.c
index 4d33c19..b7a99bf 100644
--- a/mfscommon/conncache.c
+++ b/mfscommon/conncache.c
@@ -161,7 +161,7 @@ void* conncache_keepalive_thread(void* arg) {
}
ka = keep_alive;
zassert(pthread_mutex_unlock(&glock));
- portable_usleep(10000);
+ portable_usleep(5000);
}
return arg;
}
"""

  Finally, I am curious on the progress of MooseFS 4.0. we are looking
forward for the erase-coding implementation for a quite long time. And we
also want to know how the MooseFS guys's option on Container Storage
Interface (CSI), here you can find more details on it:
https://github.com/docker/docker/issues/31923


  And at the end, thank you for this excellent project.

On Wed, Mar 29, 2017 at 6:04 PM Aleksander Wieliczko <
ale...@mo...> wrote:

> Hi.
> Did you tried the last stable MooseFS version 3.0.90?
>
> MooseFS 3.0.86 client has a few bugs, but they were fixed.
>
> Best regards
> Aleksander Wieliczko
> Technical Support Engineer
> MooseFS.com <http://moosefs.com>
> On 29.03.2017 11:46, 田忠博(Zhongbo Tian) wrote:
>
> Hi all,
>
>     We had encountered a weird issue after upgrading to moosefs 3.0.86.
> When we try to run ' TMPDIR=/some/moosefs/path python -c "import ctypes" ',
> we end up with a SIGBUS.
>     After some investigations, we found it seems related with mmap, and we
> can reproduce this bug using following C code:
> """
>
> #include <stdio.h>
> #include <fcntl.h>
> #include <unistd.h>
> #include <sys/mman.h>
>
> int main(int argc, char** argv) {
>     int fd;
>     char* filename;
>     char *c2;
>     if (argc != 2) {
>         fprintf(stderr, "usage: %s <file>\n", argv[0]);
>         return 1;
>     }
>     filename = argv[1];
>     unlink(filename);
>     fd = open(filename, O_RDWR|O_CREAT, 0600);
>     ftruncate(fd, 4096);
>     c2 = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
>     *c2 = '\0'; // SIGBUS
>     return 0;
> }
>
> """
>    Here is the strace for when we run this on a moosefs path:
>
> """
>
> $ strace ./test /mfs/user/tianzhongbo/temp/test
> execve("./test", ["./test", "/mfs/user/tianzhongbo/temp/test"], [/* 52
> vars */]) = 0
> brk(0)                                  = 0x949000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
> 0x7f825d85a000
> access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
> directory)
> open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=114873, ...}) = 0
> mmap(NULL, 114873, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f825d83d000
> close(3)                                = 0
> open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
> read(3,
> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\t\2\0\0\0\0\0"..., 832) =
> 832
> fstat(3, {st_mode=S_IFREG|0755, st_size=1697568, ...}) = 0
> mmap(NULL, 3804928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0)
> = 0x7f825d299000
> mprotect(0x7f825d430000, 2097152, PROT_NONE) = 0
> mmap(0x7f825d630000, 24576, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x197000) = 0x7f825d630000
> mmap(0x7f825d636000, 16128, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f825d636000
> close(3)                                = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
> 0x7f825d83c000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
> 0x7f825d83b000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
> 0x7f825d83a000
> arch_prctl(ARCH_SET_FS, 0x7f825d83b700) = 0
> mprotect(0x7f825d630000, 16384, PROT_READ) = 0
> mprotect(0x600000, 4096, PROT_READ)     = 0
> mprotect(0x7f825d85b000, 4096, PROT_READ) = 0
> munmap(0x7f825d83d000, 114873)          = 0
> unlink("/mfs/user/tianzhongbo/temp/test") = 0
> open("/mfs/user/tianzhongbo/temp/test", O_RDWR|O_CREAT, 0600) = 3
> ftruncate(3, 4096)                      = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7f825d859000
> --- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7f825d859000}
> ---
> +++ killed by SIGBUS +++
> Bus error
>
> """
>
>    Can anyone help to resolve this?
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> _________________________________________
> moosefs-users mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
>

[MooseFS-Users] Fwd: loading moosefs on isilon node

[Piotr R. K. <pio...@mo...>]

Hi Rupert,

thanks for all the info!

-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | moosefs.com <https://moosefs.com/>

> Begin forwarded message:
> 
> From: rupert chandler <ru...@bt...>
> Subject: Re: [MooseFS-Users] loading moosefs on isilon node
> Date: 27 March 2017 at 7:10:21 PM GMT+2
> To: Piotr Robert Konopelko <pio...@mo...>
> 
> I would guess that this would be an issue as the disks are accessed via a specialist raid card and there are also other functions such as caching etc that would be dependent on the isilon drivers and onefs os tweaks.. it’s a simple supermicro box with some very special bits inside..
>  
> Everything is isilon controlled down to drive firmware versions.
>  
> You also have to bear in mind that the onefs os itself is subject to licensing as in reality isilon gives you the hardware, they charge you for the very clever software.. in this case as the user probably bought the system from isilon this may be ok, but I would suggest being careful is a sensible way to go..
>  
> Good luck!!
>  
> Rup ;-)
>  
>  
> Rupert Chandler 
> Tel: +44(0)1672600139
> ru...@bt... <mailto:ru...@bt...>
>  
>  
>  
> From: Piotr Robert Konopelko <pio...@mo...>
> Date: Monday, 27 March 2017 at 18:01
> To: rupert chandler <ru...@bt...>
> Cc: MooseFS-Users <moo...@li...>
> Subject: Re: [MooseFS-Users] loading moosefs on isilon node
>  
> Do you maybe know if there is any possibility to install vanilla FreeBSD or any other custom OS then?
>  
>  
> PS: Please reply to all (i.e. to the MooseFS-Users list, which in CC)
>  
> Best regards,
> Peter
>  
> -- 
> Piotr Robert Konopelko
> MooseFS Technical Support Engineer | moosefs.com <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnzOtmNIRfbm4mFSMCAdNP-2B4-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA3FmQw86BIUK9pA0ZLh7XC6BY4-2BtsFGJXQHQ5krQOf3rOtlRdQjufXWGL7sgP60NXm64oAg60C2yhel3EHHaqPIzPxD8vO9TUldG4mZpj-2BddXrLzXgU2GTfQ-2BDzyjMGZsLzHUZVW9gtLF7GXGy0nDPk8xqicsku7P5CAULwDzvFOtbyz5BuX0tUhdnY7v-2BS-2Fus-3D>
>  
>> On 24 Mar 2017, at 6:14 PM, rupert chandler <ru...@bt... <mailto:ru...@bt...>> wrote:
>>  
>> The basic os is certainly freebsd but its very heavily tweaked by isilon to run their system as you will imagine.
>>  
>> Maybe possible to run moose but I would out money on there being issues switching off isilon functionality to stop it upsetting moose and vice versa..
>>  
>> rup
>>  
>>  
>>  
>> Rupert Chandler 
>> Tel: +44(0)1672600139
>> ru...@bt... <mailto:ru...@bt...>
>>  
>>  
>>  
>> From: Piotr Robert Konopelko <pio...@mo... <mailto:pio...@mo...>>
>> Date: Friday, 24 March 2017 at 18:11
>> To: Rupert Chandler <ru...@bt... <mailto:ru...@bt...>>
>> Cc: MooseFS-Users <moo...@li... <mailto:moo...@li...>>
>> Subject: Re: [MooseFS-Users] loading moosefs on isilon node
>>  
>> MooseFS can be ran on FreeBSD :)
>> We support this OS and have native and up-to-date ports for FreeBSD (see e.g. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216974 <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVv-2BwesyRx4hMxGeV1tFTtFGULntG6rYFpl2-2F4FAXho-2BmsfD4Sa8xUuTngtM2ziiYiSj1SrcD36brpegDosUmH37qDtPCPHyGwMK0ArC6K4KMTx-2F4b591rkh-2Bz5k34DfLiIkTHOMruRPVc3E-2FoGqi6xEakp3eeb0axxbRm1kKQrgvNxQtDjLJYa00O2kaaGU9dJ4T3xkqCPT1Zv4KatvgtCZCz7pRf0rhjFyH5KkOLQAd9bIjk0SV4IBGUZZDM-2B31fJ1oXom6HgC52IAyN30XSM2grS7aYee7Y65FAPKyosXEZ5QWCgZkJ5JfvI9RU-2FKC3hKpT7rDtGKlca-2FSSrrAngwQemdxavSj1BTuLpqe8-2FVnVxcjgu9YmI9bkjIjlpeOkV50xv9uLWqbMyEmauWyFkYC3fJpR0m8opXNhTQIcnvF-2FXMduOgEF5EMceNPhhzE02JGy-2BOV6hYY4tPt3yMLR2w-3D-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA3FmQw86BIUK9pA0ZLh7XC6BY4-2BtsFGJXQHQ5krQOf3rIjeT5W6DuxCQ-2BIZavPT08qb-2FQeETWX5HuSyGYyqkbgKjCpptCcyZVox7aqZnZ1CsWexwToEOAhVdWYZ4LNE4Ypt315Ikne-2BnlUyuZoW-2Bu61QoxfjKC2nG2ad9abNGKzQw6UIDLGSOKWwh8Bctx8WpE-3D>)
>>  
>> -- 
>> Piotr Robert Konopelko
>> MooseFS Technical Support Engineer | moosefs.com <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVv-2BwesyRx4hMxGeV1tFTtFLGG-2F0rr67j029dCnPFRWFbsLl9834wBCN3UXFwRIq7c06sKudO2BzWAFdUEjdY0pIt93XVKPsGWNIZt2z9ceL9QBBA33QOv-2FNTO-2Bp2C-2BXKxcp2Xa4Htfxy4Rg3ivw3SWVYi-2B5nZAlpHLHQTRgkaJ2mEc2ietqGgvLtrDdYFxR-2BCXIwh-2BrKY1NHr2raKTpNJJq2UBFgkmqCA6742RvdAxVRLlf7tdLCJhFTPbwCfokXCQXMf01WW4s0NFjdkIvmEMt9Sqhgteyj0yeBuxW6mA2ebYGs6hnjFp8pdDLDJayOHmumc4KMQTQwBUutk2Tizx2F7C6EWnCjz7fu7nM7ieamnPQd5fshs3hWnfgtTFeXE0nMkzYyPpMIJYn2WyBjmBg-3D-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA3FmQw86BIUK9pA0ZLh7XC6BY4-2BtsFGJXQHQ5krQOf3rEjFFfmLoCNLXzy68In0n1uGlFMDdMjOfx3JypFan-2BuVKzq1-2BpmBhnCMw1Unu3C9n3a7nYIe-2FuxR0XUBQOqNIJfI1CceHPvfvEkTj-2FZOsBoFbq1-2BOX2o-2FnmLJ-2Fplov4ortU9Ieg39XU1QM00Qp7kQ-2FI-3D>
>>  
>>> On 24 Mar 2017, at 6:22 PM, Rupert Chandler <ru...@bt... <mailto:ru...@bt...>> wrote:
>>>  
>>> The OS is freebsd on the isilon. 
>>> 
>>> Rup
>>> 
>>>  
>>> ___
>>> Rupert Chandler 
>>> Tel: +44(0)1672600139
>>> ru...@bt... <mailto:ru...@bt...>
>>>  
>>>  
>>> 
>>> 
>>> 
>>> 
>>> On Fri, Mar 24, 2017 at 5:05 PM +0000, "Piotr Robert Konopelko" <pio...@mo... <mailto:pio...@mo...>> wrote:
>>> 
>>>> Hi Ali,
>>>>  
>>>> if there's a possibility to install Linux on Isilon nodes, it probably will be possible to run MooseFS on them.
>>>> You can check out supported OS-es for MooseFS here: https://moosefs.com/download.html <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVv-2BwesyRx4hMxGeV1tFTtFOHK1uar9nj9bXryKkOD21MF5L079T-2FzO4MaDf4W7pB-2BpN3aNv0Pt-2BY3CMO0rUnqKYpMgf7gTC5-2ByMAXpl1eTjbiwgAEm-2Bu-2BsjEQ1TAuPUxBzGtmKxE4lLagg1iMg6WChwEUsOTi0vrKgmbQoEu-2BjwR4dMhQPUgo-2FSuwmfMz7NW-2F-2B-2FqqMcpdF8nAZ3MAlBrRZJ-2FXm1OG0eRquGCFyNWMKaDgOX8WkhmTQk2htGsB66YF-2FuI8G-2B6mhKblyNGVcjxw8W-2B2wwGAiu833IgNRkpcq6j30FAiIjqrcQXl4tp62lpxKcf8PIVpV2OGICknMBT467CTHoNPAwCCRlGcwUjdJbmsqTFL0dJE5drLTr29llSqtGq5cpFCFHNwuhkH8rjU3VhZNbDQ5jsbWJO5-2BY1WSHzcBt4K8WATwQm567mqan-2F1kdkrqyU2RZcMGh1otWLv6f2FnpuOkWqHXgNU3kp9ecxPckFhodapmGPnaA1CqJeeRG1kF72hb0hFTXAJajvhs0si6I5IV9zyfFH-2FUqXn0R0C2t5HETGo-2FK7tQZi3VUnNccx8U6CtEIeCPnOB6k86u9tBvXTN4C58hQVn9vBshTugNzmaLyq3nXzhwKw1ofPki8lZhE-2B9yhm30LqhvMjYDuTr0sKircWtvCyTj4FF6pp3ITwBNkOZCp0J8JhwuNDloPcM2tyx55oWmFZleaDaUzwmgMMLoizmVRGRwM63Wm5TKV8Pydcsw5GcmLOKnexcN1KS-2F-2FZvDYGv77CDwrlfQVNToz7m9hDFlglk63P7Dkcj-2FGEtNPCjefaR-2BJcpmV20qGPpaMeAS0Y4enhSypB4xYYMsy4cPHykBTwcH9BB6d44GyCKFid9idALL9KStd3FUyT9jPjxKAmTPdwLtezVo9qztVOpCbIFvuyDRkHAN1YEDUs7MTiggqlm3MhixZqFuYJU5yqU3SyC6cFJGd6F8QxxjWtR7uGgLjUOfszFVvXK0HLZWhAJk-2FbRxdQynzAHQ-2B5ijmCF1GbuKImA4pN-2F5buXE7hTP-2F5zm4pCiibhVw8-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA3FmQw86BIUK9pA0ZLh7XC6BY4-2BtsFGJXQHQ5krQOf3rK3ZFdMOCFxGAf2HJIaf3q-2BT9391RsMZWvsArYAwVdLKg5-2B7pMbQK1Gv-2FrhNMXd7QiIyeJr9EGToTj5D6gginGnNbiayBst7TPhqxUDmUoXighazViyJ67QedTZT-2BU-2BUr5N7QNuXsgoIFmppEFn5LxQ-3D>.
>>>>  
>>>>  
>>>> Best regards,
>>>> Peter
>>>>  
>>>> -- 
>>>> Piotr Robert Konopelko
>>>> MooseFS Technical Support Engineer | moosefs.com <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVv-2BwesyRx4hMxGeV1tFTtFOHK1uar9nj9bXryKkOD21MF5L079T-2FzO4MaDf4W7pB-2BpN3aNv0Pt-2BY3CMO0rUnqKYpMgf7gTC5-2ByMAXpl1eTjbiwgAEm-2Bu-2BsjEQ1TAuPUxB6rTmbusIhn6gQOiRQkawMLIrhL1F4fByeRT1ZSidvFmdsJAVbbW4vFDMHM1FcmMDXOOASH17JJ3WANn6IsJMwoSauoQoMd4Tnw652V7tUwAKLdWPQuicuxsaj1iyBFmDaF3hAJrdSq2yxVcp4sUy5547s1vysWWSfMElpZ86c7E4FdP0UjHxTBxlrsUj0PErKGE5RY-2BqKLqqSC8qrGypf2aZPX2GhhaMyf3hbbEgc8VBBAYojcX91-2Bn3YiUb0tL7SoPj1wwegW9dJAYZ-2BOdF522loRJvLEhHu22UHen2TRo9kinzd3HzCBlunaLSXRQSdKO5v9YlYJQyqoxZbBwC17iPfpW4tTB7ANvg-2FsJ6SEmq85VSqFEoNHwdsKe0BYcIUW3-2F8TJCozzVbbBLJDxnd422R-2FmI988JNGp8Xz3AW1yUg57GEaWC0-2F7KcH8oxSy3rmp-2FzpRbDMOBE-2BSyBXoa6pxNgO9E3BQOVgIx7haNs1eXWaa01mpTEVtE8azfK1aBRnG8Y1wmOSubsiM3o5IOuwGsGLghdg9EuIKQXPb2LUNON0-2Bw5mGelX3xmBbaPd-2FRLIs-2F-2F1BQ0ipn4AjatZd-2BKA5X1Ph2cpOd2RJIe7KMv20txkxVwHzWjzu3HkmunNaCNR8oGkpky8byH-2FaY2uotFlvhcIJTYvjFwPgs9kig1HHIBq9M6juVDnbwk1S2p8SJTueWrBAygZmuGis0gkhm6BajVliDX8UyuxbEy0dNTc8W3lpoFjXoIszeZGVsbQIv-2FMnEh6a-2FUTZ01cYfK6rZO5urJiAI47yS2HDZvN4yhn956kUy8g9aXxCk4ftnwjR4uCFO5suDKHjVNQU30MEFgpoiCXET8Dh3QFYxcQJ37B417LoFn-2F2bi1u-2FWjdHZLmX_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA3FmQw86BIUK9pA0ZLh7XC6BY4-2BtsFGJXQHQ5krQOf3rAIFNfZ7BdgRQ25F-2Bbqbe7dWn-2FB-2Fid11iBKso8lQR1yHqxlfG4aVUHxRmQ-2BioIBbn5nOj5xcyRLfCiatswD8S2X-2F6gmheR9U46UTrQee4-2BCnoAowF50WzlKGPDmI2DKV6LzN-2BCx-2FVJqOWOnVADTPmyE-3D>
>>>>  
>>>>> On 22 Mar 2017, at 8:37 PM, Ali Moeinvaziri <moe...@gm... <mailto:moe...@gm...>> wrote:
>>>>>  
>>>>> Hi,
>>>>> 
>>>>> We have a few end of support isilon nodes (NL36) that would like to load moosefs on them
>>>>> and use. I wonder if anybody has already gone down this road. 
>>>>> 
>>>>> I appreciate any notes, thoughts, or comments.
>>>>> -AM
>>>>> 
>>>>> ------------------------------------------------------------------------------
>>>>> Check out the vibrant tech community on one of the world's most
>>>>> engaging tech sites, Slashdot.org <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVv-2BwesyRx4hMxGeV1tFTtFOHK1uar9nj9bXryKkOD21MF5L079T-2FzO4MaDf4W7pB-2BpN3aNv0Pt-2BY3CMO0rUnqKayMOgwnTTa4YvBWXribzdrm1PqLcRuN1YoI8ziutyTl2586CEqcFD4Q11IgE9FkxoMVzWxkiNZwZWmPfZjYGD0NqccOR2Y7TFN-2Fj-2FrWdtQRN6chMjj3EvB9KEO6dPkHYc-2B2qSWcJsmwCtvjrg03HgPKL964mKKWoorhDDh8GitdjOPbcTdkzovinkUuGyiKFXaTo5Wr0DlbtvZr-2BpTKbUytyqC3U5wVAn4U9J1Qdekkaar2PSlUd29O-2Fr-2B-2B7FqRKbGCWC15u0vIPHadtRU8oqcf31QkRqxtRYrSSWx11b-2FLLYMbIFag8M0JYAKFJDAIsVh-2BBYPIjcJM68nGfANBWNCgLK-2F9FfHR-2Biua8f6In7eMVVd04gouPf9OYEWO0UTI04L146y9UjSQOaSKbgrKoo78ZlQkOPHxed03JHlNipWYeOgqf8TforKHZNvNDYJpJfQ9UN-2Fn08Nh6v33-2BB8GLLQsuln0G5x3Wv-2ByDeGG5ERMrqlbfOEbrd5f2bl5kcweCA5JUGWsmpek3FcRemY28uHwuVdrRjNIG-2FUwhSA8GhUmX3GKvj3LNzmWfAQd7H06mo097vyuymE8mopoURXkn3Qh-2FP-2Bhgn42lxCGK6JjUS-2FB-2FePLmEkK3aASHt-2FB-2BVHxiA85SLdQVD9WVxJU0fbkbGJkRAQJBv6PHt-2F39x5QfqqjkFdjnvk8ep9HOLJbLWcCerKwVgQZo5IGHRjOWbujX9DihajdzO04UvLMOYOuFWyAtAQmWgzlIsps-2Bpgt6bxygI1KzCuz-2Fw6IPrYrVjjbEPyM-2FCWnsQA242JoUjNKuaJCLZ-2BJ0hztrYe0zuxWNTUo3YybXQtGocst0w3KFFncOz1yZ1orfSfoS5BXuRTenxx9EvIimmeTRhgKUfYeAHsIXcjwD0Z7rCzU10dnLpv44ZA-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA3FmQw86BIUK9pA0ZLh7XC6BY4-2BtsFGJXQHQ5krQOf3rA0xMfZ8CNgFl0bEI0mHAgPm1xK64IwmOit21aMiy9T0dOK78fPpDvuIlLq9ZRCJLJUJqLJx-2BUK3nH42JPHGVOHVq-2FWjpG6M6QMOeFdgcIkpERL-2F87bm0A5ktDp9Inex1ox-2FW1n8oCi-2FGWNi2wjxEAo-3D>! http ://sdm.link/slashdot_________________________________________ <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVv-2BwesyRx4hMxGeV1tFTtFOHK1uar9nj9bXryKkOD21MF5L079T-2FzO4MaDf4W7pB-2BpN3aNv0Pt-2BY3CMO0rUnqKcFhg6zH7VHTOHwmPVbEqmNqitxCQ1rzFdQn-2FosFNtt5yyhK1LUPwO3ee55XDsg3obBXofIbJHJ3ybnYhMrEQkNtYtL8v3dxLshpxplPzBmC2QuCg-2Bl9oP8bl2tg2o5RTEbFv3ZErtmMCdv7L-2BfH8EHkp9l6vtY0wBzaZ4QKgUOOBHlSXdEeB8tksdVL1UKRX03PCNkqWBmoS1NVg8vIMHWobrMd1hLaL-2F44FTKQJQoKvLFPd7-2Bt6MwuGensep2NmeApypQvmaGorYFhVa7cAEWTjig-2BnEGTbFMwNCN9ttR7A-2F2UNoTRGQ2zD5t2AEHoXOI2hzZHP58McnM2x7OQzkVqLBlsDMfmCQR0ng-2F481cbS0UNWgberfcn8bMter-2BGeI3MKa7Nr7M3RGumGLcojarGKYFo-2F6JUMmg11lIZJP0F0G5wwMqcoSlC4HiYnm11ym1P4iMrIMPPK7LlEoIlGz1yzefqkkvrJkSohDJM-2BWqf-2FFqcmP-2FD70-2BK8Zn929UvPedUUzjtk6WOJGYHPnZgdBYZtugyIcGKbCc68VTEZqNL9qeM4LK8hMBrd5DAHOwy3h1J2ozmhKp-2FgXuMY5R2VIxQaDia25E2nTixZvWQG1tpjyFnGEulh7umcHyCyze-2BJyXoGTX1P4jSNBRK2rlo3Urcy8hfhoIrCkY5odiJi00A0bjfwr-2F-2Fc2r7xZwLQABD3AtKmTmIhn46x6qoSpQh8yNHrEHQNAKuE7vc23yLn-2Bv-2F45kYgI7Q2-2FAbqlmgqQzEiN8rDRGd7mIU2nRu-2F8aFK0R8-2F9gyub1gDD66rWkOvQjLvrRKmTqnu-2FiNIq6EHmcLRikoJ0HfFua3htXjYQme8t9xWZ-2BC26auYdwVLF7yuNyEai7NbTUqprW4CQUlHqJiWKZu5cGp4M4XvYGNZfAA9PXhIOQRXXr46CSWqk3yMN6hFfiKCms1Z5XCon-2FyWXgqkMNf4v383z6YayHk6NcZ8M5PSlxLjWGhv9G0L6ky0ILTcw6fIm1IDaw4Vzq0Z5hbr4O-2Ftwx-2BKczQ2gcnyLnMD-2FCwDhiGa6TjLWwg7TCy0Qf2_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA3FmQw86BIUK9pA0ZLh7XC6BY4-2BtsFGJXQHQ5krQOf3rH82C8pKRBh8VzCen1h74mAYEjaNniMujfAvrgPv9kN-2FreHPQxDSbb7FoAl6akA5hhEzwJoIiTA7Kpl2zSScyJwxLMs4fMfV35gj2L7cglOVirqxx3sxu-2FthD1l-2B7eF7FwgB-2FF0broHZtJDH0Ef1nV8-3D>
>>>>> moosefs-users mailing list
>>>>> moo...@li... <mailto:moo...@li...>
>>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVv-2BwesyRx4hMxGeV1tFTtFP04omKrKhuHKR5uXNKZXYmnSbbP1avEt9BkQ66YdJ8Mvh4Z5SNkz5UJxUYXTMGsQKYocHnCmnEghLzZlbuB2QwRk4NOM3kTe7zYec7InpP8oQqhLmn6OL66iQqLq0qJOwFpIhxApMuAwnJ-2BZ4cUnsP7ZBQO8OoBXZLyz-2FsaW7JJUwXjFI41UsMWXGCt7cwxcM10-2F4TLiByxVuQudkiZRJobJmdHq3y8jNfzZRSL0zQx4nOBQHJ9V05-2B0jZ7z-2FCQViCIitV6dsDyNNplQdLEDI3jjPkh8TEB3NDtfd5D-2FOXf-2FK6Z7qy1zRmbZHQZ0MdSQNILjJon4gWOWwM1x6v2TI4lyUbyfA3hfh-2F9eL-2FGH6bqglNWM7mWoGYbTng9ZHrUrM4L7CTOMXwHZ2ySy9lBkGF5rGjl7xwVNzhc-2BDn53e7GdngbQIh4JzfyKceAmA22c8RrGJMuEJoKnSD-2B1Mz64fJ9wuCnESURC9W1odWpkglt_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA3FmQw86BIUK9pA0ZLh7XC6BY4-2BtsFGJXQHQ5krQOf3rFegDhZ3nhdBBOztQzdYtrdUIG-2F1TcgAM7NsuvlLf-2FFFPq84cpYWUtJV6iTO-2FElLZ-2BdIfevtUbMKQY1fl3Sk3MCOp0YaLgeeY0ZsgRh2P5r4UDHlfZDSp1JwfLI4NZfuVOnLSHtbaS6ccsB-2B5Qtb2Q0-3D>
>>>>  
>> 
>>  
>> 
> 
>  
> 

Re: [MooseFS-Users] SIGBUS when mmap

[Aleksander W. <ale...@mo...>]

Hi.
Did you tried the last stable MooseFS version 3.0.90?

MooseFS 3.0.86 client has a few bugs, but they were fixed.

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>
On 29.03.2017 11:46, 田忠博(Zhongbo Tian) wrote:
> Hi all,
>
>     We had encountered a weird issue after upgrading to moosefs
> 3.0.86. When we try to run ' TMPDIR=/some/moosefs/path python -c
> "import ctypes" ', we end up with a SIGBUS.
>     After some investigations, we found it seems related with mmap,
> and we can reproduce this bug using following C code:
> """
>
> #include <stdio.h>
> #include <fcntl.h>
> #include <unistd.h>
> #include <sys/mman.h>
>
> int main(int argc, char** argv) {
>     int fd;
>     char* filename;
>     char *c2;
>     if (argc != 2) {
>         fprintf(stderr, "usage: %s <file>\n", argv[0]);
>         return 1;
>     }
>     filename = argv[1];
>     unlink(filename);
>     fd = open(filename, O_RDWR|O_CREAT, 0600);
>     ftruncate(fd, 4096);
>     c2 = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
>     *c2 = '\0'; // SIGBUS
>     return 0;
> }
>
> """
>    Here is the strace for when we run this on a moosefs path:
>
> """
>
> $ strace ./test /mfs/user/tianzhongbo/temp/test
> execve("./test", ["./test", "/mfs/user/tianzhongbo/temp/test"], [/* 52
> vars */]) = 0
> brk(0)                                  = 0x949000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0x7f825d85a000
> access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
> directory)
> open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=114873, ...}) = 0
> mmap(NULL, 114873, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f825d83d000
> close(3)                                = 0
> open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
> read(3,
> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\t\2\0\0\0\0\0"...,
> 832) = 832
> fstat(3, {st_mode=S_IFREG|0755, st_size=1697568, ...}) = 0
> mmap(NULL, 3804928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
> 0) = 0x7f825d299000
> mprotect(0x7f825d430000, 2097152, PROT_NONE) = 0
> mmap(0x7f825d630000, 24576, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x197000) = 0x7f825d630000
> mmap(0x7f825d636000, 16128, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f825d636000
> close(3)                                = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0x7f825d83c000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0x7f825d83b000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0x7f825d83a000
> arch_prctl(ARCH_SET_FS, 0x7f825d83b700) = 0
> mprotect(0x7f825d630000, 16384, PROT_READ) = 0
> mprotect(0x600000, 4096, PROT_READ)     = 0
> mprotect(0x7f825d85b000, 4096, PROT_READ) = 0
> munmap(0x7f825d83d000, 114873)          = 0
> unlink("/mfs/user/tianzhongbo/temp/test") = 0
> open("/mfs/user/tianzhongbo/temp/test", O_RDWR|O_CREAT, 0600) = 3
> ftruncate(3, 4096)                      = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7f825d859000
> --- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR,
> si_addr=0x7f825d859000} ---
> +++ killed by SIGBUS +++
> Bus error
>
> """
>
>    Can anyone help to resolve this?
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

[MooseFS-Users] SIGBUS when mmap

[田忠博(Zhongbo T. <win...@gm...>]

Hi all,

    We had encountered a weird issue after upgrading to moosefs 3.0.86.
When we try to run ' TMPDIR=/some/moosefs/path python -c "import ctypes" ',
we end up with a SIGBUS.
    After some investigations, we found it seems related with mmap, and we
can reproduce this bug using following C code:
"""

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>

int main(int argc, char** argv) {
    int fd;
    char* filename;
    char *c2;
    if (argc != 2) {
        fprintf(stderr, "usage: %s <file>\n", argv[0]);
        return 1;
    }
    filename = argv[1];
    unlink(filename);
    fd = open(filename, O_RDWR|O_CREAT, 0600);
    ftruncate(fd, 4096);
    c2 = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
    *c2 = '\0'; // SIGBUS
    return 0;
}

"""
   Here is the strace for when we run this on a moosefs path:

"""

$ strace ./test /mfs/user/tianzhongbo/temp/test
execve("./test", ["./test", "/mfs/user/tianzhongbo/temp/test"], [/* 52 vars
*/]) = 0
brk(0)                                  = 0x949000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f825d85a000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=114873, ...}) = 0
mmap(NULL, 114873, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f825d83d000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\t\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1697568, ...}) = 0
mmap(NULL, 3804928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) =
0x7f825d299000
mprotect(0x7f825d430000, 2097152, PROT_NONE) = 0
mmap(0x7f825d630000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x197000) = 0x7f825d630000
mmap(0x7f825d636000, 16128, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f825d636000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f825d83c000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f825d83b000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f825d83a000
arch_prctl(ARCH_SET_FS, 0x7f825d83b700) = 0
mprotect(0x7f825d630000, 16384, PROT_READ) = 0
mprotect(0x600000, 4096, PROT_READ)     = 0
mprotect(0x7f825d85b000, 4096, PROT_READ) = 0
munmap(0x7f825d83d000, 114873)          = 0
unlink("/mfs/user/tianzhongbo/temp/test") = 0
open("/mfs/user/tianzhongbo/temp/test", O_RDWR|O_CREAT, 0600) = 3
ftruncate(3, 4096)                      = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7f825d859000
--- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7f825d859000} ---
+++ killed by SIGBUS +++
Bus error

"""

   Can anyone help to resolve this?

[MooseFS-Users] MFS 3.0 Internal Rebalancing vs Undergoal Chunks

[Michael T. <mic...@ho...>]

Hi.


I recently upgraded my setup from 2.0 to 3.0.  My master, all metaloggers, and all chunkservers are already upgraded.  Most of my clients are still on 2.0 and will be upgraded this weekend.


A couple of days ago, one of the chunkservers had a disk start to fail.  So I marked it for removal and restarted the chunkserver service, and MFS started removing th chunks.  Not long after, the disk totally failed, but the removal of the chunks have not finished.  No big deal though.


So I replaced the disk with a larger sized one: old=2TB, new=4TB.  Restarted the chunkserver and the new disk shows up in the CGI GUI.  24 hours later, however, I noticed that the number of undergoal chunks did not seem to shrink much.  Looking at the logs, it seems that Internal Rebalancing (e.g. moving chunks from other disks into the new one) seems to be taking priority over reaching goal for undergoal chunks.


How can I get the master to do the reverse, i.e. prioritize reaching the goal for undergoal chunks over internal rebalancing?



--- mike t.

Re: [MooseFS-Users] loading moosefs on isilon node

[Piotr R. K. <pio...@mo...>]

Do you maybe know if there is any possibility to install vanilla FreeBSD or any other custom OS then?


PS: Please reply to all (i.e. to the MooseFS-Users list, which in CC)

Best regards,
Peter

-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | moosefs.com <https://moosefs.com/>
> On 24 Mar 2017, at 6:14 PM, rupert chandler <ru...@bt...> wrote:
> 
> The basic os is certainly freebsd but its very heavily tweaked by isilon to run their system as you will imagine.
>  
> Maybe possible to run moose but I would out money on there being issues switching off isilon functionality to stop it upsetting moose and vice versa..
>  
> rup
>  
>  
>  
> Rupert Chandler 
> Tel: +44(0)1672600139
> ru...@bt... <mailto:ru...@bt...>
>  
>  
>  
> From: Piotr Robert Konopelko <pio...@mo...>
> Date: Friday, 24 March 2017 at 18:11
> To: Rupert Chandler <ru...@bt...>
> Cc: MooseFS-Users <moo...@li...>
> Subject: Re: [MooseFS-Users] loading moosefs on isilon node
>  
> MooseFS can be ran on FreeBSD :)
> We support this OS and have native and up-to-date ports for FreeBSD (see e.g. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216974 <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbn-2Fsis1hxKCikh0qTQbXM3qlX4RA1DtbNtUrsgAX2DGiitxPmwk131bkgTGSiDejRxA-3D-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA2AT0STS9CATYpTbZwvUiOxe4HhhryLaM79oO5umb73mwn97-2FFhuZvq9itzLG3qEuyRscqS-2BrwqSQF3wlKHJSUDGQysyqxQTugd0JAnJ11G2taQmmiZedIH92exsSl32-2B-2Fe3L5CBNRGJYNb5K2dA-2BFU9ftTPBwNXFjaF0004vDCIfiWNTrICqIAL9qBouaWGys-3D>)
>  
> -- 
> Piotr Robert Konopelko
> MooseFS Technical Support Engineer | moosefs.com <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnzOtmNIRfbm4mFSMCAdNP-2B4-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA2AT0STS9CATYpTbZwvUiOxe4HhhryLaM79oO5umb73m-2Fodgc7NzVfn3fZr80gxX4Mzslstf4P-2FR14MfT3pRFtV-2Be3-2FLV8cpRiGn1WTn5NosJIHvoBuQr78e4nDNSLILS5WllHYuC72jVV343GjMbAJDV6YxJePIBLxfy1SaH0ZBqas27HQGrGZ4s5tz7T83Qc-3D>
>  
>> On 24 Mar 2017, at 6:22 PM, Rupert Chandler <ru...@bt... <mailto:ru...@bt...>> wrote:
>>  
>> The OS is freebsd on the isilon. 
>> 
>> Rup
>> 
>>  
>> ___
>> Rupert Chandler 
>> Tel: +44(0)1672600139
>> ru...@bt... <mailto:ru...@bt...>
>>  
>>  
>> 
>> 
>> 
>> On Fri, Mar 24, 2017 at 5:05 PM +0000, "Piotr Robert Konopelko" <pio...@mo... <mailto:pio...@mo...>> wrote:
>> 
>>> Hi Ali,
>>>  
>>> if there's a possibility to install Linux on Isilon nodes, it probably will be possible to run MooseFS on them.
>>> You can check out supported OS-es for MooseFS here: https://moosefs.com/download.html <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVv-2BwesyRx4hMxGeV1tFTtFAZyxWDzJsOtrAj6eWv37-2F36ABrJ0TTT33xGEFzJj95ZwkLaFCbEfboy1EhVeeRF-2FKQae-2FBpWC4yFfA7KHTzEwh99XA-2FbE1FrnRSAg45X2-2FKKb-2BbRoyCmSRXnjN0R11SgmCCAaRVLF04ZnMnaKCMDZVFhw5ogLeJoTLP5o7pahNfDDpmx-2F-2BkA-2BIXltN89y8EcksNaGxv5E1XVZzUUqsN5Ro-2BbUNmMfqqZ5nkM3MRe-2BgNKA1t3IDLa0ynt8IUUyfOSpUG8CnoD951BockubU62-2Br4VUXsdjbJJYsTbAucUmQPaHRIwpDrV5dMgqRcXYdiR7t8vf2sKuyVhmgriT3bUXZsg-2BGH-2Bou0oTugJIlyu4Tz0si4VYNUZBblJU46KUuwfdgeJkm8TqBYOJKws5pcUMayKlRXlnEc0B5cAcWQDTlz_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA2AT0STS9CATYpTbZwvUiOxe4HhhryLaM79oO5umb73m5b9kAYI-2BpMfxtGG60hDzBcxuw3QDaOy6dqrQNS2aKqGWtXhcSTU4onXrBXuWYZa9qVGHbUELAuMBQXSC9pzjTb4TWYQCkIjWjqNfaMwzPXTXgVIzI08se8WdAGFvRwn-2B-2Bli0iVePtDh1OwTAsMGRtY-3D>.
>>>  
>>>  
>>> Best regards,
>>> Peter
>>>  
>>> -- 
>>> Piotr Robert Konopelko
>>> MooseFS Technical Support Engineer | moosefs.com <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVv-2BwesyRx4hMxGeV1tFTtFLGG-2F0rr67j029dCnPFRWFbsLl9834wBCN3UXFwRIq7c06sKudO2BzWAFdUEjdY0pIt93XVKPsGWNIZt2z9ceL9QBBA33QOv-2FNTO-2Bp2C-2BXKxGk2ging45xvxGnoVY3q-2BrnAURp35K3QZYNaTd19xwlSecgV5h1fvR3A9hGF6i0RI4Ab5K-2Bqbo3MvlCyRCTZoE4NZEw9Pajqd0SouHVZwsUW6zwSadkCFw4P9XIy9rDGUlizJSzV8KJ6zRYLAGQ7GQFJhnT3xudw146UihkHvWRGlF0NNlO1vMw9x07LmVZ-2FvdkTFH0PUH8zu-2B5UcQnVbAXCpp8i2FyqB34KN5KJxtED18xD3iK3WJBqTzjwR-2FWanqgUioZmeE9G3mpo0cGFq8Q-3D-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA2AT0STS9CATYpTbZwvUiOxe4HhhryLaM79oO5umb73m-2FX10Cwg0pFQUNB28N531Jll3QhFVTZfiffz5cvC1yyCbzsB3Qi2AyFVq8I6w-2Fajg1WpAs-2B10saHwPdXt1UgpyVOaFEzfrSTs8r5lNhjdhNOSWZJA1VK8l6UnychsDn42PSfaJ-2B7kfPYDh05lcPIXC4-3D>
>>>  
>>>> On 22 Mar 2017, at 8:37 PM, Ali Moeinvaziri <moe...@gm... <mailto:moe...@gm...>> wrote:
>>>>  
>>>> Hi,
>>>> 
>>>> We have a few end of support isilon nodes (NL36) that would like to load moosefs on them
>>>> and use. I wonder if anybody has already gone down this road. 
>>>> 
>>>> I appreciate any notes, thoughts, or comments.
>>>> -AM
>>>> 
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVnzoJZLtuRrNNTJJket2AmB-2FrCgzkFXDN-2FN3JB6Gj5UqlkbnbjDkgNGbgS1u4yx-2BmdW4r4-2BwSXRpR66KwRMmSZ1S1onNjkrjECqJN8vOY2jGvfHKuC8krz-2Brv4ubyXCtM8s61ldWyZFz8s3eaWEdLZkQv0aVzHkXALueh9FdITYSgG2eZJQh7UfXvxynuZxeGRp-2B88f9xmo2kbWVUvoaOVR59TZxqOHS0BGEkaOPgbqJFNUOWtF3L-2BAUC2qmdGoCLUNlVVvBLmjFbIrbQslOH05KyfK4G2ocJysfe2irUN3S-2F6GPseqAwZmjz-2BiUddkyYGhOQKcEfKmluxciDEvtzA-2B7YXuzJIzsRt3JWSe4xx5hYMLaBQcYZ9rOsUdIo1pLX4xaIYFjwPkD0etvxiAV2wg-3D-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA2AT0STS9CATYpTbZwvUiOxe4HhhryLaM79oO5umb73mwlCPNowQL6x5LHoY2LrG-2BkreGKNqCFikZ3p-2BWuqJ1kUCsYEX3P3eNbMzt23R6nfziOwOMsDJNdKqNZHpu5bbd66nb6wffitRroKgBIY9EhzDw8vJK2JzYn34JDQI-2BtpnMIRRv0iaizdTvy3gdK4zcs-3D>! http ://sdm.link/slashdot_________________________________________ <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnyJg1K-2BNHC52rXAxvNekwKE0D5niM-2BCOtz5dXetdPEpVlNLJoz33Sj0X0yI3sOJzzSQKavXHQH8iQH9-2FfF7z8VNuW5hbx7IvAObr8O7XvOcS08xK3-2FmhTpZzNTADsky5-2BzsOIqng430EmXxrnnJiTb2Shzvt6EFnWs70N0ZjKUN72BuC1YLysWg-2B7esmgQnbpHoVkuEO-2FYgbMH6LnCQwOf8anUrn2n7WOEwZ8kWnm-2F68J83juuPO16AygzxbRVGFj8TIHgERkIX-2FYCQEz1nPsAXthnRx9yk1MF4X3vI00COIr8-2B30OHoky-2BLAXKggJlNeAhjflqh8mrqKVFTABm0G31Pl368-2FTfUw7qyG3Pyhr3a-2B4cr9x-2FUp6-2BUIWO0vYn5JBHAiIU5-2BfJMICNW-2FE244eiM4Zq5BHo-2FxNsidEawRKGywIudNc19hbKB-2BEmtrjBMua2fb6D4eyAD3owsmPcAf-2FySa7CpSma13p1CZa3QmpSRAbXkJICOFj5IU9ca41G9uFs7aOBm0sjiJOVZjIGMaZA-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA2AT0STS9CATYpTbZwvUiOxe4HhhryLaM79oO5umb73m00Sg1pMUCJpH84GvGOwVJbWwI4lAdePG-2FyVg1ErwhAki79uPFQ3ZbLd8707EswNRiVq8qpNzX7RhBTet1D-2FnRvV5Oyco-2FeBiaXwHuuUwwN-2BPjs-2Bs8t1qrrcowMACOxxtkyExIrKj1qWuY-2FISSnSJYw-3D>
>>>> moosefs-users mailing list
>>>> moo...@li... <mailto:moo...@li...>
>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbn0IXdEaT5qnUC7gIcS2XNdrgBiT-2B-2BuagrhyyC-2BAQOpyG-2BmG4gKLe4qAingLKnHCnlKCkpKKQ4sX3lGnNh63lOzI-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA2AT0STS9CATYpTbZwvUiOxe4HhhryLaM79oO5umb73mzQQ-2BlPQxZEfzRUTVf7Ts05ISaCZNnax8XrAGrQvJAf0wKbhckocX4LtMXVvopeJylF57T41h54TS1Bj8DvCmhFXPx61XLOFF1oxJiP9kv66GzFdXAcVt1pm1-2B-2B-2FEEDl6WU5XtTMoGN0JMmhVX1pbWQ-3D>
>>>  
>> 
> 
>  
> 

Re: [MooseFS-Users] loading moosefs on isilon node

[Piotr R. K. <pio...@mo...>]

MooseFS can be ran on FreeBSD :)
We support this OS and have native and up-to-date ports for FreeBSD (see e.g. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216974 <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=216974>)

-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | moosefs.com <https://moosefs.com/>
> On 24 Mar 2017, at 6:22 PM, Rupert Chandler <ru...@bt...> wrote:
> 
> The OS is freebsd on the isilon.
> 
> Rup
> 
> 
> ___
> Rupert Chandler 
> Tel: +44(0)1672600139
> ru...@bt...
> 
> 
> 
> 
> 
> 
> On Fri, Mar 24, 2017 at 5:05 PM +0000, "Piotr Robert Konopelko" <pio...@mo... <mailto:pio...@mo...>> wrote:
> 
> Hi Ali,
> 
> if there's a possibility to install Linux on Isilon nodes, it probably will be possible to run MooseFS on them.
> You can check out supported OS-es for MooseFS here: https://moosefs.com/download.html <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnzs2VURm3u13q5dMjLj1jhOSKpbUiyeyQkSEnyk-2F22Lb_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA276Isn05FbZBoOzU8L0moqUQlsHM6LqdImDbpAviSbTIxnunhVUYrD-2BjyKPAVUzC76VrMtEWy0aVeX0GqqDSFGwtjAA3EXv46h-2FVUUk8oh-2BkNe4mpLOBTyVBrx-2BFwW7ONs2RAOw5hVI0S-2B1uisLAy8aVAkJTxAzHNJjIo4d47a19vUswy3TjxHvByZlEeUftU-3D>.
> 
> 
> Best regards,
> Peter
> 
> -- 
> Piotr Robert Konopelko
> MooseFS Technical Support Engineer | moosefs.com <https://u2661533.ct.sendgrid.net/wf/click?upn=Icg0l1BNdpBnYuWGoApbnzOtmNIRfbm4mFSMCAdNP-2B4-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA276Isn05FbZBoOzU8L0moqUQlsHM6LqdImDbpAviSbTBf9hAjSK59FIPUO2c-2FVQDV5CnRf0gs0ubP5ytlyZMTlTgRdBieuKE-2FHdXsniad7266%20jkeZKw1TXDnXCu92NDu4CxBjfLe9MYUEaTcyvnhlja4PHXqYcENUjocSPunfmERqY60k4YYlzhGLSsVU4Ocs-3D>
>> On 22 Mar 2017, at 8:37 PM, Ali Moeinvaziri <moe...@gm... <mailto:moe...@gm...>> wrote:
>> 
>> Hi,
>> 
>> We have a few end of support isilon nodes (NL36) that would like to load moosefs on them
>> and use. I wonder if anybody has already gone down this road. 
>> 
>> I appreciate any notes, thoughts, or comments.
>> -AM
>> 
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org <https://u2661533.ct.sendgrid.net/wf/click?upn=FAiJY0aOU0-2BWnkh5vqxCuzDsKNuZ6E2RGQx93cRmfMk-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA276Isn05FbZBoOzU8L0moqUQlsHM6LqdImDbpAviSbTEuBjheKdBXC4p-2B1bMUEvSh3bPU6TlleTSHpm-2F5iKewd3V8xLlQKyt8QlF46nu9fUdn0iUxAZ0T-2BxsfjjmEvwEVmeqhtAPM5WX9LDyygxOr8qBSpLQGtIjDiPFvbdOMjd0F9mGyp0vihT0-2BdcrfHLh0-3D>! http ://sdm.link/slashdot_________________________________________ <https://u2661533.ct.sendgrid.net/wf/click?upn=aSVgumtVgpGQU3uYU8sWoVpneYg1QJbOcA9KD7cB3B2Df9iVPTN9nM02HJg5vK1SS-2FRtorQqEEUoCNh2jA0G8J49D54YClumZeuJDUwo9AA-3D_ZIL5ZH3kfv4h3RT93uiXeuN9lQ802wtLzdyEdpmnyA276Isn05FbZBoOzU8L0moqUQlsHM6LqdImDbpAviSbTDBMWJf3uXok3fmyryZj6-2BLCY7fK336Jgxb8Bs5GYli6pulRA-2BEmQmL4DN0ciTZGYPj-2B05APRIS9ZMwFW6efgPa-2BBjjugoXaFngI8v-2BRcKXHCSyomV1nj0RabA5O4fp56Ev9UnLclIhDYtgyVZP7Ljk-3D>
>> moosefs-users mailing list
>> moo...@li... <mailto:moo...@li...>
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> 
> 

Re: [MooseFS-Users] loading moosefs on isilon node

[Piotr R. K. <pio...@mo...>]

Hi Ali,

if there's a possibility to install Linux on Isilon nodes, it probably will be possible to run MooseFS on them.
You can check out supported OS-es for MooseFS here: https://moosefs.com/download.html <https://moosefs.com/download.html>.


Best regards,
Peter

-- 
Piotr Robert Konopelko
MooseFS Technical Support Engineer | moosefs.com <https://moosefs.com/>
> On 22 Mar 2017, at 8:37 PM, Ali Moeinvaziri <moe...@gm...> wrote:
> 
> Hi,
> 
> We have a few end of support isilon nodes (NL36) that would like to load moosefs on them
> and use. I wonder if anybody has already gone down this road. 
> 
> I appreciate any notes, thoughts, or comments.
> -AM
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot_________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

[MooseFS-Users] loading moosefs on isilon node

[Ali M. <moe...@gm...>]

Hi,

We have a few end of support isilon nodes (NL36) that would like to load
moosefs on them
and use. I wonder if anybody has already gone down this road.

I appreciate any notes, thoughts, or comments.
-AM

Re: [MooseFS-Users] MooseFS and encryption?

[Marin B. <li...@ol...>]

03 mars 2017 18:44 Warren Myers  a écrit:

>  What practices does the community suggest for encrypting files in MooseFS? 
> 
> 
> 
> Does Moose support the underlying filesystems being encrypted?
> 
> 
> 
> Is there a way to incorporate TLS or similar between chunkservers and masters?
> 
> 
> 
> Thanks,
> 
> 
> 
> Warren Myers
>  https://antipaucity.com
>  https://www.digitalocean.com/?refcode=d197a961987a
> 

Re: [MooseFS-Users] MooseFS and encryption?

[Marin B. <li...@ol...>]

Hi,  
  
 I would really love to see TLS or Kerberos support added to MooseFS, and md5   
 password hashes replaced by proper HMAC. The lack of such features makes   
 MooseFS vulnerable to several classes of exploits, mostly similar to   
 those affecting unencrypted NFS. As of now, using MooseFS on unsecure /   
 public networks is fairly dangerous.  
  
 Here is what we did to help securing the whole cluster:  
  
 - We emulate TLS with IPsec. We encrypt MooseFS traffic with IPsec between   
 all members of the cluster (including clients). Masters and chunk servers   
 are on the same subnet and use IPsec in transport mode, while clients use   
 tunnel mode through our gateways, even internally. The overhead is   
 minimal with recent CPUs and NICs.  
  
 - We use certificates to authenticate IPsec peers. This allows us to easily   
 revoke the access of a compromised system by revoking its certificate.  
  
 - We use secrets to authenticate masters and chunk servers. It's not secure,   
 but doesn't harm.  
  
 - Out of the box, MooseFS relies on DNS to locate the cluster master by   
 querying the 'mfsmaster' host name. This makes the cluster vulnerable to   
 DDoS and DNS spoofing. We use DNSSEC to authenticate DNS answers.  
  
 - We encrypt metadata (master / metaloggers) and data (chunk servers) with OS   
 level encryption. We use GELI on FreeBSD.  
  
 - Sensitive information may be further encrypted on client-side. We use PEFS   
 on FreeBSD, which is able to encrypt selected files or folders.  
  
 - Since user authentication happens on client-side, a smart user with   
 administrative access may use bogus UID/GID to bypass POSIX file system   
 permissions. Kerberos authentication would not let such a scenario   
 happen, but since it is not an option, we decided to consider file system   
 permissions as unsecure and not to rely on them. Instead, we handle   
 access control with exports: we use one export per shared directory and   
 ACL. Each export is protected by a md5 hash and we rely on the export   
 password to authenticate remote systems. Each export line includes a   
 'mapall' option which maps any remote UIDs/GIDs to an unprivilegied local   
 user/group whose permissions are limited to the shared location only.   
 This configuration protects the cluster from most exploits, except if the   
 export password is leaked.  
  
 Hope that helps,  
  
 Marin.  
  


  

Re: [MooseFS-Users] Using virtual drives

[Casper L. <cas...@pr...>]

Hi WU,

I run something similar, I have two different versions of mfs running
simultaneously, because I am transitioning from a platter-disk mfs2 cluster
to an ssd mfs3 pro cluster.

I'm not experiencing any problems except that with normal Debian packages
there is no way to simultaneously use two client versions.

Greetings, Casper

Op zo 5 mrt. 2017 23:39 schreef web user <web...@gm...>:

> I'm trying to create a second mfs server and chunks on my network
> consisting of only ssd drives. To do this I wanted to use available space
> on existing ssd drives on some of my compute servers.
>
> I see that the option of using a virtual drive is not mentioned in the
> existing docs for (2.x and 3.x). Here are the steps that I found from an
> old doc:
>
> #mkdir -p /storage/mfschunks
> #dd if=/dev/zero of=/storage/mfschunks/mfschunks1 bs=1024 count=1
> seek=$((2*1024*1024-1))
> #mkfs -t ext3 /storage/mfschunks/mfschunks1
> #mkdir -p /mnt/mfschunks1
> #mount -t ext3 -o loop /storage/mfschunks/mfschunks1 /mnt/mfschunks1
> #dd if=/dev/zero of=/storage/mfschunks/mfschunks2 bs=1024 count=1
> seek=$((2*1024*1024-1))
> #mkfs -t ext3 /storage/mfschunks/mfschunks2
> #mkdir -p /mnt/mfschunks2
> #mount -t ext3 -o loop /storage/mfschunks/mfschunks2 /mnt/mfsc
>
> I have the following questions:
>
> 1. The data for this partition is scratch and I'm not too worried if I
> loose all of it. Are the instructions still valid? I'm using ubuntu
> 14.04LTS. Any downsides that I should be aware off
> 2. Is there any harm of having two mfs servers running on two different
> machines? Will I able to mound two different mfs drives on one client
> machine.
>
> mfsmount /mnt/mfs -H mfsserever
> mfsmount /mnt/mfsssd -H mfssereverssd
>
> Thanks in advance,
>
> WU
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

[MooseFS-Users] Using virtual drives

[web u. <web...@gm...>]

I'm trying to create a second mfs server and chunks on my network
consisting of only ssd drives. To do this I wanted to use available space
on existing ssd drives on some of my compute servers.

I see that the option of using a virtual drive is not mentioned in the
existing docs for (2.x and 3.x). Here are the steps that I found from an
old doc:

#mkdir -p /storage/mfschunks
#dd if=/dev/zero of=/storage/mfschunks/mfschunks1 bs=1024 count=1
seek=$((2*1024*1024-1))
#mkfs -t ext3 /storage/mfschunks/mfschunks1
#mkdir -p /mnt/mfschunks1
#mount -t ext3 -o loop /storage/mfschunks/mfschunks1 /mnt/mfschunks1
#dd if=/dev/zero of=/storage/mfschunks/mfschunks2 bs=1024 count=1
seek=$((2*1024*1024-1))
#mkfs -t ext3 /storage/mfschunks/mfschunks2
#mkdir -p /mnt/mfschunks2
#mount -t ext3 -o loop /storage/mfschunks/mfschunks2 /mnt/mfsc

I have the following questions:

1. The data for this partition is scratch and I'm not too worried if I
loose all of it. Are the instructions still valid? I'm using ubuntu
14.04LTS. Any downsides that I should be aware off
2. Is there any harm of having two mfs servers running on two different
machines? Will I able to mound two different mfs drives on one client
machine.

mfsmount /mnt/mfs -H mfsserever
mfsmount /mnt/mfsssd -H mfssereverssd

Thanks in advance,

WU

[MooseFS-Users] MooseFS and encryption?

[Warren M. <wa...@an...>]

What practices does the community suggest for encrypting files in MooseFS?


Does Moose support the underlying filesystems being encrypted?


Is there a way to incorporate TLS or similar between chunkservers and masters?


Thanks,


Warren Myers
https://antipaucity.com
https://www.digitalocean.com/?refcode=d197a961987a

Re: [MooseFS-Users] MFS 2.0.91 Master is crashing

[Michael T. <mic...@ho...>]

First of all, my apologies if this reply is messing up the threading as I'm replying to a digest feed instead of the actual thread.

As far as trash is concerned, based on the GUI:

Trash files: 4,996
Trash size: 26GiB
Chunks with Goal=0: 1,329

There is a 19GB file named core in /var/lib/mfs:

-rw------- 1 mfs mfs  19G Mar  2 11:01 core

Is this the core dump file?  If so, I'll upload it somewhere.  Please note though that the date stamp is 1 day later, if it matters.

As far as syslog goes, I did not find any logged crash/abort entry around the time of the crash. Just a lot of messages like this: "Mar  1 10:01:19 HO-MFSMaster01 mfsmaster[1035]: chunkserver has nonexistent chunk (0000000001AC5130_00000001), so create it for future deletion"


--- mike t.


________________________________
Date: Thu, 2 Mar 2017 10:31:38 +0100
From: Aleksander Wieliczko <ale...@mo...>
Subject: Re: [MooseFS-Users] MFS 2.0.91 Master is crashing
To: moo...@li...
Message-ID: <b1e...@mo...>
Content-Type: text/plain; charset="windows-1252"

Hi.
First of all we would like to know how many objects you have in trash?
Secondly. Do you have core dump file?
System log is always good option to see, so please send it on list or
directly to 'support at moosefs.com'

I'm looking forward to hearing from you.
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>


On 02.03.2017 03:59, Michael Tinsay wrote:
>
> Hi.
>
>
> MFSmaster crashed a couple of times when I did the following:
>
>
> 1. log in to mfsmaster machine. Then su into root.
>
> 2. execute: mfsmount -m -p /mnt/mfsmeta
>
> 3. execute: cd /mnt/mfsmeta/trash
>
> 4. execute: ls -hl  <== in less than a minute mfsmaster will crash.
>
>
> If I don't do step 4, mfsmaster runs just fine.
>
>
> The mfsmaster is installed a dedicated, bare metal machine for
> mfsmaster, not a VM.  It is running Ubuntu 14.04 with the latest
> updates applied.  The machine has 32GB of ram, and mfsmaster is using
> 20GB.
>
>
> Please advise what other info is needed. Or log files. Or whatever.
>
>
> Regards.
>
>
>
> --- mike t.
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
Slashdot: News for nerds, stuff that matters<http://sdm.link/slashdot>
sdm.link
Slashdot: News for nerds, stuff that matters. Timely news source for technology related news with a heavy slant towards Linux and Open Source issues.



>
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
moosefs-users Info Page - SourceForge<https://lists.sourceforge.net/lists/listinfo/moosefs-users>
lists.sourceforge.net
To see the collection of prior postings to the list, visit the moosefs-users Archives. Using moosefs-users: To post a message to all the list members ...




-------------- next part --------------
An HTML attachment was scrubbed...

------------------------------

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

------------------------------

_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users
moosefs-users Info Page - SourceForge<https://lists.sourceforge.net/lists/listinfo/moosefs-users>
lists.sourceforge.net
To see the collection of prior postings to the list, visit the moosefs-users Archives. Using moosefs-users: To post a message to all the list members ...





End of moosefs-users Digest, Vol 87, Issue 1
********************************************

Re: [MooseFS-Users] MFS 2.0.91 Master is crashing

[Aleksander W. <ale...@mo...>]

Hi.
First of all we would like to know how many objects you have in trash?
Secondly. Do you have core dump file?
System log is always good option to see, so please send it on list or
directly to 'support at moosefs.com'

I'm looking forward to hearing from you.
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

On 02.03.2017 03:59, Michael Tinsay wrote:
>
> Hi.
>
>
> MFSmaster crashed a couple of times when I did the following:
>
>
> 1. log in to mfsmaster machine. Then su into root.
>
> 2. execute: mfsmount -m -p /mnt/mfsmeta
>
> 3. execute: cd /mnt/mfsmeta/trash
>
> 4. execute: ls -hl  <== in less than a minute mfsmaster will crash.
>
>
> If I don't do step 4, mfsmaster runs just fine.
>
>
> The mfsmaster is installed a dedicated, bare metal machine for
> mfsmaster, not a VM.  It is running Ubuntu 14.04 with the latest
> updates applied.  The machine has 32GB of ram, and mfsmaster is using
> 20GB.
>
>
> Please advise what other info is needed. Or log files. Or whatever.
>
>
> Regards.
>
>
>
> --- mike t.
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

[MooseFS-Users] MFS 2.0.91 Master is crashing

[Michael T. <mic...@ho...>]

Hi.


MFSmaster crashed a couple of times when I did the following:


1. log in to mfsmaster machine. Then su into root.

2. execute: mfsmount -m -p /mnt/mfsmeta

3. execute: cd /mnt/mfsmeta/trash

4. execute: ls -hl  <== in less than a minute mfsmaster will crash.


If I don't do step 4, mfsmaster runs just fine.


The mfsmaster is installed a dedicated, bare metal machine for mfsmaster, not a VM.  It is running Ubuntu 14.04 with the latest updates applied.  The machine has 32GB of ram, and mfsmaster is using 20GB.


Please advise what other info is needed. Or log files. Or whatever.


Regards.



--- mike t.

Re: [MooseFS-Users] Upgrade Cluster

[Aleksander W. <ale...@mo...>]

Hi.

There will be no problems.
If you like you can stop whole cluster and than execute update.
You can find instruction how to stop MooseFS cluster in our
documentation, but basically this are the steps that you have to do to
stop whole cluster.

1. Umount all clients
2. Stop master
3. Stop metaloggers
4. Stop all chunkservers

Best regards
Aleksander Wieliczko
Technical Support Engineer
MooseFS.com <moosefs.com>

On 02/16/2017 11:41 PM, Wolfgang wrote:
>
> Hi!
>
> I'm running Moosesfs 3.0.86 on Ubuntu 14.04 master and chunkservers,
> metaloggers and would like to upgrade to
> 3.0.88
>
> In the documentation I found following order to upgrade:
>
> 1. master
> 2. cigserv
> 3. metaloggers (one-by-one)
> 4. chunkservers (one-by-one)
> 5. mfs-clients (one-by-one)
>
> In my case I can allow my cluster a downtime
>
> So I could stop all mfs processes (master,chunkserver,metaloggers) and
> do an
> aptitude update
> aptitude safe-upgrade
>
> on each machine.
> Is this procedure recommended or will there be any problem with this ?
>
> Thank you & greetings
> Wolfgang
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>
>
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users