moosefs-users — Mailing list for MooseFS Users

Re: [Moosefs-users] Few Questions?

[Thomas S H. <tha...@gm...>]

Don't know how to search the list, there is a google trick to do it that I
can never remember, where you specify the site you want to search.

Yes, we had a fair amount of user error at first as well, use mfsfileinfo to
find the files with missing chunks, and then remove them, keep in mind that
when you remove them they just go to the trash, so the 0 files may last a
while unless you manually clean out the trash.

Thats odd on the meta mount, I assume you are root, can you post your
mfsexports.cfg and the command you are using to mount the mfsmeta?

Also, please post the OS you are using and the version of MooseFS, while it
does not really apply here it usually does :)

-Thomas S Hatch

On Fri, Apr 15, 2011 at 2:00 PM, Scoleri, Steven
<Sco...@gs...>wrote:

> 1.        Does anyone know a way to search this list?  There doesn’t seem
> to be a search engine on -
> http://sourceforge.net/mailarchive/forum.php?forum_name=moosefs-users
>
> 2.       I lost a couple of chunkservers (human error).  Anyway we lost a
> some chunks to the zero column.  How do I clean that up now?  I’ve deleted
> the files that were in the 0 column.
>
> 3.       When I try to mount the mfsmeta I get:  “mfsmaster register
> error: Permission denied”
>
>
>
> Thanks,
>
> -Scoleri
>
>
> ------------------------------------------------------------------------------
> Benefiting from Server Virtualization: Beyond Initial Workload
> Consolidation -- Increasing the use of server virtualization is a top
> priority.Virtualization can reduce costs, simplify management, and improve
> application availability and disaster protection. Learn more about boosting
> the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>

[Moosefs-users] Few Questions?

[Scoleri, S. <Sco...@gs...>]

1.        Does anyone know a way to search this list?  There doesn't
seem to be a search engine on -
http://sourceforge.net/mailarchive/forum.php?forum_name=moosefs-users

2.       I lost a couple of chunkservers (human error).  Anyway we lost
a some chunks to the zero column.  How do I clean that up now?  I've
deleted the files that were in the 0 column.

3.       When I try to mount the mfsmeta I get:  "mfsmaster register
error: Permission denied"

 

Thanks,

-Scoleri

[Moosefs-users] chunkserver over several offices

[eric m. <eri...@gm...>]

Hi,

 How to ensure a least a copy of each chunk on servers dispatch on several
offices. The solution is  easy if we have one chunkserver on each office.
How can I do this when some office have, for example, 5 chunkservers and
other only one?

thank for you help


-- 
Eric Mourgaya,


Respectons la planète!
Luttons contre la médiocrité!

[Moosefs-users] RE-2: Is there a limit set for network transfer of 1Gbit set in moosefs-fuse?

[Razvan D. <ra...@re...>]

 Hello again Michal!

    I agree with you that i would not get 600MB/s when i will have 10 
 reads and 10 writes
 of video files but will be close to this value as a total because in my 
 case we only deal with large video files
 2 to 10 GB in size and i can tell from experience that this storage 
 used directly via ftp deliver to the users
 around 40-5000MB/s on each controller with around 5 reads and 5 writes 
 each of ~45MB/s.
    To me from what i could see in all the tests i've done it looks like 
 somewhere the filesystem it's not reaching
 it's true transfer potential. I say this because when i read with 
 123MB/s, disks are used 15% each,
 CPU power is 70% free, RAM is also plenty but mfschunkserver processes 
 are is sleep state(S) at 40% CPU time.
 So what i can understand from this is that chunkservers instead of 
 delivering data as fast as is possible,
 they just sit and take a nap ( :D ) for 60% of the CPU time.
 If there is some filesystem IO moderation in action, that is part of 
 how the system works then in understandable
  but if is just sleeping then we can say that MooseFS is also a "Green" 
 filesystem (ECO-friendly) :P since consumes less CPU power.
    I will also try to look in the code in the following days and see if 
 i can answer myself to these questions and also to get
 a better understanding on how the FS works.
    Well hope i haven't been too obnoxious with my jokes, i'm looking 
 forward for you reply :).


 Kind regards,

 Razvan Dumitrescu


 PS: In order to justify my conclusions and answer to the questions you 
 asked, i sumbit also the following test logs:

 ====================================== start test logs 
 =================================

 newsdesk-storage-02 / # uname -a
 Linux newsdesk-storage-02 2.6.38-gentoo-r1 #1 SMP Tue Apr 5 18:28:19 
 EEST 2011 x86_64 Intel(R) Xeon(R) CPU X5355 @ 2.66GHz GenuineIntel 
 GNU/Linux

 RAID 6 128k stripe size, total 12 disks, 10 data disks, 2 parity disks
 Each volume has been set with XFS and coresponding RAID array stripe 
 aligning

 mkfs.xfs -f -d su=128k,sw=10 -l version=2,su=128k /dev/sdc
 mkfs.xfs -f -d su=128k,sw=10 -l version=2,su=128k /dev/sdd


 READ & WRITE TEST DIRECTLY TO DISKS
 ===================================
 (reads were done simultaneus on both controllers /mnt/osd0 and 
 /mnt/osd1)
 (writes were done also simultaneous on both controllers as you will be
 able to see below in atop screen)

 controller 1

 write:
 newsdesk-storage-02 / # dd if=/dev/zero of=/mnt/osd0/test.file bs=4k 
 count=2048k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 14.3845 s, 597 MB/s
 newsdesk-storage-02 / # dd if=/dev/zero of=/mnt/osd0/test1.file bs=4k 
 count=2048k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 14.2794 s, 602 MB/s
 newsdesk-storage-02 / # dd if=/dev/zero of=/mnt/osd0/test2.file bs=4k 
 count=2048k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 14.9359 s, 575 MB/s

 read:
 newsdesk-storage-02 / # dd if=/mnt/osd0/test.file of=/dev/null bs=4k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 11.7127 s, 733 MB/s
 newsdesk-storage-02 / # dd if=/mnt/osd0/test1.file of=/dev/null bs=4k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 12.3034 s, 698 MB/s
 newsdesk-storage-02 / # dd if=/mnt/osd0/test2.file of=/dev/null bs=4k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 13.2949 s, 646 MB/s

 controller 2

 write:
 newsdesk-storage-02 / # dd if=/dev/zero of=/mnt/osd1/test.file bs=4k 
 count=2048k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 14.8654 s, 578 MB/s
 newsdesk-storage-02 / # dd if=/dev/zero of=/mnt/osd1/test1.file bs=4k 
 count=2048k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 14.9257 s, 576 MB/s
 newsdesk-storage-02 / # dd if=/dev/zero of=/mnt/osd1/test2.file bs=4k 
 count=2048k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 14.6867 s, 585 MB/s

 read:
 newsdesk-storage-02 / # dd if=/mnt/osd1/test.file of=/dev/null bs=4k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 11.5464 s, 744 MB/s
 newsdesk-storage-02 / # dd if=/mnt/osd1/test1.file of=/dev/null bs=4k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 11.6056 s, 740 MB/s
 newsdesk-storage-02 / # dd if=/mnt/osd1/test2.file of=/dev/null bs=4k
 2097152+0 records in
 2097152+0 records out
 8589934592 bytes (8.6 GB) copied, 12.2195 s, 703 MB/s

 ATOP - newsdesk-storage-0 2011/04/15  16:40:41                5 seconds 
 elapsed
 PRC | sys   7.70s | user   0.22s | #proc    136 | #zombie    0 | #exit  
    0 |
 CPU | sys    147% | user      4% | irq      12% | idle    573% | wait   
  64% |
 cpu | sys     55% | user      3% | irq       5% | idle     11% | cpu000 
 w 27% |
 cpu | sys     51% | user      1% | irq       6% | idle     12% | cpu002 
 w 29% |
 cpu | sys     22% | user      0% | irq       0% | idle     78% | cpu001 
 w  0% |
 cpu | sys      7% | user      0% | irq       1% | idle     88% | cpu006 
 w  4% |
 cpu | sys      6% | user      0% | irq       0% | idle     91% | cpu004 
 w  2% |
 cpu | sys      5% | user      0% | irq       0% | idle     95% | cpu005 
 w  0% |
 CPL | avg1   1.05 | avg5    0.54 | avg15   0.31 | csw    10777 | intr   
 28418 |
 MEM | tot    3.9G | free   34.6M | cache   3.6G | buff    0.0M | slab   
 38.3M |
 SWP | tot  988.4M | free  979.4M |              | vmcom 513.7M | vmlim  
 2.9G |
 PAG | scan 1769e3 | stall      0 |              | swin       0 | swout  
    0 |
 DSK |         sdc | busy    101% | read    6635 | write      0 | avio   
 0 ms |
 DSK |         sdd | busy    101% | read    7144 | write      0 | avio   
 0 ms |
 NET | transport   | tcpi      29 | tcpo      29 | udpi       0 | udpo   
    0 |
 NET | network     | ipi       46 | ipo       29 | ipfrw      0 | deliv  
   37 |
 NET | eth1     0% | pcki      54 | pcko       7 | si    6 Kbps | so    
 4 Kbps |
 NET | eth0     0% | pcki      34 | pcko      14 | si    4 Kbps | so    
 1 Kbps |
 NET | bond0  ---- | pcki      88 | pcko      21 | si   11 Kbps | so    
 6 Kbps |
 NET | lo     ---- | pcki       8 | pcko       8 | si    0 Kbps | so    
 0 Kbps |

   PID  SYSCPU  USRCPU  VGROW  RGROW USERNAME THR  ST EXC  S  CPU CMD    
 1/1
 17681   3.19s   0.13s     0K     0K root       1  --   -  R  67% dd
 17680   3.07s   0.08s     0K     0K root       1  --   -  R  64% dd
   437   1.39s   0.00s     0K     0K root       1  --   -  R  28% 
 kswapd0
 17450   0.00s   0.01s     0K     0K mfs        1  --   -  S   0% 
 mfsmaster
 17677   0.01s   0.00s     0K     0K root       1  --   -  R   0% atop
     9   0.01s   0.00s     0K     0K root       1  --   -  S   0% 
 ksoftirqd/1
    12   0.01s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/2:0
    24   0.01s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/6:0
 16809   0.01s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/0:2
  4574   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/u:2
 13791   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 xfsbufd/sdc
 17648   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 flush-8:48

 As atop shows this is the maximum transfer RAID provides sdc/sdd being 
 busy 100%
 which is quite ok since that means around 70MB/s read and 60MB/s write 
 for each
 of the 10 data disks in each array

 so far so good and pleased how things work,

 when i try to copy from one storage to another i get the following 
 results
 which again i can understand since the limitation is the CPU.


 copy from storage to storage using cp

 newsdesk-storage-02 / # time cp /mnt/osd0/test2.file 
 /mnt/osd1/test6.file

 real	0m19.376s
 user	0m0.050s
 sys	0m15.280s
 newsdesk-storage-02 / # time cp /mnt/osd1/test2.file 
 /mnt/osd0/test6.file

 real	0m15.691s
 user	0m0.030s
 sys	0m15.560s
 newsdesk-storage-02 / # time cp /mnt/osd0/test4.file 
 /mnt/osd1/test7.file

 real	0m16.652s
 user	0m0.070s
 sys	0m16.390s
 newsdesk-storage-02 / # time cp /mnt/osd1/test4.file 
 /mnt/osd0/test7.file

 real	0m16.551s
 user	0m0.030s
 sys	0m15.760s


 8589934592 bytes (8.6 GB) copied in 19.376s = 422.79 MB/s
 8589934592 bytes (8.6 GB) copied in 15.691s = 522.08 MB/s
 8589934592 bytes (8.6 GB) copied in 16.652s = 491.95 MB/s
 8589934592 bytes (8.6 GB) copied in 16.551s = 494.95 MB/s

 The transfer reaches only this values cause cp caps in CPU while the 
 disks are used half
 of available speed as you can see below

 ATOP - newsdesk-storage-0 2011/04/15  17:04:36                5 seconds 
 elapsed
 PRC | sys   7.39s | user   0.01s | #proc    133 | #zombie    0 | #exit  
    0 |
 CPU | sys    141% | user      0% | irq       5% | idle    655% | wait   
   0% |
 cpu | sys     96% | user      0% | irq       4% | idle      0% | cpu001 
 w  0% |
 cpu | sys     27% | user      0% | irq       0% | idle     73% | cpu006 
 w  0% |
 cpu | sys     18% | user      0% | irq       0% | idle     82% | cpu002 
 w  0% |
 cpu | sys      0% | user      0% | irq       0% | idle    100% | cpu000 
 w  0% |
 cpu | sys      0% | user      0% | irq       0% | idle    100% | cpu005 
 w  0% |
 cpu | sys      0% | user      0% | irq       0% | idle    100% | cpu007 
 w  0% |
 CPL | avg1   0.11 | avg5    0.27 | avg15   0.26 | csw     6104 | intr   
 23475 |
 MEM | tot    3.9G | free   31.6M | cache   3.5G | buff    0.0M | slab  
 160.8M |
 SWP | tot  988.4M | free  978.4M |              | vmcom 513.4M | vmlim  
 2.9G |
 PAG | scan 1339e3 | stall      0 |              | swin       0 | swout  
    0 |
 DSK |         sdc | busy     56% | read       0 | write   5905 | avio   
 0 ms |
 DSK |         sdd | busy     40% | read    5216 | write      1 | avio   
 0 ms |
 DSK |         sda | busy      3% | read       0 | write      3 | avio   
 43 ms |
 DSK |         sdb | busy      1% | read       0 | write      3 | avio   
 10 ms |
 NET | transport   | tcpi      42 | tcpo      41 | udpi       0 | udpo   
    0 |
 NET | network     | ipi       85 | ipo       41 | ipfrw      0 | deliv  
   56 |
 NET | eth0     0% | pcki      70 | pcko      22 | si   10 Kbps | so    
 2 Kbps |
 NET | eth1     0% | pcki      61 | pcko       9 | si    7 Kbps | so    
 3 Kbps |
 NET | bond0  ---- | pcki     131 | pcko      31 | si   18 Kbps | so    
 6 Kbps |
 NET | lo     ---- | pcki       8 | pcko       8 | si    0 Kbps | so    
 0 Kbps |

   PID  SYSCPU  USRCPU  VGROW  RGROW USERNAME THR  ST EXC  S  CPU CMD    
 1/1
 17745   4.97s   0.01s     0K   -12K root       1  --   -  R 100% cp
   437   1.23s   0.00s     0K     0K root       1  --   -  S  25% 
 kswapd0
 17700   0.76s   0.00s     0K     0K root       1  --   -  S  15% 
 flush-8:32
 17650   0.19s   0.00s     0K     0K root       1  --   -  S   4% 
 kworker/2:2
 17702   0.13s   0.00s     0K     0K root       1  --   -  S   3% 
 kworker/2:1
    24   0.05s   0.00s     0K     0K root       1  --   -  S   1% 
 kworker/6:0
 17742   0.02s   0.00s     0K     0K root       1  --   -  R   0% atop
 17450   0.01s   0.00s     0K     0K mfs        1  --   -  S   0% 
 mfsmaster
 17456   0.01s   0.00s     0K     0K mfs       24  --   -  S   0% 
 mfschunkserver
   581   0.01s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/1:2
 17645   0.01s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/1:0
 17234   0.00s   0.00s     0K    -4K root       1  --   -  S   0% bash
 17483   0.00s   0.00s     0K     0K mfs       24  --   -  S   0% 
 mfschunkserver
   121   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 sync_supers
 13251   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 md1_raid1




 While doing dd read symultaneous with 2 clients using MooseFS i get 
 these results:
 - disks not used
 - CPU not used
 - memory again plenty

 newsdesk-storage-01 / # dd if=/mnt/mfs/test2.file of=/dev/null bs=16k
 524288+0 records in
 524288+0 records out
 8589934592 bytes (8.6 GB) copied, 201.276 s, 42.7 MB/s
 newsdesk-storage-01 / # dd if=/mnt/mfs/test2.file of=/dev/null bs=16k
 524288+0 records in
 524288+0 records out
 8589934592 bytes (8.6 GB) copied, 183.349 s, 46.9 MB/s

 fluxuri / # dd if=/mnt/mfs/test1.file of=/dev/null bs=16k
 524288+0 records in
 524288+0 records out
 8589934592 bytes (8.6 GB) copied, 144.376 s, 59.5 MB/s
 fluxuri / # dd if=/mnt/mfs/test1.file of=/dev/null bs=16k
 524288+0 records in
 524288+0 records out
 8589934592 bytes (8.6 GB) copied, 144.132 s, 59.6 MB/s

 ATOP - newsdesk-storage-0 2011/04/15  16:21:24                5 seconds 
 elapsed
 PRC | sys   1.72s | user   2.27s | #proc    132 | #zombie    0 | #exit  
    0 |
 CPU | sys     18% | user     23% | irq       6% | idle    750% | wait   
   4% |
 cpu | sys      3% | user      5% | irq       1% | idle     91% | cpu005 
 w  0% |
 cpu | sys      3% | user      3% | irq       1% | idle     94% | cpu001 
 w  0% |
 cpu | sys      2% | user      2% | irq       1% | idle     94% | cpu007 
 w  1% |
 cpu | sys      1% | user      3% | irq       1% | idle     94% | cpu003 
 w  2% |
 cpu | sys      3% | user      2% | irq       0% | idle     94% | cpu000 
 w  1% |
 cpu | sys      3% | user      3% | irq       1% | idle     93% | cpu002 
 w  0% |
 cpu | sys      2% | user      2% | irq       0% | idle     96% | cpu004 
 w  0% |
 cpu | sys      1% | user      3% | irq       1% | idle     94% | cpu006 
 w  0% |
 CPL | avg1   0.08 | avg5    0.11 | avg15   0.07 | csw    61782 | intr   
 71792 |
 MEM | tot    3.9G | free   32.1M | cache   3.6G | buff    0.0M | slab   
 37.8M |
 SWP | tot  988.4M | free  979.6M |              | vmcom 530.8M | vmlim  
 2.9G |
 PAG | scan 121157 | stall      0 |              | swin       0 | swout  
    0 |
 DSK |         sdd | busy     11% | read     492 | write      0 | avio   
 1 ms |
 DSK |         sdc | busy      5% | read     671 | write      0 | avio   
 0 ms |
 DSK |         sda | busy      4% | read       2 | write      3 | avio   
 38 ms |
 DSK |         sdb | busy      1% | read       1 | write      3 | avio   
 12 ms |
 NET | transport   | tcpi  137827 | tcpo  356731 | udpi       0 | udpo   
    0 |
 NET | network     | ipi   137862 | ipo    27661 | ipfrw      0 | deliv 
 137832 |
 NET | eth0    46% | pcki   70537 | pcko  193565 | si 7525 Kbps | so  
 463 Mbps |
 NET | eth1    39% | pcki   67369 | pcko  163226 | si 7221 Kbps | so  
 392 Mbps |
 NET | bond0  ---- | pcki  137906 | pcko  356791 | si   14 Mbps | so  
 856 Mbps |
 NET | lo     ---- | pcki       8 | pcko       8 | si    0 Kbps | so    
 0 Kbps |

   PID  SYSCPU  USRCPU  VGROW  RGROW USERNAME THR  ST EXC  S  CPU CMD    
 1/1
 17456   0.79s   1.22s   144K   -28K mfs       24  --   -  S  42% 
 mfschunkserver
 17483   0.82s   1.04s   -44K  -192K mfs       24  --   -  S  39% 
 mfschunkserver
   437   0.08s   0.00s     0K     0K root       1  --   -  S   2% 
 kswapd0
 17613   0.02s   0.00s  8676K   516K root       1  --   -  R   0% atop
 17450   0.01s   0.00s     0K     0K mfs        1  --   -  S   0% 
 mfsmaster
 16376   0.00s   0.01s     0K     0K root       1  --   -  S   0% 
 apache2
    12   0.00s   0.00s     0K     0K root       1  --   -  R   0% 
 kworker/2:0
 13251   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 md1_raid1
 13277   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 xfsbufd/md1

 Bonding works fine on newsdesk-storage-02 when i use iperf from same 2 
 clients
 i get around 1.9Gbit/s

 ATOP - newsdesk-storage-0 2011/04/15  17:29:05                5 seconds 
 elapsed
 PRC | sys   0.96s | user   0.04s | #proc    132 | #zombie    0 | #exit  
    0 |
 CPU | sys     10% | user      1% | irq       2% | idle    788% | wait   
   0% |
 cpu | sys      2% | user      0% | irq       1% | idle     97% | cpu005 
 w  0% |
 cpu | sys      1% | user      0% | irq       0% | idle     98% | cpu003 
 w  0% |
 cpu | sys      2% | user      0% | irq       0% | idle     98% | cpu000 
 w  0% |
 cpu | sys      1% | user      0% | irq       0% | idle     98% | cpu001 
 w  0% |
 cpu | sys      1% | user      0% | irq       0% | idle     98% | cpu006 
 w  0% |
 cpu | sys      1% | user      0% | irq       0% | idle     99% | cpu004 
 w  0% |
 cpu | sys      1% | user      0% | irq       0% | idle     99% | cpu002 
 w  0% |
 cpu | sys      0% | user      0% | irq       0% | idle    100% | cpu007 
 w  0% |
 CPL | avg1   0.06 | avg5    0.23 | avg15   0.28 | csw    18503 | intr   
 45720 |
 MEM | tot    3.9G | free   40.7M | cache   3.5G | buff    0.0M | slab  
 158.8M |
 SWP | tot  988.4M | free  978.4M |              | vmcom 545.8M | vmlim  
 2.9G |
 DSK |         sda | busy      3% | read       0 | write      4 | avio   
 37 ms |
 DSK |         sdb | busy      0% | read       0 | write      4 | avio   
 2 ms |
 DSK |         sdc | busy      0% | read       0 | write      1 | avio   
 0 ms |
 DSK |         sdd | busy      0% | read       0 | write      1 | avio   
 0 ms |
 NET | transport   | tcpi  782002 | tcpo  372077 | udpi       0 | udpo   
    0 |
 NET | network     | ipi   782031 | ipo   372084 | ipfrw      0 | deliv 
 782021 |
 NET | eth0    98% | pcki  406520 | pcko  372072 | si  984 Mbps | so   
 39 Mbps |
 NET | eth1    90% | pcki  375564 | pcko       7 | si  909 Mbps | so    
 2 Kbps |
 NET | bond0  ---- | pcki  782084 | pcko  372079 | si 1893 Mbps | so   
 39 Mbps |
 NET | lo     ---- | pcki       8 | pcko       8 | si    0 Kbps | so    
 0 Kbps |

   PID  SYSCPU  USRCPU  VGROW  RGROW USERNAME THR  ST EXC  S  CPU CMD    
 1/1
 17785   0.92s   0.03s     0K     0K root       5  --   -  S  19% iperf
 17450   0.01s   0.00s     0K     0K mfs        1  --   -  S   0% 
 mfsmaster
 17789   0.01s   0.00s     0K     0K root       1  --   -  R   0% atop
 17456   0.01s   0.00s     0K     0K mfs       24  --   -  S   0% 
 mfschunkserver
 17483   0.00s   0.01s     0K     0K mfs       24  --   -  S   0% 
 mfschunkserver
 16809   0.01s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/0:2
  4574   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/u:2
 13251   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 md1_raid1


 ======================================= end test logs 
 ==================================


 On Fri, 15 Apr 2011 14:51:03 +0200, Michal Borychowski wrote:
> Hi Razvan!
>
> Thanks for nice words about MooseFS! :)
>
> There is absolutely no built-in limit for transfers in MooseFS. And 
> 130 MB/s
> for SATA2 disks is a very good result - compare it to these 
> benchmarks:
> http://hothardware.com/printarticle.aspx?articleid=881. I would never 
> expect
> 600MB/s on SATA2 disks. And what are results of 'dd' for the disks 
> alone?
> But remember that such tests do not reflect real life environments 
> where
> lots of users use the system in the same time.
>
>
> Kind regards
> Michał Borychowski
> MooseFS Support Manager
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> Gemius S.A.
> ul. Wołoska 7, 02-672 Warszawa
> Budynek MARS, klatka D
> Tel.: +4822 874-41-00
> Fax : +4822 874-41-01
>
>
>
>
> -----Original Message-----
> From: Razvan Dumitrescu [mailto:ra...@re...]
> Sent: Thursday, April 14, 2011 7:11 PM
> To: moo...@li...
> Subject: [Moosefs-users] Is there a limit set for network transfer of 
> 1Gbit
> set in moosefs-fuse?
>
>  Hello guys!
>
>    First of all i have to say you made a great job with MooseFS, from
>  all the open source distributed filesystems i loooked over this is 
> by
>  far the best
>  in features, documentation and simple efficiency (installation,
>  configuration, tools). Great piece of work!
>    Now in order to be in pure extasy while looking on how good my
>  storage performs i need to get an answer :).
>    I have set up a MooseFS on a server box, 2 x Xeon X5355, 4GB RAM,
>  Intel 5000P board, 2 x Areca 1321 controllers, 24 x ST3750640NS 
> disks.
>    What puzzles me is that when i mount the MoooseFS on same machine 
> and
>  i try to make different transfers the bottleneck the i hit is always
>  around 1Gbit. My 2 storages sdc and sdd are both used around 15-20%
>  while reaching transfer from moosefs of ~130MB/s (both controllers
>  deliver around 600MB/s read and write).
>  For this local test the loopback device is used (mounted on same
>  machine) and from what i can see using atop
>  the only section that seems to cap is the network at loopback device
>  with a value slighltly under 1Gbit.
>  Using iperf on same machine i get 20Gb/s so it seems loopback device
>  can't be blamed for the limitation.
>
>  Mounts:
>
>  storage-02 / # df -h
>  Filesystem            Size  Used Avail Use% Mounted on
>  /dev/md1              149G   15G  135G  10% /
>  udev                   10M  192K  9.9M   2% /dev
>  shm                   2.0G     0  2.0G   0% /dev/shm
>  /dev/sdc              6.9T   31G  6.8T   1% /mnt/osd0
>  /dev/sdd              6.9T   31G  6.8T   1% /mnt/osd1
>  192.168.8.88:9421      14T   62G   14T   1% /mnt/mfs
>
>  Network test:
>
>
>  storage-02 / # iperf -s -B 127.0.0.1
>  ------------------------------------------------------------
>  Server listening on TCP port 5001
>  Binding to local address 127.0.0.1
>  TCP window size: 1.00 MByte (default)
>  ------------------------------------------------------------
>  [  4] local 127.0.0.1 port 5001 connected with 127.0.0.1 port 52464
>  [ ID] Interval       Transfer     Bandwidth
>  [  4]  0.0-20.0 sec  47.6 GBytes  20.4 Gbits/sec
>
>
>  storage-02 / # ls -la /mnt/mfs/
>  -rw-r--r-- 1 root root 17179869184 Apr 14 18:43 test.file
>
>  Read test from MooseFS:
>
>  dd if=/mnt/mfs/test.file of=/dev/null bs=4k
>  4194304+0 records in
>  4194304+0 records out
>  17179869184 bytes (17 GB) copied, 139.273 s, 123 MB/s
>
>  atop while doing read test:
>
>  PRC | sys   4.28s | user   3.10s | #proc    120 | #zombie    0 | 
> #exit
>     0 |
>  CPU | sys     58% | user     43% | irq       3% | idle    617% | 
> wait
>   80% |
>  cpu | sys     12% | user     12% | irq       1% | idle     57% | 
> cpu003
>  w 18% |
>  cpu | sys     14% | user      7% | irq       0% | idle     57% | 
> cpu007
>  w 21% |
>  cpu | sys      9% | user      6% | irq       1% | idle     64% | 
> cpu000
>  w 20% |
>  cpu | sys      9% | user      7% | irq       0% | idle     71% | 
> cpu004
>  w 13% |
>  cpu | sys      5% | user      5% | irq       0% | idle     90% | 
> cpu005
>  w  0% |
>  cpu | sys      5% | user      3% | irq       1% | idle     87% | 
> cpu001
>  w  4% |
>  cpu | sys      2% | user      2% | irq       0% | idle     96% | 
> cpu002
>  w  0% |
>  cpu | sys      3% | user      2% | irq       0% | idle     92% | 
> cpu006
>  w  3% |
>  CPL | avg1   0.68 | avg5    0.24 | avg15   0.15 | csw   123346 | 
> intr
>  61464 |
>  MEM | tot    3.9G | free   33.3M | cache   3.5G | buff    0.0M | 
> slab
>  184.4M |
>  SWP | tot  988.4M | free  983.9M |              | vmcom 344.5M | 
> vmlim
>  2.9G |
>  PAG | scan 297315 | stall      0 |              | swin       0 | 
> swout
>     0 |
>  DSK |         sdd | busy     10% | read     540 | write      1 | 
> avio
>  0 ms |
>  DSK |         sda | busy      6% | read       0 | write      6 | 
> avio
>  50 ms |
>  DSK |         sdc | busy      6% | read     692 | write      0 | 
> avio
>  0 ms |
>  DSK |         sdb | busy      1% | read       0 | write      6 | 
> avio
>  6 ms |
>  NET | transport   | tcpi   62264 | tcpo   80771 | udpi       0 | 
> udpo
>     0 |
>  NET | network     | ipi    62277 | ipo    62251 | ipfrw      0 | 
> deliv
>  62270 |
>  NET | eth1     0% | pcki      75 | pcko       7 | si    9 Kbps | so
>  6 Kbps |
>  NET | eth0     0% | pcki      22 | pcko      24 | si    3 Kbps | so
>  3 Kbps |
>  NET | lo     ---- | pcki   62221 | pcko   62221 | si  978 Mbps | so
>  978 Mbps |
>  NET | bond0  ---- | pcki      97 | pcko      31 | si   12 Kbps | so
>  10 Kbps |
>
>    PID  SYSCPU  USRCPU  VGROW  RGROW USERNAME THR  ST EXC  S  CPU CMD
>  1/1
>  28379   1.11s   1.31s   -44K   408K mfs       24  --   -  S  52%
>  mfschunkserver
>  28459   1.53s   0.75s     0K     0K root      19  --   -  S  49%
>  mfsmount
>  28425   0.93s   1.02s    16K  -256K mfs       24  --   -  S  42%
>  mfschunkserver
>  28687   0.48s   0.01s     0K     0K root       1  --   -  D  10% dd
>    437   0.20s   0.00s     0K     0K root       1  --   -  S   4%
>  kswapd0
>  28697   0.02s   0.00s  8676K   516K root       1  --   -  R   0% 
> atop
>  28373   0.00s   0.01s     0K     0K mfs        1  --   -  S   0%
>  mfsmaster
>  28510   0.01s   0.00s     0K     0K root       1  --   -  S   0%
>  kworker/7:0
>   4574   0.00s   0.00s     0K     0K root       1  --   -  S   0%
>  kworker/u:2
>  13251   0.00s   0.00s     0K     0K root       1  --   -  S   0%
>  md1_raid1
>  17959   0.00s   0.00s     0K     0K root       1  --   -  S   0%
>  xfsbufd/sdc
>
>
>    Is there a limitation set in moosefs or fuse client than sets the
>  maximum socket transfer at 1Gbit?
>  Is there a trick to unleash the moose beast? :P
>    If you have any ideea how can i get the full potential from my
>  storage please let me know how.
>    Looking forward for you reply!
>
>
>  Kind regards,
>
>  Razvan Dumitrescu
>  System engineer
>  Realitatea TV
>
>
>
>
> 
> ----------------------------------------------------------------------------
> --
> Benefiting from Server Virtualization: Beyond Initial Workload
> Consolidation -- Increasing the use of server virtualization is a top
> priority.Virtualization can reduce costs, simplify management, and 
> improve
> application availability and disaster protection. Learn more about 
> boosting
> the value of server virtualization. 
> http://p.sf.net/sfu/vmware-sfdev2dev
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Is there a limit set for network transfer of 1Gbit set in moosefs-fuse?

[Michal B. <mic...@ge...>]

Hi Razvan!

Thanks for nice words about MooseFS! :)

There is absolutely no built-in limit for transfers in MooseFS. And 130 MB/s
for SATA2 disks is a very good result - compare it to these benchmarks:
http://hothardware.com/printarticle.aspx?articleid=881. I would never expect
600MB/s on SATA2 disks. And what are results of 'dd' for the disks alone?
But remember that such tests do not reflect real life environments where
lots of users use the system in the same time.


Kind regards
Michał Borychowski 
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01




-----Original Message-----
From: Razvan Dumitrescu [mailto:ra...@re...] 
Sent: Thursday, April 14, 2011 7:11 PM
To: moo...@li...
Subject: [Moosefs-users] Is there a limit set for network transfer of 1Gbit
set in moosefs-fuse?

 Hello guys!

   First of all i have to say you made a great job with MooseFS, from 
 all the open source distributed filesystems i loooked over this is by 
 far the best
 in features, documentation and simple efficiency (installation, 
 configuration, tools). Great piece of work!
   Now in order to be in pure extasy while looking on how good my 
 storage performs i need to get an answer :).
   I have set up a MooseFS on a server box, 2 x Xeon X5355, 4GB RAM, 
 Intel 5000P board, 2 x Areca 1321 controllers, 24 x ST3750640NS disks.
   What puzzles me is that when i mount the MoooseFS on same machine and 
 i try to make different transfers the bottleneck the i hit is always
 around 1Gbit. My 2 storages sdc and sdd are both used around 15-20% 
 while reaching transfer from moosefs of ~130MB/s (both controllers 
 deliver around 600MB/s read and write).
 For this local test the loopback device is used (mounted on same 
 machine) and from what i can see using atop
 the only section that seems to cap is the network at loopback device 
 with a value slighltly under 1Gbit.
 Using iperf on same machine i get 20Gb/s so it seems loopback device 
 can't be blamed for the limitation.

 Mounts:

 storage-02 / # df -h
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/md1              149G   15G  135G  10% /
 udev                   10M  192K  9.9M   2% /dev
 shm                   2.0G     0  2.0G   0% /dev/shm
 /dev/sdc              6.9T   31G  6.8T   1% /mnt/osd0
 /dev/sdd              6.9T   31G  6.8T   1% /mnt/osd1
 192.168.8.88:9421      14T   62G   14T   1% /mnt/mfs

 Network test:


 storage-02 / # iperf -s -B 127.0.0.1
 ------------------------------------------------------------
 Server listening on TCP port 5001
 Binding to local address 127.0.0.1
 TCP window size: 1.00 MByte (default)
 ------------------------------------------------------------
 [  4] local 127.0.0.1 port 5001 connected with 127.0.0.1 port 52464
 [ ID] Interval       Transfer     Bandwidth
 [  4]  0.0-20.0 sec  47.6 GBytes  20.4 Gbits/sec


 storage-02 / # ls -la /mnt/mfs/
 -rw-r--r-- 1 root root 17179869184 Apr 14 18:43 test.file

 Read test from MooseFS:

 dd if=/mnt/mfs/test.file of=/dev/null bs=4k
 4194304+0 records in
 4194304+0 records out
 17179869184 bytes (17 GB) copied, 139.273 s, 123 MB/s

 atop while doing read test:

 PRC | sys   4.28s | user   3.10s | #proc    120 | #zombie    0 | #exit  
    0 |
 CPU | sys     58% | user     43% | irq       3% | idle    617% | wait   
  80% |
 cpu | sys     12% | user     12% | irq       1% | idle     57% | cpu003 
 w 18% |
 cpu | sys     14% | user      7% | irq       0% | idle     57% | cpu007 
 w 21% |
 cpu | sys      9% | user      6% | irq       1% | idle     64% | cpu000 
 w 20% |
 cpu | sys      9% | user      7% | irq       0% | idle     71% | cpu004 
 w 13% |
 cpu | sys      5% | user      5% | irq       0% | idle     90% | cpu005 
 w  0% |
 cpu | sys      5% | user      3% | irq       1% | idle     87% | cpu001 
 w  4% |
 cpu | sys      2% | user      2% | irq       0% | idle     96% | cpu002 
 w  0% |
 cpu | sys      3% | user      2% | irq       0% | idle     92% | cpu006 
 w  3% |
 CPL | avg1   0.68 | avg5    0.24 | avg15   0.15 | csw   123346 | intr   
 61464 |
 MEM | tot    3.9G | free   33.3M | cache   3.5G | buff    0.0M | slab  
 184.4M |
 SWP | tot  988.4M | free  983.9M |              | vmcom 344.5M | vmlim  
 2.9G |
 PAG | scan 297315 | stall      0 |              | swin       0 | swout  
    0 |
 DSK |         sdd | busy     10% | read     540 | write      1 | avio   
 0 ms |
 DSK |         sda | busy      6% | read       0 | write      6 | avio   
 50 ms |
 DSK |         sdc | busy      6% | read     692 | write      0 | avio   
 0 ms |
 DSK |         sdb | busy      1% | read       0 | write      6 | avio   
 6 ms |
 NET | transport   | tcpi   62264 | tcpo   80771 | udpi       0 | udpo   
    0 |
 NET | network     | ipi    62277 | ipo    62251 | ipfrw      0 | deliv  
 62270 |
 NET | eth1     0% | pcki      75 | pcko       7 | si    9 Kbps | so    
 6 Kbps |
 NET | eth0     0% | pcki      22 | pcko      24 | si    3 Kbps | so    
 3 Kbps |
 NET | lo     ---- | pcki   62221 | pcko   62221 | si  978 Mbps | so  
 978 Mbps |
 NET | bond0  ---- | pcki      97 | pcko      31 | si   12 Kbps | so   
 10 Kbps |

   PID  SYSCPU  USRCPU  VGROW  RGROW USERNAME THR  ST EXC  S  CPU CMD    
 1/1
 28379   1.11s   1.31s   -44K   408K mfs       24  --   -  S  52% 
 mfschunkserver
 28459   1.53s   0.75s     0K     0K root      19  --   -  S  49% 
 mfsmount
 28425   0.93s   1.02s    16K  -256K mfs       24  --   -  S  42% 
 mfschunkserver
 28687   0.48s   0.01s     0K     0K root       1  --   -  D  10% dd
   437   0.20s   0.00s     0K     0K root       1  --   -  S   4% 
 kswapd0
 28697   0.02s   0.00s  8676K   516K root       1  --   -  R   0% atop
 28373   0.00s   0.01s     0K     0K mfs        1  --   -  S   0% 
 mfsmaster
 28510   0.01s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/7:0
  4574   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/u:2
 13251   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 md1_raid1
 17959   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 xfsbufd/sdc


   Is there a limitation set in moosefs or fuse client than sets the 
 maximum socket transfer at 1Gbit?
 Is there a trick to unleash the moose beast? :P
   If you have any ideea how can i get the full potential from my 
 storage please let me know how.
   Looking forward for you reply!


 Kind regards,

 Razvan Dumitrescu
 System engineer
 Realitatea TV




----------------------------------------------------------------------------
--
Benefiting from Server Virtualization: Beyond Initial Workload 
Consolidation -- Increasing the use of server virtualization is a top
priority.Virtualization can reduce costs, simplify management, and improve 
application availability and disaster protection. Learn more about boosting 
the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Questions about MooseFS before putting it on the production env

[Michal B. <mic...@ge...>]

Hi Bob!

 

My replies are in the text. And for start please read this article:

http://www.moosefs.org/news-reader/items/metadata-ins-and-outs.html

 

 

From: ailms [mailto:ai...@qq...] 
Sent: Thursday, April 14, 2011 5:52 PM
To: moosefs-users
Subject: [Moosefs-users] Questions about MooseFS before putting it on the
production env

 

Dear guys,

We are plan to use moosefs as our store mechanism, which will be more than
5TB off-line command log. 

But now we have someconcern, Please find my queries below:

-----

1.  is there any plan about resolving the single-point failure of master ?

[MB] Please check a project developed by Thomas Hatch:
https://github.com/thatch45/mfs-failover 

It still needs some improvements but is a great way to test the failovers.
And Thomas needs some beta testers.

 

2.  The data will lost if master crash without dumping metadata from RAM
into disk,is it right?

[MB] Changelogs are saved continuously on the fly, there is very little
probability you would lose your data upon crash of the master server

 

3.  How can I know which data were lost if the question #2 happend?

[MB] When you make a recovery process of metadata you would know if it was
successful or not

 

4.  Dose the master only caches metadata read-only or will update the data
in RAM and flush to disk periodically?

[MB] The changes to metadata are saved to changelogs continuously on the fly

 

5.  Are the change logs based on the metadata.mfs.back or the data in RAM ?

[MB] We can consider them as based on RAM

 

6.  Can I change the default dumpling behaviour for master metadata from
per hour to user defined value? Such as 30 min?

[MB] As far as I remember, yes you can

 

7.  How can we make sure no data lost at any time with HA（KeepAlived）+
DRBD + mfs solution ?

[MB] We have not done such tests. Please first check a solution given in
question 1.

 

8.  The mfsmetaloggr didn't re-connect automatically even though I had
stoped and restarted the mfsmater, can only by manual?

[MB] Should have reconnected automatically. What was your environment? 

 

9.  which option in mfsmaster.cfg controls the rotation of changelog ?
according to the size ?  or each time when mfsmaster dump the metadata ?

[MB] Please see "man mfsmaster.cfg” - you should have all the options
listed

 

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

 


Any help will be appreciated. Looking forward your answer.


Best Regard

Bob Lin

 

 

 


 

Re: [Moosefs-users] now,i ask more questions,please help me,thanks

[Michal B. <mic...@ge...>]

Hi!

 

In problem one do you ask about 3000 million files? You would need about 3
000 * 300 MB = 878 GB of RAM which would be rather impossible. 

 

Yes, clients need to use FUSE, there is no support for NFS. And on our
roadmap there is no plan to support NFS, though we could think about
developing such a module for you commercially. 

 

 

Kind regards

Michal Borychowski 

 

 

From: ha...@si... [mailto:ha...@si...] 
Sent: Thursday, April 14, 2011 1:43 PM
To: moo...@li...
Subject: [Moosefs-users] now,i ask more questions,please help me,thanks

 


Hi 


First,thanks for your early reply ,now,i  ask more questions as follows: 

problem one: as you say in previous letters "there is also no limit for
number of files in the system.There are installations with 100 million
files(whether to support 3 billion file in the system??)."and i want to ask
:About MooseFS, if we insert just 4-8GB of RAM into the master
server,whether it support the storage 500T, 3 million files, operating 
500G times a day"?? 

problem two: as we know,for the moment,the clients need to use FUSE and if i
use NFS,wether to support"7 mul 24 hours"model?? 
  




	





That's all ,thanks a lot!   

Sincerely look forward to your reply! 



Best regards! 

Hanyw 

[Moosefs-users] bonnie++ test

[Fyodor U. <uf...@uf...>]

Hi!

I'm still not understand why bonnie++ "Sequental Outpet Char" and 
"Sequental Output rewrite" test show dramatically low results. May be 
anyone know about resolving this problem?

During the execution of this tests disk tps/bps, cpu, ethernet on 
chunk/meta servers not loaded. But results like:

Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
stb1-1           8G   128  30 91398  25 12128   2  4190  98 60089   3 
620.8   7
Latency             85864us   77570us     276ms    3763us     200ms     
147ms
Version  1.96       ------Sequential Create------ --------Random 
Create--------
stb1-1              -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16   825   4 32020  50  2891  10   691   4  3205   7  
1059   5
Latency             39868us    4367us   13013us   27704us    4812us     
178ms

Result in csv:
1.96,1.96,stb1-1,1,1302839383,8G,,128,30,91398,25,12128,2,4190,98,60089,3,620.8,7,16,,,,,825,4,32020,50,2891,10,691,4,3205,7,1059,5,85864us,77570us,276ms,3763us,200ms,147ms,39868us,4367us,13013us,27704us,4812us,178ms

mount command:

mfsmount -o mfscachefiles -o mfsentrycacheto=1 -o mfswritecachesize=256 /mnt

WBR,
     Fyodor.

[Moosefs-users] Is there a limit set for network transfer of 1Gbit set in moosefs-fuse?

[Razvan D. <ra...@re...>]

 Hello guys!

   First of all i have to say you made a great job with MooseFS, from 
 all the open source distributed filesystems i loooked over this is by 
 far the best
 in features, documentation and simple efficiency (installation, 
 configuration, tools). Great piece of work!
   Now in order to be in pure extasy while looking on how good my 
 storage performs i need to get an answer :).
   I have set up a MooseFS on a server box, 2 x Xeon X5355, 4GB RAM, 
 Intel 5000P board, 2 x Areca 1321 controllers, 24 x ST3750640NS disks.
   What puzzles me is that when i mount the MoooseFS on same machine and 
 i try to make different transfers the bottleneck the i hit is always
 around 1Gbit. My 2 storages sdc and sdd are both used around 15-20% 
 while reaching transfer from moosefs of ~130MB/s (both controllers 
 deliver around 600MB/s read and write).
 For this local test the loopback device is used (mounted on same 
 machine) and from what i can see using atop
 the only section that seems to cap is the network at loopback device 
 with a value slighltly under 1Gbit.
 Using iperf on same machine i get 20Gb/s so it seems loopback device 
 can't be blamed for the limitation.

 Mounts:

 storage-02 / # df -h
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/md1              149G   15G  135G  10% /
 udev                   10M  192K  9.9M   2% /dev
 shm                   2.0G     0  2.0G   0% /dev/shm
 /dev/sdc              6.9T   31G  6.8T   1% /mnt/osd0
 /dev/sdd              6.9T   31G  6.8T   1% /mnt/osd1
 192.168.8.88:9421      14T   62G   14T   1% /mnt/mfs

 Network test:


 storage-02 / # iperf -s -B 127.0.0.1
 ------------------------------------------------------------
 Server listening on TCP port 5001
 Binding to local address 127.0.0.1
 TCP window size: 1.00 MByte (default)
 ------------------------------------------------------------
 [  4] local 127.0.0.1 port 5001 connected with 127.0.0.1 port 52464
 [ ID] Interval       Transfer     Bandwidth
 [  4]  0.0-20.0 sec  47.6 GBytes  20.4 Gbits/sec


 storage-02 / # ls -la /mnt/mfs/
 -rw-r--r-- 1 root root 17179869184 Apr 14 18:43 test.file

 Read test from MooseFS:

 dd if=/mnt/mfs/test.file of=/dev/null bs=4k
 4194304+0 records in
 4194304+0 records out
 17179869184 bytes (17 GB) copied, 139.273 s, 123 MB/s

 atop while doing read test:

 PRC | sys   4.28s | user   3.10s | #proc    120 | #zombie    0 | #exit  
    0 |
 CPU | sys     58% | user     43% | irq       3% | idle    617% | wait   
  80% |
 cpu | sys     12% | user     12% | irq       1% | idle     57% | cpu003 
 w 18% |
 cpu | sys     14% | user      7% | irq       0% | idle     57% | cpu007 
 w 21% |
 cpu | sys      9% | user      6% | irq       1% | idle     64% | cpu000 
 w 20% |
 cpu | sys      9% | user      7% | irq       0% | idle     71% | cpu004 
 w 13% |
 cpu | sys      5% | user      5% | irq       0% | idle     90% | cpu005 
 w  0% |
 cpu | sys      5% | user      3% | irq       1% | idle     87% | cpu001 
 w  4% |
 cpu | sys      2% | user      2% | irq       0% | idle     96% | cpu002 
 w  0% |
 cpu | sys      3% | user      2% | irq       0% | idle     92% | cpu006 
 w  3% |
 CPL | avg1   0.68 | avg5    0.24 | avg15   0.15 | csw   123346 | intr   
 61464 |
 MEM | tot    3.9G | free   33.3M | cache   3.5G | buff    0.0M | slab  
 184.4M |
 SWP | tot  988.4M | free  983.9M |              | vmcom 344.5M | vmlim  
 2.9G |
 PAG | scan 297315 | stall      0 |              | swin       0 | swout  
    0 |
 DSK |         sdd | busy     10% | read     540 | write      1 | avio   
 0 ms |
 DSK |         sda | busy      6% | read       0 | write      6 | avio   
 50 ms |
 DSK |         sdc | busy      6% | read     692 | write      0 | avio   
 0 ms |
 DSK |         sdb | busy      1% | read       0 | write      6 | avio   
 6 ms |
 NET | transport   | tcpi   62264 | tcpo   80771 | udpi       0 | udpo   
    0 |
 NET | network     | ipi    62277 | ipo    62251 | ipfrw      0 | deliv  
 62270 |
 NET | eth1     0% | pcki      75 | pcko       7 | si    9 Kbps | so    
 6 Kbps |
 NET | eth0     0% | pcki      22 | pcko      24 | si    3 Kbps | so    
 3 Kbps |
 NET | lo     ---- | pcki   62221 | pcko   62221 | si  978 Mbps | so  
 978 Mbps |
 NET | bond0  ---- | pcki      97 | pcko      31 | si   12 Kbps | so   
 10 Kbps |

   PID  SYSCPU  USRCPU  VGROW  RGROW USERNAME THR  ST EXC  S  CPU CMD    
 1/1
 28379   1.11s   1.31s   -44K   408K mfs       24  --   -  S  52% 
 mfschunkserver
 28459   1.53s   0.75s     0K     0K root      19  --   -  S  49% 
 mfsmount
 28425   0.93s   1.02s    16K  -256K mfs       24  --   -  S  42% 
 mfschunkserver
 28687   0.48s   0.01s     0K     0K root       1  --   -  D  10% dd
   437   0.20s   0.00s     0K     0K root       1  --   -  S   4% 
 kswapd0
 28697   0.02s   0.00s  8676K   516K root       1  --   -  R   0% atop
 28373   0.00s   0.01s     0K     0K mfs        1  --   -  S   0% 
 mfsmaster
 28510   0.01s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/7:0
  4574   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 kworker/u:2
 13251   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 md1_raid1
 17959   0.00s   0.00s     0K     0K root       1  --   -  S   0% 
 xfsbufd/sdc


   Is there a limitation set in moosefs or fuse client than sets the 
 maximum socket transfer at 1Gbit?
 Is there a trick to unleash the moose beast? :P
   If you have any ideea how can i get the full potential from my 
 storage please let me know how.
   Looking forward for you reply!


 Kind regards,

 Razvan Dumitrescu
 System engineer
 Realitatea TV

[Moosefs-users] Questions about MooseFS before putting it on the production env

[a. <ai...@qq...>]

Dear guys,
 We are plan to use moosefs as our store mechanism, which will be more than 5TB off-line command log. 
 But now we have someconcern, Please find my queries below:
 -----
 1. is there any plan about resolving the single-point failure of master ?
 2. The data will lost if master crash without dumping metadata from RAM into disk,is it right?
 3. How can I know which data were lost if the question #2 happend?
 4. Dose the master only caches metadata read-only or will update the data in RAM and flush to disk periodically?
 5. Are the change logs based on the metadata.mfs.back or the data in RAM ?
 6. Can I change the default dumpling behaviour for master metadata from per hour to user defined value? Such as 30 min?
 7. How can we make sure no data lost at any time with HA（KeepAlived）+ DRBD + mfs solution ?
 8. The mfsmetaloggr didn't re-connect automatically even though I had stoped and restarted the mfsmater, can only by manual?
 9. which option in mfsmaster.cfg controls the rotation of changelog ? according to the size ?  or each time when mfsmaster dump the metadata ?
 
Any help will be appreciated. Looking forward your answer.
 
Best Regard
 Bob Lin

[Moosefs-users] now,i ask more questions,please help me,thanks

[<ha...@si...>]

Hi 


First,thanks for your early reply ,now,i  ask more questions as follows:

problem one: as you say in previous letters "there is also no limit for 
number of files in the system.There are installations with 100 million 
files(whether to support 3 billion file in the system??)."and i want to 
ask :About MooseFS, if we insert just 4-8GB of RAM into the master server
,whether it support the storage 500T, 3 million files, operating
500G times a day"??

problem two: as we know,for the moment,the clients need to use FUSE and if 
i use NFS,wether to support"7 mul 24 hours"model??
 








That's all ,thanks a lot! 

Sincerely look forward to your reply! 



Best regards! 

Hanyw

Re: [Moosefs-users] Hi, i have some doubts , please help me and answer me

[Michal B. <mic...@ge...>]

Hi!

 

Limit of 2TiB is for a single file. There is no limit for the total size of
all files in the system. 

 

There is also no limit for number of files in the system. There are
installations with 100 million files.

 

RAM needed for the master server depends only on the number of files in
your system (not on their contents or size). Please have a look here:
http://www.moosefs.org/moosefs-faq.html#sort

 

 

Kind regards

Michal

 

From: ha...@si... [mailto:ha...@si...] 
Sent: Wednesday, April 13, 2011 4:42 AM
To: moo...@li...
Subject: [Moosefs-users] Hi, i have some doubts , please help me and answer
me

 


  

hi,i have some  doubts ,please have a look as follows: 

Some days ago, i asked the question that "About MooseFS, If the storage
500T, 3 million files, operating 
500G times a day, how much memory the metadata needed? " 

Your answer is:For 3 million files you need to insert just 4-8GB of RAM
into the master server. 

However,today i see the FAQ about " 


Does MooseFS have file size limit? 


  

Currently MooseFS imposes a maximum file size limit of 2 TiB (2,199,023,255,
552 bytes). However we are considering removing this limitation in the near
future, at which point the maximum file size will reach the limits of the
operating system, which is currently 16EiB (18,446,744,073,709,551,616
bytes). 


" 

from the FAQ,such as "Currently MooseFS imposes a maximum file size limit
of 2 TiB (2,199,023,255,552 bytes),and so on." 


 i have some doubts,now i want to ask  you some questions ，please answer
me : 

problem one:  About MooseFS,what size does it support the single file? and
what size about the total numbers of file?? 

problem two: As your answer"For 3 million files you need to insert just 4-
8GB of RAM into the master server. ", 
wether it supports the situation "About MooseFS, if we insert just 4-8GB of
RAM into the master server,it'll support the storage 500T, 3 million files,
operating 
500G times a day"?? 


That's all ,thanks a lot!   

Sincerely look forward to your reply! 



Best regards! 

Hanyw 

[Moosefs-users] Hi, i have some doubts , please help me and answer me

[<ha...@si...>]

 hi,i have some  doubts ,please have a look as follows: 

Some days ago, i asked the question that "About MooseFS, If the storage 
500T, 3 million files, operating
500G times a day, how much memory the metadata needed? "

Your answer is:For 3 million files you need to insert just 4-8GB of RAM 
into the master server. 

However,today i see the FAQ about "

Does MooseFS have file size limit?
 
Currently MooseFS imposes a maximum file size limit of 2 TiB 
(2,199,023,255,552 bytes). However we are considering removing this 
limitation in the near future, at which point the maximum file size will 
reach the limits of the operating system, which is currently 16EiB 
(18,446,744,073,709,551,616 bytes). 
"

from the FAQ,such as "Currently MooseFS imposes a maximum file size limit 
of 2 TiB (2,199,023,255,552 bytes),and so on."


 i have some doubts,now i want to ask  you some questions ，please answer 
me :

problem one:  About MooseFS,what size does it support the single file? and 
what size about the total numbers of file??

problem two: As your answer"For 3 million files you need to insert just 
4-8GB of RAM into the master server. ",
wether it supports the situation "About MooseFS, if we insert just 4-8GB 
of RAM into the master server,it'll support the storage 500T, 3 million 
files, operating
500G times a day"??


That's all ,thanks a lot! 

Sincerely look forward to your reply! 



Best regards! 

Hanyw

Re: [Moosefs-users] More nexenta bugs

[Robert D. <ro...@in...>]

I'll follow up with this in a week or two, I am waiting on some more boxes
to test with.

 

-Rob

 

  _____  

From: Michal Borychowski [mailto:mic...@ge...] 
Sent: Tuesday, April 05, 2011 1:27 AM
To: 'Robert Dye'
Cc: moo...@li...
Subject: Re: [Moosefs-users] More nexenta bugs

 

Hi Robert!

 

Yes, we'd like MooseFS to be fully compatible with Nexenta. So if you still
have some issues, please send them to us.

 

We are (almost) sure that hanging up of the chunkserver was not connected
with SIGPIPE signal. To the "csserv_serve" function we added a test
fragment:

 

//sigpipe test

        int fd[2];

        pipe(fd);

        close(fd[0]);

        if (write(fd[1],"",1)<0) {

                mfs_errlog(LOG_NOTICE,"sigppipe test");

        }

        close(fd[1]);        

//sigpipe test

 

and everything worked fine, the logs showed:

 

(...) nexenta mfschunkserver[14100]: [ID 801593 daemon.notice] sigppipe
test: EPIPE (Broken pipe)

(...)

 

 

Is your problem repeatable? Where did you get information about SIGPIPE
from? Have you correctly run gdb (with the correct core)? Can you run "info
threads" in gdb? Maybe it will show us a real cause of the problem.

 

Thank you!

 

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

 

 

From: Robert Dye [mailto:ro...@in...] 
Sent: Thursday, March 17, 2011 7:28 PM
To: moo...@li...
Subject: [Moosefs-users] More nexenta bugs

 

I guess I should ask if I should be submitting Nexenta bugs, if this is
something the project will support in the future?  Either way, looks like I
was able to get specific trace information (shown below).

 

Problem:  mfschunkserver segfaults after running for a few hours.

 

Output:  

[New LWP 5]

main server module: listen on *:9422

[New LWP 6]

[New LWP 7]

[New LWP 8]

[New LWP 9]

[New LWP 10]

[New LWP 11]

[New LWP 12]

[New LWP 13]

[New LWP 14]

[New LWP 15]

[New LWP 16]

[New LWP 17]

[New LWP 18]

[New LWP 19]

[New LWP 20]

[New LWP 21]

[New LWP 22]

[New LWP 23]

[New LWP 24]

[New LWP 25]

stats file has been loaded

mfschunkserver daemon initialized properly

 

 

 

Program received signal SIGPIPE, Broken pipe.

0xfeed57d5 in __write () from /lib/libc.so.1

 

(gdb) bt

#0  0xfeed57d5 in __write () from /lib/libc.so.1

#1  0xfeebf37e in write () from /lib/libc.so.1

#2  0x0804d684 in csserv_write (eptr=0x8a32bf0) at csserv.c:1631

#3  0x0804f604 in csserv_serve (pdesc=0x8033c90) at csserv.c:1900

#4  0x0806208c in mainloop () at ../mfscommon/main.c:348

#5  0x080626fb in main (argc=<value optimized out>, argv=0x40) at
../mfscommon/main.c:1101

 

 

Re: [Moosefs-users] Core Dumped from mfsmount with Autofs

[Michal B. <mic...@ge...>]

Thanks for the information. We are now looking into it


Regards
Michal

-----Original Message-----
From: 姜智华 [mailto:fl...@gm...] 
Sent: Wednesday, April 06, 2011 2:12 PM
To: Michal Borychowski
Cc: moo...@li...
Subject: Re: [Moosefs-users] Core Dumped from mfsmount with Autofs

Hi, Michal

I tried to spent some time today on MFS and noticed both the core dump
and the Valgrind trace you gave was generated from "read_data_term",
but the patch you gave was at "read_data_end". So I tried to apply the
same strategy in both "read_data_end" AND "read_data_term", here's the
patch I made:

@@ -178,6 +178,7 @@
        }
        if (rrec->rbuff!=NULL) {
                free(rrec->rbuff);
+               rrec->rbuff=NULL;
        }

        pthread_mutex_lock(&glock);
@@ -221,6 +222,7 @@
                        }
                        if (rr->rbuff) {
                                free(rr->rbuff);
+                               rr->rbuff = NULL;
                        }
                        pthread_cond_destroy(&(rr->cond));
                        free(rr);

Now no more core files!!!

So could you please confirm if the change in "read_data_term" is also
expected (or the patch you gave in "read_data_end" should actually
happen in "read_data_term"?)

If you don't think the patch I made is reasonable or still want to
recreate the issue, I'll send my AutoFS configuration later.

Thanks
Flow

On 4/5/11, Michal Borychowski <mic...@ge...> wrote:
> Hi!
>
> We made tests on several different operating system and are not able to
> reproduce your error. Valgrind also doesn't fined anything bad.
>
> Without the patch we have:
> root@ubuntu10-64b:~/mfs-1.6.21/mfsmount# valgrind ./mfsmount -f ==1869==
> Memcheck, a memory error detector ==1869== Copyright (C) 2002-2009, and GNU
> GPL'd, by Julian Seward et al.
> ==1869== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for
> copyright info ==1869== Command: ./mfsmount -f ==1869== mfsmaster accepted
> connection with parameters:
> read-write,restricted_ip,ignore_gid,can_change_quota ; root mapped to
> root:root ==1869== Invalid free() / delete / delete[]
> ==1869==    at 0x4C270BD: free (vg_replace_malloc.c:366)
> ==1869==    by 0x41366C: read_data_term (readdata.c:224)
> ==1869==    by 0x418CF4: mainloop (main.c:698)
> ==1869==    by 0x41906B: main (main.c:941)
> ==1869==  Address 0x67f56e0 is not stack'd, malloc'd or (recently) free'd
> ==1869==
> mfsmount[1869]: master: connection lost (1)
> (...)
>
>
> But after applying the patch we have:
> ==1947== Memcheck, a memory error detector ==1947== Copyright (C) 2002-2009,
> and GNU GPL'd, by Julian Seward et al.
> ==1947== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for
> copyright info ==1947== Command: ./mfsmount -f ==1947== mfsmaster accepted
> connection with parameters:
> read-write,restricted_ip,ignore_gid,can_change_quota ; root mapped to
> root:root
> mfsmount[1947]: master: connection lost (1)
> (...)
>
>
> If your problem exists only while using mount with autofs, please send your
> autofs configuration - we'll try to recreate the problem again. You may
> again check if you applied our patch correctly.
>
> BTW. We strongly not recommend to use the "mfscachefiles" option. It forces
> to leave files in the cache forever. It's not good. From 1.6.21 its usage
> will be marked as deprecated and probably from 1.7 we will remove it
> completely.
>
>
> Regards
> Michal
>
>
> -----Original Message-----
> From: 姜智华 [mailto:fl...@gm...]
> Sent: Wednesday, March 16, 2011 4:32 AM
> To: Michal Borychowski
> Cc: moo...@li...
> Subject: Re: [Moosefs-users] Core Dumped from mfsmount with Autofs
>
> Hi,
>
> I tried the patch but the core file still gets dumped (with mfs 1.6.20)
>
> Core was generated by `mfsmount /home/fwjiang -o
> rw,mfscachefiles,mfsentrycacheto=30,mfsattrcacheto=30'.
> Program terminated with signal 6, Aborted.
> #0  0x00000031c16327f5 in raise () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install
> filesystem-2.4.30-2.fc12.x86_64 fuse-libs-2.8.5-2.fc12.x86_64
> glibc-2.11.2-3.x86_64 libgcc-4.4.4-10.fc12.x86_64
> (gdb) bt
> #0  0x00000031c16327f5 in raise () from /lib64/libc.so.6
> #1  0x00000031c1633fd5 in abort () from /lib64/libc.so.6
> #2  0x00000031c166fa1b in __libc_message () from /lib64/libc.so.6
> #3  0x00000031c1675336 in malloc_printerr () from /lib64/libc.so.6
> #4  0x000000000040e4ad in read_data_term () at readdata.c:224
> #5  0x00000000004131b5 in mainloop (args=0x7fffc23bf030,
>     mp=0xacc290 "/home/fwjiang", mt=1, fg=0) at main.c:600
> #6  0x00000000004134f8 in main (argc=<value optimized out>,
>     argv=0x7fffc23bf158) at main.c:819
>
> Any clues?
>
> Thanks
> Flow
>
> On 3/15/11, Michal Borychowski <mic...@ge...> wrote:
>> It'll be fixed in the next release. For the moment you may try this
>> "patch":
>>
>>
>> @@ -178,6 +178,7 @@ void read_data_end(void* rr) {
>>         }
>>         if (rrec->rbuff!=NULL) {
>>                 free(rrec->rbuff);
>> +               rrec->rbuff=NULL;
>>         }
>>
>>         pthread_mutex_lock(&glock);
>>
>>
>> Kind regards
>> Michal
>>
>> -----Original Message-----
>> From: Flow Jiang [mailto:fl...@gm...]
>> Sent: Friday, March 04, 2011 4:21 PM
>> To: Michal Borychowski
>> Cc: moo...@li...
>> Subject: Re: [Moosefs-users] Core Dumped from mfsmount with Autofs
>>
>> I tried to re-compile mfsmount with the "free(freecblockshead)" line
>> commented out. Now our servers (which keep running 7x24) are happy, no
>> more core files. However, core files still gets generated on our
>> workstations when they reboot. The core is generated from the
>> "read_data_term" line right after the "write_data_term" line mentioned
>> previously.
>>
>> Hopefully this will also get fixed in next release, and will even be
>> better if I can have a quick solution / patch for the issue.
>>
>> Thanks
>> Flow
>>
>> On 03/01/2011 11:37 PM, Flow Jiang wrote:
>>> Michal,
>>>
>>> Glad to know that this error could be simply solved by commenting out
>>> that line and will try tomorrow to see if it fixes this issue.
>>>
>>> It does annoying since each core file takes about 170M and I tried to
>>> disable the core dump but failed. So hopefully we can have a better
>>> solution in the next release.
>>>
>>> Thanks
>>> Flow
>>>
>>> On 03/01/2011 09:00 PM, Michal Borychowski wrote:
>>>> Hi!
>>>>
>>>> This error is not a serious one. It may happen only upon exits. If these
>>>> errors are annoying a quick solution is to comment out the
>>>> "free(freecblockshead)" line, recompile mfsmount and run again. We'll
>>>> prepare a better solution in the next release.
>>>>
>>>>
>>>> Kind regards
>>>> Michał
>>
>> ------------------------------------------------------------------------------
>> What You Don't Know About Data Connectivity CAN Hurt You
>> This paper provides an overview of data connectivity, details
>> its effect on application quality, and explores various alternative
>> solutions. http://p.sf.net/sfu/progress-d2d
>> _______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>>
>
> ------------------------------------------------------------------------------
> Colocation vs. Managed Hosting
> A question and answer guide to determining the best fit
> for your organization - today and in the future.
> http://p.sf.net/sfu/internap-sfd2d
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>

------------------------------------------------------------------------------
Xperia(TM) PLAY
It's a major breakthrough. An authentic gaming
smartphone on the nation's most reliable network.
And it wants your games.
http://p.sf.net/sfu/verizon-sfdev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] why the moosefs read speed slower than writespeed?

[Giovanni T. <gt...@li...>]

Il 07/04/2011 11:19, gujubiao ha scritto:
> 
> hello,Giovanni Toraldo,can you tell me how to specify bs parameter?thanks very much!!

*you* have used it on the first dd writing test, you should do the same
also for reading:

time dd if=/mnt/mfs/mfstest of=/dev/zero bs=1024k


-- 
Giovanni Toraldo
http://www.libersoft.it/

Re: [Moosefs-users] why the moosefs read speed slower than writespeed?

[gujubiao <guj...@ge...>]

hello,Giovanni Toraldo,can you tell me how to specify bs parameter?thanks very much!!


2011-04-07 




Best Regards!!
-----------------------------------------------------------------------------------
古举标(Juby,Gu) 存储工程师　信息生产平台 系统支持组
Mobile:13723406010
ＱＱ：190247054
Email:guj...@ge...
华大基因研究院(BGI)
地址：深圳市盐田区北山工业区综合楼1001（518083）
Addr:No.1001,Floor 10,Main Building,Beishan Industrial Zone,Yantian District,Shenzhen,China
Post Code:518083
------------------------------------------------------------------------------------




发件人： Giovanni Toraldo 
发送时间： 2011-04-07  17:15:30 
收件人： moosefs-users 
抄送： 
主题： Re: [Moosefs-users] why the moosefs read speed slower than writespeed? 
 
Il 06/04/2011 11:10, gujubiao ha scritto:
>     why the moosefs read speed slower than write speed? somebody who
> knows the problem?please help.thanks very much!!(see the figure below)
maybe because you should specify bs parameter to dd even when reading?
-- 
Giovanni Toraldo
http://www.libersoft.it/
------------------------------------------------------------------------------
Xperia(TM) PLAY
It's a major breakthrough. An authentic gaming
smartphone on the nation's most reliable network.
And it wants your games.
http://p.sf.net/sfu/verizon-sfdev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] why the moosefs read speed slower than write speed?

[Giovanni T. <gt...@li...>]

Il 06/04/2011 11:10, gujubiao ha scritto:
>     why the moosefs read speed slower than write speed? somebody who
> knows the problem?please help.thanks very much!!(see the figure below)

maybe because you should specify bs parameter to dd even when reading?

-- 
Giovanni Toraldo
http://www.libersoft.it/

Re: [Moosefs-users] Core Dumped from mfsmount with Autofs

[姜智华 <fl...@gm...>]

Hi, Michal

I tried to spent some time today on MFS and noticed both the core dump
and the Valgrind trace you gave was generated from "read_data_term",
but the patch you gave was at "read_data_end". So I tried to apply the
same strategy in both "read_data_end" AND "read_data_term", here's the
patch I made:

@@ -178,6 +178,7 @@
        }
        if (rrec->rbuff!=NULL) {
                free(rrec->rbuff);
+               rrec->rbuff=NULL;
        }

        pthread_mutex_lock(&glock);
@@ -221,6 +222,7 @@
                        }
                        if (rr->rbuff) {
                                free(rr->rbuff);
+                               rr->rbuff = NULL;
                        }
                        pthread_cond_destroy(&(rr->cond));
                        free(rr);

Now no more core files!!!

So could you please confirm if the change in "read_data_term" is also
expected (or the patch you gave in "read_data_end" should actually
happen in "read_data_term"?)

If you don't think the patch I made is reasonable or still want to
recreate the issue, I'll send my AutoFS configuration later.

Thanks
Flow

On 4/5/11, Michal Borychowski <mic...@ge...> wrote:
> Hi!
>
> We made tests on several different operating system and are not able to
> reproduce your error. Valgrind also doesn't fined anything bad.
>
> Without the patch we have:
> root@ubuntu10-64b:~/mfs-1.6.21/mfsmount# valgrind ./mfsmount -f ==1869==
> Memcheck, a memory error detector ==1869== Copyright (C) 2002-2009, and GNU
> GPL'd, by Julian Seward et al.
> ==1869== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for
> copyright info ==1869== Command: ./mfsmount -f ==1869== mfsmaster accepted
> connection with parameters:
> read-write,restricted_ip,ignore_gid,can_change_quota ; root mapped to
> root:root ==1869== Invalid free() / delete / delete[]
> ==1869==    at 0x4C270BD: free (vg_replace_malloc.c:366)
> ==1869==    by 0x41366C: read_data_term (readdata.c:224)
> ==1869==    by 0x418CF4: mainloop (main.c:698)
> ==1869==    by 0x41906B: main (main.c:941)
> ==1869==  Address 0x67f56e0 is not stack'd, malloc'd or (recently) free'd
> ==1869==
> mfsmount[1869]: master: connection lost (1)
> (...)
>
>
> But after applying the patch we have:
> ==1947== Memcheck, a memory error detector ==1947== Copyright (C) 2002-2009,
> and GNU GPL'd, by Julian Seward et al.
> ==1947== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for
> copyright info ==1947== Command: ./mfsmount -f ==1947== mfsmaster accepted
> connection with parameters:
> read-write,restricted_ip,ignore_gid,can_change_quota ; root mapped to
> root:root
> mfsmount[1947]: master: connection lost (1)
> (...)
>
>
> If your problem exists only while using mount with autofs, please send your
> autofs configuration - we'll try to recreate the problem again. You may
> again check if you applied our patch correctly.
>
> BTW. We strongly not recommend to use the "mfscachefiles" option. It forces
> to leave files in the cache forever. It's not good. From 1.6.21 its usage
> will be marked as deprecated and probably from 1.7 we will remove it
> completely.
>
>
> Regards
> Michal
>
>
> -----Original Message-----
> From: 姜智华 [mailto:fl...@gm...]
> Sent: Wednesday, March 16, 2011 4:32 AM
> To: Michal Borychowski
> Cc: moo...@li...
> Subject: Re: [Moosefs-users] Core Dumped from mfsmount with Autofs
>
> Hi,
>
> I tried the patch but the core file still gets dumped (with mfs 1.6.20)
>
> Core was generated by `mfsmount /home/fwjiang -o
> rw,mfscachefiles,mfsentrycacheto=30,mfsattrcacheto=30'.
> Program terminated with signal 6, Aborted.
> #0  0x00000031c16327f5 in raise () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install
> filesystem-2.4.30-2.fc12.x86_64 fuse-libs-2.8.5-2.fc12.x86_64
> glibc-2.11.2-3.x86_64 libgcc-4.4.4-10.fc12.x86_64
> (gdb) bt
> #0  0x00000031c16327f5 in raise () from /lib64/libc.so.6
> #1  0x00000031c1633fd5 in abort () from /lib64/libc.so.6
> #2  0x00000031c166fa1b in __libc_message () from /lib64/libc.so.6
> #3  0x00000031c1675336 in malloc_printerr () from /lib64/libc.so.6
> #4  0x000000000040e4ad in read_data_term () at readdata.c:224
> #5  0x00000000004131b5 in mainloop (args=0x7fffc23bf030,
>     mp=0xacc290 "/home/fwjiang", mt=1, fg=0) at main.c:600
> #6  0x00000000004134f8 in main (argc=<value optimized out>,
>     argv=0x7fffc23bf158) at main.c:819
>
> Any clues?
>
> Thanks
> Flow
>
> On 3/15/11, Michal Borychowski <mic...@ge...> wrote:
>> It'll be fixed in the next release. For the moment you may try this
>> "patch":
>>
>>
>> @@ -178,6 +178,7 @@ void read_data_end(void* rr) {
>>         }
>>         if (rrec->rbuff!=NULL) {
>>                 free(rrec->rbuff);
>> +               rrec->rbuff=NULL;
>>         }
>>
>>         pthread_mutex_lock(&glock);
>>
>>
>> Kind regards
>> Michal
>>
>> -----Original Message-----
>> From: Flow Jiang [mailto:fl...@gm...]
>> Sent: Friday, March 04, 2011 4:21 PM
>> To: Michal Borychowski
>> Cc: moo...@li...
>> Subject: Re: [Moosefs-users] Core Dumped from mfsmount with Autofs
>>
>> I tried to re-compile mfsmount with the "free(freecblockshead)" line
>> commented out. Now our servers (which keep running 7x24) are happy, no
>> more core files. However, core files still gets generated on our
>> workstations when they reboot. The core is generated from the
>> "read_data_term" line right after the "write_data_term" line mentioned
>> previously.
>>
>> Hopefully this will also get fixed in next release, and will even be
>> better if I can have a quick solution / patch for the issue.
>>
>> Thanks
>> Flow
>>
>> On 03/01/2011 11:37 PM, Flow Jiang wrote:
>>> Michal,
>>>
>>> Glad to know that this error could be simply solved by commenting out
>>> that line and will try tomorrow to see if it fixes this issue.
>>>
>>> It does annoying since each core file takes about 170M and I tried to
>>> disable the core dump but failed. So hopefully we can have a better
>>> solution in the next release.
>>>
>>> Thanks
>>> Flow
>>>
>>> On 03/01/2011 09:00 PM, Michal Borychowski wrote:
>>>> Hi!
>>>>
>>>> This error is not a serious one. It may happen only upon exits. If these
>>>> errors are annoying a quick solution is to comment out the
>>>> "free(freecblockshead)" line, recompile mfsmount and run again. We'll
>>>> prepare a better solution in the next release.
>>>>
>>>>
>>>> Kind regards
>>>> Michał
>>
>> ------------------------------------------------------------------------------
>> What You Don't Know About Data Connectivity CAN Hurt You
>> This paper provides an overview of data connectivity, details
>> its effect on application quality, and explores various alternative
>> solutions. http://p.sf.net/sfu/progress-d2d
>> _______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>>
>
> ------------------------------------------------------------------------------
> Colocation vs. Managed Hosting
> A question and answer guide to determining the best fit
> for your organization - today and in the future.
> http://p.sf.net/sfu/internap-sfd2d
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>

Re: [Moosefs-users] Hello, I want to ask some questions , Sincerely look forward to your reply!

[Michal B. <mic...@ge...>]

Hi Hanyw!

 

Thanks for your interest in MooseFS!

 

 

For 3 million files you need to insert just 4-8GB of RAM into the master
server.

 

 

Why do you think B-TREE would be better than HASH? B-TREEs are good when we
need to browse elements in a given order. When talking about a file system
there is no such need. 

 

Operations made on data structures are: 1) return all elements (readdir) -
here a simple list is sufficient and 2) find a descendant in X node of name
Y (lookup) - this Is a typical key->value mapping where nothing better than
HASH could be used. 

 

The two structures could be combined in one B-TREE but then both operations
would be slower. Maybe we could gain some RAM, but the gain would really be
just several per cents.

 

 

For the moment MooseFS doesn't support INFINIBAND and this is not on our
shortlist. Generally speaking network throughput is not a bottleneck. For
the moment, the speed of commodity hard drives is too low. 

Alternatively you can use IPoIB (IP over InfiniBand) and communicate through
TCP/IP - it won't be very efficient, but probably better than nothing if you
think about INFINIBAND.

 

 

And I already replied about HP UX. You may also have a look at the bottom of
the page http://www.moosefs.org/about-mfs.html. And we'd like MooseFS to be
compatible with every *NIX system.

 

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

 

From: ha...@si... [mailto:ha...@si...] 
Sent: Friday, March 25, 2011 12:36 PM
To: moo...@li...
Subject: [Moosefs-users] Hello, I want to ask some questions , Sincerely
look forward to your reply!

 


Hello, everyone! 

We intend to use moosefs at our product environment as the storage of our
product service. I want to ask some questions as follows: 

Problem One: About MooseFS, If the storage 500T, 3 million files, operating
500G times a day, how much memory the metadata needed? 

Problem Two: If you modify the management about the metadata's namespace ,
such as from the HASH to B-TREE, whether it needs a lot of work 
and  whether you feel it the feasibility and reasonable? 

Problem Three: About MooseFS, MFS whether to support INFINIBAND,  you think
whether need more work and feel reasonable if we modify it to support? 


And the last One: About MooseFS,Whether to support the IBM AIX and HP UX ?
Which OS it supports and which one it doesn't ,please tell me what list? 

That's all ,thanks a lot!   

Sincerely look forward to your reply! 



Best regards! 

Hanyw 

[Moosefs-users] MooseFS questions

[Michal B. <mic...@ge...>]

Hi Vadim!

 

Thanks for your interest in MooseFS. A good place to send questions is this
list (in cc).

 

Ad. 1 - you may perfectly use MooseFS as a video storage, the other question
is streaming and load balancing of the requests; I guess you should have
some other machines "in front of" MooseFS for serving the content and you
should consider MooseFS as a safe storage

 

Ad. 2 - yes; and metalogger can be both metalogger and client or metalogger
and chunkserver - there are no restrictions

 

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

 

 

Hello

I am planing to start using moosefs an I have some qiestions. I hope I
am writing to correct email. Sorry if not.

My questions:
1) Can I use MooseFS as storage for streamong video?
2) Can chunkservers be both cliens?

Thank you for your time!

Kind regards,
Vadim Viktorov, CTO
AWM LAB LTD

Re: [Moosefs-users] some questions about the background of Moosefs

[Alexander A. <akh...@ri...>]

Hi Jofly !
First of all I advise You to have a look at
http://www.moosefs.org/who-is-using-moosefs.html

Alexander

======================================================

Dear Sir,
Thank you for you sparing some time to read my letter.
I am a college student come from china, and my English is not good.I write the letter in order to ask you some questions about the background of Moosefs.

Frist, did the Moosefs originate in the United States and when is it first made publicly ?
Secondly, which company or who developed the software ? 
Thirdly,Is there any famous IT company using MooseFS, and can you tell me some successful cases?
 
Thanks for your time again,and I am looking forward to your reply.

Re: [Moosefs-users] some questions about the background of Moosefs

[Michal B. <mic...@ge...>]

Hi. My responses are below

 

 

From: Jofly [mailto:xh...@16...] 
Sent: Monday, March 21, 2011 2:35 PM
To: co...@mo...; moo...@li...
Subject: [Moosefs-users] some questions about the background of Moosefs
Importance: High

 

Dear Sir,
Thank you for you sparing some time to read my letter.
I am a college student come from china, and my English is not good.I write
the letter in order to ask you some questions about the background of
Moosefs.


Frist, did the Moosefs originate in the United States and when is it first
made publicly ?

[MB] No, MooseFS is developed in Poland, Warsaw. It was open sourced in May
2008.


Secondly, which company or who developed the software ? 

[MB] It's developed by Gemius, a Polish research company with several
branches abroad. Have a look here: http://gemius.com/pl/about_company 


Thirdly,Is there any famous IT company using MooseFS, and can you tell me
some successful cases?

[MB] You may see a list of companies using MooseFS here:
http://www.moosefs.org/who-is-using-moosefs.html 

 

Thanks for your time again,and I am looking forward to your reply.

[MB] You are welcome. You also may find some information about MooseFS in
chinese here: http://sery.blog.51cto.com/10037/263515,
http://blog.formyz.org/?p=299,
http://www.lsanotes.cn/moosefs-step-by-step-tutorial-cn 

 

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

Re: [Moosefs-users] Core Dumped from mfsmount with Autofs

[Michal B. <mic...@ge...>]

Hi!

We made tests on several different operating system and are not able to reproduce your error. Valgrind also doesn't fined anything bad.

Without the patch we have:
root@ubuntu10-64b:~/mfs-1.6.21/mfsmount# valgrind ./mfsmount -f ==1869== Memcheck, a memory error detector ==1869== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==1869== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info ==1869== Command: ./mfsmount -f ==1869== mfsmaster accepted connection with parameters: read-write,restricted_ip,ignore_gid,can_change_quota ; root mapped to root:root ==1869== Invalid free() / delete / delete[]
==1869==    at 0x4C270BD: free (vg_replace_malloc.c:366)
==1869==    by 0x41366C: read_data_term (readdata.c:224)
==1869==    by 0x418CF4: mainloop (main.c:698)
==1869==    by 0x41906B: main (main.c:941)
==1869==  Address 0x67f56e0 is not stack'd, malloc'd or (recently) free'd ==1869==
mfsmount[1869]: master: connection lost (1)
(...)


But after applying the patch we have:
==1947== Memcheck, a memory error detector ==1947== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==1947== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info ==1947== Command: ./mfsmount -f ==1947== mfsmaster accepted connection with parameters: read-write,restricted_ip,ignore_gid,can_change_quota ; root mapped to root:root
mfsmount[1947]: master: connection lost (1)
(...)


If your problem exists only while using mount with autofs, please send your autofs configuration - we'll try to recreate the problem again. You may again check if you applied our patch correctly.

BTW. We strongly not recommend to use the "mfscachefiles" option. It forces to leave files in the cache forever. It's not good. From 1.6.21 its usage will be marked as deprecated and probably from 1.7 we will remove it completely.


Regards
Michal


-----Original Message-----
From: 姜智华 [mailto:fl...@gm...] 
Sent: Wednesday, March 16, 2011 4:32 AM
To: Michal Borychowski
Cc: moo...@li...
Subject: Re: [Moosefs-users] Core Dumped from mfsmount with Autofs

Hi,

I tried the patch but the core file still gets dumped (with mfs 1.6.20)

Core was generated by `mfsmount /home/fwjiang -o
rw,mfscachefiles,mfsentrycacheto=30,mfsattrcacheto=30'.
Program terminated with signal 6, Aborted.
#0  0x00000031c16327f5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install
filesystem-2.4.30-2.fc12.x86_64 fuse-libs-2.8.5-2.fc12.x86_64
glibc-2.11.2-3.x86_64 libgcc-4.4.4-10.fc12.x86_64
(gdb) bt
#0  0x00000031c16327f5 in raise () from /lib64/libc.so.6
#1  0x00000031c1633fd5 in abort () from /lib64/libc.so.6
#2  0x00000031c166fa1b in __libc_message () from /lib64/libc.so.6
#3  0x00000031c1675336 in malloc_printerr () from /lib64/libc.so.6
#4  0x000000000040e4ad in read_data_term () at readdata.c:224
#5  0x00000000004131b5 in mainloop (args=0x7fffc23bf030,
    mp=0xacc290 "/home/fwjiang", mt=1, fg=0) at main.c:600
#6  0x00000000004134f8 in main (argc=<value optimized out>,
    argv=0x7fffc23bf158) at main.c:819

Any clues?

Thanks
Flow

On 3/15/11, Michal Borychowski <mic...@ge...> wrote:
> It'll be fixed in the next release. For the moment you may try this "patch":
>
>
> @@ -178,6 +178,7 @@ void read_data_end(void* rr) {
>         }
>         if (rrec->rbuff!=NULL) {
>                 free(rrec->rbuff);
> +               rrec->rbuff=NULL;
>         }
>
>         pthread_mutex_lock(&glock);
>
>
> Kind regards
> Michal
>
> -----Original Message-----
> From: Flow Jiang [mailto:fl...@gm...]
> Sent: Friday, March 04, 2011 4:21 PM
> To: Michal Borychowski
> Cc: moo...@li...
> Subject: Re: [Moosefs-users] Core Dumped from mfsmount with Autofs
>
> I tried to re-compile mfsmount with the "free(freecblockshead)" line
> commented out. Now our servers (which keep running 7x24) are happy, no
> more core files. However, core files still gets generated on our
> workstations when they reboot. The core is generated from the
> "read_data_term" line right after the "write_data_term" line mentioned
> previously.
>
> Hopefully this will also get fixed in next release, and will even be
> better if I can have a quick solution / patch for the issue.
>
> Thanks
> Flow
>
> On 03/01/2011 11:37 PM, Flow Jiang wrote:
>> Michal,
>>
>> Glad to know that this error could be simply solved by commenting out
>> that line and will try tomorrow to see if it fixes this issue.
>>
>> It does annoying since each core file takes about 170M and I tried to
>> disable the core dump but failed. So hopefully we can have a better
>> solution in the next release.
>>
>> Thanks
>> Flow
>>
>> On 03/01/2011 09:00 PM, Michal Borychowski wrote:
>>> Hi!
>>>
>>> This error is not a serious one. It may happen only upon exits. If these
>>> errors are annoying a quick solution is to comment out the
>>> "free(freecblockshead)" line, recompile mfsmount and run again. We'll
>>> prepare a better solution in the next release.
>>>
>>>
>>> Kind regards
>>> Michał
>
> ------------------------------------------------------------------------------
> What You Don't Know About Data Connectivity CAN Hurt You
> This paper provides an overview of data connectivity, details
> its effect on application quality, and explores various alternative
> solutions. http://p.sf.net/sfu/progress-d2d
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>

------------------------------------------------------------------------------
Colocation vs. Managed Hosting
A question and answer guide to determining the best fit
for your organization - today and in the future.
http://p.sf.net/sfu/internap-sfd2d
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] More nexenta bugs

[Michal B. <mic...@ge...>]

Hi Robert!

 

Yes, we'd like MooseFS to be fully compatible with Nexenta. So if you still
have some issues, please send them to us.

 

We are (almost) sure that hanging up of the chunkserver was not connected
with SIGPIPE signal. To the "csserv_serve" function we added a test
fragment:

 

//sigpipe test

        int fd[2];

        pipe(fd);

        close(fd[0]);

        if (write(fd[1],"",1)<0) {

                mfs_errlog(LOG_NOTICE,"sigppipe test");

        }

        close(fd[1]);        

//sigpipe test

 

and everything worked fine, the logs showed:

 

(...) nexenta mfschunkserver[14100]: [ID 801593 daemon.notice] sigppipe
test: EPIPE (Broken pipe)

(...)

 

 

Is your problem repeatable? Where did you get information about SIGPIPE
from? Have you correctly run gdb (with the correct core)? Can you run "info
threads" in gdb? Maybe it will show us a real cause of the problem.

 

Thank you!

 

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

 

 

From: Robert Dye [mailto:ro...@in...] 
Sent: Thursday, March 17, 2011 7:28 PM
To: moo...@li...
Subject: [Moosefs-users] More nexenta bugs

 

I guess I should ask if I should be submitting Nexenta bugs, if this is
something the project will support in the future?  Either way, looks like I
was able to get specific trace information (shown below).

 

Problem:  mfschunkserver segfaults after running for a few hours.

 

Output:  

[New LWP 5]

main server module: listen on *:9422

[New LWP 6]

[New LWP 7]

[New LWP 8]

[New LWP 9]

[New LWP 10]

[New LWP 11]

[New LWP 12]

[New LWP 13]

[New LWP 14]

[New LWP 15]

[New LWP 16]

[New LWP 17]

[New LWP 18]

[New LWP 19]

[New LWP 20]

[New LWP 21]

[New LWP 22]

[New LWP 23]

[New LWP 24]

[New LWP 25]

stats file has been loaded

mfschunkserver daemon initialized properly

 

 

 

Program received signal SIGPIPE, Broken pipe.

0xfeed57d5 in __write () from /lib/libc.so.1

 

(gdb) bt

#0  0xfeed57d5 in __write () from /lib/libc.so.1

#1  0xfeebf37e in write () from /lib/libc.so.1

#2  0x0804d684 in csserv_write (eptr=0x8a32bf0) at csserv.c:1631

#3  0x0804f604 in csserv_serve (pdesc=0x8033c90) at csserv.c:1900

#4  0x0806208c in mainloop () at ../mfscommon/main.c:348

#5  0x080626fb in main (argc=<value optimized out>, argv=0x40) at
../mfscommon/main.c:1101