Re: [MooseFS-Users] weird disk stats

[Krzysztof K. <krz...@mo...>]

Hello Alexander,

MooseFS reports disk transfers based on time it takes to complete the write operation (write() system call) for the chunkserver process. Since MooseFS does not use direct I/O, all writes are using local filesystem cache (buffer cache on chunkserver).

One could say, that mfscgi/mfscli are reporting time/transfer to physical device through buffer cache. If your system is busy, that you might see drops in transfer rates due to the fact, that system needs to do buffer cache flushes more often. 

You can observe the same behaviour with simple dd command test. Let’s simulate chunk write (64MiB using block size of 64KiB) operation using dd command:

[root@pg2 home]# sync ; sync
[root@pg2 home]# time dd if=/dev/zero of=test01.dat bs=64k count=1024
1024+0 records in
1024+0 records out
67108864 bytes (67 MB) copied, 0.0295609 s, 2.3 GB/s

real	0m0.030s
user	0m0.001s
sys	0m0.029s
 
The time (0.030 s), and in consequence, the transfer rate (2.3GB/s) that you get is from the Linux buffer cache. 

To calculate the actual transfer to the disk, you should also take into account the time it takes to do the sync operation, which flushes dirty pages from buffer cache to the disk (Please note, that I’ve performed sync command before dd, to flush any dirty pages already in buffer cache, so to have as clean buffer cache as possible before dd command)

[root@pg2 home]# time sync

real	0m0.508s
user	0m0.001s
sys	0m0.012s

As you can see to complete sync system needed additional 0.508 s for actual transfer from buffer cache to physical device. If you do the math, you’ll get the transfer rate of ~118 MiB/s

[root@pg2 home]# bc
bc 1.06.95
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
scale=4
67108864/(0.030 + 0.508)
124737665.4275
./(1024*1024)
118.9591

I hope that explanation was clear enough, but in case of further questions please don’t hesitate to contact us.

Best Regards,
Krzysztof Kielak
Moosefs.com | Director of Operations and Customer Support
Mobile: +48 601 476 440

> On 10 Jun 2016, at 02:07, Alexander Lazarev <gu...@gm...> wrote:
> 
> Hello everyone!
> 
> I’m getting weird disk statistics(write transfer) in web monitor(cgiserv). It shows like from 500 to 900 MiB/s per single sata disk. Am I doing/understanding something wrong or this stats are incorrect?
> Cheers.
> Alexander Lazarev.
> ------------------------------------------------------------------------------
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity 
> planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
> _________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users