Re: [Moosefs-users] Moosefs for logs

[Alexander A. <akh...@ri...>]

Yes, yes! I agree with Anh Ky Huynh.
Write speed to a MFS disk is very low.
We use Samba ower MFS and see that reading from
samba share has a normal speed and writing
as opposed to reading is very slow.

In connection with it I have a question.
When MFS client writes some file with goal >1
how it is performed ? Does client wait to complete
write operation of all file copyes?
Or client writes to just one chunk server
and than to achieve a goal we rely on chunk replication?

wbr
Alexander Akhobadze

======================================================
Вы писали 26 августа 2010 г., 17:03:31:
======================================================

How exactly does AWS EC2 work? Is it a web service? How do you connect to it? What is your Internet bandwidth? Speed 136MB/s is practically not achievable at 1Gbit network.

We suggest you generate a random file (from /dev/urandom) and you try to upload it by using normal "cp" or "dd" with giving at if= this file as the parameter. 

You can also repeat this test with four parallel writes:
dd if=/dev/zero of=/tmp/mfs_c0/test_1 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_2 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_3 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_4 bs=1024 count=524288 &

You can also check it with a little bigger block eg.: 
dd if=/dev/zero of=/tmp/mfs_c0/test_1 bs=1048576 count=512.


Kind regards
Michał 


-----Original Message-----
From: Anh K. Huynh [mailto:ky...@vi...] 
Sent: Wednesday, August 25, 2010 4:59 AM
To: Michał Borychowski
Cc: moo...@li...
Subject: Re: [Moosefs-users] Moosefs for logs

On Fri, 20 Aug 2010 12:58:40 +0200
Michał Borychowski <mic...@ge...> wrote:

> You can easily use MooseFS to store any kind of content, including
> log files. Performance should not be affected. But it depends on
> the logging mechanism, if for writing every line the file is
> continuously opened, data appended and the file closed, for sure it
> is not perfect way to save logs. 

Here's my simple test results on AWS EC2 environment: A 'dd' command tries to write about 540MB to a MFS disk, the speed is very low (28MB/s). On the same server with native disk (instant disk of any AWS EC2 instance), the same command yields the speed around 140MB/s.

I don't know if I can deploy a database server on any MFS disk ...  

/--------------------------------------------------------------
$ dd if=/dev/zero of=/tmp/mfs_c0/test bs=1024 count=524288

524288+0 records in
524288+0 records out
536870912 bytes (537 MB) copied, 18.9801 s, 28.3 MB/s

$ dd if=/dev/zero of=/tmp/test bs=1024 count=524288
524288+0 records in
524288+0 records out
536870912 bytes (537 MB) copied, 3.95007 s, 136 MB/s
\--------------------------------------------------------------


> And you have to remember about one thing. Different clients cannot
> write at the same moment to the same file located in MooseFS. So if
> you have several httpd servers, let them save the logs under
> different filenames (eg. access-192.168.0.1.log,
> access-192.168.0.2.log, etc.)


> 
> -----Original Message-----
> From: Anh K. Huynh [mailto:ky...@vi...] 
> Sent: Thursday, August 12, 2010 12:21 PM
> To: moo...@li...
> Subject: [Moosefs-users] Moosefs for logs
> 
> Hi,
> 
> I am a moosefs newbie. I am using EC2 instances and I intended to
> build a moosefs system to share EBS disks between instances.
> 
> My question is that: can I use moosefs for logs? My applications
> (web server, applications) need to write to logs files, but I don't
> know if there's any performance problem when logs are written to
> moosefs' disk.
> 
> Thank you for your helps,
> 
> Regards,