Re: [Moosefs-users] Moosefs for logs

[Michał B. <mic...@ge...>]

How exactly does AWS EC2 work? Is it a web service? How do you connect to it? What is your Internet bandwidth? Speed 136MB/s is practically not achievable at 1Gbit network.

We suggest you generate a random file (from /dev/urandom) and you try to upload it by using normal "cp" or "dd" with giving at if= this file as the parameter. 

You can also repeat this test with four parallel writes:
dd if=/dev/zero of=/tmp/mfs_c0/test_1 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_2 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_3 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_4 bs=1024 count=524288 &

You can also check it with a little bigger block eg.: 
dd if=/dev/zero of=/tmp/mfs_c0/test_1 bs=1048576 count=512.


Kind regards
Michał 


-----Original Message-----
From: Anh K. Huynh [mailto:ky...@vi...] 
Sent: Wednesday, August 25, 2010 4:59 AM
To: Michał Borychowski
Cc: moo...@li...
Subject: Re: [Moosefs-users] Moosefs for logs

On Fri, 20 Aug 2010 12:58:40 +0200
Michał Borychowski <mic...@ge...> wrote:

> You can easily use MooseFS to store any kind of content, including
> log files. Performance should not be affected. But it depends on
> the logging mechanism, if for writing every line the file is
> continuously opened, data appended and the file closed, for sure it
> is not perfect way to save logs. 

Here's my simple test results on AWS EC2 environment: A 'dd' command tries to write about 540MB to a MFS disk, the speed is very low (28MB/s). On the same server with native disk (instant disk of any AWS EC2 instance), the same command yields the speed around 140MB/s.

I don't know if I can deploy a database server on any MFS disk ... :(

/--------------------------------------------------------------
$ dd if=/dev/zero of=/tmp/mfs_c0/test bs=1024 count=524288

524288+0 records in
524288+0 records out
536870912 bytes (537 MB) copied, 18.9801 s, 28.3 MB/s

$ dd if=/dev/zero of=/tmp/test bs=1024 count=524288
524288+0 records in
524288+0 records out
536870912 bytes (537 MB) copied, 3.95007 s, 136 MB/s
\--------------------------------------------------------------


> And you have to remember about one thing. Different clients cannot
> write at the same moment to the same file located in MooseFS. So if
> you have several httpd servers, let them save the logs under
> different filenames (eg. access-192.168.0.1.log,
> access-192.168.0.2.log, etc.)


> 
> -----Original Message-----
> From: Anh K. Huynh [mailto:ky...@vi...] 
> Sent: Thursday, August 12, 2010 12:21 PM
> To: moo...@li...
> Subject: [Moosefs-users] Moosefs for logs
> 
> Hi,
> 
> I am a moosefs newbie. I am using EC2 instances and I intended to
> build a moosefs system to share EBS disks between instances.
> 
> My question is that: can I use moosefs for logs? My applications
> (web server, applications) need to write to logs files, but I don't
> know if there's any performance problem when logs are written to
> moosefs' disk.
> 
> Thank you for your helps,
> 
> Regards,
> 


-- 
Anh Ky Huynh

------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users 
worldwide. Take advantage of special opportunities to increase revenue and 
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users