Menu
▾
▴
moosefs-users
|
From: Anh K. H. <ky...@vi...> - 2010-08-12 11:09:47
|
Hi, I am a moosefs newbie. I am using EC2 instances and I intended to build a moosefs system to share EBS disks between instances. My question is that: can I use moosefs for logs? My applications (web server, applications) need to write to logs files, but I don't know if there's any performance problem when logs are written to moosefs' disk. Thank you for your helps, Regards, -- Anh Ky Huynh |
|
From: Michał B. <mic...@ge...> - 2010-08-20 10:58:57
|
Hi! You can easily use MooseFS to store any kind of content, including log files. Performance should not be affected. But it depends on the logging mechanism, if for writing every line the file is continuously opened, data appended and the file closed, for sure it is not perfect way to save logs. And you have to remember about one thing. Different clients cannot write at the same moment to the same file located in MooseFS. So if you have several httpd servers, let them save the logs under different filenames (eg. access-192.168.0.1.log, access-192.168.0.2.log, etc.) If you need any further assistance please let us know. Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 -----Original Message----- From: Anh K. Huynh [mailto:ky...@vi...] Sent: Thursday, August 12, 2010 12:21 PM To: moo...@li... Subject: [Moosefs-users] Moosefs for logs Hi, I am a moosefs newbie. I am using EC2 instances and I intended to build a moosefs system to share EBS disks between instances. My question is that: can I use moosefs for logs? My applications (web server, applications) need to write to logs files, but I don't know if there's any performance problem when logs are written to moosefs' disk. Thank you for your helps, Regards, -- Anh Ky Huynh ---------------------------------------------------------------------------- -- This SF.net email is sponsored by Make an app they can't live without Enter the BlackBerry Developer Challenge http://p.sf.net/sfu/RIM-dev2dev _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Anh K. H. <ky...@vi...> - 2010-08-24 07:56:06
|
On Fri, 20 Aug 2010 12:58:40 +0200 Michał Borychowski <mic...@ge...> wrote: > You can easily use MooseFS to store any kind of content, including > log files. Performance should not be affected. But it depends on > the logging mechanism, if for writing every line the file is > continuously opened, data appended and the file closed, for sure it > is not perfect way to save logs. > > And you have to remember about one thing. Different clients cannot > write at the same moment to the same file located in MooseFS. So if > you have several httpd servers, let them save the logs under > different filenames (eg. access-192.168.0.1.log, > access-192.168.0.2.log, etc.) Thank you for your tips. I am installing and testing MFS :) I will use different locations for different MFS clients. Regards, > -----Original Message----- > From: Anh K. Huynh [mailto:ky...@vi...] > Sent: Thursday, August 12, 2010 12:21 PM > To: moo...@li... > Subject: [Moosefs-users] Moosefs for logs > > Hi, > > I am a moosefs newbie. I am using EC2 instances and I intended to > build a moosefs system to share EBS disks between instances. > > My question is that: can I use moosefs for logs? My applications > (web server, applications) need to write to logs files, but I don't > know if there's any performance problem when logs are written to > moosefs' disk. > > Thank you for your helps, > > Regards, > -- Anh Ky Huynh |
|
From: Anh K. H. <ky...@vi...> - 2010-08-25 02:59:41
|
On Fri, 20 Aug 2010 12:58:40 +0200 Michał Borychowski <mic...@ge...> wrote: > You can easily use MooseFS to store any kind of content, including > log files. Performance should not be affected. But it depends on > the logging mechanism, if for writing every line the file is > continuously opened, data appended and the file closed, for sure it > is not perfect way to save logs. Here's my simple test results on AWS EC2 environment: A 'dd' command tries to write about 540MB to a MFS disk, the speed is very low (28MB/s). On the same server with native disk (instant disk of any AWS EC2 instance), the same command yields the speed around 140MB/s. I don't know if I can deploy a database server on any MFS disk ... :( /-------------------------------------------------------------- $ dd if=/dev/zero of=/tmp/mfs_c0/test bs=1024 count=524288 524288+0 records in 524288+0 records out 536870912 bytes (537 MB) copied, 18.9801 s, 28.3 MB/s $ dd if=/dev/zero of=/tmp/test bs=1024 count=524288 524288+0 records in 524288+0 records out 536870912 bytes (537 MB) copied, 3.95007 s, 136 MB/s \-------------------------------------------------------------- > And you have to remember about one thing. Different clients cannot > write at the same moment to the same file located in MooseFS. So if > you have several httpd servers, let them save the logs under > different filenames (eg. access-192.168.0.1.log, > access-192.168.0.2.log, etc.) > > -----Original Message----- > From: Anh K. Huynh [mailto:ky...@vi...] > Sent: Thursday, August 12, 2010 12:21 PM > To: moo...@li... > Subject: [Moosefs-users] Moosefs for logs > > Hi, > > I am a moosefs newbie. I am using EC2 instances and I intended to > build a moosefs system to share EBS disks between instances. > > My question is that: can I use moosefs for logs? My applications > (web server, applications) need to write to logs files, but I don't > know if there's any performance problem when logs are written to > moosefs' disk. > > Thank you for your helps, > > Regards, > -- Anh Ky Huynh |
|
From: Anh K. H. <ky...@vi...> - 2010-08-31 05:10:06
|
On Fri, 20 Aug 2010 12:58:40 +0200
Michał Borychowski <mic...@ge...> wrote:
> You can easily use MooseFS to store any kind of content, including
> log files. Performance should not be affected. But it depends on
> the logging mechanism, if for writing every line the file is
> continuously opened, data appended and the file closed, for sure it
> is not perfect way to save logs.
>
> And you have to remember about one thing. Different clients cannot
> write at the same moment to the same file located in MooseFS. So if
> you have several httpd servers, let them save the logs under
> different filenames (eg. access-192.168.0.1.log,
> access-192.168.0.2.log, etc.)
I share with you a small benchmark result. I set up a default apache server, then run
ab -n 150000 -c <C> http://localhost/ # ab is provided by Apache
to get information about web serving. The '<C>' is the concurrency level (number of requests will be sent at the same time.)
I compared the output of data in two cases
(A) * Default apache setup
* logs file will be written on local disk
(B) * Default apache setup on the same server as (A),
* logs file will be written on a MFS disk (provided by another server.)
Here's the output
(A) http://gx.viettug.org/zen/kyanh/jobs/log_on_root.png.html
(B) http://gx.viettug.org/zen/kyanh/jobs/log_on_mfs.png.html
As you can see, when using MFS for logs, the number of requests per second decreases significantly. This number relies on the red curve of the graphs: the maximum value in (B) is 4500, which is lesser than the lowest value in (A).
Conclusion: MFS isn't suitable for high load logs service (esp. when WRITING is important.)
I don't know about the READ performance. Should make some other tests.
Regards,
> -----Original Message-----
> From: Anh K. Huynh [mailto:ky...@vi...]
> Sent: Thursday, August 12, 2010 12:21 PM
> To: moo...@li...
> Subject: [Moosefs-users] Moosefs for logs
>
> Hi,
>
> I am a moosefs newbie. I am using EC2 instances and I intended to
> build a moosefs system to share EBS disks between instances.
>
> My question is that: can I use moosefs for logs? My applications
> (web server, applications) need to write to logs files, but I don't
> know if there's any performance problem when logs are written to
> moosefs' disk.
>
> Thank you for your helps,
>
> Regards,
>
--
Anh Ky Huynh
|
|
From: Michał B. <mic...@ge...> - 2010-08-26 13:03:52
|
How exactly does AWS EC2 work? Is it a web service? How do you connect to it? What is your Internet bandwidth? Speed 136MB/s is practically not achievable at 1Gbit network. We suggest you generate a random file (from /dev/urandom) and you try to upload it by using normal "cp" or "dd" with giving at if= this file as the parameter. You can also repeat this test with four parallel writes: dd if=/dev/zero of=/tmp/mfs_c0/test_1 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_2 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_3 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_4 bs=1024 count=524288 & You can also check it with a little bigger block eg.: dd if=/dev/zero of=/tmp/mfs_c0/test_1 bs=1048576 count=512. Kind regards Michał -----Original Message----- From: Anh K. Huynh [mailto:ky...@vi...] Sent: Wednesday, August 25, 2010 4:59 AM To: Michał Borychowski Cc: moo...@li... Subject: Re: [Moosefs-users] Moosefs for logs On Fri, 20 Aug 2010 12:58:40 +0200 Michał Borychowski <mic...@ge...> wrote: > You can easily use MooseFS to store any kind of content, including > log files. Performance should not be affected. But it depends on > the logging mechanism, if for writing every line the file is > continuously opened, data appended and the file closed, for sure it > is not perfect way to save logs. Here's my simple test results on AWS EC2 environment: A 'dd' command tries to write about 540MB to a MFS disk, the speed is very low (28MB/s). On the same server with native disk (instant disk of any AWS EC2 instance), the same command yields the speed around 140MB/s. I don't know if I can deploy a database server on any MFS disk ... :( /-------------------------------------------------------------- $ dd if=/dev/zero of=/tmp/mfs_c0/test bs=1024 count=524288 524288+0 records in 524288+0 records out 536870912 bytes (537 MB) copied, 18.9801 s, 28.3 MB/s $ dd if=/dev/zero of=/tmp/test bs=1024 count=524288 524288+0 records in 524288+0 records out 536870912 bytes (537 MB) copied, 3.95007 s, 136 MB/s \-------------------------------------------------------------- > And you have to remember about one thing. Different clients cannot > write at the same moment to the same file located in MooseFS. So if > you have several httpd servers, let them save the logs under > different filenames (eg. access-192.168.0.1.log, > access-192.168.0.2.log, etc.) > > -----Original Message----- > From: Anh K. Huynh [mailto:ky...@vi...] > Sent: Thursday, August 12, 2010 12:21 PM > To: moo...@li... > Subject: [Moosefs-users] Moosefs for logs > > Hi, > > I am a moosefs newbie. I am using EC2 instances and I intended to > build a moosefs system to share EBS disks between instances. > > My question is that: can I use moosefs for logs? My applications > (web server, applications) need to write to logs files, but I don't > know if there's any performance problem when logs are written to > moosefs' disk. > > Thank you for your helps, > > Regards, > -- Anh Ky Huynh ------------------------------------------------------------------------------ Sell apps to millions through the Intel(R) Atom(Tm) Developer Program Be part of this innovative community and reach millions of netbook users worldwide. Take advantage of special opportunities to increase revenue and speed time-to-market. Join now, and jumpstart your future. http://p.sf.net/sfu/intel-atom-d2d _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Alexander A. <akh...@ri...> - 2010-08-26 13:42:15
|
Yes, yes! I agree with Anh Ky Huynh. Write speed to a MFS disk is very low. We use Samba ower MFS and see that reading from samba share has a normal speed and writing as opposed to reading is very slow. In connection with it I have a question. When MFS client writes some file with goal >1 how it is performed ? Does client wait to complete write operation of all file copyes? Or client writes to just one chunk server and than to achieve a goal we rely on chunk replication? wbr Alexander Akhobadze ====================================================== Вы писали 26 августа 2010 г., 17:03:31: ====================================================== How exactly does AWS EC2 work? Is it a web service? How do you connect to it? What is your Internet bandwidth? Speed 136MB/s is practically not achievable at 1Gbit network. We suggest you generate a random file (from /dev/urandom) and you try to upload it by using normal "cp" or "dd" with giving at if= this file as the parameter. You can also repeat this test with four parallel writes: dd if=/dev/zero of=/tmp/mfs_c0/test_1 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_2 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_3 bs=1024 count=524288 & dd if=/dev/zero of=/tmp/mfs_c0/test_4 bs=1024 count=524288 & You can also check it with a little bigger block eg.: dd if=/dev/zero of=/tmp/mfs_c0/test_1 bs=1048576 count=512. Kind regards Michał -----Original Message----- From: Anh K. Huynh [mailto:ky...@vi...] Sent: Wednesday, August 25, 2010 4:59 AM To: Michał Borychowski Cc: moo...@li... Subject: Re: [Moosefs-users] Moosefs for logs On Fri, 20 Aug 2010 12:58:40 +0200 Michał Borychowski <mic...@ge...> wrote: > You can easily use MooseFS to store any kind of content, including > log files. Performance should not be affected. But it depends on > the logging mechanism, if for writing every line the file is > continuously opened, data appended and the file closed, for sure it > is not perfect way to save logs. Here's my simple test results on AWS EC2 environment: A 'dd' command tries to write about 540MB to a MFS disk, the speed is very low (28MB/s). On the same server with native disk (instant disk of any AWS EC2 instance), the same command yields the speed around 140MB/s. I don't know if I can deploy a database server on any MFS disk ... /-------------------------------------------------------------- $ dd if=/dev/zero of=/tmp/mfs_c0/test bs=1024 count=524288 524288+0 records in 524288+0 records out 536870912 bytes (537 MB) copied, 18.9801 s, 28.3 MB/s $ dd if=/dev/zero of=/tmp/test bs=1024 count=524288 524288+0 records in 524288+0 records out 536870912 bytes (537 MB) copied, 3.95007 s, 136 MB/s \-------------------------------------------------------------- > And you have to remember about one thing. Different clients cannot > write at the same moment to the same file located in MooseFS. So if > you have several httpd servers, let them save the logs under > different filenames (eg. access-192.168.0.1.log, > access-192.168.0.2.log, etc.) > > -----Original Message----- > From: Anh K. Huynh [mailto:ky...@vi...] > Sent: Thursday, August 12, 2010 12:21 PM > To: moo...@li... > Subject: [Moosefs-users] Moosefs for logs > > Hi, > > I am a moosefs newbie. I am using EC2 instances and I intended to > build a moosefs system to share EBS disks between instances. > > My question is that: can I use moosefs for logs? My applications > (web server, applications) need to write to logs files, but I don't > know if there's any performance problem when logs are written to > moosefs' disk. > > Thank you for your helps, > > Regards, |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X