Menu
▾
▴
moosefs-users
|
From: Richard C. <ric...@ai...> - 2011-05-13 14:39:49
|
Hello MFS Devs, I am evaluating MooseFS for use in various aspects of our business, and one of them is in use as a storage mechanism for virtual machines. Specifically, we're using proxmox for virtualization in our environment, and we seem to be hitting a snag when using KVM containers -- the virtual machines won't start. We currently believe that this may be because MFS (or FUSE, possibly) does not allow mmap'ing of files. I am wondering if anyone can shed some light on this type of situation (using MFS for storage of KVM virtual machines) and/or if anyone has any known or possibly unknown issues with mmap'ing files on MFS. Thanks, Richard. |
|
From: Thomas S H. <tha...@gm...> - 2011-05-13 14:50:19
|
I am not familiar with using proxmox for managing KVM, I have always used libvirt. When running KVM virtual machines with the storage on MooseFS I have had no problems, I have used QEMU 0.12 through 0.14. Can you manually start the KVM virtual machines via the qemu-kvm command? How does proxmox manage/run the virtual machines? At the end of the day KVM should have no problems running virtual machines on MooseFS, I have been doing it for quite some time. What is your underlying OS for your hypervisors and Moosefs chunkservers? What version of MooseFS are you using? -Thomas S Hatch On Fri, May 13, 2011 at 8:24 AM, Richard Chute <ric...@ai...>wrote: > Hello MFS Devs, > I am evaluating MooseFS for use in various aspects of our business, > and one of them is in use as a storage mechanism for virtual machines. > Specifically, we're using proxmox for virtualization in our environment, > and we seem to be hitting a snag when using KVM containers -- the > virtual machines won't start. We currently believe that this may be > because MFS (or FUSE, possibly) does not allow mmap'ing of files. > I am wondering if anyone can shed some light on this type of > situation (using MFS for storage of KVM virtual machines) and/or if > anyone has any known or possibly unknown issues with mmap'ing files on MFS. > > Thanks, > Richard. > > > ------------------------------------------------------------------------------ > Achieve unprecedented app performance and reliability > What every C/C++ and Fortran developer should know. > Learn how Intel has extended the reach of its next-generation tools > to help boost performance applications - inlcuding clusters. > http://p.sf.net/sfu/intel-dev2devmay > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
|
From: Ólafur Ó. <osv...@ne...> - 2011-05-13 15:24:33
|
Hi, Not sure if this is the same problem as yours, but we are running a decent Xen setup with MFS as the storage for the VM's, it works very well and the only problems we have had was when trying to use directio, if that is disabled it works fine. /Oli On 13.5.2011, at 14:24, Richard Chute wrote: > Hello MFS Devs, > I am evaluating MooseFS for use in various aspects of our business, > and one of them is in use as a storage mechanism for virtual machines. > Specifically, we're using proxmox for virtualization in our environment, > and we seem to be hitting a snag when using KVM containers -- the > virtual machines won't start. We currently believe that this may be > because MFS (or FUSE, possibly) does not allow mmap'ing of files. > I am wondering if anyone can shed some light on this type of > situation (using MFS for storage of KVM virtual machines) and/or if > anyone has any known or possibly unknown issues with mmap'ing files on MFS. > > Thanks, > Richard. > > ------------------------------------------------------------------------------ > Achieve unprecedented app performance and reliability > What every C/C++ and Fortran developer should know. > Learn how Intel has extended the reach of its next-generation tools > to help boost performance applications - inlcuding clusters. > http://p.sf.net/sfu/intel-dev2devmay > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users -- Ólafur Osvaldsson System Administrator Nethonnun ehf. e-mail: osv...@ne... phone: +354 517 3400 |
|
From: Kristofer P. <kri...@cy...> - 2011-05-13 17:31:20
|
Just curious - how has your performance been with that? Have you ran any type of performance benchmarks from within any of the domU's? ----- Original Message ----- From: "Ólafur Ósvaldsson" <osv...@ne...> To: "Richard Chute" <ric...@ai...> Cc: moo...@li... Sent: Friday, May 13, 2011 10:05:54 AM Subject: Re: [Moosefs-users] KVM on MFS Hi, Not sure if this is the same problem as yours, but we are running a decent Xen setup with MFS as the storage for the VM's, it works very well and the only problems we have had was when trying to use directio, if that is disabled it works fine. /Oli On 13.5.2011, at 14:24, Richard Chute wrote: > Hello MFS Devs, > I am evaluating MooseFS for use in various aspects of our business, > and one of them is in use as a storage mechanism for virtual machines. > Specifically, we're using proxmox for virtualization in our environment, > and we seem to be hitting a snag when using KVM containers -- the > virtual machines won't start. We currently believe that this may be > because MFS (or FUSE, possibly) does not allow mmap'ing of files. > I am wondering if anyone can shed some light on this type of > situation (using MFS for storage of KVM virtual machines) and/or if > anyone has any known or possibly unknown issues with mmap'ing files on MFS. > > Thanks, > Richard. > > ------------------------------------------------------------------------------ > Achieve unprecedented app performance and reliability > What every C/C++ and Fortran developer should know. > Learn how Intel has extended the reach of its next-generation tools > to help boost performance applications - inlcuding clusters. > http://p.sf.net/sfu/intel-dev2devmay > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users -- Ólafur Osvaldsson System Administrator Nethonnun ehf. e-mail: osv...@ne... phone: +354 517 3400 ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Ólafur Ó. <osv...@ne...> - 2011-05-24 10:00:37
|
Hi, We have not done extensive testing within the domU's, we did before deploying do tests on the dom0's to compare against the SAN and iSCSI setups we have and found the speed was acceptable in comparison to those. /Oli On 13.5.2011, at 17:15, Kristofer Pettijohn wrote: > Just curious - how has your performance been with that? Have you ran any type of performance benchmarks from within any of the domU's? > > ----- Original Message ----- > From: "Ólafur Ósvaldsson" <osv...@ne...> > To: "Richard Chute" <ric...@ai...> > Cc: moo...@li... > Sent: Friday, May 13, 2011 10:05:54 AM > Subject: Re: [Moosefs-users] KVM on MFS > > Hi, > Not sure if this is the same problem as yours, but we are running a decent Xen setup with MFS as the storage for the VM's, it works very well and the only problems we have had was when trying to use directio, if that is disabled it works fine. > > /Oli > > On 13.5.2011, at 14:24, Richard Chute wrote: > >> Hello MFS Devs, >> I am evaluating MooseFS for use in various aspects of our business, >> and one of them is in use as a storage mechanism for virtual machines. >> Specifically, we're using proxmox for virtualization in our environment, >> and we seem to be hitting a snag when using KVM containers -- the >> virtual machines won't start. We currently believe that this may be >> because MFS (or FUSE, possibly) does not allow mmap'ing of files. >> I am wondering if anyone can shed some light on this type of >> situation (using MFS for storage of KVM virtual machines) and/or if >> anyone has any known or possibly unknown issues with mmap'ing files on MFS. >> >> Thanks, >> Richard. >> >> ------------------------------------------------------------------------------ >> Achieve unprecedented app performance and reliability >> What every C/C++ and Fortran developer should know. >> Learn how Intel has extended the reach of its next-generation tools >> to help boost performance applications - inlcuding clusters. >> http://p.sf.net/sfu/intel-dev2devmay >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users > > -- > Ólafur Osvaldsson > System Administrator > Nethonnun ehf. > e-mail: osv...@ne... > phone: +354 517 3400 > > > ------------------------------------------------------------------------------ > Achieve unprecedented app performance and reliability > What every C/C++ and Fortran developer should know. > Learn how Intel has extended the reach of its next-generation tools > to help boost performance applications - inlcuding clusters. > http://p.sf.net/sfu/intel-dev2devmay > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users -- Ólafur Osvaldsson System Administrator Nethonnun ehf. e-mail: osv...@ne... phone: +354 517 3400 |
|
From: Giovanni T. <gt...@li...> - 2011-05-13 15:46:05
Attachments:
signature.asc
|
Hi Richard, On 13/05/2011 16:24, Richard Chute wrote: > Specifically, we're using proxmox for virtualization in our environment, > and we seem to be hitting a snag when using KVM containers -- the > virtual machines won't start. We currently believe that this may be > because MFS (or FUSE, possibly) does not allow mmap'ing of files. > I am wondering if anyone can shed some light on this type of > situation (using MFS for storage of KVM virtual machines) and/or if > anyone has any known or possibly unknown issues with mmap'ing files on MFS. I am managing a Proxmox cluster of 2 nodes with raw disk images on a MFS volume, and I haven't any problem with it. Actually I am mounting the mfs volume via an /etc/network/if-up.d/moosefs-mount.sh, that mount the volume under /srv/mfs, and on Proxmox I configured a new storage volume for virtual images, that's all. Bye. -- Giovanni Toraldo http://www.libersoft.it/ |
|
From: Richard C. <ric...@ai...> - 2011-05-13 17:26:39
|
Hi Giovanni,
Thank you for the encouragement... at least I know someone out there
has done this successfully.
We're running Proxmox VE 1.8, and this is the exact error message
we're getting:
5385 open("/mfs/pve/images/101/vm-101-disk-1.raw",
O_RDWR|O_DIRECT|O_CLOEXEC) = -1 EINVAL (Invalid argument)
What version of proxmox are you running?
Thanks,
Richard.
On 11-05-13 12:45 PM, Giovanni Toraldo wrote:
> Hi Richard,
>
> On 13/05/2011 16:24, Richard Chute wrote:
>> Specifically, we're using proxmox for virtualization in our environment,
>> and we seem to be hitting a snag when using KVM containers -- the
>> virtual machines won't start. We currently believe that this may be
>> because MFS (or FUSE, possibly) does not allow mmap'ing of files.
>> I am wondering if anyone can shed some light on this type of
>> situation (using MFS for storage of KVM virtual machines) and/or if
>> anyone has any known or possibly unknown issues with mmap'ing files on MFS.
> I am managing a Proxmox cluster of 2 nodes with raw disk images on a MFS
> volume, and I haven't any problem with it.
>
> Actually I am mounting the mfs volume via an
> /etc/network/if-up.d/moosefs-mount.sh, that mount the volume under
> /srv/mfs, and on Proxmox I configured a new storage volume for virtual
> images, that's all.
>
> Bye.
>
|
|
From: Giovanni T. <gt...@li...> - 2011-05-14 10:22:59
|
On Fri, May 13, 2011 at 7:26 PM, Richard Chute <ric...@ai...> wrote:
> 5385 open("/mfs/pve/images/101/vm-101-disk-1.raw",
> O_RDWR|O_DIRECT|O_CLOEXEC) = -1 EINVAL (Invalid argument)
Which command do you use to mount on /mfs?
> What version of proxmox are you running?
Proxmox VE 1.8 with 2.6.32 kernel
|
|
From: Richard C. <ric...@ai...> - 2011-05-14 19:35:03
|
> Which command do you use to mount on /mfs? > mfsmount -H 10.17.1.220 -S proxmox /mfs/pve Are there options that I should enable for mfsmount which would enable my use case? Are there options for FUSE which might be a factor as well? Thanks, Richard. |
|
From: Giovanni T. <me...@gi...> - 2011-05-25 16:21:02
Attachments:
signature.asc
|
Hi Richard,
On 13/05/2011 19:26, Richard Chute wrote:
> 5385 open("/mfs/pve/images/101/vm-101-disk-1.raw",
> O_RDWR|O_DIRECT|O_CLOEXEC) = -1 EINVAL (Invalid argument)
I was wrong, I am using proxmox 1.7 with MooseFS. Today, after trying to
ugrade to 1.8, I get the same error. :(
There is probably a regression (or some conflicts with MooseFS) on one
of the latest proxmox updated packages:
[AGGIORNATO] ksm-control-daemon 1.0-4 -> 1.0-5
[AGGIORNATO] libpve-storage-perl 1.0-16 -> 1.0-17
[AGGIORNATO] proxmox-ve-2.6.32 1.7-30 -> 1.8-33
[AGGIORNATO] pve-firmware 1.0-10 -> 1.0-11
[AGGIORNATO] pve-kernel-2.6.32-4-pve 2.6.32-30 -> 2.6.32-33
[AGGIORNATO] pve-manager 1.7-11 -> 1.8-17
[AGGIORNATO] pve-qemu-kvm 0.13.0-3 -> 0.14.0-3
[AGGIORNATO] qemu-server 1.1-28 -> 1.1-30
[AGGIORNATO] vzctl 3.0.24-1pve4 -> 3.0.26-1pve4
[AGGIORNATO] vzdump 1.2-10 -> 1.2-12
[AGGIORNATO] vzprocps 2.0.11-1dso2 -> 2.0.11-2
The only thing I tried, it's to manually downgrade pve-qemu-kvm to 0.13,
but nohting changed. After, I removed proxmox, modified my sources.list
to point to pve1.7 version, and reinstall, and everything was working again.
I'm cc-ing to pve mailing list too.
Regards,
--
Giovanni Toraldo
http://gionn.net/
|
|
From: Łukasz M. <l.m...@gm...> - 2011-05-25 15:57:16
|
Dnia środa, 25 maja 2011 o 17:51:37 Giovanni Toraldo napisał(a):
> Hi Richard,
>
> On 13/05/2011 19:26, Richard Chute wrote:
> > 5385 open("/mfs/pve/images/101/vm-101-disk-1.raw",
> > O_RDWR|O_DIRECT|O_CLOEXEC) = -1 EINVAL (Invalid argument)
>
> I was wrong, I am using proxmox 1.7 with MooseFS. Today, after trying to
> ugrade to 1.8, I get the same error. :(
>
> There is probably a regression (or some conflicts with MooseFS) on one
> of the latest proxmox updated packages:
>
> [AGGIORNATO] ksm-control-daemon 1.0-4 -> 1.0-5
> [AGGIORNATO] libpve-storage-perl 1.0-16 -> 1.0-17
> [AGGIORNATO] proxmox-ve-2.6.32 1.7-30 -> 1.8-33
> [AGGIORNATO] pve-firmware 1.0-10 -> 1.0-11
> [AGGIORNATO] pve-kernel-2.6.32-4-pve 2.6.32-30 -> 2.6.32-33
> [AGGIORNATO] pve-manager 1.7-11 -> 1.8-17
> [AGGIORNATO] pve-qemu-kvm 0.13.0-3 -> 0.14.0-3
> [AGGIORNATO] qemu-server 1.1-28 -> 1.1-30
> [AGGIORNATO] vzctl 3.0.24-1pve4 -> 3.0.26-1pve4
> [AGGIORNATO] vzdump 1.2-10 -> 1.2-12
> [AGGIORNATO] vzprocps 2.0.11-1dso2 -> 2.0.11-2
>
> The only thing I tried, it's to manually downgrade pve-qemu-kvm to 0.13,
> but nohting changed. After, I removed proxmox, modified my sources.list
> to point to pve1.7 version, and reinstall, and everything was working
> again.
>
> I'm cc-ing to pve mailing list too.
>
> Regards,
MFS is fuse based and fuse does not support O_DIRECT which is default since
1.8, You need to add cache=writeback to disk defined in /etc/qemu-
server/$VID.conf
example:
virtio0: MFS_disks:101/vm-101-disk-1.raw,cache=writeback
or
ide0: MFS_disks:101/vm-101-disk-1.raw,cache=writeback
Łukasz Mierzwa
|
|
From: Dietmar M. <di...@pr...> - 2011-05-26 03:49:20
|
>
> On 13/05/2011 19:26, Richard Chute wrote:
> > 5385 open("/mfs/pve/images/101/vm-101-disk-1.raw",
> > O_RDWR|O_DIRECT|O_CLOEXEC) = -1 EINVAL (Invalid argument)
>
> I was wrong, I am using proxmox 1.7 with MooseFS. Today, after trying to
> ugrade to 1.8, I get the same error. :(
>
> There is probably a regression (or some conflicts with MooseFS) on one of the
> latest proxmox updated packages:
Seems Moosefs does not support O_DIRECT. Try to use another cache setting for your disk.
Edit the VM config, set 'cache=writethrough' for all disks (see 'man qm')
PVE 1.8 uses "cache=none" by default.
- Dietmar
|
|
From: Upendra M. <upe...@he...> - 2011-05-27 12:04:10
|
Hi
Moosefs has problem if cache=none.
We can give either writeback or writethrough.
It also has issue if the disk is virtio.
Performance of IO becomes very very low when compared to ide bus.
I am using qemu 0.14 and mfs 1.6.20 on ubuntu 10.04.
On Thu, May 26, 2011 at 9:01 AM, Dietmar Maurer <di...@pr...> wrote:
> >
> > On 13/05/2011 19:26, Richard Chute wrote:
> > > 5385 open("/mfs/pve/images/101/vm-101-disk-1.raw",
> > > O_RDWR|O_DIRECT|O_CLOEXEC) = -1 EINVAL (Invalid argument)
> >
> > I was wrong, I am using proxmox 1.7 with MooseFS. Today, after trying to
> > ugrade to 1.8, I get the same error. :(
> >
> > There is probably a regression (or some conflicts with MooseFS) on one of
> the
> > latest proxmox updated packages:
>
> Seems Moosefs does not support O_DIRECT. Try to use another cache setting
> for your disk.
> Edit the VM config, set 'cache=writethrough' for all disks (see 'man qm')
>
> PVE 1.8 uses "cache=none" by default.
>
> - Dietmar
>
>
>
> ------------------------------------------------------------------------------
> vRanger cuts backup time in half-while increasing security.
> With the market-leading solution for virtual backup and recovery,
> you get blazing-fast, flexible, and affordable data protection.
> Download your free trial now.
> http://p.sf.net/sfu/quest-d2dcopy1
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
--
Thanks and Regards,
Upendra.M
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X