Menu
▾
▴
moosefs-users — Mailing list for MooseFS Users
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(4) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
(20) |
Feb
(11) |
Mar
(11) |
Apr
(9) |
May
(22) |
Jun
(85) |
Jul
(94) |
Aug
(80) |
Sep
(72) |
Oct
(64) |
Nov
(69) |
Dec
(89) |
| 2011 |
Jan
(72) |
Feb
(109) |
Mar
(116) |
Apr
(117) |
May
(117) |
Jun
(102) |
Jul
(91) |
Aug
(72) |
Sep
(51) |
Oct
(41) |
Nov
(55) |
Dec
(74) |
| 2012 |
Jan
(45) |
Feb
(77) |
Mar
(99) |
Apr
(113) |
May
(132) |
Jun
(75) |
Jul
(70) |
Aug
(58) |
Sep
(58) |
Oct
(37) |
Nov
(51) |
Dec
(15) |
| 2013 |
Jan
(28) |
Feb
(16) |
Mar
(25) |
Apr
(38) |
May
(23) |
Jun
(39) |
Jul
(42) |
Aug
(19) |
Sep
(41) |
Oct
(31) |
Nov
(18) |
Dec
(18) |
| 2014 |
Jan
(17) |
Feb
(19) |
Mar
(39) |
Apr
(16) |
May
(10) |
Jun
(13) |
Jul
(17) |
Aug
(13) |
Sep
(8) |
Oct
(53) |
Nov
(23) |
Dec
(7) |
| 2015 |
Jan
(35) |
Feb
(13) |
Mar
(14) |
Apr
(56) |
May
(8) |
Jun
(18) |
Jul
(26) |
Aug
(33) |
Sep
(40) |
Oct
(37) |
Nov
(24) |
Dec
(20) |
| 2016 |
Jan
(38) |
Feb
(20) |
Mar
(25) |
Apr
(14) |
May
(6) |
Jun
(36) |
Jul
(27) |
Aug
(19) |
Sep
(36) |
Oct
(24) |
Nov
(15) |
Dec
(16) |
| 2017 |
Jan
(8) |
Feb
(13) |
Mar
(17) |
Apr
(20) |
May
(28) |
Jun
(10) |
Jul
(20) |
Aug
(3) |
Sep
(18) |
Oct
(8) |
Nov
|
Dec
(5) |
| 2018 |
Jan
(15) |
Feb
(9) |
Mar
(12) |
Apr
(7) |
May
(123) |
Jun
(41) |
Jul
|
Aug
(14) |
Sep
|
Oct
(15) |
Nov
|
Dec
(7) |
| 2019 |
Jan
(2) |
Feb
(9) |
Mar
(2) |
Apr
(9) |
May
|
Jun
|
Jul
(2) |
Aug
|
Sep
(6) |
Oct
(1) |
Nov
(12) |
Dec
(2) |
| 2020 |
Jan
(2) |
Feb
|
Mar
|
Apr
(3) |
May
|
Jun
(4) |
Jul
(4) |
Aug
(1) |
Sep
(18) |
Oct
(2) |
Nov
|
Dec
|
| 2021 |
Jan
|
Feb
(3) |
Mar
|
Apr
|
May
|
Jun
|
Jul
(6) |
Aug
|
Sep
(5) |
Oct
(5) |
Nov
(3) |
Dec
|
| 2022 |
Jan
|
Feb
|
Mar
(3) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Elliot F. <efi...@gm...> - 2012-05-02 06:20:17
|
We use MooseFS for KVM VMs. We use virtio for Linux guests and IDE for FreeBSD. Have to use cache=writethrough for both. On Wed, Apr 25, 2012 at 2:31 AM, Deon Cui <deo...@gm...> wrote: > Hi, > > I am a bit of a new moose user, currently using moose as a simple smb share > for backup storage. It does this job well and I have no complaints. We're > currently running SL6.2 master servers and freebsd 9 zfs pools for chunk > servers. Performance is not an issue because bulk data and consistency is > more important in this use case. > > However I am looking into creating a unified storage space for KVM virtual > machines in a separate moosefs cluster. I've tried to do my research and one > topic which has been mentioned but not elaborated on is the compatibility > with the virtio driver? > When using moose with kvm does the IDE driver provide better performance > than virtio? > > Another issue I have seen is related to fuse and it not previously > supporting directio, which prevented kvm guests using cache=none from > starting. Are these two issues related? > > If they are related then I have found information that O_DIRECT support is > coming to fuse soon, when it does will this issue be resolved? > > Thanks > Deon > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
|
From: Ken <ken...@gm...> - 2012-05-02 02:32:52
|
hi, I am quite sure about mfschunkserver crash because of: http://sourceforge.net/mailarchive/message.php?msg_id=29149150 >> How do I figure out which file(s) are in that chunk for example? maybe find mount-point -exec mfsfileinfo {} \; help >> I assume repairing it would be done with mfsfilerepair? about the log: 0000000000000AA5_00000005 - invalid copy on (10.0.0.1 - ver:00000003) I think mfsfilerepair maybe work, maybe early version file helped. I am not official, if there is any mistake please correct me. -Ken On Sun, Apr 29, 2012 at 4:26 PM, Jens Kristian Søgaard <je...@me...> wrote: > Hi again, > >> I received this error message: >> backups: master query: receive error > > A bit more happened after I wrote this message. First I noticed that all > writes to files on the mfs mount hanged. I had to force unmount the file > system, but even after remounting mfs writes would still not come through. > > In the end I had to force unmount the filesystem everywhere, and then > close down the chunkservers, the metalogger and the master. When I > closed down the chunkservers with "mfschunkserver stop", I got the > following log message: > > Apr 29 00:23:16 localhost kernel: [284815.544120] mfschunkserver[2229]: > segfault at 0 ip 0000000000411997 sp 00007fff1429b598 error 4 in > mfschunkserver[400000+2a000] > > I got the exact same error message on all chunkservers with only the > stack pointer different in each one. > > After starting the master, metalogger and chunkservers again I was > greeted with some bad news. The log shows messages like this: > > Apr 29 05:47:45 localhost mfsmaster[5036]: chunk 0000000000000AA5 has > only invalid copies (2) - please repair it manually > Apr 29 05:47:45 localhost mfsmaster[5036]: chunk > 0000000000000AA5_00000005 - invalid copy on (10.0.0.1 - ver:00000003) > Apr 29 05:47:45 localhost mfsmaster[5036]: chunk > 0000000000000AA5_00000005 - invalid copy on (10.0.0.2 - ver:00000004) > > How do I figure out which file(s) are in that chunk for example? > > I assume repairing it would be done with mfsfilerepair? > > > On the web interface is now displayed under "Important messages": > > unavailable chunks: 4543067 > unavailable files: 4749936 > > How can I find out which files are affected? > > Weirdly in the "Regular chunks state matrix" in the web interface only 6 > chunks show as having 0 valid copies. > > I know that nothing happened to the disks backing the chunk servers at > any point (they're RAID-5 arrays). So it sounds weird to me that 6 > blocks went missing? > > The web interface lists 14 file names under "Import messages" as > "currently unavailable file". But if I check them with mfsfileinfo and > mfscheckfile - they look fine? > > I hope someone could help me a bit in the right direction! > I'm new to moosefs and currently testing it. > > Thanks in advance, > -- > Jens Kristian Søgaard, Mermaid Consulting ApS, > je...@me..., > http://www.mermaidconsulting.com/ > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Michał B. <mic...@co...> - 2012-05-01 15:25:12
|
You can easily update to 1.6.25 Kind regards Michał Borychowski MooseFS Support Manager From: wkmail [mailto:wk...@bn...] Sent: Monday, April 30, 2012 9:17 PM To: moo...@li... Subject: Re: [Moosefs-users] Latest stable release of MooseFS 1.6.25 Hi, is it safe to go to 1.6.25 from 1.6.20 or should we upgrade to 1.6.24 and then 1.6.25? -wk On 4/30/2012 9:30 AM, Michał Borychowski wrote: Hi! As promised we prepared the next release of MooseFS - 1.6.25. Most important changes include: * (master) improved chunk deletion algorithm (soft/hard limits per chunkserver) Now chunk deletion speed is configured by soft and hard limits. What is important the limits since this version are set per chunkserver (not globally). Deleting large amount of files should not slow down the whole system: - CHUNKS_SOFT_DEL_LIMIT = 10 - CHUNKS_HARD_DEL_LIMIT = 25 * We also modified default config values for replication: CHUNKS_WRITE_REP_LIMIT = 2 CHUNKS_READ_REP_LIMIT = 10 which should better fit most of the installations. You should fine tune these values according to the amount of chunkservers and average size of the chunks. Mind that too agressive replication will slow down the whole system. * (master+mount) added 'sugidclearmode' and 'mkdircopysgid' compatibility options (inspired by Sébastien Morand) * fixed Debian init scripts (thanks to Steve Wilson) The recommended upgrade order is as usual: mfsmaster, then metaloggers, then chunkservers and at last client machines. Just as a reminder: MooseFS has now worldwide professional technical support on the enterprise level which is provided by our company Core Technology. You can find all necessary details on the website at <http://www.coretechnology.pl/support.php> http://www.coretechnology.pl/support.php You can download the latest version at <http://www.moosefs.org/download.html> http://www.moosefs.org/download.html webpage. More information about this release is available here: <http://moosefs.org/news-reader/items/moose-file-system-v-1625-released.html > http://moosefs.org/news-reader/items/moose-file-system-v-1625-released.html Our social websites to join/like: <http://www.linkedin.com/groups?home=&gid=4083088&trk=anet_ug_hm> http://www.linkedin.com/groups?home=&gid=4083088&trk=anet_ug_hm <https://www.facebook.com/moosefs> https://www.facebook.com/moosefs <https://twitter.com/#%21/MooseFS> https://twitter.com/#!/MooseFS Thanks for your great support! Kind regards Michal Borychowski ---------------------------------------------------------------------------- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: wkmail <wk...@bn...> - 2012-04-30 19:43:51
|
Hi, is it safe to go to 1.6.25 from 1.6.20 or should we upgrade to 1.6.24 and then 1.6.25? -wk On 4/30/2012 9:30 AM, Micha? Borychowski wrote: > > Hi! > > As promised we prepared the next release of MooseFS - 1.6.25. Most > important changes include: > > * (master) improved chunk deletion algorithm (soft/hard limits per > chunkserver) > > Now chunk deletion speed is configured by soft and hard limits. What > is important the limits since this version are set per chunkserver > (not globally). Deleting large amount of files should not slow down > the whole system: > > - CHUNKS_SOFT_DEL_LIMIT = 10 > > - CHUNKS_HARD_DEL_LIMIT = 25 > > * We also modified default config values for replication: > > CHUNKS_WRITE_REP_LIMIT = 2 > > CHUNKS_READ_REP_LIMIT = 10 > > which should better fit most of the installations. You should fine > tune these values according to the amount of chunkservers and average > size of the chunks. Mind that too agressive replication will slow down > the whole system. > > * (master+mount) added 'sugidclearmode' and 'mkdircopysgid' > compatibility options (inspired by Sébastien Morand) > > * fixed Debian init scripts (thanks to Steve Wilson) > > The recommended upgrade order is as usual: mfsmaster, then > metaloggers, then chunkservers and at last client machines. > > Just as a reminder: MooseFS has now worldwide professional technical > support on the enterprise level which is provided by our company Core > Technology. You can find all necessary details on the website at > http://www.coretechnology.pl/support.php > > You can download the latest version at > http://www.moosefs.org/download.htmlwebpage. > > More information about this release is available here: > > http://moosefs.org/news-reader/items/moose-file-system-v-1625-released.html > > Our social websites to join/like: > > http://www.linkedin.com/groups?home=&gid=4083088&trk=anet_ug_hm > <http://www.linkedin.com/groups?home=&gid=4083088&trk=anet_ug_hm> > > https://www.facebook.com/moosefs > > https://twitter.com/#!/MooseFS <https://twitter.com/#%21/MooseFS> > > Thanks for your great support! > > Kind regards > > Michal Borychowski > > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > > > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Michał B. <mic...@co...> - 2012-04-30 16:28:41
|
Hi! As promised we prepared the next release of MooseFS - 1.6.25. Most important changes include: * (master) improved chunk deletion algorithm (soft/hard limits per chunkserver) Now chunk deletion speed is configured by soft and hard limits. What is important the limits since this version are set per chunkserver (not globally). Deleting large amount of files should not slow down the whole system: - CHUNKS_SOFT_DEL_LIMIT = 10 - CHUNKS_HARD_DEL_LIMIT = 25 * We also modified default config values for replication: CHUNKS_WRITE_REP_LIMIT = 2 CHUNKS_READ_REP_LIMIT = 10 which should better fit most of the installations. You should fine tune these values according to the amount of chunkservers and average size of the chunks. Mind that too agressive replication will slow down the whole system. * (master+mount) added 'sugidclearmode' and 'mkdircopysgid' compatibility options (inspired by Sébastien Morand) * fixed Debian init scripts (thanks to Steve Wilson) The recommended upgrade order is as usual: mfsmaster, then metaloggers, then chunkservers and at last client machines. Just as a reminder: MooseFS has now worldwide professional technical support on the enterprise level which is provided by our company Core Technology. You can find all necessary details on the website at <http://www.coretechnology.pl/support.php> http://www.coretechnology.pl/support.php You can download the latest version at <http://www.moosefs.org/download.html> http://www.moosefs.org/download.html webpage. More information about this release is available here: <http://moosefs.org/news-reader/items/moose-file-system-v-1625-released.html > http://moosefs.org/news-reader/items/moose-file-system-v-1625-released.html Our social websites to join/like: <http://www.linkedin.com/groups?home=&gid=4083088&trk=anet_ug_hm> http://www.linkedin.com/groups?home=&gid=4083088&trk=anet_ug_hm <https://www.facebook.com/moosefs> https://www.facebook.com/moosefs <https://twitter.com/#!/MooseFS> https://twitter.com/#!/MooseFS Thanks for your great support! Kind regards Michal Borychowski |
|
From: Florent B. <fl...@co...> - 2012-04-29 11:34:36
|
Nice ok :-) Thank you for your answer Michal! On 2012-04-29 07:37, Michał Borychowski wrote: > Hi Florent! > > We are aware of this behaviour and it is on our "todo" list to be > improved. > > Regards > Michał > MooseFS Support Manager > > > -----Original Message----- > From: Florent Bautista [mailto:fl...@co...] > Sent: Saturday, April 28, 2012 1:17 PM > To: moo...@li... > Subject: [Moosefs-users] Chunks replication : preference to > "rebalance" > instead of goaling > > Hi all, > > It seems that on my MooseFS installation, the master server prefers > to > rebalance over CS chunks with goal satisfied, instead of taking care > of > chunks in undergoal situation. > > I added a CS recently, and after a master restart, some chunks have > only 1 > valid copy (goal 2 or 3), and it stays like that for a few minutes > (undergoal chunks are decreasing very slowly). During this time, some > chunks > where rebalanced over CS (I see that because some chunks are getting > overgoal, then deleted, etc...). > > Is it possible to set a more important priority on chunks that are > undergoal > ? And only then rebalancing the others. It's very dangerous because > some > files could have only one valid copy during hours... > > I'm on 1.6.24, with 4 CS and 1 ML. Debian Squeeze for all, except the > last > added CS on FreeBSD. > > My config is the default one except: > > CHUNKS_WRITE_REP_LIMIT = 3 > CHUNKS_READ_REP_LIMIT = 10 > > Thank you for help :) > > Flo > > > ---------------------------------------------------------------------------- > -- > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat > landscape has changed and how IT managers can respond. Discussions > will > include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Jens K. S. <je...@me...> - 2012-04-29 08:26:43
|
Hi again, > I received this error message: > backups: master query: receive error A bit more happened after I wrote this message. First I noticed that all writes to files on the mfs mount hanged. I had to force unmount the file system, but even after remounting mfs writes would still not come through. In the end I had to force unmount the filesystem everywhere, and then close down the chunkservers, the metalogger and the master. When I closed down the chunkservers with "mfschunkserver stop", I got the following log message: Apr 29 00:23:16 localhost kernel: [284815.544120] mfschunkserver[2229]: segfault at 0 ip 0000000000411997 sp 00007fff1429b598 error 4 in mfschunkserver[400000+2a000] I got the exact same error message on all chunkservers with only the stack pointer different in each one. After starting the master, metalogger and chunkservers again I was greeted with some bad news. The log shows messages like this: Apr 29 05:47:45 localhost mfsmaster[5036]: chunk 0000000000000AA5 has only invalid copies (2) - please repair it manually Apr 29 05:47:45 localhost mfsmaster[5036]: chunk 0000000000000AA5_00000005 - invalid copy on (10.0.0.1 - ver:00000003) Apr 29 05:47:45 localhost mfsmaster[5036]: chunk 0000000000000AA5_00000005 - invalid copy on (10.0.0.2 - ver:00000004) How do I figure out which file(s) are in that chunk for example? I assume repairing it would be done with mfsfilerepair? On the web interface is now displayed under "Important messages": unavailable chunks: 4543067 unavailable files: 4749936 How can I find out which files are affected? Weirdly in the "Regular chunks state matrix" in the web interface only 6 chunks show as having 0 valid copies. I know that nothing happened to the disks backing the chunk servers at any point (they're RAID-5 arrays). So it sounds weird to me that 6 blocks went missing? The web interface lists 14 file names under "Import messages" as "currently unavailable file". But if I check them with mfsfileinfo and mfscheckfile - they look fine? I hope someone could help me a bit in the right direction! I'm new to moosefs and currently testing it. Thanks in advance, -- Jens Kristian Søgaard, Mermaid Consulting ApS, je...@me..., http://www.mermaidconsulting.com/ |
|
From: Michał B. <mic...@co...> - 2012-04-29 05:36:10
|
Hi Florent! We are aware of this behaviour and it is on our "todo" list to be improved. Regards Michał MooseFS Support Manager -----Original Message----- From: Florent Bautista [mailto:fl...@co...] Sent: Saturday, April 28, 2012 1:17 PM To: moo...@li... Subject: [Moosefs-users] Chunks replication : preference to "rebalance" instead of goaling Hi all, It seems that on my MooseFS installation, the master server prefers to rebalance over CS chunks with goal satisfied, instead of taking care of chunks in undergoal situation. I added a CS recently, and after a master restart, some chunks have only 1 valid copy (goal 2 or 3), and it stays like that for a few minutes (undergoal chunks are decreasing very slowly). During this time, some chunks where rebalanced over CS (I see that because some chunks are getting overgoal, then deleted, etc...). Is it possible to set a more important priority on chunks that are undergoal ? And only then rebalancing the others. It's very dangerous because some files could have only one valid copy during hours... I'm on 1.6.24, with 4 CS and 1 ML. Debian Squeeze for all, except the last added CS on FreeBSD. My config is the default one except: CHUNKS_WRITE_REP_LIMIT = 3 CHUNKS_READ_REP_LIMIT = 10 Thank you for help :) Flo ---------------------------------------------------------------------------- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Jens K. S. <je...@me...> - 2012-04-28 21:53:48
|
Hi all, I'm running a small mfs setup with 3 chunkservers totalling approx. 18 TB of capacity. Today I ran this command on a directory with a lot of files that were previously goal 2: mfssetgoal -r 1 backups I received this error message: backups: master query: receive error Checking with mfsgetgoal it seems the command was executed fine as the goal was indeed 1 on the files I checked. However when I went to the mfs web interface I found problems. Before running the command everything looked fine in the web inerface. First I noticed that a lot of blocks had 2 valid copies but only goal 1. By refreshing I could see blocks going from the "blue" state to the "green" state. This was as expected. Quite unexpected is the fact that I now had 1 chunk with 0 valid copies and goal 1. My understanding of this is that I have now irreversibly lost a chunk of data? Is there any way I can find out which file(s) were completely or partially lost? Another weird thing is that I now have 2 chunks with only 1 valid copy even though their goal is 2. They are shown in "orange" state in the web interface. How come this happened? Is it normal when changing the goal of files? (i.e. a chunk that contains files that now have different goals) It seems that moving the over-goal chunks to on-goal state is prioritized above moving my 2 under-goal chunks. Is this correct? As I re-goaled about 10.000.000 chunks it will be a while before my 2 under-goal chunks are replicated. -- Jens Kristian Søgaard, Mermaid Consulting ApS, je...@me..., http://www.mermaidconsulting.com/ |
|
From: Florent B. <fl...@co...> - 2012-04-28 11:17:23
|
Hi all, It seems that on my MooseFS installation, the master server prefers to rebalance over CS chunks with goal satisfied, instead of taking care of chunks in undergoal situation. I added a CS recently, and after a master restart, some chunks have only 1 valid copy (goal 2 or 3), and it stays like that for a few minutes (undergoal chunks are decreasing very slowly). During this time, some chunks where rebalanced over CS (I see that because some chunks are getting overgoal, then deleted, etc...). Is it possible to set a more important priority on chunks that are undergoal ? And only then rebalancing the others. It's very dangerous because some files could have only one valid copy during hours... I'm on 1.6.24, with 4 CS and 1 ML. Debian Squeeze for all, except the last added CS on FreeBSD. My config is the default one except: CHUNKS_WRITE_REP_LIMIT = 3 CHUNKS_READ_REP_LIMIT = 10 Thank you for help :) Flo |
|
From: Atom P. <ap...@di...> - 2012-04-26 01:11:23
|
On 4/26/2012 12:33 AM, Ricardo J. Barberis wrote: > El Martes 24/04/2012, Boris Epstein escribió: >> Hello all, >> >> This may be a dumb beginner's question but I will ask it all the same: what >> is the recommended formula for figuring out the metadata space to allocate >> for my moosefs setup? >> >> Thanks. >> We have about 8 million chunks (12mil FS objects) and use about 3GB of RAM on the metamaster; the metadata backup is about 1GB. As we grew the data from about 6 million chunks to 8 million chunks the size of the metadata in RAM fluctuated between 3GB and 4GB. It is actually using less RAM now than it was last week with 7mil chunks. This might be due to trash space getting freed or expired chunks being deleted, I wasn't watching those numbers. -- Perfection is just a word I use occasionally with mustard. --Atom Powers-- Director of IT DigiPen Institute of Technology +1 (425) 895-4443 |
|
From: Steve T. <sm...@cb...> - 2012-04-25 20:20:53
|
On Wed, 25 Apr 2012, Ricardo J. Barberis wrote: > El Mi?rcoles 25/04/2012, Steve Thompson escribi?: >> On Wed, 25 Apr 2012, Ricardo J. Barberis wrote: >>> Basically, it says for 25 million files requieres 25 GiB for metadada. >>> In my case, a MooseFS cluster with 4685180 files occupies 4,6 GiB of >>> metadata, according to du. >> >> On the other hand, I have just over 16 million files in MFS currently, >> and the total metadata size on the master, including the change logs, is >> 5.1 GB. Using MFS 1.6.20. >> >> Steve > > Interesting. Are you sure those are files and not chunks (I always mix them)? > If they're actually files, how many chunks do you have? Those are files. The "all fs objects" count at the moment is 16,387,045 (281,535 directories) and there is a total "all chunk copies" of 32,220,586 (goal is 2 for everything). My metadata.mfs.back is 1.5 GB. Steve -- ---------------------------------------------------------------------------- Steve Thompson, Cornell School of Chemical and Biomolecular Engineering smt AT cbe DOT cornell DOT edu "186,282 miles per second: it's not just a good idea, it's the law" ---------------------------------------------------------------------------- |
|
From: Ricardo J. B. <ric...@da...> - 2012-04-25 20:09:05
|
El Miércoles 25/04/2012, Steve Thompson escribió: > On Wed, 25 Apr 2012, Ricardo J. Barberis wrote: > > Basically, it says for 25 million files requieres 25 GiB for metadada. > > In my case, a MooseFS cluster with 4685180 files occupies 4,6 GiB of > > metadata, according to du. > > On the other hand, I have just over 16 million files in MFS currently, > and the total metadata size on the master, including the change logs, is > 5.1 GB. Using MFS 1.6.20. > > Steve Interesting. Are you sure those are files and not chunks (I always mix them)? If they're actually files, how many chunks do you have? The MFS cluster I mentioned above has all small files, so it has 4.6 million chunks, and I'm not sure if metadata size corresponds to files or chunks. And I'm also counting changelogs, metadata.mfs.back is only 484 MiB. Regards, -- Ricardo J. Barberis Senior SysAdmin / ITI Dattatec.com :: Soluciones de Web Hosting Tu Hosting hecho Simple! |
|
From: Steve T. <sm...@cb...> - 2012-04-25 18:02:44
|
On Wed, 25 Apr 2012, Ricardo J. Barberis wrote: > Basically, it says for 25 million files requieres 25 GiB for metadada. > In my case, a MooseFS cluster with 4685180 files occupies 4,6 GiB of metadata, > according to du. On the other hand, I have just over 16 million files in MFS currently, and the total metadata size on the master, including the change logs, is 5.1 GB. Using MFS 1.6.20. Steve -- ---------------------------------------------------------------------------- Steve Thompson, Cornell School of Chemical and Biomolecular Engineering smt AT cbe DOT cornell DOT edu "186,282 miles per second: it's not just a good idea, it's the law" ---------------------------------------------------------------------------- |
|
From: Ricardo J. B. <ric...@da...> - 2012-04-25 16:33:51
|
El Martes 24/04/2012, Boris Epstein escribió: > Hello all, > > This may be a dumb beginner's question but I will ask it all the same: what > is the recommended formula for figuring out the metadata space to allocate > for my moosefs setup? > > Thanks. > > Boris I don't have hard numbers, but you can check: http://www.moosefs.org/moosefs-faq.html#sort Basically, it says for 25 million files requieres 25 GiB for metadada. In my case, a MooseFS cluster with 4685180 files occupies 4,6 GiB of metadata, according to du. Regards, -- Ricardo J. Barberis Senior SysAdmin / ITI Dattatec.com :: Soluciones de Web Hosting Tu Hosting hecho Simple! |
|
From: Deon C. <deo...@gm...> - 2012-04-25 08:31:39
|
Hi, I am a bit of a new moose user, currently using moose as a simple smb share for backup storage. It does this job well and I have no complaints. We're currently running SL6.2 master servers and freebsd 9 zfs pools for chunk servers. Performance is not an issue because bulk data and consistency is more important in this use case. However I am looking into creating a unified storage space for KVM virtual machines in a separate moosefs cluster. I've tried to do my research and one topic which has been mentioned but not elaborated on is the compatibility with the virtio driver? When using moose with kvm does the IDE driver provide better performance than virtio? Another issue I have seen is related to fuse and it not previously supporting directio, which prevented kvm guests using cache=none from starting. Are these two issues related? If they are related then I have found information that O_DIRECT support is coming to fuse soon, when it does will this issue be resolved? Thanks Deon |
|
From: Ken <ken...@gm...> - 2012-04-25 01:11:17
|
>> store url and photo in leveldb, which use MooseFS as disks. I understand this solution now. Leveldb maybe write 'disk' 2 times in add/insert once, impact of Moosefs goal, the write performance is worse. The most important, only one writer can access one leveldb file. Single server is too limited. -Ken On Tue, Apr 24, 2012 at 3:11 PM, Davies Liu <dav...@gm...> wrote: > On Tue, Apr 24, 2012 at 2:29 PM, Ken <ken...@gm...> wrote: >> >> On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm...> wrote: >> > Maybe leveldb + MooseFS is better. >> Is this your mean? >> leveldb store url >> moosefs store photo > > > No, store url and photo in leveldb, which use MooseFS as disks. > the performance may be not perfect, but enough. > If not, TFS from taobao.com may be the better choice. > >> > >> > On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm...> wrote: >> >> >> >> We need to store tons of small files(photo), as noticed in faq, file >> >> count is limited in moosefs, I think bundle small files to a huge file >> >> maybe work. >> >> >> >> Write photos procedure like: >> >> allocate a huge file >> >> write head and photo content >> >> return (huge file, position, size) >> >> write another head and another photo >> >> return (huge file, position, size) >> >> ... >> >> >> >> Before read a photo, we should have enough information: huge file, >> >> position and length, the reading procedure is expected normally. >> >> >> >> To read a photo, we should provide an URL, like >> >> 'http://xxx.com/prefix/huge file/offset/size.jpg' >> >> >> >> And to be useful in WEB, build a fastcgi program for read/write access >> >> to the huge file. >> >> >> >> ps: >> >> * The matchup information photo and url should store outside of >> >> mooosefs. >> >> * Huge file size limited in 2G. >> >> * Huge file maybe cause race condition. >> >> >> >> Is there anyone interested in this? >> >> or better solution? >> >> >> >> -Ken >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Live Security Virtual Conference >> >> Exclusive live event will cover all the ways today's security and >> >> threat landscape has changed and how IT managers can respond. >> >> Discussions >> >> will include endpoint security, mobile security and the latest in >> >> malware >> >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> >> _______________________________________________ >> >> moosefs-users mailing list >> >> moo...@li... >> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> > >> > >> > >> > >> > -- >> > - Davies > > > > > -- > - Davies |
|
From: <ro...@mm...> - 2012-04-24 22:54:23
|
Wang Jian wrote:
> It's not that simple.
My response is going to be off-topic for this list, so this is an one-off
post (kind-of). I'm sure your team has gone over different architectures,
I'm just offering a (hopefully) insightful suggestion.
As I understand your problem, you are trying to build something like this:
Application Servers Storage Area
A B C D .... Z --------> BBKAfs (100's of TB)
where BBKAfs = Big-Bad-Kick-Ass-filesystem
Since the problem seems to lie in the BBKAfs, I am suggesting that you
should do without it altogether by employing sharding.
Device a way to partition your users (an algorithmic hash perhaps?) and
assign them to different machines. Each machine could hold a subset of
users and the relevant data. So instead of the above situation, you end up
in something like the following:
Application Servers w/Local Storage
A (how about 4 to 8TB)
B (how about 4 to 8TB)
C (how about 4 to 8TB)
...
Z ....
This way you have no need for a central repository where everything is
stored and no need for special filesystems. Each individual server is
smaller and so easier to manage (double them for fault tolerance, you are
already doing something along these line anyway). No single-point of
failure and adopt your scripts that you already must have for handling the
Application Servers in the original approach for handling the second case.
Of course, this suggestion leaves a lot of issues to be resolved, but is
enough to sketch the idea.
>
> In your scenario, one FS image can only be mounted once, that means a
> single server, thus single point of failure. And more seriously, loss of
> one chunk of FS image(I mean real one, this happens when
> two or more harddisk fail at the same time) leads to the whole image
> corruption, then you will lose all data in the FS image.
>
> Another problem is meta data overhead. Your mounting system will do
> filesystem journal, and below, MooseFS will do filesystem journal.
You could use ext2 which isn't a journaling fs.
>
> DB? I am curious if some company really did and be doing that, in the
> really large scale (tens and hundereds of 12TB servers)
I remember reading that facebook does this (goolge on mysql sharding).
Also : Apache Cassandra 1.1
http://www.h-online.com/open/news/item/Apache-Cassandra-1-1-release-comes-to-pass-1557837.html
"...According to the ASF, the largest known production cluster of
Cassandra carried over 300 terabytes of data spread over 400 machines...."
-Stathis
>
>
> δΊ 2012/4/24 17:06, ro...@mm... ει:
>> Ken wrote:
>>> This way is too Linux. ha. How about growing?
>> It should work on any decent *nix system.
>>
>> Growing?
>>
>> Option-1
>> How about using resize2fs.
>>
>> Option-2
>> [sci-fi-ON]
>> You could create a PV group out of filesystems-in-a-file.
>> Every time you need more space, you just add a new filesystem-in-a-file
>> to
>> the PV group
>> [sci-fi-OFF]
>>
>> Option-3
>> Use sparse files.
>> The file containing the file system could be a sparse file. This way,
>> you
>> can can specify, for example, a 2TB sparse file that will really only
>> consume the actual data as a storage space.
>> I don't know how MooseFS handles sparse files though...
>>
>> Option-4
>> Forget about filesystems (including MooseFS) and use a database for
>> storing the photos.
>> Use sharding or partioning to keep the database file down to a
>> reasonable(whatever this may be) size and improve performance.
>> Plenty of options for on-line replication of database data (so no single
>> point of failure)
>>
>> -Stathis
>>
>>> The business maybe same as Flickr, Facebook photo and DropBox, etc...
>>>
>>> If small files store resolved, moosefs will be used in more product
>>> applications scenarios.
>>>
>>> Thanks
>>>
>>> -Ken
>>>
>>>
>>> On Tue, Apr 24, 2012 at 4:10 PM,<ro...@mm...> wrote:
>>>> If you're on linux,
>>>>
>>>> i'd try a filesystem-in-a-file which would be used to store the photo
>>>> and
>>>> this file itselfs stored in a MooseFS volume.
>>>>
>>>> For example something along these lines:
>>>>
>>>> dd if=/dev/zero of=FILENAME bs=1024 count=1 seek=$((2*1024*1024-1))
>>>> mkfs -t ext3 FILENAME
>>>> mount -t ext3 -o loop FILENAME MOUNTPOINT
>>>>
>>>> where MOUNTPOINT is a MooseFS volume.
>>>>
>>>> -Stathis
>>>>
>>>> Wang Jian wrote:
>>>>> TFS has strict and random URL scheme, so it's difficult to do URL
>>>>> based
>>>>> tuning.
>>>>>
>>>>> δΠ2012/4/24 15:11, Davies Liu Ρ ι :
>>>>>> On Tue, Apr 24, 2012 at 2:29 PM, Ken<ken...@gm...
>>>>>> <mailto:ken...@gm...>> wrote:
>>>>>>
>>>>>> On Tue, Apr 24, 2012 at 2:22 PM, Davies
>>>>>> Liu<dav...@gm...
>>>>>> <mailto:dav...@gm...>> wrote:
>>>>>> > Maybe leveldb + MooseFS is better.
>>>>>> Is this your mean?
>>>>>> leveldb store url
>>>>>> moosefs store photo
>>>>>>
>>>>>> No, store url and photo in leveldb, which use MooseFS as disks.
>>>>>> the performance may be not perfect, but enough.
>>>>>> If not, TFS from taobao.com<http://taobao.com> may be the better
>>>>>> choice.
>>>>>>
>>>>>> >
>>>>>> > On Tue, Apr 24, 2012 at 1:59 PM, Ken<ken...@gm...
>>>>>> <mailto:ken...@gm...>> wrote:
>>>>>> >>
>>>>>> >> We need to store tons of small files(photo), as noticed in
>>>>>> faq,
>>>>>> file
>>>>>> >> count is limited in moosefs, I think bundle small files to
>>>>>> a
>>>>>> huge
>>>>>> file
>>>>>> >> maybe work.
>>>>>> >>
>>>>>> >> Write photos procedure like:
>>>>>> >> allocate a huge file
>>>>>> >> write head and photo content
>>>>>> >> return (huge file, position, size)
>>>>>> >> write another head and another photo
>>>>>> >> return (huge file, position, size)
>>>>>> >> ...
>>>>>> >>
>>>>>> >> Before read a photo, we should have enough information:
>>>>>> huge
>>>>>> file,
>>>>>> >> position and length, the reading procedure is expected
>>>>>> normally.
>>>>>> >>
>>>>>> >> To read a photo, we should provide an URL, like
>>>>>> >> 'http://xxx.com/prefix/huge file/offset/size.jpg'
>>>>>> >>
>>>>>> >> And to be useful in WEB, build a fastcgi program for
>>>>>> read/write
>>>>>> access
>>>>>> >> to the huge file.
>>>>>> >>
>>>>>> >> ps:
>>>>>> >> * The matchup information photo and url should store
>>>>>> outside
>>>>>> of
>>>>>> mooosefs.
>>>>>> >> * Huge file size limited in 2G.
>>>>>> >> * Huge file maybe cause race condition.
>>>>>> >>
>>>>>> >> Is there anyone interested in this?
>>>>>> >> or better solution?
>>>>>> >>
>>>>>> >> -Ken
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> ------------------------------------------------------------------------------
>>>>>> >> Live Security Virtual Conference
>>>>>> >> Exclusive live event will cover all the ways today's
>>>>>> security
>>>>>> and
>>>>>> >> threat landscape has changed and how IT managers can
>>>>>> respond.
>>>>>> Discussions
>>>>>> >> will include endpoint security, mobile security and the
>>>>>> latest
>>>>>> in
>>>>>> malware
>>>>>> >> threats.
>>>>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>>> >> _______________________________________________
>>>>>> >> moosefs-users mailing list
>>>>>> >> moo...@li...
>>>>>> <mailto:moo...@li...>
>>>>>> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > - Davies
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> - Davies
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Live Security Virtual Conference
>>>>>> Exclusive live event will cover all the ways today's security and
>>>>>> threat landscape has changed and how IT managers can respond.
>>>>>> Discussions
>>>>>> will include endpoint security, mobile security and the latest in
>>>>>> malware
>>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> moosefs-users mailing list
>>>>>> moo...@li...
>>>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>>> ------------------------------------------------------------------------------
>>>>> Live Security Virtual Conference
>>>>> Exclusive live event will cover all the ways today's security and
>>>>> threat landscape has changed and how IT managers can respond.
>>>>> Discussions
>>>>> will include endpoint security, mobile security and the latest in
>>>>> malware
>>>>> threats.
>>>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
>>>>> moosefs-users mailing list
>>>>> moo...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> will include endpoint security, mobile security and the latest in
>>>> malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> moosefs-users mailing list
>>>> moo...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond.
>> Discussions
>> will include endpoint security, mobile security and the latest in
>> malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
|
|
From: Boris E. <bor...@gm...> - 2012-04-24 22:25:20
|
Hello all, This may be a dumb beginner's question but I will ask it all the same: what is the recommended formula for figuring out the metadata space to allocate for my moosefs setup? Thanks. Boris |
|
From: Wang J. <jia...@re...> - 2012-04-24 18:04:55
|
于 2012/4/25 0:40, Allen, Benjamin S 写道: > Have you looked at Facebook's Haystack? It's not publicly available but likely a very good model to base your infrastructure on: > > http://static.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf > > Why not push your images to Amazon's Cloudfront or similar CDN service? > > Another option, LiveJournal built MogileFS for this specific purpose. > > Using a sparse file or FS in a file approach on top of MooseFS is a bit silly. Since you're stacking three filesystems. Ensure you look at the administration overhead of the solutions as well. > > MooseFS, as the authors freely admit, was not designed for small files. It was designed to serve big media files. I'd suggest picking something that was written to solve your problem, not shoehorn > MooseFS. > Facebook's Haystack and Taobao's TFS, are both 'packed small files in large files', the answer to manage titanic amount of small files. Their very difference is about how to store meta-meta information. Haystack, according to published docs, encodes location information in filename, i.e. big file location and small file's offset and length in the pack. But TFS store small meta-meta information in database. By comparison, Haystack goes furthur to decrease meta lookup overhead. But under the hood, they both have two steps when accessing small files: 1. Small files to big files location, offset, length, etc 2. Access the big files' parts in question You know, whether the first step is builtin or seperate from big file storage engine, there's no intrinsic performance difference. How to do the first step does matter. The solution Ken and I are referring, is to use a seperate meta service, which translates small file names to big file names, offset, length, etc. MooseFS can be used as low level big file storage, but other engine, like Ceph, should work too. > Ben > > On Apr 24, 2012, at 4:25 AM, Wang Jian wrote: > >> It's not that simple. >> >> In your scenario, one FS image can only be mounted once, that means a single server, thus single point of failure. And more seriously, loss of one chunk of FS image(I mean real one, this happens when >> two or more harddisk fail at the same time) leads to the whole image corruption, then you will lose all data in the FS image. >> >> Another problem is meta data overhead. Your mounting system will do filesystem journal, and below, MooseFS will do filesystem journal. >> >> DB? I am curious if some company really did and be doing that, in the really large scale (tens and hundereds of 12TB servers) >> |
|
From: Allen, B. S <bs...@la...> - 2012-04-24 16:41:08
|
Have you looked at Facebook's Haystack? It's not publicly available but likely a very good model to base your infrastructure on: http://static.usenix.org/event/osdi10/tech/full_papers/Beaver.pdf Why not push your images to Amazon's Cloudfront or similar CDN service? Another option, LiveJournal built MogileFS for this specific purpose. Using a sparse file or FS in a file approach on top of MooseFS is a bit silly. Since you're stacking three filesystems. Ensure you look at the administration overhead of the solutions as well. MooseFS, as the authors freely admit, was not designed for small files. It was designed to serve big media files. I'd suggest picking something that was written to solve your problem, not shoehorn MooseFS. Ben On Apr 24, 2012, at 4:25 AM, Wang Jian wrote: It's not that simple. In your scenario, one FS image can only be mounted once, that means a single server, thus single point of failure. And more seriously, loss of one chunk of FS image(I mean real one, this happens when two or more harddisk fail at the same time) leads to the whole image corruption, then you will lose all data in the FS image. Another problem is meta data overhead. Your mounting system will do filesystem journal, and below, MooseFS will do filesystem journal. DB? I am curious if some company really did and be doing that, in the really large scale (tens and hundereds of 12TB servers) 于 2012/4/24 17:06, ro...@mm...<mailto:ro...@mm...> 写道: Ken wrote: This way is too Linux. ha. How about growing? It should work on any decent *nix system. Growing? Option-1 How about using resize2fs. Option-2 [sci-fi-ON] You could create a PV group out of filesystems-in-a-file. Every time you need more space, you just add a new filesystem-in-a-file to the PV group [sci-fi-OFF] Option-3 Use sparse files. The file containing the file system could be a sparse file. This way, you can can specify, for example, a 2TB sparse file that will really only consume the actual data as a storage space. I don't know how MooseFS handles sparse files though... Option-4 Forget about filesystems (including MooseFS) and use a database for storing the photos. Use sharding or partioning to keep the database file down to a reasonable(whatever this may be) size and improve performance. Plenty of options for on-line replication of database data (so no single point of failure) -Stathis The business maybe same as Flickr, Facebook photo and DropBox, etc... If small files store resolved, moosefs will be used in more product applications scenarios. Thanks -Ken On Tue, Apr 24, 2012 at 4:10 PM,<ro...@mm...<mailto:ro...@mm...>> wrote: If you're on linux, i'd try a filesystem-in-a-file which would be used to store the photo and this file itselfs stored in a MooseFS volume. For example something along these lines: dd if=/dev/zero of=FILENAME bs=1024 count=1 seek=$((2*1024*1024-1)) mkfs -t ext3 FILENAME mount -t ext3 -o loop FILENAME MOUNTPOINT where MOUNTPOINT is a MooseFS volume. -Stathis Wang Jian wrote: TFS has strict and random URL scheme, so it's difficult to do URL based tuning. δΊ 2012/4/24 15:11, Davies Liu ε ι : On Tue, Apr 24, 2012 at 2:29 PM, Ken<ken...@gm...<mailto:ken...@gm...> <mailto:ken...@gm...>> wrote: On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu<dav...@gm...<mailto:dav...@gm...> <mailto:dav...@gm...>> wrote: Maybe leveldb + MooseFS is better. Is this your mean? leveldb store url moosefs store photo No, store url and photo in leveldb, which use MooseFS as disks. the performance may be not perfect, but enough. If not, TFS from taobao.com<http://taobao.com><http://taobao.com> may be the better choice. On Tue, Apr 24, 2012 at 1:59 PM, Ken<ken...@gm...<mailto:ken...@gm...> <mailto:ken...@gm...>> wrote: We need to store tons of small files(photo), as noticed in faq, file count is limited in moosefs, I think bundle small files to a huge file maybe work. Write photos procedure like: allocate a huge file write head and photo content return (huge file, position, size) write another head and another photo return (huge file, position, size) ... Before read a photo, we should have enough information: huge file, position and length, the reading procedure is expected normally. To read a photo, we should provide an URL, like 'http://xxx.com/prefix/huge file/offset/size.jpg' And to be useful in WEB, build a fastcgi program for read/write access to the huge file. ps: * The matchup information photo and url should store outside of mooosefs. * Huge file size limited in 2G. * Huge file maybe cause race condition. Is there anyone interested in this? or better solution? -Ken ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ moosefs-users mailing list moo...@li...<mailto:moo...@li...> <mailto:moo...@li...> https://lists.sourceforge.net/lists/listinfo/moosefs-users -- - Davies -- - Davies ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ moosefs-users mailing list moo...@li...<mailto:moo...@li...> https://lists.sourceforge.net/lists/listinfo/moosefs-users ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________ moosefs-users mailing list moo...@li...<mailto:moo...@li...> https://lists.sourceforge.net/lists/listinfo/moosefs-users ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ moosefs-users mailing list moo...@li...<mailto:moo...@li...> https://lists.sourceforge.net/lists/listinfo/moosefs-users ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ moosefs-users mailing list moo...@li...<mailto:moo...@li...> https://lists.sourceforge.net/lists/listinfo/moosefs-users ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ moosefs-users mailing list moo...@li...<mailto:moo...@li...> https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Wang J. <jia...@re...> - 2012-04-24 10:25:43
|
It's not that simple. In your scenario, one FS image can only be mounted once, that means a single server, thus single point of failure. And more seriously, loss of one chunk of FS image(I mean real one, this happens when two or more harddisk fail at the same time) leads to the whole image corruption, then you will lose all data in the FS image. Another problem is meta data overhead. Your mounting system will do filesystem journal, and below, MooseFS will do filesystem journal. DB? I am curious if some company really did and be doing that, in the really large scale (tens and hundereds of 12TB servers) 于 2012/4/24 17:06, ro...@mm... 写道: > Ken wrote: >> This way is too Linux. ha. How about growing? > It should work on any decent *nix system. > > Growing? > > Option-1 > How about using resize2fs. > > Option-2 > [sci-fi-ON] > You could create a PV group out of filesystems-in-a-file. > Every time you need more space, you just add a new filesystem-in-a-file to > the PV group > [sci-fi-OFF] > > Option-3 > Use sparse files. > The file containing the file system could be a sparse file. This way, you > can can specify, for example, a 2TB sparse file that will really only > consume the actual data as a storage space. > I don't know how MooseFS handles sparse files though... > > Option-4 > Forget about filesystems (including MooseFS) and use a database for > storing the photos. > Use sharding or partioning to keep the database file down to a > reasonable(whatever this may be) size and improve performance. > Plenty of options for on-line replication of database data (so no single > point of failure) > > -Stathis > >> The business maybe same as Flickr, Facebook photo and DropBox, etc... >> >> If small files store resolved, moosefs will be used in more product >> applications scenarios. >> >> Thanks >> >> -Ken >> >> >> On Tue, Apr 24, 2012 at 4:10 PM,<ro...@mm...> wrote: >>> If you're on linux, >>> >>> i'd try a filesystem-in-a-file which would be used to store the photo >>> and >>> this file itselfs stored in a MooseFS volume. >>> >>> For example something along these lines: >>> >>> dd if=/dev/zero of=FILENAME bs=1024 count=1 seek=$((2*1024*1024-1)) >>> mkfs -t ext3 FILENAME >>> mount -t ext3 -o loop FILENAME MOUNTPOINT >>> >>> where MOUNTPOINT is a MooseFS volume. >>> >>> -Stathis >>> >>> Wang Jian wrote: >>>> TFS has strict and random URL scheme, so it's difficult to do URL based >>>> tuning. >>>> >>>> δΊ 2012/4/24 15:11, Davies Liu ε ι : >>>>> On Tue, Apr 24, 2012 at 2:29 PM, Ken<ken...@gm... >>>>> <mailto:ken...@gm...>> wrote: >>>>> >>>>> On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu<dav...@gm... >>>>> <mailto:dav...@gm...>> wrote: >>>>> > Maybe leveldb + MooseFS is better. >>>>> Is this your mean? >>>>> leveldb store url >>>>> moosefs store photo >>>>> >>>>> No, store url and photo in leveldb, which use MooseFS as disks. >>>>> the performance may be not perfect, but enough. >>>>> If not, TFS from taobao.com<http://taobao.com> may be the better >>>>> choice. >>>>> >>>>> > >>>>> > On Tue, Apr 24, 2012 at 1:59 PM, Ken<ken...@gm... >>>>> <mailto:ken...@gm...>> wrote: >>>>> >> >>>>> >> We need to store tons of small files(photo), as noticed in faq, >>>>> file >>>>> >> count is limited in moosefs, I think bundle small files to a >>>>> huge >>>>> file >>>>> >> maybe work. >>>>> >> >>>>> >> Write photos procedure like: >>>>> >> allocate a huge file >>>>> >> write head and photo content >>>>> >> return (huge file, position, size) >>>>> >> write another head and another photo >>>>> >> return (huge file, position, size) >>>>> >> ... >>>>> >> >>>>> >> Before read a photo, we should have enough information: huge >>>>> file, >>>>> >> position and length, the reading procedure is expected >>>>> normally. >>>>> >> >>>>> >> To read a photo, we should provide an URL, like >>>>> >> 'http://xxx.com/prefix/huge file/offset/size.jpg' >>>>> >> >>>>> >> And to be useful in WEB, build a fastcgi program for read/write >>>>> access >>>>> >> to the huge file. >>>>> >> >>>>> >> ps: >>>>> >> * The matchup information photo and url should store outside >>>>> of >>>>> mooosefs. >>>>> >> * Huge file size limited in 2G. >>>>> >> * Huge file maybe cause race condition. >>>>> >> >>>>> >> Is there anyone interested in this? >>>>> >> or better solution? >>>>> >> >>>>> >> -Ken >>>>> >> >>>>> >> >>>>> >> >>>>> ------------------------------------------------------------------------------ >>>>> >> Live Security Virtual Conference >>>>> >> Exclusive live event will cover all the ways today's security >>>>> and >>>>> >> threat landscape has changed and how IT managers can respond. >>>>> Discussions >>>>> >> will include endpoint security, mobile security and the latest >>>>> in >>>>> malware >>>>> >> threats. >>>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>>>> >> _______________________________________________ >>>>> >> moosefs-users mailing list >>>>> >> moo...@li... >>>>> <mailto:moo...@li...> >>>>> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > -- >>>>> > - Davies >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> - Davies >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Live Security Virtual Conference >>>>> Exclusive live event will cover all the ways today's security and >>>>> threat landscape has changed and how IT managers can respond. >>>>> Discussions >>>>> will include endpoint security, mobile security and the latest in >>>>> malware >>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>>>> >>>>> >>>>> _______________________________________________ >>>>> moosefs-users mailing list >>>>> moo...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>>> ------------------------------------------------------------------------------ >>>> Live Security Virtual Conference >>>> Exclusive live event will cover all the ways today's security and >>>> threat landscape has changed and how IT managers can respond. >>>> Discussions >>>> will include endpoint security, mobile security and the latest in >>>> malware >>>> threats. >>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________ >>>> moosefs-users mailing list >>>> moo...@li... >>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. >>> Discussions >>> will include endpoint security, mobile security and the latest in >>> malware >>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>> _______________________________________________ >>> moosefs-users mailing list >>> moo...@li... >>> https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: 舒彦博 <shu...@gm...> - 2012-04-24 10:19:18
|
hi,
MooseFS is efficent file system for distributing. From Q&A on
www.moosefs.org, I know moosefs support snapshot feature.
I want to know whether if moose suppport incremental backup for data in
chunkdata server? If not support, will it be added int the future?
3x
Good lucky
|
|
From: <ro...@mm...> - 2012-04-24 09:06:26
|
Ken wrote: > This way is too Linux. ha. How about growing? It should work on any decent *nix system. Growing? Option-1 How about using resize2fs. Option-2 [sci-fi-ON] You could create a PV group out of filesystems-in-a-file. Every time you need more space, you just add a new filesystem-in-a-file to the PV group [sci-fi-OFF] Option-3 Use sparse files. The file containing the file system could be a sparse file. This way, you can can specify, for example, a 2TB sparse file that will really only consume the actual data as a storage space. I don't know how MooseFS handles sparse files though... Option-4 Forget about filesystems (including MooseFS) and use a database for storing the photos. Use sharding or partioning to keep the database file down to a reasonable(whatever this may be) size and improve performance. Plenty of options for on-line replication of database data (so no single point of failure) -Stathis > > The business maybe same as Flickr, Facebook photo and DropBox, etc... > > If small files store resolved, moosefs will be used in more product > applications scenarios. > > Thanks > > -Ken > > > On Tue, Apr 24, 2012 at 4:10 PM, <ro...@mm...> wrote: >> If you're on linux, >> >> i'd try a filesystem-in-a-file which would be used to store the photo >> and >> this file itselfs stored in a MooseFS volume. >> >> For example something along these lines: >> >> dd if=/dev/zero of=FILENAME bs=1024 count=1 seek=$((2*1024*1024-1)) >> mkfs -t ext3 FILENAME >> mount -t ext3 -o loop FILENAME MOUNTPOINT >> >> where MOUNTPOINT is a MooseFS volume. >> >> -Stathis >> >> Wang Jian wrote: >>> TFS has strict and random URL scheme, so it's difficult to do URL based >>> tuning. >>> >>> δΊ 2012/4/24 15:11, Davies Liu ε ι : >>>> On Tue, Apr 24, 2012 at 2:29 PM, Ken <ken...@gm... >>>> <mailto:ken...@gm...>> wrote: >>>> >>>> On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm... >>>> <mailto:dav...@gm...>> wrote: >>>> > Maybe leveldb + MooseFS is better. >>>> Is this your mean? >>>> leveldb store url >>>> moosefs store photo >>>> >>>> No, store url and photo in leveldb, which use MooseFS as disks. >>>> the performance may be not perfect, but enough. >>>> If not, TFS from taobao.com <http://taobao.com> may be the better >>>> choice. >>>> >>>> > >>>> > On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm... >>>> <mailto:ken...@gm...>> wrote: >>>> >> >>>> >> We need to store tons of small files(photo), as noticed in faq, >>>> file >>>> >> count is limited in moosefs, I think bundle small files to a >>>> huge >>>> file >>>> >> maybe work. >>>> >> >>>> >> Write photos procedure like: >>>> >> allocate a huge file >>>> >> write head and photo content >>>> >> return (huge file, position, size) >>>> >> write another head and another photo >>>> >> return (huge file, position, size) >>>> >> ... >>>> >> >>>> >> Before read a photo, we should have enough information: huge >>>> file, >>>> >> position and length, the reading procedure is expected >>>> normally. >>>> >> >>>> >> To read a photo, we should provide an URL, like >>>> >> 'http://xxx.com/prefix/huge file/offset/size.jpg' >>>> >> >>>> >> And to be useful in WEB, build a fastcgi program for read/write >>>> access >>>> >> to the huge file. >>>> >> >>>> >> ps: >>>> >> * The matchup information photo and url should store outside >>>> of >>>> mooosefs. >>>> >> * Huge file size limited in 2G. >>>> >> * Huge file maybe cause race condition. >>>> >> >>>> >> Is there anyone interested in this? >>>> >> or better solution? >>>> >> >>>> >> -Ken >>>> >> >>>> >> >>>> >> >>>> ------------------------------------------------------------------------------ >>>> >> Live Security Virtual Conference >>>> >> Exclusive live event will cover all the ways today's security >>>> and >>>> >> threat landscape has changed and how IT managers can respond. >>>> Discussions >>>> >> will include endpoint security, mobile security and the latest >>>> in >>>> malware >>>> >> threats. >>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>>> >> _______________________________________________ >>>> >> moosefs-users mailing list >>>> >> moo...@li... >>>> <mailto:moo...@li...> >>>> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>>> > >>>> > >>>> > >>>> > >>>> > -- >>>> > - Davies >>>> >>>> >>>> >>>> >>>> -- >>>> - Davies >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Live Security Virtual Conference >>>> Exclusive live event will cover all the ways today's security and >>>> threat landscape has changed and how IT managers can respond. >>>> Discussions >>>> will include endpoint security, mobile security and the latest in >>>> malware >>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>>> >>>> >>>> _______________________________________________ >>>> moosefs-users mailing list >>>> moo...@li... >>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>> >>> ------------------------------------------------------------------------------ >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. >>> Discussions >>> will include endpoint security, mobile security and the latest in >>> malware >>> threats. >>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________ >>> moosefs-users mailing list >>> moo...@li... >>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>> >> >> >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. >> Discussions >> will include endpoint security, mobile security and the latest in >> malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
|
From: Ken <ken...@gm...> - 2012-04-24 08:44:06
|
This way is too Linux. ha. How about growing? The business maybe same as Flickr, Facebook photo and DropBox, etc... If small files store resolved, moosefs will be used in more product applications scenarios. Thanks -Ken On Tue, Apr 24, 2012 at 4:10 PM, <ro...@mm...> wrote: > If you're on linux, > > i'd try a filesystem-in-a-file which would be used to store the photo and > this file itselfs stored in a MooseFS volume. > > For example something along these lines: > > dd if=/dev/zero of=FILENAME bs=1024 count=1 seek=$((2*1024*1024-1)) > mkfs -t ext3 FILENAME > mount -t ext3 -o loop FILENAME MOUNTPOINT > > where MOUNTPOINT is a MooseFS volume. > > -Stathis > > Wang Jian wrote: >> TFS has strict and random URL scheme, so it's difficult to do URL based >> tuning. >> >> δΊ 2012/4/24 15:11, Davies Liu ε ι : >>> On Tue, Apr 24, 2012 at 2:29 PM, Ken <ken...@gm... >>> <mailto:ken...@gm...>> wrote: >>> >>> On Tue, Apr 24, 2012 at 2:22 PM, Davies Liu <dav...@gm... >>> <mailto:dav...@gm...>> wrote: >>> > Maybe leveldb + MooseFS is better. >>> Is this your mean? >>> leveldb store url >>> moosefs store photo >>> >>> No, store url and photo in leveldb, which use MooseFS as disks. >>> the performance may be not perfect, but enough. >>> If not, TFS from taobao.com <http://taobao.com> may be the better >>> choice. >>> >>> > >>> > On Tue, Apr 24, 2012 at 1:59 PM, Ken <ken...@gm... >>> <mailto:ken...@gm...>> wrote: >>> >> >>> >> We need to store tons of small files(photo), as noticed in faq, >>> file >>> >> count is limited in moosefs, I think bundle small files to a huge >>> file >>> >> maybe work. >>> >> >>> >> Write photos procedure like: >>> >> allocate a huge file >>> >> write head and photo content >>> >> return (huge file, position, size) >>> >> write another head and another photo >>> >> return (huge file, position, size) >>> >> ... >>> >> >>> >> Before read a photo, we should have enough information: huge >>> file, >>> >> position and length, the reading procedure is expected normally. >>> >> >>> >> To read a photo, we should provide an URL, like >>> >> 'http://xxx.com/prefix/huge file/offset/size.jpg' >>> >> >>> >> And to be useful in WEB, build a fastcgi program for read/write >>> access >>> >> to the huge file. >>> >> >>> >> ps: >>> >> * The matchup information photo and url should store outside of >>> mooosefs. >>> >> * Huge file size limited in 2G. >>> >> * Huge file maybe cause race condition. >>> >> >>> >> Is there anyone interested in this? >>> >> or better solution? >>> >> >>> >> -Ken >>> >> >>> >> >>> >> ------------------------------------------------------------------------------ >>> >> Live Security Virtual Conference >>> >> Exclusive live event will cover all the ways today's security and >>> >> threat landscape has changed and how IT managers can respond. >>> Discussions >>> >> will include endpoint security, mobile security and the latest in >>> malware >>> >> threats. >>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>> >> _______________________________________________ >>> >> moosefs-users mailing list >>> >> moo...@li... >>> <mailto:moo...@li...> >>> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>> > >>> > >>> > >>> > >>> > -- >>> > - Davies >>> >>> >>> >>> >>> -- >>> - Davies >>> >>> >>> ------------------------------------------------------------------------------ >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. >>> Discussions >>> will include endpoint security, mobile security and the latest in >>> malware >>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>> >>> >>> _______________________________________________ >>> moosefs-users mailing list >>> moo...@li... >>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. >> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> > > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
84 messages has been excluded from this view by a project administrator.
