Menu
▾
▴
moosefs-users
|
From: Stas O. <sta...@gm...> - 2010-07-06 10:07:59
|
Hi. Is there any recommended procedure how to back-up MFS meta-data? I.e: * Where to backup from - master or logger? * Backup the flat files, or use some mfs tool? * Should I stop the logger before? Thanks! |
|
From: Laurent W. <lw...@hy...> - 2010-07-06 12:10:06
|
On Tue, 6 Jul 2010 13:07:28 +0300 Stas Oskin <sta...@gm...> wrote: > Hi. > > Is there any recommended procedure how to back-up MFS meta-data? > > I.e: > > * Where to backup from - master or logger? > * Backup the flat files, or use some mfs tool? > * Should I stop the logger before? > > Thanks! Hmmmmm I guess I'll answer a bit aside your question. I'd use drbd as a backup :) Anyway, it may be possible to deploy several metaloggers, but I don't know the code enough to know if it works. Could be a nice feature to implement, until high availability is available by default. -- Laurent Wandrebeck HYGEOS, Earth Observation Department / Observation de la Terre Euratechnologies 165 Avenue de Bretagne 59000 Lille, France tel: +33 3 20 08 24 98 http://www.hygeos.com GPG fingerprint/Empreinte GPG: F5CA 37A4 6D03 A90C 7A1D 2A62 54E6 EF2C D17C F64C |
|
From: Laurent W. <lw...@hy...> - 2010-07-06 12:42:32
|
On Tue, 6 Jul 2010 15:12:46 +0300 Stas Oskin <sta...@gm...> wrote: > I'm actually going to try HA procedure outlined in MFS docs later on. Nice, don't forget to take notes, it'll help others :) > > The question right now, if I can just backup the entire /var/mfs directory, > on master or on logger. I really can't answer for sure. Michal ? -- Laurent Wandrebeck HYGEOS, Earth Observation Department / Observation de la Terre Euratechnologies 165 Avenue de Bretagne 59000 Lille, France tel: +33 3 20 08 24 98 http://www.hygeos.com GPG fingerprint/Empreinte GPG: F5CA 37A4 6D03 A90C 7A1D 2A62 54E6 EF2C D17C F64C |
|
From: Michał B. <mic...@ge...> - 2010-07-07 10:20:36
|
Hi! Master server flushes metadata kept in RAM to the metadata.mfs.back binary file every hour on the hour (xx:00). So probably the best moment would be to copy the metadata file every hour on the half hour (30 minutes after the dump). Then you would maximally lose 1.5h of data. You can choose any method of copying - cp, scp, rsync, etc. Metalogger receives the metadata file every 24 hours so it is better to copy metadata from the master server. Having metadata backed up you won't see newly created files, files to which some data was appended would come back to the previous size, there would still exist deleted files and files you changed names of would come back to their previous names (and locations). But still you would have information and access to all the files created in the X past years. If you have not read this entry, we strongly recommend it: http://www.moosefs.org/news-reader/items/metadata-ins-and-outs.html Kind regards Michał From: Stas Oskin [mailto:sta...@gm...] Sent: Tuesday, July 06, 2010 12:07 PM To: moo...@li... Subject: [Moosefs-users] Backing up MFS metadata Hi. Is there any recommended procedure how to back-up MFS meta-data? I.e: * Where to backup from - master or logger? * Backup the flat files, or use some mfs tool? * Should I stop the logger before? Thanks! |
|
From: Stas O. <sta...@gm...> - 2010-07-08 19:03:51
|
Hi. Master server flushes metadata kept in RAM to the metadata.mfs.back binary > file every hour on the hour (xx:00). So probably the best moment would be to > copy the metadata file every hour on the half hour (30 minutes after the > dump). Then you would maximally lose 1.5h of data. You can choose any method > of copying – cp, scp, rsync, etc. Metalogger receives the metadata file > every 24 hours so it is better to copy metadata from the master server. > > Can I also save the change logs, and replay them into meta-data? How much time it will be able recover up to point of crash? > Having metadata backed up you won’t see newly created files, files to > which some data was appended would come back to the previous size, there > would still exist deleted files and files you changed names of would come > back to their previous names (and locations). But still you would have > information and access to all the files created in the X past years. > > > > If you have not read this entry, we strongly recommend it: > > http://www.moosefs.org/news-reader/items/metadata-ins-and-outs.html > > Thanks, now it much more clear. Speaking of crashes, will the master detect it has crashed, and will try to replay the logs itself? Or it's recommended to use the script described at the end of the entry, to always check for repayable logs? Regards. |
|
From: Fabien G. <fab...@gm...> - 2010-07-08 23:29:49
|
Hi Stas, 2010/7/8 Stas Oskin <sta...@gm...> > http://www.moosefs.org/news-reader/items/metadata-ins-and-outs.html >> >> > If you have not read this entry, we strongly recommend it:Thanks, now it > much more clear. > > Speaking of crashes, will the master detect it has crashed, and will try to > replay the logs itself? > Or it's recommended to use the script described at the end of the entry, to > always check for repayable logs? > If the mfsmaster process crashes, you won't be able to start it again unless you use "mfsmetarestore" first. Best regards, Fabien |
|
From: Stas O. <sta...@gm...> - 2010-07-09 08:15:35
|
Can I also save the change logs, and replay them into meta-data? How much time it will be able recover up to point of crash? Regards. |
|
From: Fabien G. <fab...@gm...> - 2010-07-09 08:34:49
|
Hello Stas, On Fri, Jul 9, 2010 at 10:15 AM, Stas Oskin <sta...@gm...> wrote: > Can I also save the change logs, and replay them into meta-data? > How much time it will be able recover up to point of crash? > Yes, you must save everything in /var/mfs/ (or wherever you placed it). When you launch "mfsmetarestore -a", it will use the last metadata dumpled file and replay changelogs on it. From that, you'll get an up-to-date metadata file. As for the time it takes, it depends on the number of chunks you have, and the hardware server you use (CPU + HDD speed). For example in our case (15 million chunks, 6 GB of metadata), it takes between 1 and 2 minutes on a Xeon processor. Fabien |
|
From: Stas O. <sta...@gm...> - 2010-07-09 13:29:55
|
Hi. > As for the time it takes, it depends on the number of chunks you have, and > the hardware server you use (CPU + HDD speed). For example in our case (15 > million chunks, 6 GB of metadata), it takes between 1 and 2 minutes on a > Xeon processor. > > I actually meant, how much time backwards could be recovered by replaying all the logs? Michael said that backup log is created every hour so up to 1.5 hour can be potentially lost. If all the logs are replayed, can the data be consistent up to the moment of crash? Regards. |
|
From: Fabien G. <fab...@gm...> - 2010-07-11 20:54:58
|
Hi, On Fri, Jul 9, 2010 at 2:27 PM, Stas Oskin <sta...@gm...> wrote: > As for the time it takes, it depends on the number of chunks you have, and >> the hardware server you use (CPU + HDD speed). For example in our case (15 >> million chunks, 6 GB of metadata), it takes between 1 and 2 minutes on a >> Xeon processor. >> > > I actually meant, how much time backwards could be recovered by replaying > all the logs? > > Michael said that backup log is created every hour so up to 1.5 hour can be > potentially lost. > If all the logs are replayed, can the data be consistent up to the moment > of crash? > By default, logs are written to disk every 1 hour. So you can loose up to 1 hour if your mfsmaster crashes just before a new disk flush occurs (everything is stored in memory for performance reasons : http://www.moosefs.org/moosefs-faq.html#cluster). Fabien |
|
From: Michał B. <mic...@ge...> - 2010-07-15 08:33:13
|
You can backup changelog files (you would need just the two newest ones - "0" and "1"). You can make these backups even every minute. So you can have potentially lost information for about 1-2 minutes. But the question is - why to back up changelogs manually? Metalogger machines are dedicated to this. You can have as many metalogger machines on the network as you like. And metalogger process can be run on any computer, even an older one. Regards Michał From: Stas Oskin [mailto:sta...@gm...] Sent: Friday, July 09, 2010 2:27 PM To: Fabien Germain Cc: moo...@li...; Michał Borychowski Subject: Re: [Moosefs-users] Backing up MFS metadata Hi. As for the time it takes, it depends on the number of chunks you have, and the hardware server you use (CPU + HDD speed). For example in our case (15 million chunks, 6 GB of metadata), it takes between 1 and 2 minutes on a Xeon processor. I actually meant, how much time backwards could be recovered by replaying all the logs? Michael said that backup log is created every hour so up to 1.5 hour can be potentially lost. If all the logs are replayed, can the data be consistent up to the moment of crash? Regards. |
|
From: Roast <zha...@gm...> - 2010-07-16 11:45:36
|
Another question. Does master server sync the change log to metalogger server in real time? And if so, we can restore the full meta info from the metalogger server. Am I right? 2010/7/15 Michał Borychowski <mic...@ge...> > You can backup changelog files (you would need just the two newest ones – > “0” and “1”). You can make these backups even every minute. So you can have > potentially lost information for about 1-2 minutes. > > > > But the question is – why to back up changelogs manually? Metalogger > machines are dedicated to this. You can have as many metalogger machines on > the network as you like. And metalogger process can be run on any computer, > even an older one. > > > > > > Regards > > Michał > > > > > > > > *From:* Stas Oskin [mailto:sta...@gm...] > *Sent:* Friday, July 09, 2010 2:27 PM > *To:* Fabien Germain > *Cc:* moo...@li...; Michał Borychowski > *Subject:* Re: [Moosefs-users] Backing up MFS metadata > > > > Hi. > > > As for the time it takes, it depends on the number of chunks you have, and > the hardware server you use (CPU + HDD speed). For example in our case (15 > million chunks, 6 GB of metadata), it takes between 1 and 2 minutes on a > Xeon processor. > > > I actually meant, how much time backwards could be recovered by replaying > all the logs? > > Michael said that backup log is created every hour so up to 1.5 hour can be > potentially lost. > If all the logs are replayed, can the data be consistent up to the moment > of crash? > > Regards. > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > -- The time you enjoy wasting is not wasted time! |
|
From: Michał B. <mic...@ge...> - 2010-07-19 06:45:09
|
Please read this article: http://www.moosefs.org/news-reader/items/metadata-ins-and-outs.html The most important part for you is here: The metalogger will download the metadata.mfs.back file on a regular basis (by default every 24 hours) from the master server. The downloaded file is saved with the file name metadata_ml.mfs.back. Similarly, it also continuously receives the current changes from the master server and writes them into its own text change log named changelog_ml.0.mfs. Where these files are also rotated every hour up to the configured maximum number of change log files (see man mfsmetalogger.cfg). Yes, you can fully restore metadata from files saved by metalogger. If you need any further assistance please let us know. Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 From: Roast [mailto:zha...@gm...] Sent: Friday, July 16, 2010 1:45 PM To: Michał Borychowski Cc: Stas Oskin; moosefs-users Subject: Re: [Moosefs-users] Backing up MFS metadata Another question. Does master server sync the change log to metalogger server in real time? And if so, we can restore the full meta info from the metalogger server. Am I right? 2010/7/15 Michał Borychowski <mic...@ge...> You can backup changelog files (you would need just the two newest ones – “0” and “1”). You can make these backups even every minute. So you can have potentially lost information for about 1-2 minutes. But the question is – why to back up changelogs manually? Metalogger machines are dedicated to this. You can have as many metalogger machines on the network as you like. And metalogger process can be run on any computer, even an older one. Regards Michał From: Stas Oskin [mailto:sta...@gm...] Sent: Friday, July 09, 2010 2:27 PM To: Fabien Germain Cc: moo...@li...; Michał Borychowski Subject: Re: [Moosefs-users] Backing up MFS metadata Hi. As for the time it takes, it depends on the number of chunks you have, and the hardware server you use (CPU + HDD speed). For example in our case (15 million chunks, 6 GB of metadata), it takes between 1 and 2 minutes on a Xeon processor. I actually meant, how much time backwards could be recovered by replaying all the logs? Michael said that backup log is created every hour so up to 1.5 hour can be potentially lost. If all the logs are replayed, can the data be consistent up to the moment of crash? Regards. ------------------------------------------------------------------------------ This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users -- The time you enjoy wasting is not wasted time! |
|
From: Roast <zha...@gm...> - 2010-07-19 08:34:26
|
Thanks,Michał Borychowski. 2010/7/19 Michał Borychowski <mic...@ge...> > Please read this article: > http://www.moosefs.org/news-reader/items/metadata-ins-and-outs.html > > > > The most important part for you is here: > > *The metalogger will download the metadata.mfs.back file on a regular > basis (by default every 24 hours) from the master server. The downloaded > file is saved with the file name metadata_ml.mfs.back. Similarly, it also > continuously receives the current changes from the master server and writes > them into its own text change log named changelog_ml.0.mfs. Where these > files are also rotated every hour up to the configured maximum number of > change log files (see man mfsmetalogger.cfg).* > > > > Yes, you can fully restore metadata from files saved by metalogger. > > > > If you need any further assistance please let us know. > > > > > > Kind regards > > Michał Borychowski > > MooseFS Support Manager > > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > Gemius S.A. > > ul. Wołoska 7, 02-672 Warszawa > > Budynek MARS, klatka D > > Tel.: +4822 874-41-00 > > Fax : +4822 874-41-01 > > > > *From:* Roast [mailto:zha...@gm...] > *Sent:* Friday, July 16, 2010 1:45 PM > *To:* Michał Borychowski > *Cc:* Stas Oskin; moosefs-users > > *Subject:* Re: [Moosefs-users] Backing up MFS metadata > > > > Another question. > > Does master server sync the change log to metalogger server in real time? > > And if so, we can restore the full meta info from the metalogger server. Am > I right? > > 2010/7/15 Michał Borychowski <mic...@ge...> > > You can backup changelog files (you would need just the two newest ones - > "0" and "1"). You can make these backups even every minute. So you can have > potentially lost information for about 1-2 minutes. > > > > But the question is - why to back up changelogs manually? Metalogger > machines are dedicated to this. You can have as many metalogger machines on > the network as you like. And metalogger process can be run on any computer, > even an older one. > > > > > > Regards > > Michał > > > > > > > > *From:* Stas Oskin [mailto:sta...@gm...] > *Sent:* Friday, July 09, 2010 2:27 PM > *To:* Fabien Germain > *Cc:* moo...@li...; Michał Borychowski > *Subject:* Re: [Moosefs-users] Backing up MFS metadata > > > > Hi. > > > As for the time it takes, it depends on the number of chunks you have, and > the hardware server you use (CPU + HDD speed). For example in our case (15 > million chunks, 6 GB of metadata), it takes between 1 and 2 minutes on a > Xeon processor. > > > I actually meant, how much time backwards could be recovered by replaying > all the logs? > > Michael said that backup log is created every hour so up to 1.5 hour can be > potentially lost. > If all the logs are replayed, can the data be consistent up to the moment > of crash? > > Regards. > > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > > > -- > The time you enjoy wasting is not wasted time! > -- The time you enjoy wasting is not wasted time! |
|
From: Stas O. <sta...@gm...> - 2010-07-29 22:44:49
|
Hi. Getting back to this to clarify the matter: > > You can backup changelog files (you would need just the two newest ones – >> “0” and “1”). You can make these backups even every minute. So you can have >> potentially lost information for about 1-2 minutes. >> >> According to article only "0" file contains the ongoing info, so why the "1" is needed? How often the checkpoint files are updated? > >> > > But the question is – why to back up changelogs manually? Metalogger >> machines are dedicated to this. You can have as many metalogger machines on >> the network as you like. And metalogger process can be run on any computer, >> even an older one. >> >> Perhaps I can put metaloggers on the chunkservers? How much resources the metalogger takes? Regards. |
|
From: Michał B. <mic...@ge...> - 2010-08-06 12:13:18
|
From: Stas Oskin [mailto:sta...@gm...] Sent: Friday, July 30, 2010 12:44 AM To: Roast Cc: moosefs-users; Michał Borychowski Subject: Re: [Moosefs-users] Backing up MFS metadata Hi. Getting back to this to clarify the matter: You can backup changelog files (you would need just the two newest ones - "0" and "1"). You can make these backups even every minute. So you can have potentially lost information for about 1-2 minutes. According to article only "0" file contains the ongoing info, so why the "1" is needed? How often the checkpoint files are updated? [MB] In most cases 0 would be enough. But there may happen some "border" cases and it is more secure to use also file 1. But the question is - why to back up changelogs manually? Metalogger machines are dedicated to this. You can have as many metalogger machines on the network as you like. And metalogger process can be run on any computer, even an older one. Perhaps I can put metaloggers on the chunkservers? How much resources the metalogger takes? [MB] You can easily put a metalogger process on a chunkserver. The resources needed for metalogger are not very high and I am sure your chunkservers have quite a lot spare processor power. Regards Michał |
|
From: Stas O. <sta...@gm...> - 2010-07-29 22:53:33
|
Sorry, forgot to forward to the list. ---------- Forwarded message ---------- From: Stas Oskin <sta...@gm...> Date: 2010/7/30 Subject: Re: [Moosefs-users] Backing up MFS metadata To: Michał Borychowski <mic...@ge...> Question. > > > Yes, you can fully restore metadata from files saved by metalogger. > > How far beyond master the meta server is? Where it's better to backup, on master or on metalogger? Thanks. |
|
From: Michał B. <mic...@ge...> - 2010-08-06 12:17:29
|
Hi! ---------- Forwarded message ---------- From: Stas Oskin < <mailto:sta...@gm...> sta...@gm...> Date: 2010/7/30 Subject: Re: [Moosefs-users] Backing up MFS metadata To: Michał Borychowski < <mailto:mic...@ge...> mic...@ge...> Question. Yes, you can fully restore metadata from files saved by metalogger. How far beyond master the meta server is? [MB] What do you mean by this question? Where it's better to backup, on master or on metalogger? [MB] Master on its own creates binary metadata.mfs.back and text changelogs and you cannot disable it. Metalogger is optionally. If you ask about recovering from the backup if the master server had only power outage and has no hdd problems, you can restore metadata from the master itself. In a more serious damages to master you would need to restore meta from the metalogger. Please read this blog entry: http://www.moosefs.org/news-reader/items/metadata-ins-and-outs.html Regards Michał Thanks. |
|
From: Stas O. <sta...@gm...> - 2010-08-09 15:18:51
|
Hi. After reading the "Metadata files ins and outs" article, and this email thread, I'm still left with several questions that I would like to clarify: 1) Does mfsmaster dumps to checkpoint files an up-to-date representation of the memory, at periods of 1-2 minutes maximum? Meaning, backing up the /var/mfs directory and restoring it back, will provide me with a 1-2 minutes maximum of data loss? 2) Does metalogger pull an up-to-date version of master checkpoint files as well? If I move the master to metalogger and restore the checkpoint files, it will provide me with a 1-2 minutes maximum of data loss as well? 3) Any benefit of lowering the default 24 hours download frequency? Or the streamed updates from master to metalogger will be enough? 4) In case I do want to backup to external storage (NAS for example), which would be the better source machine - master (sounds more logical) or metalogger? 5) If I have 3 metaloggers for example, how much load it will incur on the master server? Thanks in advance for answers, as it would clarify the backup topic for me completely! Regards. |
|
From: Michał B. <mic...@ge...> - 2010-08-11 07:48:11
|
From: Stas Oskin [mailto:sta...@gm...] Sent: Monday, August 09, 2010 5:18 PM To: moosefs-users Subject: [Moosefs-users] Fwd: Fwd: Backing up MFS metadata Hi. After reading the "Metadata files ins and outs" article, and this email thread, I'm still left with several questions that I would like to clarify: 1) Does mfsmaster dumps to checkpoint files an up-to-date representation of the memory, at periods of 1-2 minutes maximum? Meaning, backing up the /var/mfs directory and restoring it back, will provide me with a 1-2 minutes maximum of data loss? [MB] The "mfsmaster.mfs.back" file by default is saved every hour but "changelog.0.mfs" is saved on the fly so the delay is just in milliseconds needed for the writing process. So the data loss of 1-2 minutes would be really the most negative scenario. 2) Does metalogger pull an up-to-date version of master checkpoint files as well? If I move the master to metalogger and restore the checkpoint files, it will provide me with a 1-2 minutes maximum of data loss as well? [MB] This delay should also be close to 0 (on condition that metalogger has been working all the time). The file metadata_ml.mfs.back is dumped every 24hours but changelogs are saved on the fly. Each change that master saves to a file is also sent to metalogger. In case of the master server failure you would lose data stuck in buffer of outgoing network interface, times also in milliseconds. 3) Any benefit of lowering the default 24 hours download frequency? Or the streamed updates from master to metalogger will be enough? [MB] When the metalogger restarts you would have incomplete change logs (because when it is down it doesn't save the data and after restart master server doesn't send to it the not yet saved changes - but this behavior will be changed soon). So if you set the frequency to eg. 6 hours you decrease probability of "holes" in changelogs in case metalogger is down. 4) In case I do want to backup to external storage (NAS for example), which would be the better source machine - master (sounds more logical) or metalogger? [MB] Yes, master. 5) If I have 3 metaloggers for example, how much load it will incur on the master server? [MB] For the master the load is really negligible. If you have any other doubts please let us know. Regards Michal Thanks in advance for answers, as it would clarify the backup topic for me completely! Regards. |
|
From: Stas O. <sta...@gm...> - 2010-08-11 08:47:08
|
Hi. Thanks a lot - much appreciate your answers :). * * > > *If you have any other doubts please let us know.* > > * * > > * > * > > ** > Just one question - what the main benefit of keeping meta-loggers, considering the meta-data can be backed up to external additional medium? Is this the option of having high-availability, where the meta-logger could be quickly brought down, and master brought up using the meta-logger data? Regards. |
|
From: Michał B. <mic...@ge...> - 2010-08-11 09:33:35
|
Yes, we can think of metalogger as a "spare" master. When it fails, metalogger can easily and quickly take over its functions. The process of switching to metalogger by using CARP is described here: http://www.moosefs.org/mini-howtos.html#redundant-master (we have not yet tried Heartbeat and as far as we know nobody from the community have done it yet neither). And with metalogger it is just easy to back up the metadata on the fly. Of course, you can additionally back it up to external resources, but you need to prepare extra scripts. Regards Michal From: Stas Oskin [mailto:sta...@gm...] Sent: Wednesday, August 11, 2010 10:47 AM To: Michał Borychowski Cc: moosefs-users Subject: Re: [Moosefs-users] Fwd: Fwd: Backing up MFS metadata Hi. Thanks a lot - much appreciate your answers :). If you have any other doubts please let us know. Just one question - what the main benefit of keeping meta-loggers, considering the meta-data can be backed up to external additional medium? Is this the option of having high-availability, where the meta-logger could be quickly brought down, and master brought up using the meta-logger data? Regards. |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X