Menu
▾
▴
moosefs-users
|
From: Bán M. <ba...@vo...> - 2010-09-05 17:25:35
|
Hi, recently I've installed an mfs cluster with several chunk servers, one master and two metaloggers. The metaloggers download the changelog files hourly. Is it normal? Can I set them to download the recent changelogs from the master more frequently or I have to write own scripts to keep up-to date the metalogger servers? my second question, why the metalogger do not download the metadata.mfs.back immediately after it has been started? On my metalogger servers the first metadata files appeared after a day. If is this the normal way, I can't agree with this because after a big failure on the first day I can't restore the master server from the metalogger. Thanks, Miklos |
|
From: Michał B. <mic...@ge...> - 2010-09-06 11:42:58
|
Hi! Metaloggers should continuously receive the current changes from the master server and write them into its own text change logs named changelog_ml.0.mfs. How do you know that in your system they are save hourly? Don't they increment with every change in the filesystem? Regarding your second question - yes, this is right. For now, metalogger doesn't download the metadata file upon starting. We know about this shortcoming and we'll fix the behaviour soon. Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 -----Original Message----- From: Bán Miklós [mailto:ba...@vo...] Sent: Sunday, September 05, 2010 7:06 PM To: moo...@li... Subject: [Moosefs-users] changelogs on the metalogger Hi, recently I've installed an mfs cluster with several chunk servers, one master and two metaloggers. The metaloggers download the changelog files hourly. Is it normal? Can I set them to download the recent changelogs from the master more frequently or I have to write own scripts to keep up-to date the metalogger servers? my second question, why the metalogger do not download the metadata.mfs.back immediately after it has been started? On my metalogger servers the first metadata files appeared after a day. If is this the normal way, I can't agree with this because after a big failure on the first day I can't restore the master server from the metalogger. Thanks, Miklos ---------------------------------------------------------------------------- -- This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Bán M. <ba...@vo...> - 2010-09-06 14:16:32
|
On Mon, 6 Sep 2010 13:42:30 +0200 Michał Borychowski <mic...@ge...> wrote: > Hi! > > Metaloggers should continuously receive the current changes from the > master server and write them into its own text change logs named > changelog_ml.0.mfs. > > How do you know that in your system they are save hourly? Don't they > increment with every change in the filesystem? Yes, exactly. There is no new lines on metalogger server's changelog_ml.0.mfs, while the master's changelog.0.mfs updating. Nevertheless the sessions_ml.mfs is updating continuously. My server version is 1.6.7 and it is running on Ubuntu Jaunty. Here is a short log part from a metalogger: Sep 6 14:02:00 fekete mfsmetalogger[30553]: sessions downloaded 2374B/0.000435s (5.457 MB/s) Sep 6 14:02:02 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/09/chunk_0000000000000009_00000001.mfs Sep 6 14:02:12 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/0C/chunk_000000000000000C_00000001.mfs Sep 6 14:02:22 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/0E/chunk_000000000000000E_00000001.mfs Sep 6 14:02:32 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/10/chunk_0000000000000010_00000001.mfs Sep 6 14:02:42 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/0A/chunk_000000000000000A_00000001.mfs Sep 6 14:02:52 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/0F/chunk_000000000000000F_00000001.mfs Sep 6 14:03:00 fekete mfsmetalogger[30553]: sessions downloaded 2374B/0.000570s (4.165 MB/s) > > Regarding your second question - yes, this is right. For now, > metalogger doesn't download the metadata file upon starting. We know > about this shortcoming and we'll fix the behaviour soon. > > Thanx. &Miklos |
|
From: Travis H. <tra...@tr...> - 2010-09-08 18:51:57
|
On 2010-09-06, at 10:16 AM, Bán Miklós wrote: >> >> Metaloggers should continuously receive the current changes from the >> master server and write them into its own text change logs named >> changelog_ml.0.mfs. >> >> How do you know that in your system they are save hourly? Don't they >> increment with every change in the filesystem? > > Yes, exactly. There is no new lines on metalogger server's > changelog_ml.0.mfs, while the master's changelog.0.mfs updating. > Nevertheless the sessions_ml.mfs is updating continuously. > My server version is 1.6.7 and it is running on Ubuntu Jaunty. > Did you mean 1.6.17 ?,, or 1.5.7. (There was no 1.6.7 release?) I was concerned, if you were not running the latest server and metalogger, perhaps this would be a bug that has already been fixed. Travis |
|
From: Bán M. <ba...@vo...> - 2010-09-08 20:33:12
|
Hi,
Sorry, I was wrong. The version no is 1.6.17.
I've patched the mfsmetalogger/masterconn.c file with the following
lines:
140a141
>
187a189,192
> if (eptr->logfd) {
> fclose(eptr->logfd);
> eptr->logfd = NULL;
> }
which have solved that problem. So this is the bug fix. There was no
fclose after opening the changelog_ml.0.mfs.
Miklos
On Wed, 8 Sep 2010 14:33:39 -0400
Travis Hein <tra...@tr...> wrote:
>
> On 2010-09-06, at 10:16 AM, Bán Miklós wrote:
>
> >>
> >> Metaloggers should continuously receive the current changes from
> >> the master server and write them into its own text change logs
> >> named changelog_ml.0.mfs.
> >>
> >> How do you know that in your system they are save hourly? Don't
> >> they increment with every change in the filesystem?
> >
> > Yes, exactly. There is no new lines on metalogger server's
> > changelog_ml.0.mfs, while the master's changelog.0.mfs updating.
> > Nevertheless the sessions_ml.mfs is updating continuously.
> > My server version is 1.6.7 and it is running on Ubuntu Jaunty.
> >
>
> Did you mean 1.6.17 ?,, or 1.5.7. (There was no 1.6.7 release?)
>
> I was concerned, if you were not running the latest server and
> metalogger, perhaps this would be a bug that has already been fixed.
>
> Travis
>
>
>
>
> ------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
|
|
From: Michał B. <mic...@ge...> - 2010-09-13 10:18:41
|
Hi!
Unfortunately this can cause other problems. If after each save you close
the file and later open it, you'll blow the metalogger. Maybe you would
rather use: fflush(eptr->logfd). We are further investigating why the
changelogs may not save on metalogger.
Regards
Michał
-----Original Message-----
From: Bán Miklós [mailto:ba...@vo...]
Sent: Wednesday, September 08, 2010 10:33 PM
To: moo...@li...
Subject: Re: [Moosefs-users] changelogs on the metalogger
Hi,
Sorry, I was wrong. The version no is 1.6.17.
I've patched the mfsmetalogger/masterconn.c file with the following
lines:
140a141
>
187a189,192
> if (eptr->logfd) {
> fclose(eptr->logfd);
> eptr->logfd = NULL;
> }
which have solved that problem. So this is the bug fix. There was no
fclose after opening the changelog_ml.0.mfs.
Miklos
On Wed, 8 Sep 2010 14:33:39 -0400
Travis Hein <tra...@tr...> wrote:
>
> On 2010-09-06, at 10:16 AM, Bán Miklós wrote:
>
> >>
> >> Metaloggers should continuously receive the current changes from
> >> the master server and write them into its own text change logs
> >> named changelog_ml.0.mfs.
> >>
> >> How do you know that in your system they are save hourly? Don't
> >> they increment with every change in the filesystem?
> >
> > Yes, exactly. There is no new lines on metalogger server's
> > changelog_ml.0.mfs, while the master's changelog.0.mfs updating.
> > Nevertheless the sessions_ml.mfs is updating continuously.
> > My server version is 1.6.7 and it is running on Ubuntu Jaunty.
> >
>
> Did you mean 1.6.17 ?,, or 1.5.7. (There was no 1.6.7 release?)
>
> I was concerned, if you were not running the latest server and
> metalogger, perhaps this would be a bug that has already been fixed.
>
> Travis
>
>
>
>
>
----------------------------------------------------------------------------
--
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
----------------------------------------------------------------------------
--
This SF.net Dev2Dev email is sponsored by:
Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users
|
|
From: Michał B. <mic...@ge...> - 2010-09-13 11:29:24
|
Hello again Your changelog_ml.0.mfs never got filled? Saving in metalogger takes place by using write buffer which is flushed by the operating system. This is much quicker. It also means that in case of metalogger failure it will be missing some recent changes. But in case of master failure, after shutting down the metalogger process, system would write all data waiting in the buffer and metadata file on metalogger machine would be complete. We've made automatic data flushing every second which won't affect performance but possilby this behaviour would be more clear to users. Kind regards Michał -----Original Message----- From: Bán Miklós [mailto:ba...@vo...] Sent: Monday, September 06, 2010 4:16 PM To: moo...@li... Subject: Re: [Moosefs-users] changelogs on the metalogger On Mon, 6 Sep 2010 13:42:30 +0200 Michał Borychowski <mic...@ge...> wrote: > Hi! > > Metaloggers should continuously receive the current changes from the > master server and write them into its own text change logs named > changelog_ml.0.mfs. > > How do you know that in your system they are save hourly? Don't they > increment with every change in the filesystem? Yes, exactly. There is no new lines on metalogger server's changelog_ml.0.mfs, while the master's changelog.0.mfs updating. Nevertheless the sessions_ml.mfs is updating continuously. My server version is 1.6.7 and it is running on Ubuntu Jaunty. Here is a short log part from a metalogger: Sep 6 14:02:00 fekete mfsmetalogger[30553]: sessions downloaded 2374B/0.000435s (5.457 MB/s) Sep 6 14:02:02 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/09/chunk_0000000000000009_00000001.mfs Sep 6 14:02:12 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/0C/chunk_000000000000000C_00000001.mfs Sep 6 14:02:22 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/0E/chunk_000000000000000E_00000001.mfs Sep 6 14:02:32 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/10/chunk_0000000000000010_00000001.mfs Sep 6 14:02:42 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/0A/chunk_000000000000000A_00000001.mfs Sep 6 14:02:52 fekete mfschunkserver[30562]: testing chunk: /mnt/mfs_hd1/0F/chunk_000000000000000F_00000001.mfs Sep 6 14:03:00 fekete mfsmetalogger[30553]: sessions downloaded 2374B/0.000570s (4.165 MB/s) > > Regarding your second question - yes, this is right. For now, > metalogger doesn't download the metadata file upon starting. We know > about this shortcoming and we'll fix the behaviour soon. > > Thanx. &Miklos ---------------------------------------------------------------------------- -- This SF.net Dev2Dev email is sponsored by: Show off your parallel programming skills. Enter the Intel(R) Threading Challenge 2010. http://p.sf.net/sfu/intel-thread-sfd _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Bán M. <ba...@vo...> - 2010-09-06 14:17:21
|
Hi,
I've found something:
in the masterconn.c
void masterconn_metachanges_log ...
the fprintf just write the changelogs when I stop or restart the process
178 if (eptr->logfd==NULL) {
179 eptr->logfd = fopen("changelog_ml.0.mfs","a");
180 }
181
182 data++;
183 version = get64bit(&data);
184 if (eptr->logfd) {
185 fprintf(eptr->logfd,"%"PRIu64": %s\n",version,data);
186 } else {
187 syslog(LOG_NOTICE,"lost MFS change %"PRIu64":
%s",version,data); 188 }
I've append this lines:
if (eptr->logfd) {
fclose(eptr->logfd);
eptr->logfd = NULL;
}
It seems to work, but I'm not sure is it a good way or not, I did not
read the source code thoroughly.
Second problem:
If I stop, wait and start the metalogger server, the changelog will be
inconsistent. The changes from the master server, while the metalogger
was being stopped, not arriving after I turn it back on.
Is there any recommendation how can I manage the metalogger servers
to keep coherent change logs after failure situation?
Miklos
On Mon, 6 Sep 2010 13:42:30 +0200
Michał Borychowski <mic...@ge...> wrote:
> Hi!
>
> Metaloggers should continuously receive the current changes from the
> master server and write them into its own text change logs named
> changelog_ml.0.mfs.
>
> How do you know that in your system they are save hourly? Don't they
> increment with every change in the filesystem?
>
> Regarding your second question - yes, this is right. For now,
> metalogger doesn't download the metadata file upon starting. We know
> about this shortcoming and we'll fix the behaviour soon.
>
>
|
|
From: Bán M. <ba...@vo...> - 2010-09-07 09:26:31
|
On Mon, 6 Sep 2010 16:17:08 +0200 Bán Miklós <ba...@vo...> wrote: > Second problem: > If I stop, wait and start the metalogger server, the changelog will be > inconsistent. The changes from the master server, while the metalogger > was being stopped, not arriving after I turn it back on. > Is there any recommendation how can I manage the metalogger servers > to keep coherent change logs after failure situation? Hi, I wrote two initiation script. One start the mfsmaster and the other the mfsmetalogger. It is a tcp based client server socket pair. If I restart a metalogger server it is sending the last changelog line to the masterserver. If that line differ from the masters's last changelog line the masterserver restarting and send back the new metadata.mfs to metalogger server. The metalogger server after receiving a new metadata file clear the changelogs and restarting or starting. It works and I think it is not a wrong way, but there is a problem: If there is a file operations while I restart the masterserver, what happening? Is there a lock mechanism on the filesystem while the mfsmaster is restarting? more.. I think there is no this client-server process to manage this kind of metadata updating, because it is possible to implement into the main code. Opinion? It is too difficult for me. Sorry. Would it be interesting to send these two script to the maillist? &Miklos |
|
From: Bán M. <ba...@vo...> - 2010-09-07 09:53:15
|
On Tue, 7 Sep 2010 11:44:46 +0200
Michał Borychowski <mic...@ge...> wrote:
> Please, send these two scripts to the list so that we can have a look
> at them
>
>
I start these scripts from the init.d
on the masterserver:
#!/usr/bin/perl
use strict;
use IO::Socket;
use Net::hostent;
my $mfsserver = '/usr/sbin/mfsmaster';
my $port = 7990;
my $data_path = "/var/lib/mfs";
my ($server, $client, $hostinfo, $n, $foo, $buf);
$server = IO::Socket::INET->new( Proto => 'tcp',
LocalPort => $port,
Listen => SOMAXCONN,
Reuse => 1);
die "can't setup server" unless $server;
print "[Server $0 accepting clients]\n";
system("$mfsserver start");
print "[mfsmaster server has been started]\n";
while ($client = $server->accept()) {
$client->autoflush(1);
$hostinfo = gethostbyaddr($client->peeraddr);
while ( <$client>) {
next unless /\S/;
if (/^close$/i) { last; }
elsif (/^(meta:)/) {
$n = $';
chomp($n);
$foo = `tail -1 $data_path/changelog.0.mfs`;
chomp($foo);
if ($n ne $foo) {
system("$mfsserver restart");
open (FH,"<$data_path/metadata.mfs.back") or die $!;
binmode(FH);
while (sysread(FH, $buf, 1024)) {
syswrite($client, $buf, length($buf));
}
close(FH) or die $!;
}
} else {
close $client;
}
}
close $client;
}
on the metalogger server:
#!/usr/bin/perl
use strict;
use IO::Socket;
my $mfslogger = '/usr/sbin/mfsmetalogger';
my $data_path = "/var/lib/mfs";
my $host = "mfsmaster";
my $port = 7990;
my ($kidpid, $handle, $buf, $c, $last_line);
$last_line = `tail -1 $data_path/changelog_ml.0.mfs`;
$handle = IO::Socket::INET->new(Proto => "tcp",
PeerAddr => $host,
PeerPort => $port)
or die "can't connect to port $port on $host: $!";
$handle->autoflush(1);
die "can't fork: $!" unless defined($kidpid = fork());
if ($kidpid) {
open (FH,">$data_path/metadata_ml.mfs.back") or die $!;;
binmode(FH);
$c = 0;
while (sysread($handle, $buf, 1024)) {
syswrite(FH, $buf, length($buf));
$c = $c+1;
}
close(FH);
if ($c) {
unlink glob("$data_path/changelog_ml*");
system("touch $data_path/changelog_ml.0.mfs");
exec("$mfslogger $ARGV[0]");
}
kill("TERM", $kidpid);
}
else {
print $handle "meta:$last_line\n";
print $handle "close\n";
}
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X