moosefs-users — Mailing list for MooseFS Users

[Moosefs-users] Recovery and logs

[Reinis R. <r...@ro...>]

Hello,
have noticed such log lines:

Nov 12 16:31:05 mfmaster215 mfsmaster[3925]: chunk 0000000002428757 has only 
invalid copies (1) - please repair it manually
Nov 12 16:31:05 mfmaster215 mfsmaster[3925]: chunk 
0000000002428757_00000002 - invalid copy on (192.168.0.221 - ver:00000001)
Nov 12 16:31:05 mfmaster215 mfsmaster[3925]: chunk 
0000000002428757_00000002 - invalid copy on (192.168.0.221 - ver:00000001)

What does 'please repair it manually' mean in this case? Since I couldnt 
find any direct tools to interact with chunks itself.

The mfscgiserv shows that there are kinda 8 files with no copies - is there 
a way to find out what files are theese?


Second - what do theese log lines mean?

Nov 12 16:32:46 mfmaster215 mfsmount[15074]: file: 41248800, index: 0, 
chunk: 37952708, version: 1 - writeworker: connection with (D5AF4BDB:9422) 
was timed out (unfinished writes: 1; try counter: 1)
Nov 12 16:32:48 mfmaster215 mfsmount[15074]: file: 41248800, index: 0, 
chunk: 37952708, version: 1 - writeworker: connection with (D5AF4BDA:9422) 
was timed out (unfinished writes: 2; try counter: 1)
Nov 12 16:32:54 mfmaster215 mfsmount[15074]: file: 41248959, index: 0, 
chunk: 37952867, version: 1 - writeworker: connection with (D5AF4BDB:9422) 
was timed out (unfinished writes: 1; try counter: 1)


I assume D5AF4BDB and D5AF4BDA are names for chunkservers - is there a way 
to find out which are the physical nodes/ips?

rr 

[Moosefs-users] mfscgiserv not working

[Josef <pe...@p-...>]

  Hello,
     I have managed to make a log output of mfscgiserv. This is what 
happens, when I try to access the server:

telex:/opt/mfs# /opt/mfs/sbin/mfscgiserv -f -v
starting simple cgi server (host: any , port: 9425 , rootpath: 
/opt/share/mfscgi)
Asynchronous HTTP server running on port 9425
Traceback (most recent call last):
   File "/opt/mfs/sbin/mfscgiserv", line 419, in <module>
     loop(server,HTTP)
   File "/opt/mfs/sbin/mfscgiserv", line 152, in loop
     client_handlers[r_socket].handle_read()
   File "/opt/mfs/sbin/mfscgiserv", line 65, in handle_read
     self.process_incoming()
   File "/opt/mfs/sbin/mfscgiserv", line 74, in process_incoming
     self.response = self.make_response()
   File "/opt/mfs/sbin/mfscgiserv", line 257, in make_response
     return self.err_resp(500,'Internal Server Error')
   File "/opt/mfs/sbin/mfscgiserv", line 346, in err_resp
     resp_line = "%s %s %s\r\n" %(self.protocol,code,msg)
AttributeError: HTTP instance has no attribute 'protocol'

[Moosefs-users] copy from localhost

[Abuse <Abu...@go...>]

 I have a question,

I have masterserver A, chunkservers A,B,C,D,E,F. If chunkserver C as a client mounts the mfs parition and copies a file to a local drive, will it acess any of the chunks from it's own chunkserver? I see the copy going at 11.6MB/s on a 100M switch, however I assumed that any chunks locally would copy faster.

i.e. will the chunkserver when acting as an mfs client copy from it's own chunkserver?


regards

Gov.gd sysadmin

Re: [Moosefs-users] Chunking in MooseFS

[Fabien G. <fab...@gm...>]

Hi,

On Wed, Nov 10, 2010 at 11:22 AM, Laurent Wandrebeck <lw...@hy...> wrote:

> > This means that small files use 64KB blocks while big files use 64MB
> I guess so.
>

Yes, that's right :

[root@mfsmaster]# dd if=/dev/zero of=10k_file bs=10k count=1
1+0 enregistrements lus.
1+0 enregistrements écrits.
[root@mfsmaster]# mfsfileinfo 10k_file
10k_file:
    chunk 0: 00000000001F0EBE_00000001 / (id:2035390 ver:1)
        copy 1: 192.168.200.2:9422

On the chunkserver :
[root@mfschunk2]# ls -lh BE/chunk_00000000001F0EBE_00000001.mfs
-rw-r----- 1 nobody nobody 69K nov 10 13:17
chunk_00000000001F0EBE_00000001.mfs


Fabien

Re: [Moosefs-users] Chunking in MooseFS

[Laurent W. <lw...@hy...>]

On Wed, 10 Nov 2010 10:43:36 +0100
Ioannis Aslanidis <ias...@fl...> wrote:

> Hello,
> 
> From what you say, by default we have blocks of 64KB and chunks of
> 64MB. Correct?
right.
> 
> This means that small files use 64KB blocks while big files use 64MB
> chunks. Is this the case?
I guess so.
> 
> So in the end, the difference is that many small files can fill in a
> chunk, while big files take in the whole chunk.
agreed.
> 
> Are there any performance indicators that show how much space gets lost?
Not that I know of.
See http://www.moosefs.org/moosefs-faq.html#source_code for details.
HTH,
-- 
Laurent Wandrebeck
HYGEOS, Earth Observation Department / Observation de la Terre
Euratechnologies
165 Avenue de Bretagne
59000 Lille, France
tel: +33 3 20 08 24 98
http://www.hygeos.com
GPG fingerprint/Empreinte GPG: F5CA 37A4 6D03 A90C 7A1D  2A62 54E6 EF2C
D17C F64C

Re: [Moosefs-users] chunkserver can't connect to masterserver

[Laurent W. <lw...@hy...>]

On Wed, 10 Nov 2010 18:07:54 +0800
"lwxian_aha" <lwx...@16...> wrote:

> hello:
> 
> today I have new trouble with my new MFS system,chunkserver can't connect to masterserver;
> 
> my MFS system  consist of one masterserver,three chunkserver,every chunkserver with 7.2T diskspace; about 1.4T 
> data and about 14 million files ,every file with 2 copies;
> MFS version is 1.6.17
> OS version CENTOS 5.0
> FS is ext3;
> 
> Following is the error message in master server :
> 
> [root@localhost mfs]# tail /var/log/messages 
> Nov 10 17:26:49 localhost mfsmaster[23530]: CS(192.168.10.21) packet: out of memory
> Nov 10 17:26:49 localhost mfsmaster[23530]: chunkserver disconnected - ip: 192.168.10.21, port: 0, usedspace: 0 (0.00 GiB), t
> otalspace: 0 (0.00 GiB)
> 
> what's happen ?need you help!
Looks like the answer is in the first line…
Is your mfsmaster ram+swap full ? See dmesg to find if OOM killer
triggered. Do you run x86_64 ?
Oh, and update your centos to 5.5 !
Regards,
-- 
Laurent Wandrebeck
HYGEOS, Earth Observation Department / Observation de la Terre
Euratechnologies
165 Avenue de Bretagne
59000 Lille, France
tel: +33 3 20 08 24 98
http://www.hygeos.com
GPG fingerprint/Empreinte GPG: F5CA 37A4 6D03 A90C 7A1D  2A62 54E6 EF2C
D17C F64C

Re: [Moosefs-users] chunkserver can't connect to masterserver

[Anh K. H. <ky...@vi...>]

On Wed, 10 Nov 2010 18:07:54 +0800
"lwxian_aha" <lwx...@16...> wrote:

> Following is the error message in master server :
> 
> [root@localhost mfs]# tail /var/log/messages 
> Nov 10 17:26:49 localhost mfsmaster[23530]: CS(192.168.10.21)
> packet: out of memory
> what's happen ?need you help!

Lack memory on mfs master?

-- 
Anh Ky Huynh at UTC+7

Re: [Moosefs-users] Chunking in MooseFS

[Ioannis A. <ias...@fl...>]

Hello,

>From what you say, by default we have blocks of 64KB and chunks of
64MB. Correct?

This means that small files use 64KB blocks while big files use 64MB
chunks. Is this the case?

So in the end, the difference is that many small files can fill in a
chunk, while big files take in the whole chunk.

Are there any performance indicators that show how much space gets lost?

Regards.

On Wed, Nov 10, 2010 at 10:10 AM, Laurent Wandrebeck <lw...@hy...> wrote:
> On Tue, 9 Nov 2010 17:13:34 +0100
> Ioannis Aslanidis <ias...@fl...> wrote:
>
>> Hello,
>>
>> I have some pretty important questions regarding chunking.
>>
>> The first question is with respect to the default chunk size, and
>> whether it can easily be modified, perhaps in a configuration file.
> Chunk size is hardcoded, for performance reasons. Having it modified is
> possible, though you'll have to crawl through the code and modify the
> config file parser. You may have to change Chunk header size too, and
> some other things. So it's not really trivial.
>>
>> The second question is how exactly does the chunking work with small files.
>>
>> In our case we have four types of files:
>>  - several hundred thousand small files of less than 10KB
> A file is stored in a 64KB block. So here you'll lose quite a lot of
> space.
>>  - several million medium files of around 10MB
> I'm wondering if the file would use 64KB blocks or a complete 64MB
> chunk in that case. Probably 64KB blocks, but I'm really unsure.
> Michal ?
>>  - several tens of thousand large files of around 200MB
>>  - several thousand extra large files, larger than 500MB
> Here the answer is quite clear.
>>
>> What would be a good chunk size in this case to prevent space loss?
> Maybe 16 MB. But I'm afraid of the performance hit. Keep in mind
> MooseFS wasn't designed to store small files. Is lost space so
> important given your volume compared to data security ?
> HTH,
> --
> Laurent Wandrebeck
> HYGEOS, Earth Observation Department / Observation de la Terre
> Euratechnologies
> 165 Avenue de Bretagne
> 59000 Lille, France
> tel: +33 3 20 08 24 98
> http://www.hygeos.com
> GPG fingerprint/Empreinte GPG: F5CA 37A4 6D03 A90C 7A1D  2A62 54E6 EF2C
> D17C F64C
>
> ------------------------------------------------------------------------------
> The Next 800 Companies to Lead America's Growth: New Video Whitepaper
> David G. Thomson, author of the best-selling book "Blueprint to a
> Billion" shares his insights and actions to help propel your
> business during the next growth cycle. Listen Now!
> http://p.sf.net/sfu/SAP-dev2dev
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>



-- 
Ioannis Aslanidis
System and Network Administrator
Flumotion Services, S.A.
E-Mail: iaslanidis at flumotion dot com
Office Phone: +34 93 508 63 59
Mobile Phone: +34 672 20 45 75

[Moosefs-users] chunkserver can't connect to masterserver

[lwxian_aha <lwx...@16...>]

hello:

today I have new trouble with my new MFS system,chunkserver can't connect to masterserver;

my MFS system  consist of one masterserver,three chunkserver,every chunkserver with 7.2T diskspace; about 1.4T 
data and about 14 million files ,every file with 2 copies;
MFS version is 1.6.17
OS version CENTOS 5.0
FS is ext3;

Following is the error message in master server :

[root@localhost mfs]# tail /var/log/messages 
Nov 10 17:26:49 localhost mfsmaster[23530]: CS(192.168.10.21) packet: out of memory
Nov 10 17:26:49 localhost mfsmaster[23530]: chunkserver disconnected - ip: 192.168.10.21, port: 0, usedspace: 0 (0.00 GiB), t
otalspace: 0 (0.00 GiB)

what's happen ?need you help!
thanks a lot's

2010-11-10 



lwxian_aha 

Re: [Moosefs-users] Chunking in MooseFS

[Laurent W. <lw...@hy...>]

On Tue, 9 Nov 2010 17:13:34 +0100
Ioannis Aslanidis <ias...@fl...> wrote:

> Hello,
> 
> I have some pretty important questions regarding chunking.
> 
> The first question is with respect to the default chunk size, and
> whether it can easily be modified, perhaps in a configuration file.
Chunk size is hardcoded, for performance reasons. Having it modified is
possible, though you'll have to crawl through the code and modify the
config file parser. You may have to change Chunk header size too, and
some other things. So it's not really trivial.
> 
> The second question is how exactly does the chunking work with small files.
> 
> In our case we have four types of files:
>  - several hundred thousand small files of less than 10KB
A file is stored in a 64KB block. So here you'll lose quite a lot of
space.
>  - several million medium files of around 10MB
I'm wondering if the file would use 64KB blocks or a complete 64MB
chunk in that case. Probably 64KB blocks, but I'm really unsure.
Michal ?
>  - several tens of thousand large files of around 200MB
>  - several thousand extra large files, larger than 500MB
Here the answer is quite clear.
> 
> What would be a good chunk size in this case to prevent space loss?
Maybe 16 MB. But I'm afraid of the performance hit. Keep in mind
MooseFS wasn't designed to store small files. Is lost space so
important given your volume compared to data security ?
HTH,
-- 
Laurent Wandrebeck
HYGEOS, Earth Observation Department / Observation de la Terre
Euratechnologies
165 Avenue de Bretagne
59000 Lille, France
tel: +33 3 20 08 24 98
http://www.hygeos.com
GPG fingerprint/Empreinte GPG: F5CA 37A4 6D03 A90C 7A1D  2A62 54E6 EF2C
D17C F64C

[Moosefs-users] Chunking in MooseFS

[Ioannis A. <ias...@fl...>]

Hello,

I have some pretty important questions regarding chunking.

The first question is with respect to the default chunk size, and
whether it can easily be modified, perhaps in a configuration file.

The second question is how exactly does the chunking work with small files.

In our case we have four types of files:
 - several hundred thousand small files of less than 10KB
 - several million medium files of around 10MB
 - several tens of thousand large files of around 200MB
 - several thousand extra large files, larger than 500MB

What would be a good chunk size in this case to prevent space loss?

Regards.

-- 
Ioannis Aslanidis
System and Network Administrator
Flumotion Services, S.A.
E-Mail: iaslanidis at flumotion dot com
Office Phone: +34 93 508 63 59
Mobile Phone: +34 672 20 45 75

Re: [Moosefs-users] problem with KVM virtual machine

[Terry <ha...@st...>]

you can use strace to trace kvm process. if you find you got a error on 
open image file with O_DIRECTIO, add cache option to hdd driver, since 
old qemu using directio by default but mfs does not support this flag.

On 2010?11?02? 17:01, Ken Zeng wrote:
>
> I've got a problem when I start the KVM virtual machine which stored 
> in a MFS directory:
>
> Type: virsh start xx.xx.xx.xx (IP)
>
> I get:
>
> error: Failed to start domain xx.xx.xx.xx
>
> error: internal error process exited while connecting to monitor: char 
> device redirected to /dev/pts/5
> qemu: could not open disk image /share/xx.xx.xx.xx dsk: Permission denied
>
> What should I do to get the virtual machine started?
>
> Ken  from  Guangzhou china
>
>
> ------------------------------------------------------------------------------
> The Next 800 Companies to Lead America's Growth: New Video Whitepaper
> David G. Thomson, author of the best-selling book "Blueprint to a
> Billion" shares his insights and actions to help propel your
> business during the next growth cycle. Listen Now!
> http://p.sf.net/sfu/SAP-dev2dev
>
>
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users


-- 
?? ha...@st...

???? -  ???
R&D Center - Platform Dept
??:(8610)62675604
gtalk&  msn: te...@gm...
??:???????????58???????17?       100080

[Moosefs-users] Does MooseFS (Version 1.6.17) remove the file size limit?

[michael.tangzj <mic...@gm...>]

Hi, As FAQ showed below, I want to confirm whether MooseFS current
version(1.6.17）removed file size limit ( as said " in the near future")?
If not, so how can i modify source codes in order to remove this limit by
hand?  thanks for reply.

best wishes.

--------------------------------------------
Does MooseFS have file size limit?


Currently MooseFS imposes a maximum file size limit of 2 TiB
(2,199,023,255,552 bytes). However we are considering removing this
limitation in the near future, at which point the maximum file size will
reach the limits of the operating system, which is currently 16EiB
(18,446,744,073,709,551,616 bytes).


-- 
regards,
michael.tang

Re: [Moosefs-users] Auto failover with ucarp

[jose m. <let...@us...>]

El lun, 08-11-2010 a las 15:05 -0700, Thomas S Hatch escribió:
> Looks like I have clean auto failover with ucarp for MooseFS!
> 
> 
> There are a lot of little tricks needed for failover with more than 2
> nodes, and there is almost no documentation on using ucarp. With that
> said I am starting to write up how to do it. Basically, with updates
> to the mfsmetalogger found in 1.6.18 make it really possible, and
> there are a few tricks you need to do to make sure the sessions are
> valid when the failover happens.
> 
> 
> But a failover behaves much like rebooting the machine hosting the
> mfsmaster, the current write will fail when the mount becomes active
> again, but the current process will continue. In my setup the lag time
> from a master failing to the failover node coming back to life is
> about 35-40 seconds.
> 
> 
> Thanks MooseFS team, your work is amazing, I will have a paper put
> together on this soon!
> 
> 
> -Thomas S Hatch
> ------------------------------------------------------------------------------
> The Next 800 Companies to Lead America's Growth: New Video Whitepaper
> David G. Thomson, author of the best-selling book "Blueprint to a 
> Billion" shares his insights and actions to help propel your 
> business during the next growth cycle. Listen Now!
> http://p.sf.net/sfu/SAP-dev2dev
> _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users

* I appreciate your effort, I sincerely believe that the solution passes
through native MooseFS options, IMO "The proff solution with carp" in
the howto moosefs.org web page, should read "The NO proff solution".

* My non-professional solution to safeguard the integrity and coherence
of the cluster and data at all costs, so in my case the metalogger
machine is mfsmaster clone, and monitoring mfsmaster if it is offline
for whatever reason, running a number of acctions via ssh, stops
applications, umount fuse conecctions, the process stops at all
chunkservers mfschunserver, send phone alerts, stop rsync cron jobs to
syncronize with secondary backup cluster.

* it could execute more actions such as reconfiguration metalogger ip
and start the process as mfsmaster, but like I said the solution depends
on MooseFS.

* the entry into service of the cluster is done manually.

* MooseFS not provide HA, unfortunately, it appears that I have a candle
to the Black Virgin of Chestokova.

* cheers ....

[Moosefs-users] syslog messages

[jose m. <let...@us...>]

* I've implemented an alerting system through two means, google calendar
and movistar gateway to receive on mobile phone mfs processes monitored,
and apply automatic acction reduce/increase goals in filesystem, trash,
or reduce trashtime, delete in trash ..., now I need to know exactly
what gives mfs syslog messages when errors occur on a hard disk or
Damaged status to receive message alert.

* Machines which would be suitable for monitoring, mfsmaster or
chunkservers?

* sorry poor english.

[Moosefs-users] Auto failover with ucarp

[Thomas S H. <tha...@gm...>]

Looks like I have clean auto failover with ucarp for MooseFS!

There are a lot of little tricks needed for failover with more than 2 nodes,
and there is almost no documentation on using ucarp. With that said I am
starting to write up how to do it. Basically, with updates to the
mfsmetalogger found in 1.6.18 make it really possible, and there are a few
tricks you need to do to make sure the sessions are valid when the failover
happens.

But a failover behaves much like rebooting the machine hosting the
mfsmaster, the current write will fail when the mount becomes active again,
but the current process will continue. In my setup the lag time from a
master failing to the failover node coming back to life is about 35-40
seconds.

Thanks MooseFS team, your work is amazing, I will have a paper put together
on this soon!

-Thomas S Hatch

Re: [Moosefs-users] Metalogger issues

[Michał B. <mic...@ge...>]

Update: I meant "metadata_ml.mfs"

 

From: Michał Borychowski [mailto:mic...@ge...] 
Sent: Monday, November 08, 2010 12:27 PM
To: 'Thomas S Hatch'
Cc: 'moosefs-users'
Subject: RE: [Moosefs-users] Metalogger issues

 

Hi Thomas!

 

In version 1.6.18 changelog_ml.0.mfs should be immediately created upon
metalogger restart. Don't you have this behavior? Do you really have to
restart also mfsmaster after having restarted metalogger?

 

 

Kind regards

Michał 

 

From: Thomas S Hatch [mailto:tha...@gm...] 
Sent: Friday, November 05, 2010 7:23 PM
To: moosefs-users
Subject: Re: [Moosefs-users] Metalogger issues

 

Ok, but when I restart the mfsmaster with metaloggers attatched then the
metaloggers start getting regular changelogs

but I still have to wait for the metadata_ml.mfs file

and having to restart the mfsmaster like this can be cumbersome. 

 

-Thomas S Hatch

 

On Fri, Nov 5, 2010 at 12:04 PM, Thomas S Hatch <tha...@gm...> wrote:

I am looking at my mfsmetaloggers and I have some concerns, first, if I
start the metaloggers before the master my changelog_ml.0.mfs stays empty,
and I couldn't use the changelog yet anyway because the logger doesn't grab
a copy of the metadata.mfs.bak file for a few hours.

 

Then the only times the changelog_ml.0.mfs file gets updated is when I
restart the mfsmetalogger.

 

I am seeing this behavior on moosefs 1.6.17 and the yet unreleased 1.6.18.

 

This means that I am only getting information to the loggers when I restart
mfsmetalogger and when the configured 24 hour download happens.

 

I am under the impression that the intended behavior is that when the
metalogger starts it should grab the metadata.mfs file right away, and the
changelog.0.mfs from the master, then the changelog_ml.0.mfs on the
metalogger should be getting constant updates from the mfsmaster so that at
any time an mfsmetarestore -a will recover a current working metadata.mfs
file

 

Instead, I have gaps in my changelogs, I have to wait a very long time for
metadata files and half the time mfsmetaresore -a fails.

 

Please help me understand what I am doing wrong with the metalogger.

 

-Thomas S Hatch

 

Re: [Moosefs-users] Metalogger issues

[Michał B. <mic...@ge...>]

Hi Thomas!

 

In version 1.6.18 changelog_ml.0.mfs should be immediately created upon
metalogger restart. Don't you have this behavior? Do you really have to
restart also mfsmaster after having restarted metalogger?

 

 

Kind regards

Michał 

 

From: Thomas S Hatch [mailto:tha...@gm...] 
Sent: Friday, November 05, 2010 7:23 PM
To: moosefs-users
Subject: Re: [Moosefs-users] Metalogger issues

 

Ok, but when I restart the mfsmaster with metaloggers attatched then the
metaloggers start getting regular changelogs

but I still have to wait for the metadata_ml.mfs file

and having to restart the mfsmaster like this can be cumbersome. 

 

-Thomas S Hatch

 

On Fri, Nov 5, 2010 at 12:04 PM, Thomas S Hatch <tha...@gm...> wrote:

I am looking at my mfsmetaloggers and I have some concerns, first, if I
start the metaloggers before the master my changelog_ml.0.mfs stays empty,
and I couldn't use the changelog yet anyway because the logger doesn't grab
a copy of the metadata.mfs.bak file for a few hours.

 

Then the only times the changelog_ml.0.mfs file gets updated is when I
restart the mfsmetalogger.

 

I am seeing this behavior on moosefs 1.6.17 and the yet unreleased 1.6.18.

 

This means that I am only getting information to the loggers when I restart
mfsmetalogger and when the configured 24 hour download happens.

 

I am under the impression that the intended behavior is that when the
metalogger starts it should grab the metadata.mfs file right away, and the
changelog.0.mfs from the master, then the changelog_ml.0.mfs on the
metalogger should be getting constant updates from the mfsmaster so that at
any time an mfsmetarestore -a will recover a current working metadata.mfs
file

 

Instead, I have gaps in my changelogs, I have to wait a very long time for
metadata files and half the time mfsmetaresore -a fails.

 

Please help me understand what I am doing wrong with the metalogger.

 

-Thomas S Hatch

 

Re: [Moosefs-users] Metalogger issues

[Bán M. <ba...@vo...>]

Hi,

when I encountered that problem I have written and sent two metalog sync
scripts to this list which automatically restart the master if I
restart a log-server and download the actual mfsmetalog file. 
This solution with an other small patch about the instantaneous log
flush on the metaloggers eventuate a reliable metalogger.

Miklós

On Fri, 5 Nov 2010 12:04:14 -0600
Thomas S Hatch <tha...@gm...> wrote:

> I am looking at my mfsmetaloggers and I have some concerns, first, if
> I start the metaloggers before the master my changelog_ml.0.mfs stays
> empty, and I couldn't use the changelog yet anyway because the logger
> doesn't grab a copy of the metadata.mfs.bak file for a few hours.
> 
> Then the only times the changelog_ml.0.mfs file gets updated is when I
> restart the mfsmetalogger.
> 
> I am seeing this behavior on moosefs 1.6.17 and the yet unreleased
> 1.6.18.
> 
> This means that I am only getting information to the loggers when I
> restart mfsmetalogger and when the configured 24 hour download
> happens.
> 
> I am under the impression that the intended behavior is that when the
> metalogger starts it should grab the metadata.mfs file right away,
> and the changelog.0.mfs from the master, then the changelog_ml.0.mfs
> on the metalogger should be getting constant updates from the
> mfsmaster so that at any time an mfsmetarestore -a will recover a
> current working metadata.mfs file
> 
> Instead, I have gaps in my changelogs, I have to wait a very long
> time for metadata files and half the time mfsmetaresore -a fails.
> 
> Please help me understand what I am doing wrong with the metalogger.
> 
> -Thomas S Hatch

Re: [Moosefs-users] Metalogger issues

[Thomas S H. <tha...@gm...>]

Ok, but when I restart the mfsmaster with metaloggers attatched then the
metaloggers start getting regular changelogs
but I still have to wait for the metadata_ml.mfs file
and having to restart the mfsmaster like this can be cumbersome.

-Thomas S Hatch


On Fri, Nov 5, 2010 at 12:04 PM, Thomas S Hatch <tha...@gm...> wrote:

> I am looking at my mfsmetaloggers and I have some concerns, first, if I
> start the metaloggers before the master my changelog_ml.0.mfs stays empty,
> and I couldn't use the changelog yet anyway because the logger doesn't grab
> a copy of the metadata.mfs.bak file for a few hours.
>
> Then the only times the changelog_ml.0.mfs file gets updated is when I
> restart the mfsmetalogger.
>
> I am seeing this behavior on moosefs 1.6.17 and the yet unreleased 1.6.18.
>
> This means that I am only getting information to the loggers when I restart
> mfsmetalogger and when the configured 24 hour download happens.
>
> I am under the impression that the intended behavior is that when the
> metalogger starts it should grab the metadata.mfs file right away, and the
> changelog.0.mfs from the master, then the changelog_ml.0.mfs on the
> metalogger should be getting constant updates from the mfsmaster so that at
> any time an mfsmetarestore -a will recover a current working metadata.mfs
> file
>
> Instead, I have gaps in my changelogs, I have to wait a very long time for
> metadata files and half the time mfsmetaresore -a fails.
>
> Please help me understand what I am doing wrong with the metalogger.
>
> -Thomas S Hatch
>

[Moosefs-users] Metalogger issues

[Thomas S H. <tha...@gm...>]

I am looking at my mfsmetaloggers and I have some concerns, first, if I
start the metaloggers before the master my changelog_ml.0.mfs stays empty,
and I couldn't use the changelog yet anyway because the logger doesn't grab
a copy of the metadata.mfs.bak file for a few hours.

Then the only times the changelog_ml.0.mfs file gets updated is when I
restart the mfsmetalogger.

I am seeing this behavior on moosefs 1.6.17 and the yet unreleased 1.6.18.

This means that I am only getting information to the loggers when I restart
mfsmetalogger and when the configured 24 hour download happens.

I am under the impression that the intended behavior is that when the
metalogger starts it should grab the metadata.mfs file right away, and the
changelog.0.mfs from the master, then the changelog_ml.0.mfs on the
metalogger should be getting constant updates from the mfsmaster so that at
any time an mfsmetarestore -a will recover a current working metadata.mfs
file

Instead, I have gaps in my changelogs, I have to wait a very long time for
metadata files and half the time mfsmetaresore -a fails.

Please help me understand what I am doing wrong with the metalogger.

-Thomas S Hatch

Re: [Moosefs-users] writeWorker time out

[Stas O. <sta...@gm...>]

Hi.

Thanks for replies - it clarifies the matter.

Regards.


2010/11/5 Michał Borychowski <mic...@ge...>

>  *Hi!*
>
> * *
>
> *My answers are below*
>
>
>
> *From:* Stas Oskin [mailto:sta...@gm...]
> *Sent:* Sunday, October 31, 2010 3:08 PM
> *To:* Michał Borychowski
> *Cc:* moosefs-users
> *Subject:* Re: [Moosefs-users] writeWorker time out
>
> Hi.
>
> Thanks for the explanation.
>
> Is this error appears on replications of MFS servers themselves, or on
> application writing to MFS mount (and as a result, writing to single MFS
> server)?
>
> *[MB] This is message from mfsmount. It means that from the moment of
> sending data to a chunkserver passed to much time. Unless the messages occur
> very often there is nothing to worry about. *
>
>
>
> Also, if this happens on writing to MFS mount, is this error reported to
> application as error in writing file?
>
> *[MB] No. Only after several such failures application gets EIO (input /
> output error). By default after 30. Here you have not much timeouts so that
> the application would not know about this.*
>
> * *
>
> Also, if it happens on the MFS mount side - where the information is kept
> during these retries? In RAM? On disk?
>
> *[MB] In RAM. MooseFS mount has write cache with configurable size (check
> the man pages for details)*
>
>
>
> *Kind regards*
>
> *Michał *
>
>
>
> Thanks again!
>
> 2010/10/29 Michał Borychowski <mic...@ge...>
>
>
>
> There is a small timeout set for the write operation (several seconds). It
> may happen that a single write operation takes several or more seconds. If
> these messages are sent by different servers, there is nothing to worry
> about.
>
>
>
> But if the message is sent mainly by one server (IP in hex C0A8020F =
> 192.168.2.15) you should investigate it more. In CGI monitor go to the Disks
> tab and click "hour" in "I/O stats last min (switch to hour,day)" row and
> sort by "write" in "max time (switch to avg)" column. Now look if there are
> disks which obviously stay from the others. You can also look at the "fsync"
> column and sort the results. Maximum times should not exceed 2 seconds (2
> million microseconds). You should look for individual disks which may be a
> bottleneck of the system.
>
>
>
> "try counter: 1" alone is not a problem - number of trials is set as an
> option to mfsmount (by default 30). Until mfsmounts reaches this limit write
> operations are repeated and the application gets the OK status.
>
>
>
>
>
> Regards
>
> Michal
>
>
>
>
>
>
>
> *From:* Stas Oskin [mailto:sta...@gm...]
> *Sent:* Wednesday, October 20, 2010 1:04 PM
> *To:* moosefs-users
> *Subject:* [Moosefs-users] writeWorker time out
>
>
>
> Hi.
>
> We noticed the following message in logs:
> file: 28, index: 7, chunk: 992, version: 1 - writeworker: connection with
> (C0A8020F:9422) was timed out (unfinished writes: 5; try counter: 1)
>
> MFS seems to be working and functioning normally.
> It seems to be related to write process timing-out, but connection is
> normal.
>
> Can it be caused by slow speed of disks? Also, what counter 1 can do, and
> where it can be changed?
> Finally, what operation system will return to application - that write
> operation has failed?
>
> Thanks in advance!
>
>
>
>
>

Re: [Moosefs-users] writeWorker time out

[Michał B. <mic...@ge...>]

Hi!

 

My answers are below

 

From: Stas Oskin [mailto:sta...@gm...] 
Sent: Sunday, October 31, 2010 3:08 PM
To: Michał Borychowski
Cc: moosefs-users
Subject: Re: [Moosefs-users] writeWorker time out

Hi.

Thanks for the explanation.

Is this error appears on replications of MFS servers themselves, or on application writing to MFS mount (and as a result, writing to single MFS server)?

[MB] This is message from mfsmount. It means that from the moment of sending data to a chunkserver passed to much time. Unless the messages occur very often there is nothing to worry about. 



Also, if this happens on writing to MFS mount, is this error reported to application as error in writing file?

[MB] No. Only after several such failures application gets EIO (input / output error). By default after 30. Here you have not much timeouts so that the application would not know about this.

 

Also, if it happens on the MFS mount side - where the information is kept during these retries? In RAM? On disk?

[MB] In RAM. MooseFS mount has write cache with configurable size (check the man pages for details)

 

Kind regards

Michał 



Thanks again!

2010/10/29 Michał Borychowski <mic...@ge...>

 

There is a small timeout set for the write operation (several seconds). It may happen that a single write operation takes several or more seconds. If these messages are sent by different servers, there is nothing to worry about.

 

But if the message is sent mainly by one server (IP in hex C0A8020F = 192.168.2.15) you should investigate it more. In CGI monitor go to the Disks tab and click “hour” in “I/O stats last min (switch to hour,day)” row and sort by “write” in “max time (switch to avg)” column. Now look if there are disks which obviously stay from the others. You can also look at the “fsync” column and sort the results. Maximum times should not exceed 2 seconds (2 million microseconds). You should look for individual disks which may be a bottleneck of the system. 

 

"try counter: 1" alone is not a problem – number of trials is set as an option to mfsmount (by default 30). Until mfsmounts reaches this limit write operations are repeated and the application gets the OK status. 

 

 

Regards

Michal

 

 

 

From: Stas Oskin [mailto:sta...@gm...] 
Sent: Wednesday, October 20, 2010 1:04 PM
To: moosefs-users
Subject: [Moosefs-users] writeWorker time out

 

Hi.

We noticed the following message in logs:
file: 28, index: 7, chunk: 992, version: 1 - writeworker: connection with (C0A8020F:9422) was timed out (unfinished writes: 5; try counter: 1)

MFS seems to be working and functioning normally.
It seems to be related to write process timing-out, but connection is normal.

Can it be caused by slow speed of disks? Also, what counter 1 can do, and where it can be changed?
Finally, what operation system will return to application - that write operation has failed?

Thanks in advance!

 

 

Re: [Moosefs-users] problems with cgi monitor

[Michał B. <mic...@ge...>]

Hi!

CGI monitor by default works on 9425 port. Do you attach it to the URL?


Kind regards
Michał Borychowski 
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01


-----Original Message-----
From: Josef [mailto:pe...@p-...] 
Sent: Sunday, October 31, 2010 3:05 PM
To: moo...@li...
Subject: [Moosefs-users] problems with cgi monitor

  Hello,
     I'm playing with MooseFS, but I'm unable to get some running info 
about the system. CGI monitor returns just a blank page, i'm runing it 
on the master. I'm runing it using /opt/mfs/sbin/mfscgiserv (mfs is 
installed into /opt/mfs). Do I have to set any extra parameters or 
something? My mfs version is 1.6.17.

Thanks,
     Josef

----------------------------------------------------------------------------
--
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store 
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

[Moosefs-users] About mooseFS size limit

[tjing.tech <tji...@gm...>]

hi!
what is the mooseFS  file system‘s size limit？ 
what is the mooseFS name space size limit?
thanks!



tjing.tech
2010-11-04