Re: [Moosefs-users] Purging missing chunks

[Olivier T. <Oli...@lm...>]

Hi Karl,

Le 11/06/2012 18:45, Karl Zander a écrit :
> Hi Chris,
>
> Thanks for the fast response.  I'll give it a shot but there must be a way that
> isn't so time consuming.  My moosefs is 500TB and I'd imagine this operation
> could take days or a week.  I'll get it started now and report back with my
> findings.

As far as I understand, logs on the mfsmaster shows the missing chunk immediatly 
followed by the files that are stored in this chunk.
Logs on my mfsmaster look like this :
Jun 12 09:29:09 manganese mfsmaster[18364]: currently unavailable chunk 
00000000003DDE38 (inode: 5122921 ; index: 0)
Jun 12 09:29:09 manganese mfsmaster[18364]: * currently unavailable file 
5122921: 
2copies/backuppc/pc/ubuntu-lmpt/75/f%252f/fusr/fshare/fxfig/fLibraries/fFlowchart/foffline_storage.fig
Jun 12 09:29:09 manganese mfsmaster[18364]: * currently unavailable file 
5122921: 
2copies/backuppc/pc/ubuntu-lmpt/0/f%252f/fusr/fshare/fxfig/fLibraries/fFlowchart/foffline_storage.fig
Jun 12 09:29:09 manganese mfsmaster[18364]: * currently unavailable file 
5122921: 
2copies/backuppc/pc/ubuntu-lmpt/49/f%252f/fusr/fshare/fxfig/fLibraries/fFlowchart/foffline_storage.fig
Jun 12 09:29:09 manganese mfsmaster[18364]: * currently unavailable file 
5122921: 
2copies/backuppc/pc/ubuntu-lmpt/56/f%252f/fusr/fshare/fxfig/fLibraries/fFlowchart/foffline_storage.fig
Jun 12 09:29:09 manganese mfsmaster[18364]: * currently unavailable file 
5122921: 
2copies/backuppc/pc/ubuntu-lmpt/64/f%252f/fusr/fshare/fxfig/fLibraries/fFlowchart/foffline_storage.fig
Jun 12 09:29:09 manganese mfsmaster[18364]: * currently unavailable file 
5122921: 
2copies/backuppc/pc/ubuntu/0/f%252f/fusr/fshare/fxfig/fLibraries/fFlowchart/foffline_storage.fig
Jun 12 09:29:09 manganese mfsmaster[18364]: * currently unavailable file 
5122921: 
2copies/backuppc/pc/ubuntu/6/f%252f/fusr/fshare/fxfig/fLibraries/fFlowchart/foffline_storage.fig
Jun 12 09:29:09 manganese mfsmaster[18364]: * currently unavailable file 
5122921: 
2copies/backuppc/pc/ubuntu/8/f%252f/fusr/fshare/fxfig/fLibraries/fFlowchart/foffline_storage.fig
Jun 12 09:29:09 manganese mfsmaster[18364]: * currently unavailable file 
5122921: 2copies/backuppc/cpool/9/b/f/9bf659facce54906ee6f3dd1b1794e5e
Jun 12 09:29:09 manganese mfsmaster[18364]: currently unavailable chunk 
00000000000A1073 (inode: 928652 ; index: 0)
Jun 12 09:29:09 manganese mfsmaster[18364]: * currently unavailable file 928652: 
3copies/home_etu/20900601t/.dbus/session-bus/0ddb87ec776a0a55b80e2b184c09165c-1

So you can see here two missing chunks : 00000000003DDE38 and 00000000000A1073, 
with their inode number, respectively 5122921 and 928652.

What I did in such case was simply remove those files. (i know they have been 
stored in other chunks lately).
Before deletion, i set the trashtime to 1 second so the files don't even stay in 
the trash space.

Automated way could then be something like :

#!/bin/bash

# Go to the mfs root
cd /mfsroot/

for f in `grep  "currently unavailable file" mfsmaster.log | cut -d ' ' -f 11-`; do
     mfssettrashtime 1 $f
     rm -f $f
done

It may be necessary to escape some character in the file name (such as space for 
example).
Before doing this, I would suggest to be sure that every chunkservers are online 
and every disks/partitions of each chunkserver are available, in order to be 
sure that listed missing files are really missing in the moosefs space.

Comments on this way of practice are welcome.
It worked very nice for me, but i am not sure it's the best way.

Regards,

	Olivier




>
> Regards,
>
> Karl
>
> On Jun 11, 2012 12:39 PM, "Chris Picton" <ch...@ec...
> <mailto:ch...@ec...>> wrote:
>
>     Hi Karl
>
>     I use the following command to find files which have 0 copy chunks:
>
>     find /mnt/mfs/root/ -type f | xargs mfscheckfile | grep -B1 ^0 | grep :$ |
>     sed 's/:$//g'
>
>     In my case, I have mounted the root of mfs on /mnt/mfs/root
>
>     If the above finds no files, they have to be in trash (afaik)
>
>     Chris
>
>     On 2012/06/11 3:47 PM, Karl Zander wrote:
>>     Hi MooseFS Users,
>>
>>     I'm hoping someone has experienced this and can help me out.  I have a
>>     fairly large Moose and I lost 2 nodes due to hardware failure.  I know
>>     have some 30,000 missing chunks on my moosefs and I have no idea how to
>>     clear them.  I've tried recursively checking each file with mfsfileinfo
>>     and I can't seem to find the missing chunks/files.  I have files listed on
>>     the web interface that are missing but I've already deleted them and the
>>     trash time has already been exceeded.  I'm finding this to be very
>>     inefficient.  Isn't there any easier way to delete chunks that are offline
>>     and never will be online again?
>>
>>     Also, is there a better way to query the moose besides the web interface?
>>      I'd like to be able to call this information using scripts and the mfs*
>>     commands don't give me everything I need.  I appreciate any help!  Thanks.
>>
>>     MooseFS version 1.6.20-8
>>
>>
>>     Karl
>>
>>
>>     ------------------------------------------------------------------------------
>>     Live Security Virtual Conference
>>     Exclusive live event will cover all the ways today's security and
>>     threat landscape has changed and how IT managers can respond. Discussions
>>     will include endpoint security, mobile security and the latest in malware
>>     threats.http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>
>>
>>     _______________________________________________
>>     moosefs-users mailing list
>>     moo...@li...  <mailto:moo...@li...>
>>     https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
>
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>


-- 
Olivier THIBAULT
Université François Rabelais - UFR Sciences et Techniques
Laboratoire de Mathématiques et Physique Théorique (UMR CNRS 7350)
Service Informatique de l'UFR
Parc de Grandmont
37200 Tours - France
Email: olivier.thibault at lmpt.univ-tours.fr
Tel:     (33)(0)2 47 36 69 12
Fax:     (33)(0)2 47 36 70 68