Re: [Moosefs-users] Clean up chunk files

[Anh K. H. <ky...@vi...>]

On Wed, 25 Aug 2010 13:20:56 -0300
"Ricardo J. Barberis" <ric...@da...> wrote:

> El Miércoles 25 Agosto 2010, Anh K. Huynh escribió:
> > Hi,
> >
> > I have a MFS system with two chunk disks
> > 	* a disk with size 05 GB (mounted as /mnt/mfs_m0)
> > 	* a disk with size 30 GB (mounted as /mnt/mfs_m1)
> >
> > The MFS disk is /tmp/mfs_c0 was reported to have 1GB in size, as
> > below:
> >
> > /-------------------------------------------------------
> > Filesystem            Size  Used Avail Use% Mounted on
> > /mnt/mfs_chunk_db1_0  5.0G  4.4G  288M  94% /mnt/mfs_m0
> > /mnt/mfs_chunk_db1_1   30G   27G  1.2G  96% /mnt/mfs_m1
> > mfs#10.0.0.10:9421
> >                      1001M     0 1001M   0% /tmp/mfs_c0
> >
> > ls /tmp/mfs_c0/
> >
> > \-------------------------------------------------------
> >
> > All these disks were completely free (use 0%) before I tried to
> > write a very large file to MFS mount point. During the writing
> > process, my program detected that disk was full and it aborted.
> > Unfortunately, MFS server couldn't recovery from such error, and
> > though it reported that MFS disk is free (0% used), I can't write
> > anything to it. Moreover, any chunk disks are full: such disks
> > contains many chunk files that aren't valid anymore.
> >
> > How to recovery from such error? and How to clean up chunk files
> > on chunk disks? How to know if any chunk files are valid for used?
> >
> > Thanks for your helps.
> 
> Really strange, are you sure you didn't write the file
> to /mnt/mfs_m0 directly instead of /tmp/mfs_c0 ?

I'm sure that I wrote data only to /tmp/mfs_c0. A listing command show that
	* in /mnt/mfs_m1, there are 428 chunk files (*.mfs)
	* in /mnt/mfs_m0, there are 68 chunk files (*.mfs)

When `mfsmaster` starts, it report that there're 496 chunks, but actually in /tmp/mfs_c0 there are no file associated with those chunks.

/------------------------------------------------------------- 
    working directory: /opt/var/mfs
    lockfile created and locked
    initializing mfsmaster modules ...
    loading sessions ... ok
    sessions file has been loaded
    exports file has been loaded
    loading metadata ...
    loading objects (files,directories,etc.) ... ok
    loading names ... ok
    loading deletion timestamps ... ok
    checking filesystem consistency ... ok
    loading chunks data ... ok
    connecting files and chunks ... ok
    all inodes: 70
    directory inodes: 1
    file inodes: 69
    chunks: 496
    metadata file has been loaded
    stats file has been loaded
    master <-> metaloggers module: listen on *:9419
    master <-> chunkservers module: listen on *:9420
    main master server module: listen on *:9421
    mfsmaster daemon initialized properly
\-------------------------------------------------------------    

> What are /mnt/mfs_chunk_db1_0 and /mnt/mfs_chunk_db1_1, are those
> loop files? (hinted by your previous email)

Yes, they are all loop files created by `dd` command (as in MFS' documents). I created the file `/mnt/mfs_chunk_db1_0` first, then I created `/mnt/mfs_chunk_db1_1` with a bigger size while hoping that MFS would only write my hug data to the second disk (goal = 1). Unfortunately, MFS tried to write data two both chunks and error occured. Please note, all these happened on the same server.

I tried to investigate MFS source code to find someways to check/list status of chunk files; I think such program is very useful.

Regards,

-- 
Anh Ky Huynh