[MooseFS-Users] Missing chunks: "WRONG VERSION"

[Michael T. <mic...@ho...>]

Hi,


A strange thing has been happening over these past two weeks.  For three times now, one or two missing chunks appeared with "WRONG VERSION" as the type of missing chunks as shown in the GUI.  What is strange is that all these missing chunks occured only for the same file.


Here's my (partial) setup:


1x Master

3x Chunkservers -- all files have goal=3, and 1 chunkserver is doing a rebalancing

1x Client


All are running at version 3.0.100.


The client is a KVM host running small VMs whose disks are stored are .raw/.img files under /mnt/moosefs/images.  The affected file is always the raw disk image of a VM (ProxyServer02):


Missing files (gathered by previous file-loop)
# paths inode index chunk id type of missing chunk
1 ./TRASH (KVM/images/ProxyServer02-hda.raw) 12261261 200 0000000001DD5970 WRONG VERSIONS
2 ./TRASH (KVM/images/ProxyServer02-hda.raw) 12261261 206 0000000001DD5978 WRONG VERSIONS



I've been running this setup for several years now but only now have I encountered this.  It is a good thing that the affected VM is not mission critical and we can recover from this issue within an hour of detecting it.


Where should I begin to troubleshoot this?



--- mike t.