[Moosefs-users] Non-existing chunks hang the mfsmaster

[Stas O. <sta...@gm...>]

Hi.

We got a very strange that happened on our test cluster.

After a power crash, the mfsmaster syslog is full of following errors:
Feb 27 19:19:21 web1 mfsmaster[30654]: chunkserver has nonexistent chunk
(00000000005313A2_00000001), so create it for future deletion
Feb 27 19:19:21 web1 mfsmaster[30654]: chunkserver has nonexistent chunk
(00000000005393A2_00000001), so create it for future deletion
Feb 27 19:19:21 web1 mfsmaster[30654]: chunkserver has nonexistent chunk
(00000000005493A3_00000001), so create it for future deletion
Feb 27 19:19:21 web1 mfsmaster[30654]: chunkserver has nonexistent chunk
(00000000005293A3_00000001), so create it for future deletion
Feb 27 19:19:21 web1 mfsmaster[30654]: chunkserver has nonexistent chunk
(00000000005513A3_00000001), so create it for future deletion
Feb 27 19:19:21 web1 mfsmaster[30654]: chunkserver has nonexistent chunk
(00000000005313A3_00000001), so create it for future deletion
Feb 27 19:19:21 web1 mfsmaster[30654]: chunkserver has nonexistent chunk
(00000000005113A3_00000001), so create it for future deletion
Feb 27 19:19:21 web1 mfsmaster[30654]: chunkserver has nonexistent chunk
(00000000005093A3_00000001), so create it for future deletion

These errors appears all the time, and practically hang the mfsmaster.
mfscgi stops working (hangs), and mounts are aborting with following error:
error receiving data from mfsmaster: ETIMEDOUT (Operation timed out)
error receiving data from mfsmaster: ETIMEDOUT (Operation timed out)


Upgrading to .20 didn't help.

Any idea what this could be and how to resolve it?

Thanks.