Menu
▾
▴
[MooseFS-Users] [v4] master process crashed on metadata dump
|
From: Gandalf C. <gan...@gm...> - 2018-05-31 11:09:36
|
May 31 13:00:07 cs01 mfsmaster[21698]: write error May 31 13:00:07 cs01 mfsmaster[21698]: can't write metadata May 31 13:00:15 cs01 mfsmaster[21698]: write error May 31 13:00:15 cs01 mfsmaster[21698]: write error May 31 13:00:15 cs01 mfsmaster[21698]: write error May 31 13:00:15 cs01 mfsmaster[21698]: write error May 31 13:00:15 cs01 mfsmaster[20898]: child finished May 31 13:00:15 cs01 mfsmaster[20898]: store process has finished - store time: 15.774 May 31 13:00:15 cs01 mfsmaster[20898]: metadata not stored !!! (child exited) - exiting May 31 13:00:15 cs01 mfsmaster[20898]: internal terminate request May 31 13:00:15 cs01 mfsmaster[20898]: state: transition FOLLOWER -> DUMMY ; changed 52033 seconds ago ; leaderip: 10.200.1.13 May 31 13:00:15 cs01 mfsmaster[20898]: state: DUMMY ; changed 0 seconds ago ; leaderip: 10.200.1.13 May 31 13:00:16 cs01 mfsmaster[20898]: exited from main loop May 31 13:00:16 cs01 mfsmaster[20898]: exititng ... May 31 13:00:16 cs01 mfsmaster[20898]: main master server module: closing *:9421 May 31 13:00:16 cs01 mfsmaster[20898]: master <-> chunkservers module: closing *:9420 May 31 13:00:16 cs01 mfsmaster[20898]: master control module: closing *:9419 May 31 13:00:23 cs01 mfsmaster[20898]: cleaning metadata ... May 31 13:00:24 cs01 mfsmaster[20898]: metadata have been cleaned May 31 13:00:24 cs01 mfsmaster[20898]: process exited successfully (status:0) May 31 13:00:25 cs01 mfsmaster[24058]: set gid to 111 May 31 13:00:25 cs01 mfsmaster[24058]: set uid to 107 May 31 13:00:25 cs01 mfsmaster[24058]: can't find process to terminate # df -h Filesystem Size Used Avail Use% Mounted on udev 7.9G 0 7.9G 0% /dev tmpfs 1.6G 9.1M 1.6G 1% /run /dev/sdd1 12G 8.2G 3.9G 69% / tmpfs 7.9G 0 7.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup chunk0 1.8T 239G 1.6T 14% /mnt/chunks/chunk0 chunk1 1.8T 243G 1.6T 14% /mnt/chunks/chunk1 tmpfs 1.6G 0 1.6G 0% /run/user/1000 maybe i'm running out of space, but I think that master process shound't crash # ls -la /var/lib/mfs/ total 7352160 drwxr-xr-x 2 mfs mfs 311 May 31 13:00 . drwxr-xr-x 25 root root 4096 May 31 00:13 .. -rw-r----- 1 mfs mfs 5 May 30 21:39 .mfschunkserver.lock -rw-r----- 1 mfs mfs 4870620149 May 31 13:00 changelog.1.mfs -rw-r----- 1 mfs mfs 90 May 30 18:30 changelog.12.mfs -rw-r----- 1 mfs mfs 16509535 May 30 22:20 changelog.2.mfs -rw-r----- 1 mfs mfs 1164 May 30 21:32 changelog.4.mfs -rw-r----- 1 mfs mfs 10 May 30 21:17 chunkserverid.mfs -rw-r----- 1 mfs mfs 4066348 May 31 13:00 csstats.mfs -rw-r----- 1 mfs mfs 144 May 31 13:00 metadata.crc -rw-r----- 1 mfs mfs 910498282 May 31 13:00 metadata.mfs -rw-r----- 1 mfs mfs 852231792 May 31 12:00 metadata.mfs.back.1 -rw-r----- 1 mfs mfs 870973440 May 31 13:00 metadata.mfs.emergency -rw-r--r-- 1 root root 8 May 17 09:28 metadata.mfs.empty -rw-r----- 1 mfs mfs 3672832 May 31 13:00 stats.mfs |
