Menu
▾
▴
[MooseFS-Users] mfsmount stuck
|
From: <ma...@me...> - 2017-07-11 07:30:51
|
Hi!
I've experienced mfs.mount stuck while doing intensive I/O operations with many small files. I run two operations, rsync -avS /local/fs/with/thousands/of/files /mnt/mfs/mfs-filesystem/ , rsync was copying files for long time and there was no problem. Next I run du -hs /mnt/mfs/mfs-filesystem (parallelly to rsync) and after a minute or two mfs.mount stuck.
mfsnetdump shows:
1499757977.093449 : 192.168.100.190 : 26157 -> 192.168.100. 10 : 9421 CLTOMA_FUSE_STATFS
0x00000: 00 00 00 12
1499757977.093585 : 192.168.100. 10 : 9421 -> 192.168.100.190 : 26157 MATOCL_FUSE_STATFS
0x00000: 00 00 00 12 00 00 01 AA 9A 4B 20 00 00 00 01 35 DB 1B 24 00 00 00 00 00 00 00 00 4E 00 00 00 00
0x00020: 00 00 00 00 00 04 E2 89
1499757987.734365 : 192.168.100.190 : 26157 -> 192.168.100. 10 : 9421 CLTOMA_FUSE_SUSTAINED_INODES
0x00000: 00 00 6F 89 00 04 E2 8A
1499758005.736338 : 192.168.100.190 : 26157 -> 192.168.100. 10 : 9421 CLTOMA_FUSE_TIME_SYNC
0x00000: 00 00 00 00
1499758005.736455 : 192.168.100. 10 : 9421 -> 192.168.100.190 : 26157 MATOCL_FUSE_TIME_SYNC
0x00000: 00 00 00 00 00 05 54 05 9F 2E 52 8D
strace shows:
[pid 31457] 09:28:17.274753 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.277375 nanosleep({tv_sec=0, tv_nsec=2500000}, <unfinished ...>
[pid 31470] 09:28:17.278169 <... nanosleep resumed> 0x7ff9d14d9f00) = 0
[pid 31470] 09:28:17.278189 nanosleep({tv_sec=0, tv_nsec=10000000}, <unfinished ...>
[pid 31457] 09:28:17.279987 <... nanosleep resumed> 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.280028 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.282634 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.285247 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.287876 nanosleep({tv_sec=0, tv_nsec=2500000}, <unfinished ...>
[pid 31470] 09:28:17.288265 <... nanosleep resumed> 0x7ff9d14d9f00) = 0
[pid 31470] 09:28:17.288290 nanosleep({tv_sec=0, tv_nsec=10000000}, <unfinished ...>
[pid 31457] 09:28:17.290498 <... nanosleep resumed> 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.290532 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.293154 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.295804 read(31, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.295853 write(31, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.295906 nanosleep({tv_sec=0, tv_nsec=2500000}, <unfinished ...>
[pid 31470] 09:28:17.298369 <... nanosleep resumed> 0x7ff9d14d9f00) = 0
[pid 31470] 09:28:17.298388 nanosleep({tv_sec=0, tv_nsec=10000000}, <unfinished ...>
[pid 31457] 09:28:17.298487 <... nanosleep resumed> 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.298507 read(38, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.298535 write(38, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.298566 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.301151 read(13, 0x7ff9e3758f20, 8) = -1 EAGAIN (Resource temporarily unavailable)
[pid 31457] 09:28:17.301184 write(13, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.301217 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.303805 read(18, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.303846 write(18, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.303885 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.306513 read(28, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.306545 write(28, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.306572 nanosleep({tv_sec=0, tv_nsec=2500000}, <unfinished ...>
[pid 31470] 09:28:17.308463 <... nanosleep resumed> 0x7ff9d14d9f00) = 0
[pid 31470] 09:28:17.308481 nanosleep({tv_sec=0, tv_nsec=10000000}, <unfinished ...>
[pid 31457] 09:28:17.309149 <... nanosleep resumed> 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.309166 read(19, 0x7ff9e3758f20, 8) = -1 EAGAIN (Resource temporarily unavailable)
[pid 31457] 09:28:17.309191 write(19, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.309218 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.311812 read(25, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.311860 write(25, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.311904 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.314511 read(34, 0x7ff9e3758f20, 8) = -1 EAGAIN (Resource temporarily unavailable)
[pid 31457] 09:28:17.314544 write(34, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.314578 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.317166 read(22, 0x7ff9e3758f20, 8) = -1 EAGAIN (Resource temporarily unavailable)
[pid 31457] 09:28:17.317194 write(22, "\0\0\0\0\0\0\0\0", 8) = 8
[pid 31457] 09:28:17.317226 nanosleep({tv_sec=0, tv_nsec=2500000}, <unfinished ...>
[pid 31470] 09:28:17.318560 <... nanosleep resumed> 0x7ff9d14d9f00) = 0
[pid 31470] 09:28:17.318582 nanosleep({tv_sec=0, tv_nsec=10000000}, <unfinished ...>
[pid 31457] 09:28:17.319824 <... nanosleep resumed> 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.319853 nanosleep({tv_sec=0, tv_nsec=2500000}, <unfinished ...>
[pid 31472] 09:28:17.320388 <... nanosleep resumed> 0x7ff9d09b5f20) = 0
[pid 31472] 09:28:17.320420 nanosleep({tv_sec=0, tv_nsec=100000000}, <unfinished ...>
[pid 31457] 09:28:17.322446 <... nanosleep resumed> 0x7ff9e3758f10) = 0
[pid 31457] 09:28:17.322469 nanosleep({tv_sec=0, tv_nsec=2500000}, 0x7ff9e3758f10) = 0
At this moment I can't do any I/O operation on this mountpoint.
All moosefs components are 3.0.94.
Thanks for help,
Marcin
|
