From: <n-...@us...> - 2012-07-20 04:29:49
|
Revision: 6438 http://gfarm.svn.sourceforge.net/gfarm/?rev=6438&view=rev Author: n-soda Date: 2012-07-20 04:29:43 +0000 (Fri, 20 Jul 2012) Log Message: ----------- merge r6437 from the main trunk: add some description how to relax descriptor hard limit on Linux, NetBSD, FreeBSD and Solaris, for #411 - gfmd.conf "metadb_server_max_descriptors" directive to specify descriptor limit Revision Links: -------------- http://gfarm.svn.sourceforge.net/gfarm/?rev=6437&view=rev Modified Paths: -------------- gfarm_v2/branches/2.5/doc/Gfarm-FAQ.en gfarm_v2/branches/2.5/doc/Gfarm-FAQ.ja Property Changed: ---------------- gfarm_v2/branches/2.5/ gfarm_v2/branches/2.5/gftool/gfgroup/gfgroup.c Property changes on: gfarm_v2/branches/2.5 ___________________________________________________________________ Modified: svn:mergeinfo - /gfarm_v2/branches/2.4:5271,5275-5281,5290,5329 /gfarm_v2/trunk:5598,5616-5624,5627-5628,5641-5647,5650,5653,5657-5666,5670,5674,5683-5685,5693,5695-5696,5699,5719-5720,5727,5729-5730,5775,5793,5799,5808-5809,5821,5825,5827-5831,5839-5842,5844-5849,5863-5864,5868,5870,5874-5878,5886,5889,5913,5915,5924,5932,5934,5959,6001-6003,6012-6013,6016,6019,6025-6027,6029,6040,6070,6073,6077-6079,6081,6083-6084,6086,6089,6091,6093,6095,6101,6103,6118,6120,6129,6137,6139,6152-6153,6155,6159,6161,6166,6168,6173,6175,6178-6179,6184-6185,6188,6268,6272,6295,6305-6306,6310,6315,6333,6335,6354,6397,6427 + /gfarm_v2/branches/2.4:5271,5275-5281,5290,5329 /gfarm_v2/trunk:5598,5616-5624,5627-5628,5641-5647,5650,5653,5657-5666,5670,5674,5683-5685,5693,5695-5696,5699,5719-5720,5727,5729-5730,5775,5793,5799,5808-5809,5821,5825,5827-5831,5839-5842,5844-5849,5863-5864,5868,5870,5874-5878,5886,5889,5913,5915,5924,5932,5934,5959,6001-6003,6012-6013,6016,6019,6025-6027,6029,6040,6070,6073,6077-6079,6081,6083-6084,6086,6089,6091,6093,6095,6101,6103,6118,6120,6129,6137,6139,6152-6153,6155,6159,6161,6166,6168,6173,6175,6178-6179,6184-6185,6188,6268,6272,6295,6305-6306,6310,6315,6333,6335,6354,6397,6427,6437 Modified: gfarm_v2/branches/2.5/doc/Gfarm-FAQ.en =================================================================== --- gfarm_v2/branches/2.5/doc/Gfarm-FAQ.en 2012-07-20 04:25:38 UTC (rev 6437) +++ gfarm_v2/branches/2.5/doc/Gfarm-FAQ.en 2012-07-20 04:29:43 UTC (rev 6438) @@ -131,12 +131,41 @@ Nm: maximum number of metadata servers. Nf: maximum number of filesystem nodes. Nc: maximum number of client processes. (e.g. gfarm2fs) - - Then the number is: + Then the number is calculated by: (Nm - 1) * 2 + Nf * (Nf + 1) + Nc * (Nf + 1) + some more - Each term means: + You can see actual setting by looking at system log + (e.g. /var/log/messages) at gfmd startup like follows: + gfmd[...]: .... max descriptors = 131072 + If this value is smaller than the value specified by the + metadb_server_max_descriptors parameter, that means it's limited + by the hard limit setting of the Operating System. To increase + the hard limit, please follow the following procedure. + On Linux: + On Linux, gfarm limits the value to 75% of sysctl fs.file-max + setting, instead of using current hard limit setting. + Thus, to make "metadb_server_max_descriptors 131072" setting + effective, you should add something like "fs.file-max=180224" + to a file /etc/sysctl.conf, and reboot or invoke a command + "/sbin/sysctl -p". + On NetBSD: + Add something like "kern.maxfiles=180224" to a file + /etc/sysctl.conf, and reboot or invoke a command + "/sbin/sysctl -f /etc/sysctl.conf". + On FreeBSD: + Add settings like the followings to a file /etc/sysctl.conf: + kern.maxfiles=180224 + kern.maxfilesperproc=131072 + and reboot or invoke a command + /sbin/sysctl -f /etc/sysctl.conf". + On Solaris: + Add a setting like "set rlim_fd_max = 131072" to a file + /etc/system, and then reboot. + NOTE: + + Each term of the above formula means: + (Nm - 1) * 2 = "number of inter-gfmd connections":(Nm - 1) * ("descriptor of the socket":1 + @@ -155,11 +184,11 @@ for backend database connection and file descriptor to access journal file, etc. - The number of "sockets for gfsd which servers the client" may - be smaller than Nf, but it may reach Nf if many Gfarm files are - opened by the client at once. Even if number of simultaneously - opened Gfarm files is less than Nf, it may reach - min(Nf, gfsd_connection_cache parameter). + An actual number of "sockets for gfsd which servers the client" + may be smaller than Nf, but it may reach Nf if many Gfarm files + are opened by the client at once. Or, even if number of + simultaneously opened Gfarm files is less than Nf, it may reach + min(Nf, "gfsd_connection_cache" parameter). The gfmd memory size which is needed by this parameter is about 320 byte per descriptor. Modified: gfarm_v2/branches/2.5/doc/Gfarm-FAQ.ja =================================================================== --- gfarm_v2/branches/2.5/doc/Gfarm-FAQ.ja 2012-07-20 04:25:38 UTC (rev 6437) +++ gfarm_v2/branches/2.5/doc/Gfarm-FAQ.ja 2012-07-20 04:29:43 UTC (rev 6438) @@ -124,16 +124,43 @@ \xA4\xB3\xA4\xCE\xC3ͤϡ\xA4gfmd.conf \xA4\xCE metadb_server_max_descriptors \xA4ǻ\xD8\xC4ꤷ\xA4ޤ\xB9\xA1\xA5 - gfmd \xA4\xCE\xCD\xF8\xCDѤ\xB9\xA4\xEB\xA5ե\xA1\xA5\xA4\xA5\xEB\xA5ǥ\xA3\xA5\xB9\xA5\xAF\xA5\xEA\xA5ץ\xBF\xBF\xF4\xA4ϡ\xA4 + \xC0\xDF\xC4ꤹ\xA4\xEB\xC3ͤη\xBB\xCA\xFDˡ\xA4ϰʲ\xBC\xA4\xCE\xC4̤\xEA\xA4Ǥ\xB9\xA1\xA5 Nm: \xA5\xA5ǡ\xBC\xA5\xBF\xA5\xB5\xA1\xBC\xA5п\xF4\xA4\xCE\xC1\xDB\xC4\xEA\xBA\xC7\xC2\xE7\xC3\xCD Nf: \xA5ե\xA1\xA5\xA4\xA5륷\xA5\xB9\xA5ƥ\xE0\xA5Ρ\xBC\xA5ɿ\xF4\xA4\xCE\xC1\xDB\xC4\xEA\xBA\xC7\xC2\xE7\xC3\xCD Nc: Ʊ\xBB\xFE\xCD\xF8\xCDѥ\xAF\xA5饤\xA5\xA2\xA5\xF3\xA5ȡ\xA6\xA5ץ\x{D97B}\xA5\xB9\xBF\xF4\xA4\xCE\xC1\xDB\xC4\xEA\xBA\xC7\xC2\xE7\xC3\xCD - \xA4Ȥ\xB9\xA4\xEB\xA4ȡ\xA4 + \xA4Ȥ\xB9\xA4\xEB\xA4ȡ\xA4\xB2\xBC\xB5\xAD\xA4\xCE\xC3Ͱʾ\xE5\xA4\xCB\xC0\xDF\xC4ꤷ\xA4Ƥ\xAA\xA4\xAFɬ\xCDפ\xAC\xA4\xA2\xA4\xEA\xA4ޤ\xB9\xA1\xA5 + (Nm - 1) * 2 + Nf * (Nf + 1) + Nc * (Nf + 1) + \xBC㴳\xBF\xF4 - (Nm - 1) * 2 + Nf * (Nf + 1) + Nc * (Nf + 1) + \xBC㴳\xBF\xF4 - \xB0ʾ\xE5\xA4\xCB\xC0\xDF\xC4ꤷ\xA4Ƥ\xAA\xA4\xAFɬ\xCDפ\xAC\xA4\xA2\xA4\xEA\xA4ޤ\xB9\xA1\xA5 + gfmd \xB5\xAFư\xBB\xFE\xA4ˡ\xA4\xA5\xB7\xA5\xB9\xA5ƥ\xE0\xA4Υ\x{D970} (/var/log/messages \xA4ʤ\xC9) \xA4\xCB + \xB0ʲ\xBC\xA4Τ褦\xA4\xCBɽ\xBC\xA8\xA4\xB5\xA4\xEC\xA4ޤ\xB9\xA4Τǡ\xA4\xB3\xCEǧ\xA4\xB7\xA4ƤߤƤ\xAF\xA4\xC0\xA4\xB5\xA4\xA4\xA1\xA5 + gfmd[\xA1\xC1]: \xA1\xC1 max descriptors = 131072 + \xA4\xB3\xA4\xB3\xA4\xC7ɽ\xBC\xA8\xA4\xB5\xA4줿\xC3ͤ\xAC metadb_server_max_descriptors \xA4ǻ\xD8\xC4ꤷ\xA4\xBF\xC3\xCD + \xA4\xE8\xA4\xEA\xA4⾮\xA4\xB5\xA4\xA4\xBE\xEC\xB9\xE7\xA4ϡ\xA4OS \xA4\xCE hard limit \xC0\xDF\xC4\xEA\xA4ˤ\xE8\xA4\xEA\xC0\xA9\xB8¤\xF2\xBC\x{164C64}Ƥ\xA4\xA4ޤ\xB9\xA1\xA5 + hard limit \xC0\xDF\xC4\xEA\xA4\xF2\xC1\xFD\xA4䤹\xA4ˤϡ\xA4\xB0ʲ\xBC\xA4Τ褦\xA4ˤ\xB7\xA4ޤ\xB9\xA1\xA5 + Linux \xA4ξ\xEC\xB9\xE7: + Linux \xA4ξ\xEC\xB9\xE7\xA4ϡ\xA4hard limit \xC0\xDF\xC4\xEA\xA4ˤ\xE8\xA4\xEB\xC0\xA9\xB8¤ǤϤʤ\xAF\xA1\xA4Gfarm \xA4\xAC + sysctl fs.file-max \xC0\xDF\xC4\xEA\xA4\xCE 75% ̤\xCB\xFE\xA4Ȥʤ\xEB\xA4褦\xC0\xA9\xB8¤\xB7\xA4Ƥ\xA4\xA4ޤ\xB9\xA1\xA5 + \xA4\xB3\xA4Τ\xBF\xA4\xE1 gfmd.conf \xA4Ρ\xD6metadb_server_max_descriptors 131072\xA1\xD7 + \xA4Ȥ\xA4\xA4\xA6\xC0\xDF\xC4\xEA\xA4\xF2\xBCºݤ\xCBͭ\xB8\xFA\xA4ˤ\xB9\xA4\xEB\xA4ˤϡ\xA4\xA5ե\xA1\xA5\xA4\xA5\xEB /etc/sysctl.conf \xA4\xCB + \xA1\xD6fs.file-max=180224\xA1פȤ\xA4\xA4ä\xBF\xC0\xDF\xC4\xEA\xC6\xFE\xA4줿\xB8塢\xA5\xEA\xA5֡\xBC\xA5Ȥ\xB9\xA4뤫\xA1\xA4 + \xA4\xA2\xA4뤤\xA4ϥ\xB3\xA5ޥ\xF3\xA5ɡ\xD6/sbin/sysctl -p\xA1פ\xF2\xBC¹Ԥ\xB7\xA4ޤ\xB9\xA1\xA5 + NetBSD \xA4ξ\xEC\xB9\xE7: + \xA5ե\xA1\xA5\xA4\xA5\xEB /etc/sysctl.conf \xA4ˡ\xD6kern.maxfiles=180224\xA1פȤ\xA4\xA4ä\xBF + \xC0\xDF\xC4\xEA\xA4\xF2\xB9Ԥʤ\xA4\xA1\xA4\xA5\xEA\xA5֡\xBC\xA5Ȥ\xB9\xA4뤫\xA1\xA4\xA4\xA2\xA4뤤\xA4ϥ\xB3\xA5ޥ\xF3\xA5\xC9 + \xA1\xD6/sbin/sysctl -f /etc/sysctl.conf\xA1פ\xF2\xBC¹Ԥ\xB7\xA4ޤ\xB9\xA1\xA5 + FreeBSD \xA4ξ\xEC\xB9\xE7: + \xA5ե\xA1\xA5\xA4\xA5\xEB /etc/sysctl.conf \xA4˰ʲ\xBC\xA4Τ褦\xA4\xCA\xC0\xDF\xC4\xEA\xA4\xF2\xB9Ԥʤ\xA4\xA1\xA4 + kern.maxfiles=180224 + kern.maxfilesperproc=131072 + \xA5\xEA\xA5֡\xBC\xA5Ȥ\xB9\xA4뤫\xA1\xA4\xA4\xA2\xA4뤤\xA4ϥ\xB3\xA5ޥ\xF3\xA5\xC9 + \xA1\xD6/sbin/sysctl -f /etc/sysctl.conf\xA1פ\xF2\xBC¹Ԥ\xB7\xA4ޤ\xB9\xA1\xA5 + Solaris \xA4ξ\xEC\xB9\xE7: + \xA5ե\xA1\xA5\xA4\xA5\xEB /etc/system \xA4ˡ\xD6set rlim_fd_max = 131072\xA1פȤ\xA4\xA4ä\xBF + \xC0\xDF\xC4\xEA\xA4\xF2\xB9Ԥʤ\xA4\xA1\xA4\xA5\xEA\xA5֡\xBC\xA5Ȥ\xB7\xA4ޤ\xB9\xA1\xA5 + + \xBB\xB2\xB9\xCD: - \xA4\xB3\xA4\xB3\xA4ǡ\xA4\xB3ƹ\xE0\xA4ΰ\xD5̣\xA4ϰʲ\xBC\xA4\xCE\xC4̤\xEA\xA4Ǥ\xB9\xA1\xA5 + \xB7\xBB\xBC\xB0\xA4γƹ\xE0\xA4ΰ\xD5̣\xA4ϡ\xA4\xB0ʲ\xBC\xA4\xCE\xC4̤\xEA\xA4Ǥ\xB9\xA1\xA5 (Nm - 1) * 2 = gfmd \xB4\xD6\xC0\xDC³\xA4ο\xF4:(Nm - 1) * @@ -155,7 +182,7 @@ \xA4\xB3\xA4Τ\xA6\xA4\xC1\xA1\xA4\xA1֥\xAF\xA5饤\xA5\xA2\xA5\xF3\xA5\xC8\xC2б\xFE\xCD\xD1gfsd\xA4\xC8gfmd\xA4δ֤\xCE\xC0\xDC³\xA1פ\xCEʬ\xA4ϡ\xA4 \xBCºݤˤϤ\xE2\xA4äȾ\xAE\xA4\xB5\xA4\xAF\xA4ʤ\xEB\xB2\xC4ǽ\xC0\xAD\xA4⤢\xA4\xEA\xA4ޤ\xB9\xA1\xA5\xA4\xB7\xA4\xAB\xA4\xB7Ʊ\xBB\xFE\xA4\xCB¿\xBF\xF4\xA4\xCE Gfarm\xA5ե\xA1\xA5\xA4\xA5\xEB\xA4\xBC\xA5ץƤ\xA4\xA4\xEB\xBE\xEC\xB9\xE7\xA4ϡ\xA4\xBA\xC7\xC2\xE7\xA4\xC7 Nf \xA4ޤ\xC7ã\xA4\xB9\xA4\xEB\xB2\xC4ǽ\xC0\xAD\xA4\xE2 - \xA4\xA2\xA4\xEA\xA4ޤ\xB9\xA1\xA5\xA4ޤ\xBF\xA1\xA4\xA4⤷Ʊ\xBB\xFE\xA5\xAA\xA1\xBC\xA5ץ\xF3\xBF\xF4\xA4\xAC Nf \xA4\xE8\xA4꾯\xA4ʤ\xA4\xBE\xEC\xB9\xE7\xA4Ǥ\xE2 + \xA4\xA2\xA4\xEA\xA4ޤ\xB9\xA1\xA5\xA4\xA2\xA4뤤\xA4ϡ\xA4\xA4⤷Ʊ\xBB\xFE\xA5\xAA\xA1\xBC\xA5ץ\xF3\xBF\xF4\xA4\xAC Nf \xA4\xE8\xA4꾯\xA4ʤ\xA4\xBE\xEC\xB9\xE7\xA4Ǥ\xE2 gfsd_connection_cache \xA5ѥ\xE9\xA5\xA5\xBF\xA4\xF2\xC0\xDF\xC4ꤷ\xA4Ƥ\xA4\xA4\xEB\xBE\xEC\xB9\xE7\xA4ϡ\xA4 min(Nf, gfsd_connection_cache) \xA4ޤ\xC7ã\xA4\xB7\xA4ޤ\xB9\xA1\xA5 Property changes on: gfarm_v2/branches/2.5/gftool/gfgroup/gfgroup.c ___________________________________________________________________ Modified: svn:mergeinfo - /gfarm_v2/branches/2.4/gftool/gfgroup/gfgroup.c:5271,5275-5281,5290,5329,5335 /gfarm_v2/trunk/gftool/gfgroup/gfgroup.c:5598,5616-5624,5627-5628,5641-5647,5650,5653,5657-5666,5670,5674,5683-5685,5693,5695-5696,5699,5719-5720,5727,5729-5730,5775,5793,5799,5808-5809,5821,5827-5831,5839-5842,5844-5849,5863-5864,5868,5870,5874-5878,5886,5889,5913,5915,5924,5932,5934,5959,6001-6003,6012-6013,6016,6019,6025-6027,6029,6040,6073,6078,6091,6093,6118,6137,6152-6153,6155,6159,6161,6166,6168,6173,6175,6178-6179,6184-6185,6188,6268,6272,6295,6305-6306,6310,6315,6333,6335,6354,6397,6427 + /gfarm_v2/branches/2.4/gftool/gfgroup/gfgroup.c:5271,5275-5281,5290,5329,5335 /gfarm_v2/trunk/gftool/gfgroup/gfgroup.c:5598,5616-5624,5627-5628,5641-5647,5650,5653,5657-5666,5670,5674,5683-5685,5693,5695-5696,5699,5719-5720,5727,5729-5730,5775,5793,5799,5808-5809,5821,5827-5831,5839-5842,5844-5849,5863-5864,5868,5870,5874-5878,5886,5889,5913,5915,5924,5932,5934,5959,6001-6003,6012-6013,6016,6019,6025-6027,6029,6040,6073,6078,6091,6093,6118,6137,6152-6153,6155,6159,6161,6166,6168,6173,6175,6178-6179,6184-6185,6188,6268,6272,6295,6305-6306,6310,6315,6333,6335,6354,6397,6427,6437 This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site. |