Menu
▾
▴
[Postgres-xc-general] Datanode is not starting up after "No space left on device"
|
From: Juned K. <jkh...@gm...> - 2014-04-14 10:26:16
|
Hi all, Yesterday due some problem datanode slave stopped working on one of my DB server. After figuring out the issue i freed out the space on that server. now almost 50% disk is free on server. the problem is after that incident i am not able start all my pgxc components, now its uncertain some of components starts and sometimes it does not each time when i execute "stop all" and "start all" scenario is different.I am not able to figure out the problem which causing this issue. I have enabled the debug log though and tried to identify the issue and after reviewing the logs it seems memory related issue, i am not sure about this. PGXC start all Start GTM master gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist Is server running? server starting Start GTM slavegtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist Is server running? server starting Done. Starting all the gtm proxies. Starting gtm proxy gtm_pxy1. Starting gtm proxy gtm_pxy2. gtm_proxy: no process found server starting gtm_proxy: no process found server starting Done. Starting coordinator master. Starting coordinator master coord1 Starting coordinator master coord2 DEBUG: 00000: postgres: PostmasterMain: initial environment dump: LOCATION: PostmasterMain, postmaster.c:962 DEBUG: 00000: ----------------------------------------- LOCATION: PostmasterMain, postmaster.c:964 DEBUG: 00000: MAIL=/var/mail/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SSH_CLIENT=41.218.72.115 48992 59696 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: USER=postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LANGUAGE=en_ZA:en LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SHLVL=1 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: HOME=/home/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PG_GRANDPARENT_PID=19891 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LOGNAME=postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: _=/usr/local/bin/pg_ctl LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LANG=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SHELL=/bin/bash LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PWD=/home/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_MONETARY=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_NUMERIC=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_TIME=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: ----------------------------------------- LOCATION: PostmasterMain, postmaster.c:969 DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 LOCATION: SlruScanDirectory, slru.c:1312 DEBUG: 00000: removing file "pg_notify/0000" LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 LOCATION: set_max_safe_fds, fd.c:548 LOG: 00000: redirecting log output to logging collector process HINT: Future log output will appear in directory "pg_log". LOCATION: SysLogger_Start, syslogger.c:649 DEBUG: 00000: postgres: PostmasterMain: initial environment dump: LOCATION: PostmasterMain, postmaster.c:962 DEBUG: 00000: ----------------------------------------- LOCATION: PostmasterMain, postmaster.c:964 DEBUG: 00000: MAIL=/var/mail/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SSH_CLIENT=41.218.72.115 33169 59696 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: USER=postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LANGUAGE=en_ZA:en LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SHLVL=1 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: HOME=/home/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PG_GRANDPARENT_PID=19507 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LOGNAME=postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: _=/usr/local/bin/pg_ctl LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LANG=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SHELL=/bin/bash LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PWD=/home/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_MONETARY=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_NUMERIC=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_TIME=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: ----------------------------------------- LOCATION: PostmasterMain, postmaster.c:969 DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 LOCATION: SlruScanDirectory, slru.c:1312 DEBUG: 00000: removing file "pg_notify/0000" LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 LOCATION: set_max_safe_fds, fd.c:548 LOG: 00000: redirecting log output to logging collector process HINT: Future log output will appear in directory "pg_log". LOCATION: SysLogger_Start, syslogger.c:649 DEBUG: 00000: logger shutting down LOCATION: SysLoggerMain, syslogger.c:517 DEBUG: 00000: shmem_exit(0): 0 callbacks to make LOCATION: shmem_exit, ipc.c:212 DEBUG: 00000: proc_exit(0): 0 callbacks to make LOCATION: proc_exit_prepare, ipc.c:184 DEBUG: 00000: exit(0) LOCATION: proc_exit, ipc.c:135 DEBUG: 00000: shmem_exit(-1): 0 callbacks to make LOCATION: shmem_exit, ipc.c:212 DEBUG: 00000: proc_exit(-1): 0 callbacks to make LOCATION: proc_exit_prepare, ipc.c:184 pg_ctl: could not start server Examine the log output. Done. Starting all the datanode masters. Starting datanode master datanode1. LOG: redirecting log output to logging collector process HINT: Future log output will appear in directory "pg_log". Done. Starting all the datanode slaves. Starting datanode slave datanode1. LOG: redirecting log output to logging collector process HINT: Future log output will appear in directory "pg_log". Done. PGXC monitor all Running: gtm master Running: gtm slave Running: gtm proxy gtm_pxy1 Running: gtm proxy gtm_pxy2 Running: coordinator master coord1 Not running: coordinator master coord2 Not running: datanode master datanode1 Running: datanode slave datanode1 My DB server has 32 GB RAM and 260 GB of Hard disk and i am using pgxc-1.2.1. what would be the optimal memory related postgresql.conf configuration for this server. Anyone has an idea about this issue ? -- Thanks, Juned Khan iNextrix Technologies Pvt Ltd. www.inextrix.com |
