Menu
▾
▴
postgres-xc-general
|
From: Juned K. <jkh...@gm...> - 2014-04-14 10:26:16
|
Hi all, Yesterday due some problem datanode slave stopped working on one of my DB server. After figuring out the issue i freed out the space on that server. now almost 50% disk is free on server. the problem is after that incident i am not able start all my pgxc components, now its uncertain some of components starts and sometimes it does not each time when i execute "stop all" and "start all" scenario is different.I am not able to figure out the problem which causing this issue. I have enabled the debug log though and tried to identify the issue and after reviewing the logs it seems memory related issue, i am not sure about this. PGXC start all Start GTM master gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist Is server running? server starting Start GTM slavegtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist Is server running? server starting Done. Starting all the gtm proxies. Starting gtm proxy gtm_pxy1. Starting gtm proxy gtm_pxy2. gtm_proxy: no process found server starting gtm_proxy: no process found server starting Done. Starting coordinator master. Starting coordinator master coord1 Starting coordinator master coord2 DEBUG: 00000: postgres: PostmasterMain: initial environment dump: LOCATION: PostmasterMain, postmaster.c:962 DEBUG: 00000: ----------------------------------------- LOCATION: PostmasterMain, postmaster.c:964 DEBUG: 00000: MAIL=/var/mail/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SSH_CLIENT=41.218.72.115 48992 59696 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: USER=postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LANGUAGE=en_ZA:en LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SHLVL=1 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: HOME=/home/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PG_GRANDPARENT_PID=19891 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LOGNAME=postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: _=/usr/local/bin/pg_ctl LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LANG=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SHELL=/bin/bash LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PWD=/home/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_MONETARY=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_NUMERIC=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_TIME=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: ----------------------------------------- LOCATION: PostmasterMain, postmaster.c:969 DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 LOCATION: SlruScanDirectory, slru.c:1312 DEBUG: 00000: removing file "pg_notify/0000" LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 LOCATION: set_max_safe_fds, fd.c:548 LOG: 00000: redirecting log output to logging collector process HINT: Future log output will appear in directory "pg_log". LOCATION: SysLogger_Start, syslogger.c:649 DEBUG: 00000: postgres: PostmasterMain: initial environment dump: LOCATION: PostmasterMain, postmaster.c:962 DEBUG: 00000: ----------------------------------------- LOCATION: PostmasterMain, postmaster.c:964 DEBUG: 00000: MAIL=/var/mail/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SSH_CLIENT=41.218.72.115 33169 59696 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: USER=postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LANGUAGE=en_ZA:en LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SHLVL=1 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: HOME=/home/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PG_GRANDPARENT_PID=19507 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LOGNAME=postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: _=/usr/local/bin/pg_ctl LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LANG=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SHELL=/bin/bash LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PWD=/home/postgres LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_MONETARY=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_NUMERIC=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: LC_TIME=C LOCATION: PostmasterMain, postmaster.c:967 DEBUG: 00000: ----------------------------------------- LOCATION: PostmasterMain, postmaster.c:969 DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 LOCATION: SlruScanDirectory, slru.c:1312 DEBUG: 00000: removing file "pg_notify/0000" LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 LOCATION: set_max_safe_fds, fd.c:548 LOG: 00000: redirecting log output to logging collector process HINT: Future log output will appear in directory "pg_log". LOCATION: SysLogger_Start, syslogger.c:649 DEBUG: 00000: logger shutting down LOCATION: SysLoggerMain, syslogger.c:517 DEBUG: 00000: shmem_exit(0): 0 callbacks to make LOCATION: shmem_exit, ipc.c:212 DEBUG: 00000: proc_exit(0): 0 callbacks to make LOCATION: proc_exit_prepare, ipc.c:184 DEBUG: 00000: exit(0) LOCATION: proc_exit, ipc.c:135 DEBUG: 00000: shmem_exit(-1): 0 callbacks to make LOCATION: shmem_exit, ipc.c:212 DEBUG: 00000: proc_exit(-1): 0 callbacks to make LOCATION: proc_exit_prepare, ipc.c:184 pg_ctl: could not start server Examine the log output. Done. Starting all the datanode masters. Starting datanode master datanode1. LOG: redirecting log output to logging collector process HINT: Future log output will appear in directory "pg_log". Done. Starting all the datanode slaves. Starting datanode slave datanode1. LOG: redirecting log output to logging collector process HINT: Future log output will appear in directory "pg_log". Done. PGXC monitor all Running: gtm master Running: gtm slave Running: gtm proxy gtm_pxy1 Running: gtm proxy gtm_pxy2 Running: coordinator master coord1 Not running: coordinator master coord2 Not running: datanode master datanode1 Running: datanode slave datanode1 My DB server has 32 GB RAM and 260 GB of Hard disk and i am using pgxc-1.2.1. what would be the optimal memory related postgresql.conf configuration for this server. Anyone has an idea about this issue ? -- Thanks, Juned Khan iNextrix Technologies Pvt Ltd. www.inextrix.com |
|
From: Juned K. <jkh...@gm...> - 2014-04-14 11:12:02
|
Here few more logs of the component which doesn't start root@db02:~# tail -f /home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log LOG: database system was interrupted; last known up at 2014-04-14 11:00:55 WAT LOG: database system was not properly shut down; automatic recovery in progress LOG: record with zero length at 4/9A01A588 LOG: redo is not required FATAL: the database system is starting up LOG: autovacuum launcher started LOG: database system is ready to accept connections WARNING: worker took too long to start; canceled FATAL: Can not register Datanode on GTM root@db02:~# tail -f /home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log DEBUG: 00000: name: unnamed; blockState: DEFAULT; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children: LOCATION: ShowTransactionStateRec, xact.c:5238 DEBUG: 00000: Autovacuum launcher: connection to GTM closed LOCATION: CloseGTM, gtm.c:116 DEBUG: 00000: Autovacuum launcher: connection established to GTM with string host=db02 port=20002 node_name=coord1 LOCATION: InitGTM, gtm.c:84 WARNING: 01000: Xid is invalid. LOCATION: GetNewTransactionId, varsup.c:160 DEBUG: 00000: Getting snapshot. Current XID = 0 LOCATION: GetSnapshotDataCoordinator, procarray.c:3054 DEBUG: 00000: Autovacuum launcher: connection to GTM closed LOCATION: CloseGTM, gtm.c:116 DEBUG: 00000: Autovacuum launcher: connection established to GTM with string host=db02 port=20002 node_name=coord1 LOCATION: InitGTM, gtm.c:84 On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...> wrote: > Hi all, > > Yesterday due some problem datanode slave stopped working on one of my DB > server. After figuring out the issue i freed out the space on that server. > now almost 50% disk is free on server. > > the problem is after that incident i am not able start all my pgxc > components, now its uncertain some of components starts and sometimes it > does not each time when i execute "stop all" and "start all" scenario is > different.I am not able to figure out the problem which causing this issue. > > I have enabled the debug log though and tried to identify the issue and > after reviewing the logs it seems memory related issue, i am not sure about > this. > > PGXC start all > Start GTM master > gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist > Is server running? > server starting > Start GTM slavegtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" > does not exist > Is server running? > server starting > Done. > Starting all the gtm proxies. > Starting gtm proxy gtm_pxy1. > Starting gtm proxy gtm_pxy2. > gtm_proxy: no process found > server starting > gtm_proxy: no process found > server starting > Done. > Starting coordinator master. > Starting coordinator master coord1 > Starting coordinator master coord2 > DEBUG: 00000: postgres: PostmasterMain: initial environment dump: > LOCATION: PostmasterMain, postmaster.c:962 > DEBUG: 00000: ----------------------------------------- > LOCATION: PostmasterMain, postmaster.c:964 > DEBUG: 00000: MAIL=/var/mail/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SSH_CLIENT=41.218.72.115 48992 59696 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: USER=postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LANGUAGE=en_ZA:en > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SHLVL=1 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: HOME=/home/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PG_GRANDPARENT_PID=19891 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LOGNAME=postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: _=/usr/local/bin/pg_ctl > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LANG=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SHELL=/bin/bash > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PWD=/home/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_MONETARY=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_NUMERIC=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_TIME=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: ----------------------------------------- > LOCATION: PostmasterMain, postmaster.c:969 > DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) > LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 > DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 > LOCATION: SlruScanDirectory, slru.c:1312 > DEBUG: 00000: removing file "pg_notify/0000" > LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 > DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 > LOCATION: set_max_safe_fds, fd.c:548 > LOG: 00000: redirecting log output to logging collector process > HINT: Future log output will appear in directory "pg_log". > LOCATION: SysLogger_Start, syslogger.c:649 > DEBUG: 00000: postgres: PostmasterMain: initial environment dump: > LOCATION: PostmasterMain, postmaster.c:962 > DEBUG: 00000: ----------------------------------------- > LOCATION: PostmasterMain, postmaster.c:964 > DEBUG: 00000: MAIL=/var/mail/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SSH_CLIENT=41.218.72.115 33169 59696 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: USER=postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LANGUAGE=en_ZA:en > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SHLVL=1 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: HOME=/home/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PG_GRANDPARENT_PID=19507 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LOGNAME=postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: _=/usr/local/bin/pg_ctl > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LANG=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SHELL=/bin/bash > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PWD=/home/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_MONETARY=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_NUMERIC=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_TIME=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: ----------------------------------------- > LOCATION: PostmasterMain, postmaster.c:969 > DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) > LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 > DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 > LOCATION: SlruScanDirectory, slru.c:1312 > DEBUG: 00000: removing file "pg_notify/0000" > LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 > DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 > LOCATION: set_max_safe_fds, fd.c:548 > LOG: 00000: redirecting log output to logging collector process > HINT: Future log output will appear in directory "pg_log". > LOCATION: SysLogger_Start, syslogger.c:649 > DEBUG: 00000: logger shutting down > LOCATION: SysLoggerMain, syslogger.c:517 > DEBUG: 00000: shmem_exit(0): 0 callbacks to make > LOCATION: shmem_exit, ipc.c:212 > DEBUG: 00000: proc_exit(0): 0 callbacks to make > LOCATION: proc_exit_prepare, ipc.c:184 > DEBUG: 00000: exit(0) > LOCATION: proc_exit, ipc.c:135 > DEBUG: 00000: shmem_exit(-1): 0 callbacks to make > LOCATION: shmem_exit, ipc.c:212 > DEBUG: 00000: proc_exit(-1): 0 callbacks to make > LOCATION: proc_exit_prepare, ipc.c:184 > pg_ctl: could not start server > Examine the log output. > Done. > Starting all the datanode masters. > Starting datanode master datanode1. > LOG: redirecting log output to logging collector process > HINT: Future log output will appear in directory "pg_log". > Done. > Starting all the datanode slaves. > Starting datanode slave datanode1. > LOG: redirecting log output to logging collector process > HINT: Future log output will appear in directory "pg_log". > Done. > PGXC monitor all > Running: gtm master > Running: gtm slave > Running: gtm proxy gtm_pxy1 > Running: gtm proxy gtm_pxy2 > Running: coordinator master coord1 > Not running: coordinator master coord2 > Not running: datanode master datanode1 > Running: datanode slave datanode1 > > > My DB server has 32 GB RAM and 260 GB of Hard disk and i am using > pgxc-1.2.1. what would be the optimal memory related postgresql.conf > configuration for this server. > > Anyone has an idea about this issue ? > > -- > Thanks, > Juned Khan > iNextrix Technologies Pvt Ltd. > www.inextrix.com > -- Thanks, Juned Khan iNextrix Technologies Pvt Ltd. www.inextrix.com |
|
From: Masataka S. <pg...@gm...> - 2014-04-15 01:38:02
|
I'm not sure but GTM or GTM proxy seems to be wrong. Please check their log. If you find an issue, search postgres-xc-general ml for keyword "register.node". Regards. On 14 April 2014 20:11, Juned Khan <jkh...@gm...> wrote: > Here few more logs of the component which doesn't start > > root@db02:~# tail -f > /home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log > LOG: database system was interrupted; last known up at 2014-04-14 11:00:55 > WAT > LOG: database system was not properly shut down; automatic recovery in > progress > LOG: record with zero length at 4/9A01A588 > LOG: redo is not required > FATAL: the database system is starting up > LOG: autovacuum launcher started > LOG: database system is ready to accept connections > WARNING: worker took too long to start; canceled > FATAL: Can not register Datanode on GTM > > > root@db02:~# tail -f > /home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log > DEBUG: 00000: name: unnamed; blockState: DEFAULT; state: INPROGR, > xid/subid/cid: 0/1/0, nestlvl: 1, children: > LOCATION: ShowTransactionStateRec, xact.c:5238 > DEBUG: 00000: Autovacuum launcher: connection to GTM closed > LOCATION: CloseGTM, gtm.c:116 > DEBUG: 00000: Autovacuum launcher: connection established to GTM with > string host=db02 port=20002 node_name=coord1 > LOCATION: InitGTM, gtm.c:84 > WARNING: 01000: Xid is invalid. > LOCATION: GetNewTransactionId, varsup.c:160 > DEBUG: 00000: Getting snapshot. Current XID = 0 > LOCATION: GetSnapshotDataCoordinator, procarray.c:3054 > DEBUG: 00000: Autovacuum launcher: connection to GTM closed > LOCATION: CloseGTM, gtm.c:116 > DEBUG: 00000: Autovacuum launcher: connection established to GTM with > string host=db02 port=20002 node_name=coord1 > LOCATION: InitGTM, gtm.c:84 > > > > > > On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...> wrote: >> >> Hi all, >> >> Yesterday due some problem datanode slave stopped working on one of my DB >> server. After figuring out the issue i freed out the space on that server. >> now almost 50% disk is free on server. >> >> the problem is after that incident i am not able start all my pgxc >> components, now its uncertain some of components starts and sometimes it >> does not each time when i execute "stop all" and "start all" scenario is >> different.I am not able to figure out the problem which causing this issue. >> >> I have enabled the debug log though and tried to identify the issue and >> after reviewing the logs it seems memory related issue, i am not sure about >> this. >> >> PGXC start all >> Start GTM master >> gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist >> Is server running? >> server starting >> Start GTM slavegtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" >> does not exist >> Is server running? >> server starting >> Done. >> Starting all the gtm proxies. >> Starting gtm proxy gtm_pxy1. >> Starting gtm proxy gtm_pxy2. >> gtm_proxy: no process found >> server starting >> gtm_proxy: no process found >> server starting >> Done. >> Starting coordinator master. >> Starting coordinator master coord1 >> Starting coordinator master coord2 >> DEBUG: 00000: postgres: PostmasterMain: initial environment dump: >> LOCATION: PostmasterMain, postmaster.c:962 >> DEBUG: 00000: ----------------------------------------- >> LOCATION: PostmasterMain, postmaster.c:964 >> DEBUG: 00000: MAIL=/var/mail/postgres >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: SSH_CLIENT=41.218.72.115 48992 59696 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: USER=postgres >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LANGUAGE=en_ZA:en >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: SHLVL=1 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: HOME=/home/postgres >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: PG_GRANDPARENT_PID=19891 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LOGNAME=postgres >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: _=/usr/local/bin/pg_ctl >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LANG=en_ZA.UTF-8 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: SHELL=/bin/bash >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: PWD=/home/postgres >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_MONETARY=C >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_NUMERIC=C >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_TIME=C >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: ----------------------------------------- >> LOCATION: PostmasterMain, postmaster.c:969 >> DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) >> LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 >> DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 >> LOCATION: SlruScanDirectory, slru.c:1312 >> DEBUG: 00000: removing file "pg_notify/0000" >> LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 >> DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 >> LOCATION: set_max_safe_fds, fd.c:548 >> LOG: 00000: redirecting log output to logging collector process >> HINT: Future log output will appear in directory "pg_log". >> LOCATION: SysLogger_Start, syslogger.c:649 >> DEBUG: 00000: postgres: PostmasterMain: initial environment dump: >> LOCATION: PostmasterMain, postmaster.c:962 >> DEBUG: 00000: ----------------------------------------- >> LOCATION: PostmasterMain, postmaster.c:964 >> DEBUG: 00000: MAIL=/var/mail/postgres >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: SSH_CLIENT=41.218.72.115 33169 59696 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: USER=postgres >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LANGUAGE=en_ZA:en >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: SHLVL=1 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: HOME=/home/postgres >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: PG_GRANDPARENT_PID=19507 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LOGNAME=postgres >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: _=/usr/local/bin/pg_ctl >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LANG=en_ZA.UTF-8 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: SHELL=/bin/bash >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: PWD=/home/postgres >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_MONETARY=C >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_NUMERIC=C >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: LC_TIME=C >> LOCATION: PostmasterMain, postmaster.c:967 >> DEBUG: 00000: ----------------------------------------- >> LOCATION: PostmasterMain, postmaster.c:969 >> DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) >> LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 >> DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 >> LOCATION: SlruScanDirectory, slru.c:1312 >> DEBUG: 00000: removing file "pg_notify/0000" >> LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 >> DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 >> LOCATION: set_max_safe_fds, fd.c:548 >> LOG: 00000: redirecting log output to logging collector process >> HINT: Future log output will appear in directory "pg_log". >> LOCATION: SysLogger_Start, syslogger.c:649 >> DEBUG: 00000: logger shutting down >> LOCATION: SysLoggerMain, syslogger.c:517 >> DEBUG: 00000: shmem_exit(0): 0 callbacks to make >> LOCATION: shmem_exit, ipc.c:212 >> DEBUG: 00000: proc_exit(0): 0 callbacks to make >> LOCATION: proc_exit_prepare, ipc.c:184 >> DEBUG: 00000: exit(0) >> LOCATION: proc_exit, ipc.c:135 >> DEBUG: 00000: shmem_exit(-1): 0 callbacks to make >> LOCATION: shmem_exit, ipc.c:212 >> DEBUG: 00000: proc_exit(-1): 0 callbacks to make >> LOCATION: proc_exit_prepare, ipc.c:184 >> pg_ctl: could not start server >> Examine the log output. >> Done. >> Starting all the datanode masters. >> Starting datanode master datanode1. >> LOG: redirecting log output to logging collector process >> HINT: Future log output will appear in directory "pg_log". >> Done. >> Starting all the datanode slaves. >> Starting datanode slave datanode1. >> LOG: redirecting log output to logging collector process >> HINT: Future log output will appear in directory "pg_log". >> Done. >> PGXC monitor all >> Running: gtm master >> Running: gtm slave >> Running: gtm proxy gtm_pxy1 >> Running: gtm proxy gtm_pxy2 >> Running: coordinator master coord1 >> Not running: coordinator master coord2 >> Not running: datanode master datanode1 >> Running: datanode slave datanode1 >> >> >> My DB server has 32 GB RAM and 260 GB of Hard disk and i am using >> pgxc-1.2.1. what would be the optimal memory related postgresql.conf >> configuration for this server. >> >> Anyone has an idea about this issue ? >> >> -- >> Thanks, >> Juned Khan >> iNextrix Technologies Pvt Ltd. >> www.inextrix.com > > > > > -- > Thanks, > Juned Khan > iNextrix Technologies Pvt Ltd. > www.inextrix.com > > ------------------------------------------------------------------------------ > Learn Graph Databases - Download FREE O'Reilly Book > "Graph Databases" is the definitive new guide to graph databases and their > applications. Written by three acclaimed leaders in the field, > this first edition is now available. Download your free book today! > http://p.sf.net/sfu/NeoTech > _______________________________________________ > Postgres-xc-general mailing list > Pos...@li... > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > |
|
From: Juned K. <jkh...@gm...> - 2014-04-15 05:19:35
|
Hi Masataka, Thanks for help On Tue, Apr 15, 2014 at 7:07 AM, Masataka Saito <pg...@gm...> wrote: > I'm not sure but GTM or GTM proxy seems to be wrong. > > Please check their log. > If you find an issue, search postgres-xc-general ml for keyword > "register.node". > > Regards. > > On 14 April 2014 20:11, Juned Khan <jkh...@gm...> wrote: > > Here few more logs of the component which doesn't start > > > > root@db02:~# tail -f > > > /home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log > > LOG: database system was interrupted; last known up at 2014-04-14 > 11:00:55 > > WAT > > LOG: database system was not properly shut down; automatic recovery in > > progress > > LOG: record with zero length at 4/9A01A588 > > LOG: redo is not required > > FATAL: the database system is starting up > > LOG: autovacuum launcher started > > LOG: database system is ready to accept connections > > WARNING: worker took too long to start; canceled > > FATAL: Can not register Datanode on GTM > > > > > > root@db02:~# tail -f > > /home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log > > DEBUG: 00000: name: unnamed; blockState: DEFAULT; state: INPROGR, > > xid/subid/cid: 0/1/0, nestlvl: 1, children: > > LOCATION: ShowTransactionStateRec, xact.c:5238 > > DEBUG: 00000: Autovacuum launcher: connection to GTM closed > > LOCATION: CloseGTM, gtm.c:116 > > DEBUG: 00000: Autovacuum launcher: connection established to GTM with > > string host=db02 port=20002 node_name=coord1 > > LOCATION: InitGTM, gtm.c:84 > > WARNING: 01000: Xid is invalid. > > LOCATION: GetNewTransactionId, varsup.c:160 > > DEBUG: 00000: Getting snapshot. Current XID = 0 > > LOCATION: GetSnapshotDataCoordinator, procarray.c:3054 > > DEBUG: 00000: Autovacuum launcher: connection to GTM closed > > LOCATION: CloseGTM, gtm.c:116 > > DEBUG: 00000: Autovacuum launcher: connection established to GTM with > > string host=db02 port=20002 node_name=coord1 > > LOCATION: InitGTM, gtm.c:84 > > > > > > > > > > > > On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...> wrote: > >> > >> Hi all, > >> > >> Yesterday due some problem datanode slave stopped working on one of my > DB > >> server. After figuring out the issue i freed out the space on that > server. > >> now almost 50% disk is free on server. > >> > >> the problem is after that incident i am not able start all my pgxc > >> components, now its uncertain some of components starts and sometimes it > >> does not each time when i execute "stop all" and "start all" scenario is > >> different.I am not able to figure out the problem which causing this > issue. > >> > >> I have enabled the debug log though and tried to identify the issue and > >> after reviewing the logs it seems memory related issue, i am not sure > about > >> this. > >> > >> PGXC start all > >> Start GTM master > >> gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist > >> Is server running? > >> server starting > >> Start GTM slavegtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" > >> does not exist > >> Is server running? > >> server starting > >> Done. > >> Starting all the gtm proxies. > >> Starting gtm proxy gtm_pxy1. > >> Starting gtm proxy gtm_pxy2. > >> gtm_proxy: no process found > >> server starting > >> gtm_proxy: no process found > >> server starting > >> Done. > >> Starting coordinator master. > >> Starting coordinator master coord1 > >> Starting coordinator master coord2 > >> DEBUG: 00000: postgres: PostmasterMain: initial environment dump: > >> LOCATION: PostmasterMain, postmaster.c:962 > >> DEBUG: 00000: ----------------------------------------- > >> LOCATION: PostmasterMain, postmaster.c:964 > >> DEBUG: 00000: MAIL=/var/mail/postgres > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: SSH_CLIENT=41.218.72.115 48992 59696 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: USER=postgres > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LANGUAGE=en_ZA:en > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: SHLVL=1 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: HOME=/home/postgres > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: PG_GRANDPARENT_PID=19891 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LOGNAME=postgres > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: _=/usr/local/bin/pg_ctl > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: > PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LANG=en_ZA.UTF-8 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: SHELL=/bin/bash > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: PWD=/home/postgres > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_MONETARY=C > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_NUMERIC=C > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_TIME=C > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: ----------------------------------------- > >> LOCATION: PostmasterMain, postmaster.c:969 > >> DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) > >> LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 > >> DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 > >> LOCATION: SlruScanDirectory, slru.c:1312 > >> DEBUG: 00000: removing file "pg_notify/0000" > >> LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 > >> DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 > >> LOCATION: set_max_safe_fds, fd.c:548 > >> LOG: 00000: redirecting log output to logging collector process > >> HINT: Future log output will appear in directory "pg_log". > >> LOCATION: SysLogger_Start, syslogger.c:649 > >> DEBUG: 00000: postgres: PostmasterMain: initial environment dump: > >> LOCATION: PostmasterMain, postmaster.c:962 > >> DEBUG: 00000: ----------------------------------------- > >> LOCATION: PostmasterMain, postmaster.c:964 > >> DEBUG: 00000: MAIL=/var/mail/postgres > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: SSH_CLIENT=41.218.72.115 33169 59696 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: USER=postgres > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LANGUAGE=en_ZA:en > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: SHLVL=1 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: HOME=/home/postgres > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: PG_GRANDPARENT_PID=19507 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LOGNAME=postgres > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: _=/usr/local/bin/pg_ctl > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: > PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LANG=en_ZA.UTF-8 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: SHELL=/bin/bash > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: PWD=/home/postgres > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_MONETARY=C > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_NUMERIC=C > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: LC_TIME=C > >> LOCATION: PostmasterMain, postmaster.c:967 > >> DEBUG: 00000: ----------------------------------------- > >> LOCATION: PostmasterMain, postmaster.c:969 > >> DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) > >> LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 > >> DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 > >> LOCATION: SlruScanDirectory, slru.c:1312 > >> DEBUG: 00000: removing file "pg_notify/0000" > >> LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 > >> DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 > >> LOCATION: set_max_safe_fds, fd.c:548 > >> LOG: 00000: redirecting log output to logging collector process > >> HINT: Future log output will appear in directory "pg_log". > >> LOCATION: SysLogger_Start, syslogger.c:649 > >> DEBUG: 00000: logger shutting down > >> LOCATION: SysLoggerMain, syslogger.c:517 > >> DEBUG: 00000: shmem_exit(0): 0 callbacks to make > >> LOCATION: shmem_exit, ipc.c:212 > >> DEBUG: 00000: proc_exit(0): 0 callbacks to make > >> LOCATION: proc_exit_prepare, ipc.c:184 > >> DEBUG: 00000: exit(0) > >> LOCATION: proc_exit, ipc.c:135 > >> DEBUG: 00000: shmem_exit(-1): 0 callbacks to make > >> LOCATION: shmem_exit, ipc.c:212 > >> DEBUG: 00000: proc_exit(-1): 0 callbacks to make > >> LOCATION: proc_exit_prepare, ipc.c:184 > >> pg_ctl: could not start server > >> Examine the log output. > >> Done. > >> Starting all the datanode masters. > >> Starting datanode master datanode1. > >> LOG: redirecting log output to logging collector process > >> HINT: Future log output will appear in directory "pg_log". > >> Done. > >> Starting all the datanode slaves. > >> Starting datanode slave datanode1. > >> LOG: redirecting log output to logging collector process > >> HINT: Future log output will appear in directory "pg_log". > >> Done. > >> PGXC monitor all > >> Running: gtm master > >> Running: gtm slave > >> Running: gtm proxy gtm_pxy1 > >> Running: gtm proxy gtm_pxy2 > >> Running: coordinator master coord1 > >> Not running: coordinator master coord2 > >> Not running: datanode master datanode1 > >> Running: datanode slave datanode1 > >> > >> > >> My DB server has 32 GB RAM and 260 GB of Hard disk and i am using > >> pgxc-1.2.1. what would be the optimal memory related postgresql.conf > >> configuration for this server. > >> > >> Anyone has an idea about this issue ? > >> > >> -- > >> Thanks, > >> Juned Khan > >> iNextrix Technologies Pvt Ltd. > >> www.inextrix.com > > > > > > > > > > -- > > Thanks, > > Juned Khan > > iNextrix Technologies Pvt Ltd. > > www.inextrix.com > > > > > ------------------------------------------------------------------------------ > > Learn Graph Databases - Download FREE O'Reilly Book > > "Graph Databases" is the definitive new guide to graph databases and > their > > applications. Written by three acclaimed leaders in the field, > > this first edition is now available. Download your free book today! > > http://p.sf.net/sfu/NeoTech > > _______________________________________________ > > Postgres-xc-general mailing list > > Pos...@li... > > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general > > > -- Thanks, Juned Khan iNextrix Technologies Pvt Ltd. www.inextrix.com |
|
From: Juned K. <jkh...@gm...> - 2014-04-15 11:30:19
|
Hi Masataka, You answer helped but again i am having this problem. do i need to remove register.node file again ? GTM logs 1:140587359135488:2014-04-15 12:22:56.711 WAT -LOG: Saving transaction restoration info, backed-up gxid: 259975 LOCATION: GTM_WriteRestorePointXid, gtm_txn.c:2649 1:140587359135488:2014-04-15 12:22:56.712 WAT -LOG: Started to run as GTM-Active. LOCATION: main, main.c:641 1:140587359135488:2014-04-15 12:22:57.712 WAT -LOG: Any GTM standby node not found in registered node(s). LOCATION: gtm_standby_connect_to_standby_int, gtm_standby.c:381 1:140587350980352:2014-04-15 12:22:57.717 WAT -LOG: Failed to establish a connection with GTM standby. - 0x13ac390 LOCATION: gtm_standby_connect_to_standby_int, gtm_standby.c:396 1:140587321448192:2014-04-15 12:27:02.055 WAT -LOG: unexpected EOF on client connection LOCATION: ReadCommand, main.c:1374 1:140587359135488:2014-04-15 12:27:03.928 WAT -LOG: GTM shutting down. LOCATION: ServerLoop, main.c:802 1:140587359135488:2014-04-15 12:27:03.928 WAT -LOG: Saving transaction info - next_gxid: 257975 LOCATION: GTM_SaveTxnInfo, gtm_txn.c:2632 Received signal 15 GTM Proxy logs LOCATION: pgxcnode_add_info, register_common.c:249 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG: Node with the given ID number already exists LOCATION: pgxcnode_add_info, register_common.c:249 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG: Node with the given ID number already exists LOCATION: pgxcnode_add_info, register_common.c:249 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG: Node with the given ID number already exists LOCATION: pgxcnode_add_info, register_common.c:249 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG: Node with the given ID number already exists LOCATION: pgxcnode_add_info, register_common.c:249 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG: Node with the given ID number already exists LOCATION: pgxcnode_add_info, register_common.c:249 On Tue, Apr 15, 2014 at 10:49 AM, Juned Khan <jkh...@gm...> wrote: > Hi Masataka, > > Thanks for help > > > On Tue, Apr 15, 2014 at 7:07 AM, Masataka Saito <pg...@gm...> wrote: > >> I'm not sure but GTM or GTM proxy seems to be wrong. >> >> Please check their log. >> If you find an issue, search postgres-xc-general ml for keyword >> "register.node". >> >> Regards. >> >> On 14 April 2014 20:11, Juned Khan <jkh...@gm...> wrote: >> > Here few more logs of the component which doesn't start >> > >> > root@db02:~# tail -f >> > >> /home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log >> > LOG: database system was interrupted; last known up at 2014-04-14 >> 11:00:55 >> > WAT >> > LOG: database system was not properly shut down; automatic recovery in >> > progress >> > LOG: record with zero length at 4/9A01A588 >> > LOG: redo is not required >> > FATAL: the database system is starting up >> > LOG: autovacuum launcher started >> > LOG: database system is ready to accept connections >> > WARNING: worker took too long to start; canceled >> > FATAL: Can not register Datanode on GTM >> > >> > >> > root@db02:~# tail -f >> > /home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log >> > DEBUG: 00000: name: unnamed; blockState: DEFAULT; state: INPROGR, >> > xid/subid/cid: 0/1/0, nestlvl: 1, children: >> > LOCATION: ShowTransactionStateRec, xact.c:5238 >> > DEBUG: 00000: Autovacuum launcher: connection to GTM closed >> > LOCATION: CloseGTM, gtm.c:116 >> > DEBUG: 00000: Autovacuum launcher: connection established to GTM with >> > string host=db02 port=20002 node_name=coord1 >> > LOCATION: InitGTM, gtm.c:84 >> > WARNING: 01000: Xid is invalid. >> > LOCATION: GetNewTransactionId, varsup.c:160 >> > DEBUG: 00000: Getting snapshot. Current XID = 0 >> > LOCATION: GetSnapshotDataCoordinator, procarray.c:3054 >> > DEBUG: 00000: Autovacuum launcher: connection to GTM closed >> > LOCATION: CloseGTM, gtm.c:116 >> > DEBUG: 00000: Autovacuum launcher: connection established to GTM with >> > string host=db02 port=20002 node_name=coord1 >> > LOCATION: InitGTM, gtm.c:84 >> > >> > >> > >> > >> > >> > On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...> >> wrote: >> >> >> >> Hi all, >> >> >> >> Yesterday due some problem datanode slave stopped working on one of my >> DB >> >> server. After figuring out the issue i freed out the space on that >> server. >> >> now almost 50% disk is free on server. >> >> >> >> the problem is after that incident i am not able start all my pgxc >> >> components, now its uncertain some of components starts and sometimes >> it >> >> does not each time when i execute "stop all" and "start all" scenario >> is >> >> different.I am not able to figure out the problem which causing this >> issue. >> >> >> >> I have enabled the debug log though and tried to identify the issue and >> >> after reviewing the logs it seems memory related issue, i am not sure >> about >> >> this. >> >> >> >> PGXC start all >> >> Start GTM master >> >> gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not >> exist >> >> Is server running? >> >> server starting >> >> Start GTM slavegtm_ctl: PID file >> "/home/postgres/pgxc/nodes/gtm/gtm.pid" >> >> does not exist >> >> Is server running? >> >> server starting >> >> Done. >> >> Starting all the gtm proxies. >> >> Starting gtm proxy gtm_pxy1. >> >> Starting gtm proxy gtm_pxy2. >> >> gtm_proxy: no process found >> >> server starting >> >> gtm_proxy: no process found >> >> server starting >> >> Done. >> >> Starting coordinator master. >> >> Starting coordinator master coord1 >> >> Starting coordinator master coord2 >> >> DEBUG: 00000: postgres: PostmasterMain: initial environment dump: >> >> LOCATION: PostmasterMain, postmaster.c:962 >> >> DEBUG: 00000: ----------------------------------------- >> >> LOCATION: PostmasterMain, postmaster.c:964 >> >> DEBUG: 00000: MAIL=/var/mail/postgres >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: SSH_CLIENT=41.218.72.115 48992 59696 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: USER=postgres >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LANGUAGE=en_ZA:en >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: SHLVL=1 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: HOME=/home/postgres >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: PG_GRANDPARENT_PID=19891 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LOGNAME=postgres >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: _=/usr/local/bin/pg_ctl >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: >> PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LANG=en_ZA.UTF-8 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: SHELL=/bin/bash >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: PWD=/home/postgres >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_MONETARY=C >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_NUMERIC=C >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_TIME=C >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: ----------------------------------------- >> >> LOCATION: PostmasterMain, postmaster.c:969 >> >> DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) >> >> LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 >> >> DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 >> >> LOCATION: SlruScanDirectory, slru.c:1312 >> >> DEBUG: 00000: removing file "pg_notify/0000" >> >> LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 >> >> DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 >> >> LOCATION: set_max_safe_fds, fd.c:548 >> >> LOG: 00000: redirecting log output to logging collector process >> >> HINT: Future log output will appear in directory "pg_log". >> >> LOCATION: SysLogger_Start, syslogger.c:649 >> >> DEBUG: 00000: postgres: PostmasterMain: initial environment dump: >> >> LOCATION: PostmasterMain, postmaster.c:962 >> >> DEBUG: 00000: ----------------------------------------- >> >> LOCATION: PostmasterMain, postmaster.c:964 >> >> DEBUG: 00000: MAIL=/var/mail/postgres >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: SSH_CLIENT=41.218.72.115 33169 59696 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: USER=postgres >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LANGUAGE=en_ZA:en >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: SHLVL=1 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: HOME=/home/postgres >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: PG_GRANDPARENT_PID=19507 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LOGNAME=postgres >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: _=/usr/local/bin/pg_ctl >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: >> PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LANG=en_ZA.UTF-8 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: SHELL=/bin/bash >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: PWD=/home/postgres >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_MONETARY=C >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_NUMERIC=C >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: LC_TIME=C >> >> LOCATION: PostmasterMain, postmaster.c:967 >> >> DEBUG: 00000: ----------------------------------------- >> >> LOCATION: PostmasterMain, postmaster.c:969 >> >> DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) >> >> LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 >> >> DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 >> >> LOCATION: SlruScanDirectory, slru.c:1312 >> >> DEBUG: 00000: removing file "pg_notify/0000" >> >> LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 >> >> DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 >> >> LOCATION: set_max_safe_fds, fd.c:548 >> >> LOG: 00000: redirecting log output to logging collector process >> >> HINT: Future log output will appear in directory "pg_log". >> >> LOCATION: SysLogger_Start, syslogger.c:649 >> >> DEBUG: 00000: logger shutting down >> >> LOCATION: SysLoggerMain, syslogger.c:517 >> >> DEBUG: 00000: shmem_exit(0): 0 callbacks to make >> >> LOCATION: shmem_exit, ipc.c:212 >> >> DEBUG: 00000: proc_exit(0): 0 callbacks to make >> >> LOCATION: proc_exit_prepare, ipc.c:184 >> >> DEBUG: 00000: exit(0) >> >> LOCATION: proc_exit, ipc.c:135 >> >> DEBUG: 00000: shmem_exit(-1): 0 callbacks to make >> >> LOCATION: shmem_exit, ipc.c:212 >> >> DEBUG: 00000: proc_exit(-1): 0 callbacks to make >> >> LOCATION: proc_exit_prepare, ipc.c:184 >> >> pg_ctl: could not start server >> >> Examine the log output. >> >> Done. >> >> Starting all the datanode masters. >> >> Starting datanode master datanode1. >> >> LOG: redirecting log output to logging collector process >> >> HINT: Future log output will appear in directory "pg_log". >> >> Done. >> >> Starting all the datanode slaves. >> >> Starting datanode slave datanode1. >> >> LOG: redirecting log output to logging collector process >> >> HINT: Future log output will appear in directory "pg_log". >> >> Done. >> >> PGXC monitor all >> >> Running: gtm master >> >> Running: gtm slave >> >> Running: gtm proxy gtm_pxy1 >> >> Running: gtm proxy gtm_pxy2 >> >> Running: coordinator master coord1 >> >> Not running: coordinator master coord2 >> >> Not running: datanode master datanode1 >> >> Running: datanode slave datanode1 >> >> >> >> >> >> My DB server has 32 GB RAM and 260 GB of Hard disk and i am using >> >> pgxc-1.2.1. what would be the optimal memory related postgresql.conf >> >> configuration for this server. >> >> >> >> Anyone has an idea about this issue ? >> >> >> >> -- >> >> Thanks, >> >> Juned Khan >> >> iNextrix Technologies Pvt Ltd. >> >> www.inextrix.com >> > >> > >> > >> > >> > -- >> > Thanks, >> > Juned Khan >> > iNextrix Technologies Pvt Ltd. >> > www.inextrix.com >> > >> > >> ------------------------------------------------------------------------------ >> > Learn Graph Databases - Download FREE O'Reilly Book >> > "Graph Databases" is the definitive new guide to graph databases and >> their >> > applications. Written by three acclaimed leaders in the field, >> > this first edition is now available. Download your free book today! >> > http://p.sf.net/sfu/NeoTech >> > _______________________________________________ >> > Postgres-xc-general mailing list >> > Pos...@li... >> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >> > >> > > > > -- > Thanks, > Juned Khan > iNextrix Technologies Pvt Ltd. > www.inextrix.com > -- Thanks, Juned Khan iNextrix Technologies Pvt Ltd. www.inextrix.com |
|
From: Masataka S. <pg...@gm...> - 2014-04-16 06:24:40
|
It is worth trying at a glance to the proxy log. AFAIK you're needless to remove register.node with typical operations. If you meet this issue, I think you'd better review your operations. Regards. On 15 April 2014 20:30, Juned Khan <jkh...@gm...> wrote: > Hi Masataka, > > You answer helped but again i am having this problem. do i need to remove > register.node file again ? > > GTM logs > > 1:140587359135488:2014-04-15 12:22:56.711 WAT -LOG: Saving transaction > restoration info, backed-up gxid: 259975 > LOCATION: GTM_WriteRestorePointXid, gtm_txn.c:2649 > 1:140587359135488:2014-04-15 12:22:56.712 WAT -LOG: Started to run as > GTM-Active. > LOCATION: main, main.c:641 > 1:140587359135488:2014-04-15 12:22:57.712 WAT -LOG: Any GTM standby node > not found in registered node(s). > LOCATION: gtm_standby_connect_to_standby_int, gtm_standby.c:381 > 1:140587350980352:2014-04-15 12:22:57.717 WAT -LOG: Failed to establish a > connection with GTM standby. - 0x13ac390 > LOCATION: gtm_standby_connect_to_standby_int, gtm_standby.c:396 > 1:140587321448192:2014-04-15 12:27:02.055 WAT -LOG: unexpected EOF on > client connection > LOCATION: ReadCommand, main.c:1374 > 1:140587359135488:2014-04-15 12:27:03.928 WAT -LOG: GTM shutting down. > LOCATION: ServerLoop, main.c:802 > 1:140587359135488:2014-04-15 12:27:03.928 WAT -LOG: Saving transaction info > - next_gxid: 257975 > LOCATION: GTM_SaveTxnInfo, gtm_txn.c:2632 > Received signal 15 > > > GTM Proxy logs > LOCATION: pgxcnode_add_info, register_common.c:249 > 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG: Node with the given ID > number already exists > LOCATION: pgxcnode_add_info, register_common.c:249 > 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG: Node with the given ID > number already exists > LOCATION: pgxcnode_add_info, register_common.c:249 > 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG: Node with the given ID > number already exists > LOCATION: pgxcnode_add_info, register_common.c:249 > 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG: Node with the given ID > number already exists > LOCATION: pgxcnode_add_info, register_common.c:249 > 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG: Node with the given ID > number already exists > LOCATION: pgxcnode_add_info, register_common.c:249 > > > > > On Tue, Apr 15, 2014 at 10:49 AM, Juned Khan <jkh...@gm...> wrote: >> >> Hi Masataka, >> >> Thanks for help >> >> >> On Tue, Apr 15, 2014 at 7:07 AM, Masataka Saito <pg...@gm...> wrote: >>> >>> I'm not sure but GTM or GTM proxy seems to be wrong. >>> >>> Please check their log. >>> If you find an issue, search postgres-xc-general ml for keyword >>> "register.node". >>> >>> Regards. >>> >>> On 14 April 2014 20:11, Juned Khan <jkh...@gm...> wrote: >>> > Here few more logs of the component which doesn't start >>> > >>> > root@db02:~# tail -f >>> > >>> > /home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log >>> > LOG: database system was interrupted; last known up at 2014-04-14 >>> > 11:00:55 >>> > WAT >>> > LOG: database system was not properly shut down; automatic recovery in >>> > progress >>> > LOG: record with zero length at 4/9A01A588 >>> > LOG: redo is not required >>> > FATAL: the database system is starting up >>> > LOG: autovacuum launcher started >>> > LOG: database system is ready to accept connections >>> > WARNING: worker took too long to start; canceled >>> > FATAL: Can not register Datanode on GTM >>> > >>> > >>> > root@db02:~# tail -f >>> > /home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log >>> > DEBUG: 00000: name: unnamed; blockState: DEFAULT; state: >>> > INPROGR, >>> > xid/subid/cid: 0/1/0, nestlvl: 1, children: >>> > LOCATION: ShowTransactionStateRec, xact.c:5238 >>> > DEBUG: 00000: Autovacuum launcher: connection to GTM closed >>> > LOCATION: CloseGTM, gtm.c:116 >>> > DEBUG: 00000: Autovacuum launcher: connection established to GTM with >>> > string host=db02 port=20002 node_name=coord1 >>> > LOCATION: InitGTM, gtm.c:84 >>> > WARNING: 01000: Xid is invalid. >>> > LOCATION: GetNewTransactionId, varsup.c:160 >>> > DEBUG: 00000: Getting snapshot. Current XID = 0 >>> > LOCATION: GetSnapshotDataCoordinator, procarray.c:3054 >>> > DEBUG: 00000: Autovacuum launcher: connection to GTM closed >>> > LOCATION: CloseGTM, gtm.c:116 >>> > DEBUG: 00000: Autovacuum launcher: connection established to GTM with >>> > string host=db02 port=20002 node_name=coord1 >>> > LOCATION: InitGTM, gtm.c:84 >>> > >>> > >>> > >>> > >>> > >>> > On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...> >>> > wrote: >>> >> >>> >> Hi all, >>> >> >>> >> Yesterday due some problem datanode slave stopped working on one of my >>> >> DB >>> >> server. After figuring out the issue i freed out the space on that >>> >> server. >>> >> now almost 50% disk is free on server. >>> >> >>> >> the problem is after that incident i am not able start all my pgxc >>> >> components, now its uncertain some of components starts and sometimes >>> >> it >>> >> does not each time when i execute "stop all" and "start all" scenario >>> >> is >>> >> different.I am not able to figure out the problem which causing this >>> >> issue. >>> >> >>> >> I have enabled the debug log though and tried to identify the issue >>> >> and >>> >> after reviewing the logs it seems memory related issue, i am not sure >>> >> about >>> >> this. >>> >> >>> >> PGXC start all >>> >> Start GTM master >>> >> gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not >>> >> exist >>> >> Is server running? >>> >> server starting >>> >> Start GTM slavegtm_ctl: PID file >>> >> "/home/postgres/pgxc/nodes/gtm/gtm.pid" >>> >> does not exist >>> >> Is server running? >>> >> server starting >>> >> Done. >>> >> Starting all the gtm proxies. >>> >> Starting gtm proxy gtm_pxy1. >>> >> Starting gtm proxy gtm_pxy2. >>> >> gtm_proxy: no process found >>> >> server starting >>> >> gtm_proxy: no process found >>> >> server starting >>> >> Done. >>> >> Starting coordinator master. >>> >> Starting coordinator master coord1 >>> >> Starting coordinator master coord2 >>> >> DEBUG: 00000: postgres: PostmasterMain: initial environment dump: >>> >> LOCATION: PostmasterMain, postmaster.c:962 >>> >> DEBUG: 00000: ----------------------------------------- >>> >> LOCATION: PostmasterMain, postmaster.c:964 >>> >> DEBUG: 00000: MAIL=/var/mail/postgres >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: SSH_CLIENT=41.218.72.115 48992 59696 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: USER=postgres >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LANGUAGE=en_ZA:en >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: SHLVL=1 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: HOME=/home/postgres >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: PG_GRANDPARENT_PID=19891 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LOGNAME=postgres >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: _=/usr/local/bin/pg_ctl >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: >>> >> PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LANG=en_ZA.UTF-8 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: SHELL=/bin/bash >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: PWD=/home/postgres >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_MONETARY=C >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_NUMERIC=C >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_TIME=C >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: ----------------------------------------- >>> >> LOCATION: PostmasterMain, postmaster.c:969 >>> >> DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) >>> >> LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 >>> >> DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 >>> >> LOCATION: SlruScanDirectory, slru.c:1312 >>> >> DEBUG: 00000: removing file "pg_notify/0000" >>> >> LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 >>> >> DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 >>> >> LOCATION: set_max_safe_fds, fd.c:548 >>> >> LOG: 00000: redirecting log output to logging collector process >>> >> HINT: Future log output will appear in directory "pg_log". >>> >> LOCATION: SysLogger_Start, syslogger.c:649 >>> >> DEBUG: 00000: postgres: PostmasterMain: initial environment dump: >>> >> LOCATION: PostmasterMain, postmaster.c:962 >>> >> DEBUG: 00000: ----------------------------------------- >>> >> LOCATION: PostmasterMain, postmaster.c:964 >>> >> DEBUG: 00000: MAIL=/var/mail/postgres >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: SSH_CLIENT=41.218.72.115 33169 59696 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: USER=postgres >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LANGUAGE=en_ZA:en >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: SHLVL=1 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: HOME=/home/postgres >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: PG_GRANDPARENT_PID=19507 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LOGNAME=postgres >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: _=/usr/local/bin/pg_ctl >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: >>> >> PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LANG=en_ZA.UTF-8 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: SHELL=/bin/bash >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: PWD=/home/postgres >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_MONETARY=C >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_NUMERIC=C >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: LC_TIME=C >>> >> LOCATION: PostmasterMain, postmaster.c:967 >>> >> DEBUG: 00000: ----------------------------------------- >>> >> LOCATION: PostmasterMain, postmaster.c:969 >>> >> DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) >>> >> LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 >>> >> DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 >>> >> LOCATION: SlruScanDirectory, slru.c:1312 >>> >> DEBUG: 00000: removing file "pg_notify/0000" >>> >> LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 >>> >> DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 >>> >> LOCATION: set_max_safe_fds, fd.c:548 >>> >> LOG: 00000: redirecting log output to logging collector process >>> >> HINT: Future log output will appear in directory "pg_log". >>> >> LOCATION: SysLogger_Start, syslogger.c:649 >>> >> DEBUG: 00000: logger shutting down >>> >> LOCATION: SysLoggerMain, syslogger.c:517 >>> >> DEBUG: 00000: shmem_exit(0): 0 callbacks to make >>> >> LOCATION: shmem_exit, ipc.c:212 >>> >> DEBUG: 00000: proc_exit(0): 0 callbacks to make >>> >> LOCATION: proc_exit_prepare, ipc.c:184 >>> >> DEBUG: 00000: exit(0) >>> >> LOCATION: proc_exit, ipc.c:135 >>> >> DEBUG: 00000: shmem_exit(-1): 0 callbacks to make >>> >> LOCATION: shmem_exit, ipc.c:212 >>> >> DEBUG: 00000: proc_exit(-1): 0 callbacks to make >>> >> LOCATION: proc_exit_prepare, ipc.c:184 >>> >> pg_ctl: could not start server >>> >> Examine the log output. >>> >> Done. >>> >> Starting all the datanode masters. >>> >> Starting datanode master datanode1. >>> >> LOG: redirecting log output to logging collector process >>> >> HINT: Future log output will appear in directory "pg_log". >>> >> Done. >>> >> Starting all the datanode slaves. >>> >> Starting datanode slave datanode1. >>> >> LOG: redirecting log output to logging collector process >>> >> HINT: Future log output will appear in directory "pg_log". >>> >> Done. >>> >> PGXC monitor all >>> >> Running: gtm master >>> >> Running: gtm slave >>> >> Running: gtm proxy gtm_pxy1 >>> >> Running: gtm proxy gtm_pxy2 >>> >> Running: coordinator master coord1 >>> >> Not running: coordinator master coord2 >>> >> Not running: datanode master datanode1 >>> >> Running: datanode slave datanode1 >>> >> >>> >> >>> >> My DB server has 32 GB RAM and 260 GB of Hard disk and i am using >>> >> pgxc-1.2.1. what would be the optimal memory related postgresql.conf >>> >> configuration for this server. >>> >> >>> >> Anyone has an idea about this issue ? >>> >> >>> >> -- >>> >> Thanks, >>> >> Juned Khan >>> >> iNextrix Technologies Pvt Ltd. >>> >> www.inextrix.com >>> > >>> > >>> > >>> > >>> > -- >>> > Thanks, >>> > Juned Khan >>> > iNextrix Technologies Pvt Ltd. >>> > www.inextrix.com >>> > >>> > >>> > ------------------------------------------------------------------------------ >>> > Learn Graph Databases - Download FREE O'Reilly Book >>> > "Graph Databases" is the definitive new guide to graph databases and >>> > their >>> > applications. Written by three acclaimed leaders in the field, >>> > this first edition is now available. Download your free book today! >>> > http://p.sf.net/sfu/NeoTech >>> > _______________________________________________ >>> > Postgres-xc-general mailing list >>> > Pos...@li... >>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general >>> > >> >> >> >> >> -- >> Thanks, >> Juned Khan >> iNextrix Technologies Pvt Ltd. >> www.inextrix.com > > > > > -- > Thanks, > Juned Khan > iNextrix Technologies Pvt Ltd. > www.inextrix.com |
Thanks for helping keep SourceForge clean.
X