Menu
▾
▴
Re: [Postgres-xc-general] Datanode is not starting up after "No space left on device"
|
From: Juned K. <jkh...@gm...> - 2014-04-14 11:12:02
|
Here few more logs of the component which doesn't start root@db02:~# tail -f /home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log LOG: database system was interrupted; last known up at 2014-04-14 11:00:55 WAT LOG: database system was not properly shut down; automatic recovery in progress LOG: record with zero length at 4/9A01A588 LOG: redo is not required FATAL: the database system is starting up LOG: autovacuum launcher started LOG: database system is ready to accept connections WARNING: worker took too long to start; canceled FATAL: Can not register Datanode on GTM root@db02:~# tail -f /home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log DEBUG: 00000: name: unnamed; blockState: DEFAULT; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children: LOCATION: ShowTransactionStateRec, xact.c:5238 DEBUG: 00000: Autovacuum launcher: connection to GTM closed LOCATION: CloseGTM, gtm.c:116 DEBUG: 00000: Autovacuum launcher: connection established to GTM with string host=db02 port=20002 node_name=coord1 LOCATION: InitGTM, gtm.c:84 WARNING: 01000: Xid is invalid. LOCATION: GetNewTransactionId, varsup.c:160 DEBUG: 00000: Getting snapshot. Current XID = 0 LOCATION: GetSnapshotDataCoordinator, procarray.c:3054 DEBUG: 00000: Autovacuum launcher: connection to GTM closed LOCATION: CloseGTM, gtm.c:116 DEBUG: 00000: Autovacuum launcher: connection established to GTM with string host=db02 port=20002 node_name=coord1 LOCATION: InitGTM, gtm.c:84 On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...> wrote: > Hi all, > > Yesterday due some problem datanode slave stopped working on one of my DB > server. After figuring out the issue i freed out the space on that server. > now almost 50% disk is free on server. > > the problem is after that incident i am not able start all my pgxc > components, now its uncertain some of components starts and sometimes it > does not each time when i execute "stop all" and "start all" scenario is > different.I am not able to figure out the problem which causing this issue. > > I have enabled the debug log though and tried to identify the issue and > after reviewing the logs it seems memory related issue, i am not sure about > this. > > PGXC start all > Start GTM master > gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist > Is server running? > server starting > Start GTM slavegtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" > does not exist > Is server running? > server starting > Done. > Starting all the gtm proxies. > Starting gtm proxy gtm_pxy1. > Starting gtm proxy gtm_pxy2. > gtm_proxy: no process found > server starting > gtm_proxy: no process found > server starting > Done. > Starting coordinator master. > Starting coordinator master coord1 > Starting coordinator master coord2 > DEBUG: 00000: postgres: PostmasterMain: initial environment dump: > LOCATION: PostmasterMain, postmaster.c:962 > DEBUG: 00000: ----------------------------------------- > LOCATION: PostmasterMain, postmaster.c:964 > DEBUG: 00000: MAIL=/var/mail/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SSH_CLIENT=41.218.72.115 48992 59696 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: USER=postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LANGUAGE=en_ZA:en > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SHLVL=1 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: HOME=/home/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PG_GRANDPARENT_PID=19891 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LOGNAME=postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: _=/usr/local/bin/pg_ctl > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LANG=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SHELL=/bin/bash > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PWD=/home/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_MONETARY=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_NUMERIC=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_TIME=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: ----------------------------------------- > LOCATION: PostmasterMain, postmaster.c:969 > DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) > LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 > DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 > LOCATION: SlruScanDirectory, slru.c:1312 > DEBUG: 00000: removing file "pg_notify/0000" > LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 > DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 > LOCATION: set_max_safe_fds, fd.c:548 > LOG: 00000: redirecting log output to logging collector process > HINT: Future log output will appear in directory "pg_log". > LOCATION: SysLogger_Start, syslogger.c:649 > DEBUG: 00000: postgres: PostmasterMain: initial environment dump: > LOCATION: PostmasterMain, postmaster.c:962 > DEBUG: 00000: ----------------------------------------- > LOCATION: PostmasterMain, postmaster.c:964 > DEBUG: 00000: MAIL=/var/mail/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SSH_CLIENT=41.218.72.115 33169 59696 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: USER=postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LANGUAGE=en_ZA:en > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SHLVL=1 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: HOME=/home/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PG_GRANDPARENT_PID=19507 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LOGNAME=postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: _=/usr/local/bin/pg_ctl > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PGSYSCONFDIR=/usr/local/pgsql/etc > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LANG=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SHELL=/bin/bash > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PWD=/home/postgres > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: PGDATA=/home/postgres/pgxc/nodes/coord > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_COLLATE=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_CTYPE=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_MESSAGES=en_ZA.UTF-8 > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_MONETARY=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_NUMERIC=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: LC_TIME=C > LOCATION: PostmasterMain, postmaster.c:967 > DEBUG: 00000: ----------------------------------------- > LOCATION: PostmasterMain, postmaster.c:969 > DEBUG: 00000: invoking IpcMemoryCreate(size=148193280) > LOCATION: CreateSharedMemoryAndSemaphores, ipci.c:149 > DEBUG: 00000: SlruScanDirectory invoking callback on pg_notify/0000 > LOCATION: SlruScanDirectory, slru.c:1312 > DEBUG: 00000: removing file "pg_notify/0000" > LOCATION: SlruScanDirCbDeleteAll, slru.c:1277 > DEBUG: 00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6 > LOCATION: set_max_safe_fds, fd.c:548 > LOG: 00000: redirecting log output to logging collector process > HINT: Future log output will appear in directory "pg_log". > LOCATION: SysLogger_Start, syslogger.c:649 > DEBUG: 00000: logger shutting down > LOCATION: SysLoggerMain, syslogger.c:517 > DEBUG: 00000: shmem_exit(0): 0 callbacks to make > LOCATION: shmem_exit, ipc.c:212 > DEBUG: 00000: proc_exit(0): 0 callbacks to make > LOCATION: proc_exit_prepare, ipc.c:184 > DEBUG: 00000: exit(0) > LOCATION: proc_exit, ipc.c:135 > DEBUG: 00000: shmem_exit(-1): 0 callbacks to make > LOCATION: shmem_exit, ipc.c:212 > DEBUG: 00000: proc_exit(-1): 0 callbacks to make > LOCATION: proc_exit_prepare, ipc.c:184 > pg_ctl: could not start server > Examine the log output. > Done. > Starting all the datanode masters. > Starting datanode master datanode1. > LOG: redirecting log output to logging collector process > HINT: Future log output will appear in directory "pg_log". > Done. > Starting all the datanode slaves. > Starting datanode slave datanode1. > LOG: redirecting log output to logging collector process > HINT: Future log output will appear in directory "pg_log". > Done. > PGXC monitor all > Running: gtm master > Running: gtm slave > Running: gtm proxy gtm_pxy1 > Running: gtm proxy gtm_pxy2 > Running: coordinator master coord1 > Not running: coordinator master coord2 > Not running: datanode master datanode1 > Running: datanode slave datanode1 > > > My DB server has 32 GB RAM and 260 GB of Hard disk and i am using > pgxc-1.2.1. what would be the optimal memory related postgresql.conf > configuration for this server. > > Anyone has an idea about this issue ? > > -- > Thanks, > Juned Khan > iNextrix Technologies Pvt Ltd. > www.inextrix.com > -- Thanks, Juned Khan iNextrix Technologies Pvt Ltd. www.inextrix.com |
