Re: [Postgres-xc-general] Datanode is not starting up after "No space left on device"

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi Masataka,

Thanks for help

On Tue, Apr 15, 2014 at 7:07 AM, Masataka Saito <pg...@gm...> wrote:

> I'm not sure but GTM or GTM proxy seems to be wrong.
>
> Please check their log.
> If you find an issue, search postgres-xc-general ml for keyword
> "register.node".
>
> Regards.
>
> On 14 April 2014 20:11, Juned Khan <jkh...@gm...> wrote:
> > Here few more logs of the component which doesn't start
> >
> > root@db02:~# tail -f
> >
> /home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log
> > LOG:  database system was interrupted; last known up at 2014-04-14
> 11:00:55
> > WAT
> > LOG:  database system was not properly shut down; automatic recovery in
> > progress
> > LOG:  record with zero length at 4/9A01A588
> > LOG:  redo is not required
> > FATAL:  the database system is starting up
> > LOG:  autovacuum launcher started
> > LOG:  database system is ready to accept connections
> > WARNING:  worker took too long to start; canceled
> > FATAL:  Can not register Datanode on GTM
> >
> >
> > root@db02:~# tail -f
> > /home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log
> > DEBUG:  00000: name: unnamed; blockState:       DEFAULT; state: INPROGR,
> > xid/subid/cid: 0/1/0, nestlvl: 1, children:
> > LOCATION:  ShowTransactionStateRec, xact.c:5238
> > DEBUG:  00000: Autovacuum launcher: connection to GTM closed
> > LOCATION:  CloseGTM, gtm.c:116
> > DEBUG:  00000: Autovacuum launcher: connection established to GTM with
> > string host=db02 port=20002 node_name=coord1
> > LOCATION:  InitGTM, gtm.c:84
> > WARNING:  01000: Xid is invalid.
> > LOCATION:  GetNewTransactionId, varsup.c:160
> > DEBUG:  00000: Getting snapshot. Current XID = 0
> > LOCATION:  GetSnapshotDataCoordinator, procarray.c:3054
> > DEBUG:  00000: Autovacuum launcher: connection to GTM closed
> > LOCATION:  CloseGTM, gtm.c:116
> > DEBUG:  00000: Autovacuum launcher: connection established to GTM with
> > string host=db02 port=20002 node_name=coord1
> > LOCATION:  InitGTM, gtm.c:84
> >
> >
> >
> >
> >
> > On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...> wrote:
> >>
> >> Hi all,
> >>
> >> Yesterday due some problem datanode slave stopped working on one of my
> DB
> >> server. After figuring out the issue i freed out the space on that
> server.
> >> now almost 50% disk is free on server.
> >>
> >> the problem is after that incident i am not able start all my pgxc
> >> components, now its uncertain some of components starts and sometimes it
> >> does not each time when i execute "stop all" and "start all" scenario is
> >> different.I am not able to figure out the problem which causing this
> issue.
> >>
> >> I have enabled the debug log though and tried to identify the issue and
> >> after reviewing the logs it seems memory related issue, i am not sure
> about
> >> this.
> >>
> >> PGXC start all
> >> Start GTM master
> >> gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist
> >> Is server running?
> >> server starting
> >> Start GTM slavegtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid"
> >> does not exist
> >> Is server running?
> >> server starting
> >> Done.
> >> Starting all the gtm proxies.
> >> Starting gtm proxy gtm_pxy1.
> >> Starting gtm proxy gtm_pxy2.
> >> gtm_proxy: no process found
> >> server starting
> >> gtm_proxy: no process found
> >> server starting
> >> Done.
> >> Starting coordinator master.
> >> Starting coordinator master coord1
> >> Starting coordinator master coord2
> >> DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
> >> LOCATION:  PostmasterMain, postmaster.c:962
> >> DEBUG:  00000: -----------------------------------------
> >> LOCATION:  PostmasterMain, postmaster.c:964
> >> DEBUG:  00000:  MAIL=/var/mail/postgres
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  SSH_CLIENT=41.218.72.115 48992 59696
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  USER=postgres
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LANGUAGE=en_ZA:en
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  SHLVL=1
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  HOME=/home/postgres
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  PG_GRANDPARENT_PID=19891
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LOGNAME=postgres
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  _=/usr/local/bin/pg_ctl
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:
>  PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LANG=en_ZA.UTF-8
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  SHELL=/bin/bash
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  PWD=/home/postgres
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_MONETARY=C
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_NUMERIC=C
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_TIME=C
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000: -----------------------------------------
> >> LOCATION:  PostmasterMain, postmaster.c:969
> >> DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
> >> LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
> >> DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
> >> LOCATION:  SlruScanDirectory, slru.c:1312
> >> DEBUG:  00000: removing file "pg_notify/0000"
> >> LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
> >> DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
> >> LOCATION:  set_max_safe_fds, fd.c:548
> >> LOG:  00000: redirecting log output to logging collector process
> >> HINT:  Future log output will appear in directory "pg_log".
> >> LOCATION:  SysLogger_Start, syslogger.c:649
> >> DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
> >> LOCATION:  PostmasterMain, postmaster.c:962
> >> DEBUG:  00000: -----------------------------------------
> >> LOCATION:  PostmasterMain, postmaster.c:964
> >> DEBUG:  00000:  MAIL=/var/mail/postgres
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  SSH_CLIENT=41.218.72.115 33169 59696
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  USER=postgres
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LANGUAGE=en_ZA:en
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  SHLVL=1
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  HOME=/home/postgres
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  PG_GRANDPARENT_PID=19507
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LOGNAME=postgres
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  _=/usr/local/bin/pg_ctl
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:
>  PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LANG=en_ZA.UTF-8
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  SHELL=/bin/bash
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  PWD=/home/postgres
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_MONETARY=C
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_NUMERIC=C
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000:  LC_TIME=C
> >> LOCATION:  PostmasterMain, postmaster.c:967
> >> DEBUG:  00000: -----------------------------------------
> >> LOCATION:  PostmasterMain, postmaster.c:969
> >> DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
> >> LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
> >> DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
> >> LOCATION:  SlruScanDirectory, slru.c:1312
> >> DEBUG:  00000: removing file "pg_notify/0000"
> >> LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
> >> DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
> >> LOCATION:  set_max_safe_fds, fd.c:548
> >> LOG:  00000: redirecting log output to logging collector process
> >> HINT:  Future log output will appear in directory "pg_log".
> >> LOCATION:  SysLogger_Start, syslogger.c:649
> >> DEBUG:  00000: logger shutting down
> >> LOCATION:  SysLoggerMain, syslogger.c:517
> >> DEBUG:  00000: shmem_exit(0): 0 callbacks to make
> >> LOCATION:  shmem_exit, ipc.c:212
> >> DEBUG:  00000: proc_exit(0): 0 callbacks to make
> >> LOCATION:  proc_exit_prepare, ipc.c:184
> >> DEBUG:  00000: exit(0)
> >> LOCATION:  proc_exit, ipc.c:135
> >> DEBUG:  00000: shmem_exit(-1): 0 callbacks to make
> >> LOCATION:  shmem_exit, ipc.c:212
> >> DEBUG:  00000: proc_exit(-1): 0 callbacks to make
> >> LOCATION:  proc_exit_prepare, ipc.c:184
> >> pg_ctl: could not start server
> >> Examine the log output.
> >> Done.
> >> Starting all the datanode masters.
> >> Starting datanode master datanode1.
> >> LOG:  redirecting log output to logging collector process
> >> HINT:  Future log output will appear in directory "pg_log".
> >> Done.
> >> Starting all the datanode slaves.
> >> Starting datanode slave datanode1.
> >> LOG:  redirecting log output to logging collector process
> >> HINT:  Future log output will appear in directory "pg_log".
> >> Done.
> >> PGXC monitor all
> >> Running: gtm master
> >> Running: gtm slave
> >> Running: gtm proxy gtm_pxy1
> >> Running: gtm proxy gtm_pxy2
> >> Running: coordinator master coord1
> >> Not running: coordinator master coord2
> >> Not running: datanode master datanode1
> >> Running: datanode slave datanode1
> >>
> >>
> >> My DB server has 32 GB RAM and 260 GB of Hard disk and i am using
> >> pgxc-1.2.1. what would be the optimal memory related postgresql.conf
> >> configuration for this server.
> >>
> >> Anyone has an idea about this issue ?
> >>
> >> --
> >> Thanks,
> >> Juned Khan
> >> iNextrix Technologies Pvt Ltd.
> >> www.inextrix.com
> >
> >
> >
> >
> > --
> > Thanks,
> > Juned Khan
> > iNextrix Technologies Pvt Ltd.
> > www.inextrix.com
> >
> >
> ------------------------------------------------------------------------------
> > Learn Graph Databases - Download FREE O'Reilly Book
> > "Graph Databases" is the definitive new guide to graph databases and
> their
> > applications. Written by three acclaimed leaders in the field,
> > this first edition is now available. Download your free book today!
> > http://p.sf.net/sfu/NeoTech
> > _______________________________________________
> > Postgres-xc-general mailing list
> > Pos...@li...
> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
> >
>

-- 
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com