Re: [Postgres-xc-general] Datanode is not starting up after "No space left on device"

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

I'm not sure but GTM or GTM proxy seems to be wrong.

Please check their log.
If you find an issue, search postgres-xc-general ml for keyword "register.node".

Regards.

On 14 April 2014 20:11, Juned Khan <jkh...@gm...> wrote:
> Here few more logs of the component which doesn't start
>
> root@db02:~# tail -f
> /home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log
> LOG:  database system was interrupted; last known up at 2014-04-14 11:00:55
> WAT
> LOG:  database system was not properly shut down; automatic recovery in
> progress
> LOG:  record with zero length at 4/9A01A588
> LOG:  redo is not required
> FATAL:  the database system is starting up
> LOG:  autovacuum launcher started
> LOG:  database system is ready to accept connections
> WARNING:  worker took too long to start; canceled
> FATAL:  Can not register Datanode on GTM
>
>
> root@db02:~# tail -f
> /home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log
> DEBUG:  00000: name: unnamed; blockState:       DEFAULT; state: INPROGR,
> xid/subid/cid: 0/1/0, nestlvl: 1, children:
> LOCATION:  ShowTransactionStateRec, xact.c:5238
> DEBUG:  00000: Autovacuum launcher: connection to GTM closed
> LOCATION:  CloseGTM, gtm.c:116
> DEBUG:  00000: Autovacuum launcher: connection established to GTM with
> string host=db02 port=20002 node_name=coord1
> LOCATION:  InitGTM, gtm.c:84
> WARNING:  01000: Xid is invalid.
> LOCATION:  GetNewTransactionId, varsup.c:160
> DEBUG:  00000: Getting snapshot. Current XID = 0
> LOCATION:  GetSnapshotDataCoordinator, procarray.c:3054
> DEBUG:  00000: Autovacuum launcher: connection to GTM closed
> LOCATION:  CloseGTM, gtm.c:116
> DEBUG:  00000: Autovacuum launcher: connection established to GTM with
> string host=db02 port=20002 node_name=coord1
> LOCATION:  InitGTM, gtm.c:84
>
>
>
>
>
> On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...> wrote:
>>
>> Hi all,
>>
>> Yesterday due some problem datanode slave stopped working on one of my DB
>> server. After figuring out the issue i freed out the space on that server.
>> now almost 50% disk is free on server.
>>
>> the problem is after that incident i am not able start all my pgxc
>> components, now its uncertain some of components starts and sometimes it
>> does not each time when i execute "stop all" and "start all" scenario is
>> different.I am not able to figure out the problem which causing this issue.
>>
>> I have enabled the debug log though and tried to identify the issue and
>> after reviewing the logs it seems memory related issue, i am not sure about
>> this.
>>
>> PGXC start all
>> Start GTM master
>> gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not exist
>> Is server running?
>> server starting
>> Start GTM slavegtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid"
>> does not exist
>> Is server running?
>> server starting
>> Done.
>> Starting all the gtm proxies.
>> Starting gtm proxy gtm_pxy1.
>> Starting gtm proxy gtm_pxy2.
>> gtm_proxy: no process found
>> server starting
>> gtm_proxy: no process found
>> server starting
>> Done.
>> Starting coordinator master.
>> Starting coordinator master coord1
>> Starting coordinator master coord2
>> DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
>> LOCATION:  PostmasterMain, postmaster.c:962
>> DEBUG:  00000: -----------------------------------------
>> LOCATION:  PostmasterMain, postmaster.c:964
>> DEBUG:  00000:  MAIL=/var/mail/postgres
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  SSH_CLIENT=41.218.72.115 48992 59696
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  USER=postgres
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LANGUAGE=en_ZA:en
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  SHLVL=1
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  HOME=/home/postgres
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  PG_GRANDPARENT_PID=19891
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LOGNAME=postgres
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  _=/usr/local/bin/pg_ctl
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LANG=en_ZA.UTF-8
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  SHELL=/bin/bash
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  PWD=/home/postgres
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_MONETARY=C
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_NUMERIC=C
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_TIME=C
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000: -----------------------------------------
>> LOCATION:  PostmasterMain, postmaster.c:969
>> DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
>> LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
>> DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
>> LOCATION:  SlruScanDirectory, slru.c:1312
>> DEBUG:  00000: removing file "pg_notify/0000"
>> LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
>> DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
>> LOCATION:  set_max_safe_fds, fd.c:548
>> LOG:  00000: redirecting log output to logging collector process
>> HINT:  Future log output will appear in directory "pg_log".
>> LOCATION:  SysLogger_Start, syslogger.c:649
>> DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
>> LOCATION:  PostmasterMain, postmaster.c:962
>> DEBUG:  00000: -----------------------------------------
>> LOCATION:  PostmasterMain, postmaster.c:964
>> DEBUG:  00000:  MAIL=/var/mail/postgres
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  SSH_CLIENT=41.218.72.115 33169 59696
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  USER=postgres
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LANGUAGE=en_ZA:en
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  SHLVL=1
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  HOME=/home/postgres
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  PG_GRANDPARENT_PID=19507
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LOGNAME=postgres
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  _=/usr/local/bin/pg_ctl
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LANG=en_ZA.UTF-8
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  SHELL=/bin/bash
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  PWD=/home/postgres
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_MONETARY=C
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_NUMERIC=C
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000:  LC_TIME=C
>> LOCATION:  PostmasterMain, postmaster.c:967
>> DEBUG:  00000: -----------------------------------------
>> LOCATION:  PostmasterMain, postmaster.c:969
>> DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
>> LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
>> DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
>> LOCATION:  SlruScanDirectory, slru.c:1312
>> DEBUG:  00000: removing file "pg_notify/0000"
>> LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
>> DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
>> LOCATION:  set_max_safe_fds, fd.c:548
>> LOG:  00000: redirecting log output to logging collector process
>> HINT:  Future log output will appear in directory "pg_log".
>> LOCATION:  SysLogger_Start, syslogger.c:649
>> DEBUG:  00000: logger shutting down
>> LOCATION:  SysLoggerMain, syslogger.c:517
>> DEBUG:  00000: shmem_exit(0): 0 callbacks to make
>> LOCATION:  shmem_exit, ipc.c:212
>> DEBUG:  00000: proc_exit(0): 0 callbacks to make
>> LOCATION:  proc_exit_prepare, ipc.c:184
>> DEBUG:  00000: exit(0)
>> LOCATION:  proc_exit, ipc.c:135
>> DEBUG:  00000: shmem_exit(-1): 0 callbacks to make
>> LOCATION:  shmem_exit, ipc.c:212
>> DEBUG:  00000: proc_exit(-1): 0 callbacks to make
>> LOCATION:  proc_exit_prepare, ipc.c:184
>> pg_ctl: could not start server
>> Examine the log output.
>> Done.
>> Starting all the datanode masters.
>> Starting datanode master datanode1.
>> LOG:  redirecting log output to logging collector process
>> HINT:  Future log output will appear in directory "pg_log".
>> Done.
>> Starting all the datanode slaves.
>> Starting datanode slave datanode1.
>> LOG:  redirecting log output to logging collector process
>> HINT:  Future log output will appear in directory "pg_log".
>> Done.
>> PGXC monitor all
>> Running: gtm master
>> Running: gtm slave
>> Running: gtm proxy gtm_pxy1
>> Running: gtm proxy gtm_pxy2
>> Running: coordinator master coord1
>> Not running: coordinator master coord2
>> Not running: datanode master datanode1
>> Running: datanode slave datanode1
>>
>>
>> My DB server has 32 GB RAM and 260 GB of Hard disk and i am using
>> pgxc-1.2.1. what would be the optimal memory related postgresql.conf
>> configuration for this server.
>>
>> Anyone has an idea about this issue ?
>>
>> --
>> Thanks,
>> Juned Khan
>> iNextrix Technologies Pvt Ltd.
>> www.inextrix.com
>
>
>
>
> --
> Thanks,
> Juned Khan
> iNextrix Technologies Pvt Ltd.
> www.inextrix.com
>
> ------------------------------------------------------------------------------
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases and their
> applications. Written by three acclaimed leaders in the field,
> this first edition is now available. Download your free book today!
> http://p.sf.net/sfu/NeoTech
> _______________________________________________
> Postgres-xc-general mailing list
> Pos...@li...
> https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>