Re: [Postgres-xc-general] Datanode is not starting up after "No space left on device"

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

It is worth trying at a glance to the proxy log.

AFAIK you're needless to remove register.node with typical operations.
If you meet this issue, I think you'd better review your operations.

Regards.

On 15 April 2014 20:30, Juned Khan <jkh...@gm...> wrote:
> Hi Masataka,
>
> You answer helped but again i am having this problem. do i need to remove
> register.node file again ?
>
> GTM logs
>
> 1:140587359135488:2014-04-15 12:22:56.711 WAT -LOG:  Saving transaction
> restoration info, backed-up gxid: 259975
> LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
> 1:140587359135488:2014-04-15 12:22:56.712 WAT -LOG:  Started to run as
> GTM-Active.
> LOCATION:  main, main.c:641
> 1:140587359135488:2014-04-15 12:22:57.712 WAT -LOG:  Any GTM standby node
> not found in registered node(s).
> LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:381
> 1:140587350980352:2014-04-15 12:22:57.717 WAT -LOG:  Failed to establish a
> connection with GTM standby. - 0x13ac390
> LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:396
> 1:140587321448192:2014-04-15 12:27:02.055 WAT -LOG:  unexpected EOF on
> client connection
> LOCATION:  ReadCommand, main.c:1374
> 1:140587359135488:2014-04-15 12:27:03.928 WAT -LOG:  GTM shutting down.
> LOCATION:  ServerLoop, main.c:802
> 1:140587359135488:2014-04-15 12:27:03.928 WAT -LOG:  Saving transaction info
> - next_gxid: 257975
> LOCATION:  GTM_SaveTxnInfo, gtm_txn.c:2632
> Received signal 15
>
>
> GTM Proxy logs
> LOCATION:  pgxcnode_add_info, register_common.c:249
> 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG:  Node with the given ID
> number already exists
> LOCATION:  pgxcnode_add_info, register_common.c:249
> 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG:  Node with the given ID
> number already exists
> LOCATION:  pgxcnode_add_info, register_common.c:249
> 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG:  Node with the given ID
> number already exists
> LOCATION:  pgxcnode_add_info, register_common.c:249
> 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG:  Node with the given ID
> number already exists
> LOCATION:  pgxcnode_add_info, register_common.c:249
> 1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG:  Node with the given ID
> number already exists
> LOCATION:  pgxcnode_add_info, register_common.c:249
>
>
>
>
> On Tue, Apr 15, 2014 at 10:49 AM, Juned Khan <jkh...@gm...> wrote:
>>
>> Hi Masataka,
>>
>> Thanks for help
>>
>>
>> On Tue, Apr 15, 2014 at 7:07 AM, Masataka Saito <pg...@gm...> wrote:
>>>
>>> I'm not sure but GTM or GTM proxy seems to be wrong.
>>>
>>> Please check their log.
>>> If you find an issue, search postgres-xc-general ml for keyword
>>> "register.node".
>>>
>>> Regards.
>>>
>>> On 14 April 2014 20:11, Juned Khan <jkh...@gm...> wrote:
>>> > Here few more logs of the component which doesn't start
>>> >
>>> > root@db02:~# tail -f
>>> >
>>> > /home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log
>>> > LOG:  database system was interrupted; last known up at 2014-04-14
>>> > 11:00:55
>>> > WAT
>>> > LOG:  database system was not properly shut down; automatic recovery in
>>> > progress
>>> > LOG:  record with zero length at 4/9A01A588
>>> > LOG:  redo is not required
>>> > FATAL:  the database system is starting up
>>> > LOG:  autovacuum launcher started
>>> > LOG:  database system is ready to accept connections
>>> > WARNING:  worker took too long to start; canceled
>>> > FATAL:  Can not register Datanode on GTM
>>> >
>>> >
>>> > root@db02:~# tail -f
>>> > /home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log
>>> > DEBUG:  00000: name: unnamed; blockState:       DEFAULT; state:
>>> > INPROGR,
>>> > xid/subid/cid: 0/1/0, nestlvl: 1, children:
>>> > LOCATION:  ShowTransactionStateRec, xact.c:5238
>>> > DEBUG:  00000: Autovacuum launcher: connection to GTM closed
>>> > LOCATION:  CloseGTM, gtm.c:116
>>> > DEBUG:  00000: Autovacuum launcher: connection established to GTM with
>>> > string host=db02 port=20002 node_name=coord1
>>> > LOCATION:  InitGTM, gtm.c:84
>>> > WARNING:  01000: Xid is invalid.
>>> > LOCATION:  GetNewTransactionId, varsup.c:160
>>> > DEBUG:  00000: Getting snapshot. Current XID = 0
>>> > LOCATION:  GetSnapshotDataCoordinator, procarray.c:3054
>>> > DEBUG:  00000: Autovacuum launcher: connection to GTM closed
>>> > LOCATION:  CloseGTM, gtm.c:116
>>> > DEBUG:  00000: Autovacuum launcher: connection established to GTM with
>>> > string host=db02 port=20002 node_name=coord1
>>> > LOCATION:  InitGTM, gtm.c:84
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...>
>>> > wrote:
>>> >>
>>> >> Hi all,
>>> >>
>>> >> Yesterday due some problem datanode slave stopped working on one of my
>>> >> DB
>>> >> server. After figuring out the issue i freed out the space on that
>>> >> server.
>>> >> now almost 50% disk is free on server.
>>> >>
>>> >> the problem is after that incident i am not able start all my pgxc
>>> >> components, now its uncertain some of components starts and sometimes
>>> >> it
>>> >> does not each time when i execute "stop all" and "start all" scenario
>>> >> is
>>> >> different.I am not able to figure out the problem which causing this
>>> >> issue.
>>> >>
>>> >> I have enabled the debug log though and tried to identify the issue
>>> >> and
>>> >> after reviewing the logs it seems memory related issue, i am not sure
>>> >> about
>>> >> this.
>>> >>
>>> >> PGXC start all
>>> >> Start GTM master
>>> >> gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not
>>> >> exist
>>> >> Is server running?
>>> >> server starting
>>> >> Start GTM slavegtm_ctl: PID file
>>> >> "/home/postgres/pgxc/nodes/gtm/gtm.pid"
>>> >> does not exist
>>> >> Is server running?
>>> >> server starting
>>> >> Done.
>>> >> Starting all the gtm proxies.
>>> >> Starting gtm proxy gtm_pxy1.
>>> >> Starting gtm proxy gtm_pxy2.
>>> >> gtm_proxy: no process found
>>> >> server starting
>>> >> gtm_proxy: no process found
>>> >> server starting
>>> >> Done.
>>> >> Starting coordinator master.
>>> >> Starting coordinator master coord1
>>> >> Starting coordinator master coord2
>>> >> DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
>>> >> LOCATION:  PostmasterMain, postmaster.c:962
>>> >> DEBUG:  00000: -----------------------------------------
>>> >> LOCATION:  PostmasterMain, postmaster.c:964
>>> >> DEBUG:  00000:  MAIL=/var/mail/postgres
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  SSH_CLIENT=41.218.72.115 48992 59696
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  USER=postgres
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LANGUAGE=en_ZA:en
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  SHLVL=1
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  HOME=/home/postgres
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  PG_GRANDPARENT_PID=19891
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LOGNAME=postgres
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  _=/usr/local/bin/pg_ctl
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:
>>> >> PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LANG=en_ZA.UTF-8
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  SHELL=/bin/bash
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  PWD=/home/postgres
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_MONETARY=C
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_NUMERIC=C
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_TIME=C
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000: -----------------------------------------
>>> >> LOCATION:  PostmasterMain, postmaster.c:969
>>> >> DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
>>> >> LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
>>> >> DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
>>> >> LOCATION:  SlruScanDirectory, slru.c:1312
>>> >> DEBUG:  00000: removing file "pg_notify/0000"
>>> >> LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
>>> >> DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
>>> >> LOCATION:  set_max_safe_fds, fd.c:548
>>> >> LOG:  00000: redirecting log output to logging collector process
>>> >> HINT:  Future log output will appear in directory "pg_log".
>>> >> LOCATION:  SysLogger_Start, syslogger.c:649
>>> >> DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
>>> >> LOCATION:  PostmasterMain, postmaster.c:962
>>> >> DEBUG:  00000: -----------------------------------------
>>> >> LOCATION:  PostmasterMain, postmaster.c:964
>>> >> DEBUG:  00000:  MAIL=/var/mail/postgres
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  SSH_CLIENT=41.218.72.115 33169 59696
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  USER=postgres
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LANGUAGE=en_ZA:en
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  SHLVL=1
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  HOME=/home/postgres
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  PG_GRANDPARENT_PID=19507
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LOGNAME=postgres
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  _=/usr/local/bin/pg_ctl
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:
>>> >> PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LANG=en_ZA.UTF-8
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  SHELL=/bin/bash
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  PWD=/home/postgres
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_MONETARY=C
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_NUMERIC=C
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000:  LC_TIME=C
>>> >> LOCATION:  PostmasterMain, postmaster.c:967
>>> >> DEBUG:  00000: -----------------------------------------
>>> >> LOCATION:  PostmasterMain, postmaster.c:969
>>> >> DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
>>> >> LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
>>> >> DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
>>> >> LOCATION:  SlruScanDirectory, slru.c:1312
>>> >> DEBUG:  00000: removing file "pg_notify/0000"
>>> >> LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
>>> >> DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
>>> >> LOCATION:  set_max_safe_fds, fd.c:548
>>> >> LOG:  00000: redirecting log output to logging collector process
>>> >> HINT:  Future log output will appear in directory "pg_log".
>>> >> LOCATION:  SysLogger_Start, syslogger.c:649
>>> >> DEBUG:  00000: logger shutting down
>>> >> LOCATION:  SysLoggerMain, syslogger.c:517
>>> >> DEBUG:  00000: shmem_exit(0): 0 callbacks to make
>>> >> LOCATION:  shmem_exit, ipc.c:212
>>> >> DEBUG:  00000: proc_exit(0): 0 callbacks to make
>>> >> LOCATION:  proc_exit_prepare, ipc.c:184
>>> >> DEBUG:  00000: exit(0)
>>> >> LOCATION:  proc_exit, ipc.c:135
>>> >> DEBUG:  00000: shmem_exit(-1): 0 callbacks to make
>>> >> LOCATION:  shmem_exit, ipc.c:212
>>> >> DEBUG:  00000: proc_exit(-1): 0 callbacks to make
>>> >> LOCATION:  proc_exit_prepare, ipc.c:184
>>> >> pg_ctl: could not start server
>>> >> Examine the log output.
>>> >> Done.
>>> >> Starting all the datanode masters.
>>> >> Starting datanode master datanode1.
>>> >> LOG:  redirecting log output to logging collector process
>>> >> HINT:  Future log output will appear in directory "pg_log".
>>> >> Done.
>>> >> Starting all the datanode slaves.
>>> >> Starting datanode slave datanode1.
>>> >> LOG:  redirecting log output to logging collector process
>>> >> HINT:  Future log output will appear in directory "pg_log".
>>> >> Done.
>>> >> PGXC monitor all
>>> >> Running: gtm master
>>> >> Running: gtm slave
>>> >> Running: gtm proxy gtm_pxy1
>>> >> Running: gtm proxy gtm_pxy2
>>> >> Running: coordinator master coord1
>>> >> Not running: coordinator master coord2
>>> >> Not running: datanode master datanode1
>>> >> Running: datanode slave datanode1
>>> >>
>>> >>
>>> >> My DB server has 32 GB RAM and 260 GB of Hard disk and i am using
>>> >> pgxc-1.2.1. what would be the optimal memory related postgresql.conf
>>> >> configuration for this server.
>>> >>
>>> >> Anyone has an idea about this issue ?
>>> >>
>>> >> --
>>> >> Thanks,
>>> >> Juned Khan
>>> >> iNextrix Technologies Pvt Ltd.
>>> >> www.inextrix.com
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Thanks,
>>> > Juned Khan
>>> > iNextrix Technologies Pvt Ltd.
>>> > www.inextrix.com
>>> >
>>> >
>>> > ------------------------------------------------------------------------------
>>> > Learn Graph Databases - Download FREE O'Reilly Book
>>> > "Graph Databases" is the definitive new guide to graph databases and
>>> > their
>>> > applications. Written by three acclaimed leaders in the field,
>>> > this first edition is now available. Download your free book today!
>>> > http://p.sf.net/sfu/NeoTech
>>> > _______________________________________________
>>> > Postgres-xc-general mailing list
>>> > Pos...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>>> >
>>
>>
>>
>>
>> --
>> Thanks,
>> Juned Khan
>> iNextrix Technologies Pvt Ltd.
>> www.inextrix.com
>
>
>
>
> --
> Thanks,
> Juned Khan
> iNextrix Technologies Pvt Ltd.
> www.inextrix.com