Re: [Postgres-xc-general] Datanode is not starting up after "No space left on device"

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi Masataka,

You answer helped but again i am having this problem. do i need to remove
register.node file again ?

GTM logs

1:140587359135488:2014-04-15 12:22:56.711 WAT -LOG:  Saving transaction
restoration info, backed-up gxid: 259975
LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2649
1:140587359135488:2014-04-15 12:22:56.712 WAT -LOG:  Started to run as
GTM-Active.
LOCATION:  main, main.c:641
1:140587359135488:2014-04-15 12:22:57.712 WAT -LOG:  Any GTM standby node
not found in registered node(s).
LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:381
1:140587350980352:2014-04-15 12:22:57.717 WAT -LOG:  Failed to establish a
connection with GTM standby. - 0x13ac390
LOCATION:  gtm_standby_connect_to_standby_int, gtm_standby.c:396
1:140587321448192:2014-04-15 12:27:02.055 WAT -LOG:  unexpected EOF on
client connection
LOCATION:  ReadCommand, main.c:1374
1:140587359135488:2014-04-15 12:27:03.928 WAT -LOG:  GTM shutting down.
LOCATION:  ServerLoop, main.c:802
1:140587359135488:2014-04-15 12:27:03.928 WAT -LOG:  Saving transaction
info - next_gxid: 257975
LOCATION:  GTM_SaveTxnInfo, gtm_txn.c:2632
Received signal 15

GTM Proxy logs
LOCATION:  pgxcnode_add_info, register_common.c:249
1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG:  Node with the given ID
number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG:  Node with the given ID
number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG:  Node with the given ID
number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG:  Node with the given ID
number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:140179440826112:2014-04-15 12:27:22.893 WAT -LOG:  Node with the given ID
number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249

On Tue, Apr 15, 2014 at 10:49 AM, Juned Khan <jkh...@gm...> wrote:

> Hi Masataka,
>
> Thanks for help
>
>
> On Tue, Apr 15, 2014 at 7:07 AM, Masataka Saito <pg...@gm...> wrote:
>
>> I'm not sure but GTM or GTM proxy seems to be wrong.
>>
>> Please check their log.
>> If you find an issue, search postgres-xc-general ml for keyword
>> "register.node".
>>
>> Regards.
>>
>> On 14 April 2014 20:11, Juned Khan <jkh...@gm...> wrote:
>> > Here few more logs of the component which doesn't start
>> >
>> > root@db02:~# tail -f
>> >
>> /home/postgres/pgxc/nodes/dn_master/pg_log/postgresql-2014-04-14_114619.log
>> > LOG:  database system was interrupted; last known up at 2014-04-14
>> 11:00:55
>> > WAT
>> > LOG:  database system was not properly shut down; automatic recovery in
>> > progress
>> > LOG:  record with zero length at 4/9A01A588
>> > LOG:  redo is not required
>> > FATAL:  the database system is starting up
>> > LOG:  autovacuum launcher started
>> > LOG:  database system is ready to accept connections
>> > WARNING:  worker took too long to start; canceled
>> > FATAL:  Can not register Datanode on GTM
>> >
>> >
>> > root@db02:~# tail -f
>> > /home/postgres/pgxc/nodes/coord/pg_log/postgresql-2014-04-14_114612.log
>> > DEBUG:  00000: name: unnamed; blockState:       DEFAULT; state: INPROGR,
>> > xid/subid/cid: 0/1/0, nestlvl: 1, children:
>> > LOCATION:  ShowTransactionStateRec, xact.c:5238
>> > DEBUG:  00000: Autovacuum launcher: connection to GTM closed
>> > LOCATION:  CloseGTM, gtm.c:116
>> > DEBUG:  00000: Autovacuum launcher: connection established to GTM with
>> > string host=db02 port=20002 node_name=coord1
>> > LOCATION:  InitGTM, gtm.c:84
>> > WARNING:  01000: Xid is invalid.
>> > LOCATION:  GetNewTransactionId, varsup.c:160
>> > DEBUG:  00000: Getting snapshot. Current XID = 0
>> > LOCATION:  GetSnapshotDataCoordinator, procarray.c:3054
>> > DEBUG:  00000: Autovacuum launcher: connection to GTM closed
>> > LOCATION:  CloseGTM, gtm.c:116
>> > DEBUG:  00000: Autovacuum launcher: connection established to GTM with
>> > string host=db02 port=20002 node_name=coord1
>> > LOCATION:  InitGTM, gtm.c:84
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Apr 14, 2014 at 3:56 PM, Juned Khan <jkh...@gm...>
>> wrote:
>> >>
>> >> Hi all,
>> >>
>> >> Yesterday due some problem datanode slave stopped working on one of my
>> DB
>> >> server. After figuring out the issue i freed out the space on that
>> server.
>> >> now almost 50% disk is free on server.
>> >>
>> >> the problem is after that incident i am not able start all my pgxc
>> >> components, now its uncertain some of components starts and sometimes
>> it
>> >> does not each time when i execute "stop all" and "start all" scenario
>> is
>> >> different.I am not able to figure out the problem which causing this
>> issue.
>> >>
>> >> I have enabled the debug log though and tried to identify the issue and
>> >> after reviewing the logs it seems memory related issue, i am not sure
>> about
>> >> this.
>> >>
>> >> PGXC start all
>> >> Start GTM master
>> >> gtm_ctl: PID file "/home/postgres/pgxc/nodes/gtm/gtm.pid" does not
>> exist
>> >> Is server running?
>> >> server starting
>> >> Start GTM slavegtm_ctl: PID file
>> "/home/postgres/pgxc/nodes/gtm/gtm.pid"
>> >> does not exist
>> >> Is server running?
>> >> server starting
>> >> Done.
>> >> Starting all the gtm proxies.
>> >> Starting gtm proxy gtm_pxy1.
>> >> Starting gtm proxy gtm_pxy2.
>> >> gtm_proxy: no process found
>> >> server starting
>> >> gtm_proxy: no process found
>> >> server starting
>> >> Done.
>> >> Starting coordinator master.
>> >> Starting coordinator master coord1
>> >> Starting coordinator master coord2
>> >> DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
>> >> LOCATION:  PostmasterMain, postmaster.c:962
>> >> DEBUG:  00000: -----------------------------------------
>> >> LOCATION:  PostmasterMain, postmaster.c:964
>> >> DEBUG:  00000:  MAIL=/var/mail/postgres
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  SSH_CLIENT=41.218.72.115 48992 59696
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  USER=postgres
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LANGUAGE=en_ZA:en
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  SHLVL=1
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  HOME=/home/postgres
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  PG_GRANDPARENT_PID=19891
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LOGNAME=postgres
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  _=/usr/local/bin/pg_ctl
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:
>>  PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LANG=en_ZA.UTF-8
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  SHELL=/bin/bash
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  PWD=/home/postgres
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 48992 41.218.72.115 59696
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_MONETARY=C
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_NUMERIC=C
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_TIME=C
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000: -----------------------------------------
>> >> LOCATION:  PostmasterMain, postmaster.c:969
>> >> DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
>> >> LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
>> >> DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
>> >> LOCATION:  SlruScanDirectory, slru.c:1312
>> >> DEBUG:  00000: removing file "pg_notify/0000"
>> >> LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
>> >> DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
>> >> LOCATION:  set_max_safe_fds, fd.c:548
>> >> LOG:  00000: redirecting log output to logging collector process
>> >> HINT:  Future log output will appear in directory "pg_log".
>> >> LOCATION:  SysLogger_Start, syslogger.c:649
>> >> DEBUG:  00000: postgres: PostmasterMain: initial environment dump:
>> >> LOCATION:  PostmasterMain, postmaster.c:962
>> >> DEBUG:  00000: -----------------------------------------
>> >> LOCATION:  PostmasterMain, postmaster.c:964
>> >> DEBUG:  00000:  MAIL=/var/mail/postgres
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  SSH_CLIENT=41.218.72.115 33169 59696
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  USER=postgres
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LANGUAGE=en_ZA:en
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  SHLVL=1
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  HOME=/home/postgres
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  PG_GRANDPARENT_PID=19507
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LOGNAME=postgres
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  _=/usr/local/bin/pg_ctl
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  PGSYSCONFDIR=/usr/local/pgsql/etc
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:
>>  PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LANG=en_ZA.UTF-8
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  SHELL=/bin/bash
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  PWD=/home/postgres
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  SSH_CONNECTION=41.218.72.115 33169 41.218.72.114 59696
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  PGDATA=/home/postgres/pgxc/nodes/coord
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_COLLATE=en_ZA.UTF-8
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_CTYPE=en_ZA.UTF-8
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_MESSAGES=en_ZA.UTF-8
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_MONETARY=C
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_NUMERIC=C
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000:  LC_TIME=C
>> >> LOCATION:  PostmasterMain, postmaster.c:967
>> >> DEBUG:  00000: -----------------------------------------
>> >> LOCATION:  PostmasterMain, postmaster.c:969
>> >> DEBUG:  00000: invoking IpcMemoryCreate(size=148193280)
>> >> LOCATION:  CreateSharedMemoryAndSemaphores, ipci.c:149
>> >> DEBUG:  00000: SlruScanDirectory invoking callback on pg_notify/0000
>> >> LOCATION:  SlruScanDirectory, slru.c:1312
>> >> DEBUG:  00000: removing file "pg_notify/0000"
>> >> LOCATION:  SlruScanDirCbDeleteAll, slru.c:1277
>> >> DEBUG:  00000: max_safe_fds = 984, usable_fds = 1000, already_open = 6
>> >> LOCATION:  set_max_safe_fds, fd.c:548
>> >> LOG:  00000: redirecting log output to logging collector process
>> >> HINT:  Future log output will appear in directory "pg_log".
>> >> LOCATION:  SysLogger_Start, syslogger.c:649
>> >> DEBUG:  00000: logger shutting down
>> >> LOCATION:  SysLoggerMain, syslogger.c:517
>> >> DEBUG:  00000: shmem_exit(0): 0 callbacks to make
>> >> LOCATION:  shmem_exit, ipc.c:212
>> >> DEBUG:  00000: proc_exit(0): 0 callbacks to make
>> >> LOCATION:  proc_exit_prepare, ipc.c:184
>> >> DEBUG:  00000: exit(0)
>> >> LOCATION:  proc_exit, ipc.c:135
>> >> DEBUG:  00000: shmem_exit(-1): 0 callbacks to make
>> >> LOCATION:  shmem_exit, ipc.c:212
>> >> DEBUG:  00000: proc_exit(-1): 0 callbacks to make
>> >> LOCATION:  proc_exit_prepare, ipc.c:184
>> >> pg_ctl: could not start server
>> >> Examine the log output.
>> >> Done.
>> >> Starting all the datanode masters.
>> >> Starting datanode master datanode1.
>> >> LOG:  redirecting log output to logging collector process
>> >> HINT:  Future log output will appear in directory "pg_log".
>> >> Done.
>> >> Starting all the datanode slaves.
>> >> Starting datanode slave datanode1.
>> >> LOG:  redirecting log output to logging collector process
>> >> HINT:  Future log output will appear in directory "pg_log".
>> >> Done.
>> >> PGXC monitor all
>> >> Running: gtm master
>> >> Running: gtm slave
>> >> Running: gtm proxy gtm_pxy1
>> >> Running: gtm proxy gtm_pxy2
>> >> Running: coordinator master coord1
>> >> Not running: coordinator master coord2
>> >> Not running: datanode master datanode1
>> >> Running: datanode slave datanode1
>> >>
>> >>
>> >> My DB server has 32 GB RAM and 260 GB of Hard disk and i am using
>> >> pgxc-1.2.1. what would be the optimal memory related postgresql.conf
>> >> configuration for this server.
>> >>
>> >> Anyone has an idea about this issue ?
>> >>
>> >> --
>> >> Thanks,
>> >> Juned Khan
>> >> iNextrix Technologies Pvt Ltd.
>> >> www.inextrix.com
>> >
>> >
>> >
>> >
>> > --
>> > Thanks,
>> > Juned Khan
>> > iNextrix Technologies Pvt Ltd.
>> > www.inextrix.com
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Learn Graph Databases - Download FREE O'Reilly Book
>> > "Graph Databases" is the definitive new guide to graph databases and
>> their
>> > applications. Written by three acclaimed leaders in the field,
>> > this first edition is now available. Download your free book today!
>> > http://p.sf.net/sfu/NeoTech
>> > _______________________________________________
>> > Postgres-xc-general mailing list
>> > Pos...@li...
>> > https://lists.sourceforge.net/lists/listinfo/postgres-xc-general
>> >
>>
>
>
>
> --
> Thanks,
> Juned Khan
> iNextrix Technologies Pvt Ltd.
> www.inextrix.com
>

-- 
Thanks,
Juned Khan
iNextrix Technologies Pvt Ltd.
www.inextrix.com