[Postgres-xc-general] GTM proxy issue

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

We started noticing outages with one of the nodes in our cluster this morning.  When I looked at the box, gtm proxy was running hot.  So, I looked at the logs of the GTM proxy - which was spooling rapidly.

LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746454378240:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746445985536:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746454378240:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249
1:139746445985536:2014-05-28 16:06:44.641 UTC -LOG:  Node with the given ID number already exists
LOCATION:  pgxcnode_add_info, register_common.c:249

After several restarts, it was painfully obvious that this proxy was not going to start.  In an attempt to recover the GTM proxy, I saved the GTM proxy directory and rebuilt it from scratch expecting to rebuild its state from the master.  No dice, here's what the data coordinator reported.

26885 | 2014-05-28 16:01:49 UTC | LOG:  autovacuum launcher started
26885 | 2014-05-28 16:02:50 UTC | ERROR:  GTM error, could not obtain snapshot XID = 1193752
26885 | 2014-05-28 16:03:51 UTC | WARNING:  Xid is invalid.
26885 | 2014-05-28 16:04:31 UTC | WARNING:  Xid is invalid.
26863 | 2014-05-28 16:04:49 UTC | FATAL:  Can not register Coordinator on GTM
26885 | 2014-05-28 16:05:31 UTC | WARNING:  Xid is invalid.
26885 | 2014-05-28 16:05:31 UTC | ERROR:  GTM error, could not obtain snapshot XID = 0
26885 | 2014-05-28 16:06:12 UTC | WARNING:  Xid is invalid.

I believe the first portion of the log occurred when the autovacuum tried to connect to the new proxy.

Here's what I'd like to know.  Clearly the GTM proxy failure wasn't expected, but the problem here is one of recoverability.  How does one go about getting the coordinator and data node to a sane place given that the old GTM proxy was in an unrecoverable state and a new one can't seemingly be put in place to replace it?

Aaron