XC_Watchdog

XC_Watchdog why?

To monitor if each XC component is running, psql is not sufficient because it does not check gtm/gtm_proxy/datanode. Also, psql detection may take time. As discussed in the cluster summit (http://wiki.postgresql.org/wiki/PgCon2012CanadaClusterSummit), watchdog time will be nice for this purpose.

Here's a design of watchdog timer:

• Have separate shared memory for each component,
• Postmaster and gtm/gtm_proxy server main loop increment each watchdog timer,
• ( Timer will be detected by separate command to report any fault

For this purpose, need some GUC and GTM/GTM-Proxy configuration parameters to specify

• If watchdog time is on
• Timer increment interval (maybe in milliseconds)

Shmid for each component will be kept in pg_control, gtm.control and gtm_proxy.control files.

API to attach shared memory for watchdog and read the timer value will be provided too.

Implementation

1. Add GUC

   1. xc_watchdog onoff

   2. xc_watchdog_interval (in millisecond)

   3. Add gtm.opt option
   4. xc_watchdog on/off
   5. xc_watchdog_interval in millisecond
   6. Add gtm_proxy.opt option
   7. xc_watchdog on/off
   8. xc_watchdog_interval in millisecond
   9. Write the following
   10. watchdog structure definition
   11. function to get, initialize and attach watchdog memory
   12. Check the magic and node category (coord/datanode/gtm/gtm_proxy)
   13. function to imcrement the watchdog timer
   14. Individual functions for coord/datanode/gtm/gtm_proxy to do the following:
   15. Initialize pointer to the watchdog structure
   16. Set interval
   17. Increment the timer (in each server loop)
   18. Handle status file.

When status file is first read, check shmid of the watchdog structure. If set, try to attach this and check if it is valid. If not set or not valid, then shmget, attach and initialize it, and then write back the status file.

?? When watchdog structure can be removed? ---> When each component terminate gracefully, then the structure can be removed.

1. Watchdog detector
   1. API for
   2. watchdog detector application to attach and monitor.
   3. (maybe) some asynchronous means to notify the target failure
   4. Detect XC cluster configuration and build XC cluster configuration structure.
   5. various supporting API for c)
   6. Command to
   7. Detect a failurem
   8. Report to external entities --> could be smnp trap, could be pacemaker monitoring tool, could be any others.
   9. Collect XC cluster configuration and advice what to do when any failure occurs --> Split brain handling will be important here.