Menu
▾
▴
Reconnect_GTM-Proxy
Initial Design
(April 6th, 2011, Koichi Suzuki)
Now we have GTM-Standby, which backs up all the current GTM status in synchronous way and fails over when master GTM fails. Now we need a mechanism in GTM-Proxy to reconnect to new GTM in such case.
When GTM-Standby fails over the GTM, all the status has been copied to the Standby and GTM-Proxy should just disconnect from the old GTM, reconnect to the new one and register itself. If the last command to the old GTM did not respond, the command can be reissued to the new GTM to get correct response.
Because reconnect itself will be triggered by XCM module, xcwatcher and monitoring agent, this document provides initial design how to implement the reconnect in GTM-Proxy.
Reconnect can be initiated by invoking gtm_ctl with a new command "reconnect". The syntacs will be as follows:
gtm_ctl -S gtm_proxy reconnect -D dir -o "-s xxx -t xxx"
where -D option describes gtm_proxy's working directory, which must be the same as it started. -s and -t specifies address and the port number of the new GTM.
Also, at the time of start, gtm_ctl will have new options, namely, -w and -n. They indicates how long and how many times GTM-Proxy waits when it detects connection closure from GTM. During this wait, users can issue gtm_ctl reconnect to GTM-Proxy.
Within gtm_ctl, these options are backed up to gtm_proxy.opts file in GTM's working directory, merged with existing options. Then gtm_ctl will prepare gtm_proxy_sighup.opt (file name may change) file to indicate to reconnect and issue SIGHUP signal to the gtm_proxy.
Gtm_proxy SIGHUP signal handler will check grm_proxy_sighup_opt and determines it should reconnect to the new GTM (at present, this is only one option of SIGUP action), and update a flag to indicate reconnect. Such a flag can be stored in thread-specific structure.
Each thread checks this flag before it send commands to GTM. If error is detected to receive response from GTM, it will be the time to check this structure. If it is no set, then the thread can wait for a little while and recheck this.
If "should_lock" bit is set, then the thread disconnects current connection to (old) GTM and reconnect to the new one and can continue service to coordinator or datanode.
Current Code Study
gtm_ctl
It was very straightforward code.
The point is: new GTM host and port will be give as -o option. This will be passed to the target GTM Proxy through "newgtm" file under -D option directory.
Only one issue is that we have to leave option argument check to gtm proxy's signal handler. So the result of gtm_ctl can only be known by testing the log.
gtm proxy
Signal hander is straightforward too. I need new signal handler because current one just exit whole process. GTMProxy_ThreadInfo contains all the information needed for each thread. We can add a flag to reconnect (and necessary lock) to this structure.
Because this thread_info is not exposed, I need to have one anchor in heap area and chain all the thread info from here, so that signal handler can find where to set the info (need lock to write the value)
Each thread polls this value. Because signal handler is only one writer and write will be done in one instruction, we don't need a lock to read this. When it recognizes the flag is set, then it reconnects to the new GTM and (after acquiring the lock) set the flag off.
worker thread main loop
GTMProxy_ThreadMain() is the main thread loop.
We need to add the polling of SIGUSR2 handler flag. If it is set, then GTMProxy disconnects the current Proxy and reconnect to the new GTM (promoted one).
How to detect gtm_ctl reconnect ?
It's not simple.
1. Need to backup command from clients. --> Is memory context safe?
2. Need to detect gtm_ctl while it is waiting reply of send() to GTM.
send() may return with the error (disconnected) --> need error handling. Send/receive handling: need handling send()
