Here's what I'm currently using. It seems to work so far. Looking through the PyMol binary I see about 50 possible names though, so it would be nice if there were a built-in way to detect X11.
""" Make an educated guess as to whether Tk is installed,
hopefully without triggering any installation on Macs
hasTk = True
if sys.platform=="darwin": #Mac
# Hack: check the path for entries containing 'X11'
hasTk = any([ "X11" in p.upper() for p in sys.path])
hasTk = False