building without X11 is not possible:
clang++ -c -pipe -stdlib=libc++ -mmacosx-version-min=10.7 -O2 -arch x86_64 -Wall -W -DDATADIR="/usr/local/Cellar/qjackctl/0.4.1/share" -DQT_NO_DEBUG -DQT_XML_LIB -DQT_GUI_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/local/Cellar/qt/4.8.7_1/mkspecs/unsupported/macx-clang-libc++ -I. -I/usr/local/Cellar/qt/4.8.7_1/lib/QtCore.framework/Versions/4/Headers -I/usr/local/Cellar/qt/4.8.7_1/lib/QtCore.framework/Versions/4/Headers -I/usr/local/Cellar/qt/4.8.7_1/lib/QtGui.framework/Versions/4/Headers -I/usr/local/Cellar/qt/4.8.7_1/lib/QtGui.framework/Versions/4/Headers -I/usr/local/Cellar/qt/4.8.7_1/lib/QtXml.framework/Versions/4/Headers -I/usr/local/Cellar/qt/4.8.7_1/lib/QtXml.framework/Versions/4/Headers -I/usr/local/Cellar/qt/4.8.7_1/include -I. -I/usr/local/Cellar/qt/4.8.7_1/include -I/usr/local/include -I.moc -I.ui -F/usr/local/Cellar/qt/4.8.7_1/lib -o .obj/qrc_qjackctl.o qrc_qjackctl.cpp
clang++ -headerpad_max_install_names -stdlib=libc++ -mmacosx-version-min=10.7 -arch x86_64 -o qjackctl.app/Contents/MacOS/qjackctl .obj/qjackctl.o .obj/qjackctlAlsaConnect.o .obj/qjackctlConnect.o .obj/qjackctlConnectAlias.o .obj/qjackctlInterfaceComboBox.o .obj/qjackctlJackConnect.o .obj/qjackctlPatchbay.o .obj/qjackctlPatchbayFile.o .obj/qjackctlPatchbayRack.o .obj/qjackctlSession.o .obj/qjackctlSetup.o .obj/qjackctlSystemTray.o .obj/qjackctlAboutForm.o .obj/qjackctlConnectionsForm.o .obj/qjackctlMainForm.o .obj/qjackctlMessagesStatusForm.o .obj/qjackctlPatchbayForm.o .obj/qjackctlSessionForm.o .obj/qjackctlSetupForm.o .obj/qjackctlSocketForm.o .obj/moc_qjackctlConnect.o .obj/moc_qjackctlPatchbay.o .obj/moc_qjackctlPatchbayRack.o .obj/moc_qjackctlSystemTray.o .obj/moc_qjackctlAboutForm.o .obj/moc_qjackctlConnectionsForm.o .obj/moc_qjackctlMainForm.o .obj/moc_qjackctlMessagesStatusForm.o .obj/moc_qjackctlPatchbayForm.o .obj/moc_qjackctlSessionForm.o .obj/moc_qjackctlSetupForm.o .obj/moc_qjackctlSocketForm.o .obj/qrc_qjackctl.o -F/usr/local/Cellar/qt/4.8.7_1/lib -L/usr/local/Cellar/qt/4.8.7_1/lib -L/usr/local/Cellar/qt/4.8.7_1/lib -L/usr/local/lib -L/usr/lib -ljack -lX11 -framework CoreAudio -framework CoreFoundation -framework QtXml -L/usr/local/opt/openssl/lib -F/usr/local/Cellar/qt/4.8.7_1/lib -framework QtCore -framework QtGui
ld: library not found for -lX11
It looks like it's linked against X11 even if it is not needed (https://github.com/rncbc/qjackctl/blob/master/configure.ac#L364).
merged https://github.com/rncbc/qjackctl/pull/8
qjackctl v0.4.1.7