geany --list-documents launches a new full-fledged geany instance when there isn't another one currently running. In fact, it doesn't even return the list of documents after closing the new instance, it behaves as if it were a no args geany invocation.
Geany man page states "This [--list-documents] can be used to read the currently opened documents in Geany from an external script or tool...". But the current behavior is not very amenable to automation because it depends on a context (another geany running or not) that can't be reliably checked (because of non atomicity).
IMO the right behavior would be to return an empty document list, because in fact the is no document currently being edited when no other geany instance is running.