[Doxygen-develop] segfault with cyclic subgroups

[Helmut G. <he...@su...>]

Dear Doxygen developers,

I am faced with a segmentation fault of Doxygen 1.8.8 (as used by Debian
unstable on amd64) when trying to process the following (invalid?)
header:

| /**
|  * @addtogroup foo1 foo2
|  * @{
|  * @addtogroup foo4 foo5
|  * @{
|  * @defgroup foo1 foo3
|  * @}
|  * @}
|  */

While this example looks artificial, it was processed by earlier
versions of Doxygen and the Enlightenment libraries reproduce the same
segmentation fault (see https://bugs.debian.org/762272).

What happens is that stack space is exhausted in a recursion of
recursivelyAddGroupListToTitle. Turns out the recursion alternates
between group foo1 and foo4. Conceptually, I have no clue what cyclic
subgroup membership is supposed to mean. One way to look at this problem
is to argue that Doxygen fails to refuse processing cyclic subgroups.
Alternatively the depth of the recursion can be limited.

I am proposing the attached patch (suitable to "git am") to make Doxygen
error out when faced with cyclic subgroups. Is this patch reasonable for
inclusion upstream and does it fully fix the problem?

I would like to add some minimal fix to the Debian package as Debian
jessie is almost frozen (i.e. a new upstream release of Doxygen cannot
be added to Debian atm).

Thanks for considering

Helmut