I tried to enumerate the PageLabels just to find a bug in the Tree Enumerator.
NumberTree and NameTree override the PairsKey property to be either "Nums" or "Names".
The Tree class uses then this property to access the PDF Objects.
The problem is that the Enumerator doesn't, it's using hardcoded PdfName.Names and hence only works with the NameTree.
I have created a private property pairsKey, which I set in the constructor to tree.pairsKey.
Then I replaced all usage of PdfName.Names in the iterator with pairsKey and it's working fine now.