If an extension function implemented according to the JAXP XPath API (bound via the XPathFunctionResolver) returns an object of type NodeList, Saxon does not recognize the type and treats it as a general-purpose external object. In fact this applies generally to any extension that returns an object whose dynamic type is NodeList, if the extension did not declare NodeList as the static return type. (JAXP XPath extension functions unfortunately have no way of declaring the return type statically.)
A patch for this particular case will be placed in Subversion for the 9.1 branch. It doesn't solve the general problem, because relying on dynamic type information is generally a poor strategy given the rich type system of XPath 2.0.