I’m not sure that you need to go through all that trouble to find out the media-type. I had a need to find out the media-type assigned to the transform output and all I did was:
String mediaType = xf.getOutputProperty(OutputKeys.MEDIA_TYPE);
String encoding = xf.getOutputProperty(OutputKeys.ENCODING);
Maybe this would work in your situation.
That will only get the media-type for the default (unnamed) xsl:output declaration. With xsl:result-document you can use a named xsl:output declaration or you can compute the serialization properties dynamically.