Currently I have implemented a "getFile" method that simply returns an array of bytes back to the requester. However, if the file is too large, it overloads my system by creating way too many bytes and creating a huge response.
Now, I know I can take an input stream in, but how can I send an output stream out?
This is currently not supported, I'm afraid. Behind the scenes the response objects that are serialized back to the client pass through a StringWriter that accumulates the complete response message before it is sent back to the client. The XML-RPC spec mandates that the Content-Length header is set so the whole message needs to be collected in memory before that can be determined.
That said, you may get away with saving some memory by using a custom serializer. If you have a look at one of the built-in serializers:
That one indicates to the library that it is prepared to serialize any object of type List. In the same way you may create your own that serializes any InputStream. Your implementation could then Base64-encode the stream on-the-fly without having to put it into a byte array first.
Remember, however, that the whole file will end up in its Base64-encoded form in the internal StringWriter before being passed on to the client. You will have saved yourself the byte-array version of the file, though.
I don't think it's a very good idea o send files over XML-RPC, which is a bad fit here. Is it at all possible to change the exposed XML-RPC API on your server such that it returns the URI of the file instead? Then clients can use some simple HTTP client to fetch the actual file data.