#4 UTF-8 is not handled correctly

[Andreas Koch]

When converting an UTF-8 encoded string to an InputStream the encoding of the string is not taken into account, which causes all letters to be converted to question marks. The attached Rapid Miner process illustrates the problem.

I can fix the problem by giving the encoding as an argument to the getBytes() method called in the loadDocument() method of the SourceAsTextLoader class:
/** Open the document and return an input stream on it. */
public InputStream loadDocument(WVTDocumentInfo d) throws WVToolException {

try
{
byte[] bArray = d.getSourceName().getBytes(d.getContentEncoding());
stream = new ByteArrayInputStream(bArray);
}
catch(Exception e)
{
throw new WVToolException("Could not decode string with "+d.getContentEncoding());
}

return stream;
}

[Andreas Koch]

Rapidminer process which illustrates the problem