CSVReader 3 Doesn't Read the Entire Stream
Brought to you by:
aruckerjones,
sconway
Menu
▾
▴
#108 CSVReader 3 Doesn't Read the Entire Stream
When CSVReader closes the stream whenever there are no bytes immediately available for reading (without blocking). This can lead to incomplete consumption of the stream, when not reading from a file nor in-memory data.
This is apparent, for example, when the underlying stream has been wrapped in Channels.newInputStream, since that implementation always returns 0 bytes available when it's not a file channel.
http://sourceforge.net/p/opencsv/source/ci/4b071f94e15a912cae73692ac00a2d94d1bdc69f/
Could you please send a small piece of sample code that demonstrates the
defect or (better yet) a unit test that reproduces the issue.
On Thu, Feb 19, 2015 at 9:34 AM, James Leigh jamesrdf@users.sf.net wrote:
--
Scott Conway
scott.conway@gmail.com
http://www.conwayfamily.name
Related
Bugs:
#108The following worked in 2.0, but fails in 3.0
byte[] bytes = "name\r\nvalue\r\n".getBytes("UTF-8");
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
ReadableByteChannel ch = Channels.newChannel(bais);
InputStream in = Channels.newInputStream(ch);
InputStreamReader reader = new InputStreamReader(in, Charset.forName("UTF-8"));
CSVReader csv = new CSVReader(reader);
assertEquals(2, csv.readAll().size());
Channels don't support available(), unless it is a file channel. Readers across a slow network may also return 0 from available() before the end.
I was able to take your example and make a unit test and successfully recreated the issue.
The reason it is failing in version 3 but not version 2 was code that we added for an issue with Groovy - http://stackoverflow.com/questions/23058244/csvparser-stream-closed-error-when-parsing-multiple-files/23058601#23058601
What we added was a check to see if the file was closed before reading it. We did that by checking if the ready method in the BufferedReader built using the reader passed in returned true. After debugging the code I see that the issue is that the reader built using the Channel method you gave me will always return false until the first read is done. And after that it works like the other Readers.
I have not worked with the nio classes before so I don't know a workaround. If I cannot find one then I will add a boolean like I did with issue 106 that will allow you to bypass the first isClosed and see if that will work.
I have merged a fix into the trunk. If you can do a get pull of the source and test it out. Otherwise in the next couple of weeks I will push out the next version with this fix.
I added a value to the CSVReaderBuilder withVerifyReader by default the value will be true which is the current default behavior. By passing in false opencsv will mimic the behavior of version 2.4 and before. This is needed when using readers whose ready methods return false until a read is done.
Here is a test that now passes that shows how to use the verify reader.
@Test
public void issue108ReaderPlaysWellWithChannels() throws IOException {
byte[] bytes = "name\r\nvalue\r\n".getBytes("UTF-8");
ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
ReadableByteChannel ch = Channels.newChannel(bais);
InputStream in = Channels.newInputStream(ch);
InputStreamReader reader = new InputStreamReader(in, Charset.forName("UTF-8"));
CSVReaderBuilder builder = new CSVReaderBuilder(reader);
CSVReader csv = builder.withVerifyReader(false).build();
assertEquals(2, csv.readAll().size());
}
Pushed fix in 3.3.
I also hit this issue after upgrading from version 2.3 to 3.3. CSV reader doesn't read anything at all though calling readLine() on the buffered reader returns data.
My understanding is that isClosed() method is incorrect in using br.ready() because that method doesn't indicate the stream is closed, it just says some data is immediately available. If it's not available it doesn't mean it won't get available after some delay needed for data fetching.
Unfortunately the workaround with CSVReaderBuilder doesn't work for me as I need to pass a separator and quote char and there are no corresponding methods for those in CSVReaderBuilder.
I possibly could work around this issue by creating a class in com.opencsv package so that I could access the package protected constructor of CSVReader to pass verifyReader parameter. But I think I'll just roll back to version 2.3 until this issue is fixed.