#196 CsvToBean parse empty lines error

[gunten]

Situation as the attachments.
I use CSVReader with ColumnPositionMappingStrategy readed the first 7 lines, then use CsvToBean to parse the left part. code like this:

 CsvToBean<ReconRepayTempInfoDO> csvToBean = new CsvToBeanBuilder(reader)
                            .withType(ReconRepayTempInfoDO.class)
                            .withSkipLines(1)//jumpover header
                            .withIgnoreLeadingWhiteSpace(true)
                            .build();

                        Iterator<ReconRepayTempInfoDO> iterator = csvToBean.iterator();//exception
                    while (iterator.hasNext()) {
                            T reconRepayTempInfo = iterator.next();
                            ....
                    }

or

List<T> list = csvToBean.parse();

Both of them throw RunTimeException "Error capturing CSV header!" .
After looking up the source code, i found reason at ColumnPositionMappingStrategy.java line 72(version 4.5)

String[] firstLine = reader.peek();// return null

Is this a bug ?
I think when encounter an empty line, csvToBean.parse() return null could be nicer

[Scott Conway]

Hello Gunten

Could you please send me your CSV file, the java class with your ReconRepayTempInfoDO and a small sample program that gets the error that you are seeing because I cannot recreate the error you are experiencing.

By default you the CsvToBean creates a HeaderColumnNameMappingStrategy and it requires that your first line be the header and the header matches fields in the object. If so then one of the two fields in your first line does not exist in your ReconRepayTempInfoDO object and that would give you an error capturing CSV Header message. But that said if I see your full test program I can see where you overrode with a ColumnPositionMappingStrategy and can come up with a test that matches it.

Also with the default HeaderColumnNameMappingStrategy you will get an error on your last line because you have more fields than you do with your header. But that error would be "Caused by: com.opencsv.exceptions.CsvRequiredFieldEmptyException: Number of data fields does not match number of headers." and that is on purpose!

The other thing is that the skipLines will not skip the header it should skip the line after the header. But there again I am using the default CsvToBean.

Here is the test I added to CsvToBeanTest. If you can modify it to reproduce your result that would be awesome!!

@Test
public void verifyBug196() {
    String testString = "name,id,orderNumber\n" +
            "1,\"foo\", 3\n" +
            "1,\"a string\", 3\n" +
            "1,\"a string\", 3, \"more\",\"fields\"\n" +
            "1,\"bar\", 3";

    StringReader stringReader = new StringReader(testString);

    CsvToBean<MockBean> csvToBean = new CsvToBeanBuilder(stringReader)
            .withType(MockBean.class)
            .withIgnoreLeadingWhiteSpace(true)
            //.withThrowExceptions(false)
            .withSkipLines(1)
            .build();

    List<MockBean> rows = csvToBean.parse();
    assertEquals(2, rows.size());

}

Notice I have the .withThrowsException(false) commented out. With that uncommented out it would not throw an exception with the third data line (the one with more fields) but instead just skip it. So with throws exception as false and skip line as 1 I get back two data lines and with skip lines removed I get three.

..... Reading what you are saying it sounds line you are using a CSVReader to read the first seven lines of the file and then pass the same reader (without closing it) into the CsvToBeanBuilder. Is that correct? If so your line 8 must be the Header (all the fields must match what is in the ReconRepayTempInfoDO. I will try and modify my test and see what happens but I will say if true that is the first time I have seen a csv file formatted this way.

Scott :)

[Scott Conway]

Okay I rewrote the test based on the last part of my previous post and it is not throwing any errors.

@Test
public void verifyBug196() throws IOException {
    String testString =
            "dummy, line\n" +
            "another, dummy, line\n" +
            "name,id,orderNumber\n" +
            "1,\"foo\", 3\n" +
            "2,\"a string\", 3\n" +
            "3,\"bar\", 3";

    StringReader stringReader = new StringReader(testString);

    CSVReader csvReader = new CSVReaderBuilder(stringReader).build();

    // read the first two lines to get to the part that csv to bean needs.

    String[] line1 = csvReader.readNext();
    String[] line2 = csvReader.readNext();

    assertEquals(2, line1.length);
    assertEquals(3, line2.length);

    CsvToBean<MockBean> csvToBean = new CsvToBeanBuilder(csvReader)
            .withType(MockBean.class)
            .withSkipLines(1)
            .withIgnoreLeadingWhiteSpace(true)
            .build();

    List<MockBean> rows = csvToBean.parse();
    assertEquals(3, rows.size());

}

Now in this case the withSkipLines does nothing because it is for the CSVReader that the CsvToBeanBuilder would create but since you made one yourself the withSkipLines is effectively ignored.

If this matches what you are trying please redo the test to where you can recreate your issue.

Scott :)

[Scott Conway]

Sorry realized iterator could be the issue so here is another version of the test using an iterator.

@Test
public void verifyBug196() throws IOException {
    String testString =
            "dummy, line\n" +
            "another, dummy, line\n" +
            "name,id,orderNumber\n" +
            "1,\"foo\", 3\n" +
            "2,\"a string\", 3\n" +
            "3,\"bar\", 3";

    StringReader stringReader = new StringReader(testString);

    CSVReader csvReader = new CSVReaderBuilder(stringReader).build();

    // read the first two lines to get to the part that csv to bean needs.

    String[] line1 = csvReader.readNext();
    String[] line2 = csvReader.readNext();

    assertEquals(2, line1.length);
    assertEquals(3, line2.length);

    CsvToBean<MockBean> csvToBean = new CsvToBeanBuilder(csvReader)
            .withType(MockBean.class)
            .withSkipLines(1)
            .withIgnoreLeadingWhiteSpace(true)
            .build();

    Iterator<MockBean> iterator = csvToBean.iterator();
    List<MockBean> beans = new ArrayList<>();

    while (iterator.hasNext()) {
        beans.add(iterator.next());
    }

    assertEquals(3, beans.size());
    assertEquals("1", beans.get(0).getName());
    assertEquals("2", beans.get(1).getName());
    assertEquals("3", beans.get(2).getName());
}

Scott :)

[gunten]

Hi Scott,
my case likes the follow

@Test
public void verifyBug196() throws IOException {
    String testString =
            "dummy, line\n" +
            "another, dummy, line\n" +
            "name,id,orderNumber\n";  //no data
            //data-part

    StringReader stringReader = new StringReader(testString);

    CSVReader csvReader = new CSVReaderBuilder(stringReader).build();

    // read the first two lines to get to the part that csv to bean needs.

    String[] line1 = csvReader.readNext();
    String[] line2 = csvReader.readNext();

    assertEquals(2, line1.length);
    assertEquals(3, line2.length);

    CsvToBean<MockBean> csvToBean = new CsvToBeanBuilder(stringReader)//this's the reader i use
            .withType(MockBean.class)
            .withSkipLines(1)
            .withIgnoreLeadingWhiteSpace(true)
            .build();

    Iterator<MockBean> iterator = csvToBean.iterator();
    List<MockBean> beans = new ArrayList<>();

    while (iterator.hasNext()) {
        beans.add(iterator.next());
    }

    assertEquals(3, beans.size());
    assertEquals("1", beans.get(0).getName());
    assertEquals("2", beans.get(1).getName());
    assertEquals("3", beans.get(2).getName());
}

@Data
public class MockBean implements Serializable {

    private static final long serialVersionUID = 1L;

    @CsvBindByPosition(position = 1)
    private String name;

    @CsvBindByPosition(position = 0)
    private Integer id;

    @CsvBindByPosition(position = 2)
    private Integer orderNumber;
}

the "Error capturing CSV header!" error appears.

In my project ,i use BufferedReader instead of StringReader, when the "data-part" has lines,code's woking well , also withSkipLines works. It fails only when the "data-part" hsa no data. So i catch the exception to judge whether the "data-part" is empty in production env.

But in this testcase, i meet a new problem , whether the "data-part" has data or not, "capturing CSV header!" appears.

....a little confused ,seeking for help
: -)

[Scott Conway]

Not that it makes any difference but is the column positions out of order from the headers by design?

Does the test above recreate the error? Hopefully I will be able to try it tonight after work. Because you can create a BufferedReader from a StringReader and I will try that as well.

http://www.java2s.com/Tutorial/Java/0180__File/CreateBufferedReaderfromStringReader.htm

Scott :)

[Scott Conway]

Okay I was able to recreate your problem but only if there is no data at all.

@Test
public void verifyBug196NoDataAfterHeader() throws IOException {
String testString =
"dummy, line\n" +
"another, dummy, line\n"; // no more data

    StringReader stringReader = new StringReader(testString);

    CSVReader csvReader = new CSVReaderBuilder(stringReader).build();

    // read the first two lines to get to the part that csv to bean needs.

    String[] line1 = csvReader.readNext();
    String[] line2 = csvReader.readNext();

    assertEquals(2, line1.length);
    assertEquals(3, line2.length);

    CsvToBean<CPMockBean> csvToBean = new CsvToBeanBuilder(csvReader)
            .withType(CPMockBean.class)
            .withSkipLines(1)
            .withIgnoreLeadingWhiteSpace(true)
            .build();

    Iterator<CPMockBean> iterator = csvToBean.iterator();
    List<CPMockBean> beans = new ArrayList<>();

    while (iterator.hasNext()) {
        beans.add(iterator.next());
    }

    assertTrue(beans.isEmpty());
}

I did not need a BufferedReader - I was able to duplicate this with just a StringReader.

Now originally I was going to say this is a bug but when I added a null check to prevent the NullPointerException the following test, also in CsvToBeanTest, failed:

@Test(expected = RuntimeException.class)
public void throwRuntimeExceptionIfNoData() {
    List<MinimalCsvBindByPositionBeanForWriting> result =
            new CsvToBeanBuilder<MinimalCsvBindByPositionBeanForWriting>(new StringReader(""))
                    .withType(MinimalCsvBindByPositionBeanForWriting.class)
                    .build()
                    .parse();
}

Looking at the git log the commit had the following comment

 Added test to better call out that an exception was expected from CsvToBean when there is no data/header to process.

So really this is NOT a bug. If you are using CsvToBean you have to have some data in the reader at the moment of its inception. That was the original agreed upon behavior and the test was put in place as a way of documenting it in case I forget (embarrassingly six weeks later :) ).

Scott :)

[gunten]

Haha.. Now i see the comment
Tks a lot
:-)