#129 expand CsvDataTypeMismatchException to include the field name failed on

[Andrew M]

With a bean having many of the same type fields it's difficult to immediately know which one caused the exception. What do you think about expanding CsvDataTypeMismatchException to include the field name?

Exception in thread "pool-171-thread-7" java.lang.RuntimeException: com.opencsv.exceptions.CsvDataTypeMismatchException: Conversion of 10 Nov 2047 to java.time.LocalDate failed.
        at com.opencsv.bean.util.OpencsvUtils.handleException(OpencsvUtils.java:122)
        at com.opencsv.bean.concurrent.ProcessCsvLine.run(ProcessCsvLine.java:108)
ex=java.lang.RuntimeException: Error parsing CSV line: 3. [<snip>]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: com.opencsv.exceptions.CsvDataTypeMismatchException: Conversion of 10 Nov 2047 to java.time.LocalDate failed.
        at com.opencsv.bean.ConverterPrimitiveTypes.convertToRead(ConverterPrimitiveTypes.java:130)
        at com.opencsv.bean.BeanFieldSingleValue.convert(BeanFieldSingleValue.java:99)
        at com.opencsv.bean.AbstractBeanField.setFieldValue(AbstractBeanField.java:180)
2020-05-05 18:14:04.467  INFO 33085 --- [           main] c.c.bondloader.parsers.BondScreener      : parse /public/data/bond_screener2/bond_screener_20200402.csv
        at com.opencsv.bean.AbstractMappingStrategy.setFieldValue(AbstractMappingStrategy.java:581)
        at com.opencsv.bean.AbstractMappingStrategy.populateNewBean(AbstractMappingStrategy.java:328)
        at com.opencsv.bean.concurrent.ProcessCsvLine.processLine(ProcessCsvLine.java:135)
        at com.opencsv.bean.concurrent.ProcessCsvLine.run(ProcessCsvLine.java:87)
        ... 3 more
Caused by: org.apache.commons.beanutils.ConversionException: Can't convert value '10 Nov 2047' to type class java.time.LocalDate
        at org.apache.commons.beanutils.converters.AbstractConverter.conversionException(AbstractConverter.java:474)
        at org.apache.commons.beanutils.converters.StringConverter.convertToType(StringConverter.java:96)
        at org.apache.commons.beanutils.converters.AbstractConverter.convert(AbstractConverter.java:169)
        at org.apache.commons.beanutils.converters.ConverterFacade.convert(ConverterFacade.java:61)
        at org.apache.commons.beanutils.ConvertUtilsBean.convert(ConvertUtilsBean.java:491)
        at com.opencsv.bean.ConverterPrimitiveTypes.convertToRead(ConverterPrimitiveTypes.java:119)
        ... 9 more

[Scott Conway]

At first I was going to say yes with the caveat that you understand that the field name you would see would be the name of the varable in the object NOT the name of the column in the file. When I thought about that afterwards it threw up a red flag but I will get to that in a bit.

The first thing I did though was find a unit test that tested result of CsvDataTypeMismatchException just to get a feel for what is going on - CollectionSplitTest::testUnknownElementType if you want to look at it. And it was there I realized that for exceptions thrown from the converters we cannot do this without alot of work because we do not pass the field information down that low of a level. The only field information we pass into the converter is the type.

But going back to my red flag. On cursory glance at your stack trace you know the line the error happened on (line 3) and the value of the individual variable (10 Nov 2047) and the type (LocalDate) for a customer that is enough to either correct the data (not in this case because your text is clearly a date) or give a report to you because you know the object and how the conversion works if it was a custom conversion. To hand out a field name that may not match their csv file will cause them more confusion.

So no I do not think it is worth the effort to put the name of the bean field into the exception.

Okay so that's out of the way lets work on the real problem. And that is this exception you are getting, unless you created it on purpose just to show the exception message.... but I will ASSume that is not the case <bg></bg>

Looking at the stacktrace the exception stems from the fact that the apache commons beanutils library we are using, the latest version in fact, does not know how to convert to a java.time.LocalDate. And looking at the api for bean utils shows it was written for Java 5 and above so it does not support data types introduced after java 5. Hence the exception.
http://commons.apache.org/proper/commons-beanutils/

http://commons.apache.org/proper/commons-beanutils/apidocs/org/apache/commons/beanutils/ConvertUtilsBean.html

My suggestion to get around this I would suggest using the annotations that @aruckerjones added in opencsv 5.0.

http://opencsv.sourceforge.net/#annotations_2

Namely the @CSVDate as you can see in this simple example from the asciidocs

     public class Employees {

     @CsvBindByName(required = true)
     private String username;

     @CsvBindByName(column = "valid since")
     @CsvDate("dd.MM.yyyy")
     private Date validSince;

     @CsvBindByName(column = "annual salary", locale = "de-DE")
     @CsvNumber("#.###¤")
     private int salary;

     // Getters and setters go here.
     }

Or a more wicked example from the AnnotatedMockBeanTemporal class used in some of the JUnit tests

    /**

     * Field for {@link java.time.temporal.TemporalAccessor}-based annotation
     * tests.
     * <p>Used for the following test cases, reading:<ul>
     * <li>{@link TemporalTest#testReadGoodDataByName()}</li>
     * <li>{@link TemporalTest#testReadGoodDataByPosition()} ()}</li>
     * </ul></p>
     * <p>Used for the following test cases, writing:<ul>
     * <li>{@link TemporalTest#testWriteGoodDataByName()}</li>
     * <li>{@link TemporalTest#testWriteGoodDataByPosition()}</li>
     * </ul></p>
     */
    @CsvBindByName(locale = "de")
    @CsvBindByPosition(position = 5, locale = "de")
    @CsvDate(
            value = "G yyyy MMMM dd HH mm ss z",
            writeFormatEqualsReadFormat = false,
            writeFormat = "G yyyy MMMM dd")
    private LocalDate localDateLocale;

Hope that helps.

Scott :)

[Andrew Rucker Jones]

Scott's suggestion is not just a suggestion—it's the only way to perform date conversions in opencsv, and the annotation CsvDate has been there since version 3.8, not 5.0. Check the documentation for more details if you are in doubt.

As to the original request behind this ticket, I concur with Scott. The information in the exception should be enough in almost all cases unless you have the same data in two different fields on the same line.

[Andrew M]

In my case I had just forgotten to annotate one of my LocalDates with @CsvDate. Until I wrote this original post I'd actually overlooked the row number being provided in the error. Having that helps a lot. As long as your values are unique you can search for that value in the input data on that row, figure out which column it's in (count the commas?) and then look for that colum in the code. thanks.