Recent changes to Home

Discussion for Home page

Norbert Becker — Mon, 23 Nov 2020 11:08:51 -0000

Ok - I found the reason for the behavior and created ticket https://sourceforge.net/p/opencsv/feature-requests/142/

You may think that it makes no sense to do such a coding. But that's how it looked liked before and worked with older versions of OpenCSV.
So for me not really a bug, but some changes in implementation that may force users to adapt there code.

Thanks again for your support!
We are now happy (again) with OpenCSV in version 5.3 ;-).

Discussion for Home page

Scott Conway — Fri, 20 Nov 2020 16:54:44 -0000

Yes please send us a test case! In 5.4 we refactored the CsvToBeanBuilder withThrowsException method to essentially call the setThrowException on the CsvToBean in the build method. We do have tests on that and they are passing. So I am curious as to what we missed.

Discussion for Home page

Andrew Rucker Jones — Thu, 19 Nov 2020 18:05:14 -0000

I'm very bothered by your claim that exceptions are not being collected and reported with setThrowExceptions(false) and getCapturedExceptions(). Could you possibly open a ticket and provide us with a verifyable test case?

Discussion for Home page

Norbert Becker — Thu, 19 Nov 2020 10:43:45 -0000

Hi Andrew & Scott,
thanks for the swift reply!

Regarding syntax handling aligned to RFC4180: Ok - that's understandable and acceptable :-). The main problem had been that I got no error and the lines where just skipped. I checked our error handling: We use CsvToBean.setThrowExceptions(false) and retrieve the list of exceptions by CsvToBean.getCapturedExceptions() after parsing has finished. This list is still empty?! I changed the code and now use CsvToBean.setErrorHandler() to collect the exceptions. Now it works and I get an exception for each corrupted line ;-).
So maybe a bug regarding the compat code for collecting exceptions? The JavaDoc recomments the use of an exception handler anyway and I leave it up to you to fix it or not. We no longer need the flag/getter...

Regarding usage of setter: You are right - setters are used if they exist. But the finding out whether a setter exist is done by the field name: myProperty -> setMyProperty().
Our "mistake" has been that we use different names which is indeed not standard for Java Beans: property/setMyProperty(). In Version 3.8 this works if you use myProperty as column name. Only to let you know ... we adjusted this codes.

So to wrap it up: The problems where solved.
Thanks for your support!

Discussion for Home page

Scott Conway — Wed, 18 Nov 2020 21:02:35 -0000

Hello

Sorry but Andrew is actually being nice. You can blame me for this and I will gladly take it.

It was not just #155 but we had several bugs in a short period of time where malformed (missing or extra column) data was causing parse errors. Some benign like what you actually want (null or empty field) but some were very malicious as a missing column caused all the data to be shifted and opencsv was trying to figure out how to convert "some string" into an Integer or Date.

And the last part was the problem. We can ASSume that the data was not malformed and that missing columns were the last columns intentionally removed but that causes issues when it IS malformed and could potentially make it harder to debug - especially when everything is a String.
So when dealing with this we went back to the RFC4180 specification on CSV files:

*4. Within the header and each record, there may be one or more
fields, separated by commas. Each line should contain the same
number of fields throughout the file. Spaces are considered part
of a field and should not be ignored. The last field in the
record must not be followed by a comma. For example:

aaa,bbb,ccc*

This is in section 2 of the document: https://tools.ietf.org/html/rfc4180

So what we do parse the first record to get the "required" number of fields and any record after that must have that number or it is in error.

Honestly I am surprised no one has taken us to task on the second part of number four (last field must not be followed by a comma). But if no one is complaining then I am okay with our current behavior (treat it as null or empty string depending on settings).

Sorry, I know that was not what you wanted to hear. But there was alot of history behind making that decision and wanted you to know it was not made lightly because we really do take backwards compatibility seriously and only break it when we feel it is absolutely necessary. And it this case it was as it we needed to follow the standards to stop all the bugs we were getting from malformed csv files.

Discussion for Home page

Scott Conway — Wed, 18 Nov 2020 21:00:44 -0000

Hello

Sorry but Andrew is actually being nice. You can blame me for this and I will gladly take it.

Within the header and each record, there may be one or more
fields, separated by commas. Each line should contain the same
number of fields throughout the file. Spaces are considered part
of a field and should not be ignored. The last field in the
record must not be followed by a comma. For example:

aaa,bbb,ccc

This is in section 2 of the document: https://tools.ietf.org/html/rfc4180

So what we do parse the first record to get the "required" number of fields and any record after that must have that number or it is in error.

Discussion for Home page

Andrew Rucker Jones — Wed, 18 Nov 2020 19:40:14 -0000

Would have prefered to see a ticket for this, but whatever. :)

Lazy separators: I'm surprised this ever worked. It looks like maybe it was bug #155 that fixed this. I doubt we would be willing to support this, since that kind of input is simply malformed.

Using setters: I'm surprised to hear you say this. The code for assigning values to fields is consistent in opencsv, no matter what binding method is used. (That was the reason we got rid of Introspection—half the code used it, half didn't.) opencsv always uses a setter if it's available. That said, the setter is not a setter in the true bean/Introspection sense. The bean definition allows one to choose any name one wants for setters as long as a property file exists to define the mapping from setter to field. (At least that's what I remember reading.) opencsv does nothing more than honor the convention that the setter for "myProperty" is called "setMyProperty".

Discussion for Home page

Norbert Becker — Wed, 18 Nov 2020 18:43:31 -0000

Hi,
we have been using OpenCSV Version 3.8 for a longer time.
Now we migrated our code to Version 5.3 and have (by now :-) two issues:

First issue:
We have an input file like this:

"A-column","B-column"
1
2

The bean is mapped to "A-column" and "B-column" by HeaderColumnNameTranslateMappingStrategy. The B-column value is optional.
The effect in 5.3 is that NO line is parsed: We got an emty list - no error is thrown
Changing it to ...

"A-column","B-column"
1,
2,

... makes it work. But the problem is that users of our applications use such files.
Questions:
Is there a may to make this "lazy" separator handling work again?
Is it a bug that no error is raised for those input lines?

Second issue:
We do not use CsvBindByName annotations. No we found out that OpenCSV 5.3 only uses the fields for setting the value. The former Version 3.8 did also use setter. These is possible the case because the old version used the java.beans.PropertyEditor framework.
Is there a way to use the setter to?

Thanks for a swift answer!

Discussion for Home page

andres reynoso — Sun, 26 Apr 2020 20:22:54 -0000

Ok, going to try that, thank you very much

Discussion for Home page

Andrew Rucker Jones — Sun, 26 Apr 2020 20:21:22 -0000

I'm not sure what you mean. You pass a BeanVerifier to CsvToBeanBuilder.withVerifier() and opencsv uses it internally. You personally never pass anything to a BeanVerifier in your code.

Your code sample has a few problems. First of all, the first line of your verifyBean() method is a no-op. I think you mean to assign rowDto to this.rowDto. That would lead to the next problem, though. As stated in the Javadoc, BeanVerifiers must be thread-safe, and that is not thread-safe behavior. I don't know what you're doing in fieldValidOrThrow(), but I would recommend moving that code directly into verifyBean(), where it's supposed to be, and definitely remove the private instance variable rowDto.