opencsv Activity
Brought to you by:
aruckerjones,
sconway
Activity for opencsv
-
Scott Conway modified ticket #168
Create Compressed CSV Files
-
There is no need as Java already has built in libraries to read and write gzip files. https://medium.com/javarevisited/efficient-handling-and-processing-of-compressed-files-in-java-7d023551168c Using the GZIPOutputStream - https://docs.oracle.com/javase/8/docs/api/index.html?java/util/zip/ZipOutputStream.html Using an OutputStreamWriter as a wrapper of the GZIPOutputStream you have a writer that you can create a CSVWriter with and you are now creating compressed csv files. https://www.tutorialsp...
-
Prashant created ticket #169
Create Multiple CSV For SQL Result Set
-
To be able to create/write to compressed csv to save disk space and faster file transfer
-
Create Compressed CSV Files
-
Alexey Pelykh created ticket #126
Support Java's record
-
Jeff Thomas posted a comment on ticket #251
Hi Scott, I actually did message Gary a few months ago and ended up on the Apache mailing list. Gary Gregory announced yesterday via the mailing list that they expect to have a beanutils2 release-candidate in around one month.
-
Gordon created ticket #167
Support validation against unknown columns
-
Added additional tests after Merge #36. Corrected issues brought up by mvn site:site build
-
What's new
-
b'Scott Conway committed [0bdd1d] on Source
Merge /u/sdaubin/opencsv/ branch master into master
-
Enhancement - add withType method to mapping strategy builders
-
b'Scott Conway committed [fc1a2f] on Source
Merge /u/cy2008/opencsv/ branch master into master
-
Enhancement
-
czp posted a comment on ticket #166
Thanks for your patience, now I know that 'profile' means maven profile. But that's what I don't understand. opencsv as a dependency library, rather than as a standalone application, only needs a few additional resource files to support graalvm. This does not cause jdk version incompatibility or jdk provider incompatibility. It also doesn't ask for the jdk version at compile time, unlike java9 module-info. The graalvm files (as written above) are just normal resource files that don't affect anything...
-
Hello czp. When I talked about a profile I meant that I was willing to add or allow someone to add a graalvm profile in our maven pom.xml. You can see an exampe of that in the maven section of the graalvm page. The benefit of this is that IF changes required to make the native image are incompatible with our current requirements (like requiring a newer version of Java than Java8) we can loosen those restrictions for the graalvm profile for people who priortize that but leave the defaults alone. If...
-
I'm not sure what you mean by "profile", do you mean https://www.graalvm.org/latest/reference-manual/native-image/guides/optimize-native-executable-with-pgo/ ? PGO is a graalvm native image optimization, and does not critically determine whether or not opencsv will run properly on the native image. "Support for graalvm native image" means that opencsv, as a library, can execute correctly when used as a dependency in a standalone application. I'm sorry, but I've always used github and don't really...
-
I took a look at the graalvm and have no problem creating a profile - though wouldn't it need to be a native shared library as opencsv is not a stand alone application? As long as it is added as a new profile feel free to make the change yourself and submit a merge request as it will be a while before I have free time to take a look at this.
-
Hello Saxon Thanks for the merge request. I will try and get a closer look at it next week as this week is very busy at work and at home we are getting ready for the labor day weekend :) If you get a chance, if not I will do it if everything else looks good, please copy the tests you modified and have one with the setType and the other with the withType to show that the code change maintains backwards compatibility. Sincerely Scott :)
-
Saxon created merge request #36 on Source
Enhancement - add withType method to mapping strategy builders
-
support graalvm native image
-
Go ahead and message Gary Gregory to see when they are planning on a release of commons-beanutils. Yeah they are still making code changes but the last release they made was almost five years ago (https://central.sonatype.com/artifact/commons-beanutils/commons-beanutils/versions) so there are a whole bunch of apache commons libraries that are in what I call "development hell" because changes are constantly being made but no releases are being made.
-
Hi Scott, came here looking for this exact ticket because of the CVE vulnerability lighting up in my IDE. Just a hint, the last code changes in beanutils were just 3 days ago. Maybe it is possible to message the maintainer Gary Gregory about a release? :) The master was already switched to commons-collections4 back in September 2023.
-
Cx78f40514-81ff,
-
Sorry I am closing this one. If you do a mvn dependency:tree on the opencsv project or just look at the pom file you see we are using the latest version (4.4) of common-collections4 mvn dependency:tree [INFO] Scanning for projects... [INFO] [INFO] ------------------------< com.opencsv:opencsv >------------------------- [INFO] Building opencsv 5.9.1-SNAPSHOT [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- dependency:3.6.1:tree (default-cli)...
-
MDR created ticket #125
Cx78f40514-81ff,
-
Yeah I am going to close this one. I agree with you with the sentiment about commons-beanutils being a dead project but there is a story here and I wished I knew what it was. If you check you will find that back in 2017 there was a apache commons-beanutils2 created in 2014 but looking at the git branches the first checkin referencing beanutils2 was 2017 (https://commons.apache.org/sandbox/commons-beanutils2/index.html). And looking at the git repo (https://github.com/apache/commons-beanutils) it...
-
Hristo Ivanov posted a comment on ticket #251
Hi Scott, upon further analysis, it appears that commons-beanutils uses FastHashMap, which transitively imports commons-collections 3.2.2 into OpenCSV. This means any developer using OpenCSV who has excluded older versions of commons-collections in the pom file due to the CVE will likely encounter this error. Unfortunately, there has not been an update to commons-beanutils since 2019, suggesting that the project might have been abandoned. It would be great, if you could take a look into this in your...
-
If you have a simple code sample that you can make into a unit test I will try it out. Otherwise I will test it out in the next week or so depending on when I can get free. Thanks for finding this. I am surprised our existing unit tests did not catch it.
-
OpenCSV 5.9
-
Yat Man, Wong modified a comment on ticket #124
thank you both. The reason I have this use case is so that the user can see all the errors during an import, avoiding the need to fix the errors one by one. I could workaround by reading the CSV rows into a Bean where all properties are string and run my validation logic after. For the case of data miss one semicolon, I see OpenCSV is already throwing "CsvRequiredFieldEmptyException: Number of data fields does not match number." I agree just this error is sufficient to indicate the file is corrupted...
-
thank you both. The reason I have this use case is so that the user can see all the errors during an import, avoiding the need to fix the errors one by one. I could workaround by reading the CSV rows into a Bean where all the are type string and run my validation logic after. For the case of data miss one semicolon, I see OpenCSV is already throwing "CsvRequiredFieldEmptyException: Number of data fields does not match number." I agree just this error is sufficient to indicate the file is corrupted...
-
To expand on the cascading errors problem we have had quite a bit of support requests where data was mis quoted causing shifts in the columns so after the first error you really don't know if anything after that is really an error or if its because you are looking at the wrong data for a given column.
-
Andrew Rucker Jones posted a comment on ticket #124
No, BeanVerifier is not meant to support multiple errors in the same row. As far as the implementation goes, BeanVerifier throws an exception if anything is wrong, which interrupts the normal flow of the program. It would be difficult to make that work for multiple errors in one row. Conceptually, I decided not to provide for the case of multiple errors per row because I figured if there was one error, it might create other errors by its mere existence. I'm thinking of programming in compiled languages...
-
Thank you Scott! I tested moving the verifiers into their own class. The behavior I found is only the exception from the first verifier is returned from getCapturedExceptions(), the latter verifiers are not executed after the first verifier thrown an exception for that row. For example in this change, CountryVerifier was set first so getCapturedExceptions() return the exception from that verifier. https://github.com/yatw/openCsvValidationDemo/commit/657a9121e97b0a48ecad938e978e8ab58d34ef99#diff-b238fe60176ed02237cddabacb5b4ab05b6f7c26ede636fe8550faa77797979e...
-
So if you want to look at a working example download the opencsv code and look at AnnotationTest. Specifically the testMultipleExceptionsPerLine method. Looking at your example, which thank you for loading that to github for me as that made it way easier to see what is going on., the first thing that jumped out at me was that you don't need the .withExceptionHandler(new ExceptionHandlerQueue()) as that is the default that is used when you call the .withThrowExceptions(false). You only need to use...
-
Support: Can beanVerifer validate all errors in a row
-
Print header separately
-
Nope. I went back and checked the code, or rather my javadoc, to refresh my memory. The reason an actual bean is necessary is because of the joining and splitting functionality we added a few years ago. Any fields joined on reading and split on writing are represented as a MultiValuedMap, with the exact column names in the index of the map. These names, as in the documented example, might be "Track1," "Track2," "Track3," and so on. These column names do not exist in the class. They only exist in...
-
Automaticaly sorting columns on HeaderColumnNameMappingStrategy
-
I just had a chance to read this. I had only half-understood it while scanning over the ticket early in the morning when I got up two weeks ago. I agree with Scott; this is a pretty good solution, and I can't think of a better way to do this with the built-in functionality of opencsv if you are determined to keep ordering information solely in the annotations. Well done.
-
Six I apologize for taking so long to get back to this. Work and family has been keeping me busy. Me personally I am going to pass on this for two reasons: The first is I actually like your solution!! I believe it was what Andrew was envisioning when he designed the setColumnOrderOnWrite and seeing what you did made me actually understand what Andrew did. Hopefully between now and our next release (nothing planned) I am going to update the documentation ( https://opencsv.sourceforge.net/#changing_the_write_order)...
-
I believe its a chicken and egg thing. The bean is probably the best way to get all the fields and annotations. Though if we were to ever try a ground up rewrite we might try and use the class first.
-
Six Douglas posted a comment on ticket #164
Hello Scott, In deed, this a possibility but I don't see why an existing bean is required to export the header?
-
Yes that's a possibility and I'm using the setColumnOrderOnWrite method in my sample code above, passing it comparator. But my point is about giving the possibility to use the CsvBindByPosition annotation in conjonction with the HeaderColumnNameMappingStrategy to do the column sorting automatically. In my use case, I'm using CSV files as output to validate the behaviour of my program. Our QA team give us some CSV files, directly extracted from the Database. We are planning to use plain text comparison...
-
Hello Six I don't have as much experience with this one as I have never personally had a use case where order of the output mattered. But that said there already exists an non-annotation way of handling it. There is some crude documentation at https://opencsv.sourceforge.net/#changing_the_write_order The headerColumnNameMappingStrategy setColumnOrderOnWrite method takes a comparator of strings. You can use something like the supplied LiteralComparator and a sample of its usage in ComparatorTest.java....
-
Print header separately
-
Hello Six All mapping strategies have a generateHeader that creates the header as a array of Strings. It can be called at will. If that does not work for you then please explain your use case with some sample code. Scott :)
-
Automaticaly sorting columns on HeaderColumnNameMappingStrategy
-
Print header separately
-
Hello Jeremy - thanks for the offer but so far sourceforge has done everything we needed so there is really no need to move. I actually use github for my personal projects (though nowhere near the number of commits you have) but opencsv started on sourceforge in 2005 and there are millions of bookmarks in browsers in the world for opencsv.sourceforge.net and there is no need to cause such a disruption because a newer, sexier developer platform comes out, because there will always be a newer, sexier...
-
Jeremy Landis posted a comment on ticket #152
Find me at github at https://github.com/hazendaz so you know you have someone with great experience that can help get this over.
-
Scott, please reconsider this. No one uses sourceforge these days that care about open source. Every single thing you have here can move to github. I can help you. I've been on github for 13 years now. There are issues with this library and I can guarantee no one reports them due to it being here. Whatever you need I want to help. Lets get this on github and start releasing more frequently and get rid of the legacy baggage this currently contains. Even if you don't port everything, just port the...
-
Jon Schewe posted a comment on ticket #69
My specific case is that I want to represent NULL with an explicit string value. If a non-null value evaluates to the NULL string value, then prepend the NULL string value. Then when loading the CSV file back if the value read is the NULL string value then insert a NULL into the database. If the value read starts with NULL string value, then remove it and insert into the database. So it's both an export and import. At this point opencsv doesn't do it's own import, so I suspect this isn't something...
-
Sorry I missed your original post. But for the original question - it depends. If it is a single change that if left unset or by default leaves the original functionality unchanged (so by default it does not change established behavior) then I am okay with a simple setter. If it is a very pervasive change that is difficult not to modify existing behavior then extension would be the route to go - which you can see in the ResultSetColumnNameHelperService. Let me know if you need/want to change the...
-
I did some more work on this and realized that the solution I need is likely more specialized than would make sense to add to the library. I believe I can handle this by writing a custom ResultSetHelper class.
-
Patch to support modifying the default value of ResultSetHelperService
-
Sorry for the late response but work was pretty busy this week. So if your change did not break any of the existing tests. And more importantly you have a junit that will break if someone removes your fix, thus showing why the fix is there in the first place, please send it as a merge request or a patch.
-
Jason Nyberg posted a comment on ticket #250
One solution to this dilemma of "to interpret or not to interpret carriage returns" is to use a modified LineReader class that interprets both CR and LF as line endings, but appends them to the resulting string just like other characters. Then, you wouldn't need to interpose newlines when reconstructing multi-line fields, as LineReader has preserved the original line-ending characters. Then, it's just a matter of trimming them strategically in CSVParser and RFC4180Parser. This way you get the best...
-
CsvToBean.stream() does not preserve order of input
-
Okay I see what you are talking about but you are using the wrong file. Doing the diff showed me that while both are ordered the same the but differed in the order of creation. diff output_iterator.csv output_stream.csv 90,91c90,91 < 90;newyawdrive02_13a < 89;newyawdrive02_14a --- > 89;newyawdrive02_13a > 90;newyawdrive02_14a 126,127c126,127 < 126;newyawdrive03_23a < 125;newyawdrive03_24a --- > 125;newyawdrive03_23a > 126;newyawdrive03_24a 201,202c201,202 < 201;newyawdrive06_20a < 200;newyawdrive06_21a...
-
Carriage returns considered part of quoted data
-
So why do you have withKeepCarriageReturn set to true? Your test, and thank you for supplying a test!!, does not show that use case and my answer for this particular test is to set the carriage return to false. I created the following test and it passes: @Test public void bug250CarriageReturnsAtEndOfLine() throws IOException { String input = "\"Line 1\"\r\n\"Line 2\""; CSVReader csvReader = new CSVReaderBuilder(new StringReader(input)) .withCSVParser(new RFC4180Parser()) .withKeepCarriageReturn(false)...
-
Jacob Shields created ticket #250
Carriage returns considered part of quoted data
-
Stefan Meier posted a comment on ticket #249
I can reproduce my problem, but turns out it is a little bit different than I thought. So the order in which the elements are returned is actually the same as in the input. But in my Bean class I have a field called sequenceNumber which is assigned its value in the constructor based on a global counter. import com.opencsv.bean.CsvBindByName; public class CsvSymbol { @CsvBindByName(column = "Designer ID") public String designerId; // this is just a counter not included in the CSV file public static...
-
CsvToBean.stream() does not preserve order of input
-
If possible please send a test sample so we can see what is going on. My original gut feeling was I was not surprised at all because calling the iterator from the CsvToBean returns a CsvToBeanIterator whereas calling the stream method called from the CsvToBean returns a java stream and then you are calling the iterator on the java stream. But digging into the java stream code it calls the spliterator code which is the opencsv LineExecutor. So in theory even though they are different they should be...
-
CsvToBean.stream() does not preserve order of input
-
Of course you can - though I would recommend you extending the CSVParserBuilder as well to have a builder for your own extended class as well. We went the builder route because after several years and many, many modifications later we realized that our classes had 8-9 constructors some with a dozen parameters so we could maintain backwards compatibility while still allowing for the new features being requested. It just got to be too much so we created a Factory/Builder class and never looked back....
-
Zakaria Fadili posted a comment on ticket #93
Thanks for this solution, btw can we extend CSVParser and override this method like you did while changing some fields like escape char? Fields are final and only builder can set them. Thanks!
-
What's new
-
ConverterPrimitiveTypes calls 'new ConvertUtilsBean()' over and over
-
yes - my apologies but aparently I did not update the wiki. I missed several of those in the 5.8 release for some reason. After ensuring that the fix is in by looking at the git history I did update the wiki.
-
What's new
-
Operational risk in opencsv
-
K - so I looked up the issues in the 4.4 version of commons-collections4 - https://mvnrepository.com/artifact/org.apache.commons/commons-collections4/4.4 The issue is from an older version of juni4 (4.12) which is a test scope dependency so it is not compiled into the system. PLUS the website noted that the issue was fixed in 4.13.1. Because of our use of the junit5-vintage-engine we are pulling in a newer version of junit4. mvn dependency:tree | grep junit [INFO] +- org.junit.jupiter:junit-jupiter-api:jar:5.10.1:test...
-
CSVReader readAll method exist attack risks
-
closed for lack of response.
-
readNextSilently method of CSVReader not found
-
closed - not a bug per se but a jar hell scenario from a refactor we made a decade ago where the package locations were changed causing two CSVReader classes to exist for a user using a library with a dependency on the old version of opencsv.
-
Unexpected behavior in OpenCSV when parsing CSV file with backslash characters
-
close for lack of response.
-
Kevin S posted a comment on ticket #163
Hi Scott, Thanks very much for responding and thanks for the suggestions.
-
So on the first problem are you seeing a comma at the end of every row or the end of every line? That is important for two reasons - the first is that csv files can have data that has multiple new line characters. And second opencsv uses the java Reader class which reads the file one line at a time to create a row. If your data has no newlines in it so each line in the file is exactly one row of data or you have a comma at the end of each line then you are good. In this case I would recommend you...
-
Allow header transformation in CsvReaderHeaderAware
-
Youssef El Houti created ticket #248
Unable to set locale dynamically
-
Nelly Barret posted a comment on ticket #247
Hi again @sconway, Sorry for the delay! I have seen the ticket that you opened for DL4J, thank you! Our full deployment pipeline is a bit more complex because we have four projects, which build on (-->) top of the previous one: ConnectionStudio --> Pathways --> Abstra --> ConnectionLens After looking at our recent Abstra POM, we do not use either DL4J anymore, we replaced it by a much more compact library, mainly because we couldn't deploy our JAR anymore due tu DL4J size. Apart from that, I started...
-
I reached out to them yesterday and they sent me a link to their github this morning so I created a request there. https://github.com/deeplearning4j/deeplearning4j/issues/10048
-
If there is no net.sf.opencsv or au.com.bytecode references then there should not be a jar hell situation. please run the following command on the astra project mvn dependency:tree -Dverbose > somefile.txt and attach that file to the ticket. looking at the uncompressed core jar I do see the net.sf.opencsv find . -name pom.xml -exec grep -il opencsv {} \; ./META-INF/maven/net.sf.opencsv/opencsv/pom.xml ./META-INF/maven/org.apache.commons/commons-csv/pom.xml ./META-INF/maven/com.opencsv/opencsv/pom.xml...
-
Dear Scott Conway, First, thank you very much for you quick and detailled answer, it is very much appreciated! Yes, Abstra is huge, mainly because ConnectionLens is huge, due to language models... Sorry that you had to go through this! The Abstra project that you downloaded is the public one, with a stable release: that is why there is no pom.xml, we only provide a jar with all its dependencies for a simpler installation. However, as you can see, it has been lastly updated 5 months ago, we work obviously...
-
K - for giggles I downloaded the astra project. And it was HUGE!! I am not sure what you are using to build the project as there was no pom file nor did I see a build.xml (ant) or build.gradle (gradle) but I can definitely tell you that you have a jar hell situation. I extracted the abstra-core-full-1.1-SNAPSHOT-develop-1daa256-20230223-1808.jar and did a grep for opencsv and came across two different directories: find . -type d -name opencsv ./META-INF/maven/net.sf.opencsv/opencsv ./META-INF/maven/com.opencsv/opencsv...
-
readNextSilently method of CSVReader not found
-
So your connectionlens project is working without issue but when it is included into the abstra project you are getting this issue. Actually I have seen this quite a bit and it goes by several names, typically dependency conflict or "jar hell". So what is happening is that your connectionlens project is using a version of opencsv that has the readNextSilently method in it - doing a little bit of git forensics (the fact that the call is 129 of CSVReaderHeaderAware) you are using up to version 5.5...
-
readNextSilently method of CSVReader not found
-
James Saffery posted a comment on ticket #120
Awesome, thank you. Works perfectly.
-
CsvNumber - rounding mode
-
Version 5.9 has been released with this feature.
-
how to transform data when the column header in csv file is not the same as the pojo
-
Spaces in Header and CsvBindByName.required
-
I don't know what changed today other than the fact that I was taking screen shots of everything so I could file a Jira with sonatype but it worked and 5.9 has been released. :)
1 >