#247 readNextSilently method of CSVReader not found

[Nelly Barret]

Dear opencsv creators,

We are trying to use opencsv for a project. We are doing the following:

• project connectionlens imports opencsv in the pom.xml and a class CSVDatasource in project connectionlens uses opencsv
• project abstra imports project connectionlens using Maven and calls CSVDatasource using AbstraExperiment.registerDataInCL() function.

Class CSVDatasource works very well when called from project connectionlens. But when used from project abstra we are getting the following error:

Exception in thread "main" java.lang.NoSuchMethodError: 'java.lang.String[] com.opencsv.CSVReader.readNextSilently()'
    at com.opencsv.CSVReaderHeaderAware.initializeHeader(CSVReaderHeaderAware.java:129)
    at com.opencsv.CSVReaderHeaderAware.<init>(CSVReaderHeaderAware.java:34)
    at fr.inria.cedar.connectionlens.source.CSVDataSource.traverseEdges(CSVDataSource.java:74)
    at fr.inria.cedar.connectionlens.ConnectionLens.register(ConnectionLens.java:586)
    at fr.inria.cedar.connectionlens.ConnectionLens.register(ConnectionLens.java:425)
    at fr.inria.cedar.connectionlens.ConnectionLens.completeDataSourceSetRegistration(ConnectionLens.java:296)
    at fr.inria.cedar.abstra.AbstraExperiment.registerDataInCL(AbstraExperiment.java:231)
    at fr.inria.cedar.abstra.AbstraExperiment.run(AbstraExperiment.java:155)
    at fr.inria.cedar.abstra.AbstraExperiment.main(AbstraExperiment.java:188)

Process finished with exit code 1

Could you please help us out in case we are missing something?

Thank you,

[Scott Conway]

So your connectionlens project is working without issue but when it is included into the abstra project you are getting this issue.

Actually I have seen this quite a bit and it goes by several names, typically dependency conflict or "jar hell".

So what is happening is that your connectionlens project is using a version of opencsv that has the readNextSilently method in it - doing a little bit of git forensics (the fact that the call is 129 of CSVReaderHeaderAware) you are using up to version 5.5 of opencsv. But your abstra project has a dependency (direct or transitive) on a much older version of opencsv that does not have readNextSilently and is overriding the dependency from connectionlens thus you are getting the error.

The easiest way to illustrate this is run the following command on both projects.

mvn dependency:tree | grep opencsv

And you will see the version of opencsv that is used by both projects. Like I said my guess is connectionlens will have a much later version than abstra.

Then in the abstra project run the following

mvn dependency:tree -Dverbose

what the verbose command does is give you the FULL dependency tree so not only will you see the version of opencsv that was used but will see the ones that were omitted.

Now if abstra has a direct dependency on opencsv and it is overriding everything you need to update your dependency in abstra to the latest version you see in the verbose dependency tree.

If abstra only has transitive dependencies on opencsv then the first thing I would try is make the project with the largest opencsv version the first dependency in your pom.xml. If that does not work then you need to add opencsv to the dependencyManagement section of your pom file and put at least the latest version there. What the dependencyManagment section does is tell maven "I am not telling you to use X but if you do then you will use version Y" so it does not bring in a dependency but it does dictate the version if it is used. A better explanation can be found at the apache maven site.

Let me know how that works.

Scott :)

[Scott Conway]

K - for giggles I downloaded the astra project. And it was HUGE!!
I am not sure what you are using to build the project as there was no pom file nor did I see a build.xml (ant) or build.gradle (gradle) but I can definitely tell you that you have a jar hell situation.
I extracted the abstra-core-full-1.1-SNAPSHOT-develop-1daa256-20230223-1808.jar and did a grep for opencsv and came across two different directories:

find . -type d -name opencsv
./META-INF/maven/net.sf.opencsv/opencsv
./META-INF/maven/com.opencsv/opencsv
./au/com/bytecode/opencsv
./com/opencsv

com/opencsv is from your current opencsv. the au/com/bytecode/opencsv is from the versions prior to 2014 (3.0 and before). You need to remove this older version of opencsv. Hopefully that is all you need to do. If you have anything that is actually dependent on it you will have to find an updated version of that.

[Nelly Barret]

Dear Scott Conway,

First, thank you very much for you quick and detailled answer, it is very much appreciated!

Yes, Abstra is huge, mainly because ConnectionLens is huge, due to language models... Sorry that you had to go through this!
The Abstra project that you downloaded is the public one, with a stable release: that is why there is no pom.xml, we only provide a jar with all its dependencies for a simpler installation. However, as you can see, it has been lastly updated 5 months ago, we work obviously on a more recent (still private) version :).

The private Abstra has its own pom.xml, as well as ConnectionLens.
I looked at the MVN dependency trees and I found out the following:

In ConnectionLens, I have OpenCSV 5.1:

[INFO] fr.inria.cedar.connection-lens:connection-lens-core:jar:3.4-SNAPSHOT
[INFO] +- com.opencsv:opencsv:jar:5.1:compile

In Abstra, I can see that:

• no openCSV version is manually defined in Abstra's pom.xml
• when Abstra imports CL dependencies, it fetches Open CSV 5.1
• RDFQuotient (another Abstra dependency) uses openCSV 3.2 but this dependency is "omitted for a conflict with 5.1"

[INFO] fr.inria.cedar.abstra:abstra-core:jar:1.4-SNAPSHOT
[INFO] +- fr.inria.cedar:RDFQuotient:jar:2.2:compile
[INFO] |  +- fr.inria.cedar.ontosql:ontosql-rdfdb:jar:1.1.0-SNAPSHOT:compile
[INFO] |  |  +- fr.inria.cedar.ontosql:ontosql-conjunctivequerycardinalityestimator:jar:1.1.0-SNAPSHOT:compile
[INFO] |  |  |  +- fr.inria.cedar.commons:commons-miscellaneous:jar:1.1.0-SNAPSHOT:compile
[INFO] |  |  |  |  \- fr.inria.cedar.ontosql:ontosql-db:jar:1.1.0-SNAPSHOT:compile
[INFO] |  |  |  |     +- com.franz:agraph-java-client:jar:2.2.1:compile
[INFO] |  |  |  |     |  +- org.eclipse.rdf4j:rdf4j-queryresultio-text:jar:2.2.2:compile
[INFO] |  |  |  |     |  |  \- (com.opencsv:opencsv:jar:3.2:compile - omitted for conflict with 5.1)
[INFO] +- fr.inria.cedar.connection-lens:connection-lens-core:jar:3.3-SNAPSHOT:compile
[INFO] |  +- com.opencsv:opencsv:jar:5.1:compile

Also, I uncompresseed our latest full JAR archive, and I see that there are no more /au/com/bytecode/opencsv or net/sf/opencsv references. These libraries have been removed since.

I think the conclusion is that ConnectionLens and Abstra both use openCSV 5.1, and that no spurious openCSV library exists in the maven dependency tree. I will continue to dig in the code and try to fix this hopefully.

Any further idea, if any, would appreciated.
Thank you again for your time and help! :)
Nelly

[Scott Conway]

If there is no net.sf.opencsv or au.com.bytecode references then there should not be a jar hell situation.

please run the following command on the astra project

mvn dependency:tree -Dverbose > somefile.txt

and attach that file to the ticket.

looking at the uncompressed core jar I do see the net.sf.opencsv

find . -name pom.xml -exec grep -il opencsv {} \;
./META-INF/maven/net.sf.opencsv/opencsv/pom.xml
./META-INF/maven/org.apache.commons/commons-csv/pom.xml
./META-INF/maven/com.opencsv/opencsv/pom.xml
./META-INF/maven/org.datavec/datavec-api/pom.xml
./META-INF/maven/fr.inria.cedar.abstra/abstra-core/pom.xml

and just doing some editing I see that the net.sf.opencsv is used by the datavec-api which is included in the org.deeplearning4j libraries which are included in the abstra-core pom.xml. So if you are still using or.deeplearning4j you still have the net.sf.opencsv.

If you are then sadly I don't know how to help you. I do not know what a good replacement of org.deeplearning4j is and looking at its latest version published in 2022 it is still using the older version of opencsv. If you do not have a replacement for deplearning4j my suggestion is to research and contact whoever is maintaining that project and ask that they update their opencsv to com.opencsv. No earlier than 3.10 though as there were issues with the java 7 update in some of the early 3.x versions but preferably in the 5.x range.

[Scott Conway]

I reached out to them yesterday and they sent me a link to their github this morning so I created a request there.

https://github.com/deeplearning4j/deeplearning4j/issues/10048

[Nelly Barret]

Hi again @sconway,

Sorry for the delay!

I have seen the ticket that you opened for DL4J, thank you!

Our full deployment pipeline is a bit more complex because we have four projects, which build on (-->) top of the previous one:

ConnectionStudio --> Pathways --> Abstra --> ConnectionLens

After looking at our recent Abstra POM, we do not use either DL4J anymore, we replaced it by a much more compact library, mainly because we couldn't deploy our JAR anymore due tu DL4J size.

Apart from that, I started again from an "old" ConnectionLens branch, before the problem with openCSV happened. I brought back manually the (same) changes carefully. Now, everything seems to work in ConnectionLens and Abstra, but still not in the final product Studio (I have the same error as I had in Abstra three days ago). I will investigate whether Pathways has some "hell dependencies" that could be in conflict with openCSV. I will inspect all the four POM dependency trees as you suggested.

I will not take more of your time. You have been of great help! I will continue to debug on my own.

Have a great day.

[Scott Conway]

closed - not a bug per se but a jar hell scenario from a refactor we made a decade ago where the package locations were changed causing two CSVReader classes to exist for a user using a library with a dependency on the old version of opencsv.