#165 RFC4180Parser reads additional line when last line value is line break only

[srzc]

Hi OpenCSV team,

we are trying to read the following CSV written accordingly to RFC4180 (at least from our point of view :)
We use OpenCSV 4.1.

"value 1.1","
"
"value 2.1","value 2.2"

The following JUnit-Test shows this as an example

    @Test
    public void testOpenCSVWithLineBreak000() throws IOException {
        List<String[]> lines = new ArrayList<String[]>();

        lines.add(new String[] {"value 1.1", "\n"});
        lines.add(new String[] {"value 2.1", "value 2.2"});

        File file = new File("opencsv_test.txt");

        CSVWriter csvWriter = new CSVWriter(new FileWriter(file));
        for (String[] strings : lines) {
            csvWriter.writeNext(strings);
        }
        csvWriter.close();

        CSVReader csvReader = new CSVReaderBuilder(new FileReader(file)).withCSVParser(new RFC4180Parser()).build();
        String[] readLine = null;

        while ((readLine = csvReader.readNext()) != null) {
            System.out.println("#####################################");
            for (String string : readLine) {
                System.out.println("read [" + string + "]");
            }
        }
        csvReader.close();
    }

This prints:

#####################################
read [value 1.1]
read []
#####################################
read []
#####################################
read [value 2.1]
read [value 2.2]

From my point of view it should print (as OpenCSV 1.8 did):

#####################################
read [value 1.1]
read []
#####################################
read [value 2.1]
read [value 2.2]

When modifying the first line in the test code to:

        lines.add(new String[] {"value 1.1", "before\nafter"});

It prints (correctly):

#####################################
read [value 1.1]
read [before
after]
#####################################
read [value 2.1]
read [value 2.2]

I am currently struggeling how to parse this file. Maybe this is just a configuration issue of the parser? Any help is appreciated.

Best wishes
Stefan

[srzc]

Hi!

I did some research in the OpenCSV 4.1 file RFC4180Parser.

From my point of view the following method fails when the line ends with a doubleqoute because then the first AND the last character is a doubleqoute.

    private boolean lastElementStartedWithQuoteButDidNotEndInOne(List<String> elements) {
        String lastElement = elements.get(elements.size() - 1);
        return lastElement.startsWith(Character.toString(quotechar)) && !lastElement.endsWith(Character.toString(quotechar));
    }

I modified the method the following way:

    // assume that a non equal number of quotechars in the last element means an unclosed value which continues next line
    private boolean lastElementStartedWithQuoteButDidNotEndInOne(List<String> elements) {
        String lastElement = elements.get(elements.size() - 1);
        int quotecharCount = lastElement.length() - lastElement.replace(Character.toString(quotechar), "").length();
        return quotecharCount % 2 != 0;
    }

With this modification my test works as expected. I extended my test and added the code. Will do some more tests next week.
Looking forward for feedback - maybe this modification breaks some other OpenCSV JUnit test.

Best wishes
Stefan

[srzc]

Hi!

I did some more testing. Found no issues with the suggested change (until now).

Best wishes
Stefan

[Scott Conway]

Hello Stefan. I looked at your solution and it has given me doubts to my solution. Namely if you have a field with embedded quotes and that has a new line for some reason (I.E. "I talked with Stefan and he asked ""\nWhat about odd number of quotes?"" and now I have doubts about my solution") . I will add a unit test with the above and see what comes out.

Thanks for your testing and suggestion!

Scott :)

[Daniel Fedak]

Hi

... Sorry ... your original test case worked... still trying to figure out a test case that will prove the problem I'm having...

Dan

[Daniel Fedak]

Just to share... still can't get a 'fail' test for the data. Java class and dummy test CSV attached.

[Daniel Fedak]

Here is a final version that works in all cases for our CSV imports with many 'strange' scenarios:

"field1',"field 2","

field 3

""moretext"""
---
"1","2","3""
sometext"

[srzc]

Hi Daniel,

thanks for posting your RFC4180Parser.java. It fails for the test cases "testOpenCSVWithLineBreak003", "testOpenCSVWithLineBreak004" and "testOpenCSVWithLineBreak005" I posted in OpenCSVTest.java on 19/Jan/18.

Could you post your test cases as JUnit tests? I would like to run them with the RFC4180Parser method "lastElementStartedWithQuoteButDidNotEndInOne(...)" I postet on 19/Jan/18.

Thank you
Stefan

[Scott Conway]

Hello Daniel. Sorry it took so long to get back to the project but I added an addtional test and it the solution I came up with is still passing all the tests. You can see the RFC4180ParserSpec for all the test but here is the one that I specifically added.

def 'bug 165 - No character line showing up as an extra record with CSVParser'() {
    given:
    List<String[]> lines = new ArrayList<String[]>();

    lines.add(["value 1.1", "\n"])
    lines.add(["value 2.1", "value 2.2"])
    lines.add(["\"value 3.1\"", "\"I talked with Stefan and he asked \"\"\nWhat about odd number of quotes?\"\" and now I have doubts about my solution\""])

    when:
    StringWriter stringWriter = new StringWriter(128);

    CSVWriter csvWriter = new CSVWriter(stringWriter);
    for (String[] strings : lines) {
        csvWriter.writeNext(strings);
    }
    csvWriter.close();

    StringReader stringReader = new StringReader(stringWriter.toString());

    CSVReader csvReader = new CSVReaderBuilder(stringReader).withCSVParser(new CSVParser()).build();

    List<String[]> readLines = csvReader.readAll();

    csvReader.close()

    then:
    lines == readLines
}

For now I will go with this solution but if you can come up with a test that fails I will be happy to revisit the solution.

Scott :)

[srzc]

Hi Scott,

I ran my tests with the latest version of RFC4180Parser I found in the repository. Three tests failed :( Below the settings that did not work. The complete and running test code can be found in the OpenCsvTest.java I posted in this thread at the 01/19/2018.

Best wishes
Stefan

        List<String[]> lines = new ArrayList<String[]>();
        lines.add(new String[] {"value 1.1", "\"\n\""});
        lines.add(new String[] {"value 2.1", "value 2.2"});

        List<String[]> lines = new ArrayList<String[]>();
        lines.add(new String[] {"value 1.1", "\"\n\"\n\""});
        lines.add(new String[] {"value 2.1", "value 2.2"});

        List<String[]> lines = new ArrayList<String[]>();
        lines.add(new String[] {"value 1.1", "\"\n"});
        lines.add(new String[] {"value 2.1", "value 2.2"});

[Scott Conway]

Ahh I see the differencees in your tests and what I have in the RFC4180ParserSpec. But here is the thing - all three of the examples you are posting above is illegal under the RFC4180 specification. When you have quotes inside a value to be parsed the value must have quotes around it (which you have) and the quote must be escaped by another quote (which you do not have).

The acutal text is

   7.  If double-quotes are used to enclose fields, then a double-quote
   appearing inside a field must be escaped by preceding it with
   another double quote.  For example:

   "aaa","b""bb","ccc"

Which can be found at https://www.ietf.org/rfc/rfc4180.txt

Because of that we cannot parse those examples.

Regards

Scott.

[srzc]

Hi Scott,

thank your for your analysis. Currently I am a little confuesed. Let me explain this by the following example:

I have the following values:

        List<String[]> lines = new ArrayList<String[]>();
        lines.add(new String[] {"value 1.1", "\"\n\""});
        lines.add(new String[] {"value 2.1", "value 2.2"});

I create a CSVWriter with the following command:

        CSVWriter csvWriter = new CSVWriter(new FileWriter(file));

Then I use the CSVWriter to write the lines above in a file. The file content is then:

"value 1.1","""
"""
"value 2.1","value 2.2"

Then I read the file with a CSVReader constructed with this command:

CSVReader csvReader = new CSVReaderBuilder(fileReader).withCSVParser(new RFC4180Parser()).build();

And compare the lines I wrote to the file with the lines I got from the reader. They do not match.

What confuses me is:

(1) The following CSV matches RFC4180 from my point of view. Every value is inclosed in doubleqoutes and doubleqoutes are qouted by doubleqoutes.

"value 1.1","""
"""
"value 2.1","value 2.2"

(2) When writing a file with CSVWriter built like the one above, anything I write with the writer should be readable with the RFC4180Parser.

Best wishes
Stefan

[Scott Conway]

Ahhh - so here is the thing: openCSV actually predates the RFC4180 specification by a year. And even then it was about four or five years before we got the first "your parser does not follow the specification" defect. Eventually we did come up with a test case that we could not pass by modifying the configuration of the CSVParser or Reader and thus came up with the RFC4180Parser. The CSVWriter has its on built in parser and thus is not RFC4180 compliant (having predated it). As it stands it is barely compatible with CSVParser which has lead to several bugs like above where the default CSVWriter has output different than what the default CSVReader is set for.

All that said I hope to correct that in the 4.2 release. If you look at the javadocs you will see in 4.1 we added a parseToLine method in the CSVParser and RFC4180 Parser to basically reverse parse a string array into a csv formatted line based on that parsers settings. In 4.2 we are introducing a CSVParserWriter that allows you to inject parser you wish to use so you can use either the CSVParser or the RFC4180Parser to generate your csv output.

Please wait until 4.2 is released and then retry your test.

Regards

Scott Conway :)

[Andrew Rucker Jones]

Is this problem fixed now that 4.2 has been released?