#151 Fields containing commas in CSV Download format files

Defect
closed-fixed
Sara Beck
9
2014-07-17
2010-02-16
David Benn
No

For CSV AAVSO download format files, if the comment field (or any other text field) contains commas, this will cause VStar (or any other program) to misinterpret subsequent fields. In VStar this sometimes manifests as invalid observations with more than the max number of expected fields.

One way to handle this is to wrap each field in quotes. This is an option provided by spreadsheet apps like Excel.

Assigning to you Sara to raise with Doc et al since obviously this affects other consumers of AAVSO download CSV files.

For much more detail, see:

http://www.aavso.org/comma-separated-values-and-aavso-data-download-web-form

Although this is an annoying problem, it can be worked around by requesting TSV instead of CSV (assuming there are no tabs in comment fields etc, otherwise you have the same problem, different delimiter).

Discussion

1 2 > >> (Page 1 of 2)
  • David Benn
    David Benn
    2010-02-27

    • summary: AAVSO Download format files as CSV and comment field --> Comment fields and AAVSO Download format files as CSV
     
  • Sara Beck
    Sara Beck
    2010-03-01

    After doing a little research, I discovered that the AAVSO download program makes an effort to avoid this problem by changing the commas to blank spaces in many of the fields. However, it does NOT do this for the commentcode field, and there are about 1300 records in the database which currently do carry commas in this field. The solution to this problem is to change the download program so that it will change commas to blank spaces for ALL of the fields in the download. I have contacted Doc about this and will follow up with his response.

     
  • David Benn
    David Benn
    2010-03-05

    • priority: 3 --> 9
     
  • David Benn
    David Benn
    2010-05-14

    • summary: Comment fields and AAVSO Download format files as CSV --> Fields containing commas in CSV Download format files
     
  • David Benn
    David Benn
    2011-01-27

    Has there been any resolution re: this? Does this really now need to be an AAVSO tracker and closed here?

     
  • Sara Beck
    Sara Beck
    2011-01-28

    Seems like the problem still exists with AID download data in the comment field still containing commas. I have contacted Doc about fixing the data download utility to replace them with blank spaces. There is nothing wrong with the way VStar works.

     
  • Sara Beck
    Sara Beck
    2011-01-28

    • status: open --> closed-invalid
     
  • David Benn
    David Benn
    2011-06-23

    • status: closed-invalid --> open
     
  • David Benn
    David Benn
    2013-03-29

    • Description has changed:

    Diff:

    --- old
    +++ new
    @@ -3,3 +3,9 @@
     One way to handle this is to wrap each field in quotes. This is an option provided by spreadsheet apps like Excel.
    
     Assigning to you Sara to raise with Doc et al since obviously this affects other consumers of AAVSO download CSV files.
    +
    +For much more detail, see:
    +
    +  http://www.aavso.org/comma-separated-values-and-aavso-data-download-web-form
    +
    +Although this is an annoying problem, it can be worked around by requesting TSV instead of CSV (assuming there are no tabs in comment fields etc, otherwise you have the same problem, different delimiter).
    
    • priority: 9 --> 8
     
1 2 > >> (Page 1 of 2)