#151 Fields containing commas in CSV Download format files

Defect
closed-fixed
9
2014-07-17
2010-02-16
David Benn
No

For CSV AAVSO download format files, if the comment field (or any other text field) contains commas, this will cause VStar (or any other program) to misinterpret subsequent fields. In VStar this sometimes manifests as invalid observations with more than the max number of expected fields.

One way to handle this is to wrap each field in quotes. This is an option provided by spreadsheet apps like Excel.

Assigning to you Sara to raise with Doc et al since obviously this affects other consumers of AAVSO download CSV files.

For much more detail, see:

http://www.aavso.org/comma-separated-values-and-aavso-data-download-web-form

Although this is an annoying problem, it can be worked around by requesting TSV instead of CSV (assuming there are no tabs in comment fields etc, otherwise you have the same problem, different delimiter).

Discussion

  • David Benn

    David Benn - 2010-02-27
    • summary: AAVSO Download format files as CSV and comment field --> Comment fields and AAVSO Download format files as CSV
     
  • Sara Beck

    Sara Beck - 2010-03-01

    After doing a little research, I discovered that the AAVSO download program makes an effort to avoid this problem by changing the commas to blank spaces in many of the fields. However, it does NOT do this for the commentcode field, and there are about 1300 records in the database which currently do carry commas in this field. The solution to this problem is to change the download program so that it will change commas to blank spaces for ALL of the fields in the download. I have contacted Doc about this and will follow up with his response.

     
  • David Benn

    David Benn - 2010-03-05
    • priority: 3 --> 9
     
  • David Benn

    David Benn - 2010-05-14
    • summary: Comment fields and AAVSO Download format files as CSV --> Fields containing commas in CSV Download format files
     
  • David Benn

    David Benn - 2011-01-27

    Has there been any resolution re: this? Does this really now need to be an AAVSO tracker and closed here?

     
  • Sara Beck

    Sara Beck - 2011-01-28

    Seems like the problem still exists with AID download data in the comment field still containing commas. I have contacted Doc about fixing the data download utility to replace them with blank spaces. There is nothing wrong with the way VStar works.

     
  • Sara Beck

    Sara Beck - 2011-01-28
    • status: open --> closed-invalid
     
  • David Benn

    David Benn - 2011-06-23
    • status: closed-invalid --> open
     
  • David Benn

    David Benn - 2013-03-29
    • Description has changed:

    Diff:

    --- old
    +++ new
    @@ -3,3 +3,9 @@
     One way to handle this is to wrap each field in quotes. This is an option provided by spreadsheet apps like Excel.
    
     Assigning to you Sara to raise with Doc et al since obviously this affects other consumers of AAVSO download CSV files.
    +
    +For much more detail, see:
    +
    +  http://www.aavso.org/comma-separated-values-and-aavso-data-download-web-form
    +
    +Although this is an annoying problem, it can be worked around by requesting TSV instead of CSV (assuming there are no tabs in comment fields etc, otherwise you have the same problem, different delimiter).
    
    • priority: 9 --> 8
     
  • David Benn

    David Benn - 2013-03-29
    • priority: 8 --> 6
     
  • David Benn

    David Benn - 2014-05-21

    Progress! At some point since the last update in 2013, fields containing the delimiter are now quoted. I suspect that headers were added to the generated download files around the same time (which VStar handles). For example:

    2456362.04142,8.2,,,Vis.,BDJB,BU,82,76,10 star,"""8\"" SCT, 24.5mm eyepiece""",,,Z,,,,R CAR,,STD,,,

    Now I need to update VStar accordingly.

     
  • David Benn

    David Benn - 2014-05-21
    • labels: Suggestion Box --> Observation Sources
    • Priority: 6 --> 9
     
  • David Benn

    David Benn - 2014-07-17
    • status: open --> closed-fixed
     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks