#74 vstar does not support space delimited files

Defect
closed-fixed
mikeu
1
2009-12-20
2009-11-21
mikeu
No

Download all data for S Ori from the aavso website choosing "Space Delimited (not column-delimited)" format. Opening the file in vstar gives "...error in observation source."

Personally, I find the thought of space delimited data to be abhorrent. But if this is a supported format to download from the aavso website, then vstar should open it. I had no problem opening csv or tsv formated data files for the same star.

Discussion

  • mikeu

    mikeu - 2009-11-21
    • priority: 5 --> 1
     
  • David Benn

    David Benn - 2009-11-21

    Hi Mike. It's interesting to read the current spec, discussion, and requirements for input file support in VStar, which along with field validation and creating a generic framework to be able to handle database access and file input sources uniformly within VStar, consumed a lot of my time early on. See these pages:

    https://sourceforge.net/apps/mediawiki/vstar/index.php?title=Input
    https://sourceforge.net/apps/mediawiki/vstar/index.php?title=Talk:Input
    https://sourceforge.net/apps/mediawiki/vstar/index.php?title=Phase_I_VStar_Requirements

    The summary is that only CSV and TSV are supported for AAVSO download files (I've gone further than the spec and requirements by supporting both because once you've handled one, the other is trivial), but not space delimited files. I started out trying to support that too but gave up after awhile. For simple format files, space delimiters are doable (and in the spec), but for AAVSO download format, space delimited files are human readable but not really machine readable. Due to the optionality of some fields and the free-form text allowed in comments fields, not only would a context-free grammar be impossible, a context-sensitive parse of each line of such a file would be very challenging, to say the least.

    So, although I have not taken a close look at examples of such space-delimited (e.g. The same as TSV and CSV but just a single space, or is it multiple spaces per delimiter, in which case how to handle optional fields? Are comment fields quoted?) downloaded files, I do not believe we need to concern ourselves with this format. If you think we should pursue this with AAVSO, then let's keep this issue open until resolved.

    However, this tracker has raised two issues:

    1. Is the Help documentation clear about this?
    2. Does the simple file format (as currently implemented) permit space delimiters?

    The answer to 1. is almost certainly "no". Do you want to have a go at this?

    The answer to 2. is no. Simple format also requires TSV and CSV. That's interesting. The requirements I derived from the spec say that simple format file rows will be TSV or CSV, but the spec and discussion pages suggest that space delimiters should permitted. Even here however, any of the last 3 (of 5) fields may be optional. Sorting out space delimited rows when parsing each simple format file line is no big drama, but arguably unnecessary. Again, we can raise this with Sara et al again. Given that this simple file format is intended for use when creating one's own observation files, or perhaps later, importing data from other sources (e.g. Photometrica), we should be able to define what this looks like. So, I would be arguing for TSV and CSV even here, and that is indeed what the format requirement states. There were individual, ad hoc informal reviews of the phase I requirements I derived but nothing formal. I recall some discussions about this, but I think this may have fallen through the cracks.

     
  • mikeu

    mikeu - 2009-11-21

    I agree that space delimited is not something we want to support and that this is a documentation issue. I'll assign this to myself. I've also modified the summary of this tracker to reflect this.

     
  • mikeu

    mikeu - 2009-11-21
    • labels: 1207008 --> Documentation
    • assigned_to: david_benn --> mu301
    • summary: vstar gives error opening space delimited files --> vstar does not support space delimited files
     
  • mikeu

    mikeu - 2009-12-10
    • status: open --> open-fixed
     
  • mikeu

    mikeu - 2009-12-10

    There is a note in help about space delimited file being unsupported in svn 298

    We might want to gather some information about what formats are produced by default, or optionally, using various software that reduces images. For example MaximDL or other packages that are commonly used. However, csv and tsv are by far the most common formats for data such as this and are handled by all spreadsheet software and other common tools.

     
  • mikeu

    mikeu - 2009-12-20
    • status: open-fixed --> closed-fixed
     

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks