Menu
▾
▴
#557 Reloading Data Files Saved from VStar
From VStar CHOICE VI (https://www.aavso.org/content/8-bug-reports-and-feature-suggestions)
"If you import a data file into VStar and save it from the observations tab, the saved files of most file formats cannot be reloaded. Files downloaded from AAVSO database can be reloaded as can simple files. This seems to be simply because the data VStar needs to recognize the file as a given format is not saved with it and/or files with certain file formats are only downloadable from a web location not a file stored locally. The only files that I have been able to save and re-load are files downloaded from AID and Simple files. I have not been able to re-load files from any other source including AAVSOnet and APASS that have been saved from VStar. It seems there should be a way to re-load a file saved from VStar regardless of the format in which it was originally downloaded without manually reformatting the saved file."
Brad Walter
One approach to this is to write observations as XML and have a reader for this. Given that we have a defined XML format (AID web service), I would suggest we use that.
It will have to be generalised however, to allow arbitrary key-value pairs for arbitrary observation source formats. This could be a simple sub-class of the code we use now for web service reading, but perhaps generalising all string value reading, i.e. pretty much everything else except JD, mag, error, ...
David,
I tried this "round trip" approach. While it initially failed, I analyzed the saved file format. It appears to be tab delimited and line up with the download file format . After some experimentation, it appears that often the star name is blank, as when loading from a source like Gaia, Kepler or ASASSN. When I simply added a star name by hand, the file loaded with the "New Star from File..." option.
Does this make sense?
Hi Cliff
Yes, I think it does. Let’s choose a particular round trip target +n JD range scenario to explore this.
David
Related
Bugs and Features: #557
Consider Y Oph which is Gaia ID 4175017625462647168 and is located at RA/DEC 268.16125 -6.14358.
So do a load from AID from 2457000 to 2458500. I get 297 observations. No errors. Save to file AID.tab.
Loaded Observations: 297
Johnson B: 5
Johnson V: 11
Tri-Color Blue: 6
Visual: 275
Note the AID.tab file contains only the Visual and V observations???
Now load AID.tab with New Star from file. It loads 286 observations.
Now do a New Star from APASS, NOT ADDITIVE. All filters.
Loaded Observations: 9
Johnson B: 3
Johnson V: 3
Sloan g: 1
Sloan i: 1
Sloan r: 1
Save to APASS.tab. File has 9 lines.
Now load APASS.tab with New Star from file. It loads NO observations.
Now do a New Star from Gaia. Do NOT do a transform or additive load.
Loaded Observations: 53
Blue: 14
Discrepant: 11
Green: 14
Red: 14
Save to Gaia.tab. File contains 53 lines. Note the plugin has put Gaia_4175017625462647168 in the starname.
Now load Gaia.tab with New Star from file. It loads 53 observations.
Now repeat the loads from AID, APASS and Gaia, but do additive loads for APASS and Gaia.
Loaded Observations: 359
Blue: 14
Discrepant: 11
Green: 14
Johnson B: 8
Johnson V: 14
Red: 14
Sloan g: 1
Sloan i: 1
Sloan r: 1
Tri-Color Blue: 6
Visual: 275
Save the observations to AID+APASS+Gaia.tab. File contains 359 lines.
Now load AID+APASS+Gaia.tab with New Star from file. It loads 350 observations. The 9 APASS observations are rejected with the error "The name field cannot be empty.".
Attached file contains my filesystem results.
Cliff
Hi Cliff
Thanks for this analysis.
AID: You’re seeing this because the save to file operation only saves what is visible in the obs list. You can remedy this by selecting the extra series in the plot control dialog before saving to a file. I tried this and it worked OK.
APASS: I confirm your results
Gaia: I confirm your results
AID+APASS+Gaia.tab: I confirm your results
Very cool.
As you say, the saved file is indeed AAVSO download format and VStar recognises it as such.
So the upshot is that any plugin that does not add a name to each observation will result in non-loadable files when observations are saved, e.g. APASS.
I wonder which others have this problem. I see that Kepler does, for example.
I should have noticed this pattern in the past.
Progressing this would be a very Good Thing! Please feel free to run with it and ask for help if needed, if that’s okay.
On a related note. There are some other aspects of AAVSO download file format that VStar is fussy about, e.g. legal transformed (yes, no) values. So, an AID dataset might load OK, but then upon save and reload, VStar might complain that some observations have illegal field values. Perhaps that’s another ticket which may already or should exist.
David
Related
Bugs and Features: #557
David,
I looked at this a bit. Not unlike DIY home and auto repairs, it seems more complex after getting into it.
So I thought I should understand the edits within the code that reads the download file format/”Now star from file…”. There is enough indirection involved in the implementation that I am unsure I can easily determine this.
So I thought I’d look at the APASS plugin since it was like the Gaia plugin. I now realize that the APASS and AAVSONet plugins are implemented mostly in the main VStar executable. Changing this would involve solving a couple problems on my PC. First, I have been unable to get my svn client to get the whole project – some timeout/limit sort of issue. I settled for just getting the plugin portion. The other issue is that when I downloaded a ZIP file of the whole project and tried to run an ant build, I seem to lack some tool(s) that are required. I probably could solve these issues, but at this point do not know how.
I looked through the other observation source plugins and they seem to have the implementation within the plugin code itself, so I should be able to address them. My thought would be to build a name from a unique identifier, if available. For example in the Kepler FITS file for a K2 object we might see “OBJECT = 'EPIC 211785852'”. Where there may only be a filename, I suppose we use that.
Of course another way around this would be to just remove the requirement for the name when loading or perhaps plug in “NA” on the load when it is missing.
What are your thoughts?
Cliff
Cliff Kotnik
clkotnik@att.net
From: David Benn david_benn@users.sourceforge.net
Sent: Monday, November 26, 2018 5:15 AM
To: [vstar:bugs-and-features] 557@bugs-and-features.vstar.p.re.sourceforge.net
Subject: [vstar:bugs-and-features] Re: #557 Reloading Data Files Saved from VStar
Hi Cliff
Thanks for this analysis.
Very cool.
As you say, the saved file is indeed AAVSO download format and VStar recognises it as such.
So the upshot is that any plugin that does not add a name to each observation will result in non-loadable files when observations are saved, e.g. APASS.
I wonder which others have this problem. I see that Kepler does, for example.
I should have noticed this pattern in the past.
Progressing this would be a very Good Thing! Please feel free to run with it and ask for help if needed, if that’s okay.
On a related note. There are some other aspects of AAVSO download file format that VStar is fussy about, e.g. legal transformed (yes, no) values. So, an AID dataset might load OK, but then upon save and reload, VStar might complain that some observations have illegal field values. Perhaps that’s another ticket which may already or should exist.
David
On 26 Nov 2018, at 11:07, Cliff Kotnik clkotnik@users.sourceforge.net clkotnik@users.sourceforge.net wrote:
Consider Y Oph which is Gaia ID 4175017625462647168 and is located at RA/DEC 268.16125 -6.14358.
So do a load from AID from 2457000 to 2458500. I get 297 observations. No errors. Save to file AID.tab.
Loaded Observations: 297
Johnson B: 5
Johnson V: 11
Tri-Color Blue: 6
Visual: 275
Note the AID.tab file contains only the Visual and V observations???
Now load AID.tab with New Star from file. It loads 286 observations.
Now do a New Star from APASS, NOT ADDITIVE. All filters.
Loaded Observations: 9
Johnson B: 3
Johnson V: 3
Sloan g: 1
Sloan i: 1
Sloan r: 1
Save to APASS.tab. File has 9 lines.
Now load APASS.tab with New Star from file. It loads NO observations.
Now do a New Star from Gaia. Do NOT do a transform or additive load.
Loaded Observations: 53
Blue: 14
Discrepant: 11
Green: 14
Red: 14
Save to Gaia.tab. File contains 53 lines. Note the plugin has put Gaia_4175017625462647168 in the starname.
Now load Gaia.tab with New Star from file. It loads 53 observations.
Now repeat the loads from AID, APASS and Gaia, but do additive loads for APASS and Gaia.
Loaded Observations: 359
Blue: 14
Discrepant: 11
Green: 14
Johnson B: 8
Johnson V: 14
Red: 14
Sloan g: 1
Sloan i: 1
Sloan r: 1
Tri-Color Blue: 6
Visual: 275
Save the observations to AID+APASS+Gaia.tab. File contains 359 lines.
Now load AID+APASS+Gaia.tab with New Star from file. It loads 350 observations. The 9 APASS observations are rejected with the error "The name field cannot be empty.".
Attached file contains my filesystem results.
Cliff
Attachments:
RoundTrip.zip https://sourceforge.net/p/vstar/bugs-and-features/_discuss/thread/554707ca/5e43/attachment/RoundTrip.zip (10.2 kB; application/x-zip-compressed)
[bugs-and-features:#557] https://sourceforge.net/p/vstar/bugs-and-features/557/ https://sourceforge.net/p/vstar/bugs-and-features/557/ Reloading Data Files Saved from VStar
Status: open
Group: Defect
Created: Fri Dec 09, 2016 12:29 PM UTC by David Benn
Last Updated: Sun Nov 25, 2018 08:05 PM UTC
Owner: nobody
From VStar CHOICE VI (https://www.aavso.org/content/8-bug-reports-and-feature-suggestions)
"If you import a data file into VStar and save it from the observations tab, the saved files of most file formats cannot be reloaded. Files downloaded from AAVSO database can be reloaded as can simple files. This seems to be simply because the data VStar needs to recognize the file as a given format is not saved with it and/or files with certain file formats are only downloadable from a web location not a file stored locally. The only files that I have been able to save and re-load are files downloaded from AID and Simple files. I have not been able to re-load files from any other source including AAVSOnet and APASS that have been saved from VStar. It seems there should be a way to re-load a file saved from VStar regardless of the format in which it was originally downloaded without manually reformatting the saved file."
Brad Walter
Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/vstar/bugs-and-features/557/ https://sourceforge.net/p/vstar/bugs-and-features/557/
To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/ https://sourceforge.net/auth/subscriptions/
[bugs-and-features:#557] https://sourceforge.net/p/vstar/bugs-and-features/557/ Reloading Data Files Saved from VStar
Status: open
Group: Defect
Created: Fri Dec 09, 2016 12:29 PM UTC by David Benn
Last Updated: Mon Nov 26, 2018 12:37 AM UTC
Owner: nobody
From VStar CHOICE VI (https://www.aavso.org/content/8-bug-reports-and-feature-suggestions)
"If you import a data file into VStar and save it from the observations tab, the saved files of most file formats cannot be reloaded. Files downloaded from AAVSO database can be reloaded as can simple files. This seems to be simply because the data VStar needs to recognize the file as a given format is not saved with it and/or files with certain file formats are only downloadable from a web location not a file stored locally. The only files that I have been able to save and re-load are files downloaded from AID and Simple files. I have not been able to re-load files from any other source including AAVSOnet and APASS that have been saved from VStar. It seems there should be a way to re-load a file saved from VStar regardless of the format in which it was originally downloaded without manually reformatting the saved file."
Brad Walter
Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/vstar/bugs-and-features/557/
To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/
Related
Bugs and Features: #557
Ok. I'll start with examining the code for loading the download format.
I completed a test of all the ob src plugins except Hipparcos and DASCH. I could not locate any data for those two.
I was successful doing a round trip with only AID and Gaia. Oddly, the upload file format would not round trip in spite of having a star name in the original file.
Cliff