I have used them both in the past. CLEANest was used when working with gaps in the data. SLICK is a "prewhitening" routine that I used to use a lot, but haven't in many years. I think both are more useful when working with long period variables. These could also be possible options for a plug-in framework, if one is implemented. My vote is to put them on the list, but perhaps make it late phase 2 or even a phase 3 thing.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for increasing the priority of this. As I mentioned to David, there is a Peranso implementation but it does not produce correct or usable residuals (which are important!) and Tonny V has confirmed the bug and also the fact that it won't be fixed anytime soon!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No problem Doug. Now that I'm back on the other side of the pond (just got to Sydney airport from San Francisco) I took a quick look at the ts1201.f Fortran code I'm planning to use. I don't know what the status of that implementation is, but I'll check with Matt T. I assume it's fine though, and should have no problem in translating this for VStar. It may take me a week or so before I start doing much development again, but I'll keep you posted. If you have any particular ideas about presentation of the output of the CLEANest spectrum algorithm, feel free to make suggestions. Without a closer look, I imagine there'll at least be a dialog with a line plot.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
First implementation of CLEANest that passes a unit test based upon the tcas.dat example scenario provided in the docs accompanying ts12.f. Locked and variable periods have not yet been incorporated.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Finished initial complete CLEANest implementation with variable and locked periods now requested (via dialog); added more unit tests to match new functionality.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Arising from email discussion with Doug and Grant, Grant recommended that ultimately VStar's CLEANest implementation should permit the inclusion of harmonics per frequency specified for refinement. Here is the last email exchange on the topic, a reply from Grant on July 17 2011 to comments made by Doug and I:
> > The TS (and VStar) implementation of CLEANest permits these frequencies/periods categories to be specified:
> >
> > 1. User selected
> > 2. Locked
> > 3. Variable
> >
> > So are you saying that we essentially need another category:
> >
> > 4. Harmonics
> >
> > and that unlike 3. (variable), although these can vary, they must do so such that they maintain their harmonic relationship with their "parent" frequency?
Yes, I think that describes it quite well
> Ideally, the category would be "Variable fundamental frequency with
> integer-value-locked harmonics up to some specified degree".
There is merit in this description also. The harmonics must be tied to the fundamental, and must be integer multiples of same.
In my R version, I pass it a list of frequencies (to be refined by CLEANEST) together with a list of harmonic orders. Then it finds the combination of fundamental frequencies which, *together with their integer multiples up to the specified order*, best fit the data.
For example, if I had identified a frequency of f=0.177 for which I wanted 6 harmonics (harmonics up to 6f), and another frequency f=0.202 for which I only wanted 2 harmonics (f and 2f), applied to data "x" at times "t", I would say
Maybe instead of including a new "type" of frequency, you can simply set a "number of harmonics" parameter for each frequency in the list (with all of them defaulting to 1)
By the way -- some folks (most in fact) call 6 times fundamental the 5th harmonic, so if you want multiples up to 6 times then you'd have fundamental plus 5 harmonics. I call 6 times fundamental the 6th harmonic, so if you want multiples up to 6 times then you have 6 harmonics. It's nonstandard terminology -- I'm such a maverick! -- but maybe I'll be able to change the world (over the course of several decades) and introduce a new lexicon. But just so you know, when I say up to 6th harmonic I mean f, 2f, 3f, 4f, 5f, and 6f -- whereas most would say that's only up to the 5th harmonic.
For VSTAR it might be best to go with the more common convention. You could then have check-boxes or a drop-down menu for the number of harmonics *starting with zero* (no harmonics), which is what I would refer to as "starting with 1" (just the fundamental).
Sincerely,
Grant
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Fixed a bug in which the wrong period was being sent to the phase plot dialog after CLEANest. This lead to a rewrite of period analysis selection message generation which also fixed other bugs relating to selection of rows and plot coordinates. In:
Implemented unified CLEANest parameter dialog including harmonics per user frequency. These are passed to refinement method on period analyis algorithm but not yet implemented in CLEANest implementation.
Moved CLEANest source files into a "refinement" package and renamed CLEANest dialog to RefinementParameterDialog. Added temporary code to identify harmonics specified for each user-requested frequency.
Apart from the current work towards including harmonics in the analysis, here are some other possible features to implement:
1. Be able to reset CLEANest, which would entail removing additions to the top-hits table, and remove plot annotations.
2. Permit addition of frequencies to the set of user-selected frequencies other than locked or variable, assuming these categories won't suffice.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have used them both in the past. CLEANest was used when working with gaps in the data. SLICK is a "prewhitening" routine that I used to use a lot, but haven't in many years. I think both are more useful when working with long period variables. These could also be possible options for a plug-in framework, if one is implemented. My vote is to put them on the list, but perhaps make it late phase 2 or even a phase 3 thing.
Doug Welch's comments at Citizen Sky 2 suggest that the priority of this should be higher.
Thanks for increasing the priority of this. As I mentioned to David, there is a Peranso implementation but it does not produce correct or usable residuals (which are important!) and Tonny V has confirmed the bug and also the fact that it won't be fixed anytime soon!
No problem Doug. Now that I'm back on the other side of the pond (just got to Sydney airport from San Francisco) I took a quick look at the ts1201.f Fortran code I'm planning to use. I don't know what the status of that implementation is, but I'll check with Matt T. I assume it's fine though, and should have no problem in translating this for VStar. It may take me a week or so before I start doing much development again, but I'll keep you posted. If you have any particular ideas about presentation of the output of the CLEANest spectrum algorithm, feel free to make suggestions. Without a closer look, I imagine there'll at least be a dialog with a line plot.
A couple of conversations with Doug since this have continued to keep this uppermost. Raising the priority.
In http://vstar.svn.sourceforge.net/viewvc/vstar?view=revision&revision=744:
Initial, extremely rough translation of CLEANest from Fortran to Java.
In http://vstar.svn.sourceforge.net/viewvc/vstar?view=revision&revision=745:
First implementation of CLEANest that passes a unit test based upon the tcas.dat example scenario provided in the docs accompanying ts12.f. Locked and variable periods have not yet been incorporated.
At this stage, only support for CLEANest from ts12.f is planned, not Slick. That can change however, depending upon demand from end users.
In http://vstar.svn.sourceforge.net/viewvc/vstar?view=revision&revision=746:
Initial GUI CLEANest exposure implemented; currently, top hits table in DC DFT results dialog updates only.
In http://vstar.svn.sourceforge.net/viewvc/vstar?view=revision&revision=747:
Updated dist jar; added T Cas data file for use with CLEANest testing.
First implementation of adding CLEANest spikes to power spectrum in http://vstar.svn.sourceforge.net/viewvc/vstar?view=revision&revision=760
One curiosity I've noticed is that for each CLEANest run for a given DC DFT, the same amplitude value is returned! Is that the case for TS. Why?
Checking that the user cannot select CLEANest results as input to CLEANest in http://vstar.svn.sourceforge.net/viewvc/vstar?view=revision&revision=761
In http://vstar.svn.sourceforge.net/viewvc/vstar?view=revision&revision=778:
Finished initial complete CLEANest implementation with variable and locked periods now requested (via dialog); added more unit tests to match new functionality.
Arising from email discussion with Doug and Grant, Grant recommended that ultimately VStar's CLEANest implementation should permit the inclusion of harmonics per frequency specified for refinement. Here is the last email exchange on the topic, a reply from Grant on July 17 2011 to comments made by Doug and I:
> > The TS (and VStar) implementation of CLEANest permits these frequencies/periods categories to be specified:
> >
> > 1. User selected
> > 2. Locked
> > 3. Variable
> >
> > So are you saying that we essentially need another category:
> >
> > 4. Harmonics
> >
> > and that unlike 3. (variable), although these can vary, they must do so such that they maintain their harmonic relationship with their "parent" frequency?
Yes, I think that describes it quite well
> Ideally, the category would be "Variable fundamental frequency with
> integer-value-locked harmonics up to some specified degree".
There is merit in this description also. The harmonics must be tied to the fundamental, and must be integer multiples of same.
In my R version, I pass it a list of frequencies (to be refined by CLEANEST) together with a list of harmonic orders. Then it finds the combination of fundamental frequencies which, *together with their integer multiples up to the specified order*, best fit the data.
For example, if I had identified a frequency of f=0.177 for which I wanted 6 harmonics (harmonics up to 6f), and another frequency f=0.202 for which I only wanted 2 harmonics (f and 2f), applied to data "x" at times "t", I would say
freq.raw = c(0.177,0.202)
harmons = c(6,2)
freq = clean(t,x,freq.raw,nharm=harmons)
Maybe instead of including a new "type" of frequency, you can simply set a "number of harmonics" parameter for each frequency in the list (with all of them defaulting to 1)
By the way -- some folks (most in fact) call 6 times fundamental the 5th harmonic, so if you want multiples up to 6 times then you'd have fundamental plus 5 harmonics. I call 6 times fundamental the 6th harmonic, so if you want multiples up to 6 times then you have 6 harmonics. It's nonstandard terminology -- I'm such a maverick! -- but maybe I'll be able to change the world (over the course of several decades) and introduce a new lexicon. But just so you know, when I say up to 6th harmonic I mean f, 2f, 3f, 4f, 5f, and 6f -- whereas most would say that's only up to the 5th harmonic.
For VSTAR it might be best to go with the more common convention. You could then have check-boxes or a drop-down menu for the number of harmonics *starting with zero* (no harmonics), which is what I would refer to as "starting with 1" (just the fundamental).
Sincerely,
Grant
Fixed a bug in which the wrong period was being sent to the phase plot dialog after CLEANest. This lead to a rewrite of period analysis selection message generation which also fixed other bugs relating to selection of rows and plot coordinates. In:
http://vstar.svn.sourceforge.net/viewvc/vstar?view=revision&revision=785
Implemented unified CLEANest parameter dialog including harmonics per user frequency. These are passed to refinement method on period analyis algorithm but not yet implemented in CLEANest implementation.
In http://vstar.svn.sourceforge.net/viewvc/vstar?view=revision&revision=793
Moved CLEANest source files into a "refinement" package and renamed CLEANest dialog to RefinementParameterDialog. Added temporary code to identify harmonics specified for each user-requested frequency.
In http://vstar.svn.sourceforge.net/viewvc/vstar?view=revision&revision=794
Apart from the current work towards including harmonics in the analysis, here are some other possible features to implement:
1. Be able to reset CLEANest, which would entail removing additions to the top-hits table, and remove plot annotations.
2. Permit addition of frequencies to the set of user-selected frequencies other than locked or variable, assuming these categories won't suffice.