Extraction of all source in an image / serie of images
Stacking program, astrometric (plate) solver, fits file viewer
Brought to you by:
han59
Menu
▾
▴
Hello Han,
I recently discovered that Astap was able to find several hundreds of gaia stars in a given field, in a few seconds, so as to find a transformation coefficient.
So I was wondering if there was any way to extract all sources photometry in a field, beyond a certain SNR, for every images in a time serie, so as 1) automatically record photometry in an observed field for every already known variable star in that field, and 2) discovering new stars exhibiting variations in a given time serie.
I hope that possibility exists ;-) I tried to do that manually, but found it practically unfeasible.
Thanks in advance
Christophe
Hello Han,
A few days later ...
I discovered that the tool "Magnitude (measured) Annotation" was already available, which is almost what I could dream of.
So, maybe a few questions if you like, about this very interesting function.
Is there a way to adjust the parameters of this tool ? Particularly is it possible to set the minimum SNR for detection, so as to get more or less stars detected above the noise ?
Is there any way to export the complete list of the detected stars, in each image of a serie, with maybe 3 decimals, and astrometric position ?
If you have a few minutes, I would be a very happy photometrist ;-)
Thanx in advance,
Christophe
The measured magnitude annotation will report any detectable star. That is about down to SNR 7. It is not adjustable.
Getting the star info from the viewer image is already available in the viewer pop-up menu:
http://www.hnsky.org/astap.htm#positional_and_photometric_report
If you select in the photometry tab "Gaia online" it should give the magnitudes for all colour bands
If you pull a square (hold right mouse down) prior to release the right mouse button it will give only the info of the stars inside the selection. Else for all stars.
You could do more or less the same in the photometry tab by measure all:
http://www.hnsky.org/astap.htm#measure_all
That is probably easier because you could measure all the stars of a series of images in one step and select and copy and paste the info to a spreadsheet.
In the photometry tab older versions of ASTAP reported standard four outliers in the series but I removed that a few months ago. I did not have the impression it worked very well. But you can try again if you select report all stars in http://www.hnsky.org/astap.htm#measure_all
If you have any other question, please ask.
Cheers, Han
Hello Han,
Thank you for your last message. It helped really greatly.
I think I'm almost there !
I was able to extract more than 3000 stars in a tested star field at once, with many gaia mags, and the associated astrometry, and to transfert all the data to a spreadsheet. All of this in a single mouse click.
Wow. That is very impressive !
Now, that worked only for a selected and lonely image.
I tried to apply the same to a time serie, but when trying to use the options "Measure all annoted" or "Measure all stars, total nr", that give always the same result : I got new columns only for the already kown variable stars, and comps, in the field. Not all stars, as I could extract in the single image, loaded in the viewer. Maybe 50 to 80 kown variable and comps stars. Not the 3000 stars.
At this point, I didn't test the find outliers method.
Most probably something I did bad in the process ?
Can you be nice to help further ?
Anyway, I had a good time, discovering the all sources in a single image.
Clear skies,
Christophe
Yes I see the same here. I also noticed I get a run time error when I try to paste the date to a spreadsheet because I made a mistake in column count.
I will check why there are less stars.
It is pretty untested. An other problem is that older Excel version can can only handle 256 columns. Maybe the latest Excel version has less. Open office much more. I could swap the rows and columns when pasting.
Measure all stars only works for all detectable stars if you select the local annotation database. I'm working on a fix.
Hello Han,
I'm very pleased to be a beta tester for this new function !
I also used Open Office.
Maybe the actual form of the extracted stars file for a single image is good. I mean : stars in row, dates in columns should be good. There will be possibly more stars in an observed field than images over a single night.
Even, if it is limited to 256 dates overnight, this is enought to detect any unkown variation on a star. Let's say one observe continuously for 12 hours overnight, with an image each 3 minutes, that is 240 images in total, so 240 points in a light curve. That's good.
At this point, one don't really need to know the total ADU nor the SNR for each star/each date. Maybe one mean ADU/SNR value for each star / all dates should be enough. So you can save the room in the columns for the dates.
We only need to kown if a star is possibly variable or not. If we find a new variable star, then we normally coud do a more classical photometry for this star, so as to verify if the variation is real or not.
May I suggest you to put the gaia mag to the left columns in the xls file, and to reject the measured mag to the right ?
I had tought about a possibly good way to sort the variables stars in an sample of stars.
Maybe it could be easy to compute, for each measured star in a field :
(max measured photometric value overnight) - (min measured photometric value overnight)
and to put all those computed values in a graph, versus magnitude dispersion.
So that the really variable stars will most probably have a larger dispersion than the non variable stars, and will pop-up in the graph.
That's something any observer could do in the spreadsheet, anyway.
Thank you soooo much for your attention,
Christophe
Last edit: ChrisCiel 2024-11-12
Best way to detect a variable is to calculate the standard deviation. The practical problem is likely the many false alarms you will get. What I could do if you select all (ctrl+A) in the photometry tab and then copy (ctrl+C), I could do some math to the values. This can be shown in the data pasted (ctrl+V) to a spreadsheet.
Open office can handle many more columns then my old Excel 2003.
First I will fix the "annotation all" I'm reorganising the code structure now which use less code but need some testing. Your help will be appreciated.
I assume your using the Window version?
Yes. Win 64bit.
Do you need some data to test ? I can share some spreadsheet for single images, with several thousands of detected stars.
I have enough test files at this moment.
I made several large modifications. The functionality is the same but it should be more efficient. Did also a long testing, so it should be good and I decided to skip an initial beta release. . You can already download this version 2024-11-13..
The listview (tab photometry) becomes a little sluggish if you have a few thousand stars measured. But it should work. Tell me if it works for you.
Han
Hello han,
Thanx for the extensive work on this new function. ;-)
I'm now dowloading the lastest version, and will test it this evening with some time series.
I'll report soon.
Christophe
Yeah !
What a great Job you did ! IT WORKS !
At least, it works as well as a beta version should work ;-)
I'm totally and definitively impressed by all the possibilities that this function will open to amateur astronomers searching for new variable stars, as well as for those waiting for a tool able to measure a lot of kown variables stars in a given field.
I thinks as soon as the word will spread in the community, that this tool exists, the more and more amateur will use Astap. A very long awaited tool for me, at least.
Well, I did some early testings, and for now, I have a few observations to share, and maybe I noticed some possible details yet to improve.
First at all, I did a search on a lonely image, the first of a time serie taken overnight, so to a moderate altitude (airmass 1.4). The algorithm found about 2300 stars with SNR > 7 in that single image. I tested a second image, taken in the middle of the night (airmass 1.2), with an improved signal, as the field moved higher in the sky. The software detected 2700 stars in the field.
When using the "Measure all stars. Nr" 3000, the function did found a really remarquable bunch of about 1900 stars, but failed to found all the previously detected stars. The aperture/annulus were set to 20/30.
Then I set the values to a more realistic 6/10 and then 7/20, but the algorithm, at this time, detected almost 5000 stars, which is probably over optimistic. Then, a few minute later, it detected around 6700 stars, which I think is an error, obviously.
(maybe, when deleting a list of images, the table do not return to the zero state. I can see that the first line in the table stay the same, with thousand of detected star names/astrometric positions, staying on the screen, even if there is no longer any data below. (do you see what I mean ? I can take some screenshots if needed. My english is not very good)
Then I tried to look at a selected image, after having used the “Measure all stars Nr” function, so as to have a check on the detected stars, but the image displayed in the viewer did only shown a few hundred of stars, most probably the already kown variable stars and the aavso comps, and maybe a few other stars, annoted in squares.
I think it would be interesting to get a view on the full field at that point, at any pointed image, so as to control the process.
Also, maybe any means to control/test and set the circle/annulus should be great, before lunching the process of detecting all stars.
Now, I tried to select and copy the list of detected stars (ctrl A, ctrl C), and to paste it to Open Office. I get an error message (in french : Impossible d'Insérer le contenu du presse papier = which could be translated to “Impossible to paste the clipboard contain”).
Indeed some data were present, but how much ? Impossible to say. The spreadsheet was full of columns, but I think its capacity was over. As you said, it was very slow and reluctant to work properly, and the inversion of row and columns was impossible.
Most probably, it would be needed to inverse row and columns before copying them to the clipboard ; at this stage I couldn't go any further with Open Office.
Well ! Here and now, we are almost there !
Hope you understood what I was stating ; don't hesitate to ask me. My english is somewhat imperfect. I could write in french and try an automated goog translation ; maybe should be better.
Clear skies,
Christophe
That is not impossible for longer exposures and a fov of one dregee or more
I will look into this. You could change a setting like annotationdatabase and they will be cleared. Maybe even if you use the clear button. I will check. If not, adding it to the clear button would good idea anyhow.
The viewer is typically limited to 500. See setting max nr star to use in tab Alignment. But increase it for experimenting only. Leave it at 500 for solving.
Best way it to reduce the max nr of star in tab photometry.
No idea why you had problems. Libre Openoffice worked fine for me. Newer Excel versions have 16,384 columns. I will test swapping columsn and rows. Maybe just a second menu option.
Feedback helps.
Cheers, Han
I have added an new menu to copy with the row and columns swapped. See attached.
Download version 2024-11-14:
http://www.hnsky.org/astap_setup.exe
You could process images from different parts of the sky in one step. This could theoretical be used to detect novas, minor planets and so on. But practically that will be too difficult unless you have a very large telescope on a top position at a mountain. Th existing surveys scan the sky in a few days with one meter telescopes. You can't beat that with amateur equipment. But try it and tell me if it works for you.
Cheers, Han
Hello Han,
First : once again, thank you for spending your time to offer this fantastic software to the community, and to expand some functions to a level only available to the professionnal astronomers until now.
Your help is priceless.
Well, our project is not to beat the professionnal in their field. We do not own any one meter class telescope, but only amateur optics ! Anyway we could try to detect some new and interesting objects. Our optics are short focal length, with full frame camera, so the FOV are a little more than 4.5 x 3 deg large, so about 14 square degrees, on automated mounts. Not far from the optics of ASASSN, for example (140mm f2.8 telelenses). We could most probably image half of the sky (or even more) each clear night to mag 16 or 18.5, and maybe deeper, but that will not be our strategy. Minor planets and comets, are not our primary goal, because this is not our area of expertise ; anyway, let's see what will happens in our scopes.
We will concentrate our efforts on short and very short period variable stars, eclipsing binaries, and long period variable also, and search for transients, and maybe for microlensing in the northern Milkyway (we will not waste our time on the southern Milkyway, as the OGLE system is unbeatable in this area, in summer). At least, we will try.
We are now only testing and validating our observations and analysing methods, so this is only the beginning of this adventure.
So, for now I'll download the new version and test it, and then report. I think many observers will use those new automated functions when they will be fully available.
The option for swapping columns and row is certainly a very good idea, and will be much appreciated. Beside the fact that the clipboard had troubles with OpenOffice, I was feeling that it was easier to have a very long tab with many many rows and only a few columns.
Also, a “clear” button could help.
I noticed (on the screenshot you provided) that you did detect the “all stars” with local database only, not Gaia V online, and with “Measure All Annotated” selected. How many stars do you detect, with such configuration, on your typical images ? I noted about 3500 to 4000 stars in our typical field on a single image, outside of the Milkyway, to a level of SNR >7, using the menu “Tools>Magnitude (measured) annotation”. Sometimes no more than 2500 when the sky conditions where less than favourable. This is the reason why I was skeptical, when the system detected 5000 to 6000 stars in the same image.
Also, I have seen that the AAVSO (and gaia ?) would not transmit data for fields larger than 3degrees, and mag 12 ? Do you think it could be possible to bypass this limit by dividing a larger field in a serie of smaller FOV ? Hence a field of 5x3 degrees could be divided in two parts, with two requests to the servers, and the data assembled to a more deeper level for a larger area ? A 9 by 6 deg field could be divided in six smaller fields, and so on.
Another idea was about the actual limit SNR >7 for the stars detections, I told you previously. I understand that, at this level of measurement, the photometry is more reliable than at smaller SNR, which sound very logical. But staying at this level of confidence will prevent any user to detect any transient, or SN which could be otherwise detected just above the noise. Practically, I've seen we were loosing about one half to a full magnitude of data (depending on the sky conditions), as the system detected stars to a limiting magnitude 15.5, for example, where I could see stars to a deeper 16.5 (depending of the B-V of the stars) to the eye.
Do you think it would be feasible to have an option, someday, for the user to choose which limiting SNR he would like for the detection, provided he knows that the extracted photometry is (far) less reliable at low SNR ? It would not be for fine photometric measurements, but for detection purpose only.
Well, enough suggestions for today ! ;-). Let's try this new version of Astap, and appreciate all the works you did on it.
I will report asap.
Clear skies,
Christophe
Last edit: ChrisCiel 2024-11-14
The ASTAP "Clear" button at the photometry tab clears the columns.
More later.
This tool will first calibrate the photometry MZERO with all stars brighter then SNR=30. So the in the log reported number of stars are only the brighter ones. The reported limiting magnitude is for SNR>=7. So many more stars are detectable then reported for calibration.
Next step is to annotate the stars. This is arbitrary set at SNR>=10. But stars with an SNR of 7 are reliable detectable. I think I selected SNR>=10 to avoid too many annotations.
The photometry tab will report any star with a peak value 7 times the standard deviation of the noise and a SNR above 10. If I take one of my images of M31, 200 seconds exposure, field 2.3 x 1,54 degrees, 6126 x4152 pixels, I measure about 7800 stars. Limiting magnitude 18. In the tab photometry this will give 2x7800 columns . The photometry listview (spreadsheet) building block can handle it but is far beyond it's intended use.
So for your 5x3 field it could be 40000 stars or more. The listview (spreadsheet) building block could struggle with this. Maybe it is better add multi file star detection to the batch menu. Then all data could be exported to a .CSV file.
Han
Last edit: han.k 2024-11-14
Anyway, it works very well !
Wow.
I could extract about 1900 LC for a time survey overnight, and just sorted the stars with the function I was thinking :
(Max Mag overnight) - (Min Mag overnight) versus (Mean Mag of the star)
It's simple, not very elaborate, but it works.
This graph is an example.
I noticed that all the mags, in the spreadsheet were without (dot) :
13645 for 13.645
More tomorrow ; I will test the new function extensively.
;-)
Cheers,
Christophe
Last edit: ChrisCiel 2024-11-14
Correction
The photometry tab will report any star with a peak value 7 times the standard deviation of the noise and a SNR above 10.
If I take one of my images of M31, 200 seconds exposure, field 2.3 x 1,54 degrees, 6126 x4152 pixels, I measure about 7800 stars. Limiting magnitude 18. In the tab photometry this will give 2x7800 columns . The photometry listview (spreadsheet) building block can handle it but is far beyond it's intended use.
This peak value is to speed up the detection. It will only search at locations with a peak values 7 x sigma of the noise
The tool “Tools>Magnitude (measured) annotation”. can detect close to 10000 stars in my 200 second exposure of M31 if I reduce the SNR limit to 7. The initial detection is set at 5 x sigma of the noise.
I will correct that. It should follow the Windows settings.
ASTAP follows Windows setting. Loosing the dot or comma is caused by OpenOffice import. There is a setting where you can set 'Locale". That should fix it. There doesn't seems to be a setting to follow Windows regional settings. See attached
Last edit: han.k 2024-11-14
For the menu l “Tools>Magnitude (measured) annotation”, I have added a new log message to report the number of stars detected and annotated. Maybe that helps:
http://www.hnsky.org/astap_setup.exe
Hello Han,
Thanks for the help and the clarification on some important points.
Here are a few observations, and thoughts, if this could help.
I performed more test today. Here is what I have observed so far.
Always the same time serie and same 90 images, so as being able to compare the results of the modifications from day to day.
First, I noted that the “Tools>Magnitude (measured) annotation” performs always more (Around 2600 detected stars) than the detection by “Mesure all Stars. Total Nr” (around 1900 stars found), whatever the setting for annulus and aperture I choose. The result remains constant, and under that of the Tools. No need to say that I prefere the Tools results, with its 2600 stars detected ;-) It detect almost any visible stars. Yes, some are missed, but not too much (see image).
More interesting, I have noted that when I ask the algorithm to detect all stars from image 1 (taken just after sunset) to image 90 (middle of the night), so all the time serie, the detection sticks to around 1900 detected stars and the same limiting mag all over the time serie. Even if the later images of the serie are obviously of better quality, with a better limiting mag.
Then If I ask the algorithm to detect all stars only on the later images, it found about 2330 stars, so 20% more. It is not able to detect all the 2600 stars found with the “Tools>Mag Measured annotation”, but not too far.
By the way, I convinced myself that there is not much more than 3000 stars in total in those images – not 40000. My images are not as deep as yours. The optic is only 90mm in diameter.
The fact that there was, at the beginning, about 2000 detected stars, then 4000, then 6000 was most probably due to a problem with the memory not cleared. It is no longer present since I use the clear button, so far.
Beside of the improvement of the number of the detected stars, what should help analysing those results, should be that the algorithm detects automatically all the already known variable stars in the field, and that the variable stars are correctly labelled in the spreadsheet. There is a lot of variable stars here : between 50 to 80 in the field, but only half a dozen of them are correctly labelled in the spreadsheet, the other ar missing, lurking beyond their coordinates. So, most of the detected variability in the spreadsheet is about known stars, which I need to “rediscover” and discard manually before going deeper. That's a long process, alas.
Finally, having an image with all detected stars could help. I'm not able to find how to get this image. When I go to the viewer, I allway get an annotation only for the known variables and comps – which I do not even have in the spreadsheet hahaha !
About the dot in the numbers, I do not stick with the french coma in number like this 13,432. I had to give this out, because this caused many issue with most of the other astronomical software I use daily, which are only able to deal with US number notation 13.345.
There is no more dots at this point when importing the data from the clipboard, but, to be honest, this is a very small issue ! Don't waste your time on this, I think ;-)
A last suggestion, if you permits : I'm not sure that having an SNR for any detected stars at this stage is really an advantage. We all know that the SNR will be lower at mag 16 than mag 10, and that's not important for detecting new variable stars, I think. In case of discovery, anyway, we will surely re-do another and classic photometry, with all important parameter.
Maybe given up with the SNR could help to keep a reasonable table and the keep the software not too slow with a huge number of columns ?
By the way, so as not to slow the software, maybe the table could not be display at all on the screen, but only exported as .xls or .csv file at the users's choice, with a button, when available, at the end of the process, in a new window/message ?
Well, I hope you understood what I have to say.
If needed, don't hesitate to ask.
I will not be able to test more for a couple of day, but be back soon.
Clear skies,
Sincerely,
Christophe
Hello Han,
I did perform more tests since last week ; well, I'm not sure you made some improvements in the code in the last version of the sofware, but all in all, I found that the all stars extraction function is a great one. This is a great achievment, very very accurate and usefull, and so far, the best possibility in such area for the amateur astronomers.
Thank you soooo much !
Clear skies,
Christophe
Thanks. Yes in revision 2024-11-13 I implemented some major changes and assume photometry extraction is now working flawlessly.
Cheers, Han