Master dark & cosmic rays
Stacking program, astrometric (plate) solver, fits file viewer
Brought to you by:
han59
Menu
ā¾
ā“
With a view to better understanding calibration I recently took a deep dive into the data I've been using. I'm lucky to have an ASI2600 so am not plagued with noise or amp glow but nevertheless have been using a full set of calibration frames. Pixel peeping into my master dark - from 50 300 second exposures (gain 100, temp -5C) has perhaps led me down a rabbit hole...
The master showed some bright spots. I still had the constituent exposures and soon found that the bright spot only appeared in one frame. As an (ex) particle physicist this appeared to be a cosmic ray cascade. Looking more carefully I found other examples in my subs, some brighter than others. Of course, I'd expect to see these in exposures this long.
My question to the group is whether these should be rejected in the master dark? If these were lights then I'd use sigma clipping to exclude such outliers. I can't determine what calculation ASTAP does for master darks but could understand that such a clipping calculation could be seen as too complicated. If a simple average is used to get the master dark is this the mean or the median? I'm guessing the mean as I'm still seeing evidence in the master...is there a good reason why the median is not used to calculate the master dark pixels? Outliers would then be ignored.
I've been painstakingly deleting frames by hand but have convinced myself this is ridiculous as I'm excluding ~60% of my frames. Should I load these darks as lights and use sigma clipping? I can't be the first person to see cosmic rays in dark frames. Am I overthinking this?
Many thanks in advance for any insights!
Clear skies, Mike
I now have found others detailing how taking the median of pixel values effectively ignores outliers caused by cosmic rays. DSS documents that it defaults to a median calculation for master darks and bias. For Astap I can only find Appendix 1 (stack process) stating that images are calculated using:
(image- {1/nā [darks]} ) / master flat
which implies a mean is used. But this could be simplifying the description and I guess I have to explore the source :-(
Hi Michael,
Yes ASTAP is taking the mean. What understand from other experiments is that the median result in more noisy images/darks.
My Touptek camera has the same sensor as your or 2600. I have never see any significant cosmic rays in darks but I never looked for them very well. What I see is sometimes blinking pixels and telegram noise in the dark frames. I'm open for new ideas and suggestion but I never seen a convincing test that median is better then mean. More the opposite. Note you can always stack more darks to average anything flat.
You could also process the darks with method sigma clip in the lights tab. But only if you remove the keyword FRAME =DARK. Otherwise it will move the frame to the dark tab.
Cheers, Han
Thanks for the response Han. What I'm seeing makes sense then ;-).
I'm pretty certain these are cosmic rays, and in some cases cascades. I need to think about how median use could be noisier, esp in a low noise sensor. I'd have thought that excluding such huge bright outliers could only remove significant pixel to pixel variation and ultimately result in less noise? But now I have the confirmation I can try creating master darks for my library either by using the lights approach you outline or back in DSS (sorry!). I'm somewhat conscious that many are simply using bias frames for darks with this sensor - which would massively reduce the likelihood of cosmic rays in the frames. I dither fairly aggressively so may not need true darks to exclude hot pixels. But it's a once/twice a year task to build up this library, so why not?
BTW - I switched to ASTAP from DSS after seeing several posts/videos praising low noise compared to other stacking apps. I've since seen you have the best multi-night input & built-in calibration frame matching that I've come across so far, so I'll be continuing using ASTAP for stacking (as well as solving, obv!). Thanks for all you've done.
Hi Michael.
The Sony sensors have a pretty small dark current but is becomes noticable/measurable after 60 seconds or so. Up to now i would recommend to keep the same exposure as the light but is doesn't matter so much. You could use a dark of 20 seconds exposure for lights of 60 seconds exposure. But I would not use a bias. In the first second of exposure something happens.
More important is to have darks with the same gain and temperature as the light frames.
For swapping dark with DSS, you have to be very carefull that no flipping occurs. Furthermore DSS uses a very rare TIFF format setting , which ASTAP can't read and also ASTAP can not read 32 bit TIFF. I don't know if DSS can export 32 bit FITS, PPM but that would be the best format. So better to make the dark witin ASTAP. Furthermore ASTAP will also record all details of the master dark inside the DARK header for later use and recording in the stack header. So inthe final stack you will have a record of all frame details. ASTAP can also autmatically select a dark with the correct gain, exposure time, temperature which will be only possible if you make the master darks in ASTAP
So I would just make 50 darks or so, combine them into a master dakrrand if no cosmic rays are visible in the master dark then just proceed.
Cheers, Han
Can you share a few dark frames where cosmic rays are visible?
The rabbit hole has just widened! Good points. I'll stick with ASTAP although may briefly look into processing darks as lights, keeping a keen eye on the fits headers.
If I'd been sensible I wouldn't have deleted the subs until I concluded this query, but I did :-(. However, I have the original master from ~50 subs which shows some - which is why I investigated. It's over 100Mb; how do I best share this. Can I post a OneDrive link?
Mike
Doesn't matter as long it is accessible. You could upload it in a zip file (one file please) at this location at send me the link:
https://ufile.io/
Calculation the median of a series of images is not easy. You would need to keep all frames in memory. If it helps yes, I could look into an implementation. But up to now I have indications that mean/average is better for lights but I have no information about the effect on dark frames.
In principle it is easy to measure. Take two set of darks and process both to a master dark using method median. Then subtract and measure the noise of the subtraction. Do the same but for method mean. You could use DSS for that. Then you have measurement to proof which concept is better.
An other possibility would be to detect cosmic rays and exclude the frames effected.
Han
Apologies for delay responding - I'm away from home.
Here's my original master dark, from 50 frames (300s, gain 100, temp -5 - I use gain 100 for NB): https://1drv.ms/u/s!AgyMCIfwIgxggroONm70hlAXAdqCeg?e=VCPt58
Newbie mistake - 300s is too long and I'd be better using 180s. I need to redo my darks accordingly!
If you zoom 1:1 and look top left there are a couple of what I believe to be cosmic rays visible. These were a lot brighter in the original sub but the mean calculation even with 50 frames means they are still very visible (admittedly looking at an auto-stretch). I was concerned that subtracting these from lights would leave a sort of ghost in the resultant image? Likely only if in a really dark area?? I know there are more tracks and somewhere there was what looked like a cascade with multiple resultant tracks - again more evident in the original subs, now sadly deleted.
By zealously looking around each frame at 1:1 and excluding all subs with evident tracks I ended up with 17 frames - basically killing 2/3! The resultant master dark is here:
https://1drv.ms/u/s!AgyMCIfwIgxgg5hqpl4QNjuuHb0ZmA?e=CKKGp5
I hadn't given thought to implementation issues with a median calculation :-(. I know others do this, but as you say, likely at a significant cost if all data has to be held in memory. Although it's painful to throw away multiple 5 minute subs, having an automatic way to detect tracks from cosmic rays would be a big improvement. And with a cooled camera dark library generation is only an occasional task so this would be a good work-around. Any pointers to a tool that would do this would be most welcome!
Clear skies,
Mike
I can see maybe about three the cosmic rays in the master dark made of 50 darks. I do not see that in my darks. Maybe you are on a high mountain or the natural background radio activity is higher at your location.
I took 54 of my dark made with the Touptek camera having a Sony IMX571 chip. Split them in two groups of 27 darks. Made master darks of them in DSS using method median and average without other options. Subtracted them from each other like one was the lights. See table below for the result.
As expected the method average/mean give a little less noise. For the median the hotpixe values where a little lower. So no clear winner. That's why they likely introduced Median Kappa-Sigma Clipping.
I do not think DSS does a mathematical true median. The memory size doesn't change in the process and stays at about 700 mbytes. It is impossible to calculate the median of 54 frames with a size of 50 mbytes withing 700 mbytes. That would require 2.5 gbytes memory. So I don't know what median process DSS does in practice but not the mathematical median. value for each pixel.
Furthermore I noted in your darks there are several star like structures. High gain? Mixed up with a light?
DSS uses 16 bit format for the master dark. That gives a rounding effect. Better to use the 32 bit float format.
Share some raw darks in future. Maybe I will look into detecting cosmic rays.
Cheers, Han
Thanks for these investigations. Interesting w.r.t. the median implementation in DSS... I did an hour or two of searching and couldn't find any mechanism to efficiently (from a memory usage perspective) determine true median. I wonder how they do it? I did briefly look at the harmonic mean which, whilst being more expensive then arithmetic mean, could be efficiently calculated. It would help with my cosmic ray outliers, but I think it might suppress hot pixels and amp glow too much . Maybe those aren't a problem for me (minimal amp glow, cooled camera, dither) but I'm guessing it's not a general solution. And I've not seen it mentioned in other apps.
Yes, the star-like structures are worrying. Perhaps I have a bad sensor? I hope not. They appear in all my subs but only clearly in the master. These were at gain 100; fairly standard for this camera. (I use gain 0 for LRGB, 100 for SHO)
I'm overdue for redoing my darks. I'll be very careful with these, look at the points above, etc. I took all darks in my fridge and can't believe I had any leakage. There are parts of the UK on granite where background radiation is high, but this doesn't apply to me. Should I continue to use darks - many don't with this camera model and I need to weigh up whether the issues in this topic outweigh the advantage of initial hot pixel removal.
Again, many thanks for taking the time to respond. It's helped my understanding of these calibrations frames and some of the issues you've faced coding ASTAP. I'm in awe!
As a comparison here is master dark from my IMX571 camera.
https://ufile.io/7cpebj4h
Maybe you have to click a few times on the above link and ignore the advertisements popping up.
The hot pixels are single. No grouping
Not using darks is ill advised. The latest Sony sensors including the IMX571 have still clearly visible hot pixels. Using darks is absolutely required to filter these hot pixels out and only then you are using the full potential of the sensor. Darks are also required (or a pedestal frame) for the correct correction by the master flat. Without a master dark you will get over correction by the master flat.
What you could do is take a few hundred darks with a shorter exposure time like 5 seconds since that will reduce the influence of the cosmic rays. Or just make darks with DSS. That will work as well.
Share a link to your new darks so I can use them for some testing how to avoid/detect cosmic rays.
Cheers, Han
Thanks. I'll share darks - but it'll be a while (weeks) due to my current situation.
Currently I can't get to ufile.io. I'll keep trying!