Hot pixels not being removed after stack

[michael michael]

My latest project is the Cygnus Wall. I took 209 x 180subs and stacked them in ASTAP using Sigma clip average method. But the result was covered in clumps of blue, green and to a lesser degree red pixels. See the attached image. You may need to zoom in to see the hot pixels.

It looks to me like the clumps hot pixels are because of the dithering moving them around. The thing I dont understand is why stacking is not removing them. I have created several stacks in ASTAP before and never noticed this.

I've tried both the AstroSimple and AstroC methods of De-mosaic on somebodies advice. The Simple method seems to give slightly better results, but neither get rid of the problem. The image attached is the AstroSimple version.

Can anybody offer any insights as to why this is happening? I usually love the results from ASTAP, but I can't seem to get this project off the ground. I really don't want to stack using another tool. I've done my experimentation in the past and always preferred the results from ASTAP.

[han.k]

Hi Michael,

That looks abnormal. I checked the Sigma Clip routine here just in case since I had a great modification but it works fine here.

It could be that the background is different so drifting in the series. Then the spread (sigma) of the pixels will be larger and sigma clip will not work so effectively. Can you have a look to the background values in the tab lights. If some have a much higher value then others then uncheck these frames.

You can also set the sigma clip factor lower to filter out more outliers. See attached screenshot. You could set it at 1.5 for this series. A normal value would be 1.8.

If this doesn't help can you share a part of the series for testing here in house so I can have look /trace what happens exactly.

Han

[michael michael]

Thanks Han. I'm going to try all your suggestions.

I just analysed the frames again and the background range goes from 2976 to 3568 across the entire 209 subs. Its quite a slow linear change across the range - with no big jumps in background at any point.

I did discover something else that might be related. Maybe I should have given this context before: The 209 subs were taken over 6 different nights. When I was up to 145 frames I did a quick stack to see how it was coming along and I didn't have anywhere near the vivid hot pixels. I could see them if I really looked, but it was something much more manigable in post processing.

So I had another look ( by eye ) over all the subs and I noticed that the last bunch of frames had a different camera rotation to the 145. Not by much - It looks like about 3-4 degrees. So now I'm thinking: if the 145 sub stack was good but the 209 stack had the problems above. AND the last 64 subs I took contained a bunch of frames that had a different rotation to the 145 - could that have a bearing here?

I'm buidling a new stack of the 209 frame right now ( takes a while! ) with the sigma clip factor set to 1.5 as you suggest. I'll report back when its done - but figured I should outline the above as it might help us trouble shoot.

EDIT: heres a screenshot of my current stack settings. Maybe you might spot something amiss.

Cheers.

[han.k]

A rotation of 3 or 5 degrees should not make a difference.

Settings look fine to me. Note the latest ASTAP version stacks significant faster.

The drift in background makes a difference. A range 2976 to 3568 will result in a mean of about 3250 but any hot pixel in this 2976 to 3568 range will not be easily detected as an outlier.

What you in principle could do is to stack in groups of about the same background value in sigma clip. Then combine the few resulting stack results again with average stacking. But it is much easier to reduce the sigma clip factor to maybe 1.5 so it is more stricter in outlier detection. You will lose some info by reducing the sigma value. A sigma of 2 will filter out about 4% of the pixels. A sigma of 1.5 will filter out about 2x7=14% of the pixels.

Again, if you share the images for experimenting, I could look into this problem and see if I can make something more advanced. In principle you could equalise the light backgrounds prior applying sigma clip. I have never tried it but it could be an interesting option I could consider adding.

Han

Background info "normal distribution", so the effect of sigma in sigma clip routine:
https://en.wikipedia.org/wiki/Normal_distribution

https://itfeature.com/statistical-tables/standard-normal-table

[michael michael]

Hey Han. I tried to stack all 209 frames again using a sigma of 1.5 but it didnt really help. Ive attached 2 images - one from the 209 sub stack I just did, and the other is my 145 sub stack from a few days ago. The files are named accordingly. The ASTAP settings were the same on both aside from the 209 stack using 1.5 sigma.

I performed the same stretch on both and clipped out the same-ish bit of sky. The 145 sub stack doesn't show the pixels but the 209 stack does. Its this that leads me to believe that the latest 64 frames I took are causing this. Although I dont know if its all 64 or just a few.

I might try - as you suggest - stacking the 64 seperately and seeing how that looks. If its fine - then I could stack the 145 + 64 sub stacks together using average.

I would be happy to send you some subs - but there 209 of them! After considering all the above, what would you like me to send you so we can troubleshoot further?

[han.k]

Share about 10 lights with a low background value and 10 with high background value. Zip them together with a master flat and master dark and upload them e.g. to:

https://ufile.io/

And send me the link here or via private message.

Or share them via Google drive or something similar.

[michael michael]

Ok - I'll do that. Although my internet is painfully slow ( I live out in the UK countryside ) so It might take me a little while.

Also, I'm not taking darks or flats for this project so I cant include those. Is that a problem? I find my ASI533 does not need darks as it produces very clean data with enough integration. And usually ASTAP takes care of hot pixels so never felt the need to take them. And I have very little vignetting with my combination of small sensor and wide angle scope. I have stacked lots of projects using this image train and ASTAP and never had any issues thus far.

[han.k]

without the master dark and master flat is fine.

[michael michael]

Thanks Han. Here's a link to my G-Drive zip with the 20 subs: https://drive.google.com/file/d/1XJcqLekVvzUgaZ8rZjqHcbsmFo5kmfbK/view?usp=sharing

The weird thing is, the 20 subs are distributed across all the 209 frames. I expected many of the high background images to be part of the latest 64 frames I took, but that's not the case. So I'm confused as to why my 145 frame stack was so clean compared to the 209 frame stack.

Regardless, if the above files don't help, let me know and I'll try and gather 2 more sets of around 20 subs each - one that stacks fine without the hot pixels and one that shows them. Maybe that might help. Let me know how you get along.

I just want to take this opportunity to thank you for your help. And also for ASTAP which is fantastic and has served me superbly well up till now. I very much appreciate your efforts. Thanks.

[han.k]

A new development version 2023.09.26 is available (Windows):

http://www.hnsky.org/astap_setup.exe

It works well. I had this idea already longer but now it is implemented. Tell me if this work for you or if your using an other operating system then Windows.

Han

[michael michael]

Thank you Han. I'll try this a bit later. Should I leave Sigma set to 1.5 or put it back to 2.0 for this test?

[michael michael]

I gave it a go with your new version. I used Sigma 1.8 and stacked all 209 again. The result is quite a bit cleaner, so whatever you did had a positive effect.

However, its not entirely gone. There are still some clumps of green pixels, but the are much feinter than before and probably easier to deal with in post processing. I've not tried yet, but I will soon.

I've attached a close up of the latest stack using your updated version of ASTAP. I applied the same stretch as I did previously. As you can see, the hot pixels are much less vibrant - but still there. Does this look more normal to you now or do you feel there is still something not quite right?

[han.k]

Removing those pixels manually will be difficult. You better make and use some darks. That should fix it.

Was the temperature reasonable constant and low?

[michael michael]

Yep. -5c across the entire integration.

I just ran the stack through NoiseXterminator and it further reduced the green pixels to the point that they are not really noticeable unless you know they are there and pixel peep looking for them.

[han.k]

Darks are so easy and will make the result much smoother and clean.

[michael michael]

Ok - I'll give it a go. Thank you so much Han. Amazing support.

Also - I noticed it did indeed stack faster than the last version. Good stuff! Cheers.