FITS captured by Astroberry won't work with ASTAP's created FITS

[Tom Oskar Ortleb]

FITS captured by Astroberry (INDI, I guess) won't work with ASTAP-generated FITS files from Canon DSLRCR2 raws.

Hello Han,
recently I started with a freshly installed Astroberry on Raspi4. First, I was not aware that Astroberry creates and stores FITS files only, even when a CF/SD card is available in Camera. Today I know I could set it up to save to both the camera's CF/SD card and/or the local/remote disk (eg. MicroSD).

Now a have some tens of FITS files from capturing sessions which I knew before, ASTAP could (an will) work with. I also made darks in these sessions. The files are stored locally in astroberry@astroberry's Light and Dark folders respectively.

I rsync-ed everything to my Linux desktop, opened the files in ASTAP and added flats and bias CR2s. During the stacking process, ASTAP complained about bad flats, size differed.

It took me some time to find out that the Astroberry FITS files have been created with a dimension of 5184x3456px while ASTAP creates FITS out of the CR2s with a dimension of 5202x3465px.

We talked about this recently: Reported image size versus sensor size.

Because I didn't get RAWs (CR2s) from Astroberry, I cannot let ASTAP do all the CR2>FITS conversions like I used to do before starting with Astroberry. Thus, I now wish ASTAP could create FITS files with the same image dimensions as the raw files reports (eg. by exiftool).

I did a test with one Canon CR2 file (test.CR2) to find out which conversion does what. Please note that I wrote the conversion call in the filenames. The testfile's name and dimension is always test.CR2.5184x3456px (name and initial dimension)

me@linux% ls -l test*:
-rw-r--r-- 1 frank frank 24990632 Okt 21 11:27 test.CR2.5184x3456px.CR2
-rw-rw-r-- 1 frank frank 36043200 Nov 16 21:41 test.CR2.5184x3456px-unprocessed_raw-astap-f-5202x3464px.fits
-rw-rw-r-- 1 frank frank 37696320 Nov 16 21:36 test.CR2.5184x3456px-unprocessed_raw-astap-F-5360x3516px.fits
-rw-rw-r-- 1 frank frank 54074921 Nov 17 12:56 test.CR2.5184x3456px-usr.lib.unprocessed_raw-5202x3465px.ppm

As you can see, I get 3 different dimensions depending on the "unprocessed_raw" version used. I don't now what program creates the FITS on Astroberry and how. I only saw that libraw and files are dated 2019, well over 2 years ago – seemingly no update available (if that mattered at all):

astroberry@astroberry:~ $ ls -al /usr/lib/libraw/
test 140
drwxr-xr-x  2 root root  4096 Jun  3  2020 .
drwxr-xr-x 89 root root  4096 Okt 11 20:41 ..
-rwxr-xr-x  1 root root  9976 Jan 10  2019 4channels
-rwxr-xr-x  1 root root 14148 Jan 10  2019 dcraw_emu
-rwxr-xr-x  1 root root  5804 Jan 10  2019 dcraw_half
-rwxr-xr-x  1 root root  9932 Jan 10  2019 half_mt
-rwxr-xr-x  1 root root  9984 Jan 10  2019 mem_image
-rwxr-xr-x  1 root root  9964 Jan 10  2019 multirender_test
-rwxr-xr-x  1 root root  9984 Jan 10  2019 postprocessing_benchmark
-rwxr-xr-x  1 root root 22480 Jan 10  2019 raw-identify
-rwxr-xr-x  1 root root 10032 Jan 10  2019 simple_dcraw
-rwxr-xr-x  1 root root  9996 Jan 10  2019 unprocessed_raw

So, as I have no clue by whom and how the Astroberry FITSs are created (I suspect by INDI libs and unprocessed_raw), I hope you can enlighten me as ASTAP is also available in Astroberry.

I need consistent image dimensions in the FITSs. Preferably, I would store all CR2 files, because they are much smaller than the FITSs and the FITSs can always be re-generated out of the RAWs – but not the other way round.

Currently, I can't use my already available darks and flats (for lens vignetting compensation) with the Astroberry created FITSs ... and of course I don't want to mix and mess everything up.

Because Astroberry delivers 5184x3456px lights and darks, I cannot use the lights together with darks, flats or dark flats (biases) with different dimensions in ASTAP (and probably elsewhere).

Suggestion: Is it perhaps possible to implement a switch or a drop-down selection in ASTAP to chose the right image dimension for all involved files? With a switch, there could eg. be the default to use "sensor width and height" like ASTAP does it currently and when the switch is off (not checked) it could be "image dimensions" as it is set (or reported) reported by Astroberry's RAWs and DSLR RAWs via exiftool and others.

NB: As I only work with Canon DSLRs, this sensor versus image dimension thing might not be a problem with dedicated astro cameras ...

Regards,
Tom

Edit / Important:
The FITSs created by Astroberry (INDI) carry no other image dimension than

NAXIS1 = 5184 / length of data axis 1
NAXIS2 = 3456 / length of data axis 2

... which is what eg. exiftool reads. I don't now if a more current libraw (or what else) in Astroberry could work wonders?!

[han.k]

Hello Tom,

5184x3456
5202x3464
5360x3516

Yes this can be confusing. Libraw can give you the three different dimensions as listed above.

The largest one (5360x3516) is the default if you use the standard unprocess_raw for CR2 to TIFF conversion. The result info from the full sensor but at two sides it contains a black part which is not iluminated. I assume this part is used by DSLR camera as a dark reference.

The smallest one (5184x3456) is the official sensor size which is fully illuminated. I assume Astroberry is producing that size.

The 5202x3464 is the format produced by the special Libraw program unprocessed_raw-astap. All pixels are illuminated. I made this modified version unprocessed_raw-astap and selected this format (reported by Libraw) because it contains all illuminated pixels. It would be a waste not to use it.

To fix your problem there are two ways.

1) You could switch to DCRAW as conversion program. Go to tab "stack mmethod and at the left bottom you can choose raw conversion. Select DCRAW. The only problem you will miss some information in the header. But exposure time should be available. Also this DCRAW version is a special one for the purpose to retrieve the exposure time (dcraw-astap?)

2) I have to make a new version of unprocessed_raw where you can select the standard size. I have to make a menu option in ASTAP to select that. This could take some days.

Regards, Han

[han.k]

I think the camera supplier(s) don't report these extra pixels because they want standard sizes and standard length x width ratios.

[han.k]

Using the Libraw utility program raw-identify you can get three different dimensions identified from a .CR2 file:

Thumb size:  5184 x 3456
Full size:   **5344 x 3516**
Raw inset, width x height: **5184 x 3456** left: 152 top: 56
Aspect: 3:2
Image size:  **5202 x 3464**
Output size: 3464 x 5202
Image flip: 6
Canon record mode: 6, CR2
SensorWidth          = 5344
SensorHeight         = 3516
SensorLeftBorder     = 152
SensorTopBorder      = 56
SensorRightBorder    = 5335
SensorBottomBorder   = 3511
Raw colors: 3

[han.k]

Dcraw gives here a 4th type format 5202x3465 so it doesn't help. Only option 2) will help. But for programming, I have to be careful because it crops on the other side... You want the pixels to match exactly for dark/hotpixels compensation.

[Tom Oskar Ortleb]

Hello Han,

the RasPi4 with Astroberry ist fixed below the mount outside. It is connected to the home network via WiFi. From my (Linux) desktop I catch the Astroberry's desktop (with KStars, Ekos and INDI) via VNC over WiFi – this works good.

Capturing images in a session delivers 5184x3456px RAWs with no additional information about sensor size. Thus, I must use 5184x3456px for the calibration frames, too. Of course, I take the darkframes after the lightframes with the same settings. Again, 5184x3456px is the single available information available in the RAWs created by INDI / Astroberry.

Because I took (or take when needed) the bias and flat frames as CR2 RAWs, these then also offer the sensor dimensions and borders which ASTAP (on my desktop) makes use of. This is generally a good idea, the best approach, but unfortunately issues trouble when raw-identify can only offer the so called "Thumb size" image dimension.

I thus have to "tell" (read: force!) ASTAP to only use the "least common denominator" available in the lightframes and in all calibration frames. Whatever biggest dimension a lightframe may offer – it dictates the correct image dimension any calibration frame must offer, too.

Han says: "2) I have to make a new version of unprocessedraw where you can select the standard size. I have to make a menu option in ASTAP to select that. This could take some days."

This sounds good! Thanks for your great support (as always)!

Regards,
too

[han.k]

Yes it will be a new option for Libraw conversion inside ASTAP. I testing/debugging now.

[han.k]

Version 0.9.594 is ready. Download from here:

https://www.hnsky.org/astap.htm#alpha_version

Select in tab "Stack method", for raw conversion "LibRaw Thumb size"

Please check if the pixels match. Look for a hot pixel in the AstroBerry files and if it exactly matches with a hot pixel of the other files. The should be precisely at the same location. Otherwise darks and flats will not work optimal. Otherwise you have to make the dark and flats in Astroberry using the same software.

I noted that SGP is producing the same size images as ASTAP (5202 x3464). But "LibRaw Thumb size" will produce 5184 x3456 dimensions.

Please tell me if this works.

Han

[Tom Oskar Ortleb]

Before going to sleep, I tried a set of biases / darks / flats and lights with 5 images each. Please remind, the lights and darks are FITS only and were created by INDI (Astroberry) without using/keeping CR2 RAWs. Thus, their image dimension dictates the dimensions of flats and biases to be used for calibration. I reuse flats and biases regularly.

Unfortunately, the ASTAP converted flats and biases RAWs to FITSs look very false: full of color patterns and in particular the biases way too bright and colorful.

I noticed that, unsing the sensor dimension from the RAWs, the Bayer pattern is actually GBRG while when using the image (thumb) dimension, the Bayer pattern is RGGB. Thus, I tried both settings for the Bayer pattern manually in the Stack method tab ... with no useful result in both cases.

For further investigation I put each 5 FITSs from INDI (Astroberry) in their respective lights and darks folders and each 5 RAWs each from my readily available biases and flats in their. I zipped these folders (b/d/f/l) with 20 files into b-d-f-l.zip and uploaded the ZIP to my server.

You will get the download link in a minute via private message ...

Good night,
Tom

[han.k]

Cropping will influence the Bayer pattern unless you do it with a multiply of 4 pixels. ASTAP is following the LibRaw values which is the standard. If Astroberry software and ASTAP do it the same is should not make a difference but INDI should update the keywords correctly after applying cropping of the data.

I will test you files later in the day.

GBRG and RGGB differ only in a vertical flip.

I there a reason not to make the darks and bias in Astroberry software? If you make all images in the same way there is no problem.

Han

[Tom Oskar Ortleb]

Hello Han,

I there a reason not to make the darks and bias in Astroberry software?

I make new darks whenever needed for a given combination of camera body + exposure time + temperature. The flats and biases (and/or dark-flats) are usually done next day as for the flats with a lens ("closed" refractor) we only need to image the vignetting.

Darks, flats and biases are stored in a folder structure, so that I can reuse them whenever the lights fit. I have to setup and polar align my unguided mobile setup every single time. Thus, I mostly do 60s subs at ISO 800 (~gain) so that I don't need new calibration files every time. Nevertheless, with the uncooled DSLR I almost always take darks because temperature matters and changes during a session.

It is IMO not possible (or at least not recommended) to take flats with my balcony setup because this location and the observable sky sector suffers 1 strong direct light from the neighborhood in addition to light pollution from my town directly south (I live on the north end) – my only home observation option.

As I only need flats for good vignette imaging ... I really don't like the idea to unmount and rebuild the Astroberry setup, cables and connections and carry it elsewhere just for that. I used to make my flats (all RAWs) with camera and tripod alone against a well selected wall inside with only little diffuse light (to get longer exposure times), even and without reflections or double shadows.

My "flat library" already contains many camera body + lens + aperture + focal length combinations, all RAWs (CR2) which I re-used many times. It was a whole lot of careful work, tagging, naming.

Only since I started with Astroberry some weeks ago, troubles appeared due to this sensor / thumb / image dimension dilemma. Thanks to your hint (raw-identify) I now understand the problem and thus suggested earlier in this thread to use what the camera makers officially announce only, that would be the image dimensions, or in this case misleadingly called "Thumb dimension".

Of course I see and understand that FITS is the preferred and most useful image format in AP, with all these nice informations in the FITS header. Unfortunately, I cannot easily switch to "all FITS" with astroberry. The best compromise seems to let Astroberry (INDI) take only all RAWs. Thus, I dont' get the header information but at least the filenames are meaningful and the RAWs need much less storage.

Happy sunday,
Tom

[han.k]

I checked some more things of the .CR2 conversion.

1) The cropped applied by ASTAP based on LibRaw is a multiply of 2 so it doesn't influence the Bayer pattern.

2) The image orientation is compatible with MaximDl and SGP. So hotpixels are at the same place. So no need to change that.

3) MaximDL and SGP produce 5202x3464 pixel files. This is the same as the standard setup in ASTAP but different from the Astroberry files.

4) Astroberry/INDI converts the Bayer to RGGB by adding the keyword
ROWORDER= 'TOP-DOWN' / Row Order

So RGGB
is
RG
GB
but flipped vertical you get
GB
RG

equals GBRG

That is the correct one for the image produced by ASTAP. I checked terrestrial images in colour and it looks all correct. I do not want to change that to RGGB & ROWORDER= 'TOP-DOWN' because the roworder keyword is not universal used.

I could not test the your BIAS.CR2 images orientation on hotpixels. The hotpixels are too weak for a good detection and it is not possible to compare them with the lights from Astroberry/INDI. That will require longer exposed darks in CR2 format.

The gain is universal 800 ISO. I will add the keyword ISOSPEED to ASTAP and handle it as gain.

So it should work. Still I think the best approach is to capture lights, darks, flats, dark & flats all in the Astroberry software and not separate.

Han

[han.k]

This is an interesting link showing the frustration of the DSS team with LibRaw and the different sensor areas:
https://groups.io/g/DeepSkyStacker/topic/choice_between_activearea_and/76429886?p=

I'm still interested in a long exposure dark in CR2 format and .FITS from Astroberry to compare and see exactly which sensor area they extract.

Han

[han.k]

Using Canon "Digital Photo Professional" to convert CR2 to PNG indicates that the current conversion is 10 pixels in the left and 5 pixels in the top shifted. I will fix that. The definition will change in:

1) Full sensor including masked areas (5360x3516) (not used in ASTAP)
2) Active area (5202 x3464)
3) Default cropped active area (5184 x3456)

The description "Thumb" in DCRAW and Libraw is strange to me.

[Tom Oskar Ortleb]

On Linux Canon's DPP ist not available, thus I never used it in 15+ years. I also stumbled upon this strange "Thumb" naming ... it'ss inventor must have a huge thumb ... :D

[han.k]

Hello Tom,

Can you send me a long exposed dark in CR2? A dark produced by INDI/Astroberry I have already from you. I just want to know if Astrobery software is maybe producing flipped vertical.

DPP and Photoshop produce the same images from CR2. I checked that just to be sure.

Attached the latest conversion program. Option -i is for the "Default cropped active area (5184 x3456)". That should produce compatible files as Astroberry. To be checked by hotpixels. They should match. You could copy it to the latest ASTAP at \opt\astap

I like to fix this and then do other Sunday things. :)

[han.k]

You can now also download the lastest ASTAP version which contains the latest updated unprocessed_raw included.

I like to have the RAW conversion fixed and stable otherwise people starts to create darks which are later incompatible.

Regards, Han

[Tom Oskar Ortleb]

Hello Han,
here is the link to a Canon EOS 7D CR2 raw with image dimension 5184x3456px. This dark is from march 2021. I also attach an ASTAP created FITS out of this raw.

Please download: see link via personal message!

[han.k]

Thanks Tom.

I just a fixed an mix up in the code between DCRAW and LibRaw. You have to download ASTAP v1.0.0RC2

The files where very useful. But they don't match:

Hotpixels

DPP (Adobe) 18, 3141 and 152, 2802 (counting starts at zero)
ASTAP: 19, 3142 and 153, 2803
INDI: 29, 2147 and 163, 2808

So there is a shift of 10 in width and 5 in height. The fault lies in Ekos/INDI. They made the same mistake as I made in the beginning. For ASTAP the position of the hot pixels is exactly the same as with Canon "Digital Photo Professional"

So INDI and ASTAP images do not match. You can't mix them. This is a bug in Ekos. I will post something on the INDI webpage.

Han

So C lies not in the middle of A. See image.

[Tom Oskar Ortleb]

"So INDI and ASTAP images do not match. You can't mix them. This is a bug in Ekos. I will post something on the INDI webpage."

This es exactly the insight needed to understand why I had no success to get good stacking results with ASTAP when using lights and darks created by INDI@astroberry together with flats (and biases) from my desktop workflow: horrible backgrounds!

There should really be a consent about what "active region" means for RAWs.

[han.k]

This is my post at INDI forum:
https://indilib.org/forum/ekos/10767-ekos-extracts-the-wrong-raw-area-shift-of-10-and-5-pixels.html#77882

[Tom Oskar Ortleb]

Thanks for precisely describing the error in the Ekos/INDI forum. Hopefully this bug gets fixed this year (hint: Christmas ahead ... gifts usually on their way ... :D )

[han.k]

Somebody has to report as bug. I don't where to put it and at the moment I like to leave it to other.

For a solution now, can't you run Ekos in house for darks and flats? Or do it outside during a cloudy night to get the correct low temperature. I make my darks always at night because during daylight, some light can leak inside the camera system.

Han

[han.k]

This is the situation. Ekos is blue: