#13 Combine datasets with different pixel sizes

[Grant Myers]

I have a series of geologic thin section image files at various magnifications and thus different voxel sizes. Ultimately, I'd like to create a 'virtual microscopy slide' in jpx or hdf5 format. I have defined both the 'bounding box' and 'voxel size' parameters for the images and resaved them in hopes of combining them into a (mostly) registered stack. But when I 'combine selected datasets' in the 'directory contents' frame, the pixel size and bounding box definitions from the individual files are not correctly interpreted and the images in the result are distorted.

For example, I have two RGB TIF single-frame images. The first is 2873x1446 with a pixel size of 13.28 µm:
BoundingBox 0.00000 38128.09760 59.00000 19242.53100 0.00000 13.27580

The second is 1978x1617 pixels with a pixel size of 1.14 µm:
BoundingBox 30740.47000 32996.68171 6436.03000 8280.25768 0.00000 1.14123

The resulting combined data set has dimensions of 2873x1617 (i.e. the max of pixel dimensions from the two images). The BoundingBox matches exactly that of the second image, and the resulting pixel size is X: 0.786, Y:1.412 and Z1.412.

Suggestions? Is it necessary to resample lower-resolution images to match the resolution/bounding box of the highest resolution image?

[Ilya Belevich]

Hi Grant,
MIB takes metadata and pixel size from the first file of the series when you combine the datasets. I think the best strategy is to, as you said, resample all datasets to the same pixel size and after that combine them together.
Most likely you will need to register the individual images after that. For that you can use the manual multi-point landmarks alignment (from MIB version 2.3) or in the current beta version I have an alignment using automatically detected features, these are located in Menu->Dataset->Alignment

[Grant Myers]

Ilya - thanks for the quick response.

It would appear to me that it's not as simple as taking the metadata/pixel size from one file or the other. The pixel dimensions of the combined result come from the maximum of the number of pixels in all the images. Then the boundingbox information appears to be taken from the last image file. And the resulting pixel sizes are scaled accordingly. So in the example above, the result of combining the two images is 2873x1617 pixels. 2873 pixels wide because that's the width of the first image, which is larger than the width of the second (1978px). The result is also 1617 pixels tall because that's the height of the second image, which is larger than the first (1446px). The pixel size in the result looks to be based on those dimensions and the boundingbox of the second.

OK I've tried some rescaling and combining. I scaled up the first image to match the resolution of the second, which results in a massive file that takes a full minute to load up and 2 Gb of memory. To get the combined result to scale properly, I had to rename the first file so that it comes last alphabetically. After combining, the smaller (higher-resolution) image is located in the top left corner and I need to register it in some way. Manual single/multi point alignment could work, but maybe only after downsampling the higher resolution image to match the resolution of the lower resolution (but massive in size image)...Ironic. Furthermore, manual alignment will get onerous if I've got to do that for many different images. On the other hand, I've already done the heavy lifting: I know (roughly) the coordinates of the bounding box for the higher resolution image. Isn't there some other way?

What's the point of defining Bounding boxes? What about either 'insert into the open dataset' or using Chop?

The former appears to hang with a note in the matlab window that there were errors:

Error using pause (line 20)
Error while evaluating Menu Callback.

The latter fails with a note that there's an error: pixel sizes mismatch.

[Ilya Belevich]

could you specify which version do you use, is it under Matlab or compiled?
General note: here, we mostly work with 3d electron microscopy volumes and this mostly defines the logic behind many actions in MIB. Your task is quite different in many way. Developing a tool for your specific task is not difficult, but quite unique and with large rasters it would require implementation of image pyramids (as in google maps) and virtual mode for reading the files without loading them into memory. The latter was implemented in 2.40,but still can only be used for browsing the images. Are you also interested in some segmentation?

Since we work with volumes of similarly imaged pieces we don't really expect the differences between the pixel sizes in the individual blocks, because of that MIB does not take into account the actual pixel sizes of each image/volume that are combined. It takes the pixel size and the bounding box from the last file of the sequence (I tested it now; I think it is a bug because, it suppose to take it from the first one, which makes more sense to me). In normal workflows, we specify the bounding box for the assembled dataset rather than obtaining it from individual 2d slices.
The reason of the Bounding box is to correctly show the volumes in 3d visualization program. We may have different areas imaged in parallel and we would like to see tham at their correct positions. This way we can visualize multiple volumes and place them above overview images, as for example shown here (http://mib.helsinki.fi/images/gallery/mitotic_er2.jpg), with two models modeled in cropped subvolumes. The second use of bounding box is during cropping: it is possible to crop out a subvolume, process it and import it back using the Chop/Rechop tool as shown here: https://youtu.be/lY0XjNy4Dr8

When you combine the images, the resulting dataset has width/height equal to the largest of those from all the images that are combined together.

I scaled up the first image to match the resolution of the second, which results in a massive file that takes a full minute to load up and 2 Gb of memory.

and how many of such images do you have?
With the available tools you will need the following:
1. resample all available images to the final resolition that you would like to have in the combined stack. It is probably wise to start with just 4 images for tests first.
2. load the first slice
3. go to the Alignement tool and select:
- Mode: two stacks,
- provide a "volume in a single file",
- uncheck the Automatic mode
- specify the ShiftX and ShiftY for the second slice
4. repeat with the third one...
5. this will do a rough alignement, after that it will be possible to run an automatic alignment procedure for correction.
6. at the end, provide/check the pixel sizes and save images in h5 format. You will be able to see it in the virtual mode later, i.e. without loading the whole dataset into memory, as shown here: https://youtu.be/qYoz_fY664o

Alternative solution is to write a special script that does the job for this specific task.

What's the point of defining Bounding boxes? What about either 'insert into the open dataset' or using Chop?
"insert into the open dataset" allows to insert selected file into the open dataset to any position. (there was a bug with it in 2.40 but it is already fixed in beta). So, for example, you already spend a lot of time loading images but at the end realized that you've missed a file, so you can use it to add it to the open volume.

The 'Chop' mode will insert the subvolume into the open one, but it has to be inside the open volume and has the same pixel size.

Best regards,
Ilya

[Grant Myers]

All right -I appreciate the thorough response, recognizing that what I'm doing isn't necessarily the intended application of the software.

Running the compiled windows version of the software: would much prefere to run the Matlab-native version, but unfortunately don't have the image processing toolbox.

Yes image pyramids is probably what I want to get to ultimately, and I think that jp2/jpx or hdf5 files could be the way, but I'm still learning how to get there and hoping MIB will help me along.

For a given set, probably only one image that's this large. However, I've got a higher resolution sub-frame (0.24 µm/px) that's going to mean additional resampling and an even larger size - 35x the pixels! I think I'll have to start with the smallest and work my way out.

Your steps to manually align after rescaling appear to be working, but note that the X-shift and Y-shift are in units of (whole) pixels.

I'm still working out the kinks in automatic alignment - right now the plot of Displacement vs. Frame number comes up blank and the algorithm hangs after choosing yes.

Thanks again, Ilya - I'll keep at it.

GM

[Ilya Belevich]

X-shift and Y-shift are in units of (whole) pixels
yes, that's right. The features in MIB are usually developed for some specific projects, later those parts may be further extended and supplemented with other routines, as, for example, in the alignment tool. Thus in some cases the units may be pixels (as in this case or with the annotations) or in physical units, as with bounding box and 3d-lines. I try to unify these things, but it goes quite slow due because it is boring and I don't really have that much time to do it.
plot of Displacement vs. Frame number comes up blank and the algorithm hangs after choosing yes
which method did you use? You can also try the beta with automatic alignement, I will put it somewhere tomorrow, so that you can download it (I hope you have windows...)

BR,
Ilya

[Ilya Belevich]

here is a compiled beta :
http://www.biocenter.helsinki.fi/~ibelev/temp/MIB2/MIB2_beta_deploy.exe
here is a demo of automatic alignment:
http://www.biocenter.helsinki.fi/~ibelev/temp/MIB2/MIB_brief_AutomaticAlignment.mp4

Ilya