OpenCV Interface

The YOUTUBE tutorial can be found HERE.

Setting the right Properties
Fusing data seems straightforward, until one is confronted with the details. If the different data sets were acquired by the same instrument and have the same coordinate reference system it is simple, but if data acquired by different instruments is fused, things get complicated rather fast. All data sets are flat projections of 3D objects and each instrument distortions the data in its own way.

We can correct for this and align and fuse data sets, but it is not always easy. The Open Computer Vision Library (OpenCV) supports the stitching/fusion of images (http://www.opencv.org). This GUI allows one to apply the same to spectroscopic imaging data.

Fundamentally, this GUI is using images derived from the data sets to calculate the correct alignment of the data and then fuse them. This means that at any given moment only one dataset is in the memory of the computer and allows for handling large data volumes.

The “Set Properties” is the first of the three tabs in the right of the window shown in Figure 2. From top to bottom: the droplist allows to select the active set. “New Set” and “Delete Set” add a new set or remove the current one. “Move Set Up/Down” means that the current set is moved up or down in the list of sets. This is important as when data is fused the data sets are inserted on their order of appearance in this list, i.e. the last one is the top one that overwrites all previous data in overlapping areas. “Name” allows setting the name of the Set.

There are three ways to set the image used by the program, called “flat image”:

• a) An external image is imported. This image should be derived from the data set and have exactly the same dimensions as the original data. This allows using filters in external programs to enhance them. Attention: Sometimes image and data are rotated.
• b) A monochrome image can be obtained by integrating the data. For this a string is entered here. An example is shown here: “5∗(90-100);-(56-60);20-30∗∗.5”. Different contributions are separated by “;”. “()” mean that the spectral range between them is normalized. A leading “-“ indicates a negative contribution, between two numbers it indicates a spectral range. A leading “∗” means that this range is multiplied by a factor, while a tailing “∗∗” means that it is taken to the power of. This means that our example picture is five times the normalized spectral range between the channels 90 and 100, thus between 0 and 5. From this we subtract the normalized spectral range between the channels 56 and 60. And finally we add the square root of the summed content between the channels 20 and 30. The image needs to be adjusted to the best of the operators knowledge.
• c) An RGB image is created by filling the same form of equation for each of the Red, Green and Blue color channels.

The buttons below allow to export the raw flat image, the geometrically modified flat image as .png file. ”Inspect Data” starts the [Data Inspector] that allows to inspect the data cube. If anything more is needed the use of the ROI tool of PyMCA is recommended.
The last row concerns the Master image. The Master image is a reference that is used for aligning the data. The Master image can either be replaced by the current flat image or the current image can be added to it. Finally, it can also be exported as a .png file.
In the bottom left corner of the main Window one can select if the currently active flat image is put on top of the Master Image, is hidden or shown transparent above it.

Image Transform
This tab allows for geometrical modification of the flat image. The image is subject to an affine transformation (upper part) and/or a warping. The “Offset” is the position of the image relative to the origin of the coordinate system. The “Stretch” is extending or shrinking the image. “General Scale” is a linear factor applied to the image and all data for fusing, taking linear differences into account, e.g. different integration times. “Tilt Scale” allows to take signal variations that depend linear on one position into account. “Small Rotation” is a detailed rotation by a few degrees. Here information of the image is always lost. Rotations in steps of 90 degrees and inversion full under “Large Rotation”. This can be necessary when image and data were written with different protocols.
The next part shows the transformation matrix. It cannot be edited, it’s calculated in the next section. But it is important to realize that there are just nine values in this matrix and thus the degree of correction in the warping is limited, especially if several sources interfere.

OpenCV
In this section features are detected. Two algorithms can be used for this SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF). SIFT yields better results and is in general to prefer. However, as it is not a free algorithm, i.e. if used for non-scientific applications one is supposed to pay for it, it is no longer in the OpenCV standard library. To use it one needs to install an extension, which requires a certain skill level. But it is included in the executable version of this software. This selection is made in the “Options” window (Figure 3), which can be accessed in the upper right corner of the main Window (Figure 2). This window also allows to set parameters for SIFT and ORB, but the default values are rather good. Here also the borders for cropping the image after fusion can be set, i.e. to cut of small bits of data extending out of the original square.
In the OpenCV tab a table is showing the matching features found by pressing “Detect Features”. Selecting one will highlight it in the plot window. The coordinates of the active couple of features are also given in two lines below. Their values can be modified by clicking on new coordinates in the image. The image can also be changed to either show the entire area or add two additional plot highlighting the features by selecting between “Normal” and “Split” screen. “Delete” removes the current couple of features in the table. It can also be “Replac”ed by the modified pair of features, which can also be “Add”ed as a new pair.
The detection of features depends a lot on how similar the images are in contrast and also lateral resolution. So, optimizing an image to be similar to the Master before feature detection might be a good idea.
Once all features are selected one can either “Calculate Transformation” for warping the image or “Estimate the Translation”. The first allows for a better alignment, but requires that features are found throughout the warped image. The capabilities of an affine transformation are limited, but they do not distort the image that strongly.
The content of the Table can also be loaded and saved for debugging purposes.

Data Fusion
Once all data sets are aligned they can be fused. Either by “Merge to Master”, which means that the active set is saved in the area it overlaps with the master image, in case leaving several pixels empty, or by “Merge All”, i.e. merging all Sets in the list. For this it is necessary that all data sets have the same spectral dimension. The buttons for this are in the bottom of the main window, together with buttons that allow to save and load the list of Sets.

---

This tool was derived from experiences gathered with the project referenced below, tow which should be referred if it is used.
Reference: M. Alfeld, S. Pedetti, P. Martinez, P. Walter, "Joint data treatment for Vis–NIR reflectance imaging spectroscopy and XRF imaging acquired in the Theban Necropolis in Egypt by data fusion and t-SNE", Comptes Rendus Phys. 19 (2018) 625–635. https://doi.org/10.1016/j.crhy.2018.08.004.