[PanoTools-devel] How can libpano support 3D alignment? From: Thomas Sharpless - 2013-04-15 20:46 ```Hi All I'm getting very serious about camera arrays and 3D stitching -- that is, combining images taken from multiple positions, with general parallax correction. And I'm trying to make a version of libpano13 to support some experiments. Given a reasonable array geometry and plenty of overlap between the cameras' fovs, it is perfectly feasible in theory to construct panoramic views on any single center, and to choose that center pretty freely. Of course the 900 pound gorilla is that in order to correct parallax you need a depth map of the scene. But assuming that is possible, there are several other matters that need attention. First, defining the positions and orientations of the cameras. With TrX, TrY, TrZ, PT has the means to specify the positions, and with y, p, r the orientations. The origin of these coordinates would no longer be the pano center -- we haven't chosen one yet -- so let's call it the array center. To fix the array coordinate system it is enough to designate one camera as having all these parameters zero, (except possibly TrZ) . Then array X, Y, Z are parallel to the reference camera's axes. Given the array geometry we can easily compute camera directions and distances with respect to any panocenter, and any rotation of the array as a whole. The relevant quantities for parallax correction are the direction of the optic axis and the direction and distance of the lens pupil. With those facts and a depth map we can compute parallax-corrected image coordinates from 'ideal' pano coordinates, and thus do a parallax corrected stitch. It has to be possible to determine these 6 parameters by optimization, because nobody can build a perfectly rigid array, and because we will surely want to support hand held "pseudo arrays". I'm assuming this can be done using control points found in the usual way, by assigning a depth to each control point and optimizing those along with y,p,r,TrX,TrY,TrZ. This optimization could use any hypothetical pano center, for instance the array center. It would be pointless to try to optimize lens parameters at the same time, as lens distortion is basically indistinguishable from parallax. So all this assumes excellent prior camera calibration. That's as far as I want to go now. Your thoughts, please, on how to insert a true 3D camera / control point model into libpano13's optimization scheme. -- Tom ```