#26 invert (c, transpose (b, a))

[Anonymous]

Given an AFFINE Matrix44f with translation components in column [3], this sequence gives an incorrect result:

Matrix44f a, b, c;
// Set matrix a with non-zero translation and a.mState = AFFINE
invert (c, transpose (b, a));

The problem is that transpose merely copies the mState (AFFINE), so invert calls invertAffine, which contains this:

// handle matrices with translation
if (COLS == 4)
{
// The right column vector of the matrix should always be [ 0 0 0 s ]
// this represents some shear values
result[3][0] = result[3][1] = result[3][2] = 0;

There are two work-arounds:

* Reverse the order of operations (which is mathematically equivalent and avoids the bug):

// invert (c, transpose (b, a)); // transpose fails to change mState, invert makes bad assumption
transpose (c, invert (b, a)); // avoid the bug

* Force mState to FULL before invert:

transpose (b, a);
b.mState = FULL;
invert (c, b);