[Lcms-user] Handling of error return from MAT3Inverse

[Glenn Randers-P. <gl...@gm...>]

This appeared (after some discussion of Mozilla's handling of the PNG cHRM
chunk) on png...@li....  See also bug #460520 at
bugzilla.mozilla.org:

... little-cms succeeds when it should fail.

I think it fails to detect the error in MAT3Inverse (cmxmtrx.c):

int MAT3inverse()

it returns 1 on success and -1 for what the comment asserts is a singular
matrix.

But the code in cmswtpnt.c tests the result as a Boolean:

LCMSBOOL LCMSEXPORT cmsBuildRGB2XYZtransferMatrix(LPMAT3 r, LPcmsCIExyY
WhitePt,
                                          LPcmsCIExyYTRIPLE Primrs)
{
...
      // Result = Primaries ^ (-1) inverse matrix
      if (!MAT3inverse(&Primaries, &Result))
                      return FALSE;

The same bug exists in cmsmtrx.c itself (!) but not in cmsxform.c - which
tests for <0.

Glenn Randers-Pehrson (forwarding report by John Bowler)