[Gimp-print-devel] Faulty floating point comparisons lead to false errors

[Johannes M. <js...@su...>]

Hello,

this is a follow up of the mail thread with subject

"Gutenprint 5.3.4: only on 32-bit failures with
  'Image is too long/wide for the page'
for certain PPDs"

see

https://sourceforge.net/p/gimp-print/mailman/gimp-print-devel/thread/5676098bc0fd4489557f650e2199ebd9%40suse.de

At SUSE we did some deeper investigation of the root cause
and came to the following conclusions:

Rounding effects in intermediate calculations
throw results off by some least significant bits
with any limited-precision floating point formats.

If you do:
------------------------------------------------------
floattype a = <some calculation 1>;
floattype b = <some other calculation 2>;
------------------------------------------------------
where mathematically calc1 and calc2 are the same value,
then
------------------------------------------------------
if (a > b)   /* which imples a != b */
   error();
------------------------------------------------------
calls for unpredictable problems.

On hardware 1 it might work, on hardware 2 it might not,
when calc1 and calc2 happen to be implemented
with different sequences of CPU instructions
on hardware 1 versus hardware 2.

This cannot be properly fixed with compiler settings
because then calc1 and calc2 can still be implemented
with different sequences of CPU instructions
on hardware 1 versus hardware 2.

The only way out (for what is needed in this case)
is using an epsilon neighbourhood for comparisons
with floating point formats.

For example like the following:

With an open epsilon neighbourhood of b:
------------------------------------------------------
/* consider a greater than b
  * only if a is significantly greater than b
  */
if ( a >= b + epsilon )

/* consider a less than b
  * only if a is significantly less than b
  */
if ( a <= b - epsilon )

/* consider a equals b
  * when a and b do not significantly differ
  */
if ( a < b + epsilon && a > b - epsilon )
------------------------------------------------------
or alternatively
with a closed epsilon neighbourhood of b:
------------------------------------------------------
/* consider a greater than b
  * only if a is significantly greater than b
  */
if ( a > b + epsilon )

/* consider a less than b
  * only if a is significantly less than b
  */
if ( a < b - epsilon )

/* consider a equals b
  * when a and b do not significantly differ
  */
if ( a <= b + epsilon && a >= b - epsilon )
------------------------------------------------------

The point is to not tinker with the calculations
but to only fix the comparisons.


Kind Regards
Johannes Meixner
-- 
SUSE Software Solutions Germany GmbH
Frankenstr. 146 - 90461 Nuernberg - Germany
GF: Ivo Totev, Andrew McDonald, Werner Knoblich
(HRB 36809, AG Nuernberg)