From: Richard C. W. <drsavage@q.com> - 2010-06-04 16:37:04
|
Kein-Hong and Ben, Thanks for your responses. Kein-Hong wrote: > A float has about 7 decimal digits of precision. For whatever > reasons, your result has a 1 plus 'a very small number'. So, > taking 8 digits of displayed result (since the '1' at the front > doesn't 'contribute' a full 1 digits of precision): > > 1.0000014 > 1.0000015 > 1.0000017 > > Nothing wrong with the compiler or the machine. and Ben added: > There is something very wrong somewhere because if you are performing > > x / 640.0 * 9e-6 > > For 0.0 <= x <= 640.0 > > Your results should be way less than 1. Then Kein-Hong correctly deduced: > Sounds like a throwaway test of some sort when zooming in failed > (or turned blocky) due to lack of numeric precision. Sorry, my description of my test wasn't clear. The results I posted were from a stand-alone, throw away test program that made the same calculation that I'm using in my main program. The calculation is: Domain_x = ((Pixel_x / 640.0) * 9.0e-6) + 1.0 I've run the same test using another programming language, on the very same machine. With the second programming language, I have full control over how the calculation is performed. In this case, I'm starting with the integer value Pixel_x, pushing it onto the FPU's stack, then pushing the other floating point values onto the stack and performing the math ops. The results are in the third column of numbers below. Pixel_x = 100 Domain_x = 1.00000143051147460000 1.000001406250000 Pixel_x = 101 Domain_x = 1.00000143051147460000 1.000001420312500 Pixel_x = 102 Domain_x = 1.00000143051147460000 1.000001434375000 Pixel_x = 103 Domain_x = 1.00000143051147460000 1.000001448437500 Pixel_x = 104 Domain_x = 1.00000143051147460000 1.000001462500000 Pixel_x = 105 Domain_x = 1.00000143051147460000 1.000001476562500 Pixel_x = 106 Domain_x = 1.00000154972076420000 1.000001490625000 Pixel_x = 107 Domain_x = 1.00000154972076420000 1.000001504687500 Pixel_x = 108 Domain_x = 1.00000154972076420000 1.000001518750000 Pixel_x = 109 Domain_x = 1.00000154972076420000 1.000001532812500 Pixel_x = 110 Domain_x = 1.00000154972076420000 1.000001546875000 Pixel_x = 111 Domain_x = 1.00000154972076420000 1.000001560937500 Pixel_x = 112 Domain_x = 1.00000154972076420000 1.000001575000000 Pixel_x = 113 Domain_x = 1.00000154972076420000 1.000001589062500 Pixel_x = 114 Domain_x = 1.00000154972076420000 1.000001603125000 Pixel_x = 115 Domain_x = 1.00000166893005370000 1.000001617187500 Pixel_x = 116 Domain_x = 1.00000166893005370000 1.000001631250000 Pixel_x = 117 Domain_x = 1.00000166893005370000 1.000001645312500 Pixel_x = 118 Domain_x = 1.00000166893005370000 1.000001659375000 Pixel_x = 119 Domain_x = 1.00000166893005370000 1.000001673437500 Note the difference in granularity. The third column changes with each different integer input, whereas the MinGW results are almost an average of the third-column result over their range, and change only after an integer difference of 9. I do a great deal of CFD and numerical modeling work, and the third-column results are the kind of precision I'm used to seeing from floating point on the PC. I hope that helps to better illustrate the problem. Any ideas? Regards, Rich Wagner |