|
From: Steve C. <ste...@gm...> - 2010-06-04 16:48:42
|
On 06/04/2010 12:36 PM, Richard C. Wagner wrote: > Kein-Hong and Ben, > > Thanks for your responses. Kein-Hong wrote: > > >> A float has about 7 decimal digits of precision. For whatever >> reasons, your result has a 1 plus 'a very small number'. So, >> taking 8 digits of displayed result (since the '1' at the front >> doesn't 'contribute' a full 1 digits of precision): >> >> 1.0000014 >> 1.0000015 >> 1.0000017 >> >> Nothing wrong with the compiler or the machine. >> > and Ben added: > > >> There is something very wrong somewhere because if you are performing >> >> x / 640.0 * 9e-6 >> >> For 0.0<= x<= 640.0 >> >> Your results should be way less than 1. >> > Then Kein-Hong correctly deduced: > > >> Sounds like a throwaway test of some sort when zooming in failed >> (or turned blocky) due to lack of numeric precision. >> > > Sorry, my description of my test wasn't clear. The results I posted were > from a stand-alone, throw away test program that made the same calculation > that I'm using in my main program. The calculation is: > > Domain_x = ((Pixel_x / 640.0) * 9.0e-6) + 1.0 > > I've found that the problem is usually in the source code that *isn't* shown. Could you post a complete, self-contained, short program that illustrates the problem? |