Menu

#2 MathFP.div() seems to have problems

open-accepted
Math Engine (6)
5
2003-04-23
2003-03-28
No

Hi David,

I think I found a bug, but I'm not sure. Very easy to reproduce:

a.set( ".029558" );
b.set( ".29559" );
b.div( a );

b.toString() returns "0.690729009" instead of "10"!!!

I'm using MathFP 0.99b on:
- midp2.0fcs on Linux (Sun reference implementation)
- Java 1.4.1-01

I tried setting the precision to 12, but it doesn't change anything.

Any idea what could be the problem? Is there a work around?

Thanks

Hugo Frappier

Discussion

  • David DUTOUR

    David DUTOUR - 2003-04-23
    • status: open --> open-accepted
     
  • David DUTOUR

    David DUTOUR - 2003-04-23
    • assigned_to: nobody --> looky
     

Log in to post a comment.