Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#2493 erf inaccurate for small bigfloat values

closed
nobody
5
2012-11-14
2012-11-14
Raymond Toy
No

erf(1b-20) -> 1.387778780781446b-17

The correct answer is closer to 1.128379167095513b-20.

This is caused by bfloat-erf computing erf using 1-gamma_incomplete(1/2,x^2)/sqrt(%pi). For small x, gamma_incomplete(1/2,x^2) is very close to sqrt(%pi), so we lose lots of precision using this formula.

For small x, we should just use the taylor series.

Discussion

  • Raymond Toy
    Raymond Toy
    2012-11-14

    Oops. This is duplicate of 3587191.

     
  • Raymond Toy
    Raymond Toy
    2012-11-14

    • status: open --> closed