#386 Inconsistent application of NUMERIC DIGITS

open
5
2012-09-14
2012-09-14
Anonymous
No

In the example below, Regina 3.7 (and possibly pror versions) seems to return a result that is less precise than called for by the NUMERIC DIGITS specification.

numeric digits 40
>
two24 = 2**24
>
say two24
16777216
>
say 1/two24
5.9604644775390625E-8
>
say 1/3
0.3333333333333333333333333333333333333333

5.9604644775390625E-8 would be
0.0000000059604644775390625 give or take a zero but in any case is a shorter string than the NUMERIC DIGITS specification calls for.

Discussion


Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:





No, thanks