[ojAlgo-user] Precision

[Andy T. <A.G...@le...>]

With functions how do I know what precision is returned and can a specific precision be requested?

Running the following code for example shows me that by default rootThousand0 is returned with a scale of 48 where as the rootTwo is returned with a scale of 52. Setting the scale of what is input seems not to have an effect.

        BigDecimal rootThousand0 = BigFunction.POW.invoke(
                BigMath.THOUSAND,
                BigMath.HALF);
        System.out.println("" + BigMath.THOUSAND + " precision " + BigMath.THOUSAND.scale());
        System.out.println("" + BigMath.HALF + " precision " + BigMath.HALF.scale());
        System.out.println("" + rootThousand0 + " precision " + rootThousand0.scale());
        System.out.println("Math.pow(1000.0d,0.5d) " + Math.pow(1000.0d,0.5d));

        BigDecimal thousand = BigMath.THOUSAND.setScale(100);
        BigDecimal half = BigMath.HALF.setScale(100);
        BigDecimal rootThousand1 = BigFunction.POW.invoke(
                thousand,
                half);
        System.out.println("" + rootThousand1 + " precision " + rootThousand1.scale());
        
        BigDecimal rootTwo = BigFunction.POW.invoke(
                BigMath.TWO,
                BigMath.HALF);
        System.out.println("" + rootTwo + " precision " + rootTwo.scale());
        System.out.println("Math.pow(2.0d,0.5d) " + Math.pow(2.0d,0.5d));


TIA for any help or guidance.

Andy
http://www.geog.leeds.ac.uk/people/a.turner/