Re: [ojAlgo-user] Precision

[Anders P. <an...@op...>]

You should look at precision rather than scale.

If you look at the BigFunction source code you'll se that most of the implementations either delegate to BigDecimal or to PrimitiveFunction. When it delegates to PrimitiveFunction the precision is given by double otherwise MathContext.DECIMAL128 is used. You cannot control this.

Did you notice the RationalNumber class?

http://ojalgo.org/generated/org/ojalgo/scalar/RationalNumber.html

/Anders


On 14 apr 2010, at 17.50, Andy Turner wrote:

> With functions how do I know what precision is returned and can a specific precision be requested?
> 
> Running the following code for example shows me that by default rootThousand0 is returned with a scale of 48 where as the rootTwo is returned with a scale of 52. Setting the scale of what is input seems not to have an effect.
> 
>        BigDecimal rootThousand0 = BigFunction.POW.invoke(
>                BigMath.THOUSAND,
>                BigMath.HALF);
>        System.out.println("" + BigMath.THOUSAND + " precision " + BigMath.THOUSAND.scale());
>        System.out.println("" + BigMath.HALF + " precision " + BigMath.HALF.scale());
>        System.out.println("" + rootThousand0 + " precision " + rootThousand0.scale());
>        System.out.println("Math.pow(1000.0d,0.5d) " + Math.pow(1000.0d,0.5d));
> 
>        BigDecimal thousand = BigMath.THOUSAND.setScale(100);
>        BigDecimal half = BigMath.HALF.setScale(100);
>        BigDecimal rootThousand1 = BigFunction.POW.invoke(
>                thousand,
>                half);
>        System.out.println("" + rootThousand1 + " precision " + rootThousand1.scale());
> 
>        BigDecimal rootTwo = BigFunction.POW.invoke(
>                BigMath.TWO,
>                BigMath.HALF);
>        System.out.println("" + rootTwo + " precision " + rootTwo.scale());
>        System.out.println("Math.pow(2.0d,0.5d) " + Math.pow(2.0d,0.5d));
> 
> 
> TIA for any help or guidance.
> 
> Andy
> http://www.geog.leeds.ac.uk/people/a.turner/
> 
> 
> 
> ------------------------------------------------------------------------------
> Download Intel® Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> ojAlgo-user mailing list
> ojA...@li...
> https://lists.sourceforge.net/lists/listinfo/ojalgo-user
> 
>