Re: [WebERP-developers] Float accuracy problem

[rfthomas <rf...@as...>]

This is an old problem that is on all computer systems and actually predates
computer systems.  Floating point does not solve the problem, rather it will
make it worse.  Our company works with floating point calculations in
engineering and manufacturing.  Loss of precision is a major problem.  There
are numerous references explaining this problem in detail:

Numerical Mathematical Analysis - Scarborough
Computerized Accounting Methods and Controls - Tyran
Error Propagation for DIfference Methods - Henrici

Cobol and other accounting oriented languages normally utilize integers in
fixed formats to store numbers.

Order of operations, rounding and truncation all create specific problems. 
For example:

if the cost of an item is 3 for $1.00, then if one buys one the price is
$0.3333... Assuming that this unit price is stored to 2 places, i.e. $0.33
if a customer buys 3 pieces then the cost to the customer is $0.99 and the
business must absorb the $0.01 shortage.

There are no clean solutions.  One must use finite values and live with
results.  If rounding of the unit prices is performed then a consistent
policy needs to be established.

Floating point numbers should not be used since by their nature certain
values are difficult to store.  It is normal to see a value of 1. stored as
.9999999 if using Real*4 or .999999999999999 if using Real*8.

Storage is cheap and readily expanded.  Computers are very fast and the
numerical calculations simple.  We would prefer to expand fields and use
character or integer representations for all numbers.



--
View this message in context: http://weberp-accounting.1478800.n4.nabble.com/Float-accuracy-problem-tp4657899p4657949.html
Sent from the web-ERP-developers mailing list archive at Nabble.com.