sql-ledger-users

[SL] Rounding Error or Human Error?

[Luke <sl...@li...>]

Hello

I have an invoice which appears to be calculating the discount 
incorrectly.

The line item cost is $2.95 (sellprice--the list price is $2.99, and I 
lowered this on the invoice to compensate for another seeming error).

Setting the discount to 10, with a quantity of 12, yealds a cost of 
$31.80, while my calculater seems to think that it should equal $31.86.

SL 2.8.10; PGSQL 8.2.

Am I missing something obvious here?

Luke

Re: [SL] Rounding Error or Human Error?

[Luke <sl...@li...>]

More on this:

If I change the USD currency precision to 6, from its former setting of 2, 
the numbers come out right.  However, I am then left with totals like 
31.860000.  Changing the defaults precision setting seems to do nothing.

Is there any way to have calculations be done on a precision above 2, 
without displaying values with a precision above 2?

Or, better yet, have a final rounding be done for totals to a precision of 
2?

A calculation precision and a display precision, perhaps.

The problem seems to result from a calculation of the discount based upon 
the sellprice, not upon the (sellprice * quantity).

precision: 2
How I thought it was being done: 2.95 * 12 - 10% = 31.86
How it is being done: (2.95 - 10%) * 12 = 31.80

Luke


On Tue, 1 Jan 2008, Luke wrote:

> Hello
> 
> I have an invoice which appears to be calculating the discount 
> incorrectly.
> 
> The line item cost is $2.95 (sellprice--the list price is $2.99, and I 
> lowered this on the invoice to compensate for another seeming error).
> 
> Setting the discount to 10, with a quantity of 12, yields a cost of 
> $31.80, while my calculator seems to think that it should equal $31.86.
> 
> SL 2.8.10; PGSQL 8.2.
> 
> Am I missing something obvious here?
> 
> Luke
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> sql-ledger-users mailing list
> sql...@li...
> https://lists.sourceforge.net/lists/listinfo/sql-ledger-users
> 

Re: [SL] Rounding Error or Human Error?

[R P. <arc...@mt...>]

On Tuesday 01 January 2008, Luke wrote:
> Hello
> 
> I have an invoice which appears to be calculating the discount 
> incorrectly.
> 
> The line item cost is $2.95 (sellprice--the list price is $2.99, and I 
> lowered this on the invoice to compensate for another seeming error).
> 
> Setting the discount to 10, with a quantity of 12, yealds a cost of 
> $31.80, while my calculater seems to think that it should equal $31.86.
> 
> SL 2.8.10; PGSQL 8.2.
> 
> Am I missing something obvious here?
> 
> Luke

Hello Luke

Yup you are missing something...

This is because the total amount depends on where, in the process, the discount is applied (combined with the resulting rounding)

For example... if you calculate (and round if needed) the  line total.... and then calculate the discount on that line total...as you are doing...
(item price * quantity) * discount
2.95*12 = 35.40 (round to 35.40)
35.40 * 0.90 = $31.86

whereas SQL-ledger is applying the discount to the single item....rounding that to the nearest cent if needed..... and then generating a line total.
(This method makes sense since you would usually  discount the actual item price not the total)

$2.95 * 0.90 =$2.655  (round to $2.65)
2.65 * 12 = $31.80


Rounding can be a pain but unfortunately it's part of the hassle when you have math that doesn't come out to exact cents.


Cheers,
Roy Pennington

Re: [SL] Rounding Error or Human Error?

[R P. <arc...@mt...>]

On Tuesday 01 January 2008, Luke wrote:
> Hello
> 
> I have an invoice which appears to be calculating the discount 
> incorrectly.
> 
> The line item cost is $2.95 (sellprice--the list price is $2.99, and I 
> lowered this on the invoice to compensate for another seeming error).
> 
> Setting the discount to 10, with a quantity of 12, yealds a cost of 
> $31.80, while my calculater seems to think that it should equal $31.86.
> 
> SL 2.8.10; PGSQL 8.2.
> 
> Am I missing something obvious here?
> 
> Luke

Hello Luke

Yup you are missing something...

This is because the total amount depends on where, in the process, the discount is applied (combined with the resulting rounding)

For example... if you calculate (and round if needed) the  line total.... and then calculate the discount on that line total...as you are doing...
(item price * quantity) * discount
2.95*12 = 35.40 (round to 35.40)
35.40 * 0.90 = $31.86

whereas SQL-ledger is applying the discount to the single item....rounding that to the nearest cent if needed..... and then generating a line total.
(This method makes sense since you would usually  discount the actual item price not the total)

$2.95 * 0.90 =$2.655  (round to $2.65)
2.65 * 12 = $31.80


Rounding can be a pain but unfortunately it's part of the hassle when you have math that doesn't come out to exact cents.


Cheers,
Roy Pennington