Menu

#29 DPTXlator2ByteFloat: inaccurate precision

General
closed-fixed
nobody
None
7
2014-08-07
2014-07-23
No

DPTXlator2ByteFloat cannot represent the maximum and minimum values for this type.
Root cause seems to be in private method: fromDPT(). The accuracy when calculating the value is Float but might need to be Double.
The following test cases test the erroneous calculation: 0x7FFF should result in 670760.96 but results in 670760.94 and 0xF800 should result in -671088.64 but results in -671088.6.

Discussion

  • Boris Malinowsky

    • status: open --> closed-fixed
     
  • Boris Malinowsky

    Fixed in dev for v2.2.

     

Log in to post a comment.