[Hardmad-devel] imdct numeric precision

[Rafael A. de <raf...@ic...>]

=2D----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I was studding the numeric precision problem reported against the IMDCT=20
module.=20

The serious problems occur when some intermediate result overflows and beco=
mes=20
negative. I discovered that removing the optimisation proposed by Leslie=20
(multiplying the input of the DCT4 by 2 and dividing the output) the=20
precision on the average case decreases but the overflow become rare.

It is not clear if in real streams there are values that may generate an=20
overflow. Tomorrow I will contract the libmad development list to see if th=
ey=20
have some evidence on this fact.

I am closing the bug.

Rafael

=2D----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/xB7NLlrfGJ8JUHwRAkwLAJ9eu9p3yF3a7G8oenmO2VIE5Rmc6QCgsKNY
OHWNidWnyFK3jCeSHPhbO7Y=3D
=3D3Cal
=2D----END PGP SIGNATURE-----