Re: [SoX-users] Convolution, channel-selectiveness of effects and spectrograms

[Dr. T. T. <t....@gm...>]

Hello Martin, hello Sergei,


Sergei's observation can also be analyzed as follows:

We are looking for DFT bin counts n, where cos(0) - cos(1/n) is
less than the machine precision (indicated by the number of bits in
the mantissa).  In that case cos(1/n) does not differ enough from
cos(0) so that the difference is zero (this is referred to as
"numerical cancellation" by the way).

For floats the machine precision m is 2^(-24), for doubles it
is 2^(-53).

So we have:
     cos(0) - cos(1/n) = 1 - cos(1/n) < m
     ==> n > 1/arccos(1 - m)

For floats this leads to n > 2896, for doubles n > 67108864.

So Sergei is correct: a bin count of 4096 is too big for float and
a bin count of 2^27 is too big for double.

But to be honest, I have not yet understood how this cancellation
propagates within the DFT/FFT algorithm and whether it invalidates
the complete transformation or only part of it.


	Best regards,

		Thomas