Re: [Libxtract-devel] libxtract, SC UGens and FFT

[Jamie B. <ja...@po...>]

Hi Dan,

Great to know you're still interested in LibXtract! I've just posted the
official 0.4.2 release tarball on sf.net, so it's a good time for you to
start the SC coding!

On Mon, 2007-03-12 at 10:17 +0000, Dan Stowell wrote:
> The main thing I want to do right now is FFT-based feature extraction. I 
> would much rather use SC's existing FFT infrastructure, rather than have 
> libxtract "repeat" the FFT (mainly for efficiency's sake). So, for 
> example, to use libxtract's flatness measure it would look something like:
> 
>    chain = FFT(buf, sig);
>    flatness = Flatness.kr(chain);
> 
> The problem with this is that libxtract wants to see the FFT data in a 
> different format. So the Flatness UGen would need to convert the FFT 
> data to a list of magnitudes + a list of frequencies. Again, for 
> efficiency it would be bad for each ugen to repeat this conversion, so 
> I'm thinking it may be best to create a UGen which does the conversion, 
> which the feature extractors can hang off the end of.
> 

This is pretty much what I _used_ to do in PD, I used PD's internal
fft~, took the mag spectrum, rearranged the data and then fed it into
the xtract~ objects. I actually don't do this now, because the
efficiency gain is so minimal on a fast computer, and to me it doesn't
outweigh the convenience of just doing [xtract~ spectrum] and getting
the results how they are needed. Using libxtract to get the spectrum
also has the advantage that in the (unlikely) event that I change the
data format again you (theoretically) shouldn't need to alter your code
because I will maintain compatibility between the various functions. 

The story is different of course if you are doing large FFT's say
>=2048. Then I guess there will be more of an advantage to doing the FFT
only once. Perhaps I should do some benchmarks for this...

> If for example it's called "Mags", the process would go something like this:
> 
>    chain = FFT(buf, sig);
>    Mags(chain, magbuf); // "magbuf" will be filled with data in 
> libxtract-style format
>    flatness = Flatness.kr(chain, magbuf); // "chain" is needed so we 
> know when to fire, but "magbuf" contains the data that will be processed
> 
> It might seem an odd intermediate stage from an SC user's point of view, 
> but actually the Mags UGen would be quite useful - it would have the 
> side effect of loading the magnitudes into a buffer in a form that could 
> easily be accessed and manipulated from the lang side.
> 

Yes that makes sense to me, and again, it's approximately how I used to
do things in PD with libxtract, and its predecessor flib. You just need
to be careful about the data format expected by the target. I'm sure you
are already aware of this, but don't forget to check the header files
(xtract_scalar.h and xtract_vector.h) to determine what the function
expects. The useful thing in 0.4.x is that the magnitudes are always in
the first half of any spectral-type array, so all you need to change for
functions that only need the magnitudes is the array size 'N'.

Perhaps your SC implementation should include an (optional), arraysize
argument, so:

chain = FFT(buf, sig);
Spectrum(chain, specbuf); //specbuf has N/2 amps and N/2 freqs
flatness = Flatness.kr(chain, specbuf, N/2); 

This would make things more flexible, but maybe you only want to
implement a subset of libxtract, and the functions you want only take
the mags...

> I'd appreciate your thoughts on this. I'll probably be doing some coding 
> this week along these lines.
> 

If the SC implementation were analogous to the PD one, then you would do
it like:

chain = FFT(buf, sig);
Spectrum(chain, specbuf); 
flatness = Xtract.kr(flatness, chain, specbuf, N/2, argv); //argv
corresponds to the argv in the function def.

It is a lot less work for the developer because you can just iterate
over the descriptors array to get the expected data types etc. but I can
see that it is not so user friendly as the much nicer Flatness.kr,
Irregularity.kr etc. approach.

Hmmm.. maybe I should rethink the PD version...

Jamie