Re: [Libxtract-devel] libxtract, SC UGens and FFT

[Dan S. <dan...@el...>]

Hi Jamie,

Jamie Bullock wrote:
> Hi Dan,
> 
> Great to know you're still interested in LibXtract! I've just posted the
> official 0.4.2 release tarball on sf.net, so it's a good time for you to
> start the SC coding!

Jolly good!

I've just committed a small fix: my first commit for libxtract, I think, 
and very minor. The placeholders for FFT-based functions when compiled 
without FFT support weren't quite right.

By the way, I've lost track: have you got MFCCs working to your 
satisfaction?


> On Mon, 2007-03-12 at 10:17 +0000, Dan Stowell wrote:
>> The main thing I want to do right now is FFT-based feature extraction. I 
>> would much rather use SC's existing FFT infrastructure, rather than have 
>> libxtract "repeat" the FFT (mainly for efficiency's sake). So, for 
>> example, to use libxtract's flatness measure it would look something like:
>>
>>    chain = FFT(buf, sig);
>>    flatness = Flatness.kr(chain);
>>
>> The problem with this is that libxtract wants to see the FFT data in a 
>> different format. So the Flatness UGen would need to convert the FFT 
>> data to a list of magnitudes + a list of frequencies. Again, for 
>> efficiency it would be bad for each ugen to repeat this conversion, so 
>> I'm thinking it may be best to create a UGen which does the conversion, 
>> which the feature extractors can hang off the end of.
>>
> 
> This is pretty much what I _used_ to do in PD, I used PD's internal
> fft~, took the mag spectrum, rearranged the data and then fed it into
> the xtract~ objects. I actually don't do this now, because the
> efficiency gain is so minimal on a fast computer, and to me it doesn't
> outweigh the convenience of just doing [xtract~ spectrum] and getting
> the results how they are needed. Using libxtract to get the spectrum
> also has the advantage that in the (unlikely) event that I change the
> data format again you (theoretically) shouldn't need to alter your code
> because I will maintain compatibility between the various functions. 
> 
> The story is different of course if you are doing large FFT's say
>> =2048. Then I guess there will be more of an advantage to doing the FFT
> only once. Perhaps I should do some benchmarks for this...
> 
>> If for example it's called "Mags", the process would go something like this:
>>
>>    chain = FFT(buf, sig);
>>    Mags(chain, magbuf); // "magbuf" will be filled with data in 
>> libxtract-style format
>>    flatness = Flatness.kr(chain, magbuf); // "chain" is needed so we 
>> know when to fire, but "magbuf" contains the data that will be processed
>>
>> It might seem an odd intermediate stage from an SC user's point of view, 
>> but actually the Mags UGen would be quite useful - it would have the 
>> side effect of loading the magnitudes into a buffer in a form that could 
>> easily be accessed and manipulated from the lang side.
>>
> 
> Yes that makes sense to me, and again, it's approximately how I used to
> do things in PD with libxtract, and its predecessor flib. You just need
> to be careful about the data format expected by the target. I'm sure you
> are already aware of this, but don't forget to check the header files
> (xtract_scalar.h and xtract_vector.h) to determine what the function
> expects. The useful thing in 0.4.x is that the magnitudes are always in
> the first half of any spectral-type array, so all you need to change for
> functions that only need the magnitudes is the array size 'N'.
> 
> Perhaps your SC implementation should include an (optional), arraysize
> argument, so:
> 
> chain = FFT(buf, sig);
> Spectrum(chain, specbuf); //specbuf has N/2 amps and N/2 freqs
> flatness = Flatness.kr(chain, specbuf, N/2); 
> 
> This would make things more flexible, but maybe you only want to
> implement a subset of libxtract, and the functions you want only take
> the mags...

I'd prefer the user not to have to specify buffer size. The allocated 
buffer size is a known quantity in SC so I can access that, and assume 
that the buffer contains data in libxtracty format (N/2 mags, N/2 
freqs), whether that's an ordinary spectrum or a peak spectrum.


>> I'd appreciate your thoughts on this. I'll probably be doing some coding 
>> this week along these lines.
>>
> 
> If the SC implementation were analogous to the PD one, then you would do
> it like:
> 
> chain = FFT(buf, sig);
> Spectrum(chain, specbuf); 
> flatness = Xtract.kr(flatness, chain, specbuf, N/2, argv); //argv
> corresponds to the argv in the function def.
> 
> It is a lot less work for the developer because you can just iterate
> over the descriptors array to get the expected data types etc. but I can
> see that it is not so user friendly as the much nicer Flatness.kr,
> Irregularity.kr etc. approach.

Yes, I appreciate that a direct approach would be easiest to develop and 
maintain but I do want the UGens to be comprehensible to others! As well 
as fitting with SC's general way of doing things.

It might however, be straightforward to create an "Xtract.kr" and then 
add some pseudo-ugen conveniences such as "Flatness.kr" that in reality 
just call on Xtract.kr (and constructs the N and argv arguments 
appropriately). Not sure if that really would be a pleasant way around 
but it's worth thinking about. I will think...

Thanks
Dan