libxtract-devel Mailing List for LibXtract
Status: Alpha
Brought to you by:
postlude
Menu
▾
▴
libxtract-devel — Discussion and announcements relating to the LibXtract library
You can subscribe to this list here.
| 2007 |
Jan
|
Feb
|
Mar
(6) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|---|
|
From: Jamie B. <ja...@po...> - 2007-03-20 08:12:11
|
On Fri, 2007-03-16 at 16:56 +0000, Dan Stowell wrote: > A couple more MFCC questions, hope you don't mind: > > Windowing: the xtract_mfcc method takes time-domain data in but doesn't > seem to do any windowing, so I guess the windowing is expected to be > handled before the data reaches libxtract. Is that right? It's possible > that the filters somehow have the windowing implicit, but I'm guessing not. > Actually there was an error in the docs which I have now fixed. For xtract_mfcc() *data should point to an array of spectral magnitudes. The assumption throughout libxtract is that windowing and/or pre-filtering will be done by some external function. > Alloc'ing: I can see in the PD example how you're allocating memory for > the mel filter, assuming "getbytes" is a PD realtime-malloc-type-thing. > (Is t_float something PD-specific, or just the same as float?) > It's just a float, included (I think) for portability reasons. > One thing that I'm wondering about is the malloc in the xtract_mfcc() > method. I'm not sure why you malloc some new space to copy the data into > - you don't seem to do anything there except read the data. Would it be > safe to get rid of that malloc/memcpy/free? It might be a hangover from > an older way of doing things. > Yes, I think it's a hangover. I've removed it. > Similarly there's a malloc in the DCT routine, not sure why that's > needed. Maybe there's a reason for these that I've missed (thread safety?). > I've removed that too. I've haven't even started considering things like thread safety yet. best, Jamie |
|
From: Dan S. <dan...@el...> - 2007-03-16 16:57:41
|
>> By the way, I've lost track: have you got MFCCs working to your >> satisfaction? >> > > Yes, as far as I can tell, it works as it should, and the vamp-libxtract > plugin code is in sync with the 0.4.x API. You've probably noticed that > you need to use the FFTW functionality to use xtract_mfcc(). However, > the 'meat' of the MFCC implementation is in the filters, which are built > with xtract_init_mfcc(). In your SC implementation you could construct > the filters with libxtract, and then do the rest with yourself if you > don't want the 'bloat' of linking to fftw just for this. No, that's fine, I'm happy to let libxtract do the work on this bit... > Does SC have a DCT ugen? No, it doesn't. A couple more MFCC questions, hope you don't mind: Windowing: the xtract_mfcc method takes time-domain data in but doesn't seem to do any windowing, so I guess the windowing is expected to be handled before the data reaches libxtract. Is that right? It's possible that the filters somehow have the windowing implicit, but I'm guessing not. Alloc'ing: I can see in the PD example how you're allocating memory for the mel filter, assuming "getbytes" is a PD realtime-malloc-type-thing. (Is t_float something PD-specific, or just the same as float?) One thing that I'm wondering about is the malloc in the xtract_mfcc() method. I'm not sure why you malloc some new space to copy the data into - you don't seem to do anything there except read the data. Would it be safe to get rid of that malloc/memcpy/free? It might be a hangover from an older way of doing things. Similarly there's a malloc in the DCT routine, not sure why that's needed. Maybe there's a reason for these that I've missed (thread safety?). Best Dan |
|
From: Jamie B. <ja...@po...> - 2007-03-14 17:31:46
|
Hi Dan, On Tue, 2007-03-13 at 10:32 +0000, Dan Stowell wrote: > I've just committed a small fix: my first commit for libxtract, I think, > and very minor. The placeholders for FFT-based functions when compiled > without FFT support weren't quite right. > Thanks, there was an old function name in there as well, so I've just changed that. > By the way, I've lost track: have you got MFCCs working to your > satisfaction? > Yes, as far as I can tell, it works as it should, and the vamp-libxtract plugin code is in sync with the 0.4.x API. You've probably noticed that you need to use the FFTW functionality to use xtract_mfcc(). However, the 'meat' of the MFCC implementation is in the filters, which are built with xtract_init_mfcc(). In your SC implementation you could construct the filters with libxtract, and then do the rest with yourself if you don't want the 'bloat' of linking to fftw just for this. Does SC have a DCT ugen? Jamie |
|
From: Dan S. <dan...@el...> - 2007-03-13 10:33:39
|
Hi Jamie, Jamie Bullock wrote: > Hi Dan, > > Great to know you're still interested in LibXtract! I've just posted the > official 0.4.2 release tarball on sf.net, so it's a good time for you to > start the SC coding! Jolly good! I've just committed a small fix: my first commit for libxtract, I think, and very minor. The placeholders for FFT-based functions when compiled without FFT support weren't quite right. By the way, I've lost track: have you got MFCCs working to your satisfaction? > On Mon, 2007-03-12 at 10:17 +0000, Dan Stowell wrote: >> The main thing I want to do right now is FFT-based feature extraction. I >> would much rather use SC's existing FFT infrastructure, rather than have >> libxtract "repeat" the FFT (mainly for efficiency's sake). So, for >> example, to use libxtract's flatness measure it would look something like: >> >> chain = FFT(buf, sig); >> flatness = Flatness.kr(chain); >> >> The problem with this is that libxtract wants to see the FFT data in a >> different format. So the Flatness UGen would need to convert the FFT >> data to a list of magnitudes + a list of frequencies. Again, for >> efficiency it would be bad for each ugen to repeat this conversion, so >> I'm thinking it may be best to create a UGen which does the conversion, >> which the feature extractors can hang off the end of. >> > > This is pretty much what I _used_ to do in PD, I used PD's internal > fft~, took the mag spectrum, rearranged the data and then fed it into > the xtract~ objects. I actually don't do this now, because the > efficiency gain is so minimal on a fast computer, and to me it doesn't > outweigh the convenience of just doing [xtract~ spectrum] and getting > the results how they are needed. Using libxtract to get the spectrum > also has the advantage that in the (unlikely) event that I change the > data format again you (theoretically) shouldn't need to alter your code > because I will maintain compatibility between the various functions. > > The story is different of course if you are doing large FFT's say >> =2048. Then I guess there will be more of an advantage to doing the FFT > only once. Perhaps I should do some benchmarks for this... > >> If for example it's called "Mags", the process would go something like this: >> >> chain = FFT(buf, sig); >> Mags(chain, magbuf); // "magbuf" will be filled with data in >> libxtract-style format >> flatness = Flatness.kr(chain, magbuf); // "chain" is needed so we >> know when to fire, but "magbuf" contains the data that will be processed >> >> It might seem an odd intermediate stage from an SC user's point of view, >> but actually the Mags UGen would be quite useful - it would have the >> side effect of loading the magnitudes into a buffer in a form that could >> easily be accessed and manipulated from the lang side. >> > > Yes that makes sense to me, and again, it's approximately how I used to > do things in PD with libxtract, and its predecessor flib. You just need > to be careful about the data format expected by the target. I'm sure you > are already aware of this, but don't forget to check the header files > (xtract_scalar.h and xtract_vector.h) to determine what the function > expects. The useful thing in 0.4.x is that the magnitudes are always in > the first half of any spectral-type array, so all you need to change for > functions that only need the magnitudes is the array size 'N'. > > Perhaps your SC implementation should include an (optional), arraysize > argument, so: > > chain = FFT(buf, sig); > Spectrum(chain, specbuf); //specbuf has N/2 amps and N/2 freqs > flatness = Flatness.kr(chain, specbuf, N/2); > > This would make things more flexible, but maybe you only want to > implement a subset of libxtract, and the functions you want only take > the mags... I'd prefer the user not to have to specify buffer size. The allocated buffer size is a known quantity in SC so I can access that, and assume that the buffer contains data in libxtracty format (N/2 mags, N/2 freqs), whether that's an ordinary spectrum or a peak spectrum. >> I'd appreciate your thoughts on this. I'll probably be doing some coding >> this week along these lines. >> > > If the SC implementation were analogous to the PD one, then you would do > it like: > > chain = FFT(buf, sig); > Spectrum(chain, specbuf); > flatness = Xtract.kr(flatness, chain, specbuf, N/2, argv); //argv > corresponds to the argv in the function def. > > It is a lot less work for the developer because you can just iterate > over the descriptors array to get the expected data types etc. but I can > see that it is not so user friendly as the much nicer Flatness.kr, > Irregularity.kr etc. approach. Yes, I appreciate that a direct approach would be easiest to develop and maintain but I do want the UGens to be comprehensible to others! As well as fitting with SC's general way of doing things. It might however, be straightforward to create an "Xtract.kr" and then add some pseudo-ugen conveniences such as "Flatness.kr" that in reality just call on Xtract.kr (and constructs the N and argv arguments appropriately). Not sure if that really would be a pleasant way around but it's worth thinking about. I will think... Thanks Dan |
|
From: Jamie B. <ja...@po...> - 2007-03-13 09:24:52
|
Hi Dan, Great to know you're still interested in LibXtract! I've just posted the official 0.4.2 release tarball on sf.net, so it's a good time for you to start the SC coding! On Mon, 2007-03-12 at 10:17 +0000, Dan Stowell wrote: > The main thing I want to do right now is FFT-based feature extraction. I > would much rather use SC's existing FFT infrastructure, rather than have > libxtract "repeat" the FFT (mainly for efficiency's sake). So, for > example, to use libxtract's flatness measure it would look something like: > > chain = FFT(buf, sig); > flatness = Flatness.kr(chain); > > The problem with this is that libxtract wants to see the FFT data in a > different format. So the Flatness UGen would need to convert the FFT > data to a list of magnitudes + a list of frequencies. Again, for > efficiency it would be bad for each ugen to repeat this conversion, so > I'm thinking it may be best to create a UGen which does the conversion, > which the feature extractors can hang off the end of. > This is pretty much what I _used_ to do in PD, I used PD's internal fft~, took the mag spectrum, rearranged the data and then fed it into the xtract~ objects. I actually don't do this now, because the efficiency gain is so minimal on a fast computer, and to me it doesn't outweigh the convenience of just doing [xtract~ spectrum] and getting the results how they are needed. Using libxtract to get the spectrum also has the advantage that in the (unlikely) event that I change the data format again you (theoretically) shouldn't need to alter your code because I will maintain compatibility between the various functions. The story is different of course if you are doing large FFT's say >=2048. Then I guess there will be more of an advantage to doing the FFT only once. Perhaps I should do some benchmarks for this... > If for example it's called "Mags", the process would go something like this: > > chain = FFT(buf, sig); > Mags(chain, magbuf); // "magbuf" will be filled with data in > libxtract-style format > flatness = Flatness.kr(chain, magbuf); // "chain" is needed so we > know when to fire, but "magbuf" contains the data that will be processed > > It might seem an odd intermediate stage from an SC user's point of view, > but actually the Mags UGen would be quite useful - it would have the > side effect of loading the magnitudes into a buffer in a form that could > easily be accessed and manipulated from the lang side. > Yes that makes sense to me, and again, it's approximately how I used to do things in PD with libxtract, and its predecessor flib. You just need to be careful about the data format expected by the target. I'm sure you are already aware of this, but don't forget to check the header files (xtract_scalar.h and xtract_vector.h) to determine what the function expects. The useful thing in 0.4.x is that the magnitudes are always in the first half of any spectral-type array, so all you need to change for functions that only need the magnitudes is the array size 'N'. Perhaps your SC implementation should include an (optional), arraysize argument, so: chain = FFT(buf, sig); Spectrum(chain, specbuf); //specbuf has N/2 amps and N/2 freqs flatness = Flatness.kr(chain, specbuf, N/2); This would make things more flexible, but maybe you only want to implement a subset of libxtract, and the functions you want only take the mags... > I'd appreciate your thoughts on this. I'll probably be doing some coding > this week along these lines. > If the SC implementation were analogous to the PD one, then you would do it like: chain = FFT(buf, sig); Spectrum(chain, specbuf); flatness = Xtract.kr(flatness, chain, specbuf, N/2, argv); //argv corresponds to the argv in the function def. It is a lot less work for the developer because you can just iterate over the descriptors array to get the expected data types etc. but I can see that it is not so user friendly as the much nicer Flatness.kr, Irregularity.kr etc. approach. Hmmm.. maybe I should rethink the PD version... Jamie |
|
From: Dan S. <dan...@el...> - 2007-03-12 10:18:52
|
Hi Jamie, Sorry you haven't heard much from me re libxtract. Would still like to use it, and I'm currently thinking about how best to use it in SC UGens. The main thing I want to do right now is FFT-based feature extraction. I would much rather use SC's existing FFT infrastructure, rather than have libxtract "repeat" the FFT (mainly for efficiency's sake). So, for example, to use libxtract's flatness measure it would look something like: chain = FFT(buf, sig); flatness = Flatness.kr(chain); The problem with this is that libxtract wants to see the FFT data in a different format. So the Flatness UGen would need to convert the FFT data to a list of magnitudes + a list of frequencies. Again, for efficiency it would be bad for each ugen to repeat this conversion, so I'm thinking it may be best to create a UGen which does the conversion, which the feature extractors can hang off the end of. If for example it's called "Mags", the process would go something like this: chain = FFT(buf, sig); Mags(chain, magbuf); // "magbuf" will be filled with data in libxtract-style format flatness = Flatness.kr(chain, magbuf); // "chain" is needed so we know when to fire, but "magbuf" contains the data that will be processed It might seem an odd intermediate stage from an SC user's point of view, but actually the Mags UGen would be quite useful - it would have the side effect of loading the magnitudes into a buffer in a form that could easily be accessed and manipulated from the lang side. I'd guess that it should be possible to create an analogous UGen using the peak-picker, to make it easy to apply the same measures but only to the located peaks. That's not my focus though. I'd appreciate your thoughts on this. I'll probably be doing some coding this week along these lines. Dan -- Dan Stowell Centre for Digital Music Queen Mary, University of London Mile End Road, London E1 4NS http://www.elec.qmul.ac.uk/department/staff/research/dans.htm http://www.mcld.co.uk/ |
