Re: [Audacity-nyquist] beat analysis plugin help

[Alex S. B. <ale...@al...>]

I worked to get a memory-safe version of beat detector a while back. I am
attaching my code as a sample. I am not exactly sure if it implements the
"moving window" that you are aiming for, but it should give you a better
starting point.

Here are some specific recommendations for your code, too.

> (setf sFn "c:\\temp3.wav")
> (setf s (s-read sFn))
An Audacity plug-in does not need to set "s". Audacity sets it to be the
currently selected music passage. If you are debugging in the Nyquist
environment, you need a statement like this to load a sound, but not in the
final plug-in.

You have a few calls that just gobble up memory. Some of the ones that jumped
out at me are snd-length and snd-array. These functions are fine to use
sometimes, especially just for a small section of a sound, but using them
across the whole sample causes Nyquist to load the whole sound into memory.
Because "s" is a global variable, the samples will probably never be "garbage
collected".  Look back on my threads about "silence detection" for some really
good background on this issue; I learned a lot from the folks in this forum.

Once you got all those samples, you assigned them to global array variables.
Then you created a global array for the entire set of average values. Each of
these actions allocates large chunks of memory, and they will never be garbage
collected. Because there are so many samples in the sound, the memory use gets
ugly very quickly.

If your program truly needed all that information in memory at once, you might
have to live with the high memory use. You really only need enough information
in memory to handle a few samples at a time. Loading the samples using "do"
loops and the "snd-fetch" command (see the attachment for a sample), and
Nyquist has the opportunity to garbage-collect all your unused samples. It will
only load the samples into memory one at a time, and it will garbage collect
the ones that are in the past. The downside of "snd-fetch" is that you can
never go backwards again -- it is destructive. But it is possible to do beat
detection this way.

Another good piece of advice is to find as many built-in Nyquist functions as
possible to do the work for you. I noticed that you use the averaging function
at one point, but then have your own custom code to calculate a series of
average energies over a window of time. The snd-avg function can be taken over
fixed-length windows, replacing your custom code.

You can also transform your sound using mathematical functions like expt, mult,
and so on. You can do a transformation of "s" to make it into an energy value,
then apply the snd-avg function to that tranformed "s". Again, the attached
plug-in uses this technique. You can even resample this tranformed sound
variable and do other sound-related functions that would be really
memory-consuming if you had to use arrays. Nyquist will only evaluate the sound
and its transformations as it needs to. If you iterate over the sound using
snd-fetch, it will also garbage collect all the information that it calcuated
for samples in the past. The result is a complex, but memory-efficient
plug-in.

A final minor note is that your code assumes that "s" is a stereo signal. You
can add code pretty simply to check whether it is mono or stero, and do the
proper transformations either way. Check out the snd-energy function in the
attached code.

If you really enjoy this stuff, I started working on an FFT-tranformed beat
detection algorithm, featured further down in the article you cited. That is a
real pain to figure out with arrays of arrays. I used some of the
object-oriented features of Nyquist to make that work, and I still have not got
it working yet. It does the job, but I do not consider the results usable for
most sound files. If you want to help with that one, send me a note privately.

Good luck. It looks like you fell into some of the same traps I did with the
memory use. Do not give up on Nyquist! With a little understanding, you can get
control of that problem without needing to drop into C++.

--Alex

Quoting Adam <ada...@ya...>:

> Hi - new to Nyquist (and LISP). Working on a beat
> detection plugin.  Code is here - would appreciate any
> help and pointers can get.  Think is commented fairly
> well.  But not working right and horrendous use of
> memory. Would appreciate *any* comments on it's
> working.  Based on article here -
> http://www.gamedev.net/reference/articles/article1952.asp
> 
> I'm trying to create a (more) intelligent "moving
> window" so the instant energy falls inside the average
> energy window properly.
> 
> The code *is* wrong - I've spent a few days on it and
> getting to grips with Nyquist but am on verge of
> writing something in C++ instead. Stop me.
> 
> ;nyquist plug-in
> ;version 1
> ;type analyze
> ;name "Beat Analysis Tool (BAT) DEBUG .5"
> ;action "Finding beats..."
> ;info "Beat Analysis Tool by Adam T.\nCreates beat
> pattern track. 
> ;control THRESHOLD "Threshold" real "c" 1.3 0.1 100.0
> ;control INSTANT_CHUNK_SIZE "Instant Num Samples" int
> "samples" 1024 128 4096
> 
> ; debug
> (setq THRESHOLD 1.2)
> (setq INSTANT_CHUNK_SIZE 1024) ; in samples
> ;;(setf sfn "H:\\temp2.wav")
> (setf sFn "c:\\temp3.wav")
> (setf s (s-read sFn))
> 
> ; get left & right channel sounds from the stereo
> array
> (setf sLeft (aref s 0))
> (setf sRight (aref s 1))
> 
> ; snd-srate works with sounds - not stereo arrays
> (setq sSampleRate (snd-srate sLeft))
> (display "sample rate" sSampleRate)
> (setq EAR_CHUNK_SIZE sSampleRate)  ; in samples
> 
> (setq nInstantChunksInEarChunk (/ EAR_CHUNK_SIZE
> INSTANT_CHUNK_SIZE))
> (display "num instant chunks in ear"
> nInstantChunksInEarChunk)
> 
> ; get number total samples in sound
> (setq nTotalSamples (snd-length sLeft NY:ALL))
> (display "total num samples" nTotalSamples)
> 
> ; convert channel sounds to arrays
> (setf array_sLeft (snd-samples sLeft nTotalSamples))
> (setf array_sRight (snd-samples sRight nTotalSamples))
> 
> ; number of instant energy chunks
> (setq nTotalInstantChunks (/ nTotalSamples
> INSTANT_CHUNK_SIZE))
> (display "total num instant chunks"
> nTotalInstantChunks)
> 
> ; create array for storing processed instant chunks
> (setq array_s_processed (make-array
> nTotalInstantChunks))
> (display "array created" array_s_processed)
> 
> ; process one instant energy block from individual
> samples and return 'e
> (defun instant-energy(asL asR offset)
>   (display "instant energy chunk" offset)
>   (do ((e 0.0) (i 0) (v0 (aref asL offset)) (v1 (aref
> asR offset))) ((= i (- INSTANT_CHUNK_SIZE 1)) e)
>     (setq a (expt v0 2))
>     (setq b (expt v1 2))
>     (setq e (+ e (+ a b)))
>     (incf i)
>     (setq v0 (aref asL (+ offset i)))
>     (setq v1 (aref asR (+ offset i))) ))
> 
> ; average ear energy persistence throughout sound
> (defun avg-ear(s) (snd-avg s (truncate (/ srate
> INSTANT_CHUNK_SIZE)) 1 OP-AVERAGE))
> 
> (setq E_ear 0.0)
> 
> ; fetch a moving ear persistence window and return 'E
> (defun get-ear-window(offset)
> ; (display "get-ear-window" offset)
>    ;get pre-num_instant_chunks to sum
>   (setq nPreChunks (truncate (/
> nInstantChunksInEarChunk 2))) ; tend to mid-point
>   (if (> nPreChunks offset) (setq nPreOffset offset)
> (setq nPreOffset (- offset nPreChunks)))
> ;   (display "PreOffset" nPreOffset)
>   
>   ; TODO: compensate for not enough history (look
> forward in buffer)
>   
>    ;get post-num_instant_chunks to sum
>   (setq nPostChunks (truncate (/
> nInstantChunksInEarChunk 2))) ; tend to mid-point
>   (if (> (+ nPostChunks offset) nTotalInstantChunks)
> (setq nPostOffset nTotalInstantChunks) (setq
> nPostOffset (+ offset nPostChunks)))
> ;  (display "PostOffset" nPostOffset)
>   
>   ; TODO: compensate for not enough future data (use
> more history)
>   
>   ;sum instant chunks
>   (setq E 0.0)
>   (do ((i nPreOffset) ) ((= i nPostOffset) E)
>     ; lookout for Nil values in the array
>     (setq e_instant (aref array_s_processed i))
>     (setq E (+ E e_instant))
>     ;(display "E" E)
>     (incf i)
>   )
>   
>   ;multiply by 1 / number of instant chunks in an
> *this* ear chunk (slightly dynamic)
> ; (display "PreOffset" nPreOffset)
> ; (display "PostOffset" nPostOffset)
>   (setq num_chunks (- nPostOffset nPreOffset ))
> ;  (display "num chunks" num_chunks)
>   (setq result (/ E num_chunks))
>   ;(display "ear-window result" result)
>  ; (return result)
>   (setq E_ear result)
> )
> 
> ; process and store entire sample array into instant
> energy chunks
> (defun process-samples ()
> ;  (display "process-samples" 0)
>   (do ((i 0) ) ((= i nTotalInstantChunks )
> array_s_processed)
>     (setq e (instant-energy array_sLeft array_sRight
> i))
>     (setf (aref array_s_processed i) e)
>     (incf i)
>     )
>   )
> 
> ; main
> (process-samples)
> ;(display "process-samples" 1)
> (setq labels Nil)
> ; time t, index i
> (do ((t 0.0) (i 0)) ((= i nTotalInstantChunks) labels)
>   ;(display "fetch" i)
>   (setq e_instant (aref array_s_processed i))
>   (get-ear-window i)
>   (setq c (* THRESHOLD E_ear))
>   (display "c" c)
>   (if (> e_instant c)
>       (setq labels (cons (list t "B") labels)))
>   (setq t (+ t (float (/ INSTANT_CHUNK_SIZE
> sSampleRate) )))
>   (setq e_instant (aref array_s_processed i))
>   (incf i)
>  )
> 
> ; return
> labels
> 
----------------------
Alex S. Brown, PMP
ale...@al...
http://www.alexsbrown.com/