[Audacity-nyquist] New "Silence Finder" Plug-In, Need Help on Memory Use (Alex S. Brown)

[Roger B. D. <rb...@cs...>]

Sorry for the length of this reply, but there are some subtle issues =
here...

> I believe that the biggest memory-use growth is in the "do" loop,=20
> not the initial SETF.  I calculated that for a 20-minute segment,=20
> the loop is going 1.2 million times, so just a 200 bytes of memo>ry=20
> use each round would lead to over 200 MB over the course of the =
function.=20
> Maybe that is part of the problem.

Memory use is subtle in Lisp and especially in Nyquist. When memory =
usage
increases, Nyquist runs the garbage collector, which will free anything =
that
is not being used. So normally, anything allocated in the do loop or
elsewhere will soon be freed and not cause any problems.

Problems occur when there are unintended references to data. Typically =
this
happens when you set a global variable to some signal value. Once you do
this, Nyquist is unable to free any of the signal's samples.

You might say, "so what, the signal is already in memory, so what harm =
does
it do to assign it to a global?" The answer is that signals are only
evaluated on demand. E.g. the variable "s" that Audacity passes to =
Nyquist
plug-ins is really just an object that is capable of producing samples =
(by
reading them from disk) when asked to do so by Nyquist. Ordinarily, =
Nyquist
will read a block of samples from "s", process those samples, deliver =
the
result to Audacity, and free any intermediate samples that were used in =
the
computation. Nyquist's runtime system figures out automatically that =
once a
block of samples from "s" have been processed, they are no longer needed =
and
can be freed.

If you assign "s" (or some function of "s") to a global, Nyquist's =
runtime
system assumes that you may want to use these samples in the future for
something else. It cannot free the samples as they are processed, and
instead appends them to a linked list in memory. When the garbage =
collector
runs, it sees that the huge list of samples are referenced by a (global)
variable and thus cannot be freed.

You are right that the do loop is involved -- the do loop is forcing =
Nyquist
to read "s" and reading "s" causes samples to be accumulated in memory. =
But
the trick is to let the garbage collection free samples after you read =
them.
SND-FETCH is good for this. It will read from the head of a sound and
advance the "reader" to the next sample, implying sequential access and
allowing the initial samples to be deallocated once they are read. SREF
allows random access to the sound, which means you are going to have =
some
variable pointing to the sound. As you SREF further and further into the
sound, you'll amass more and more samples in memory -- again, Nyquist =
cannot
know that you will not perform an SREF back to the beginning of the =
sound,
so all the sound's samples must be kept in memory. Nyquist will only =
read in
samples as you ask for them, so you would expect to see memory growing =
in
the do loop.