Re: [Audacity-devel] Real Time Effects on the 'Road Map'?

[James C. <cr...@in...>]

Joshua Haberman wrote:

> I think it is a good idea to support background rendering for exactly
> the reasons you cite.  However, this should not get in the way of us
> implementing a true real-time effects system.  That's why it makes sense
> to think of them as separate things.

I agree that we want both.  Far from getting in the way of each other
I think that planning both together will lead to a better system.  My own
preference is to see background rendering implemented first, but I can
understand that other people may make the opposite choice.  Really
the 'order of implementation' ultimately comes down to the preferences 
of whoever actually puts the development time in.  At best a discussion
such as we are having now can only be to lobby for a point of view.

> In my view, the capability for real-time effects is not an enhancement
> to the existing effects system, it is a new system entirely.  What you
> propose could be a first incremental step to that, but I think we could
> do better.

Well, it actually looks as if it may be getting harder to see exactly 
what the differences in our points of view are.  I certainly don't think
that background-rendering is everything there is to real-time effects,
but they are a usable step towards them.

Getting down to the details of how we'd go about background 
rendering as an incremental step towards 'true real time', how about 
in the implementation of background rendering we offer a function 
in the Effect class called something like 'get-next-chunk'?  This 
function may itself call other get-next-chunk functions in other 
Effect class instances.   Crucially we make the chunk size a 
parameter of that function.

We make sure that the special case of a chunk size of one
sample is a valid value for the parameter.

The earliest trial implementations of rtv just uses this
get-next-chunk function, passing in a chunk size of one
sample.  As we get around to implementing rtv more
efficiently, we can special case the chunk-size-of-one
case to make it faster.  There are probably many clever 
optimisations that can be done.  What matters to me is 
whether they can be done incrementally or whether we 
will hit a wall where we suddenly have to abandon this 
approach and start over to get the level of performance 
or some other aspect that we want.  Currently I see an
incremental approach as being a good one.  I can't
see a wall ahead.  If someone can see a place where it
would break down, I'd like to know.

This is not to say that we MUST implement background
rendering first.  We could write code in the effects classes
to support a buffer-of-one approach in implementing
rtv - and then gradually extend onwards to background
rendering, if that's how someone actually doing the
development wants to tackle it.

> > > * How will effects chaining work?  Do we want to support making
> > >    complicated effects routing diagrams like this?
> > >
> > >                        Input
> > >                 +--------+-----------+
> > >              Low-pass            High-pass
> > >                 |                    |
> > >              Reverb-1            Reverb-2
> > >                 +------+-------------+
> > >                       Mix
> > >                        |
> > >                    Compressor
> > >                        |
> > >                     Output
> >
> > To my eyes this is a script.

> I don't think is very accurate to call this a script.  It is a signal
> graph, where signal flows from node to node and can be modified by each
> node.  Calling it a script implies that you perform a bunch of
> coarsely-grained operations in sequence.  What you want is a signal-flow
> system where each plugin sends its output to the next plugin in the
> graph.

Hmm.  If the 'chunk size' is the length of the whole of the input
waveform it really *IS* a script.  If the chunk size is one sample 
in size it *IS* a flow graph.  If we treat scripts and flow graphs as
completely distinct things we end up with two implementations
of what in my eyes is the same underlying object.  I can live with
calling the two extremes different things, as long as the underlying
implementation has a common base class in which we can put
methods that belong to both that we identify.  

To hammer the point home, if I haven't already, the flow graph
diagram could be presented like this:

Out = Compressor( Mix(
    Reverb-1( Low-pass(In)),
    Reverb-2( High-pass(In))
))

It is nothing like as pretty but it says the same thing.  It's an
*implementation choice* how this script is evaluated, i.e. what 
chunk size is used.  It's not inherent in the script.

> This kind of signal-flow graph is very common: this concept is at the
> core of gstreamer, JACK, the BeOS Media Kit, modular synths, etc.

My take is that we can add fancy graphics for flow-scripts later.
Personally I think we should allow developing the graphics to lag 
on the sound processing in this area.  We can get a lot of mileage 
out of rtv without the swish graphics for them.  We can do this 
incrementally.  We don't have to implement a raft of difficult 
code for editing and displaying before seeing any benefit from 
using them.

--James.