Re: [Libvisual-devel] VisAudio overhaul proposal.

[Dennis S. <sy...@yo...>]

On Tue, 2005-03-08 at 16:23 +0200, Vitaly V. Bursov wrote:
> On Tue, 08 Mar 2005 14:26:46 +0100
> Dennis Smit <sy...@yo...> wrote:
> 
> > On Tue, 2005-03-08 at 15:15 +0200, Vitaly V. Bursov wrote:
> > > I think it would be nice to have an "audio filter" stack, tree or
> > > whatever and some "control" api (on/off, value from a range)
> > 
> > Audio filter stack ? what do you mean ?
> Well, Actor gets not "raw" stream but processed by filters.
> Including channel mixers, speed up/down, pitch, voice remove, etc.
> It would be a nice place to put a resampler too.
> 
> Ofcourse some data should be grabbed from a VJ app if there's one.
> 
> Just an idea...

This is out of the scope for just libvisual, however a friend of
mine is semi into audio, and I was talking to him about a lib on top of
libvisual for audio processing. So that nicely fits :)

> > > I don't think it's good idea to use fixed single frequency...
> > > Anyway, does it matter?
> > 
> > Well the more we generalise the less the Actor plugin has to think.
> > And keep in mind, more than one actor can connect to one VisAudio.
> Yeah, probably you're right.

I was thinking, we could store samples in a different type,
VisAudioSample, timestamp it there, connect a type to it.

And then make it possible to go from VisAudio to a few common types
where while converting stuff is put together, (we can store a history of
samples).

> > > > 	How large do we want our buffers ?
> > > Large buffer means high latency. Small buffer (upto 64 bytes) measns low
> > > latency.
> > 
> > True, how many samples of how many bytes are pushed out a second ? :)
> If sampling rate is 44100, you'll get 44100*2channels*16bit = 176400 bps  :)
> so, with a 2048 sample buffer there will be (2048*2channels*16bit)/176400
> ~= 46 msec lag if a frame is rendered in tenth of a millisecond. Or
> around 21 "real" FPS.

Ok so we prolly want to fetch with buffers of around 256 ? and combine 
multiple samples together if the plugin needs a larger worker array ?

(Thanks for the calculus btw :))

> > > Personally I believe buffer should be variable in size. It will be
> > > useful if we're going to use "audio filters".
> > 
> > What do you want to do with these audio filters ?, what we could
> > also do is take a non naieve approach, where a processed VisAudio
> > is rather a VisAudioPool that negotiates with the different
> > actors. However this does not make things easier, and I think we should
> > keep goal focused, proof me wrong tho :)
> 
> Well, I'm not sure... 

What about one sample array, from which plugins can ask a array version 
themself ?, something like:

In VisAudio:

        VisAudioSample         *samples; // We might want to make 
				this a rotating buffer... anyway just 
				for concept
        int                     samples_count;


VisAudioSample {
        VisObject                       object;

        VisAudioSampleFormatType        format;

        VisTime                         timestamp;

        sampledata...
        origdata...
}


void visual_audio_convert_to (VisAudio *audio, VisAudioSampleFormatType
format, void *data, int size);


> =====
> AudioSource (N channels, interleaved or not)
>     ||       ||
>     \/       ||
>   Filter1    ||
>    |     \   ||
>    |      \  ||
>    v       v \/
>  Filter2  Filter3
>    |         |
>    v         v
>  Actor1    Actor2
> =====
> 
> or, if there's a VJ app
> =====
> AudioSource (N channels, interleaved or not, better not)
>    Ch1      Ch2   ...   ChN
>     |        |
>     v        v
>   Actor1   Actor2
> =====

Yes that setup makes a lot of sense. Tho for the real connection of
this I think we really should introduce 'another' new lib :)

Anyway my focus isn't there atm, however, we need to keep open
the possibility.

> > > What about audio frame timestamping?
> > 
> > GOOD one, putting that on the list. We need that anyway, to use it for
> > the Beat Detection. Maybe we should keep a general sample history,
> > of which a [3][size] buffer can be derived ?
> Hm, I still think we should have small and variable buffer size.
> buffer size can specify how much data is available(no guarantee
> that frame proccessing will take exactly N usec);
> small buffer means low latency;
> all data should be processed to detect beats accurately.

Agree. I am incorporating all the comments in a new version
of the draft.

> It's virtually impossible to sync a/v if libvisual has no control
> over an audio stream (i.e. nobody can delay it - live music).
> The only solution here is a small buffer.

Yep. Since we can use a sample history with the new design, we might
want to add a 'delay' control that keeps back newer samples for
that amount of msecs ? Or would it be better to implement this
as a plugin within an audio lib on top ?

Cheers,
Dennis