Re: [Libvisual-devel] VisAudio overhaul proposal.

[Vitaly V. B. <vit...@uk...>]

On Tue, 08 Mar 2005 14:26:46 +0100
Dennis Smit <sy...@yo...> wrote:

> On Tue, 2005-03-08 at 15:15 +0200, Vitaly V. Bursov wrote:
> > I think it would be nice to have an "audio filter" stack, tree or
> > whatever and some "control" api (on/off, value from a range)
> 
> Audio filter stack ? what do you mean ?
Well, Actor gets not "raw" stream but processed by filters.
Including channel mixers, speed up/down, pitch, voice remove, etc.
It would be a nice place to put a resampler too.

Ofcourse some data should be grabbed from a VJ app if there's one.

Just an idea...

> > > 	Audio energy registration, calculation (already done).
> > > 	normalized spectrum (logified).
> > > 	Internally, use floats to represent audio. (from 0.0 to 1.0 or -1.0 to
> > > 1.0 ?)
> > > 	Use floats to represent the spectrum (from 0.0 to 1.0)
> > > 	Have conversion code to go from short audio to float.
> > there's K7 (semi) optimized verions within scivi
> 
> That is neat!, will look at it.
that's plugin.c, "USE_ASM_K7" def

> > I don't think it's good idea to use fixed single frequency...
> > Anyway, does it matter?
> 
> Well the more we generalise the less the Actor plugin has to think.
> And keep in mind, more than one actor can connect to one VisAudio.
Yeah, probably you're right.

> > > 	How large do we want our buffers ?
> > Large buffer means high latency. Small buffer (upto 64 bytes) measns low
> > latency.
> 
> True, how many samples of how many bytes are pushed out a second ? :)
If sampling rate is 44100, you'll get 44100*2channels*16bit = 176400 bps  :)
so, with a 2048 sample buffer there will be (2048*2channels*16bit)/176400
~= 46 msec lag if a frame is rendered in tenth of a millisecond. Or
around 21 "real" FPS.

or something like that. :)

> > Personally I believe buffer should be variable in size. It will be
> > useful if we're going to use "audio filters".
> 
> What do you want to do with these audio filters ?, what we could
> also do is take a non naieve approach, where a processed VisAudio
> is rather a VisAudioPool that negotiates with the different
> actors. However this does not make things easier, and I think we should
> keep goal focused, proof me wrong tho :)

Well, I'm not sure... 

=====
AudioSource (N channels, interleaved or not)
    ||       ||
    \/       ||
  Filter1    ||
   |     \   ||
   |      \  ||
   v       v \/
 Filter2  Filter3
   |         |
   v         v
 Actor1    Actor2
=====

or, if there's a VJ app
=====
AudioSource (N channels, interleaved or not, better not)
   Ch1      Ch2   ...   ChN
    |        |
    v        v
  Actor1   Actor2
=====

> > What about audio frame timestamping?
> 
> GOOD one, putting that on the list. We need that anyway, to use it for
> the Beat Detection. Maybe we should keep a general sample history,
> of which a [3][size] buffer can be derived ?
Hm, I still think we should have small and variable buffer size.
buffer size can specify how much data is available(no guarantee
that frame proccessing will take exactly N usec);
small buffer means low latency;
all data should be processed to detect beats accurately.

> > How to guarantee (do our's best) to synchronize video with audio?
> > Hard video processing requires some time...
> 
> This is hard, I haven't put much thought into this, but timestamping
> should help right ? :)
I really don't know how to synchronize it :)

It's virtually impossible to sync a/v if libvisual has no control
over an audio stream (i.e. nobody can delay it - live music).
The only solution here is a small buffer.

-- 
Vitaly
GPG Key ID: F95A23B9