#373 Sound track and sound handling

[Pete]

Currently there is no way of syncronizing sound and animation in AoI. For that purpose I'm requestin a sound trak and sound management features to be added.

There are three basic function that should be possible:
#1 "Lip sync" to a pre-recorded sound
#2 Sycronizing sound effects to events on the animation
#3 Producing movies with sound

Case #1 simply involves the need to import a sound file and position it on the time line, where the "talk" is supposed to happen. It would also be necessary to place markers on the timeline that could be projected to the animation score to set keyframes of the animation correctly.

Case #2 means, that it should be possible to place several sound samples on the timeline according to the events on the animation. -- Say, sounds of walking steps or a bouncing ball, that can not necessarily be predicted before the animation is done. Here the workin direction of timemarkers should be opposite to case #1 -- from the animation tracks to the sound tracks. Then there should be a way to render and export the sample track(s) as a sound file for further editing.

Case #3 .. self evident I think ... :)

At minimum level there should be an editor* (score, mixer) for handling the sound tracks, but I think it would be very useful to have the possibilty to add soundtracks under objects as many sounds are related to objects. The possibility of placing sound tracks among animation tracks**, would facilitate editing greatly.

So basically I'm suggesting two userinterfaces: Sound tracks among animation tracks and a "master sound score" (maybe a specialiced editor window) to handle all soundtracks in one view. The master score could be expanded into to a sound editor, with the basic effects like delay/reverb, graphic/parametric EQ but I don't see those belonging to the minimum set of features.

On the score you could add sound tracks and name them appropriately ("walk", "talk", "car motor"). On each sound track you could add sound samples, multiply and place them where you need them. The samples should also visualize the soundwaves they are carrying, so that time markers could placed correctly. There also should be an easy way of locking/unlocking each sample (like using a handbrake) to avoid moving them accidentally.

On sound track level there should be at least volume control and a mute button. Preferably also pan/balance controls and a solo button. As an additional feature it would be great to be able control volume and pan/bal etc. settings by a control track of some kind. (And why not the volume of each sound sample internally as well...). Maybe the control track option could be visualized as a different display mode of the soundtrack?

When it comes to visualization of the sound samples, the most common way of doing it seems to be to convert (and scale) bytes directly into pixels. However sometimes a desibel based view would be very useful (though not a minimum req in this context), The volume (etc.) controls though should be desibel based.

Thinkin of the time markers, there are a few options that should be considered_ They could be independent of the sound- an animation tracks or they could be attached to a point in a sound sample or a keyframe of an animation. All these could be useful and I believe one possible way to use them would be to fix a sound sample with (a group of) keyframes so that they all would move in sync.

Actually the time markers and fixing keyframe seguences could be implemented as general features that don't need to be connected to any sound features. Also the possibilities of naming, labeling, colouring (or selecting the appearance or the markers) would be welcome. In preceding discussion the idea came out that on animation tracks the timemarkes could simpy appear as vertical lines and that it would be useful to have a snap-to-marker feature for placing keyframes.

The possibility of connecting sounds to objects could later on be expanded to "sound tracing" on the scene, so that distances, directions, obscuring or reflecting objects could be taken into account in the output sound. Naturally this would lead to the need of determinig acoustic propertied of 3D-objects too. However, for future, it might be a good idea to have an interface for 4-dimensional radiation patterns. That probably could be handled by a "sound object" where you could model the radiation patterns and attach sound tracks. By four dimensions I mean directions in space as function of frequency. Though this definitely does not belong to the minimum set of features. :)

On the output side it should be "wyhiwyg", so the sound should be rendered according to current volume, balance, mute etc, settings.

Latest discussion here:
http://sourceforge.net/p/aoi/discussion/47782/thread/1346fa8d/
http://www.friendlyskies.net/aoiforum/viewtopic.php?id=2895

[Pete]

I forgot to mention: If sound samples on one track overlap, there are a few options how to handle the situation. 1) Play both, 2) Play the latter one, 3) Crossfade or 4) Play the earlier. I think at least 1 and 2 should be available and the selection could be per track, not necessarily per sample.

EDIT: On second thought, the track should provide a default way of handling the overlap and each sample should have the possibility to override the deafult.

Example: Putting together a soundrack for a toy xylophone. If you hit the same key again while it is still ringing the later sample is the only one you should hear. If you hit a different key, they both of them ring. If the switch between the modes can not be done you end up using multiple sound tracks while otherwise it could be handled on just one.

[YardStick]

Just a comment:

I think the essential features would be
(1) A possibility to bring at least one soundfile (stereo 16-32bit/48-96kHz.wav - Maybe other formats like .mp3 too.) to the score.
(2) You should be able to see the waveform there, and it should scale with the score. That way you can time your keyframes with the sound.
(3) To do this you should be able to play the animation on edit mode in real time with sound.

(4) And maybe output rendered videos with sound.

What comes to sound editing, I believe the standard method, when there are several samples overlapping on same track, is to play the latter one. I haven't seen other options in any software I've used so far. (Could also be that I haven't been looking hard enough. Who knows.) And if sound editing and multiple samples on oone track is made possible, there should also be some kind of autocrossfade option to prevent pops and snaps between samples.

But in general I think the sound editing features in AoI should be kept to minimum. There are plenty of decent sound and video editing tools available for free, and even open source.

[Pete]

Thanks. Very good points there. I quite agree, that the basic functionality should be kept simple and easy to use.

Above I was pretty much collecting together all ideas, that have ever come to my mind, but most of those are just possiblities for futher development and maybe not that practical, when you come to think of it. ;)

[Konrad W.]

Hello,
I'm no programmer only an amateur user and fan of AoI just reading all this marvelous ideas, while I'm actually working on an animation along a soundtrack in cooperation to Klaus B..
I'd like to bring in two thoughts:
1.The pianoroll of a sequenzer (at least like up to date in Audacity, later on along with its midifile-sound) should be useable, too. Aside it could be able to present distinguishable instruments (with different coloring) it shows the precise separated ins, holds and outs of (midi) soundevents. Animating a piano (using also Audacity) I found it much more helpful than the corresponding wavefile.
2. Looking around I saw that 2D Synfig is able to work with Parrot (lip syncing software). Afaik Audacity and Parrot both are Open Source projects. Could there be established some kind of cooperation to AoI?

[Pete]

Could there be established some kind of cooperation to AoI?

Well there is at least the obvious obstacle, that Audacity and Parrot are
programmed in C (I can't tell what kind of C) and AoI is programmed in
Java. Though it certainly would be helpful to be able to build and sync a
soudtrack (and why not music as well) in the same tool with the animation.
-- Currently there are no tools for that in AoI at all.

It probably would not be an overwhelmingly difficult job to add at least a
very basic sound track capability to AoI. Then it could be extended by
plugins. That's at least what I was thinking with this feature request.

[Konrad W.]

Thanks for Your clearing reply, Pete. You are right: It's much more comfortable to have a built in solution for the Sound in AoI than only a 'link' to another program (perhaps with difficulties in translations). To open Audacity aside to AoI is already possible but not really convenient. So Your position is what I actually was thinking of, not knowing the problem of the different programming languages.
Edit: I have to beg Your pardon. I just saw the second programs name (for lipsynchronisation) is Papagayo and it's written in Python. This error of mine depends on being a native German, wherein 'Papagei' means parrot. But the problem with different programming languages isn't solved by that however.

[Konrad W.]

There is another sound program I've found recently, java based, open source and maybe an intersting partner to AoI: Frinika.