[Lives-users] LiVES news 23 July 2012

[salsaman <sal...@gm...>]

Hi all,
it's been a while since the last LiVES news, so I thought it was worth
an update. Last month saw the release of version 1.6.2 which added
support for fast opening of hdv clips amongst other things. Since then
onlt two minor bugs have been reported: a problem with initial startup
when pulseaudio is not installed
(http://lives.svn.sourceforge.net/viewvc/lives/trunk/src/startup.c?r1=1446&r2=1509)
and a problem with filenames being truncated in the plugin interfaces
(http://lives.svn.sourceforge.net/viewvc/lives/trunk/src/paramspecial.c?r1=1447&r2=1510).

Now, some of you may recal that I am currently working for a Masters
in Applied IT, and my research project involves digital signal
processing in LiVES. I have just completed the first half of my course
and so it is a good time to mention my progress on that.

The idea behind the project is that LiVES should be able to process
other type of data as well as video, audio and subtitles. In order to
achieve this I have created some new types of realtime plugins.

Video analyser plugins:
- blank frame detector:
this plugin looks for frames where the luminosity (brightness) of all
the pixels is below a certain user-defined threshold value. As well as
the threshold, there is also a "count" parameter which defines how
many consecutive frames must pass this test. The plugin has a boolean
(int) output, which is triggered (TRUE or 1) when the conditions are
met.

- farneback motion detector:
this plugin uses openCV, in particular the Farneback dense motion
detection algorithm. The outputs are two float alpha channels, the x
and y values for each pixel indicating its motion vector.

- haar analyser:
this plugin performs a Haar transform (a kind of 2 dimensional Fourier
transform), on YUV version of the input frames. The outputs are 3
arrays of integer (one for each plane Y,U,V). The array length can be
set by the user. This type of plugin can be used to classify similar
looking images.


Data processing plugins:
data can be processed without touching video or audio or subtitles. I
created an experimental plugin which linked to Weka (a java based data
processing tool). However this plugin was a little complicated to get
installed and running (due to use of java JNI), so it is isabled by
default. However the code can be found in the "experimental"
subdirectory of weed-plugins in case anybody wants to experiment with
linking LiVES with java.


Data visualiser plugins:
- cairo vector visualiser:
this plugin takes two float alpha channel inputs and overlays arrows
on the video frame. This may be used as the output for the Farneback
analyser for example.


Audio generator plugins:
- tone generator:
this plugin takes two float inputs - a frequency (Hz) value, and a
multiplier value (which can be used to scale the frequency). It
generates a square wave at the selected frequency.


Audio analyser plugins:
- audio_fft plugin:
a work-in-progress, this plugin takes the audio and outputs a float
amplitude for a selected frequency.

- beat detector:
this has been requested on here before, but it will use FFT and
statistical methods to detect beats in the audio. It has two outputs:
- a beat pulse output, and a beat & hold output. There is also a reset
input which will reset the beat & hold output. This is so that video
effects can log the beat from this filter (which runs at audio speed),
and then reset.



Still to come:
Audio processing plugins:
I hope to be able to create a wrapper around some LADSPA plugins to
enable realtime audio effects.



So, as you can see this work is coming on quite well in the
background. The next phase will be to create a graphical interface
which will allow the data connections to be made within LiVES (most
probably in the realtime effect mapper). The details for this are
still up in the air, so if anybody has any good suggestions we can
discuss this on the lives-devel mailing list. And of course, it should
be possible to connect and disconnect data pipes through the LiVES OSC
interface.

Other ideas which I would like to implement if time allows:
- chaining of plugins into a single effect slot (for example motion
detector -> averager -> frame shifter could give a nice camera shake
reduction filter).
- more plugins: - speech to text, text to speech, music generator,
"kicker" plugin (for output from the beat detector).


If anybody is interested in knowing more about this, please feel free
to ask. I am open to suggestions if anybody has any ideas related to
this.


Regards,
Gabriel "salsaman", main developer, LiVES.








http://lives.sourceforge.net
https://www.ohloh.net/accounts/salsaman