From: Henning T. <le...@he...> - 2012-07-25 11:27:14
|
I want to do advanced video processing in a functional way and wonder whether gstreamer can help me. I looked into VorbisPlay.hs example and it looks like I have to assemble a pipeline manually and there seem to be no types that can tell me what elements can be linked. Further I have no idea how to find the elements that do what I need or how I can say 'demux' without the need to know the input format. I also have no idea how to integrate custom audio or image processors. Are there more examples of how to use gstreamer that cover these issues? This is what I want to do: Task no. 1: I have recorded a talk both with a camera and the microphone of the speaker with audacity. Whenever someone asks a question in the auditory, the question shall be mixed into the target video. This is done by some custom signal processing by me called 'audioMerge'. Also the camera splits the movie into chunks of approximately 2 GB. Now I want to merge these parts into one video like this: merge1 duration (cameraStart, cameraAudioVideoParts) (microphoneStart, microphoneAudio) = let (cameraAudio, cameraVideo) = unzipAudioVideo $ drop cameraStart $ concat cameraAudioVideoParts in take duration $ zipAudioVideo (audioMerge cameraAudio $ drop microphoneStart microphoneAudio) cameraVideo Very Haskellish and very declarative, isn't it? Is it possible to write it that way with gstreamer? Task no. 2: I want to merge a camera recording and a screencast like this: merge2 (cameraStart, cameraAudioVideoParts) (screenStart, screenAudioVideo) = let (cameraAudio, cameraVideo) = unzipAudioVideo $ drop cameraStart $ concat cameraAudioVideoParts (screenAudio, screenVideo) = unzipAudioVideo $ drop screenStart screenAudioVideo in zipAudioVideo (audioMerge cameraAudio screenAudio) (pictureByPicture cameraVideo screenVideo) Additional problem is that the frame rate of the screen cast (10 fps) and the camera are different (25 fps) and I want to fill the screen cast with frames in order to match the frame rate of the camera. Also the audio clocks of both recording devices seem to mismatch such that I want to resample the audio stream of the screen cast. That is, what I really want to do is this: merge3 (cameraStart, cameraAudioVideoParts) (screenStart, screenAudioVideo) = let (cameraAudio, cameraVideo) = unzipAudioVideo $ drop cameraStart $ concat cameraAudioVideoParts (screenAudio, screenVideo) = unzipAudioVideo $ drop screenStart screenAudioVideo in zipAudioVideo (audioMerge cameraAudio $ adjustSamplerate screenAudio) (pictureByPicture cameraVideo $ adjustFramerate screenVideo) Best, Henning |
From: Conrad P. <co...@me...> - 2012-07-25 23:03:45
|
Hi Henning, I'm a bit rusty on the details, but there's components like playbin2: http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-playbin2.html which automatically builds an internal playing pipeline. IIRC there's a plugin called "spider" which can automatically build a pipeline from a given sink type to a given source type. cheers, Conrad. On 25 July 2012 18:50, Henning Thielemann <le...@he...> wrote: > > I want to do advanced video processing in a functional way and wonder > whether gstreamer can help me. I looked into VorbisPlay.hs example and it > looks like I have to assemble a pipeline manually and there seem to be no > types that can tell me what elements can be linked. Further I have no idea > how to find the elements that do what I need or how I can say 'demux' > without the need to know the input format. I also have no idea how to > integrate custom audio or image processors. Are there more examples of how > to use gstreamer that cover these issues? > > This is what I want to do: > > Task no. 1: I have recorded a talk both with a camera and the microphone of the > speaker with audacity. Whenever someone asks a question in the auditory, the > question shall be mixed into the target video. This is done by some custom > signal processing by me called 'audioMerge'. Also the camera splits the movie > into chunks of approximately 2 GB. Now I want to merge these parts into one > video like this: > > merge1 duration (cameraStart, cameraAudioVideoParts) (microphoneStart, microphoneAudio) = > let (cameraAudio, cameraVideo) = > unzipAudioVideo $ > drop cameraStart $ > concat cameraAudioVideoParts > in take duration $ > zipAudioVideo > (audioMerge cameraAudio $ > drop microphoneStart microphoneAudio) > cameraVideo > > Very Haskellish and very declarative, isn't it? > Is it possible to write it that way with gstreamer? > > > Task no. 2: I want to merge a camera recording and a screencast like this: > > merge2 (cameraStart, cameraAudioVideoParts) (screenStart, screenAudioVideo) = > let (cameraAudio, cameraVideo) = > unzipAudioVideo $ drop cameraStart $ concat cameraAudioVideoParts > (screenAudio, screenVideo) = > unzipAudioVideo $ drop screenStart screenAudioVideo > in zipAudioVideo > (audioMerge cameraAudio screenAudio) > (pictureByPicture cameraVideo screenVideo) > > > Additional problem is that the frame rate of the screen cast (10 fps) and > the camera are different (25 fps) and I want to fill the screen cast with > frames in order to match the frame rate of the camera. Also the audio > clocks of both recording devices seem to mismatch such that I want to > resample the audio stream of the screen cast. That is, what I really want > to do is this: > > merge3 (cameraStart, cameraAudioVideoParts) (screenStart, screenAudioVideo) = > let (cameraAudio, cameraVideo) = > unzipAudioVideo $ drop cameraStart $ concat cameraAudioVideoParts > (screenAudio, screenVideo) = > unzipAudioVideo $ drop screenStart screenAudioVideo > in zipAudioVideo > (audioMerge cameraAudio $ adjustSamplerate screenAudio) > (pictureByPicture cameraVideo $ adjustFramerate screenVideo) > > > Best, > Henning > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Gtk2hs-users mailing list > Gtk...@li... > https://lists.sourceforge.net/lists/listinfo/gtk2hs-users > |
From: <wag...@se...> - 2012-07-26 15:17:25
|
Gtk2Hs provides pretty low-level interfaces to the C bindings. If C can't do it... probably Haskell can't! I guess there's probably nothing stopping somebody from creating a slightly higher-level interface built on top of the basic gstreamer building blocks provided, but I don't know of any such that's already been made. On the other hand, the upside of this status is that it's usually quite easy to translate knowledge in the C documentation into knowledge about the Haskell bindings. =) ~d Quoting Henning Thielemann <le...@he...>: > > I want to do advanced video processing in a functional way and wonder > whether gstreamer can help me. I looked into VorbisPlay.hs example and it > looks like I have to assemble a pipeline manually and there seem to be no > types that can tell me what elements can be linked. Further I have no idea > how to find the elements that do what I need or how I can say 'demux' > without the need to know the input format. I also have no idea how to > integrate custom audio or image processors. Are there more examples of how > to use gstreamer that cover these issues? > > This is what I want to do: > > Task no. 1: I have recorded a talk both with a camera and the > microphone of the > speaker with audacity. Whenever someone asks a question in the auditory, the > question shall be mixed into the target video. This is done by some custom > signal processing by me called 'audioMerge'. Also the camera splits the movie > into chunks of approximately 2 GB. Now I want to merge these parts into one > video like this: > > merge1 duration (cameraStart, cameraAudioVideoParts) > (microphoneStart, microphoneAudio) = > let (cameraAudio, cameraVideo) = > unzipAudioVideo $ > drop cameraStart $ > concat cameraAudioVideoParts > in take duration $ > zipAudioVideo > (audioMerge cameraAudio $ > drop microphoneStart microphoneAudio) > cameraVideo > > Very Haskellish and very declarative, isn't it? > Is it possible to write it that way with gstreamer? > > > Task no. 2: I want to merge a camera recording and a screencast like this: > > merge2 (cameraStart, cameraAudioVideoParts) (screenStart, screenAudioVideo) = > let (cameraAudio, cameraVideo) = > unzipAudioVideo $ drop cameraStart $ concat cameraAudioVideoParts > (screenAudio, screenVideo) = > unzipAudioVideo $ drop screenStart screenAudioVideo > in zipAudioVideo > (audioMerge cameraAudio screenAudio) > (pictureByPicture cameraVideo screenVideo) > > > Additional problem is that the frame rate of the screen cast (10 fps) and > the camera are different (25 fps) and I want to fill the screen cast with > frames in order to match the frame rate of the camera. Also the audio > clocks of both recording devices seem to mismatch such that I want to > resample the audio stream of the screen cast. That is, what I really want > to do is this: > > merge3 (cameraStart, cameraAudioVideoParts) (screenStart, screenAudioVideo) = > let (cameraAudio, cameraVideo) = > unzipAudioVideo $ drop cameraStart $ concat cameraAudioVideoParts > (screenAudio, screenVideo) = > unzipAudioVideo $ drop screenStart screenAudioVideo > in zipAudioVideo > (audioMerge cameraAudio $ adjustSamplerate screenAudio) > (pictureByPicture cameraVideo $ adjustFramerate screenVideo) > > > Best, > Henning > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Gtk2hs-users mailing list > Gtk...@li... > https://lists.sourceforge.net/lists/listinfo/gtk2hs-users > > |