#112 Support for video call

[Anonymous]

Originally created by: boss...@yahoo.fr

Hi all,
First of all I would like to say that yo have done a great job!
I'm developing an IMS/LTE (4G) client for Android (https://code.google.com/p/imsdroid/) and looking for client which support video call (H264 or Theora). I have tried SipDroid but it cannot receive video and only support H263+.
Do you plan to support video call in the coming releases?

[Anonymous]

Originally posted by: r3gis...@gmail.com

Yes it's planned but in a *far* future.

The main technical problem is that as I use a native stack that doesn't support video yet. So it has to be modified to support video and connected to the android video system.
So not so easy to do, and many things are remaining to support only voice and provide a good user experience.

But maybe if somebody hears you and want to start something on that point ... he would be welcome ... it's an opensource project ;) .

Labels: -Type-Defect -Priority-Medium Type-Enhancement Priority-Low

[Anonymous]

Originally posted by: boss...@yahoo.fr

Thanks for the quick response.
I'll stay tuned.

[Anonymous]

Originally posted by: r3gis...@gmail.com

Issue 173 has been merged into this issue.

[Anonymous]

Originally posted by: r3gis...@gmail.com

Quick update

Excellent news.... pjsip (the sip stack on which CSipSimple rely) support video media.

Now work for me... do a driver between android and pjsip :D

Too much things to do, but this is an exciting feature so I'll try to do that AZAP :)

Status: Accepted

[Anonymous]

Originally posted by: sahadeva...@gmail.com

Hi,

Provide me the steps for setting up CsipSimple for video call

Thanks in Advance

[Anonymous]

Originally posted by: r3gis...@gmail.com

As explained several time above CSipSimple does not support video yet.

There is a current effort on this task. If you want to join the development effort you'll be welcome !

Status: Started

[Anonymous]

Originally posted by: da...@styleflare.com

I am looking into this as well; the only thing I see as the "show" stopper is only h.263 encoding support; Does anyone know if there is a h.264 encoder?

I am not an expert at video but I did spend some time looking into the possibility;

The 2 problems I see are;

1. Video Support in the SIP stack;  (Solved)
2. Video Encoding (Solved if you want h.263)

I would be curious to see what other "projects/companies" are doing about this.

(Please update me if I am completely wrong about my assumptions;)

Thanks.

[Anonymous]

Originally posted by: r3gis...@gmail.com

> Video Support in the SIP stack;  << actually not fully yet...

In fact what is released by pjsip guys is for now only the pjmedia part. So no SDP nor higher level integration.

The pjmedia encoding part relies on ffmpeg. As consequence even if for now few encoders are integrated, would not be so hard to get more.

The biggest thing we are waiting for now is for the release of teluu for the integration with the rest of the stack.

The thing I'm trying to treat on my side and that is far to be an minor thing to do is the device glue to get and write in the android UI.

I know that there is somebody that is trying to hardcode some stuff inside the current pjsua to get some proof of concept. They'll commit as soon as they'll get something. But the driver for android has to be done before...

[Anonymous]

Originally posted by: boss...@yahoo.fr

I've never used pjsip but I can help adding video support by:
- Building and adding FFMpeg
- Developing these video codecs: h263,h264,theora and mp4-es
- Writing the java glue & UI (jni,surfaceview,camera management,...)
What should I do to contribute to the project?

[Anonymous]

Originally posted by: r3gis...@gmail.com

You're welcome :)

As for building and adding ffmpeg that's already done and available in the video branch of csipsimple.
As for the codecs if I don't mistake we will automatically benefit it from ffmpeg, isn't it?

For the java glue, if you want to try to do it, it will be very welcome.
To do so, try to checkout the video branch :
https://code.google.com/p/csipsimple/source/browse/#svn%2Fbranches%2Fvideo
What is missing is something there :
https://code.google.com/p/csipsimple/source/browse/#svn%2Fbranches%2Fvideo%2Fpjsip_android-2.0%2Fapps%2Fpjsip%2Fproject%2Fpjmedia%2Fsrc%2Fpjmedia-videodev
to make the glue with openGL (I guess that's the best choice to render video) and one with the android java API for cameras.

For now, pjsip guys has not yet finished everything that is missing between the pjsua (user agent) and their pjmedia library. However that's planned to be released soon as far as I understood :).

Meanwhile it should be possible to test the video dev using their unitary tests (for example the aviplayer.c project).

If you have a first patch for the video branch let me know I'll open you the svn write access to contribute ;)

[Anonymous]

Originally posted by: boss...@yahoo.fr

FFmpeg will give encoding/ decoding(avcodec), chroma conversion,scaling...(swscale) but it will be up to you to develop the rtp packetization/depacketization([rfc3984],draft-barbato-avt-rtp-theora-01,[rfc4629],...) unless you have it in avformat or pjsip.
OpenGL-es is a good idea but it's too heavy for a voip application where we only want to display the video stream "as-is". Using android "surfaceview" is easier even if it gives better performance (see main loop threading issue). The real benefit of opengl-es 2.0 could be the fact that you can draw yuv420 frames without converting them to rgb565 (to be checked). As you know the chroma conversion is one blocking point on some armv5te devices (at least 90% of the market). swscale doesn't really provide amrv5 optimized code for chroma conversion.

[Anonymous]

Originally posted by: r3gis...@gmail.com

As for RTP packetization, I think (but I'm absolutely not sure ;) ) that it is already provided by pjsip or will be ;) (anyway pjsip is the good place to put it and already provides some utilities functions for rtp packetization).

For OpenGL-ES vs surfaceview you have probably better knowledge than me on this field.
I was thinking about OpenGL-ES cause that's a "standard" and could be reused on other platforms than android.
Besides with the very bad experience I had with the audio java API (1 month of dev and still not working on all devices) and the excellent experience I had with OpenSL-ES (1 day of dev), I was thinking that OpenGL-ES could be easier but it was just a guess.
I was also thinking about all stuff that could be done through the GPU for image optimization (contrast/balance/effects/scaling). And maybe other stuff on hardware acceleration I'm not aware of.

Could be interesting to test both :). But, if you think it's easier to start with surfaceview I entirely trust you, as I never really tried nor opengl-es neither surfaceview :).

[Anonymous]

Originally posted by: ferna...@bugabundo.net

so pjsip 2.0 with video support soon?

[Anonymous]

Originally posted by: r3gis...@gmail.com

:)... http://twitter.com/#!/CSipSimple

And in csipsimple video branch there is the dev for the display (using opengl -- if boss has an implementation with surface view it will be welcome to bench both implementation).

As for capturing I started something too. For now I have preview. next step will be to push frames to native stack.

There is also something to do about converting frames from different color space. ffmpeg lib should be helpful, but I'd like to use the pjsip wrapper around ffmpeg libswscale.

I was not very reactive on the issue list all this week end and probably all the rest of the week cause of that point... :) Obviously I can't do support + dev at the same time ;) . So for now it's more dev than support. To the others that ask for help sorry if I'm not responsive but video is a very exciting feature to implement ;).

[Anonymous]

Originally posted by: ferna...@bugabundo.net

Make audio bulletproof, then code video.
Not that many front facing camera devices, and even less using it with sip

[Anonymous]

Originally posted by: r3gis...@gmail.com

Remember, I do csipsimple for my pleasure ;).
So if *I* don't do video it does not mean I'll do more stuff on audio.
It just mean that I spend time playing with some other stuff.

Other stuff can be video on csipsimple, but could have been completely different project on android or another project on something different than android. I also do some html5/css3/svg projects ;).

My personal roadmap are only based on what I want to do on my free time ;). If some company or user want to have a feature on one device... JUST CODE IT !!! CSipSimple is opensource !!!

[Anonymous]

Originally posted by: r3gis...@gmail.com

First video call done with csipsimple ;).

It was done between csipsimple on nexus one and my pc (ubuntu) running pjsip.
For now pjsip-2.0 only support (I think so) h263 CIF which result poor quality. As it relies on ffmpeg lib I guess other codecs will come shortly.

For now hard to say how quality will finally be using a good codec, but video flaw seems to be pretty good. As the N1 camera already support CIF size there is no scale operation on capture and as I use opengl-es to display it the resize of display in done by GPU (I guess...) so the only CPU work is to encode/decode codec + change pixel format.
Unfortunately, opengl-es 1 does not support pixel formats used in the codec I tested - however I'm wondering if for some conversion a pixel shader could not do the trick...

So this is a good news, however do not expect something for daily use soon.
pjsip-2.0 is still in dev and other things has to be done before having this landing. (Modular stuff (tls/video/codecs) in csipsimple for example).

If you want to build and test it it's available on the video branch of this svn. Build it yourself if you want to test : I've not yet checked all licenses of ffmpeg parts so I do not release anything. Besides as it require to build on pc it must be used by expert only for now... I don't want users to cry about the fact it's incomplete ;).

Of course, any contribution to the video branch will be welcome if you build and improve it :)

[Anonymous]

Originally posted by: ferna...@bugabundo.net

congrats!

[Anonymous]

Originally posted by: sahadeva...@gmail.com

Congratz!!! I have been waiting for this feature for some time :)

[Anonymous]

Originally posted by: ocean.ra...@gmail.com

congrats!!!I think this is the great news for this community.

[Anonymous]

Originally posted by: peterfri...@gmail.com

Well done 3R! Let's hope this becomes available soon...

[Anonymous]

Originally posted by: kishanra...@gmail.com

Hi. First off great work. I followed this thread and see that you were able to create a version that supports video. Do you know where I can get it? I would like to test video calling using my Nexus S. Thanks!

[Anonymous]

Originally posted by: r3gis...@gmail.com

@kishanra : you could try to build it from video branch. However that's still complicated to build.
I'll release soon on the market and then will focus more to provide a build of csipsimple with a real support of video. Pjsip project has now h264 support which would make video calls much more interesting than just QCIF h263 which I was able to test ;).

[Anonymous]

Originally posted by: Bart.Van...@gmail.com

Great news!  Keep up the good work!  Can't wait to test SIP video calls on my Nexus S!

[Anonymous]

Originally posted by: gasc...@gmail.com

For who is interested in compiling video branch these are the instructions:

download branches Video:
- svn checkout http://csipsimple.googlecode.com/svn/branches/video/ folder_name

Then download ffmpeg for Android from Bambuser project (http://bambuser.com/opensource)

Follow the instructions in the file README to compile the project ffmpeg-android of Bambuser.

Once compiled ffmpeg for Android, put the folder in

/home/gio/..../folder_name/pjsip_android-2.0/apps (at the same level of pjsip).

Replace the folders: swig_tools and apps in the pjsip of the trunk version and then follow the build instruction at: https://code.google.com/p/csipsimple/wiki/HowToBuild

Your build are only under armeabi-v7a and not armeabi

../CSipSimple/libs/armeabi-v7a

ps: when I run video branches under my htc desire with frojo I have the following error:

E/AndroidRuntime( 5177): java.lang.ExceptionInInitializerError
E/AndroidRuntime( 5177):     at com.csipsimple.pjsip.PjSipService.addAccount(PjSipService.java:519)
E/AndroidRuntime( 5177):     at com.csipsimple.service.SipService.addAllAccounts(SipService.java:977)
E/AndroidRuntime( 5177):     at com.csipsimple.service.SipService.reAddAllAccounts(SipService.java:1038)
E/AndroidRuntime( 5177):     at com.csipsimple.service.SipService.access$3(SipService.java:1035)
E/AndroidRuntime( 5177):     at com.csipsimple.service.SipService$1.reAddAllAccounts(SipService.java:155)
E/AndroidRuntime( 5177):     at com.csipsimple.wizards.BasePrefsWizard$4.run(BasePrefsWizard.java:320)

that is the same of the  issue 768 : https://code.google.com/p/csipsimple/issues/detail?id=768

I think the problem is my android phone. How can i compile video trunk also for armeabi and not only for armv7a? thanks in advance!