#2378 ZRTP crashes

[Anonymous]

Originally created by: privus...@gmail.com

What steps will reproduce the problem?
1. Make a ZRTP call between 2 CSipSimple clients running latest nightly build
2. After some random time (from a few seconds to a few minutes - at most) ZRTP crashes and the call hangs up. CS registers again immediately on the PBX.
3. Sometimes I get the infamous PJ_EEOF error too at seemingly random times. This could be related.

What is the expected output? What do you see instead?
I expect the call to continue until I hang it up, not suddenly dropping the call and unregistering then re-registering CS on the PBX.

What version of the product are you using? On what device / operating
system?

Samsung Galaxy S3 LTE (i9305) running LiquidSmooth JB 2.4 Official (Android 4.2.2) with CS latest nightly build (2239). I've also tested with earlier CS builds and the problem persists.

Please provide any additional information below.

Like I said above, I also have the PJ_EEOF error happen randomly, so this ZRTP crash could also be related to that.

I see in the logs tons of "zrtp_android.c  ZRTP warning message: Dropping packet because SRTP replay check failed!" messages until the call eventually drops.

It's not a network issue as the call works well until the crash and both clients are on the same LAN with a good connection.

This may or may not be due to running a custom ROM (which is very stable) and/or Android 4.2.2, as evidenced in issues 2316 and 2280 (which is now closed?).

I'm running Freeswitch (very recent build) in bypass media mode so FS shouldn't even touch the RTP, so I think that is not the problem.

Here are the logs. http://pastebin.com/bgQiiuaU

[Anonymous]

Originally posted by: privus...@gmail.com

Can someone running Android 4.2.2 confirm that ZRTP is working OK for them so that I can try to narrow down the cause of the problem?

Here are the relevant messages in the logs:

18:58:04.275 zrtp_android.c  jzrtp_getInfoFromContext : user data 7288d334
18:58:04.281 pjsua_jni_addo  Get secure for media type 1
18:58:04.282 zrtp_android.c  jzrtp_getInfoFromContext : user data 7288d334
18:58:04.282 pjsua_jni_addo  ZRTP :: V 1
18:58:04.282 pjsua_jni_addo  ZRTP :: S L 4
18:58:04.282 pjsua_jni_addo  ZRTP :: C L 12
18:58:04.282 zrtp_android.c  jzrtp_getInfoFromContext : user data 7288d334
18:58:04.300 pjsua_jni_addo  Setup android capturer for all calls
18:58:04.329 strm0x72b6c274 !Jitter buffer starts returning normal frames (after 35 empty/lost)
18:58:04.355 zrtp_android.c !ZRTP warning message: Dropping packet because SRTP replay check failed!
18:58:04.360 pjsua_jni_addo !Setup android capturer for all calls
18:58:04.368 pjsua_jni_addo  Setup android capturer for all calls
18:58:04.760   sound_port.c !EC activated
18:58:07.428 zrtp_android.c !ZRTP warning message: Dropping packet because SRTP replay check failed!
18:58:07.525 zrtp_android.c !ZRTP warning message: Dropping packet because SRTP replay check failed!
18:58:08.257 zrtp_android.c  ZRTP warning message: Dropping packet because SRTP replay check failed!
18:58:08.565 zrtp_android.c  ZRTP warning message: Dropping packet because SRTP replay check failed!

As you can see, although ZRTP seems to succeed, it immediately starts dropping packets because the SRTP replay check failed. Sometimes the call drops right away, other times it drops after a few minutes.
I'm hoping someone out there can shed further information on this.

Thanks

[Anonymous]

Originally posted by: r3gis...@gmail.com

Can you try with the fullOpenSSL version available here and tell me how it goes :
http://nightlies.csipsimple.com/specific_builds/CSipSimple-fullOpenSSL.apk

Thanks !

Status: Need-Details

[Anonymous]

Originally posted by: privus...@gmail.com

Hi regis,

Thanks for your quick reply.
I just tested with the OpenSSL build and the issue remains.
Logs show the "SRTP replay check failed" messages, same as before. I can send them to you if you like. No call has lasted even 30 seconds before it gets dropped.
On the plus side, it seems that indeed the PJ_EEOF error referenced in issue 2280 has been resolved with the use of the OpenSSL library.

Let me know how I can help resolve this issue.

[Anonymous]

Originally posted by: privus...@gmail.com

This may be relevant so I post it here for future reference.

Given that there are many people reporting issues with ZRTP crashing after updating to Jelly Bean, I speculate that the problem may be related to some security changes in 4.1 and higher.

http://android-developers.blogspot.pt/2013/02/security-enhancements-in-jelly-bean.html
http://stackoverflow.com/questions/13389870/android-4-2-broke-my-aes-encrypt-decrypt-code

[Anonymous]

Originally posted by: privus...@gmail.com

Well I've done some experimenting with a Sony Xperia running stock Android 4.0.4 and the full OpenSSL.apk referenced above by Regis.

Unfortunately the exact same errors and dropped calls were observed, and the logs show the infamous "SRTP replay check failed" message.

This seems to indicate that this is not an Android Jelly Bean problem after all, but most likely a PJSIP bug with the ZRTP implementation.

Since this issue is likely upstream from CS, I imagine this would be better discussed in a PJSIP forum rather than here. I guess I'll have to look around and see if I can find the right place and bring this issue to their attention.

Unless someone has any better ideas?

[Anonymous]

Originally posted by: privus...@gmail.com

What further details are needed to classify this as a proper issue?

I have exchanged some info with Werner Dittmann (author of the PJSIP ZRTP implementation) on the PJSIP list with the title "possible bug in ZRTP implementation - SRTP replay check failed!".

Werner seemed to be keen on helping debug this issue but he hasn't responded in over 1 week, so I'm not sure if he's too busy or lost interest.

It would be great if others could confirm this bug with ZRTP so I can be sure it is not some strange behavior with my i9305 4.2.2 S3 phone. But since I got the same errors testing with the Xperia running stock 4.0.4, I still believe it is a CS/PJSIP/ZRTP integration bug.

Let me know what else I can do to help debug this very frustrating behavior. In practice it means that I cannot make any ZRTP calls.

[Anonymous]

Originally posted by: r3gis...@gmail.com

Issue 2399 has been merged into this issue.

[Anonymous]

Originally posted by: r3gis...@gmail.com

(No comment was entered for this change.)

Status: Accepted

[Anonymous]

Originally posted by: r3gis...@gmail.com

(No comment was entered for this change.)

Summary: ZRTP crashes

[Anonymous]

Originally posted by: htu...@gmail.com

With last nightly build which I have compiled has the same problem for many phones. After some random time call crashes. But with the stable CS in the market the problem did not occurred(But for samsung mini although call is not ended, ZRTP session is not started). And then I have checkout [r2230] (stable tag) and compiled, but the problem continues. I have check the my .so libs with the stable CS's.
my [r2230] compile
   libpj_opensl_dev.so   17,580 bytes
   libpjsipjni.so        2,000,540 bytes
   libstlport_shared.so  361,784 bytes
stable
   libpj_opensl_dev.so   21,740 bytes
   libpjsipjni.so        1,968,576 bytes
   libstlport_shared.so  378,224 bytes

Is this the problem about external repo zrtp,openssl?
Which version of zrtp,openssl did you used in stable version?

And also with my same nightly build I got "Can't load native library. CPU arch invalid for this build" error for HTC Wildfire S and  LG Optimus L5 Dual E612, although I have all the libs with all arch. But with the stable version CS works.

[Anonymous]

Originally posted by: privus...@gmail.com

Sorry to be a pain, but has there any progress at all on this bug?

Is anyone looking into this or is ZRTP basically useless at the moment?

[Anonymous]

Originally posted by: htu...@gmail.com

Now I could not look into this bug too much. In my case it is working for some devices and not working for other some devices now.

[Anonymous]

Originally posted by: privus...@gmail.com

Could you share exactly what device (and OS version) you have gotten ZRTP
working?

I've tried with S3 (i9305) running 4.2.2 with no success. I've also tried
the Sony Xperia running 4.0.4 with no success.

Also, anyone know which CS version broke ZRTP? I remember it seemed to work
well back in 2012, but I don't know which version exactly.

[Anonymous]

Originally posted by: htu...@gmail.com

For example, Samsung s3 mini with original CS and Sony Xperia U with compiled CS works. But calls from xperia U to s3 mini most of the time does not initiate ZRTP, but calls vice versa ZRTP initiates.

[Anonymous]

Originally posted by: js6439...@gmail.com

Have you guys tried the current CSipSimple nightly ([r2264])? [r2250] updated the ZRTPCPP library*.

I made a successful 15 minute ostel.co echo call with the new revision BUT haven't been able to try a real device-to-device call yet AND I can't remember if that echo call service was affected by the crash before.

* GNU ZRTP 3.2.0 changelog:

"The main ZRTP modules contain fixes for three vulnerabilities found by Mark
Dowd. Thus we advise application developers to use this version of the
library. The vulnerabilities may lead to application crashes during ZRTP
negotiation if an attacker sends prepared ZRTP packets. The fixes remove these
attack vectors.

Some small other enhancements and cleanup, mainly inside client code.

Some enhancements in cache handling and the handling of retained shared
secrets. This change was proposed by Phil, is a slight security enhacement and
is fully backward comaptible."

[Anonymous]

Originally posted by: q...@mt2014.com

I have the same issue. Only basic stable version 0.4..something is good. Even stable RC 1.0.0 is bad. I can say nighties are even worse then "stable" 1.0.0...trunk. I installed my own SIP server on the local machine, on the same lan, the same switch, the same wifi router. All devices are within the subnet mask 198.168.1.*
CyanogenMod 10.1 4.2.2 on Galaxy SII i9100 and the same system on Galaxy Tab 2 3100.
ZRTP OFF = All is good
ZRTP - ON random crashes without any message.
For me it looks there is a try {} catch{} somewhere which prevents "force close" but is makes our connection diconnected. It can be a null pointer or a simple buffer index problem. I also tested on Galaxy Note 2 Cyanogen 10.1 4.2.2 and it also closes.

And bluetooth earpiece doesn't work. When you try to use BT headset you get only silence in all sound sources.

[Anonymous]

Originally posted by: q...@mt2014.com

>> The vulnerabilities may lead to application
>> crashes during ZRTP negotiation if an attacker
>> sends prepared ZRTP packets."

No no no. I installed my own http://www.myvoipapp.com/ SIP server to eliminate such possibility that provider sends malformed packets. First I tested CSipSimple with free online VoIp providers and I thought they disconnected ZRTP calls in purpose after they had discoverd ZRTP. But it was not true. This is a bug in software. Both phones connect to eachother IP to IP without further server intervention. It happens on isolated LAN.
The common thing is CyanogenMod 10.1 4.2.2

Ha! Only the oldest version 0.4... can set up ZRTP with the latest Jitsi.

[Anonymous]

Originally posted by: js6439...@gmail.com

qtqtqtqt...:

Nevertheless, the ZRTPCPP commit changed things around in an interesting area of code, so the latest CSipSimple nightly might fix the problem (however it is caused) by accident.

Like you, I use CyanogenMod 10.1 nightly / Android 4.2.2.

[Anonymous]

Originally posted by: q...@mt2014.com

I tested the latest trunk build from the /trunk dir. I made many tests an hour before I wrote my previous post. The same crash. ZRTP=OFF=OK, ZRTP=ON=fail regardless of other settings and SIP provider. I am also a programmer, I write in Java for Android, C for embedded and C++ so I know how to test software. The only operational version is 0.4... from the main page of the CSipSimple project and maybe from the Play store however I don't use google apps because I have the clean Cyanogen.
What I want to say more ... v. 0.4.. this working one has very poor sound quality. Noise (in bacground) is not cancelled but amplified. I tested many codecs, bitrates and sampling rates from 8 to 44kHz and echo canceler doesn't work as expected. This software is still far from being called stable.

[Anonymous]

Originally posted by: r3gis...@gmail.com

@qtqt... : you're right zrtp feature integration is not yet stable... and the purpose of this issue is to track down why zrtp crashes on some devices+some ROM.

Any help to improve that is welcome. The project is opensource and if you have skills to debug the problem feel free to join the effort !

v0.4 is obsolete and is running an old version of the ZRTP protocol. It's not a good solution to revert on this old version.

The fact that js64 asked to re-test with latest nightlies was a very good idea ! Because, indeed, the recent changes integrated to csipsimple about zrtp overflow problem could have been an explanation for the problem. If you have experience on C/C++ you should understand that overflow problems found in zrtp could have been the root problem of the crashes observed.
It seems it doesn't help thanks to your additional tests. Don't think we are doubting of the fact you did the tests the right way. The results of your tests are very interesting to confirm the problem was not solved by lasts ZRTPCPP fixes. But was good to try :)

Another point you have to be aware : android custom ROM such as cyanogen ROM are not always robust as well regarding android audio layer. I don't know if you already dig into this field, but it's a very big factor of problem when coming to android sound quality. The clock rate is far to be the only setting that could help (the audio troubleshooting options has to be tweaked to match what is expected by the driver implementation). It's usually not a problem from Cyanogen maker but more a problem with proprietary audio driver and weird implementation in internals of the drivers.

[Anonymous]

Originally posted by: r3gis...@gmail.com

Also, I'm just realizing I forgot to mention that on this issue. Current ZRTP integration is known to be not thread safe. So it's good idea to disable multiple media threads while testing. It should not have big impact if your network quality is good enough and using a good codec (that does not flood network), but maybe it can help... so good to try that too.

To do so, switch to expert mode (https://code.google.com/p/csipsimple/wiki/ExpertSettingMode?wl=en#General_settings) and go in settings > media > Threads count for media. If you see 2 here (usually means you have a dual core CPU on your phone), change the value to 1.

Let me know if it changes something, maybe could increase priority of making zrtp4pj thread safe.

[Anonymous]

Originally posted by: q...@mt2014.com

I have DDMS log from Eclipse from Galaxy S2 i9100 just after crash. (Cyanogen 10.1 4.2.2 the latest from get.cm) The file is too large to be posted here. i9100 crashed but the other phone on the second table had been thinking is still connected until now and never disconnected. I see its screen now i the call with ZRTP is in progress. But mine here already reconnected to the server.

[Anonymous]

Originally posted by: q...@mt2014.com

One comment:
CsipSimple was 1.00.00-trunk
I did not mess with options. I just configured ZRTP, connected to my local SIP server on the second windows XP, then I called from i9100 to tablet and i9100 simply disconnected the call after several seconds.

[Anonymous]

Originally posted by: r3gis...@gmail.com

Please send the logs by emails (better for readability of the logs -- see the HowToCollectLogs wiki page to get more info about how to do just like regular users).

Two points after fast analysis of your logs :
- Video is enabled. This is a pre-alpha feature and will surely not work with ZRTP. Try do disable that first.
- The logs you get are also typicals one from issue about multithreading when codec is bad (typically G711) and network is bad too. Try to ensure media thread count is 1. On the SGS2 which has 2 core the default is probably 2 media threads. So the indication given in comment #21 are worth to try.

[Anonymous]

Originally posted by: q...@mt2014.com

I did not enable video. Maybe the Video plugin is installed but I did not click anything
I have it connected now. What do you want me to do ?