Menu
▾
▴
#72 Stale connection causing Halef to hang up at random
[java] 2015-08-19 01:07:18,350 DEBUG client.DefaultHttpClient [cbde02c5-a6ce-4839-be6e-56c04b9f5eef]: Stale connection detected
select distinct b.cairoCallId from haleflogs a, callsExt b where message like '%Stale connection detected%' and a.id between b.minId and b.maxId;
214 rows int set
This is
callID 53aa2d62497a65b3b9b173ac7dbae46d@141.31.8.62
jvxmlSessionId cbde02c5-a6ce-4839-be6e-56c04b9f5eef
Last edit: David Suendermann-Oeft 2015-08-19
There is an issue in that this call did not log anything into openvxmllogs. Whilenon-null calls most often do have entries in the openvxml table (87%):
create table cTmp as select cairoCallId from callsExt where jvxmlSessionId in (select sessionId from openvxmllogs);
select count(1) from completeCalls where callId in (select * from cTmp);
Calls not logging into openvxmllogs will always be null due to the missing code record:
select count(1) from completeCalls where callId not in (select * from cTmp) and code is null;
Oops. This is what I found in the Catalina logs:
INFO|2015/08/19|01:24:43|p=Deploy_Workflow|s=25C0AAF7FADE96F3218AA4751E5FE525|e=0001> completing key.toString()
org.mozilla.javascript.WrappedException: Wrapped java.io.IOException: Server returned HTTP response code: 500 for URL: http://141.31.8.161/openvxml_logger.php (<script>#52)<br> at org.mozilla.javascript.Context.throwAsScriptRuntimeEx(Context.java:1754)<br> at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:148)<br> at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:225)<br> at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1473)</p></script>
In the Catalina logs, I found evidence of 962 calls:
tjr@it-tjr18:~$ cat ./software/apache-tomcat-8.0.12/logs/catalina.out | grep '^sessionId' | sort -u | wc
962 1924 45183
and 20 HTTP 500 issues:
tjr@it-tjr18:~$ cat ./software/apache-tomcat-8.0.12/logs/catalina.out | grep 'Server returned HTTP response code: 500' | wc
20 260 3924
The HTTP 500 issue should be properly treated. It could even result in completed calls not being logged in the database since the code never gets stored.
BTW, according to the catalina logs, this seems to have been the reason for the premature hangup. Right after the JAVA IO exception, the following hang-up page is being called:
<vxml^m xmlns="http://www.w3.org/2001/vxml" version="2.1">^M
<form id="user-content-EndMessageForm" scope="document">^M</vxml^m>
<block name="EndMessageBlock">^M
<exit>^M
</exit></block>^M
</form>^M
<catch event="connection.disconnect.hangup">^M
<goto next="/7704/-/abort">^M
</goto></catch>^M
^M
Last edit: David Suendermann-Oeft 2015-08-19
I had a case (on August 3rd) where my call got stuck after I had responded to a prompt (but eventually recovered after ~10 sec) -- possibly because of a similar stale connection issue. Here is the corresponding JVXML log, in case it helps with debugging: