Menu ▾ ▴

#61 Hangup after first prompt / user input due to no received RTP stream

1.0
open
nobody
None
2015-11-04
2015-07-13
David Suendermann-Oeft
No

During the most recent data collection in the period July 4 to 6, the following 16 Ext 3333 calls terminated right after the first user input due to "No RTP stream received":

7ff0cd4ff09c9345d9ac1c74e31f889c@141.31.8.62
36a00b646edf8af3786657281598f2bc@141.31.8.62
7bd36fa2780a89275847a8284cf4d2ba@141.31.8.62
8c760ded627b5792a3df96190bcf92a4@141.31.8.62
21b1a8cab630a51f0c139cf284fdb747@141.31.8.62
802116f89f5d5eb6a665403232e1cddd@141.31.8.62
23936f70d4bf8c5a146e62a5902cec0d@141.31.8.62
bbdafef2e6bc79253304773ff9ecd86e@141.31.8.62
60b2fbeea831e8ccf4e1c31edfccb809@141.31.8.62
146585232b569ac56f96dcf567818026@141.31.8.62
196fd6a6e9298fdca580a92e9d7eb60a@141.31.8.62
1f3ca58420df44e0ff71a545ac198660@141.31.8.62
963efac2494a557615c0bd86d63cd756@141.31.8.62
dce5c3d6f5738ed9fe47a0d26fded6c9@141.31.8.62
6a9778e2997922ef458ca87ea5dde6e9@141.31.8.62
457c5548e958c8b618841fc00b098f2f@141.31.8.62

It shall be mentioned that these were all successive calls in a single batch and that the problem was resolved by restarting Halef.

This is a typical excerpt of the issue:

| 2877401 | 2015-07-05 01:20:49 | OK, processor was null |
| 2877402 | 2015-07-05 01:20:59 | java.lang.IllegalStateException: No RTP stream yet received! |
| 2877403 | 2015-07-05 01:20:59 | MrcpRequestProcessorImpl got response from request handler. |
| 2877404 | 2015-07-05 01:20:59 | WRITTEN:
MRCP/2.0 86 1436051568814 402 COMPLETE
Channel-Identifier:14e4d54b5eb@speechrecog

  |

| 2877405 | 2015-07-05 01:20:59 | Mrcpv2 invocation exception while initiating a recognition request |
| 2877406 | 2015-07-05 01:20:59 | caught JVoiceXML event while processing 'error.noresource'

Discussion

  • David Suendermann-Oeft

    David Suendermann-Oeft - 2015-07-13
    • Description has changed:

    Diff:

    --- old
+++ new
@@ -17,6 +17,8 @@
 6a9778e2997922ef458ca87ea5dde6e9@141.31.8.62
 457c5548e958c8b618841fc00b098f2f@141.31.8.62

+It shall be mentioned that these were all successive calls in a single batch and that the problem was resolved by restarting Halef.
+
 This is a typical excerpt of the issue:

 | 2877401 | 2015-07-05 01:20:49 | OK, processor was null                                                                               |
     

  • David Suendermann-Oeft

    David Suendermann-Oeft - 2015-07-13

    In the 337 calls logged in the DB so far, there are 29 affected by this issue:

    select callId, timestamp, duration, extension from calls where callid in (select callid from hli where message like '%no RTP stream%');

    +----------------------------------------------+---------------------+----------+-----------+
    | callId | timestamp | duration | extension |
    +----------------------------------------------+---------------------+----------+-----------+
    | 0d4cf04e5aa9d5c246bb4f08c3dcaadf@141.31.8.62 | 2015-06-16 16:24:00 | 24 | 5001 |
    | 546bc8e525a7aa359cdb560adf68f124@141.31.8.62 | 2015-06-16 16:32:24 | 23 | 5002 |
    | e858d353f8639096d5c5e0813a1920eb@141.31.8.62 | 2015-06-23 22:43:12 | 36 | 1010 |
    | b87e499323583987da09fbb5002d93a5@141.31.8.62 | 2015-06-23 22:44:06 | 39 | 1010 |
    | 2502e4b6e5063d30b4538f2d53ae9c7b@141.31.8.62 | 2015-06-23 22:45:45 | 35 | 1010 |
    | 73953515d5e77810b2cfb75887187d89@141.31.8.62 | 2015-06-24 02:01:56 | 39 | 1000 |
    | b8727f2a1d0a9faa8a17b845a4abe945@141.31.8.62 | 2015-06-24 02:03:31 | 41 | 1000 |
    | f9472f410534e6c8924b18939a724822@141.31.8.62 | 2015-06-24 02:07:53 | 40 | 1000 |
    | 08f0cfc3df2c4db7e318540e3ae07aea@141.31.8.62 | 2015-06-24 02:13:00 | 37 | 1000 |
    | 65effeb87d3a22233fb096b56d133d1e@141.31.8.62 | 2015-07-01 05:16:03 | 30 | 7624 |
    | e99b9ef45b0a707b666c31fded58aa7d@141.31.8.62 | 2015-07-02 14:04:32 | 32 | 2222 |
    | d91d94896edb32250af8f7090c825b4a@141.31.8.62 | 2015-07-02 14:04:32 | 32 | 2222 |
    | 457c5548e958c8b618841fc00b098f2f@141.31.8.62 | 2015-07-04 20:47:49 | 43 | 3333 |
    | 6a9778e2997922ef458ca87ea5dde6e9@141.31.8.62 | 2015-07-04 20:58:02 | 41 | 3333 |
    | dce5c3d6f5738ed9fe47a0d26fded6c9@141.31.8.62 | 2015-07-04 20:59:19 | 40 | 3333 |
    | 963efac2494a557615c0bd86d63cd756@141.31.8.62 | 2015-07-04 21:05:32 | 41 | 3333 |
    | 1f3ca58420df44e0ff71a545ac198660@141.31.8.62 | 2015-07-04 21:16:17 | 42 | 3333 |
    | 196fd6a6e9298fdca580a92e9d7eb60a@141.31.8.62 | 2015-07-04 21:17:08 | 41 | 3333 |
    | 146585232b569ac56f96dcf567818026@141.31.8.62 | 2015-07-04 21:35:25 | 37 | 3333 |
    | 60b2fbeea831e8ccf4e1c31edfccb809@141.31.8.62 | 2015-07-04 21:36:16 | 41 | 3333 |
    | bbdafef2e6bc79253304773ff9ecd86e@141.31.8.62 | 2015-07-04 21:39:28 | 42 | 3333 |
    | 23936f70d4bf8c5a146e62a5902cec0d@141.31.8.62 | 2015-07-04 21:43:17 | 42 | 3333 |
    | 802116f89f5d5eb6a665403232e1cddd@141.31.8.62 | 2015-07-04 21:59:09 | 41 | 3333 |
    | 21b1a8cab630a51f0c139cf284fdb747@141.31.8.62 | 2015-07-04 22:20:33 | 60 | 3333 |
    | 8c760ded627b5792a3df96190bcf92a4@141.31.8.62 | 2015-07-05 00:24:22 | 41 | 3333 |
    | 7bd36fa2780a89275847a8284cf4d2ba@141.31.8.62 | 2015-07-05 00:25:07 | 37 | 3333 |
    | 36a00b646edf8af3786657281598f2bc@141.31.8.62 | 2015-07-05 01:01:33 | 42 | 3333 |
    | 7ff0cd4ff09c9345d9ac1c74e31f889c@141.31.8.62 | 2015-07-05 01:20:59 | 44 | 3333 |
    | 91fbbc9c2e8f6abaec98bb211ba5d17d@141.31.8.62 | 2015-07-05 01:23:26 | 37 | 7709 |

     

  • Patrick L. Lange

    Patrick L. Lange - 2015-07-13

    This happend to me before. I thought this was related to a bad internet connection that day.

     

  • David Suendermann-Oeft

    David Suendermann-Oeft - 2015-07-14

    Hm, perhaps, but in the above case, a Halef restart resolved the problem then. Does this make sense?

     

  • Patrick L. Lange

    Patrick L. Lange - 2015-07-14

    Not really. So maybe the problem was not cause by my internet connection. Do you know the callId of the call that happend before this chain of NO RTP STREAM calls? Maybe something in there can help us find the problem.

     

  • David Suendermann-Oeft

    David Suendermann-Oeft - 2015-07-14

    Here are some observations about calls around no-RTP-stream-received issues. This is how you read the list:

    • a call ID followed by a date-time is a call affected by the no-RTP-stream-received issue
    • a line featuring the statement "VoiceXML interpreter 0.7.6.EA started" indicates a Halef restart
    • a call ID followed by a full-call recording file name is a call not affected by the no-RTP-stream-received issue

    You also find the last call before the long chain of affected calls below (call ID 6fa0b605da5c8caa1663bdf7b41448c5@141.31.8.62).

    Test call Patrick (issue with very slow grammar initialization)
    | 0d4cf04e5aa9d5c246bb4f08c3dcaadf@141.31.8.62 | 2015-06-16 16:24:00 | 24 | 5001 |
    | 1443430 | 2015-06-16 16:28:58 | 141.31.8.62 | VoiceXML interpreter 0.7.6.EA started !. |
    | 1900dbbc371d3adc196020b5e43c10b5@141.31.8.62 | HALEF_MixMonitor_audio_ext_5001_16062015_222937.wav | 2015-06-16 16:30:43 | 68 | 5001 | NULL |

    Test call DSO (issue with very slow grammar initialization)
    | 546bc8e525a7aa359cdb560adf68f124@141.31.8.62 | 2015-06-16 16:32:24 | 23 | 5002 |
    | 1450222 | 2015-06-16 16:37:13 | 141.31.8.62 | VoiceXML interpreter 0.7.6.EA started !. |
    | bdb74a6fcbffa65cf695b2c40bdcd6c6@141.31.8.62 | HALEF_MixMonitor_audio_ext_5001_17062015_053145.wav | 2015-06-16 23:32:53 | 65 | 5001 | NULL |

    Test call Vikram (scheduling item)
    | e858d353f8639096d5c5e0813a1920eb@141.31.8.62 | 2015-06-23 22:43:12 | 36 | 1010 |
    | b87e499323583987da09fbb5002d93a5@141.31.8.62 | 2015-06-23 22:44:06 | 39 | 1010 |
    | 2502e4b6e5063d30b4538f2d53ae9c7b@141.31.8.62 | 2015-06-23 22:45:45 | 35 | 1010 |
    | 1770109 | 2015-06-23 22:47:47 | 141.31.8.62 | VoiceXML interpreter 0.7.6.EA started !. |
    | f291315cf49ca7c7455605e4f164f64e@141.31.8.62 | HALEF_MixMonitor_audio_ext_1010_24062015_044800.wav | 2015-06-23 22:51:13 | 203 | 1010 | 7928 |

    Test call DSO (peanut item, likely test calls for Ticket 39)
    | 73953515d5e77810b2cfb75887187d89@141.31.8.62 | 2015-06-24 02:01:56 | 39 | 1000 |
    | b8727f2a1d0a9faa8a17b845a4abe945@141.31.8.62 | 2015-06-24 02:03:31 | 41 | 1000 |
    | 2136342 | 2015-06-24 02:05:21 | 141.31.8.62 | VoiceXML interpreter 0.7.6.EA started !. |
    | 326ad28df2ad95acd9c2f1d04a50b532@141.31.8.62 | HALEF_MixMonitor_audio_ext_1000_24062015_080541.wav | 2015-06-24 02:06:40 | 70 | 1000 | 4304 |

    Test call DSO (peanut item, immediately following, likely test calls for Ticket 39)
    | f9472f410534e6c8924b18939a724822@141.31.8.62 | 2015-06-24 02:07:53 | 40 | 1000 |
    | 2156116 | 2015-06-24 02:09:48 | 141.31.8.62 | VoiceXML interpreter 0.7.6.EA started !. |
    | 4b950e3aa4497cab1fff22ccdae46593@141.31.8.62 | HALEF_MixMonitor_audio_ext_1000_24062015_081010.wav | 2015-06-24 02:11:05 | 65 | 1000 | 7955 |

    Test call DSO (peanut item, immediately following, likely test calls for Ticket 39)
    | 08f0cfc3df2c4db7e318540e3ae07aea@141.31.8.62 | 2015-06-24 02:13:00 | 37 | 1000 |
    | 2182167 | 2015-06-24 02:15:10 | 141.31.8.62 | VoiceXML interpreter 0.7.6.EA started !. |
    | 8869e04f14b640712018c746a5480e0d@141.31.8.62 | HALEF_MixMonitor_audio_ext_1000_24062015_081526.wav | 2015-06-24 02:16:21 | 65 | 1000 | 7940 |

    | 65effeb87d3a22233fb096b56d133d1e@141.31.8.62 | 2015-07-01 05:16:03 | 30 | 7624 |
    ... followed by many calls, multiple of them with rand codes, none of them with RTP issue, including production calls
    | 2806435 | 2015-07-02 06:47:38 | 141.31.8.62 | VoiceXML interpreter 0.7.6.EA started !. |

    No recordings for the first calls, then test call Vikram (pizza item)
    | e99b9ef45b0a707b666c31fded58aa7d@141.31.8.62 | 2015-07-02 14:04:32 | 32 | 2222 |
    | d91d94896edb32250af8f7090c825b4a@141.31.8.62 | 2015-07-02 14:04:32 | 32 | 2222 |
    | 01b225052bd5f0995a8158094f74e75d@141.31.8.62 | HALEF_MixMonitor_audio_ext_7711_02072015_235930.wav | 2015-07-02 18:02:48 | 208 | 7711 | NULL |
    | 7471bc05822d64e7f9532bf7909a5c46@141.31.8.62 | HALEF_MixMonitor_audio_ext_7709_05072015_003139.wav | 2015-07-04 18:31:59 | 26 | 7709 | NULL |
    | e8bbcba16ef767cdfcd7d1c1a168dadd@141.31.8.62 | HALEF_MixMonitor_audio_ext_7709_05072015_003224.wav | 2015-07-04 18:36:41 | 268 | 7709 | 6055 |
    | e54a9e898a8b4f022f2a9d9dac8162d5@141.31.8.62 | HALEF_MixMonitor_audio_ext_7709_05072015_003708.wav | 2015-07-04 18:37:24 | 27 | 7709 | NULL |
    | 6911617fb25fb428b35a88f22ed4aa15@141.31.8.62 | HALEF_MixMonitor_audio_ext_3333_05072015_003808.wav | 2015-07-04 18:38:20 | 22 | 3333 | NULL |
    | 4bdbdee3ec165b69cb4f59d66384f390@141.31.8.62 | HALEF_MixMonitor_audio_ext_3333_05072015_014208.wav | 2015-07-04 19:42:27 | 48 | 3333 | NULL |
    | 6fa0b605da5c8caa1663bdf7b41448c5@141.31.8.62 | HALEF_MixMonitor_audio_ext_3333_05072015_014309.wav | 2015-07-04 19:48:23 | 330 | 3333 | 8393 |
    | 457c5548e958c8b618841fc00b098f2f@141.31.8.62 | 2015-07-04 20:47:49 | 43 | 3333 |
    | 6a9778e2997922ef458ca87ea5dde6e9@141.31.8.62 | 2015-07-04 20:58:02 | 41 | 3333 |
    | dce5c3d6f5738ed9fe47a0d26fded6c9@141.31.8.62 | 2015-07-04 20:59:19 | 40 | 3333 |
    | 963efac2494a557615c0bd86d63cd756@141.31.8.62 | 2015-07-04 21:05:32 | 41 | 3333 |
    | 1f3ca58420df44e0ff71a545ac198660@141.31.8.62 | 2015-07-04 21:16:17 | 42 | 3333 |
    | 196fd6a6e9298fdca580a92e9d7eb60a@141.31.8.62 | 2015-07-04 21:17:08 | 41 | 3333 |
    | 146585232b569ac56f96dcf567818026@141.31.8.62 | 2015-07-04 21:35:25 | 37 | 3333 |
    | 60b2fbeea831e8ccf4e1c31edfccb809@141.31.8.62 | 2015-07-04 21:36:16 | 41 | 3333 |
    | bbdafef2e6bc79253304773ff9ecd86e@141.31.8.62 | 2015-07-04 21:39:28 | 42 | 3333 |
    | 23936f70d4bf8c5a146e62a5902cec0d@141.31.8.62 | 2015-07-04 21:43:17 | 42 | 3333 |
    | 802116f89f5d5eb6a665403232e1cddd@141.31.8.62 | 2015-07-04 21:59:09 | 41 | 3333 |
    | 21b1a8cab630a51f0c139cf284fdb747@141.31.8.62 | 2015-07-04 22:20:33 | 60 | 3333 |
    | 8c760ded627b5792a3df96190bcf92a4@141.31.8.62 | 2015-07-05 00:24:22 | 41 | 3333 |
    | 7bd36fa2780a89275847a8284cf4d2ba@141.31.8.62 | 2015-07-05 00:25:07 | 37 | 3333 |
    | 36a00b646edf8af3786657281598f2bc@141.31.8.62 | 2015-07-05 01:01:33 | 42 | 3333 |
    | 79cd0a6ac13143a657535db37058e6bf@141.31.8.62 | HALEF_MixMonitor_audio_ext_7709_05072015_071600.wav | 2015-07-05 01:16:18 | 30 | 7709 | NULL |
    | 7ff0cd4ff09c9345d9ac1c74e31f889c@141.31.8.62 | 2015-07-05 01:20:59 | 44 | 3333 |
    | 91fbbc9c2e8f6abaec98bb211ba5d17d@141.31.8.62 | 2015-07-05 01:23:26 | 37 | 7709 |
    | 2893284 | 2015-07-05 01:25:43 | 141.31.8.62 | VoiceXML interpreter 0.7.6.EA started !. |
    | ebd76f51b448304ff8e223a991ed266a@141.31.8.62 | HALEF_MixMonitor_audio_ext_7709_05072015_072803.wav | 2015-07-05 01:28:55 | 62 | 7709 | NULL |

     

  • Patrick L. Lange

    Patrick L. Lange - 2015-07-14

    Does this mean that the no-RTP stream issue occured directly after a restart and was gone for Vikram's calls without a restart?

    | 2806435 | 2015-07-02 06:47:38 | 141.31.8.62 | VoiceXML interpreter 0.7.6.EA started !. |
    | e99b9ef45b0a707b666c31fded58aa7d@141.31.8.62 | 2015-07-02 14:04:32 | 32 | 2222 |
    | d91d94896edb32250af8f7090c825b4a@141.31.8.62 | 2015-07-02 14:04:32 | 32 | 2222 |
    | 01b225052bd5f0995a8158094f74e75d@141.31.8.62 | HALEF_MixMonitor_audio_ext_7711_02072015_235930.wav | 2015-07-02 18:02:48 | 208 | 7711 | NULL |
    | 7471bc05822d64e7f9532bf7909a5c46@141.31.8.62 | HALEF_MixMonitor_audio_ext_7709_05072015_003139.wav | 2015-07-04 18:31:59 | 26 | 7709 | NULL |
    | e8bbcba16ef767cdfcd7d1c1a168dadd@141.31.8.62 | HALEF_MixMonitor_audio_ext_7709_05072015_003224.wav | 2015-07-04 18:36:41 | 268 | 7709 | 6055 |
    | e54a9e898a8b4f022f2a9d9dac8162d5@141.31.8.62 | HALEF_MixMonitor_audio_ext_7709_05072015_003708.wav | 2015-07-04 18:37:24 | 27 | 7709 | NULL |
    | 6911617fb25fb428b35a88f22ed4aa15@141.31.8.62 | HALEF_MixMonitor_audio_ext_3333_05072015_003808.wav | 2015-07-04 18:38:20 | 22 | 3333 | NULL |
    | 4bdbdee3ec165b69cb4f59d66384f390@141.31.8.62 | HALEF_MixMonitor_audio_ext_3333_05072015_014208.wav | 2015-07-04 19:42:27 | 48 | 3333 | NULL |
    | 6fa0b605da5c8caa1663bdf7b41448c5@141.31.8.62 | HALEF_MixMonitor_audio_ext_3333_05072015_014309.wav | 2015-07-04 19:48:23 | 330 | 3333 | 8393 |
    | 457c5548e958c8b618841fc00b098f2f@141.31.8.62 | 2015-07-04 20:47:49 | 43 | 3333 |
    | 6a9778e2997922ef458ca87ea5dde6e9@141.31.8.62 | 2015-07-04 20:58:02 | 41 | 3333 |

     

    • David Suendermann-Oeft

      David Suendermann-Oeft - 2015-07-14

      Right. At least, I did not see evidence of a restart in haleflogs:

      select databasedate from haleflogs where message like 'VoiceXML interpreter 0.7.6.EA started%' and databasedate between '20150702' and '20150703';

      returns a single restart early that day:

      +---------------------+
      | databasedate |
      +---------------------+
      | 2015-07-02 06:47:38 |
      +---------------------+

       

  • David Suendermann-Oeft

    David Suendermann-Oeft - 2015-08-24

    There are still calls suffering from this issue:

    select distinct code, count(1) from completeCalls where callid in (select callId from hli where message like '%No RTP stream yet received%') group by code;

    shows 54 calls almost all of which died:

    +------+----------+
    | code | count(1) |
    +------+----------+
    | NULL | 51 |
    | 3823 | 2 |
    | 4790 | 1 |
    +------+----------+

    This is almost 6% of all the calls that hit our systems;

    Also, this issue seems to happen consistently over time:

    +-----------------------------+----------+
    | round(timestamp/1000000, 0) | count(1) |
    +-----------------------------+----------+
    | 20150616 | 2 |
    | 20150623 | 3 |
    | 20150624 | 4 |
    | 20150701 | 1 |
    | 20150702 | 2 |
    | 20150704 | 12 |
    | 20150705 | 5 |
    | 20150727 | 1 |
    | 20150811 | 2 |
    | 20150812 | 5 |
    | 20150817 | 8 |
    | 20150818 | 7 |
    | 20150824 | 2 |
    +-----------------------------+----------+

    select round(timestamp/1000000, 0), count(1) from completeCalls where callid in (select callId from hli where message like '%No RTP stream yet received%') group by round(timestamp/1000000, 0) ;

     

  • David Suendermann-Oeft

    David Suendermann-Oeft - 2015-11-04

    We got 14 calls with this issue today after deploying Kaldi and a few Cairo bug fixes around speech timeout etc. (I hope there is no relationship):

    select * from rtp where databasedate > '20151104';

     

Log in to post a comment.

MongoDB Logo MongoDB