#498 Crash in lame in psymodel.c:calc_energy

[David]

VLC recently updated to lame 3.100 in its contrib system, and it uses lame over libavcodec to encode mp3 audio for its chromecast streaming. We get several crashes in lame, which are only visible with our macOS port, not on windows.

Here is the relevant stack trace we have, the full trace is attached as a file:

Crash reason: EXC_SOFTWARE / SIGABRT
Crash address: 0x7fff7b50c23e
Process uptime: 78 seconds

Thread 14 (crashed)
0 libsystem_kernel.dylib!__pthread_kill + 0xa
rax = 0x0000000000000000 rdx = 0x0000000000000000
rcx = 0x000070000cdd62d8 rbx = 0x000070000cde0000
rsi = 0x0000000000000006 rdi = 0x000000000000d90b
rbp = 0x000070000cdd6310 rsp = 0x000070000cdd62d8
r8 = 0x0000000000000240 r9 = 0xcccccccccccccccd
r10 = 0x0000000000000000 r11 = 0x0000000000000206
r12 = 0x000000000000d90b r13 = 0x000000010c525000
r14 = 0x0000000000000006 r15 = 0x000000000000002d
rip = 0x00007fff7b50c23e
Found by: given as instruction pointer in context
1 libsystem_c.dylib!abort + 0x7f
rbp = 0x000070000cdd6340 rsp = 0x000070000cdd6320
rip = 0x00007fff7b4751c9
Found by: previous frame's frame pointer
2 libsystem_c.dylib!__assert_rtn + 0x140
rbp = 0x000070000cdd6380 rsp = 0x000070000cdd6350
rip = 0x00007fff7b43d868
Found by: previous frame's frame pointer
3 libavcodec_plugin.dylib!L3psycho_anal_vbr [psymodel.c : 576 + 0x1f]
rbp = 0x000070000cddb9f0 rsp = 0x000070000cdd6390
rip = 0x000000010f007617
Found by: previous frame's frame pointer
4 libavcodec_plugin.dylib!lame_encode_mp3_frame [encoder.c : 374 + 0x30]
rbx = 0x0000000000000010 rbp = 0x000070000cddfa20
rsp = 0x000070000cddba00 r12 = 0x00007fc5f5a73518
r13 = 0x00007fc5f5a6f800 r14 = 0x0000000000000001
r15 = 0x00007fc5f5a772c8 rip = 0x000000010eff6abf
Found by: call frame info
5 libavcodec_plugin.dylib!lame_encode_buffer_template [lame.c : 1793 + 0xb]
rbx = 0x00007fc5f5a41c00 rbp = 0x000070000cddfad0
rsp = 0x000070000cddfa30 r12 = 0x00007fc5f5a6f800
r13 = 0x0000000000000000 r14 = 0x00007fc5f5a13600
r15 = 0x0000000000001200 rip = 0x000000010f000038
Found by: call frame info
6 libavcodec_plugin.dylib!lame_encode_buffer_float [lame.c : 1925 + 0x11]
rbx = 0x00007fc5f4ca98c0 rbp = 0x000070000cddfaf0
rsp = 0x000070000cddfae0 r12 = 0x00007fc5f5bcda00
r13 = 0x000070000cddfc7c r14 = 0x00007fc5f4ca9e20
r15 = 0x00007fc5f4ca8f80 rip = 0x000000010f000255
Found by: call frame info
7 libavcodec_plugin.dylib!mp3lame_encode_frame [libmp3lame.c : 216 + 0x22]
rbx = 0x00007fc5f4ca98c0 rbp = 0x0000000000000002
rsp = 0x000070000cddfb00 r12 = 0x00007fc5f5bcda00
r13 = 0x000070000cddfc7c r14 = 0x00007fc5f4ca9e20
r15 = 0x00007fc5f4ca8f80 rip = 0x000000010e99a525
Found by: call frame info

This is in the following code part:

FLOAT const el = fftenergy[j];
assert(el >= 0);

Do you have any hint why this variable could be negative?

[Marc DUMONT]

Hi. Same for me (*), currently reproductible in DEBUG version. I'm stuying it (development mode). This problems occurs since a long time for me, at least 3.99 version, but I was not using the encoder so much. Now I want to track this problem.
For now what I have is that "el" value comes to 'UNDEFINED' float value (not correct negative) like -1.#IND ... I'm tracking the call stack a bit upper to see what causes fftenergy[] and perhaps wsampl_l[] to become invalid .... all around FFT code anyway.

(*) My context:
Windows platform (Windows 7 x64), Visual Studio 10, realtime streaming.

[Marc DUMONT]

[Currently validating]
I saw, at least, uninitialized arrays in psymodel.c like wsamp_L, wsamp_S (and probably the other ones), that may have some influence after many frames encoding. It seems I have many "NaN" values finally propagated.... on the FFT result "energy" and then the assertion on "el".

I'm not sure if this is a MSBuild / gcc compilers perhaps known difference ? A fact is non-static arrays are never initialized.

I remember some years a go I (privately) fixed a same kind of bug in layer2.c/decode_layer2_frame initializing "fraction" array to 0 also, else I was sometimes listening to audio artifacts, sometimes, especially on the right audio channel. I was not subscribed to this mailing list so sorry, I never submitted that bug. Now the liblame authors will know about that ;-)

[Marc DUMONT]

Continuing my investigation shows that the encoder is maybe not-so-responsible but the decoder is.... In fact my test application decodes MP3 then recodes it over a real-time stream. But sometimes the input stream "looses" some packets, which causes MP3 decoding errors but concealed (not crashing). But it seems to leave some uninitialized float-data at the MP3 decoder output....

[David]

Hello Marc, thank you very much for your feedback.
On VLC side, our current assumption is that this mostly happens when the user heavily seeks in the input data stream. This probably also causes some "bad" or invalid data being passed to the lame encoder, after the stream is flushed. This sound similar to what you are describing.

However, at least my assumption is even with invalid raw audio input data, lame should still be able to somehow encode those instead of crashing (even if it might sound wrong of course).

[Marc DUMONT]

I almost agree with you, but may be float "NaN" is not a real case to push into the encoder anyway.... That one, for sure, immediately produces the assertion. This is not a real 'crash', by the way, I mean this does not run into access violation/buffer overrun or something like this. This is an assertion (critical last-chance check) which causes program abort() at the end.

[Marc DUMONT]

OK. "My" problem is solved, I was indeed pushing bad data to the encoder, on some incoming stream packet losses, but LAME library is not responsible for that. The only open point left is the capability of LAME encoder to be resilient to error from input buffer... I leave you, authors, solving or not this point. Thanks.