#496 A critical bug in init_xrpow_core_sse

[tmkk]

A pure c version of the function init_xrpow_core_c calculates in range from xrpow[0] to xrpow[upper]:

for (i = 0; i <= upper; ++i) {

but the SSE version calculates in range from xrpow[0] to xrpow[upper-1]:

int     upper4 = (upper / 4) * 4;
int     rest = upper-upper4;

Actually this should be

int     upper4 = ((upper+1) / 4) * 4;
int     rest = upper+1-upper4;

Sometimes this results in assertion failure like

Assertion failed: (vbrsf[sfb] >= vbrsfmin[sfb]), function long_block_constrain, file vbrquantize.c, line 869.

In my test the attached file (5 seconds) causes the problem when encoded with -V 0 option, but the result may vary according to the binary used.

I think bug #490 is also affected by this issue.

[Gabriel Bouvigne]

As you can see in init_xrpow_core_sse, there is a for loop looping over sets of 4 values (up to upper4), and then a switch processing the remaining values (up to 3 values).

Let's consider the case of upper ==3:
Lame 3.99: upper4 <- 0, rest <- 3
your proposal: upper4 <-4, rest 0

Let's consider the case of upper ==4:
Lame 3.99: upper4 <- 4, rest <- 0
your proposal: upper4 <-4, rest 1

...and then, writing this, I am only now noticing that in init_xrpow_core_c the loop goes up to upper (included), while in the SSE version, it goes up to upper-1 (it it processes upper values).

It looks like you have identified (and solved) a 13 years old bug. Thank you very much.

Could someone who still have a proper setup give a second opinion on this and check it?

--
Gabriel

[Robert Hegemann]

A fix is in SVN.

[Brad Lowe]

EDIT: This bug appears fixed for my case below:

Would this also fix an error in short_block_constrain with the same type of assertion failure?

Assertion failed!
Program: ffmpeg4.2.1.exe
File: ../../src/lame-3.100/libmp3lame/vbrquantize.c, Line 783
Expression: vbrsf[sfb] >= vbrsfmin[sfb]

I've seen it in several versions of ffmpeg found on https://ffbinaries.com Windows 64 but not Mac or LinuxAMD64. (Which is odd.. would think the OS wouldn't matter..) I am unable to run with a debugger to print out the values of sfb and so forth.

I'm also unsure how to determine the version of lame included with ffmpeg. Is there a pre-built binary of ffmpeg that uses your latest lame with the above fix?

This seems to only affect ffmpeg and isn't (easily?) reproducible in lame.exe by itself.

Thanks in advance for any assistance!
EDIT: I rebuilt latest from source using Docker and ffmpeg-windows-build-helpers and bug went away. THANKS ALL.