#143 LTO type mismatch error

[Sam James]

Originally reported downstream in Gentoo at https://bugs.gentoo.org/927103.

When building with LTO (-flto) as well as options to diagnose possible runtime issues with LTO (-Werror=lto-type-mismatch -Werror=strict-aliasing), I get the following:

libtool: link: x86_64-pc-linux-gnu-g++ -m32 -mfpmath=sse  -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../lib/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/15/32/
crtbeginS.o  .libs/conform.o .libs/elemstrmwriter.o .libs/encoderparams.o .libs/macroblock.o .libs/motionest.o .libs/mpeg2coder.o .libs/mpeg2encoptions.o .libs/imageplanes.o .libs/mpeg2e
ncoder.o .libs/picture.o .libs/picturereader.o .libs/predict.o .libs/putpic.o .libs/streamstate.o .libs/seqencoder.o .libs/quantize.o .libs/ratectl.o .libs/stats.o .libs/tables.o .libs/t
ransfrm.o .libs/fdct.o .libs/idct.o .libs/predict_ref.o .libs/quantize_ref.o .libs/transfrm_ref.o .libs/fdct_x86.o .libs/fdct_mmx.o .libs/idct_mmx.o .libs/quant_mmx.o .libs/predict_mmx.o
 .libs/predcomp_mmx.o .libs/predcomp_mmxe.o .libs/predict_x86.o .libs/quantize_x86.o .libs/transfrm_x86.o .libs/ontheflyratectlpass1.o .libs/ontheflyratectlpass2.o .libs/rate_complexity_
model.o   -Wl,-rpath -Wl,/var/tmp/portage/media-video/mjpegtools-2.2.1-r2/work/mjpegtools-2.2.1-abi_x86_32.x86/utils/.libs ../utils/.libs/libmjpegutils.so -L/usr/lib/gcc/x86_64-pc-linux-
gnu/15/32 -L/usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../x86_64-pc-linux-gnu/lib/../lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../lib -L/lib/../lib -L/usr/lib/../lib -L/usr/li
b/gcc/x86_64-pc-linux-gnu/15 -L/usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../x86_64-pc-linux-gnu/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/15/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gc
c/x86_64-pc-linux-gnu/15/32/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/15/../../../../lib/crtn.o  -m32 -mfpmath=sse -O3 -march=native -mtls-dialect=gnu2 -flto=jobserver -fdiagnostics-color=always -frecord-gcc-switches -Wa,-O2 -Wa,-mtune=znver2 -Werror=lto-type-mismatch -Werror=strict-aliasing -Werror=odr -ggdb3 -Wl,-O1 -Wl,--as-needed -Wl,-z -Wl,pack-relative-relocs -fdiagnostics-color=always -flto=jobserver -Wl,--defsym=__gentoo_check_ldflags__=0 -ggdb3   -Wl,-soname -Wl,libmpeg2encpp-2.2.so.0 -o .libs/libmpeg2encpp-2.2.so.0.0.0
/var/tmp/portage/media-video/mjpegtools-2.2.1-r2/work/mjpegtools-2.2.1/mpeg2enc/quantize_x86.c:44:5: error: type of 'quant_weight_coeff_sum_mmx' does not match original declaration [-Wer
ror=lto-type-mismatch]
   44 | int quant_weight_coeff_sum_mmx (int16_t *blk, uint16_t *i_quant_mat );
      |     ^
/var/tmp/portage/media-video/mjpegtools-2.2.1-r2/work/mjpegtools-2.2.1/mpeg2enc/quant_mmx.c:188:9: note: 'quant_weight_coeff_sum_mmx' was previously declared here
  188 | int32_t quant_weight_coeff_sum_mmx(int16_t *src, int16_t *i_quant_mat)
      |         ^
/var/tmp/portage/media-video/mjpegtools-2.2.1-r2/work/mjpegtools-2.2.1/mpeg2enc/quant_mmx.c:188:9: note: code may be misoptimized unless '-fno-strict-aliasing' is used
lto1: some warnings being treated as errors
lto-wrapper: fatal error: //usr/bin/x86_64-pc-linux-gnu-g++ returned 1 exit status
compilation terminated.

This patch works for me:
```
--- a/mpeg2enc/quantize_x86.c
+++ b/mpeg2enc/quantize_x86.c
@@ -41,7 +41,7 @@
#include "quantize_precomp.h"
#include "quantize_ref.h"

-int quant_weight_coeff_sum_mmx (int16_t blk, uint16_t i_quant_mat );
+int32_t quant_weight_coeff_sum_mmx(int16_t src, int16_t i_quant_mat);

void iquantize_non_intra_m1_mmx(int16_t src, int16_t dst, uint16_t qmat);
void iquantize_non_intra_m2_mmx(int16_t src, int16_t dst, uint16_t qmat);
```

It makes the prototype in quantise_x86.c for quant_weight_coeff_sum_mmx consistent with the actual definition inquant_mmx.c.

[Sam James]

Attaching patch.

[Sam James]

I hit another issue with a 64-bit build:

/var/tmp/portage/media-video/mjpegtools-2.2.1-r2/work/mjpegtools-2.2.1/lavtools/lav_io.c:37:12: error: type of 'AVI_errno' does not match original declaration [-Werror=lto-type-mismatch]   37 | extern int AVI_errno;                                                                                                                                                                   |            ^
/var/tmp/portage/media-video/mjpegtools-2.2.1-r2/work/mjpegtools-2.2.1/lavtools/avilib.c:57:6: note: type 'long int' should match type 'int'
   57 | long AVI_errno = 0;
      |      ^
/var/tmp/portage/media-video/mjpegtools-2.2.1-r2/work/mjpegtools-2.2.1/lavtools/avilib.c:57:6: note: 'AVI_errno' was previously declared here
/var/tmp/portage/media-video/mjpegtools-2.2.1-r2/work/mjpegtools-2.2.1/lavtools/avilib.c:57:6: note: code may be misoptimized unless '-fno-strict-aliasing' is used
lto1: some warnings being treated as errors

Another patch for that:

--- a/lavtools/lav_io.c
+++ b/lavtools/lav_io.c
@@ -34,7 +34,7 @@
 #include <libdv/dv.h>
 #endif

-extern int AVI_errno;
+extern long int AVI_errno;

 static char video_format=' ';
 static int  internal_error=0;

[Steven Schultz]

paatches applied

[Sam James]

Thank you!