#431 A bunch of undefined behavior

[Dingbao Xie]

After running afl-fuzzer on flac for one night, I found over 200 undefined behavior in flac.
To reproduce them, first build the source code with flag '-fsanitize=undefined' and then execute
command 'flac -f -e -o test.ogg $file'. You should be able to see the following error information:

fixed.c:390:27: runtime error: signed integer overflow: -496475550 + -2021049322 cannot be represented in type 'int'

fixed.c:395:39: runtime error: left shift of negative value -3071
fixed.c:395:44: runtime error: signed integer overflow: -2135848804 - 386639488 cannot be represented in type 'int'

bitreader.c:425:7: runtime error: left shift of 182 by 24 places cannot be represented in type 'FLAC__int32' (aka 'int')
bitwriter.c:523:16: runtime error: left shift of negative value -73
bitreader.c:426:7: runtime error: shift exponent 32 is too large for 32-bit type 'FLAC__int32' (aka 'int')
stream_decoder.c:2545:42: runtime error: left shift of negative value -31
fixed.c:395:39: runtime error: left shift of negative value -504043600
fixed.c:395:27: runtime error: signed integer overflow: 412551703 + 2123433058 cannot be represented in type 'int'
fixed.c:395:44: runtime error: signed integer overflow: 1707089161 - -2132115330 cannot be represented in type 'int'

fixed.c:361:52: runtime error: left shift of negative value -73
fixed.c:403:52: runtime error: left shift of negative value -12
fixed.c:403:57: runtime error: signed integer overflow: -1512129672 + -756064836 cannot be represented in type 'int'
fixed.c:403:82: runtime error: signed integer overflow: 2026772754 + 696512590 cannot be represented in type 'int'
fixed.c:403:41: runtime error: signed integer overflow: 847134422 - -1571681952 cannot be represented in type 'int'
fixed.c:403:69: runtime error: signed integer overflow: 847134422 - -1571681952 cannot be represented in type 'int'

fixed.c:411:51: runtime error: left shift of negative value -8
fixed.c:411:69: runtime error: left shift of negative value -8
fixed.c:411:86: runtime error: left shift of negative value -8
fixed.c:411:74: runtime error: signed integer overflow: -1747526760 + -873763380 cannot be represented in type 'int'
fixed.c:411:56: runtime error: signed integer overflow: 1387533973 - -1378975848 cannot be represented in type 'int'
fixed.c:411:40: runtime error: signed integer overflow: -1922739012 + -861792809 cannot be represented in type 'int'
fixed.c:411:92: runtime error: signed integer overflow: 1989299140 - -861792809 cannot be represented in type 'int'

lpc.c:854:36: runtime error: shift exponent -16 is negative

lpc.c:870:11: runtime error: signed integer overflow: 1468705162 + 1782934516 cannot be represented in type 'int'
lpc.c:868:11: runtime error: signed integer overflow: 1212200402 + 1466414430 cannot be represented in type 'int'
lpc.c:867:11: runtime error: signed integer overflow: 1233218992 + 977609620 cannot be represented in type 'int'
lpc.c:868:27: runtime error: signed integer overflow: 6 * 445733629 cannot be represented in type 'int'
lpc.c:866:11: runtime error: signed integer overflow: 782769282 + 1466414430 cannot be represented in type 'int'
lpc.c:866:27: runtime error: signed integer overflow: 6 * 445733629 cannot be represented in type 'int'
lpc.c:865:11: runtime error: signed integer overflow: 1712171384 + 891467258 cannot be represented in type 'int'
lpc.c:864:11: runtime error: signed integer overflow: 1339660100 + 1782934516 cannot be represented in type 'int'
lpc.c:862:11: runtime error: signed integer overflow: 976866726 + 1466414430 cannot be represented in type 'int'
lpc.c:862:27: runtime error: signed integer overflow: 6 * 445733629 cannot be represented in type 'int'
lpc.c:861:11: runtime error: signed integer overflow: 1466414430 + 1782934516 cannot be represented in type 'int'
lpc.c:860:28: runtime error: signed integer overflow: 6 * 445733629 cannot be represented in type 'int'
lpc.c:883:11: runtime error: signed integer overflow: -1375361737 + -1370216580 cannot be represented in type 'int'
lpc.c:883:27: runtime error: signed integer overflow: 251 * -9724013 cannot be represented in type 'int'
lpc.c:884:27: runtime error: signed integer overflow: 75 * 51523392 cannot be represented in type 'int'
lpc.c:885:11: runtime error: signed integer overflow: 1506154158 + 1138343777 cannot be represented in type 'int'
lpc.c:884:11: runtime error: signed integer overflow: 1719225939 + 982547591 cannot be represented in type 'int'
lpc.c:886:11: runtime error: signed integer overflow: -1100233892 + -1344207748 cannot be represented in type 'int'
lpc.c:881:27: runtime error: signed integer overflow: -512 * -9724013 cannot be represented in type 'int'
lpc.c:882:27: runtime error: signed integer overflow: -127 * 51523392 cannot be represented in type 'int'
lpc.c:882:11: runtime error: signed integer overflow: 660238720 + 2046463808 cannot be represented in type 'int'
lpc.c:888:11: runtime error: signed integer overflow: -1237982387 + -1944868653 cannot be represented in type 'int'
lpc.c:879:27: runtime error: signed integer overflow: -448 * -9724013 cannot be represented in type 'int'
lpc.c:881:11: runtime error: signed integer overflow: -1607643136 + -997610496 cannot be represented in type 'int'

lpc.c:888:27: runtime error: signed integer overflow: 6 * -517398563 cannot be represented in type 'int'
lpc.c:887:11: runtime error: signed integer overflow: -2023655768 + -1034797126 cannot be represented in type 'int'
lpc.c:886:27: runtime error: signed integer overflow: 6 * -517398563 cannot be represented in type 'int'
lpc.c:885:11: runtime error: signed integer overflow: -770896152 + -2069594252 cannot be represented in type 'int'
lpc.c:888:11: runtime error: signed integer overflow: 1592544682 + 1731773580 cannot be represented in type 'int'
lpc.c:884:27: runtime error: signed integer overflow: 6 * -517398563 cannot be represented in type 'int'
lpc.c:884:11: runtime error: signed integer overflow: 1425082692 + 1190575918 cannot be represented in type 'int'
lpc.c:883:27: runtime error: signed integer overflow: -6 * -517398563 cannot be represented in type 'int'
lpc.c:886:11: runtime error: signed integer overflow: -1445743110 + -1455973554 cannot be represented in type 'int'

lpc.c:889:36: runtime error: shift exponent -15 is negative
lpc.c:903:11: runtime error: signed integer overflow: 651904477 + 1876584215 cannot be represented in type 'int'
lpc.c:900:27: runtime error: signed integer overflow: 6 * 375316843 cannot be represented in type 'int'
lpc.c:900:11: runtime error: signed integer overflow: 2131795721 + 1645249396 cannot be represented in type 'int'
lpc.c:899:27: runtime error: signed integer overflow: 5 * -441619650 cannot be represented in type 'int'
lpc.c:899:11: runtime error: signed integer overflow: 397106362 + 2086869046 cannot be represented in type 'int'
lpc.c:897:27: runtime error: signed integer overflow: 6 * 375316843 cannot be represented in type 'int'
lpc.c:897:11: runtime error: signed integer overflow: 1629738114 + 1645249396 cannot be represented in type 'int'
lpc.c:896:11: runtime error: signed integer overflow: -1766478600 + -581683428 cannot be represented in type 'int'
lpc.c:898:11: runtime error: signed integer overflow: 1750324760 + 1963515384 cannot be represented in type 'int'
lpc.c:901:11: runtime error: signed integer overflow: 1227342121 + 1318656642 cannot be represented in type 'int'
lpc.c:902:11: runtime error: signed integer overflow: -1748968533 + -1501267372 cannot be represented in type 'int'
lpc.c:903:27: runtime error: signed integer overflow: 5 * -441619650 cannot be represented in type 'int'
lpc.c:904:36: runtime error: shift exponent -8 is negative

lpc.c:914:27: runtime error: signed integer overflow: -8 * 330766035 cannot be represented in type 'int'
lpc.c:915:11: runtime error: signed integer overflow: 1648839016 + 571594851 cannot be represented in type 'int'

lpc.c:916:27: runtime error: signed integer overflow: 8 * -651719830 cannot be represented in type 'int'
lpc.c:917:11: runtime error: signed integer overflow: -1429684760 + -1492093844 cannot be represented in type 'int'
lpc.c:918:11: runtime error: signed integer overflow: 1373188692 + 1900562712 cannot be represented in type 'int'
lpc.c:918:27: runtime error: signed integer overflow: 6 * -651719830 cannot be represented in type 'int'
lpc.c:919:11: runtime error: signed integer overflow: 1483481470 + 1733383093 cannot be represented in type 'int'
lpc.c:920:27: runtime error: signed integer overflow: 30 * 233528841 cannot be represented in type 'int'
lpc.c:921:11: runtime error: signed integer overflow: -294561549 + -1885527855 cannot be represented in type 'int'
lpc.c:921:27: runtime error: signed integer overflow: -55 * 233528841 cannot be represented in type 'int'
lpc.c:922:36: runtime error: shift exponent -5 is negative
lpc.c:928:27: runtime error: signed integer overflow: -16384 * 4164143 cannot be represented in type 'int'

lpc.c:929:11: runtime error: signed integer overflow: 2053212131 + 850621975 cannot be represented in type 'int'
lpc.c:930:11: runtime error: signed integer overflow: 1380175543 + 850621975 cannot be represented in type 'int'
lpc.c:932:11: runtime error: signed integer overflow: -1300984993 + -1984948308 cannot be represented in type 'int'

lpc.c:933:11: runtime error: signed integer overflow: 981641733 + 2053212131 cannot be represented in type 'int'
lpc.c:934:27: runtime error: signed integer overflow: 13037 * 4164143 cannot be represented in type 'int'
lpc.c:935:36: runtime error: shift exponent -5 is negative
lpc.c:947:27: runtime error: signed integer overflow: 1090 * 339802111 cannot be represented in type 'int'
lpc.c:943:27: runtime error: signed integer overflow: -12800 * 311792 cannot be represented in type 'int'
lpc.c:947:11: runtime error: signed integer overflow: -666163770 + -1553618750 cannot be represented in type 'int'
lpc.c:946:11: runtime error: signed integer overflow: -1811984896 + -1379499109 cannot be represented in type 'int'

lpc.c:949:36: runtime error: shift exponent -10 is negative
lpc.c:959:27: runtime error: signed integer overflow: -1171 * -3392359 cannot be represented in type 'int'
lpc.c:958:27: runtime error: signed integer overflow: -1171 * -3392359 cannot be represented in type 'int'
lpc.c:957:27: runtime error: signed integer overflow: -1171 * -3392359 cannot be represented in type 'int'
lpc.c:959:11: runtime error: signed integer overflow: -1845384124 + -1354550575 cannot be represented in type 'int'
lpc.c:956:27: runtime error: signed integer overflow: -1184 * -3392359 cannot be represented in type 'int'
lpc.c:958:11: runtime error: signed integer overflow: -1688486145 + -1354550575 cannot be represented in type 'int'
lpc.c:957:11: runtime error: signed integer overflow: -1235001568 + -1354550575 cannot be represented in type 'int'

lpc.c:960:36: runtime error: shift exponent -3 is negative
lpc.c:973:27: runtime error: signed integer overflow: -15 * 1102339620 cannot be represented in type 'int'
lpc.c:972:27: runtime error: signed integer overflow: -12 * 1102339620 cannot be represented in type 'int'
lpc.c:973:11: runtime error: signed integer overflow: -1814971804 + -508401255 cannot be represented in type 'int'
lpc.c:974:36: runtime error: shift exponent -4 is negative

lpc.c:980:27: runtime error: signed integer overflow: 112 * -158668606 cannot be represented in type 'int'
lpc.c:981:11: runtime error: signed integer overflow: 1978282576 + 941651375 cannot be represented in type 'int'
lpc.c:982:27: runtime error: signed integer overflow: -4 * 804067549 cannot be represented in type 'int'
lpc.c:982:11: runtime error: signed integer overflow: -1241225422 + -1037948544 cannot be represented in type 'int'
lpc.c:981:11: runtime error: signed integer overflow: -1584706128 + -735817710 cannot be represented in type 'int'

lpc.c:993:36: runtime error: shift exponent -5 is negativ
lpc.c:998:59: runtime error: shift exponent -2 is negative
lpc.c:998:46: runtime error: signed integer overflow: -256 * -1626996735 cannot be represented in type 'int'
represented in type 'int'
lpc.c:1009:18: runtime error: signed integer overflow: 306372314 + 2071085304 cannot be represented in type 'int'
lpc.c:1010:35: runtime error: signed integer overflow: 1670 * 3293944 cannot be represented in type 'int'
lpc.c:1008:18: runtime error: signed integer overflow: 2126599864 + 1981984542 cannot be represented in type 'int'
lpc.c:1010:18: runtime error: signed integer overflow: -342813802 + -1936155756 cannot be represented in type 'int'

lpc.c:1016:35: runtime error: signed integer overflow: -296 * 8677533 cannot be represented in type 'int'
lpc.c:1017:35: runtime error: signed integer overflow: -147 * 115975311 cannot be represented in type 'int'
lpc.c:1018:18: runtime error: signed integer overflow: 1857915995 + 410579324 cannot be represented in type 'int'
lpc.c:1019:35: runtime error: signed integer overflow: -293 * 8677533 cannot be represented in type 'int'
lpc.c:1020:35: runtime error: signed integer overflow: -2341 * -6284946 cannot be represented in type 'int'
lpc.c:1020:18: runtime error: signed integer overflow: 743076873 + 1828156698 cannot be represented in type 'int'

lpc.c:1021:35: runtime error: signed integer overflow: 24 * 127855205 cannot be represented in type 'int'
lpc.c:1021:18: runtime error: signed integer overflow: -2014899967 + -1226442376 cannot be represented in type 'int'
lpc.c:1022:35: runtime error: signed integer overflow: 52 * 127855205 cannot be represented in type 'int'
lpc.c:1023:35: runtime error: signed integer overflow: -30 * 127855205 cannot be represented in type 'int'
lpc.c:1024:35: runtime error: signed integer overflow: -64 * 33770640 cannot be represented in type 'int'
lpc.c:1024:18: runtime error: signed integer overflow: 326754985 + 2133646336 cannot be represented in type 'int'
lpc.c:1025:35: runtime error: signed integer overflow: 17 * 127855205 cannot be represented in type 'int'
lpc.c:1025:18: runtime error: signed integer overflow: -1834565975 + -2121428811 cannot be represented in type 'int'
lpc.c:1026:18: runtime error: signed integer overflow: -1507376327 + -751297484 cannot be represented in type 'int'
lpc.c:1027:18: runtime error: signed integer overflow: 347744086 + 2143202880 cannot be represented in type 'int'
lpc.c:1027:35: runtime error: signed integer overflow: 36 * 127855205 cannot be represented in type 'int'
lpc.c:1028:18: runtime error: signed integer overflow: 1250012795 + 1534262460 cannot be represented in type 'int'
lpc.c:1028:35: runtime error: signed integer overflow: 13 * -197452218 cannot be represented in type 'int'
lpc.c:1029:35: runtime error: signed integer overflow: -5 * 540731445 cannot be represented in type 'int'
lpc.c:1029:18: runtime error: signed integer overflow: 1922691538 + 987261090 cannot be represented in type 'int'
lpc.c:1030:35: runtime error: signed integer overflow: -10 * 540731445 cannot be represented in type 'int'
lpc.c:1030:18: runtime error: signed integer overflow: -1385014668 + -1112347154 cannot be represented in type 'int'
lpc.c:1031:18: runtime error: signed integer overflow: 1797605474 + 1067963180 cannot be represented in type 'int'
lpc.c:1031:35: runtime error: signed integer overflow: 13 * -197452218 cannot be represented in type 'int'
lpc.c:1032:35: runtime error: signed integer overflow: 11 * -197452218 cannot be represented in type 'int'
lpc.c:1032:18: runtime error: signed integer overflow: 1922710738 + 2122992898 cannot be represented in type 'int'
lpc.c:1033:18: runtime error: signed integer overflow: -1908953333 + -1008351456 cannot be represented in type 'int'
lpc.c:1033:35: runtime error: signed integer overflow: 2 * -1532029167 cannot be represented in type 'int'

lpc.c:1034:35: runtime error: signed integer overflow: 4 * 540731445 cannot be represented in type 'int'
lpc.c:1034:18: runtime error: signed integer overflow: -1921056628 + -449235864 cannot be represented in type 'int'
lpc.c:1035:35: runtime error: signed integer overflow: 12 * -197452218 cannot be represented in type 'int'
lpc.c:1035:18: runtime error: signed integer overflow: 511091894 + 1925540680 cannot be represented in type 'int'
lpc.c:1036:35: runtime error: signed integer overflow: -14 * -197452218 cannot be represented in type 'int'
lpc.c:1036:18: runtime error: signed integer overflow: -351433127 + -1879090556 cannot be represented in type 'int'
lpc.c:1037:18: runtime error: signed integer overflow: 625112360 + 1636464400 cannot be represented in type 'int'
lpc.c:1038:18: runtime error: signed integer overflow: -1920934159 + -296611792 cannot be represented in type 'int'

lpc.c:1038:35: runtime error: signed integer overflow: 6 * 818232200 cannot be represented in type 'int'

lpc.c:1040:33: runtime error: shift exponent -11 is negative
lpc.c:1158:49: runtime error: shift exponent -7 is negative
lpc.c:1125:49: runtime error: shift exponent -5 is negative

lpc.c:1176:29: runtime error: signed integer overflow: 537071617 + 2032956606 cannot be represented in type 'int'

lpc.c:1176:49: runtime error: shift exponent -8 is negativ
lpc.c:1189:49: runtime error: shift exponent -5 is negative
lpc.c:1203:49: runtime error: shift exponent -3 is negative
lpc.c:1214:49: runtime error: shift exponent -15 is negative

lpc.c:1228:49: runtime error: shift exponent -8 is negative
lpc.c:1237:29: runtime error: signed integer overflow: -1016066047 + -1605692791 cannot be represented in type 'int'
lpc.c:1237:49: runtime error: shift exponent -13 is negative

lpc.c:1247:49: runtime error: shift exponent -13 is negative
lpc.c:1252:85: runtime error: shift exponent -6 is negative
lpc.c:1294:46: runtime error: shift exponent -14 is negative
lpc.c:1294:26: runtime error: signed integer overflow: -659054596 + -1591434921 cannot be represented in type 'int'

[Erik]

This is interesting, but I don't know if its something worth fixing.

I have already spent well over a week of CPU time fuzzing FLAC with AFL and the Address Sanitizer. That found some genuine bugs that were definitely worth fixing.

Anyone else care to weight in?

[Erik]

I've looked at this some more. If I run the command you suggest over each of these files in turn I find that the exit code for the UBSan compiled flac executable is either 0 or 1. In the case where its 1, the flac executable prints an error message before calling exit(1).

At this stage I do not consider any of these undefined baehaviour warnings to be a security issue.

[Erik]

Have recently fixed a large number of UB warnings when running the test suite. It may still be possible to find more via fuzzing, but I don't really care about UBSan warnings, unless they also trigger an ASan error.

Closing this.