test fail with crypto++-5.6.1

[dominique_libre]

When merging crypto++ on my gentoo amd64 system with FEATURES="test", it fail with the following error:

Using seed: 1290465767

Testing Settings...

passed: Your machine is little endian.
passed: Your machine allows unaligned data access.
passed: sizeof(byte) == 1
passed: sizeof(word16) == 2
passed: sizeof(word32) == 4
passed: sizeof(word64) == 8
passed: sizeof(word128) == 16
passed: sizeof(hword) == 4, sizeof(word) == 8, sizeof(dword) == 16
passed: hasMMX == 1, hasISSE == 1, hasSSE2 == 1, hasSSSE3 == 0, hasAESNI == 0, hasCLMUL == 0, isP4 == 0, cacheLineSize == 64, AESNI_INTRINSICS == 1

Testing operating system provided blocking random number generator...

passed: it took 0 seconds to generate 16 bytes
passed: 16 generated bytes compressed to 19 bytes by DEFLATE

Testing operating system provided nonblocking random number generator...

passed: 100000 generated bytes compressed to 100020 bytes by DEFLATE

CRC-32 validation suite running...

passed 00000000 ""
passed 43beb7e8 "a"
passed c2412435 "abc"
passed 7f9d1520 "message digest"
passed bd50274c "abcdefghijklmnopqrstuvwxyz"
passed d2e6c21f "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
passed 724aa97c "12345678901234567890123456789012345678901234567890123456789012345678901234567890"
passed 2639f4cb "123456789"

Adler-32 validation suite running...

passed 00000001 ""
passed 00620062 "a"
passed 024d0127 "abc"
passed 29750586 "message digest"
passed 90860b20 "abcdefghijklmnopqrstuvwxyz"
passed 8adb150c "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
passed 15d870f9 "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" repeated 15625 times

MD2 validation suite running...

passed 8350e5a3e24c153df2275c9f80692773 ""
passed 32ec01ec4a6dac72c0ab96fb34c0b5d1 "a"
passed da853b0d3f88d99b30283a69e6ded6bb "abc"
passed ab4f496bfb2a530b219ff33031fe06b0 "message digest"
passed 4e8ddff3650292ab5a4108c3aa47940b "abcdefghijklmnopqrstuvwxyz"
passed da33def2a42df13975352846c30338cd "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
passed d5976f79d83d3a0dc9806c3c66f3efd8 "12345678901234567890123456789012345678901234567890123456789012345678901234567890"

MD5 validation suite running...

passed d41d8cd98f00b204e9800998ecf8427e ""
passed 0cc175b9c0f1b6a831c399e269772661 "a"
passed 900150983cd24fb0d6963f7d28e17f72 "abc"
passed f96b697d7cb7938d525a2f31aaf161d0 "message digest"
passed c3fcd3d76192e4007dfb496cca67e13b "abcdefghijklmnopqrstuvwxyz"
passed d174ab98d277d9f5a5611c2c9f419d9f "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
passed 57edf4a22be3c955ac49da2e2107b67a "12345678901234567890123456789012345678901234567890123456789012345678901234567890"

SHA validation suite running...

Testing MessageDigest? algorithm SHA-1.
make: *** [test] Segmentation fault

##############
When merging it with the patch from comment #8 at http://bugs.gentoo.org/show_bug.cgi?id=322713
all the test will pass. The patch disable the inline assembler code from crypto++.

[zooko]

I was able to reproduce a segfault in sha1 test on Ubuntu 10.04 simply by adding -DCRYPTOPP_DISABLE_ASM to the CXXFLAGS in the GNUmakefile and running make && ./cryptest.exe v. It segfaults every time when it gets to the sha1 test.

g++ (Ubuntu 4.4.3-4ubuntu5) 4.4.3

[zooko]

See also http://tahoe-lafs.org/trac/pycryptopp/ticket/66 and http://bugs.gentoo.org/show_bug.cgi?id=322713 .

[weidai]

fix for this bug

[weidai]

Turns out the code for deciding whether to use portable code, SSE2 assembly, or AESNI in Rijndael.cpp introduced an infinite recursion when on x64, assembly disabled, and no AESNI.

[weidai]

Fixed in SVN as r523.

BTW, it showed up under SHA testing probably because it uses an AES-based RNG.

[zooko]

Dear Wei Dai:

Thanks for the quick turn-around time on fixing this.