[Mcrypt-developers] [PATCH] Fix for cast-256 on Alpha

[Matt T. <mat...@gm...>]

Hi,

I submitted this to the bug tracker, so I'm emailing the list for an
ack/nack on the patch.
https://sourceforge.net/tracker/index.php?func=detail&aid=2424445&group_id=87941&atid=584895

Originally reported in Gentoo Bugzilla.
(https://bugs.gentoo.org/show_bug.cgi?id=132356)

The KNOWN-BUGS file lists the following problem:
- cast-256 and rc6 do not work properly on Alpha (64 bit) machines

The problem was the rotl32 macro in lib/mcrypt_modules.h.

Consider the case where the number to be shifted is 1 and the shift
argument is 32. The expected result is that the original number is the same
as the final number, since it rotated the entire width of the field.

On Alpha (and others?), the 64-bit registers prevent it from rotating,
since a shift of 32 actually rotates bits into the upper 32-bits, yielding
a result of 0.

This did not affect AMD64 because gcc is able to generate 32-bit shift
instructions, whereas all shifts on Alpha are 64-bit.

The fix is to mask the shift argument before shifting. The included patch
masks the shift arguments in the rot{r,l}{16,32} macros and deletes the
KNOWN-BUGS file as there are no other known bugs listed.

Please provide a new version of libmcrypt with this fix.

Thanks,

Matt Turner