[Perl-openssl-users] Crypt::OpenSSL::RSA 0.23 segfaults in rsa_crypt - signed>unsigned conversion

[Mark M. <Mar...@ij...>]

Noticing that selftest on installing Mail::DKIM 0.20 causes perl
to segfault while checking the last signature in multiple_1.txt
(test t/verifier.t), I recompiled RSA.xs with debugging and
checked what is going on:

Program received signal SIGSEGV, Segmentation fault.

(gdb) bt
#0  0x2825f78e in memmove () from /lib/libc.so.6
#1  0x280f53ae in Perl_sv_setpvn () from .../5.8.8/mach/CORE/libperl.so
#2  0x280f7859 in Perl_newSVpv () from .../5.8.8/mach/CORE/libperl.so
#3  0x283964ff in rsa_crypt (p_rsa=0x8561164, p_from=0x849238c,
  p_crypt=0x28332a10 <RSA_public_encrypt>) at RSA.xs:203
#4  0x28397ddb in XS_Crypt__OpenSSL__RSA_encrypt (cv=0x849196c) at RSA.xs:404
#5  0x280f01c0 in Perl_pp_entersub () from .../5.8.8/mach/CORE/libperl.so
#6  0x280e91a5 in Perl_runops_standard () from .../5.8.8/mach/CORE/libperl.so
#7  0x28098b8a in perl_run () from .../5.8.8/mach/CORE/libperl.so
#8  0x08048fe2 in main ()

(gdb) up
#1  0x280f53ae in Perl_sv_setpvn () from .../5.8.8/mach/CORE/libperl.so
(gdb) up
#2  0x280f7859 in Perl_newSVpv () from .../5.8.8/mach/CORE/libperl.so
(gdb) up
#3  0x283964ff in rsa_crypt (p_rsa=0x8561164, p_from=0x849238c,
  p_crypt=0x28332a10 <RSA_public_encrypt>) at RSA.xs:203
203         sv = newSVpv(to, to_length);

(gdb) p to_length
$2 = 4294967295

The code in RSA.xs in question is:

SV* rsa_crypt(rsaData* p_rsa, SV* p_from,
  int (*p_crypt)(int, const unsigned char*, unsigned char*, RSA*, int))
{
    STRLEN from_length, to_length;
...
    to_length = p_crypt(
       from_length, from, (unsigned char*) to, p_rsa->rsa, p_rsa->padding);

    if (to_length < 0)  <-- is false because to_length is unsigned
    {
        Safefree(to);
        CHECK_OPEN_SSL(0);
    }
--> crashes here:
    sv = newSVpv(to, to_length);

So the problem is that p_crypt returns -1, which is converted
to unsigned 4294967295, and memmove crashes trying to copy
4 GB of nonexistent data to some allocated buffer.

I'd be surprised if there is no security problem, although at the
moment I'm more concerned with keeping Perl application programs stable.
There are other potentially troublesome signed/unsigned conversions
in RSA.xs and typecasting 64-bit pointers to smaller integers,
although I haven't examined them closely.

Btw, STRLEN boils down to size_t, which is unsigned 32-bit
value. This is on FreeBSD 6.1, Perl 5.8.8 from ports,
security/p5-Crypt-OpenSSL-RSA 0.23 from ports, although this
probably applies to most modern platforms.

Regards
   Mark