Re: [Openvpn-devel] More on the George Ross failure

[Timothe L. <li...@ac...>]

Having stolen a few minutes, a bit closer...

Backtracking, I believe George's error must be coming from 
openssl/ssl/s3_clnt.c: ssl3_send_client_verify(), the block of code 
starting 36 lines in, shown below.

There is a call to EVP_SignFinal, that I believe will turn out to 
dispatch to RSA_sign (openssl/crypto/rsa/rsa_sign.c ).  There is an 
intermediate maze of twisty passages that abstracts the signature 
mechanisms, but I'm pretty sure that's where we'll end up.

RSA_sign emits RSA_R_DIGEST_TOO_BIG_FOR_RSA_KEY when the size of the 
signature generated by i2d_X509_SIG is larger than the size allocated 
for it in the RSA structure.

Again, this is dependent (indirectly) on TLS version (the check is for 
md5_sha1 digest, which is used before TLS1.2).
The TLS1.1 path is too well-worn to be likely.  The code (abbreviated) 
goes like this:

     if(type == NID_md5_sha1) {
        ...
         i = SSL_SIG_LENGTH;
         s = m;
     } else {
            ...
         i=i2d_X509_SIG(&sig,NULL);
     }
     j=RSA_size(rsa);
     if (i > (j-RSA_PKCS1_PADDING_SIZE))
         {
         RSAerr(RSA_F_RSA_SIGN,RSA_R_DIGEST_TOO_BIG_FOR_RSA_KEY);
         return(0);
         }

This leaves the question of why the RSA structure doesn't have enough space.

Getting further back would be a lot easier with a reproducer and a 
debugger...

The ssl3_send_client_verify +36:

         if (TLS1_get_version(s) >= TLS1_2_VERSION)
             {
             long hdatalen = 0;
             void *hdata;
             const EVP_MD *md = s->cert->key->digest;
             hdatalen = BIO_get_mem_data(s->s3->handshake_buffer,
                                 &hdata);
             if (hdatalen <= 0 || !tls12_get_sigandhash(p, pkey, md))
                 {
                 SSLerr(SSL_F_SSL3_SEND_CLIENT_VERIFY,
                         ERR_R_INTERNAL_ERROR);
                 goto err;
                 }
             p += 2;
#ifdef SSL_DEBUG
             fprintf(stderr, "Using TLS 1.2 with client alg %s\n",
                             EVP_MD_name(md));
#endif
             if (!EVP_SignInit_ex(&mctx, md, NULL)
                 || !EVP_SignUpdate(&mctx, hdata, hdatalen)
                 || !EVP_SignFinal(&mctx, p + 2, &u, 
pkey))                   <===== breakpoint goes here
                 {
                 SSLerr(SSL_F_SSL3_SEND_CLIENT_VERIFY,
                         ERR_R_EVP_LIB);
                 goto err;
                 }
             s2n(u,p);
             n = u + 4;
             if (!ssl3_digest_cached_records(s))
                 goto err;
             }

.


Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.

This communication may not represent my employer's views,
if any, on the matters discussed.