[Mhash-dev] Bug in Libmhash GOST R 34.11-94! PATCH ATTACHED!

[Aleksey <ale...@gm...>]

Hello!
I'm new here! :)

The GOST R 34.11-94 algorithm in Libmhash doesn't work as expected. The 
following tests fails:
  GOST( <100000 characters of 'a'> ) = 
5C00CCC2734CDD3332D3D4749576E3C1A7DBAF0E7EA74E9FA602413C90A129FA
  GOST( <128 characters of 'U'> ) = 
53A3A3ED25180CEF0C1D85A074273E551C25660A87062A52D926A9E8FE5733A4

The bugfix-gosthash.patch file is attached. It reverts the problem code 
block back to the Markku-Juhani Saarinen's reference implementation 
(http://www.autochthonous.org/crypto/ ) and adds two tests from above to 
the src/hash_test.sh script.

The problem occurs due to bug in cumulative sum calculation (128-bit 
integer arithmetics).
In mhash-0.9.9.9/lib/gosthash.c,  function gosthash_bytes():   when

ctx->sum[] = ffffffff ffffffff ffffffff ffffffff  ffffffff ffffffff 
ffffffff ffffff9f
      + 61616161 61616161 61616161 61616161  61616161 61616161 61616161 
61616161

mhash    sum[] = 61616161 61616160 61616161 61616160  61616161 61616160 
61616161 61616100
expected sum[] = 61616161 61616161 61616161 61616161  61616161 61616161 
61616161 61616100

The cause is libmhash code:
        c = a + c + ctx->sum[i];
        
        if ((a==0xFFFFFFFFUL) && (ctx->sum[i]==0xFFFFFFFFUL)) {
            ctx->sum[i] = c;
            c = 1;
        } else {
            ctx->sum[i] = c;
            c = c < a ? 1 : 0;
        }
When c=1 and a=0xFFFFFFFFUL we obtain c=0, athough c=1 is expected!