[Tcl-bugs] [ tcl-Bugs-2800740 ] libtommath crash

[SourceForge.net <no...@so...>]

Bugs item #2800740, was opened at 2009-06-03 17:30
Message generated for change (Comment added) made by dgp
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110894&aid=2800740&group_id=10894

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
>Category: 48. Number Handling
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Don Porter (dgp)
>Assigned to: Kevin B KENNY (kennykb)
Summary: libtommath crash

Initial Comment:

% set x [expr {1<<(2**31-1)}]; concat
% set x [expr {$x<<(2**31-1)}]; concat
% set x [expr {$x<<(2**31-1)}]; concat
% set x [expr {$x<<(2**31-1)}]; concat
% set x [expr {$x<<(2**31-1)}]; concat
% set x [expr {$x<<(2**31-1)}]; concat
% set x [expr {$x<<(2**31-1)}]; concat

Program received signal SIGSEGV, Segmentation fault.
0x000000000046dfca in TclBN_mp_grow (a=0x7fff9fd04f20, size=536870920)
    at /home/dgp/cvs/tcl8.5/unix/../libtommath/bn_mp_grow.c:48
48            a->dp[i] = 0;
(gdb) print a
$1 = (mp_int *) 0x7fff9fd04f20
(gdb) print a->dp
$2 = (mp_digit *) 0x861fc68
(gdb) print i
$3 = 460175076
(gdb) print a->alloc
$4 = 536870920
(gdb) print a->dp[i]
Cannot access memory at address 0xe3cfb388
(gdb) print size
$5 = 536870920

Suspect there's been an overflow in the
second argument to XREALLOC() a few
lines earlier, and we didn't really request
as much memory as we actually need.

----------------------------------------------------------------------

>Comment By: Don Porter (dgp)
Date: 2009-06-03 17:44

Message:

Tcl's use of libtommath sets mp_digit
to be 'unsigned long' which on my test
system is 4 bytes, so sizeof(mp_digit) * size
is 2147483680, which is bigger than INT_MAX.

If something in the machinery giving meaning
to the XREALLOC() macro tries to force it
through a signed int, I'd expect trouble at this
point.  Can kbk (or anyone) shed light on just
what XREALLOC() means in this context ?



----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110894&aid=2800740&group_id=10894