#4808 stack space detection defeated by inlining

obsolete: 8.5.9

-------- Original Message --------
Subject: [Pkg-tcltk-devel] Bug#617628: tcl8.5: Tcl/Tk applications on powerpc crash on startup with out of stack space message
Date: Wed, 9 Mar 2011 21:47:51 -0500
From: Paul Mackerras <paulus@samba.org>
Reply-To: Paul Mackerras <paulus@samba.org>, "617628@bugs.debian.org" <617628@bugs.debian.org>
To: submit@bugs.debian.org <submit@bugs.debian.org>

Subject: tcl8.5: Tcl/Tk applications on powerpc crash on startup with out of stack space message
Package: tcl8.5
Version: 8.5.9-2
Justification: renders package unusable
Severity: grave
Tags: upstream

*** Please type your report below this line ***

The tcl libraries now end up being built with -DTCL_STACK_GROWS_UP=1
on the cc command line, which causes the check for stack overflow to
have the wrong polarity. Consequently, every Tcl and Tcl/Tk application
exits immediately on startup with a message about being out of stack

The configure test that causes this flag to be set works by running
a small program that looks like this:

int StackGrowsUp(int *parent) {
int here;
return (&here < parent);
int main (int argc, char *argv[]) {
int foo;
return StackGrowsUp(&foo);

What happens is that gcc now inlines the StackGrowsUp function, meaning
that the result of the program depends only on how gcc's algorithm for
assigning stack slots for local variables happens to lay out the stack.
On powerpc with gcc-4.5.2 (Debian version 4:4.5.2-1), gcc allocates
`here' above `foo', which makes it look like the stack grows up when
in fact it grows down.

This could happen on any architecture where gcc automatically inlines
functions. It so happens that it doesn't on x86 at the moment and does
on powerpc.

-- System Information:
Debian Release: wheezy/sid
APT prefers unstable
APT policy: (990, 'unstable')
Architecture: powerpc (ppc64)

Kernel: Linux 2.6.36-rc4+ (SMP w/4 CPU cores)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages tcl8.5 depends on:
ii libc6 2.11.2-11 Embedded GNU C Library: Shared lib


  • miguel sofer

    miguel sofer - 2011-03-10

    This happens in unix/configure.in at or near line 704.

    Can that thing be try-compiled without any optimizations to avoid inlining?

  • Jan Nijtmans

    Jan Nijtmans - 2011-03-10

    Or, replace the little program by the piece below.

    No matter what compiler optimizations, this
    funcion can never be inlined, because it
    calls itself ;-)
    int StackGrowsUp(int *parent) {
    int here;
    if (parent)
    return (&here < parent);
    return StackGrowsUp(&here);
    int main (int argc, char *argv[]) {
    return StackGrowsUp(0);

  • miguel sofer

    miguel sofer - 2011-03-10
    • assigned_to: stwo --> nijtmans
  • miguel sofer

    miguel sofer - 2011-03-10

    jan, could you take care of it?

  • Jan Nijtmans

    Jan Nijtmans - 2011-03-10

    At first sight, this looked related to:
    but taking a further look, it's a totally different thing. However,
    the mentioned trick should work. I'll take care of it.

  • Jan Nijtmans

    Jan Nijtmans - 2011-03-10
    • status: open --> closed-fixed
  • Jan Nijtmans

    Jan Nijtmans - 2011-03-10

    Fixed in core-8-5-branch. Not applicable to trunk (just merge-marked there)

  • Jan Nijtmans

    Jan Nijtmans - 2013-04-03

    See: <http://code.activestate.com/lists/tcl-core/13192/>

    It turns out that this bug was never fixed in the CROSS_COMPILE case. Now it is.


Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.

No, thanks