#4808 stack space detection defeated by inlining

obsolete: 8.5.9
closed-fixed
Jan Nijtmans
8
2011-03-10
2011-03-10
miguel sofer
No

-------- Original Message --------
Subject: [Pkg-tcltk-devel] Bug#617628: tcl8.5: Tcl/Tk applications on powerpc crash on startup with out of stack space message
Date: Wed, 9 Mar 2011 21:47:51 -0500
From: Paul Mackerras <paulus@samba.org>
Reply-To: Paul Mackerras <paulus@samba.org>, "617628@bugs.debian.org" <617628@bugs.debian.org>
To: submit@bugs.debian.org <submit@bugs.debian.org>

Subject: tcl8.5: Tcl/Tk applications on powerpc crash on startup with out of stack space message
Package: tcl8.5
Version: 8.5.9-2
Justification: renders package unusable
Severity: grave
Tags: upstream

*** Please type your report below this line ***

The tcl libraries now end up being built with -DTCL_STACK_GROWS_UP=1
on the cc command line, which causes the check for stack overflow to
have the wrong polarity. Consequently, every Tcl and Tcl/Tk application
exits immediately on startup with a message about being out of stack
space.

The configure test that causes this flag to be set works by running
a small program that looks like this:

int StackGrowsUp(int *parent) {
int here;
return (&here < parent);
}
int main (int argc, char *argv[]) {
int foo;
return StackGrowsUp(&foo);
}

What happens is that gcc now inlines the StackGrowsUp function, meaning
that the result of the program depends only on how gcc's algorithm for
assigning stack slots for local variables happens to lay out the stack.
On powerpc with gcc-4.5.2 (Debian version 4:4.5.2-1), gcc allocates
`here' above `foo', which makes it look like the stack grows up when
in fact it grows down.

This could happen on any architecture where gcc automatically inlines
functions. It so happens that it doesn't on x86 at the moment and does
on powerpc.

-- System Information:
Debian Release: wheezy/sid
APT prefers unstable
APT policy: (990, 'unstable')
Architecture: powerpc (ppc64)

Kernel: Linux 2.6.36-rc4+ (SMP w/4 CPU cores)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages tcl8.5 depends on:
ii libc6 2.11.2-11 Embedded GNU C Library: Shared lib

Discussion

  • miguel sofer
    miguel sofer
    2011-03-10

    This happens in unix/configure.in at or near line 704.

    Can that thing be try-compiled without any optimizations to avoid inlining?

     
  • Jan Nijtmans
    Jan Nijtmans
    2011-03-10

    Or, replace the little program by the piece below.

    No matter what compiler optimizations, this
    funcion can never be inlined, because it
    calls itself ;-)
    =================================
    int StackGrowsUp(int *parent) {
    int here;
    if (parent)
    return (&here < parent);
    else
    return StackGrowsUp(&here);
    }
    int main (int argc, char *argv[]) {
    return StackGrowsUp(0);
    }

     
  • miguel sofer
    miguel sofer
    2011-03-10

    • assigned_to: stwo --> nijtmans
     
  • miguel sofer
    miguel sofer
    2011-03-10

    jan, could you take care of it?

     
  • Jan Nijtmans
    Jan Nijtmans
    2011-03-10

    At first sight, this looked related to:
    <https://sourceforge.net/tracker/?func=detail&atid=110894&aid=3166410&group_id=10894>
    but taking a further look, it's a totally different thing. However,
    the mentioned trick should work. I'll take care of it.

     
  • Jan Nijtmans
    Jan Nijtmans
    2011-03-10

    • status: open --> closed-fixed
     
  • Jan Nijtmans
    Jan Nijtmans
    2011-03-10

    Fixed in core-8-5-branch. Not applicable to trunk (just merge-marked there)

     
  • Jan Nijtmans
    Jan Nijtmans
    2013-04-03

    See: <http://code.activestate.com/lists/tcl-core/13192/>

    It turns out that this bug was never fixed in the CROSS_COMPILE case. Now it is.