#36 Error on large stack support (4K)


I was trying to build openssl-0.9.8l with MingW under X64 architecture. Compiler generates incorrect code.

The file to compile is openssl-0.9.8l\crypto\pkcs7\pk7_smime.c (uploaded as attachment).
Compiler switch is: "-g -Os -fshort-wchar -fno-strict-aliasing -Wall -Werror -Wno-missing-braces -Wno-array-bounds -c -mno-red-zone -Wno-address"

Let's take function PKCS7_verify() as example. This function declars "char buf[4096];" on stack.

Compiler generates wrong prologue for this function:

0000000000000492: 41 57 push r15
0000000000000494: B8 28 11 00 00 mov eax,1128h
0000000000000499: 41 56 push r14
000000000000049B: 41 55 push r13
000000000000049D: 41 54 push r12
000000000000049F: 55 push rbp
00000000000004A0: 57 push rdi
00000000000004A1: 56 push rsi
00000000000004A2: 53 push rbx
00000000000004A3: E8 00 00 00 00 call ___chkstk
00000000000004A8: 48 85 C9 test rcx,rcx
00000000000004AB: 49 89 CD mov r13,rcx
00000000000004AE: 48 89 D3 mov rbx,rdx
00000000000004B1: 4C 89 84 24 80 11 mov qword ptr [rsp+1180h],r8
00 00
00000000000004B9: 4C 89 8C 24 88 11 mov qword ptr [rsp+1188h],r9
00 00
00000000000004C1: 44 8B B4 24 98 11 mov r14d,dword ptr [rsp+1198h]

// function body omitted here. Epilogue follows.

00000000000008D3: 48 81 C4 28 11 00 add rsp,1128h
00000000000008DA: 89 D8 mov eax,ebx
00000000000008DC: 5B pop rbx
00000000000008DD: 5E pop rsi
00000000000008DE: 5F pop rdi
00000000000008DF: 5D pop rbp
00000000000008E0: 41 5C pop r12
00000000000008E2: 41 5D pop r13
00000000000008E4: 41 5E pop r14
00000000000008E6: 41 5F pop r15
00000000000008E8: C3 ret

It is obviously incorrect that only "add rsp 1128h" is generated for epilogue, but there is NO "sub rsp, 1128h" for prologue, which causes stack crash.

I have tried adding "static" modifier before "char buf[4096];", to move it out of stack, and the generated code becomes CORRECT.

It seems that this is a bug of supporting large stack. Thanks for any support on this.


  • Nobody/Anonymous

    Source file

  • Nobody/Anonymous

    To reproduce this issue, a small HelloWorld application with large stack is enough.

    This is what I get from a tiny function with 4K stack:

    // Prologue
    0000000000000000: 56 push rsi
    0000000000000001: B8 28 10 00 00 mov eax,1028h
    0000000000000006: 53 push rbx
    // Here should be “sub rsp, 1028h” in prologue, but it is missing.

    0000000000000007: E8 00 00 00 00 call ___chkstk

    0000000000000042: 48 81 C4 28 10 00 add rsp,1028h
    0000000000000049: 31 C0 xor eax,eax
    000000000000004B: 5B pop rbx
    000000000000004C: 5E pop rsi
    000000000000004D: C3 ret

  • Ozkan Sezer

    Ozkan Sezer - 2010-05-18

    Should this not be reported to the gcc bugzilla along with the gcc version used and a small testcase?

  • Kai Tietz

    Kai Tietz - 2010-05-18
    • status: open --> pending
  • Kai Tietz

    Kai Tietz - 2010-05-18

    Please notice that gcc's ___chkstk is different in behavior to VC's __chkstk. The gcc version allocates stack, too. There os no bug AFAIKS.

    So please describe in more details the issue you have, but I can see any problems about prologue/eplilogue here.


  • Doug Semler

    Doug Semler - 2010-05-18
    • status: pending --> open
  • Doug Semler

    Doug Semler - 2010-05-18

    I don't see any issues either -
    push registers
    call __chkstack (reserving 4392 bytes - parameter in eax)
    reset stack pointer 4392 bytes
    pop registers

    Maybe the compiler's scheduling of the mov into eax of the chkstack parameter before the pushes isn't obvious? Well, actually, probably the behavior of __chkstk itself is probably the confusing thing (it's required because it's actually aligning the stack as well as allocating any pages required, etc)

  • Kai Tietz

    Kai Tietz - 2010-05-18
    • status: open --> pending
  • SourceForge Robot

    • status: pending --> closed
  • SourceForge Robot

    This Tracker item was closed automatically by the system. It was
    previously set to a Pending status, and the original submitter
    did not respond within 14 days (the time period specified by
    the administrator of this Tracker).


Log in to post a comment.