#727 Crash during destruction of thread_local objects

[Martin Stumpf]

This code crashes in the destructor of 'map':

#include <thread>
#include <unordered_map>
#include <vector>
#include <sstream>

#define print(msg) {std::stringstream ss; ss << std::this_thread::get_id() << ": " << msg; printf("%s\n", ss.str().c_str());}

struct A
{
    std::unordered_map<int, int> map;
};

A &getA()
{
    static thread_local A inst;
    return inst;
}

int main()
{
    std::vector<std::thread> threads;

    for (size_t i = 0; i < std::thread::hardware_concurrency() * 4; i++)
    {
        threads.emplace_back([]()
        {
            print(getA().map.size());
        });
    }

    for (auto &thread : threads)
        thread.join();

    return 0;
}

The correct output would be http://coliru.stacked-crooked.com/a/e4e4c21b32719ca9.

[Martin Stumpf]

Link correction: http://coliru.stacked-crooked.com/a/e4e4c21b32719ca9

[Zufu Liu]

It's fine with gcc version 7.3.0 (x86_64-posix-seh-rev0, Built by MinGW-W64 project) with compiler flags g++ -mthreads test.cpp. To use C++ threads, you need posix thread model GCC with -mthreads flag.

[Martin Stumpf]

I would appreciate if this gets reopened, this is a serious problem which got confirmed after a short talk on #mingw-w64 irc. It seems to concern every thread_local object with a non-trivial destructor. If I understood it right, the memory of the thread_local object gets freed before the destructor gets called. For a compiler as significant as mingw-w64, this is a serious issue.

[niXman]

done

[Martin Stumpf]

thanks :)

[Martin Stumpf]

I'm sorry to nag, but I'm very unsatisfied with that response.

First of all, 'g++ -mthreads test.ccp' doesn't even compile the code, it's 'g++ -mthreads -std=c++11 test.cpp'.

This specific source file does not seem to crash with -O3, please verify that you didn't use optimization. (Seems to mess up the timing required for this bug to show)
It did happen to us with -O3 in more complex scenarios, though.

Second, this is not at all invalid, and if it didn't crash for you in your setup doesn't mean this bug doesn't exist. As we weren't able to find a Win10 configuration where this code does not crash, it should be fairly simple to reproduce. It currently crashes on all our production machines.

We confirmed this bug on Windows 10 on both the latest version on https://sourceforge.net/projects/mingw-w64/ and on the up-to-date version of winbuilds.

To make sure, I reinstalled the newest version of MinGW-w64 builds from sourceforge.
This is the command line output:

C:\Users\Martin\Downloads\ThreadLocalCrash>g++ --version
g++ (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

C:\Users\Martin\Downloads\ThreadLocalCrash>g++ -std=c++11 -mthreads -O3 test.cpp

C:\Users\Martin\Downloads\ThreadLocalCrash>a.exe
1: 0
2: 0
3: 0
4: 0
5: 0
6: 0
7: 0
8: 0
9: 0
10: 0
11: 0
12: 0
13: 0
14: 0
15: 0
16: 0

C:\Users\Martin\Downloads\ThreadLocalCrash>g++ -std=c++11 -mthreads test.cpp

C:\Users\Martin\Downloads\ThreadLocalCrash>a.exe
1: 0
2: 0

Then, crash. The computer running is a Lenovo T450s with Windows 10 Pro Version 10.0.16299 Build 16299. It's a quadcore, the numbers should therefore count to 16. That said, it's not computer specific but happens on all the machines I tested it on.

[Zufu Liu]

Try to build with g++ -std=c++11 -mthreads -g -O0 test.cpp, and run in gdb to get some stacktrace.

[Zufu Liu]

A related bug "#527 thread_local stl object's destructor causing crash": https://sourceforge.net/p/mingw-w64/bugs/527/

[LIU Hao]

This is a duplicate of 527. The crash will not happen on x64 where __stdcall and __cdecl are virtually identical. The crash will only be observed on x86.

[Zufu Liu]

This seems different with 527, and both can reproduced in my Lenovo desktop with AMD Athlon II X64 + Windows 10 Pro 1083 (10.0.17134.1) x64.

backtrace for 727 using

g++ -std=gnu++17 -mthreads -g -O0 test727.cpp -o t727g17
g++ -std=c++17 -mthreads -g -O0 test727.cpp -o t727c17
g++ -std=gnu++14 -mthreads -g -O0 test727.cpp -o t727g14
g++ -std=c++14 -mthreads -g -O0 test727.cpp -o t727c14
g++ -std=gnu++11 -mthreads -g -O0 test727.cpp -o t727g11
g++ -std=c++11 -mthreads -g -O0 test727.cpp -o t727c11

Thread 6 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1128.0x1694]
0x0000000000408c5c in std::__detail::_Hash_node<std::pair<int const, int>, false>::_M_next (this=0xfeeefeeefeeefeee)
    at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/bits/hashtable_policy.h:298
298           { return static_cast<_Hash_node*>(this->_M_nxt); }
(gdb) bt
#0  0x0000000000408c5c in std::__detail::_Hash_node<std::pair<int const, int>, false>::_M_next (this=0xfeeefeeefeeefeee)
    at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/bits/hashtable_policy.h:298
#1  0x0000000000409a8b in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<int const, int>, false> > >::_M_deallocate_nodes (
    this=0x26039e8, __n=0xfeeefeeefeeefeee) at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/bits/hashtable_policy.h:2096
#2  0x0000000000408e21 in std::_Hashtable<int, std::pair<int const, int>, std::allocator<std::pair<int const, int> >, std::__detail::_Select1st, std::equal_to<int>, std::hash<int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::clear (this=0x26039e8) at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/bits/hashtable.h:2029
#3  0x0000000000408f05 in std::_Hashtable<int, std::pair<int const, int>, std::allocator<std::pair<int const, int> >, std::__detail::_Select1st, std::equal_to<int>, std::hash<int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::~_Hashtable (this=0x26039e8, __in_chrg=<optimized out>) at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/bits/hashtable.h:1355
#4  0x0000000000409308 in std::unordered_map<int, int, std::hash<int>, std::equal_to<int>, std::allocator<std::pair<int const, int> > >::~unordered_map (
    this=0x26039e8, __in_chrg=<optimized out>) at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/bits/unordered_map.h:101
#5  0x00000000004085a8 in A::~A (this=0x26039e8, __in_chrg=<optimized out>) at test727.cpp:8
#6  0x000000006fc599f9 in libstdc++-6!_ZN11__gnu_debug30_Safe_unordered_container_base7_M_swapERS0_ () from D:\Dev\gcc\bin\libstdc++-6.dll
#7  0x0000000064944741 in libwinpthread-1!.pth_gpointer_locked () from D:\Dev\gcc\bin\libwinpthread-1.dll
#8  0x0000000064944b12 in pthread_create_wrapper () from D:\Dev\gcc\bin\libwinpthread-1.dll
#9  0x00007ffd0cfbaa96 in msvcrt!_beginthreadex () from C:\WINDOWS\System32\msvcrt.dll
#10 0x00007ffd0cfbab6c in msvcrt!_endthreadex () from C:\WINDOWS\System32\msvcrt.dll
#11 0x00007ffd0f193034 in KERNEL32!BaseThreadInitThunk () from C:\WINDOWS\System32\kernel32.dll
#12 0x00007ffd0f841551 in ntdll!RtlUserThreadStart () from C:\WINDOWS\SYSTEM32\ntdll.dll
#13 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

backtrace for 527 using

g++ -std=gnu++17 -mthreads -g -O0 test527.cpp -o t527g17
g++ -std=c++17 -mthreads -g -O0 test527.cpp -o t527c17

warning: Critical error detected c0000374

Thread 5 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 6080.0x1dc4]
0x00007ffd0f8c4eab in ntdll!RtlIsNonEmptyDirectoryReparsePointAllowed () from C:\WINDOWS\SYSTEM32\ntdll.dll
(gdb) bt
#0  0x00007ffd0f8c4eab in ntdll!RtlIsNonEmptyDirectoryReparsePointAllowed () from C:\WINDOWS\SYSTEM32\ntdll.dll
#1  0x00007ffd0f8cc9b6 in ntdll!RtlpNtSetValueKey () from C:\WINDOWS\SYSTEM32\ntdll.dll
#2  0x00007ffd0f8ccc81 in ntdll!RtlpNtSetValueKey () from C:\WINDOWS\SYSTEM32\ntdll.dll
#3  0x00007ffd0f869af5 in ntdll!RtlRaiseStatus () from C:\WINDOWS\SYSTEM32\ntdll.dll
#4  0x00007ffd0f8774eb in ntdll!memset () from C:\WINDOWS\SYSTEM32\ntdll.dll
#5  0x00007ffd0cf998bc in msvcrt!free () from C:\WINDOWS\System32\msvcrt.dll
#6  0x0000000000403300 in __gnu_cxx::new_allocator<char>::deallocate (this=0x652a58,
    __p=0xfeeefeeefeeefeee <error: Cannot access memory at address 0xfeeefeeefeeefeee>)
    at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/ext/new_allocator.h:125
#7  0x000000000040382b in std::allocator_traits<std::allocator<char> >::deallocate (__a=...,
    __p=0xfeeefeeefeeefeee <error: Cannot access memory at address 0xfeeefeeefeeefeee>, __n=18369900232523579119)
    at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/bits/alloc_traits.h:462
#8  0x000000000040395d in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy (this=0x652a58, __size=18369900232523579118)
    at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/bits/basic_string.h:226
#9  0x00000000004039a0 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose (this=0x652a58)
    at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/bits/basic_string.h:221
#10 0x0000000000403e75 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string (this=0x652a58, __in_chrg=<optimized out>)
    at D:/Dev/gcc/lib/gcc/x86_64-w64-mingw32/7.3.0/include/c++/bits/basic_string.h:647
#11 0x000000006fc599f9 in libstdc++-6!_ZN11__gnu_debug30_Safe_unordered_container_base7_M_swapERS0_ () from D:\Dev\gcc\bin\libstdc++-6.dll
#12 0x0000000064944741 in libwinpthread-1!.pth_gpointer_locked () from D:\Dev\gcc\bin\libwinpthread-1.dll
#13 0x0000000064944b12 in pthread_create_wrapper () from D:\Dev\gcc\bin\libwinpthread-1.dll
#14 0x00007ffd0cfbaa96 in msvcrt!_beginthreadex () from C:\WINDOWS\System32\msvcrt.dll
#15 0x00007ffd0cfbab6c in msvcrt!_endthreadex () from C:\WINDOWS\System32\msvcrt.dll
#16 0x00007ffd0f193034 in KERNEL32!BaseThreadInitThunk () from C:\WINDOWS\System32\kernel32.dll
#17 0x00007ffd0f841551 in ntdll!RtlUserThreadStart () from C:\WINDOWS\SYSTEM32\ntdll.dll
#18 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

backtrace for 527 using

g++ -std=gnu++14 -mthreads -g -O0 test527.cpp -o t527g14
g++ -std=c++14 -mthreads -g -O0 test527.cpp -o t527c14
g++ -std=gnu++11 -mthreads -g -O0 test527.cpp -o t527g11
g++ -std=c++11 -mthreads -g -O0 test527.cpp -o t527c11

warning: Critical error detected c0000374

Thread 5 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 5668.0x540]
0x00007ffd0f8c4eab in ntdll!RtlIsNonEmptyDirectoryReparsePointAllowed () from C:\WINDOWS\SYSTEM32\ntdll.dll
(gdb) bt
#0  0x00007ffd0f8c4eab in ntdll!RtlIsNonEmptyDirectoryReparsePointAllowed () from C:\WINDOWS\SYSTEM32\ntdll.dll
#1  0x00007ffd0f8cc9b6 in ntdll!RtlpNtSetValueKey () from C:\WINDOWS\SYSTEM32\ntdll.dll
#2  0x00007ffd0f8ccc81 in ntdll!RtlpNtSetValueKey () from C:\WINDOWS\SYSTEM32\ntdll.dll
#3  0x00007ffd0f869af5 in ntdll!RtlRaiseStatus () from C:\WINDOWS\SYSTEM32\ntdll.dll
#4  0x00007ffd0f8774eb in ntdll!memset () from C:\WINDOWS\SYSTEM32\ntdll.dll
#5  0x00007ffd0cf998bc in msvcrt!free () from C:\WINDOWS\System32\msvcrt.dll
#6  0x000000006fc599f9 in libstdc++-6!_ZN11__gnu_debug30_Safe_unordered_container_base7_M_swapERS0_ () from D:\Dev\gcc\bin\libstdc++-6.dll
#7  0x0000000064944741 in libwinpthread-1!.pth_gpointer_locked () from D:\Dev\gcc\bin\libwinpthread-1.dll
#8  0x0000000064944b12 in pthread_create_wrapper () from D:\Dev\gcc\bin\libwinpthread-1.dll
#9  0x00007ffd0cfbaa96 in msvcrt!_beginthreadex () from C:\WINDOWS\System32\msvcrt.dll
#10 0x00007ffd0cfbab6c in msvcrt!_endthreadex () from C:\WINDOWS\System32\msvcrt.dll
#11 0x00007ffd0f193034 in KERNEL32!BaseThreadInitThunk () from C:\WINDOWS\System32\kernel32.dll
#12 0x00007ffd0f841551 in ntdll!RtlUserThreadStart () from C:\WINDOWS\SYSTEM32\ntdll.dll
#13 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[Martin Stumpf]

All of the machines I confirmed this bug on were x64.

[LIU Hao]

There are at least two bugs related to thread_local on mingw-w64. Other than the calling convention one on i686, destructors of thread_local objects might be invoked AFTER those of static objects, which violates the c++ standard and results in crashes. There is already a report for it. Please check whether this is the case here.

[Ryan Prichard]

I also see a thread_local crash affecting mingw-w64 that might be a duplicate:

https://gist.github.com/rprichard/d0fad2161a3e2b7474b561d040ec5b40

I think this is what's going on:

• The __cxa_thread_atexit function in libsupc++ uses a pthread key to call the C++ destructors (i.e. pass a destructor to pthread_key_create).
• The storage for __thread / thread_local variables is allocated with emutls (in libgcc), which uses a different pthread key to call emutls_destroy.
• emutls_destroy can be called before the __cxa_thread_atexit destructors, and if that happens, the destructors access freed memory.

The same problem affects some (older) Android configurations: https://github.com/android-ndk/ndk/issues/687