[Foxgui-users] Segfault in FXString.vformat on MINGW

[Lars K. <la...@gr...>]

Hi,

in the context of FXRuby I debugged an segfault issue on Windows 10 when
using MINGW-w64 compiler. It writes to readonly memory.

On a newly created string, the string pointer points to emptystring
which is defined as static const.
When using FXString.vformat() on it, it writes a terminating zero byte
into emptystring at the first call of vsnprintf().
This results in a segfault on Windows-10 like this:

C:\WINDOWS\SYSTEM32\ntdll.dll(KiUserExceptionDispatcher+0x3a)
[0x00007ff9ce169cba]
C:\WINDOWS\System32\msvcrt.dll(ungetwc+0xc74) [0x00007ff9cbf1fe04]
C:\WINDOWS\System32\msvcrt.dll(ungetwc+0xce7) [0x00007ff9cbf1fe77]
C:\WINDOWS\System32\msvcrt.dll(ungetwc+0x880) [0x00007ff9cbf1fa10]
C:\WINDOWS\System32\msvcrt.dll(vsnprintf_l+0x80) [0x00007ff9cbf1a750]
C:\WINDOWS\System32\msvcrt.dll(vsnprintf+0x11) [0x00007ff9cbf1a6c1]
C:\Ruby24-x64\lib\ruby\gems\2.4.0\gems\fxruby-1.6.35-x64-mingw32\ports\x86_64-w64-mingw32\bin\libFOX-1.6-0.dll(ZN2FX8FXString7vformatEPKcPc+0x73)
[0x00000000696031bb]

The invalid memory access can be avoided by checking for an empty string
at first. This is done in the first attached patch file, which fixes the
sagfault.

However MINGW provides a C99 conforming vsnprintf() function as well.
Therefore the second variant of vformat() can be used instead of the third.
This is enabled in the second patch, which fixes the segfault equally.

--
Kind regards,
Lars