On macOS, sizeof(wchar_t) == 4 (UTF-32LE), but libiconv maps the encoding name "wchar_t" to UCS-2 (2 bytes). When makensis processes a Unicode script with shell constants like $INSTDIR, the internal NSIS-encoded string {0x2, 0x2081, 0x0} is passed to iconv which returns EILSEQ, causing CharEncConv::Convert() to return NULL → std::bad_alloc crash.
Fix: use "UTF-32LE" instead of "wchar_t" when sizeof(wchar_t) == 4. Patch attached covers Source/utf.cpp and Source/util.cpp.
Reproduced on: macOS 14 arm64, macOS 13 x86_64. Also affects any Linux where sizeof(wchar_t) == 4.
The latest commit doesn't compile on linux because the definition odering is wrong. I also did a little research on this issue, and found that Apple changed the definition between macOS 13 and 14, so I made a note of it too.
Thank you Jason.