#323 Support installers larger than 2 GiB (up to 4 GiB)

[Andreas Wimmer]

Patches against trunk r7511 (git a96303ea), 2 patch files attached (git am or patch -p1):

• 0001-Support-installers-larger-than-2-GiB-up-to-4-GiB.patch (code)
• 0002-Document-the-4-GiB-installer-size-limit.patch (documentation and changelog)

Summary

All datablock offsets and lengths in the installer format are 32-bit values, but makensis and the exehead treat them as signed int (and use ftell, SetFilePointer without the high part, MulDiv, ...), which limits installers to 2 GiB. These patches handle them as unsigned values, which raises the limit to 4 GiB - 1 byte, the largest file Windows can execute.

• The layout of the file format is unchanged. Installers below 2 GiB are byte-identical to the ones of the current trunk, when compiled with the same stubs.
• A single input file > 2 GiB is supported if it compresses below 2 GiB (non-solid), or with /SOLID. The top bit of a block length is still the "compressed" flag, so a single non-solid block stays below 2 GiB.
• With /SOLID the uncompressed data is limited to 4 GiB as well, because the offsets in the instructions are 32-bit.
• When a limit is exceeded, makensis stops with a clear error message in the script line that caused it, instead of producing a broken installer. A partially written output file is deleted.

Changes

exehead (fileform.c, fileform.h)

• firstheader.length_of_all_following_data is UINT. NSIS_MAX_EXEDATASIZE is 0xffffffff; new NSIS_MAX_DATABLOCKSIZE (2 GiB - 1).
• File positions are set with the high part of SetFilePointer (SetFilePointerFromBegin()), so they are unsigned 32-bit. _dodecomp() uses offset != -1 as the "continue" sentinel instead of offset >= 0. -1 can never be a valid offset because the whole file is < 4 GiB.
• Solid mode: the temp file positions are UINT. Differences are computed with __int64 (add/sub/compare only, no CRT helpers needed).
• Decompressed byte counts are clamped (DECOMP_RETVAL), so extracting a file > 2 GiB is not reported as an error.
• The progress percentage no longer overflows MulDiv().

makensis

• IMMap/MMapFile/MMapBuf use 64-bit offsets and sizes (CreateFileMapping/MapViewOfFile with high and low DWORD, off_t on POSIX). MMapBuf::getlen() (IGrowBuf, int) reports an internal error instead of silently overflowing when used on a buffer > 2 GiB.
• add_db_data() returns INT64 (-1 on error). datablock_optimize()/datablock_finddata() use 64-bit offsets. The new datablock_check_size() enforces the limit when data is added.
• ftell64()/fseek64() (_ftelli64/_fseeki64, ftello/fseeko) for the output file.
• Size checks for non-solid, solid and the uninstaller.
• New NPRIu64 format macro (I64u on Windows, llu on POSIX) next to NPRIs.

Docs: features list (4 GB), File, SetCompress, SetCompressor, changelog.

Testing

MSVC 2022, x86 stubs, scons (Windows 11):

• Installer of 2 516 623 420 bytes, non-solid, CRC check, a file and the uninstaller behind the 2 GiB boundary: installs, SHA-256 of all files identical, uninstaller works.
• Same with SetCompressor /SOLID zlib: pass.
• Installers of 4 294 901 791 and 4 294 963 256 bytes (4 040 bytes below 4 GiB): run, all files identical.
• One byte changed at offset 4 294 900 791: rejected by the CRC check.
• 4 294 967 352 bytes: Error: installer too large (4294967352 bytes) ..., no output file left.
• Single 2.5 GiB file (compressible), non-solid: extracted correctly (before: "failed creating mmap").
• Single 2.3 GiB random file: extracted correctly with /SOLID; clear error without /SOLID.
• Plug-in DLL located behind the 2 GiB boundary (AccessControl::GrantOnFile): works.
• Output compared with trunk (same stubs): example1, example2 and a test script with uninstaller, zlib/bzip2/lzma, solid/non-solid: byte-identical (8 variants).
• Install/uninstall with all compressors, solid and non-solid: pass.

Not tested: amd64/arm64 stubs, MinGW and POSIX builds of makensis. The code is prepared for them (off_t, ftello/fseeko, NPRIu64); 32-bit POSIX needs _FILE_OFFSET_BITS=64.

Notes

• The separate ticket "Support paths longer than MAX_PATH in makensis, fix installer hang with a long %TMP%" is independent of this one. Both touch the changelog and the File documentation, so whichever is applied second needs a trivial doc merge; the code applies cleanly together.
• Installers > 4 GiB would need a format change (64-bit offsets in the instructions plus an external data file, like the NSISBI fork). That is out of scope here.

[Jason]

I was going to mention the NSISBI project for big installers, but it looks like you are already aware of my fork. I've been developing it since 2016 (gosh it's been that long now!), and I have upstreamed several fixes from my side over the years. It supports stub mode, so you can have a small .exe and use an internet plugin to download the datablock file. I recently added support for multiple file output too (but not disk spanning), so if you have a 5.5GB installer and use OutFileSize 1024, it will create 5 1GB files and a partial 6th file. I had to implement on-the-fly crc checking to make this possible. I also implemented codec multithreading to speed up those big installers.

I have a reasonably important enablement patch coming up which I will release on my fork first, and I'll also show the dev team privately to get some feedback before I submit it for upstreaming.

[Jason]

Now that I have looked at the patch, there are some things I don't like in there, like using __int64 which is not supported on certain older compilers (should be INT64), and using size_t on offsets when it should be a fixed width, and the percentage overflow I fixed just by bit shifting the inputs down 1 so that the top bit is always zero. I also used a union on the datablock offsets to convert a 64 bit offset to two 32 bit values. In the exehead I also force typecast the datablock size (all_following_data) from int to unsigned int. I also left the uninstaller sizes alone, because who needs an uninstaller bigger than 2GB? I've already run into the issue of using fseek64/ftell64 on other platforms, I fixed it by writing my own helper that works on all platforms. There are other little issues, like splitting one line of code into two and creating an extra variable just for that call, when the existing line was just fine as it was. And instead of directly printing 64 bit integers to the screen, I used FriendlySize to give a much nicer output.

I do appreciate the effort put into this, but this code looks alot like what my code was when I first started developing NSISBI back in 2016.