#324 Support paths longer than MAX_PATH in makensis, fix installer hang with a long %TMP%

[Andreas Wimmer]

Patches against trunk r7511 (git a96303ea), 2 patch files attached (git am or patch -p1):

• 0001-Support-paths-longer-than-MAX_PATH-260-in-makensis.patch (makensis, documentation, changelog)
• 0002-Fix-installer-hang-at-startup-with-a-TMP-longer-than.patch (exehead)

Problem

Input files whose path is longer than 260 characters cannot be used with makensis, even when LongPathsEnabled is set in Windows. Results with the current trunk:

• File "<long relative or absolute path>\file.txt" and wildcards: -> no files found
• !include "<long path>\inc.nsh": !include: could not find
• File /r dir with long subdirectories: builds without error, but silently misses the files
• File "\\?\<long absolute path>\file.txt": works

Causes:

1. makensis.exe has no manifest, so it is not longPathAware. Win32 file functions stay limited to MAX_PATH even if the system allows long paths.
2. win32_dir_reader::read() treats a failing FindFirstFile as an empty directory. This is why File /r silently loses files (the same happens for directories without read access).
3. There are fixed path buffers: get_full_path() (1024, checked only by assert, so a release build returns an uninitialized buffer), ${__FILEDIR__} (260), get_executable_path(), temp files (GetTempPath/GetTempFileName with MAX_PATH), OutFile (1024, silently truncated).

While testing, a Windows issue showed up as well: in a long path aware process, GetTempPathW and GetTempPath2W never return (endless loop) if TMP is longer than 260 characters. Processes without the manifest get an error code instead. That means:

• makensis would hang after adding the manifest, so the first patch handles it.
• Installers built with ManifestLongPathAware true already hang at startup on machines with such a %TMP% (reproduced with trunk). The second patch fixes the exehead.

Changes

Patch 1: Support paths longer than MAX_PATH (260) in makensis

• Source/makensis.manifest + Source/makensis.rc (RT_MANIFEST with longPathAware, asInvoker, Windows 10 supportedOS), added in Source/SConscript for Windows builds. The makensisw manifest gets longPathAware as well.
• dir_reader keeps the error of read() (error(), error_is_not_found(); Windows and POSIX).
  • do_add_file() reports a subdirectory that can not be read as an error. A missing top-level directory is still "no files found" (or a warning with /nonfatal).
  • If the path is longer than MAX_PATH and long paths are not enabled in Windows (RtlAreLongPathsEnabled), the error message explains why. The same hint is added to "failed opening file".
• No fixed size buffers in get_full_path(), ${__FILEDIR__}, get_executable_path() and temp files (create_tempfile_path_win32() falls back to its own unique name if the directory is too long for GetTempFileName). OutFile accepts up to 32767 characters and reports an error instead of truncating.
• get_temp_dir_win32() reads TMP/TEMP/USERPROFILE itself (the same lookup order as Windows) when the value is longer than MAX_PATH, and only calls GetTempPath() otherwise.
• Docs (File) and changelog.

Patch 2: Fix installer hang at startup with a %TMP% longer than MAX_PATH

• GetTempPathSafe() in exehead/Main.c uses the same approach. A long value then fails in ValidateTempDir() because GetTempFileName() is limited to MAX_PATH, so the installer falls back to %WINDIR%\Temp, exactly like installers without the manifest already do.

Behavior change

File /r over a directory that can not be read (for example access denied) now stops with an error, also with /nonfatal. Before, its files were silently skipped. Scripts that relied on that need /x for those directories. Silent data loss seemed worse, but a warning for /nonfatal would be an easy alternative if preferred.

Testing

MSVC 2022, x86, Windows 11 with LongPathsEnabled=1. "Before" is the current trunk, "after" is with both patches:

• File relative / absolute / wildcard / File /r / \\?\ / !include, path of 341 characters: before 4 failures and File /r misses 2 files; after all pass, all files included.
• Script, input file and OutFile in a directory of 1167 characters, ${__FILEDIR__}: before Can't open script; after pass, the installer extracts correctly.
• %TEMP% with 386 characters: !tempfile, !appendfile, !include, 20 MB datablock (MMapBuf temp file), uninstaller: before Unable to create temporary file; after pass, no temp files left.
• File /r with a subdirectory without read access: before the installer was built silently without its files; after failed reading directory ... (error 5).
• Missing directory, with and without /nonfatal: unchanged (error / warning).
• Installer with ManifestLongPathAware true, TMP of 315 and 1120 characters: before the installer hangs; after it runs (uses %WINDIR%\Temp).
• Same installer, TMP of 210 characters: unchanged, uses TMP.
• Output compared with trunk (same stubs): example1, example2 and test installers are byte-identical.
• Install/uninstall with all compressors, solid and non-solid: pass.

Not tested:

• the hint text with LongPathsEnabled=0 (that would mean changing the machine's policy)
• MinGW (windres) and POSIX builds (the POSIX dir_reader change was not compiled)
• makensisw

Notes

• The separate ticket "Support installers larger than 2 GiB (up to 4 GiB)" is independent of this one. Both touch the changelog and the File documentation, so whichever is applied second needs a trivial doc merge; the code applies cleanly together.
• cmd.exe also hangs with a %TEMP% longer than 260 characters, so !system/!execute/!finalize cannot work in that environment. That is Windows behavior, not addressed here.

[Anders]

• GetTempPath/GetTempPath2 are documented as "The maximum possible return value is MAX_PATH+1 (261)" so your machine setup is broken if you hit that.
• A broken cmd.exe is a bit of a deal breaker.

I would almost be more in favor of aborting with an error in makensis when %TEMP% and %TMP% are too long.

Why are you hitting this issue? Can't you just use symlinks/junctions/subst to shorten the file root path?