FreeBSD, big-endian, and many new architectures

This release finally includes FreeBSD in the list of supported operating systems. That should make it much easier to port DwarFS to other *BSDs as well. Big-endian platforms are also supported, and the file system images (which use little-endian) are fully portable between architectures. Binary releases are now available for a wide range of architectures: aarch64, arm, i386, loongarch64, ppc64, ppc64le, riscv64, s390x, and x86_64.

Metadata manipulation

Before this release, file system metadata was immutable once a DwarFS image had been created, and the only way to manipulate metadata was to build a new image from scratch. This release adds two options to mkdwarfs:

• --rebuild-metadata allows changes/upgrades to the metadata block. This makes it easy to change how the metadata is packed, or even perform manipulations such as --chmod after the fact.
• --change-block-size allows you to change the physical block size of the file system image.

Bug fixes

• The linker configuration for the release binaries was broken. The symptom was that, very occasionally, tests would fail in CI with std::terminate being called after the exception handling code failed to unwind the stack. The root cause was that, in clang builds, code from libunwind and libstdc++ was arbitrarily mixed, which—depending on the order in which individual threads were scheduled in the unit tests—could lead to stack unwinding working flawlessly or not at all. This was one of the hardest bugs to track down this year; fortunately, the fix was quite simple. It’s possible this issue is present in previously released binaries, although there have been no reports.

• Made section index discovery more robust. Fixes [#264].

• A recent kernel change (LKML thread, 2025-05-05) caused tools_test to fail on Linux 6.14 and later. This has been fixed by accepting both EPERM and ENOSYS as valid error codes for link() calls.

Features

• FreeBSD support. Everything that works on Linux should also work on FreeBSD. No static binaries are provided for FreeBSD, but the build should work out of the box once all dependencies are installed.

• Big-endian architectures. This is still experimental, even though all unit tests pass under QEMU and the benchmark suite runs on real hardware. This currently requires forked versions of folly and fsst. The changes are small, and the pull requests will hopefully be merged upstream soon. (https://github.com/facebook/folly/pull/2484, https://github.com/cwida/fsst/pull/36)

• Experimental 32-bit support. While DwarFS should largely “just work” on 32-bit with small images (a few hundred megabytes), limited address space is a problem due to DwarFS’s extensive use of memory-mapped files. There will be changes to limit the use of mmap in the future (primarily due to other issues), which should improve 32-bit compatibility as a side effect. Fixes [#268].

• Wider CPU architecture coverage. Static binary releases (including universal binaries) are now available for x86_64, aarch64, i386, arm, ppc64, ppc64le, riscv64, s390x, and loongarch64. Building the new release binaries uncovered a few bugs in clang (https://github.com/llvm/llvm-project/issues/150913), binutils (https://sourceware.org/bugzilla/show_bug.cgi?id=33223), mold (https://github.com/rui314/mold/issues/1490, https://github.com/rui314/mold/issues/1496, https://github.com/rui314/mold/issues/1497, https://github.com/rui314/mold/issues/1498), and UPX (https://github.com/upx/upx/issues/925), not all of which have been fixed. As a result, the binaries use slightly different toolchains and configurations depending on the architecture. Fixes [#266], [#268].

• Custom self-extracting stub for universal binaries. It aims for simplicity and portability and should work on most Linux systems. It is used if UPX support for an architecture is unavailable, or if the binaries extract much faster than the UPX-compressed version. The stub also supports --extract-wrapped-binary <file> to extract the embedded binary.

• Category metadata stored by default. The category metadata for categorized blocks is now stored in the metadata block by default. This allows recompressing blocks with a metadata-dependent algorithm (e.g., FLAC) even if they were previously compressed with a metadata-independent algorithm. You can disable this with --no-category-metadata. See the mkdwarfs man page for details.

• Options for smaller metadata. The --no-category-names and --no-category-metadata options can be used to reduce metadata size. However, this makes it impossible to use metadata-dependent compression algorithms (e.g., FLAC) or to select category-specific compression when recompressing the image.

• Metadata rebuilding in mkdwarfs. In addition to recompressing, it is now possible to change metadata packing and apply operations such as --set-owner, --set-group, --set-time, --time-resolution, --chmod, or --no-create-timestamp. Note that these operations are potentially lossy and may be irreversible. By default, the history of metadata rebuilds is tracked in the metadata itself; you can disable this with --no-metadata-version-history.

• Change block size on existing images. You can now change the block size of an existing image using --change-block-size. This implies --rebuild-metadata and --recompress=all and can be useful for tuning performance without recreating the image from scratch.

• Runtime memory display in mkdwarfs. mkdwarfs now shows its current memory usage while running. Note that -L/--memory-limit still only limits the memory used for the block queue, not overall memory usage. Fixing this is on the roadmap; there’s no need to file an issue.

• dwarfsextract format controls. New options --format-options and --format-filters control the output format. There is also --format=auto to automatically “guess” the format and filters based on the output file name. (Thanks to @oxalica for the pull request.)

• dwarfsck detail level. New frozen_details detail level shows frozen_analysis content ordered by memory location instead of memory usage, and also shows the address range of each section.

• Lean LRU cache. Replaced folly’s EvictingCacheMap with a simple in-repo LRU cache implementation. This reduces external dependencies and binary size without sacrificing performance.

• Windows extended attributes. The pxattr utility now supports all extended attribute operations on Windows, including setxattr() and removexattr(). Error handling and reporting for extended attributes on Windows has also been improved.

Docs

• Updated dwarfs-format.md with more information on section types, compression metadata, and the Frozen2 binary metadata layout. (Thanks to @oxalica for asking the right questions, reporting bugs, and ultimately releasing a Rust library to read/write DwarFS images.)

• Added a notable users section to the README. (Thanks to Vitaly Zdanevich for the PRs.)

• Updated mkdwarfs docs with more information on worker threads and the requirements for bit-identical images.

• Major README overhaul: added a Quick Start section, added more links, and fixed typos and wording.

Full Changelog: https://github.com/mhx/dwarfs/compare/v0.12.4...v0.13.0