btrfs has recently learnt about zstd (and a while ago about lz4 I think). zstd appears to grant really high compression ratios at tremendous speed.
Also, compression levels can now be adjusted (since 4.15 I think). I'm already a happy user of zram with zstd ;-).
Looking at the current btrfs-filesystem driver for rEFInd, it looks like compression support is limited to LZO and ZLIB.
Is this true? Is there effort to add support for the new compression algorithms?
Even though I know C and C++ rather well, I'm sadly not an expert in FS driver writing, so I doubt it's a good starting point for a contribution :-(.
Cheers,
Oliver
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
That driver was written by Samuel Liao, IIRC based on GRUB 2 code. As I didn't write it myself, I've e-mailed Samuel to ask him for comments. With any luck it'll be easy for him to update the driver, or at least provide some tips for somebody (me or somebody else with more EFI or filesystem driver experience) to add the support.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
A version of the btrfs driver with this support is now available; however, I haven't tested this specific feature. You can grab the x86-64 binary here:
The source code is in the git archive. Note that I needed to make some changes to Samuel's code to get it to compile for 32-bit (x86, IA32) systems, and there's a small chance I've messed something up with that, so be careful with your testing. I recommend removing the btrfs driver you're using now and loading this one from an EFI shell, or installing this to a USB flash drive and booting from that (bypassing your current rEFInd and using the EFI's own boot manager to boot the USB flash drive). This may be over-cautious, but recovering from a buggy driver that loads automatically can be a pain.
Sorry this took so long to arrive. That's entirely on me; I just didn't get around to processing Samuel's submission for about a month.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Wow - this is still faster than I expected, so many thanks for taking care of that, to both of you!
I'll be mostly at work and far from my testing system this week, but I'll try it out as soon as possible, latest during the upcoming weekend, using the flashdrive approach. If it works fine, I'll give it broader testing on at least two systems which I want to "upgrade" to zstd.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Here's how I tested:
1. Create a USB pendrive with refind 0.11.3 (which should not support zstd).
2. Create a second partition with btrfs on the same pendrive.
3. Mount that with "compress-force=zstd".
4. Copy a kernel + initramfs on that partition.
Trying to boot that on a machine, refind got stuck a long while at the loading screen, and could not detect the kernel on the btrfs partition.
After replacing the x64 binaries of refind and drivers with the ones from 0.11.3.4 on the pendrive and rebooting, refind detects all kernels fast and can boot kernel and initramfs just fine from the zstd partition!
So I think this proves the implementation is working :-).
I'll roll that out to a machine in the upcoming week and like that give it more testing over time, but I don't see why it should fail me.
So many thanks!
Oliver
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It looks like you're using the make tiano build process, which has been deprecated for a while. I've not been testing it recently, and it looks like it's finally broken. I suggest you change to make edk2 (or make fs_edk2 to build the filesystem drivers).
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've poked at this some more, and I cannot reproduce the problem except by using the tiano target -- and on closer inspection, that's throwing an unrelated error. Because I cannot reproduce the problem, I cannot fix it myself. My guess is that it's something about the Arch build environment that's causing a different set of #include files to be used, compared to my Ubuntu environment.
That said, I can offer some suggestions for how to isolate it, fix it yourself, and supply a patch, if you're at all familiar with C programming. The problem appears to be undefined references to the memset() and memcpy() functions in the newly-added btrfs zstd decompression code. Chances are some strategically-placed #define statements to alias memset to fsw_memset and memcpy to fsw_memcpy, or to some other equivalent functions, will fix the problem. That said, it's imperative that whatever change you make be fully tested on a btrfs volume that uses the new zstd compression. If you can fix and test the problem, please feel free to send me a patch.
If you don't know enough about programming to begin with the preceding task, I suggest you file a bug report with the Arch developers.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Certainly the issue isn't the gcc version as I have used gcc 4.9.2.2 on Debian 8.8 and gcc 4.9.2-10 on Debian 9.13 and after successfully compile the TianoCore toolkit (UDK2018) as suggested, the build of rEFIn 0.12 fails with the same error on both Debian releases. Here the final output (showing 'memcopy' and 'memset' errors on 'zstd_decompress.c' that blocks the build of btrfs:
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This tiano/edk build is fixed here: https://github.com/startergo/docker-makepkg
btrfs file build is skipped. It can always be built with GNU-EFI setting _USE_GNU_EFI='1'
in the beginning of the PKGBUILD
Hi together,
btrfs has recently learnt about zstd (and a while ago about lz4 I think). zstd appears to grant really high compression ratios at tremendous speed.
Also, compression levels can now be adjusted (since 4.15 I think). I'm already a happy user of zram with zstd ;-).
Looking at the current btrfs-filesystem driver for rEFInd, it looks like compression support is limited to LZO and ZLIB.
Is this true? Is there effort to add support for the new compression algorithms?
Even though I know C and C++ rather well, I'm sadly not an expert in FS driver writing, so I doubt it's a good starting point for a contribution :-(.
Cheers,
Oliver
That driver was written by Samuel Liao, IIRC based on GRUB 2 code. As I didn't write it myself, I've e-mailed Samuel to ask him for comments. With any luck it'll be easy for him to update the driver, or at least provide some tips for somebody (me or somebody else with more EFI or filesystem driver experience) to add the support.
Many thanks! That's all I've asked for :-). So I'll keep pressing my thumbs that Samuel is still reachable / active.
A version of the btrfs driver with this support is now available; however, I haven't tested this specific feature. You can grab the x86-64 binary here:
https://www.rodsbooks.com/refind-bin-0.11.3.4.zip
The source code is in the git archive. Note that I needed to make some changes to Samuel's code to get it to compile for 32-bit (x86, IA32) systems, and there's a small chance I've messed something up with that, so be careful with your testing. I recommend removing the btrfs driver you're using now and loading this one from an EFI shell, or installing this to a USB flash drive and booting from that (bypassing your current rEFInd and using the EFI's own boot manager to boot the USB flash drive). This may be over-cautious, but recovering from a buggy driver that loads automatically can be a pain.
Sorry this took so long to arrive. That's entirely on me; I just didn't get around to processing Samuel's submission for about a month.
Wow - this is still faster than I expected, so many thanks for taking care of that, to both of you!
I'll be mostly at work and far from my testing system this week, but I'll try it out as soon as possible, latest during the upcoming weekend, using the flashdrive approach. If it works fine, I'll give it broader testing on at least two systems which I want to "upgrade" to zstd.
It seems to work!
Here's how I tested:
1. Create a USB pendrive with refind 0.11.3 (which should not support zstd).
2. Create a second partition with btrfs on the same pendrive.
3. Mount that with "compress-force=zstd".
4. Copy a kernel + initramfs on that partition.
Trying to boot that on a machine, refind got stuck a long while at the loading screen, and could not detect the kernel on the btrfs partition.
After replacing the x64 binaries of refind and drivers with the ones from 0.11.3.4 on the pendrive and rebooting, refind detects all kernels fast and can boot kernel and initramfs just fine from the zstd partition!
So I think this proves the implementation is working :-).
I'll roll that out to a machine in the upcoming week and like that give it more testing over time, but I don't see why it should fail me.
So many thanks!
Oliver
I can't build the btrfs driver:
I neglected to add a new subdirectory required by the driver to the git repository. I've done so now. Please re-sync your git repo and try again.
Some progress, but still fails to build.
It looks like you're using the
make tianobuild process, which has been deprecated for a while. I've not been testing it recently, and it looks like it's finally broken. I suggest you change tomake edk2(ormake fs_edk2to build the filesystem drivers).Nope, I'm using
make fs_edk2, https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=refind-efi-git#n203 .make fs_gnuefibuilds fine, butmake fs_edk2fails.I've poked at this some more, and I cannot reproduce the problem except by using the
tianotarget -- and on closer inspection, that's throwing an unrelated error. Because I cannot reproduce the problem, I cannot fix it myself. My guess is that it's something about the Arch build environment that's causing a different set of#includefiles to be used, compared to my Ubuntu environment.That said, I can offer some suggestions for how to isolate it, fix it yourself, and supply a patch, if you're at all familiar with C programming. The problem appears to be undefined references to the
memset()andmemcpy()functions in the newly-added btrfs zstd decompression code. Chances are some strategically-placed#definestatements to aliasmemsettofsw_memsetandmemcpytofsw_memcpy, or to some other equivalent functions, will fix the problem. That said, it's imperative that whatever change you make be fully tested on a btrfs volume that uses the new zstd compression. If you can fix and test the problem, please feel free to send me a patch.If you don't know enough about programming to begin with the preceding task, I suggest you file a bug report with the Arch developers.
It's probably do to different gcc versions, I'm using gcc 8.2.1 20180831.
A similar situation happened in the past: https://sourceforge.net/p/refind/discussion/general/thread/341a3633/ .
Unfortunately I'm not familiar with C programming, so I can't fix it. And I can't report it as a bug to Arch devs since this is not an issue with the stable refind-efi package, but with the unsupported refind-efi-git AUR package.
Certainly the issue isn't the gcc version as I have used gcc 4.9.2.2 on Debian 8.8 and gcc 4.9.2-10 on Debian 9.13 and after successfully compile the TianoCore toolkit (UDK2018) as suggested, the build of rEFIn 0.12 fails with the same error on both Debian releases. Here the final output (showing 'memcopy' and 'memset' errors on 'zstd_decompress.c' that blocks the build of btrfs:
here the output:
This tiano/edk build is fixed here:
https://github.com/startergo/docker-makepkg
btrfs file build is skipped. It can always be built with GNU-EFI setting _USE_GNU_EFI='1'
in the beginning of the PKGBUILD
Workflow
Last edit: Startergo 2021-12-31
Here is the log file from the GNU-EFI build:
Workflow
Last edit: Startergo 2021-12-31
@srs5694 btrfs EDK-2 build was fixed by @joevt:
https://github.com/joevt/RefindPlus/commit/0568967733ffc8c680fd82113c23f8d0b1cfd126
I have tested the build:
https://github.com/startergo/docker-makepkg/blob/main/PKG/PKGBUILD
You can update the upstream rEFInd code.
How much testing did you do? I did not test it at all, except to have it loaded by RefindPlus.
Testing has been done:
https://github.com/dakanji/RefindPlus/issues/86#issuecomment-980480193
Don't know how to test it, but if someone can here it is attached. Just untar it.
Now you can build rEFInd with GCC5 through Docker (has to be installed on the machine). Btrfs.efi and ntfs.efi build too.
https://sourceforge.net/u/startergo/refind/ci/2b44715c94e9c2cd0bfdc4165eccfe58908fce7a/