Error: Not Found while loading vmlinuz... only for specific kernels?

[Devin A Hill]

I'm stumped -- rEFInd was working beautifully for quite a while, and now suddenly any kernel I build is hitting a Not Found error, even though my older kernels still work, and I don't see any enlightening details in the logs either. It's particularly strange since I'm not even using a manual boot stanza, so these kernels which are "not found" have already been auto-detected and validated to begin with, based on the logs as well.

Grub appears to be OK with these new kernels but I'd rather use rEFInd as it's much simpler to maintain and use with secure boot. However, maybe that means it's some kind of issue with the EFI stub? Could it be a problem with kernel configuration that I should fix? This issue did also happen eerily close to upgrading my GCC compiler version from 14.2.1 to 14.3, but I also tried building with LLVM and I see the same result.

Any advice about how I might check for differences between these kernels that rEFInd cares about would be greatly appreciated. Logs attached for reference.

[Scott Ellis]

I have the same issue. In my debugging, I've noticed..

1) Invoking the "not working" kernel from the EFI shell shows the same "Not found" error
2) Copying the kernel to the ESP (aka FAT32 partition) lets it boot
3) Changing the compression of /boot (and redoing the directory or cat vmlinuz>vmlinuz2) has no effect (still "Not found")
4) This happens on two different systems (ASUS ProArt x870E desktop and Lenovo Thinkpad X-something laptop), so doesn't seem system-specific.

I think this points to a btrfs EFI driver issue, not necessarily a refind issue, but I'm not sure how else to debug.

[Scott Ellis]

I think I've concluded that this is a BTRFS driver issue.

Using the btrfs_x64.efi that comes with rEFInd, I get the weird "Not Found" behavior. If I instead use the driver from https://github.com/maharmstone/btrfs-efi I can launch the kernel.

The caveat is that rEFInd doesn't see a valid kernel with the btrfs-efi driver, likely because it's zero sized due to limitations in the driver. I'll see if I can patch it up and make it fully work, but the fact that from an EFI shell I get "Not Found" vs a booted kernel by changing the driver makes me pretty darned confident it's a driver issue not a core-rEFInd or a filesystem issue.

[dakanji]

@scotteai ... Do you mind giving the BtrFS driver bundled with RefindPlus a shot?
https://github.com/RefindPlusRepo/RefindPlus/releases

You will find it under OtherBinaries/Drivers within the package.

[Scott Ellis]

I'd tried an earlier version of that, and tried the latest (0.14.2.AD) just now, both with the same result.

With that driver, it can see the BTRFS filesystem okay, but not navigate it. For example, I can go to FS2: (my Gentoo root partition) and ls the directories, but when I do cd boot I get an error that it's not a directory.

So far https://github.com/maharmstone/btrfs-efi/tree/master holds the most promise, I'm just struggling to rebuild it (so I can try patching the rest of the FS info - I see a pull request for that change from Jan 22).

[Scott Ellis]

Okay, I've got something that works. It's basically:
1) Clone https://github.com/maharmstone/btrfs-efi
2) Apply the patch from https://github.com/maharmstone/btrfs-efi/compare/master...heipiao233:btrfs-efi:heipiao233-fix-file_get_info.diff
3) Update to current build dependencies for gnu-efi, zstd, and zlib.

Build with cmake.

I have the changes at https://github.com/ScottESanDiego/btrfs-efi along with a binary that works for me. Would love to have feedback if it works elsewhere. I've only tested amd64 using mingw.

[dakanji]

Don't use BtrFS myself but have gotten reports of issues such as this:
https://github.com/RefindPlusRepo/RefindPlus/issues/194
While it was apparently affecting other items, Ext4, now seems likely related to the BtrFS driver.

I had noted a bug in the driver here
https://github.com/RefindPlusRepo/rEFInd/blob/253abe5c4af58d044912517daa567c4440612c46/filesystems/fsw_btrfs.c#L2161-L2167

Added a comment to RefindPlus on this here:
https://github.com/RefindPlusRepo/RefindPlus/blob/753b1681b0d93d7f3651a3ff6c0260434e08d396/filesystems/fsw_btrfs.c#L2166-L2176. The note actually meant to say "!err" is always matched

This means that the for loop, as written, will always only run once and always immediately return FSW_SUCCESS after processing the first item. Would not be a loop if intentional.

This went in with the original creation of the driver back in 2013. Not sure if relevant to the issues being reported but can't be a good thing either way. Might also be some other bugs lurking.

All said and done, while good to have a working option, would be nice to fix the driver here.

[Scott Ellis]

Agreed - that issue does look like it's relevant for the subdir navigation problem I was seeing. Would be good to have a supported solution, but not the issue I'm wanting to throw my time into.

It's easy for me to test a fix if you come up with something.

[dakanji]

Well, try the attached build. [REMOVED]

Took a guess at likely reason the parent function will be called and from this, assumed likely condition to check for instead of the problem !err check.

[dakanji]

@scotteai ... Awaiting your feedback on the post above

[Scott Ellis]

That binary seems worse. Now it detects the btrfs filesystems, but can't ls the root....well technically it doesn't throw an error, it just reports 0 files/0 dirs.

[dakanji]

@scotteai ... Thanks.

Well, that was actually useful as it confirms the relevance of that block to directory traversal.

Please test the attached tweaked version. [REMOVED]

[Scott Ellis]

No change with that version. I still get 0 File(s), 0 bytes, 0 Dir(s) on the BTRFS partition when ls'ing it.

Do you want to make a version with some debugging output that might shed some light on what is happening?

[dakanji]

The debug infrastructure is broken/inactive on the FS drivers.
Will look into resurrecting. Likely needed if the [REMOVED] still doesn't work

[Scott Ellis]

Okay, trying with the above version...no change. Still 0 files, etc. when trying to ls fs2: (my btrfs partition).

Seems the binary from https://sourceforge.net/p/refind/discussion/general/thread/172b9cbf09/#b783 was better since it at least got the top-level directory working, but none of these are functional-enough to be useful so far.

[dakanji]

Thanks for the tests. It was a stab in the dark.

Will look at sorting the debug logging on the FS drivers out and also consider installing a BtrFS instance at some point to troubleshoot if a fix has not been found in the interim..

[dakanji]

@scotteai ... Please try the [REMOVED] .

Has another attempt at a fix but importantly, includes a resurrection of the Debug Print macros from rEFIt: FSG_MSG_ASSERT, FSG_MSG_DEBUG, and FSG_MSG_DEBUGV.

You should see debug output on the screen in EFI Shell.

[Scott Ellis]

Way too much debug info - been scrolling as the driver loads (or maybe as rEFInd is enumerating files after load?) for about 10m so far. :-) Ah, how I love poor scrolling of the framebuffer by the FW. :-)

What should I be looking for?

[Scott Ellis]

Here's two photos...one just after the driver load (e.g., load foo.efi) and another after an ls of the root of the btrfs partition. I can't capture the start of the output easily.

[dakanji]

Way too much debug info

I gave you a build version with FSW_DEBUG_LEVEL=3 which is the most detailed level.
The [REMOVED] is FSW_DEBUG_LEVEL=2 Hopefully more manageable.

BTW, it seems your issue with the original driver is NOT universal and it works for many.
Is there anything unusual with your setup? RAID or something else?

[Scott Ellis]

Mine is RAID1.

[Scott Ellis]

Apologies for the screenshot again - I can't figure out how to get stderr output to go to a file (stdout works, but isn't that interesting!).

[dakanji]

The screenshot is fine. Thanks.

So far seems the "Bug" is critical to making things "work".
Still looks odd to have such a redundant loop.

Will have one more stab at it presently.
Can then look at what debug output is gotten with it in place now that we can get this.
That is, as per your first check here: https://sourceforge.net/p/refind/discussion/general/thread/172b9cbf09/#c3b7

[dakanji]

@scotteai ... Please try the attached.

Eliminates the Redundant Loop "Bug" and tries to improve the overall iteration process.
Works in a regular setup. Hopefully also does in your RAID setup. If not, then the issue is likely in another part.