rEFInd stalls while scanning for bootloaders

[Ville Halkola]

I'm trying to set up rEFInd / RefindPlus for boot management on my computer, but during the boot process both stall while scanning for bootloaders. Details on my computer and BIOS settings are attached. I've already discussed the matter in detail with the main developer for RefindPlus, mostly to no avail. On my previous computer, an HP 8200 Elite with the same GPU I'm using now (Nvidia GeForce GTX 645 OEM), 0.13.1 and earlier versions did their job as intended.

[Ville Halkola]

And as it's obviously not possible to attach more than one file, the log from rEFInd attached here.

[joevt]

I wonder if the log is missing info (if log lines are cached and not flushed after every line)? In that case it would be useful to have the log output to the display. A key to toggle this feature would be useful since outputting to the display would not look pretty. Might be cool to have a feature where the log goes to one display while the main UI remains on the main display.

[Roderick W. Smith]

The logging function flushes its output after every write; rEFInd hung just when starting its scan.

[Roderick W. Smith]

The hang is happening when rEFInd is trying to read a volume identified as Basic data partition. There's precisely one partition with that name (which is a GPT partition name, not a filesystem name), and it uses NTFS. This is a bit unusual, because it appears that rEFInd has not loaded an NTFS driver, and most EFIs don't include an NTFS driver. Thus, rEFInd would not normally even try to read that partition.

My best guess is that your EFI does include an NTFS driver (I have heard of this, but I've never seen it), and that it's buggy and hanging when rEFInd tries to access the volume. Another possibility is that another filesystem driver (the built-in FAT driver or the ext4fs driver) has misidentified the filesystem type because of an unfortunate coincidence or because of leftover data if the partition used to hold a different filesystem and the filesystem identification bytes weren't completely changed when creating NTFS on it. In either case, you might be able to work around the problem by telling rEFInd to not scan that volume, by using dont_scan_volumes in refind.conf:

dont_scan_volumes "Basic data partition"

(You could identify the volume by its partition GUID value or in other ways, as described in the rEFInd documentation.)

It's also possible that doing a filesystem check/repair on the volume might correct the problem. You'd do this with CHKDSK or a GUI front-end to it in Windows.

[Ville Halkola]

Excluding every single NTFS partition from the search indeed solves the problem. I could try and query HP about the drivers included in their EFI, but there is more than fair chance that this will prove to be a fruitless approach.

EDIT: I did so and the user support proved as helpful as I suspected.

[Ville Halkola]

As the problem was not present on my old HP Compaq 8200 Elite, the explanation is probably that HP added a faulty NTFS driver to their UEFI (or messed up something else), and the robust approach would be to refuse using any drivers not explicitly loaded by rEFInd itself.

[dakanji]

I don't think that is a viable option as the firmware loads several drivers and disconnecting these would create larger issues.

Try providing the NTFS driver from the current release to be loaded.
1. It might "override" the flawed one (probably unlikely)
2. If it doesn't work, then it could mean it could not be loaded due to the one from the firmware being already present.

State 2 could mean that they share the same ID. If so, then a step could be run to disconnect drivers with that ID as opposed to a blanket disconnection.

[Roderick W. Smith]

How old is this computer? I tried Googling it, and I'm getting dates that suggest both 2013/2014 era and 2019, which are wildly different in terms of the EFI implementation. The rEFInd log says it's EFI 2.31, which is old-ish. This is relevant because I've not heard of built-in NTFS drivers on such an old computer -- but then too, I've only heard of built-in NTFS drivers from one or two other people. If the problem is not a buggy built-in NTFS driver, then it's likely a bug in the ext4fs driver (or maybe the FAT driver) that's misidentifying an NTFS partition as being ext4fs.

In principle, rEFInd could do something like not scan NTFS volumes by default or attempt to disconnect firmware-provided NTFS drivers; however, in the absence of more bug reports similar to this one, such solutions strike me as likely to create as many problems as they solve. For instance, a user with an exotic setup might really want to be able to read from an NTFS volume, so ignoring or disconnecting from NTFS volumes would create problems for such a user.

Overall, at this point I think the easiest solution is to just use the workaround of adding problem NTFS volumes to dont_scan_volumes. If I start seeing more such problem reports, though, I'll give it more thought.

You're under absolutely no obligation to do more, Ville, but I would be curious to know what happens when:

• You comment out the dont_scan_volumes line in refind.conf and disable the ext4fs driver (by moving it out of the drivers_x64 subdirectory). If the problem goes away in this case, then it means that ext4fs is misidentifying at least one NTFS volume as an ext4fs volume, and fixing that bug in the ext4fs driver would be desirable. This would likely be something quirky about one or more of your specific NTFS volumes that's triggering the bug, since many people use rEFInd and its ext4fs driver on systems that dual-boot with Windows without any problems, but it still would count as an ext4fs driver bug.
• You launch an EFI shell and try to access the NTFS volume(s) from it. Chances are the system would hang, just as rEFInd does; but some more information might shake loose from the attempt.

[Ville Halkola]

I got the computer second-hand, but as the processor was launched 2016, it should be about 4 to 5 years old. Doing some more testing should not be of much more trouble, so I'll try your suggestions. And thank you very much for help.

[Ville Halkola]

Observations so far:

• moving the ext4-driver and commenting out dont_scan_volumes still leads to rEFInd stalling
• I can read the NTFS volumes from an EFI shell launched from rEFInd; however, there is an option in the BIOS boot menu to boot from a file, and an attempt to read the NTFS volumes (or anything else apart from FAT partitions) from this menu leads to the computer freezing to a point where only powering off using the power button has any effect
• listing drivers in the EFI shell indeed shows "Hp Ntfs File System Driver" version 01000237 being loaded

[Roderick W. Smith]

That's pretty conclusive proof that your firmware includes a buggy NTFS driver. It's conceivable that there's an updated firmware with a bug fix for this problem, so you might want to check that out.

FWIW, rEFInd's own NTFS driver has always been troublesome -- enough so that I stopped compiling it by default beginning with rEFInd 0.11.5, although it's still included in the source code.

In case anybody is interested, there's a new NTFS driver by Pete Batard, based on the NTFS-3g code:

https://github.com/pbatard/uefi-ntfs

I haven't yet had the chance to check it out, but it looks interesting, in part because it provides read/write support, which is a first for a non-FAT filesystem driver in EFI, AFAIK.

All of this is likely irrelevant to you, Ville. As far as I can tell from your posts, you don't really need EFI NTFS support; you just want to get rEFInd working, and HP's buggy NTFS driver was preventing that from happening. The dont_scan_volumes workaround should be adequate for you.

[joevt]

Can you post the info for the HP driver? Does the guid of the Hp driver match the NTFS driver that rEFInd includes?

You can pipe output from the EFI command to a text file on an EFI partition using command > commandoutput.txt.

Or you can create a .nsh script with a set of commands to gather info about your EFI environment. The script should be UTF-16 Little Endian (or UCS-2LE to be more precise). The first two bytes should be two bytes for the UTF-16LE BOM ("\xff\xfe").

Run the script when you make a change, then you can compare the results. Here's an example script:

set -v efishellmode 1.1.2

alias > alias.txt
cr > cr.txt
date > date.txt
devices > devices.txt

devtree > devtree.txt
devtree -d > devtree_d.txt

dh > dh.txt
dh -d > dh_d.txt
dh -v > dh_v.txt
dh -d -v > dh_d_v.txt

dmem > dmem.txt

drivers > drivers.txt

guid > guid.txt

map > map.txt
map -v > map_v.txt
map -c > map_c.txt
map -f > map_f.txt

ver > ver.txt
ver -s > ver_s.txt

Assuming fs0: is an EFI or FAT partition and the script is in the root directory:

fs0:
mkdir do_all_1
cd do_all_1
fs0:\do_all.nsh

Use a different folder name for each change.

Use the unload command to unload the Hp driver. Maybe do the following: load rEFInd's NTFS driver if it isn't already loaded. connect rEFInd's NTFS driver if it isn't already connected.

[dakanji]

There is an updated/improved ReadOnly NTFS driver (OpenNtfsDxe.efi) under the \X64\EFI\OC\Drivers folder of recent OpenCore releases:
https://github.com/acidanthera/OpenCorePkg/releases

[Arthur Roberts]

I recently ran into this same situation with rEFInd stalling while scanning for bootloaders. It occurred on two different brand-new HP desktop systems given to me by my IT department. I found a way around the issue by updating my boot environment to first load the EFI shell, which runs the startup.nsh script by default. In this script, it lists all drivers loaded by the EFI firmware, finds the handle for the one named "Hp Ntfs File System Driver", unloads it, and then continues on to rEFInd. I supplemented rEFInd to load the new NTFS-3g driver by Pete Batard listed above. I haven't attempted to get rEFInd to find bootloaders on NTFS yet, but at least it's not stalling anymore. Here's the script, in theory it could be used to unload any EFI firmware driver :

@echo -off

for %f in fs0 fs1 fs2 fs3 fs4 fs5 fs6 fs7 fs8 fs9
    if exists(%f:\EFI\BOOT\startup.nsh) then
        echo "rEFInd boot file system label is %f"
        %f:
        goto startup
    endif
endfor
:startup

echo "Searching for drivers incompatible with rEFInd..."
drivers -sfo > drivers_info.txt
if %lasterror% ne 0 then
    echo "Unable to get loaded driver information"
    goto unloaded
endif

# parse drivers info to unload drivers based on name
for %n run (1 256)
    parse drivers_info.txt DriversInfo 9 -i %n >v name
    if %lasterror% ne 0 then
        echo "Unable to get name for instance %n"
        goto unloaded
    endif
    parse drivers_info.txt DriversInfo 2 -i %n >v handle
    if %lasterror% ne 0 then
        echo "Unable to get handle for instance %n"
        goto unloaded
    endif
    if /i "%name%" == "hp ntfs file system driver" then
        echo "Unloading driver: %name% (%handle%)"
        unload -n %handle%
        goto unloaded
    endif
endfor

:unloaded
echo "Incompatible drivers unloaded, booting to rEFInd..."
if exists(drivers_info.txt) then
    rm drivers_info.txt
endif

# Hand off to rEFInd
\efi\boot\refind_bootx64.efi

:finished

[Ray Gardner]

See my post below that I should have made as a reply.

[dakanji]

That's a viable option to rely on.

An alternative added to RefindPlus is to activate its ransom_drives config token to handle such situations:
https://github.com/dakanji/RefindPlus/blob/64bf469aa91d0dbca92362e7a53dba5693263a02/config.conf-sample#L355-L369

Token name imagines some drives are being held ransom by firmware supplied filesystem drivers. When that token is active however, RefindPlus will free any such "hostage drives" it finds and use user supplied drivers instead.

The RefindPlus efi can be used as a drop-in replacement for the rEFInd efi:
https://github.com/dakanji/RefindPlus#installation

[Ray Gardner]

@relish02, Arthur Roberts: Thank you! This workaround lets me use rEFInd on my HP Probook laptop, which I could not otherwise do. Adding all the NTFS partition PARTUUIDs to my config was not viable, as I use a number of external NTFS drives all the time.

I tweaked your script a bit for my situation, copied refind_x64.efi to refind_x64_orig.efi, copied Pete Batard's EFI shellx64.efi to refind_x64.efi, and put my startup.nsh into the EFI\refind directory. Now my laptop thinks it's booting to rEFInd but really boots to the shell, which dumps the UEFI NTFS driver, and then starts the real rEFInd.

Here's the tweaked startup.nsh script:

@echo -off

for %f in fs0 fs1 fs2 fs3 fs4 fs5 fs6 fs7 fs8 fs9
    #if exists(%f:\EFI\BOOT\startup.nsh) then
    if exists(%f:\EFI\refind\startup.nsh) then
        echo "rEFInd boot file system label is %f"
        %f:
        goto startup
    endif
endfor
:startup

echo "Searching for drivers incompatible with rEFInd..."
drivers -sfo > drivers_info.txt
if %lasterror% ne 0 then
    echo "Unable to get loaded driver information"
    goto unloaded
endif

# parse drivers info to unload drivers based on name
for %n run (1 256)
    parse drivers_info.txt DriversInfo 9 -i %n >v name
    if %lasterror% ne 0 then
        echo "Unable to get name for instance %n"
        goto unloaded
    endif
    parse drivers_info.txt DriversInfo 2 -i %n >v handle
    if %lasterror% ne 0 then
        echo "Unable to get handle for instance %n"
        goto unloaded
    endif
    if /i "%name%" == "hp ntfs file system driver" then
        echo "Unloading driver: %name% (%handle%)"
        unload -n %handle%
        goto unloaded
    endif
endfor

:unloaded
echo "Incompatible drivers unloaded, booting to rEFInd..."
if exists(drivers_info.txt) then
    echo "rm drivers_info.txt"
    rm drivers_info.txt
endif

# Hand off to rEFInd
# \efi\boot\refind_bootx64.efi
\EFI\refind\refind_x64_orig.efi

:finished

It works!