Auto-scanned directories; Win10 Recovery

[Jsf]

Hi, before everything else thank you very much for great tool, updates and your work & time.

Could you explain what is the priority of auto-scanned directories?
I have no problem with configuring (possibly hiding sometimes) bootloaders for linux os'es, live images or installers

With Win10 ISO however we have:
- bootmgr.efi at root
- bootx64.efi at /efi/boot
- cdboot_noprompt.efi + cdboot.efi + memtest.efi at /efi/microsoft/boot
The same time rEFInd is showing only fallback bootloader to launch from this mine usb flashdrive partition. So in what order results are shown or taken into account?

And second question is about WIndows 10 recovery (recovery partition). What we have there is only Winre.wim at /Recovery/WindowsRE. Have to say I haven't started google on this yet but I understand this file is not startable?

ps. Ok, "scans the root directory and every subdirectory of the /EFI directory for additional boot loaders, but it doesn't recurse into these directories" - so /efi/microsoft/boot is omitted?

[Roderick W. Smith]

By default, rEFInd scans the root directory and most subdirectory of EFI of every partition it can read. (EFI/tools and rEFInd's own directory are the exceptions to this rule.) It also looks for EFI/Microsoft/Boot/bootmgfw.efi (the Windows boot loader), System/Library/CoreServices/boot.efi (the macOS boot loader), and System/Library/CoreServices/xom.efi (an old boot loader for Windows on Macs). You can add locations via the also_scan_dirs option in refind.conf and skip them with dont_scan_volumes or dont_scan_dirs.

Note that driver support also influences what's scanned. All EFIs support FAT, Macs support HFS+, and a few support ISO-9660. (Most bootable optical media use El Torito, which embeds a FAT image that rEFInd can scan; but if the firmware supports ISO-9660, rEFInd can read loaders on the ISO-9660 filesystem that are not in the El Torito image.) I've seen claims that some EFIs now have NTFS drivers, but I haven't seen this myself, nor have I seen sufficient documented details for me to be 100% convinced this is true. rEFInd ships with several drivers that add support for other filesystems, so if you're seeing entries on ext4fs, Btrfs, or other filesystems, you can keep rEFInd from scanning them by removing the relevant drivers from its drivers or drivers_x64 (or other architecture-specific drivers) subdirectory.

On any given partition, rEFInd looks for the specific boot loaders (for macOS and Windows) first, then scans the root directory, and then scans the subdirectories of EFI. Beyond that, the order of scanning is difficult to predict, because rEFInd scans both partitions and subdirectories of EFI in the order in which the EFI reveals them to rEFInd. This order can be rather arbitrary, although for subdirectories of EFI, my impression is that it's likely to correspond to the order in which the subdirectories were created, at least on FAT.

Within a directory, rEFInd sorts entries by time stamp, with the most recent entry first. This is done mainly for the benefit of Linux kernels, the idea being to keep the most recent kernel first in the entry. This is usually effective, but sometimes it's not, particularly if you've got two different kernel series installed. (For instance, in Ubuntu, you might have two 4.4-series kernels and two 4.8-series kernels installed simultaneously, and one of the 4.4-series kernels might have a more recent timestamp than either of the 4.8-series kernels. rEFInd's algorithm will then put the latest 4.4-series kernel first in the list.) You can easily adjust the sort order within each directory by using the touch command in a Unix-like OS to give the one you want first the most recent timestamp. I'm sure the same thing must be possible in Windows, but I don't know the command, offhand.

Unfortunately, sorting on a broader level would be difficult to implement, given rEFInd's data structures, which it inherited from rEFIt. There's also the question of how to sort. Most people who want this sort of thing want to be able to specify an order that's pretty arbitrary, from a computer's point of view, so this might be best handled by a new refind.conf option. That would add to the implementation complexity. This isn't to say that such changes will never happen, but the benefit is low compared to the effort to implement them, so this feature is not high on my priority list. If you really want this, your best bet right now is to disable scanning for the boot loaders you want sorted in a particular way and instead create manual boot stanzas for them. You'd then put manual in the scanfor line in refind.conf wherever you want it relative to the other scan options.

If your goal is to make it easier to launch particular boot loaders, you can look at the default_selection option. This won't sort the OS tags, but it will enable you to set whichever one you want as the default. The default for default_selection is +, which causes it to boot the most-recently-booted OS. If there's no record or if that loader can't be found, rEFInd drops back to the OS that appears first (leftmost) in its boot list.

Note also that rEFInd exempts certain filenames from being scanned. This is mostly done to reduce clutter; some boot loaders (like MokManager.efi) are known to be tools, which get their own second-row tags. Others are irrelevant or redundant -- for instance, Ubuntu often stores two kernels with names that are identical except that one ends in .efi.signed, so rEFInd presents only the one with the extra extension. The fallback boot loader (EFI/BOOT/bootx64.efi on x86-64 systems) is often a duplicate of another boot loader, so rEFInd hides the fallback boot loader if it's byte-for-byte identical with another boot loader on the same partition. Hiding extra programs has become increasingly difficult over the years because the number of such programs has grown. That's the point of the new feature I have under development, described here, which enables users to easily hide tags they consider pointless clutter. To be sure, I'm likely to continue to update rEFInd's hard-coded rules and configuration file defaults as developments warrant, but with obscure new programs coming along every couple of months, it's best that rEFInd enable hiding such programs as easily as possible.

[Roderick W. Smith]

Oh, I forgot: Most EFI executables have filenames that end in .efi (case-insensitive). Linux kernels with an EFI stub loader are the only major exception to this rule, AFAIK. The Winre.wim file you mention is not an EFI program file, so it's not shown as a boot option by rEFInd. The .wim extension indicates that this file is a Windows Imaging file -- probably a backup or emergency recovery image. It's likely used by either an EFI binary or a Windows recovery OS launched by an EFI binary.

[Jsf]

Thank you for the info, nearly all things were already clear to me however it will be useful for the others for sure.

Re.2* Yes, now I understand that W10 recovery partition contains WinRE tools but it is launched by Win OS only some way. I thought it could be started also for example by rEFInd.

Re1.
"On any given partition, rEFInd looks for the specific boot loaders (for macOS and Windows) first, then scans the root directory, and then scans the subdirectories of EFI"
... and if one doesn't specify 'also_scan_dirs' also /boot directory by default.
So what is the priority between root directory and boot directory? - that was my question in fact because lately I saw I had vmlinuz and initrd files on root also with KDE Neon. I deleted these 2 files and it was one of the kernels that was also on boot but without filename string version info.
So here is question because from what I remember rEFInd reported only kernels from /boot with string version information.

About "default_selection".
As I am reading in refind.conf (on of the latest versions) "The default behavior is to boot the previously-booted OS" what is I am happily using. But now you write " particular boot loader" selection.
You didn't mean kernel? Possibility to auto-launch previously selected OS and also kernel would be useful but there is no such thing for kernels, right?

For scanning priority why is the rEFnd choosing fallback bootlader from Win10 ISO (bootx64.efi at /efi/boot) instead of bootmgr.efi at root ?
Ok, with this question I will make a quick test to confirm, would be quickier.
Edit: rEFinfd does not "see" bootmgr.efi at root of USB flashdrive Win10 ISO partition. After renaming /efi/boot/bootx64.efi to .org it is stopping to report bootloader on this partition at all.
or I have some kind of problem with my new flashdrive

By the way.
GPT flashdrive with "a lot" of partitions. There are: FAT16 for ESP, a few FAT32 for linux live images and Win10 ISO, NTFS for data and ext4 for linux system.
Should my EFI partition has boot,esp flag ticked or not? - this is my first partition. It looks to me it is booting withour those flags set. Does name or label of the partition has sth. to do here?
Ok - I have K-ESP (to differentiate from SSD ESP) name and label and it's booting without problem without those flags. Would it be booted also it it would have been second, third or last partition?
And probably that is also question of PC UEFI software...
OK. Too much questions probably!

[Roderick W. Smith]

Re.2* Yes, now I understand that W10 recovery partition contains WinRE tools but it is launched by Win OS only some way. I thought it could be started also for example by rEFInd.

Sometimes it can be. The trouble is that OEMs put recovery tools in all sorts of odd locations, many of which are not scanned by rEFInd by default. If rEFInd happens to know about a specific Windows recovery tool, it should show up in the tools row with a windows icon inside a life preserver, as shown in the screen shot on the rEFInd main page. I'd like to have a comprehensive list of locations where the Windows recover boot loader tools go, but I don't have such a thing. Even gathering information from public sources like peoples' posted Boot Repair output is tricky, since it's usually unclear from such output what particular boot loader files do. Thus, I really need to test this myself, or have somebody willing to do so help out, before I can add another entry for Brand X's Windows recovery tool.

"On any given partition, rEFInd looks for the specific boot loaders (for macOS and Windows) first, then scans the root directory, and then scans the subdirectories of EFI"
... and if one doesn't specify 'also_scan_dirs' also /boot directory by default. So what is the priority between root directory and boot directory?

Yes, I forgot about /boot, since it's handled as part of also_scan_dirs and is therefore not explicitly flagged in the code. The order is:

1. Partition root (/)
2. EFI/* subdirectories
3. Subdirectories specified by also_scan_dirs. This is set to boot,@/boot by default.

I saw I had vmlinuz and initrd files on root also with KDE Neon. I deleted these 2 files and it was one of the kernels that was also on boot but without filename string version info.

Some distributions put a kernel in the root (/) directory and one or more in /boot. Frequently the one in the root (/) directory is a symbolic link to one in /boot. In this case, the symbolic link will probably be ignored by rEFInd; it's got code to ignore symbolic links, although that code is based on quirks of the way they appear when using rEFInd's drivers. If the distrubution uses a hard link or copies the file, rEFInd will show both entries. I've even seen cases where /boot contains a symbolic link called boot that points back to /boot (hence, there's a valid /boot/boot symbolic link to a directory), which can create duplicate entries if /boot is a separate partition.

About "default_selection".
As I am reading in refind.conf (on of the latest versions) "The default behavior is to boot the previously-booted OS" what is I am happily using. But now you write " particular boot loader" selection.
You didn't mean kernel? Possibility to auto-launch previously selected OS and also kernel would be useful but there is no such thing for kernels, right?

default_selection applies to any boot program, whether it's a Linux kernel, a Windows boot loader, a macOS boot loader, GRUB, ELILO, or anything else.

For scanning priority why is the rEFnd choosing fallback bootlader from Win10 ISO (bootx64.efi at /efi/boot) instead of bootmgr.efi at root ?

Honestly, I'm not sure. I'd need to see the complete set of all boot loaders on the media and trace the code. Is it important? This seems like a trivial issue to me.

Should my EFI partition has boot,esp flag ticked or not? - this is my first partition. It looks to me it is booting withour those flags set.

The "boot,esp flag" is simply libparted's way of reporting a partition with a type code of C12A7328-F81F-11D2-BA4B-00A0C93EC93B -- that is, an EFI System Partition. Most EFIs ignore partition type codes for most purposes, even though the EFI spec implies that the ESP type code must be set on the partition used to hold boot loaders. As a practical matter, setting it correctly can do several things:

1. The ESP will be hidden in Windows, macOS, and perhaps some other OSes.
2. Most Linux distributions' installers will know to mount the ESP at /boot/efi.
3. Some utilities, including the refind-install script, will be able to locate and mount the ESP when this is appropriate.

[Jsf]

"Sometimes it can be. The trouble is that OEMs put recovery tools in all sorts of odd locations, many of which are not scanned by rEFInd by default.
I'd like to have a comprehensive list of locations where the Windows recover boot loader tools go, but I don't have such a thing."

What I write about is proper, clean install of most current Windows 10 edition from Creators Update ISO - not OEM.
My 4 initial partitions were created automatically from scratch, these are Recovery, ESP, MSR & System. It is default Microsoft partitioning and content for an empty hard drive. On 450MiB Recovery partition we have only root/Recovery/WindowsRE and 3 files here (boot.sdi, ReAgent.xml + Winre.wim).
I just thought recovery is totally standalone but looks like it is "chainloaded" from system partition to start.
If you want any info of other files or sth let me know.

"If the distrubution uses a hard link or copies the file, rEFInd will show both entries."
Probably no. I had 2 kernels (v30 + v28) at boot. At root had copies without string version info, just for v28 was "-old" added. rEFInd reported only v30 + v28 - ///edit but I am not so sure now, maybe don't remember how many positions were in the F2 submenu.///
The same time copies at root had the same file sizes and probably even dates so if you have some magic to resolve such the situation and do not bother users with copies we are all happy.
The proper test should include for example putting v26 or v32 on root and checking if rEFInd reports all possibilities from all locations but that is outside of my scope.

"default_selection applies to any boot program, whether it's a Linux kernel, a Windows boot loader, a macOS boot loader, GRUB, ELILO, or anything else"
If we have folded kernels which is default setting then automatically always kernel with most recent timpe stamp is loaded.
It could possibly be a small missconception here so even with folded kernels last choosen kernel could be started - but that is only my personal opinion.

"Honestly, I'm not sure. I'd need to see the complete set of all boot loaders on the media and trace the code. Is it important? This seems like a trivial issue to me."
Yes - it is nothing important. However that is most current default Windows 10 install media. I've described all bootloaders on the media in first post.
How I see that is that for linux isos rEFInd reports all the bootloaders bur for most current windows iso from some reason only fallback bootloader is reported and bootmgr.efi at root is omitted.

Thank you for all the info.
How UEFI works then? I am talking about flashdrive so no NVRAM paths set.
I understand that if make second ESP partition, both would not have boot flags set then first found ESP partition would be "started"? - ok - UEFI is showing boot menu then.

[Jsf]

Hi Roderick - I was just thinking about 'folded kernels' for another moment.

I think you should know or you already know it that your software is very popular.
Just take a look here, https://medium.com/@manujarvinen/setting-up-a-multi-boot-of-5-linux-distributions-ca1fcf8d502, a professional illustrator is multi-booting 5 linux distros + Windows.
So people will multi-boot even more I think because even having a nice new Mac or PC there are also other interesting possibilities that one could load into.

So for multi-boot with more OS'es and uncluttered environment "fold_linux_kernels" option is a must. And then you have to take automatic last kernel boot or use F2. So possibly (without complicating things too much) new option 'default_folded_kernel' could be added? With possibilities: newest (default), oldest (safe solution) and last. Newest and oldest that is simple and for "last" option it takes oldest if last is no longer available during boot (kernel was removed by user).

That is just what I was thinking, I use backup software that takes on-line snapshots but it is possible only with arch lts-kernel. So I work sometimes with lts and sometimes with newest kernel but have to remember about F2 switch unfortunately.

Once again thank you for great boot manager!