Are kernel bootloaders much slower than GRUB?

[Shom Ghosh]

My rEFInd setup gives me the choice of bootloading using the kernel stub loaders or via an intermediate GRUB bootloading step. I have full-disk encryption enabled, and the crypt-password prompt shows up after choosing any option. I find that with the stub-loaders there is a significant wait period of ~30s AFTER I choose that option in rEFInd before I get to the password prompt, while with the GRUB option this is almost instantaneous.

I have the proper refind_linux.conf files in the correct places, gives all the right kernel options to the stub loaders. (The initrd images are also compiled with the encryption options, so crypt kernel options not strictly necessary.) The refind.conf file is the default one from the installation.

Question: is this slowdown to be expected because kernel stub loading is inherently slow? I would like to take out the GRUB middleman out of the picture and have rEFInd just automatically give me the kernel loaders in the menu, but the long wait period is annoying.

------- configuration -------------
My disk is partitioned into :
sda1 [efi] || sda2 [ first boot part.] || sda3 [second boot part.] || sda4 [LUKS container]

All linux installations and data partitions are within the encrypted disk sda4, which is an LVM container. I have two flavours of linux, and each has a boot partition (sda2 and sda3) outside the encrypted partition.

There are three options in rEFInd: one kernel stub loader from each installation in sda2 and sda3, and a GRUB2 installation by the system in sda2. If I make kernel changes in sda3, I need to log into sda2 via chroot and ensure the grub config remains current. I want to avoid this.

[Roderick W. Smith]

The kernel stub loader can be as fast as GRUB. The sorts of delays you report are usually caused by problems with EFI filesystem drivers. The original rEFIt drivers, early versions of the rEFInd drivers, and some of the efifs drivers are the worst offenders in this regard. More recent versions of the rEFInd drivers are better, but can still be sluggish on some computers. For some reason, speed problems show up with some EFIs but not others.

There are also differences from one filesystem to another. With the rEFInd drivers, in my experience, the ReiserFS driver is fastest and the ext2fs driver is slowest. If you don't want to use the (rapidly fading in popularity) ReiserFS, you could try Btrfs or ext4fs. Note that simply switching the driver from ext2fs to ext4fs will get little in the way of gains; the speed improvements from ext2fs to ext4fs have to do with the on-disk data structures, so you must switch drivers and re-write the filesystem.

In some cases, switching to FAT can be a good option, too. Most distributions don't really rely on any advanced filesystem features under /boot, so you can use a FAT /boot just fine, which bypasses the extra drivers entirely. Also, you can test whether this would help by copying your kernel, its matching initrd file, and refind_linux.conf to the ESP (usually /boot/efi). This should show up as a separate entry in rEFInd. If it boots faster, go ahead and experiment with other filesystems.

[Shom Ghosh]

Excellent tip, the filesystem driver must have been the culprit!

If the kernel and initrd were copied directly to the EFI partition (which needs to be fat32), no delay of any sort is noticed. This is not a long-term solution because I have to remember to copy every time the OS updates the kernel.

If I re-format the /boot partition to fat32, we again blaze through. This is not a long term solution however, because the kernel packagers tend to require symlinking in the /boot partitions. The symlinks are for harmless stuff like icons etc. and booting works fine w/o these links, but the whole apt-get-update process in debian , ubuntu etc. won't stop complaining.

Reformatting /boot to ext4 seems to be the right choice. There's a moment of delay now, about 1-2s compared to ~30s for ext2, but this is quite OK with me. This way, I don't have to do a thing and all OSes co-exist happily.

Thanks very much for the pointer!

One question:
The ext2fs must have been loaded before the rEFInd menu shows up , and I see all kernels on the ext2 boot partitions. Is the speed degradation happening when the entire kernel / initrd has to be read in once I choose a kernel on an ext2 partition?
you say "speed problems show up with some EFIs but not others." Does this mean that the problem is also from the hardware vendor who provides the EFI firmware? This is an HP laptop with an AWARD BOIS/EFI system.

[Roderick W. Smith]

I'm glad to hear you found a reasonable compromise. FWIW, I use a FAT /boot partition on one computer that doesn't work well with drivers (it seems to be a firmware bug that prevents the drivers from registering their filesystems). This works fine even with Ubuntu, although I did encounter glitches with one kernel update that changed a sub-sub-version number (from 3.13.0-ab-cd to 3.13.0-ab-ef; I don't recall the exact versions). That update seemed to involve symlinks. Others don't. Some other distributions don't even rely on symlinks that much. There's also the boot loader spec proposal, which calls for a FAT /boot partition. If that proposal ever gains traction, a FAT /boot will become standard. I'm skeptical that this proposal will go anywhere, though.

The speed problems with some drivers on some hardware aren't noticeable when reading directories or small files, but kernels and initrds are both huge compared to most files that rEFInd might read.

As to the cause, my guess is that some EFIs provide a built-in caching mechanism for their firmware and/or their disk drivers are more efficient. Combine the lack of performance on some EFIs with what's probably a not-very-efficient core to the rEFIt/rEFInd drivers and you end up with problems. Presumably the problem systems have more efficient FAT driver code to make up for the deficiencies elsewhere. I hacked in a very crude cache to the rEFInd driver core a while ago to overcome this problem, and it greatly improved performance on some systems; but a few still seem to lag. (IIRC, under VirtualBox the ext2fs driver had produced kernel load times on the order of five minutes, and my ugly little cache brought that down to about 10 or 20 seconds.) Clearly there's still room for improvement in the rEFInd driver core. Unfortunately, I don't know a lot about filesystem driver design, so I'm not the best person to improve this aspect of it. OTOH, that's an opportunity for me to learn, so maybe I'll take another look at it....

[Shom Ghosh]

That boot loader spec you linked to is a very sensible proposal and exactly what's needed for a situation where multiple linux/mac OSes co-exist.

As I understand it, /boot partitions need to just contain the kernel and initrd, and all options, icons etc. can be packaged into the initrd. I don't know why the debian maintainers require these symlinks, but they are being lazy in assuming an advanced fs for /boot.

Using ext2 for /boot has been a default choice for me for years because as I understood it none of the journaling / caching of ext3/4 are really used when loading a kernel. I stand corrected, and I am perfectly fine with having to use ext4fs for /boot now.

Thanks so much for your work with rEFInd! Configuration and use has been perfectly easy other than this slowdown issue. And I'm finally happy with dumping the tyranny of grub, which has always been painful to config-update when two or more linux systems co-exist. Now I don't have to do a thing, rEFInd works perfectly as configured.

[Roderick W. Smith]

You're right about the journal of ext3/4fs being largely unnecessary on /boot; a /boot partition is just not big enough to benefit from it. The speedup in the drivers occurs because of other performance-enhancing features specific to ext4fs, which are absent in ext3fs, such as defining files via extents rather than as block lists. You'd probably get bad performance if you used ext3fs.

FWIW, I also used ext2fs until I ran into this same issue -- although it's minor enough on most of my computers that I may still use ext2fs on one or two systems. On others, I use a variety of filesystems so as to help me catch bugs.

I'm glad you like rEFInd! I forked it from rEFIt so I wouldn't have to fight GRUB, either! ;-)