I am trying refind on Ubuntu Vivid 15.04 with secure boot. I followed the
instructions on the refind website, and installed refind using Ubuntu's
shim, but for some reason I cannot register the refind.cer key. When I
reboot, refind gives the 'Access Denied' error, MokManager starts, I
navigate to the key and add it, but when I reboot refind goes again to
'Access Denied' etc...
If I boot my distro, Ubuntu, and use mokutil to see the keys that have been
enrolled, I see the refind key. However, I am still not able to boot refind.
I also tried using shim from Fedora, but I have the same problem. The same
thing happens with the shim 0.2 program suggested by Roderick.
Everything works perfect with Secure Boot disabled
I am using refind version 0.9.0-0ppa1 from the Ubuntu PPA
Thank you for your help
Valerio
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If you're using the PPA, then it's signed locally, not using the rEFInd key. (The reason is that the rEFInd private key exists only in my possession, but the rEFInd PPA binaries are built on Canonical's servers. Thus, the PPA binaries cannot be signed by the rEFInd key.) Thus, the key you need to enroll is the refind_local.cer key, which should exist in the same directory as the other keys on the ESP. This key is unique to your computer.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks Roderick. In the /boot/efi/EFI//refind/keys folder, I found a key called refind_local.cer , and I enrolled it. Refind now starts with secure boot. In Refind I now see the entry for the EFI/ubuntu/grubx64.efi bootloder of the Ubuntu installation. However, if I select it, the verification fails and I cannot boot. But I don't understand: since I am using shim from the Ubuntu installation, shouln't the key with which the Ubuntu grub bootloader is signed be already enrolled? I am sorry but all these key enrolling is not completely clear to me.....
I thought of enrolling the Canonical key, but if I try to launch the EFI/refit/MokManager.efi entry, the verification fails, preventing me from enrolling the key.
Thank you for all the help you are providing, Roderick
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You mentioned in your first post that you tried Fedora's Shim. If you're still using it, that would explain why you can't launch Ubuntu's GRUB. If this is the case, adding the canonical-uefi-ca.der key should get things working. Of course, if you can't launch MokManager from rEFInd, that will cause problems. You could try enrolling the key with Secure Boot temporarily disabled, though. Alternatively, you could try another copy of Shim, being sure to copy both the shimx64.efi (or shim.efi, in some cases) and MokManager.efi binaries from the same source.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear Roderick. I tried with both Ubuntu and Fedora Shims. All give the same result, after enrolling the local refind key, refind starts, but everything except Windows gives a validation error. Enrolling the keys with secure boot off works, but once secure boot is back on, it has no effects, everything still gives a validation error. Could this be a bug in refind? This behavior seems independent of the shim version...
Thank you again
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Please enroll the keys for the distribution(s) you're trying to launch -- canonical-uefi-ca.der for Ubuntu, fedora-ca.cer for Fedora, and so on. Don't bother re-enrolling the rEFInd (or rEFInd local) key.
If that fails, you might give PreLoader a try rather than Shim. PreLoader does things a little differently, and might conceivably work better in some cases. See the rEFInd documentation on PreLoader for details.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear Roderick, thank you so much for your help. Please understand that I am not trying to criticize or suggest that refind is not working, I am just trying to get to the bottom of this key enrolling businness. Let me explain to you what I do, so we see if I am doing something wrong.
So, I either do this:
(clean start) -> install refit from the ppa -> reboot -> enroll local_refind.cer -> refind boots -> everything (ubuntu, fedora and all the MokManagers) except Windows gives "Validation Error" -> turn off secure boot -> enroll canonical and Fedora keys -> turn on secure boot -> reboot -> everything gives 'Validation Error' except windows as before, cannot boot anything
Or this:
(clean start) -> install refit from the ppa -> copy over shim and MokManager from Fedora -> reboot -> enroll local_refind.cer -> refind boots -> ubuntu and all the MokManagers give "Validation Error", and the fedora gubx64.efi entry does the same, but the Fedora gcdx64.efi entry boots and I can boot fedora -> turn off secure boot -> enroll canonical keys -> turn on secure boot -> reboot -> everything as before, cannot boot Ubuntu or all the MokManagers, can boot gcdx64.efi
Please notice that when I say that I cannot boot Ubuntu, I don't mean that I cannot boot the kernels. My kernels are in a subvolume on a btrfs partition and refind does not detect them, apparently. What I mean is that I cannot boot the grubx64.efi entry of Ubuntu, that is detected by refind and launches the grub bootloader from Ubuntu (this would be ok for me)
Am I doing something wrong? It seems to me that something must be wrong when I enroll the canonical keys, after installing refind....
Thank you again for all the help
Valerio
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
What you're doing SHOULD work, and does work on most computers. My best guess at this point is that your firmware doesn't implement the methods that rEFInd uses to register itself (and Shim) as a way to authenticate binaries. (As background, know that Shim is a one-off deal; it launches a follow-on program using very non-standard means. The follow-on program -- normally GRUB -- then calls back to Shim for authentication. GRUB is a boot loader, which means it doesn't rely on EFI system calls to launch the kernel, so the EFI doesn't really need to know what Shim does. rEFInd is a boot manager, not a boot loader, so it relies on EFI system calls to launch follow-on programs, including the kernel. Thus, rEFInd does what Shim doesn't and ties Shim more directly into the EFI authentication system. It's this linking together of the EFI and Shim that I think is failing.)
I'd be interested in knowing more about your computer and firmware. What make and model is it, and what's the firmware version? (You can type dmesg | grep -i efi | less and look for a line early in the output identifying the firmware version and manufacturer.) When did you buy the computer? (In part, I'm trying to determine if this is a problem with an obsolete EFI or if it's a problem with very recent firmware that might start affecting more people in the future.)
As to moving forward, there are a number of options:
You can try upgrading your firmware. (Most manufacturers call it a "BIOS upgrade," although an EFI isn't really a BIOS.) If the problem is a bug in the EFI, an upgrade might fix it. Note that an update might wipe out your MOKs, and perhaps your boot order, so you may need to re-register things to get even as far as you've gotten.
If you're interested, I can provide debugging versions of rEFInd that display status information as it tries to register Shim with the EFI. You can take digital photos of its displays to feed back to me and I can provide further binaries. It may be possible to track down and correct the problem in this way; however, I'm not optimistic about this. If this were to yield a fix, though, it would be my preferred solution, since this would ensure that others won't run into the same problem (with the same cause) in the future. If you want to pursue this course, e-mail me directly at rodsmith@rodsbooks.com.
You can try the rEFInd Debian package that I provide. This version is compiled with a different library than the one in the PPA. There's a slim chance that this will fix the problem, but again, I'm not optimistic that it would help. If it were to help, then chances are that properly debugging the problem, as above, would yield a solution, too.
You can disable Secure Boot. This is the easiest solution, but of course if you must have Secure Boot, this is not an acceptable solution.
You can modify your Secure Boot keys at a deeper level than Shim and MOKs permit, as described on this page of mine. Basically, you'd use your firmware and/or third-party tools to add your own key, the keys for whatever distribution(s) you want to boot, and perhaps my own rEFInd key, to your firmware. This way, there will be no need for Shim, and everything should be validated correctly by the firmware. This solution is a bit tedious to implement, but it gives you the best control possible over your Secure Boot operation.
You can abandon rEFInd in favor of GRUB.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear Roderick, thanks for your detailed answer. I am a programmer by profession, so I would really like to pin down the problem and help debugging it, if you are willing. Not only this will make refind work for me, but also for others who might have the same problem.
So, regarding the options you suggest: I would prefer to disable secure boot, and I cannot go back to grub2 because I want to install DragonflyBSD on my machine. That does not have an EFI bootloader and refind is the only bootloader that can boot both EFI and BIOS bootloaders.
After that introduction, let me say that I do not think it is BIOS-related. I have the same behavior on two laptops. One is a Dell Inspiron 15 3551 with BIOS revision A2. The other is a Lenovo Yoga 2 Pro (I don't have it with me now and I don't remember the BIOS revision). I updated the BIOS on both machines but the behavior stayed the same.
I will try the debian packages next. Please feel free to send be the debugging build of refit, I will be happy to help debugging...
If you're seeing this on multiple computers, then my suspicion is that you're doing something that's causing unsigned packages to be installed or something is going wrong when you register your keys, unrelated to a rEFInd or EFI bug. I say this because you're the only person to have reported this problem, and the probability of one person encountering an obscure bug that's dependent on the EFI version on multiple computers is extremely low.
I therefore recommend running a couple more tests:
Install multiple copies of rEFInd, side-by-side. For instance, if rEFInd is in EFI/refind, copy that entire directory tree to EFI/refind_test, but delete shimx64.efi and MokManager.efi just to keep things uncluttered. You can then see if you can launch the second rEFInd from the first. If you can, then Shim (associated with the first rEFInd) is working correctly.
Install an EFI filesystem driver for whatever filesystem holds your kernels. You should then be able to launch your Linux kernel(s) directly from rEFInd. Note that you'll need to sign the filesystem driver yourself, as described here (especially step #6; the preceding steps should already be done). The keys are stored in /etc/refind.d/keys. If you can see the kernels in the rEFInd menu, then that means that Shim is working correctly, since the signed driver is launched like any other binary. If you can see the kernels but get a Secure Boot error, then that indicates a problem with the specific key.
Since you're a programmer, you can play with the code yourself, if you like. If there's a problem registering the Secure Boot support, it will likely be in the mok/security_policy.c file, in particular in the security_policy_install() function. (This function is in turn called from refind/main.c's SecureBootSetup() function.) You can add Print() statements (similar to a standard C print() function; see examples throughout the code) to display messages. You'll probably need a PauseForKey() somewhere to keep the messages on screen long enough to read them. Any debugging version I'd send you would do just that at strategic points to let me know what's going on as the program runs.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear Rockerick, ok! I run some tests and finally got it to work! I completely agree, I don't think the problem is refind, but as you say, it is the interaction with shim that seems to fail.
First of all, I run the tests you described. I created a second copy of refind, and the first one cannot boot it. I then had refind detect all the kernels for ubuntu and fedora, and refind cannot boot them. All give verification error.
At this point I concluded that the problem was shim. I thought that the problem was the version of shim that came with ubuntu, so I booted fedora and tried to install refind from the fedora side, using the rpm package you provide. This was all useless. Everything gave verification error.
I finally tried the version of shim that you link on your website (0.2). This now worked! I could add the refind key and then the fedora and canonical. This gave no error and I could boot everything, including MokManager.
So I am also now convinced that the problem is the interaction of refind with shim. I checked and both ubuntu 15.04 and fedora 22 ship with shim 0.7. Do you know of any problem in the interaction with this version of shim?
Even if I got it to work, I would still like to help debugging this. I am available for tests and debugging, if you are interested. In the next few weeks I don't have time to look at the code, but I will be happy to run any debugging version and to report the logs.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for continuing with this -- I've managed to reproduce the problem using Shim 0.8 from the Ubuntu 15.10 pre-release. I'll look into it this week, and with any luck find a fix soon. In the meantime, sticking with a pre-0.7 Shim should be a suitable workaround.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I spent a bit of time this morning looking into this. I was unable to reproduce the problem with Shim 0.7 from Ubuntu 15.04 (Shim 0.8 is available as an update to 15.04), but Shim 0.8 from the pre-release Ubuntu 15.10 does show the problem. Could you please verify that you've had the problem with Shim 0.7? Here's a URL to a tarball with Shim 0.7 from a non-updated Ubuntu 15.04:
FWIW, I've discovered that the problem is that the first call to Shim 0.8 from rEFInd succeeds, but subsequent calls always return a failure (as if the binary were not signed by a MOK). Note that if you've got a filesystem driver installed, that first call will be to validate the driver, so it will be impossible to launch anything else after that. If you had no driver installed, you should at least be able to launch GRUB -- but I don't yet know if its calls would succeed or fail.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Rodrick. I too have encountered the exact same symptoms as Valerio Mariani has in this thread. I've been pulling my hair out all day fighting with this, even going to the lenghts of a full wipe-and-reinstall of linux mint 17.2! I was worried that microsoft actually pulled the trigger and perma-revoked all my enrolled keys during my last windows update (win8.1) in some sort of misguided push-everyone-to-win10-only move.
I'm glad to find you were already working on this problem. For the record, my machine is an HP model 2000 laptop, with an EFI v2.31 by INSYDE corporation....if that makes any difference or helps at all. I do know its a late 2012 model.
Anyway, will test your provided shim files. Question, though -- should I put them in /EFI/refind/ or /EFI/ubuntu/ or both?
-edit-
Your 0.7 shim files worked nicely, I'm back up and running. Thank you Rodrick
Last edit: Joel Brown 2015-09-16
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've traced the problem to a change in Shim 0.8. It seems that, to work around a problem with a particular boot path involving the fallback.efi binary and two Shims, the Shim developers now make Shim essentially unregister itself from the system when the first follow-on program is launched. Thus, rEFInd can authenticate one program and no more. The good news is that I've found a workaround. Here's a test version, in case you care to try it:
This is a preliminary workaround, and it's a bit of an ugly and delicate hack. (It could break again if the Shim developers change their logic.) I've e-mailed the Shim developers about this, so I hope we'll be able to coordinate something better in the long term. Unfortunately, Shim binaries often go without updates for a long time, so rEFInd will have to cope with Shim 0.8 problems for months to come. :-(
This is obviously a high-priority issue, so unless I hear of problems with my workaround or discover a better way to do it, I'll probably push that (perhaps with one or two other minor updates) out as rEFInd 0.9.2 by the end of the week.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear Roderick, it works perfectly for me on my two laptops. I am curious, however: if shim unregisters itself, how can a bootloader, for example grub in a normal Ubuntu installation, authenticate the kernels? Also, did you receive an answer from the shim developers?
Thank you again for fixing this!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
My workaround takes advantage of the way Shim is coded to trick it into not unregistering itself. (Slightly technical: Launching a program in EFI involves two system calls that Shim intercepts. The first call loads a binary into memory and the second one executes it. Shim unregisters itself on the second of these calls, but only if the program being launched was the last one to be loaded with the first call. My workaround causes rEFInd to load a copy of itself after it loads the program it intends to launch, so Shim's check doesn't match and Shim doesn't unregister itself. This is an ugly solution and it might fail if Shim's developers change the relevant logic, so we could be back at the same point again in the future.)
Yes, I'm in contact with a Shim developer, but there's no perfect solution as of yet. Even if we settle on a long-term solution, there'll still be Shim 0.8 binaries floating around for a long time, so I might need that ugly workaround in rEFInd, at least for a while....
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Joel, I don't think that is possible, if I understand correctly how this works, shim needs to be signed with the Microsoft key, and Roderick does not have access to the signing program. I don't think Roderick can distribute someone else's binary . Plus shim 0.7 has the same problem.
But I wonder, what was the technical reason for this change, do we know? This is not needed to solve the bug, it is already solved. This is just for my curiosity, I like to learn things....
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thus, redistributing the binary should be OK, AFAIK. The trouble is that if I redistribute Distribution A's Shim, then somebody using Distribution B or C will have to register both my (or their, for locally-signed) key and their distribution's key. If the user can use their own distribution's Shim, then they still need to register the rEFInd (or their local) key, but not their distribution's key. I could pay the $99 to get access to the Microsoft signing program, which would enable me to distribute a Shim with rEFInd's key built in, in which case users would have to register their distributions' own keys. This is no advantage to users compared to using a distribution-provided key; and since the signing program requires a lot of bureaucratic hoop-jumping, it's not seemed worth the effort. That's something I'm reconsidering, though.
In my own tests, Shim 0.7 does not have the problem, but I'd appreciate it if Valerio could check that using the Ubuntu Shim 0.7 to which I linked earlier:
Note that some distributions, including Ubuntu 15.04, shipped originally with one Shim but then updated to a later one. That tarball is named for Ubuntu 15.04, but it includes the original 0.7 version of Shim, not the latest version for that distribution.
As to the technical reason for the change, that's described in a comment in the Shim 0.8 source code:
See the start_image() function. Basically, in a boot path that involves Shim --> fallback.efi --> Shim --> GRUB, you can wind up with GRUB using a mixture of the first and second Shims' features, which is undesirable. This path, it should be noted, is how Red Hat has worked around an EFI bug that's really the bane of the Linux world: Some EFIs forget or ignore the EFI boot order, so Red Hat wrote the fallback.efi program to reside in the fallback position (EFI/BOOT/bootx64.efi) to restore the boot order variable and seamlessly continue the boot process. The end result is that we've got a cascading set of problems and workarounds to those problems (Secure boot yields Shim, EFI bugs yield fallback.efi, fallback.efi yields a Shim "self-destruct" feature, and the Shim "self-destruct" feature yields a crippled rEFInd.). Problems caused by a preceding program require adjustments to follow-on programs because the preceding program can't always be controlled by the person who needs it fixed. At least with rEFInd I have the option of releasing my own Shim, but it will take time and effort for me to do this. There are other possible solutions within rEFInd that I'm investigating, but they've got problems of their own (a lot of extra programming effort, program bloat, or the possibility of crashes, for instance).
So the bottom line is that the workaround in the 0.9.1.1 binary is likely to be a temporary band-aid solution at best. Moving forward, options are:
For 0.9.2, I may use the band-aid solution or distribute a Shim 0.7 signed by somebody else and require users to enroll two keys if they don't use that "somebody else's" distribution.
Maybe for 0.9.2, and definitely for later, some other in-rEFInd solution may become available.
For later versions, I may distribute my own signed Shim or I may find some other technical solution within rEFInd. There's also the possibility of better communication between Shim and rEFInd (and, in principle, other follow-on programs) to bypass the problem.
At the moment, it's just too early to say what will happen; I'm still mulling the possibilities, and it's possible that some new technical solution will appear.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Here I am. Sorry for the delay. So, shim-1504 that you posted works. What I mean is that this works with the regular refit in the PPA, not with the updated version you posted for testing
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Dear Refind developers,
I am trying refind on Ubuntu Vivid 15.04 with secure boot. I followed the
instructions on the refind website, and installed refind using Ubuntu's
shim, but for some reason I cannot register the refind.cer key. When I
reboot, refind gives the 'Access Denied' error, MokManager starts, I
navigate to the key and add it, but when I reboot refind goes again to
'Access Denied' etc...
If I boot my distro, Ubuntu, and use mokutil to see the keys that have been
enrolled, I see the refind key. However, I am still not able to boot refind.
I also tried using shim from Fedora, but I have the same problem. The same
thing happens with the shim 0.2 program suggested by Roderick.
Everything works perfect with Secure Boot disabled
I am using refind version 0.9.0-0ppa1 from the Ubuntu PPA
Thank you for your help
Valerio
If you're using the PPA, then it's signed locally, not using the rEFInd key. (The reason is that the rEFInd private key exists only in my possession, but the rEFInd PPA binaries are built on Canonical's servers. Thus, the PPA binaries cannot be signed by the rEFInd key.) Thus, the key you need to enroll is the
refind_local.cerkey, which should exist in the same directory as the other keys on the ESP. This key is unique to your computer.Thanks Roderick. In the /boot/efi/EFI//refind/keys folder, I found a key called refind_local.cer , and I enrolled it. Refind now starts with secure boot. In Refind I now see the entry for the EFI/ubuntu/grubx64.efi bootloder of the Ubuntu installation. However, if I select it, the verification fails and I cannot boot. But I don't understand: since I am using shim from the Ubuntu installation, shouln't the key with which the Ubuntu grub bootloader is signed be already enrolled? I am sorry but all these key enrolling is not completely clear to me.....
I thought of enrolling the Canonical key, but if I try to launch the EFI/refit/MokManager.efi entry, the verification fails, preventing me from enrolling the key.
Thank you for all the help you are providing, Roderick
You mentioned in your first post that you tried Fedora's Shim. If you're still using it, that would explain why you can't launch Ubuntu's GRUB. If this is the case, adding the
canonical-uefi-ca.derkey should get things working. Of course, if you can't launch MokManager from rEFInd, that will cause problems. You could try enrolling the key with Secure Boot temporarily disabled, though. Alternatively, you could try another copy of Shim, being sure to copy both theshimx64.efi(orshim.efi, in some cases) andMokManager.efibinaries from the same source.Dear Roderick. I tried with both Ubuntu and Fedora Shims. All give the same result, after enrolling the local refind key, refind starts, but everything except Windows gives a validation error. Enrolling the keys with secure boot off works, but once secure boot is back on, it has no effects, everything still gives a validation error. Could this be a bug in refind? This behavior seems independent of the shim version...
Thank you again
Please enroll the keys for the distribution(s) you're trying to launch --
canonical-uefi-ca.derfor Ubuntu,fedora-ca.cerfor Fedora, and so on. Don't bother re-enrolling the rEFInd (or rEFInd local) key.If that fails, you might give PreLoader a try rather than Shim. PreLoader does things a little differently, and might conceivably work better in some cases. See the rEFInd documentation on PreLoader for details.
Dear Roderick, thank you so much for your help. Please understand that I am not trying to criticize or suggest that refind is not working, I am just trying to get to the bottom of this key enrolling businness. Let me explain to you what I do, so we see if I am doing something wrong.
So, I either do this:
(clean start) -> install refit from the ppa -> reboot -> enroll local_refind.cer -> refind boots -> everything (ubuntu, fedora and all the MokManagers) except Windows gives "Validation Error" -> turn off secure boot -> enroll canonical and Fedora keys -> turn on secure boot -> reboot -> everything gives 'Validation Error' except windows as before, cannot boot anything
Or this:
(clean start) -> install refit from the ppa -> copy over shim and MokManager from Fedora -> reboot -> enroll local_refind.cer -> refind boots -> ubuntu and all the MokManagers give "Validation Error", and the fedora gubx64.efi entry does the same, but the Fedora gcdx64.efi entry boots and I can boot fedora -> turn off secure boot -> enroll canonical keys -> turn on secure boot -> reboot -> everything as before, cannot boot Ubuntu or all the MokManagers, can boot gcdx64.efi
Please notice that when I say that I cannot boot Ubuntu, I don't mean that I cannot boot the kernels. My kernels are in a subvolume on a btrfs partition and refind does not detect them, apparently. What I mean is that I cannot boot the grubx64.efi entry of Ubuntu, that is detected by refind and launches the grub bootloader from Ubuntu (this would be ok for me)
Am I doing something wrong? It seems to me that something must be wrong when I enroll the canonical keys, after installing refind....
Thank you again for all the help
Valerio
What you're doing SHOULD work, and does work on most computers. My best guess at this point is that your firmware doesn't implement the methods that rEFInd uses to register itself (and Shim) as a way to authenticate binaries. (As background, know that Shim is a one-off deal; it launches a follow-on program using very non-standard means. The follow-on program -- normally GRUB -- then calls back to Shim for authentication. GRUB is a boot loader, which means it doesn't rely on EFI system calls to launch the kernel, so the EFI doesn't really need to know what Shim does. rEFInd is a boot manager, not a boot loader, so it relies on EFI system calls to launch follow-on programs, including the kernel. Thus, rEFInd does what Shim doesn't and ties Shim more directly into the EFI authentication system. It's this linking together of the EFI and Shim that I think is failing.)
I'd be interested in knowing more about your computer and firmware. What make and model is it, and what's the firmware version? (You can type
dmesg | grep -i efi | lessand look for a line early in the output identifying the firmware version and manufacturer.) When did you buy the computer? (In part, I'm trying to determine if this is a problem with an obsolete EFI or if it's a problem with very recent firmware that might start affecting more people in the future.)As to moving forward, there are a number of options:
Dear Roderick, thanks for your detailed answer. I am a programmer by profession, so I would really like to pin down the problem and help debugging it, if you are willing. Not only this will make refind work for me, but also for others who might have the same problem.
So, regarding the options you suggest: I would prefer to disable secure boot, and I cannot go back to grub2 because I want to install DragonflyBSD on my machine. That does not have an EFI bootloader and refind is the only bootloader that can boot both EFI and BIOS bootloaders.
After that introduction, let me say that I do not think it is BIOS-related. I have the same behavior on two laptops. One is a Dell Inspiron 15 3551 with BIOS revision A2. The other is a Lenovo Yoga 2 Pro (I don't have it with me now and I don't remember the BIOS revision). I updated the BIOS on both machines but the behavior stayed the same.
I will try the debian packages next. Please feel free to send be the debugging build of refit, I will be happy to help debugging...
Valerio
PS: I found another thread of someone with the same problem: https://sourceforge.net/p/refind/discussion/general/thread/44a83d67/
I tried the deb package you provide, but I get exactly the same behavior, at least on the Dell.
Incidenta;ly, I checked that grubx64.efi is signed with the canonical key as provided in refind, and the check was positive
Last edit: Valerio Mariani 2015-09-13
If you're seeing this on multiple computers, then my suspicion is that you're doing something that's causing unsigned packages to be installed or something is going wrong when you register your keys, unrelated to a rEFInd or EFI bug. I say this because you're the only person to have reported this problem, and the probability of one person encountering an obscure bug that's dependent on the EFI version on multiple computers is extremely low.
I therefore recommend running a couple more tests:
EFI/refind, copy that entire directory tree toEFI/refind_test, but deleteshimx64.efiandMokManager.efijust to keep things uncluttered. You can then see if you can launch the second rEFInd from the first. If you can, then Shim (associated with the first rEFInd) is working correctly./etc/refind.d/keys. If you can see the kernels in the rEFInd menu, then that means that Shim is working correctly, since the signed driver is launched like any other binary. If you can see the kernels but get a Secure Boot error, then that indicates a problem with the specific key.Since you're a programmer, you can play with the code yourself, if you like. If there's a problem registering the Secure Boot support, it will likely be in the
mok/security_policy.cfile, in particular in thesecurity_policy_install()function. (This function is in turn called fromrefind/main.c'sSecureBootSetup()function.) You can addPrint()statements (similar to a standard Cprint()function; see examples throughout the code) to display messages. You'll probably need aPauseForKey()somewhere to keep the messages on screen long enough to read them. Any debugging version I'd send you would do just that at strategic points to let me know what's going on as the program runs.Dear Rockerick, ok! I run some tests and finally got it to work! I completely agree, I don't think the problem is refind, but as you say, it is the interaction with shim that seems to fail.
First of all, I run the tests you described. I created a second copy of refind, and the first one cannot boot it. I then had refind detect all the kernels for ubuntu and fedora, and refind cannot boot them. All give verification error.
At this point I concluded that the problem was shim. I thought that the problem was the version of shim that came with ubuntu, so I booted fedora and tried to install refind from the fedora side, using the rpm package you provide. This was all useless. Everything gave verification error.
I finally tried the version of shim that you link on your website (0.2). This now worked! I could add the refind key and then the fedora and canonical. This gave no error and I could boot everything, including MokManager.
So I am also now convinced that the problem is the interaction of refind with shim. I checked and both ubuntu 15.04 and fedora 22 ship with shim 0.7. Do you know of any problem in the interaction with this version of shim?
Even if I got it to work, I would still like to help debugging this. I am available for tests and debugging, if you are interested. In the next few weeks I don't have time to look at the code, but I will be happy to run any debugging version and to report the logs.
Thanks for continuing with this -- I've managed to reproduce the problem using Shim 0.8 from the Ubuntu 15.10 pre-release. I'll look into it this week, and with any luck find a fix soon. In the meantime, sticking with a pre-0.7 Shim should be a suitable workaround.
Thanks Roderick. Please feel free to send me binaries to test if you need
I spent a bit of time this morning looking into this. I was unable to reproduce the problem with Shim 0.7 from Ubuntu 15.04 (Shim 0.8 is available as an update to 15.04), but Shim 0.8 from the pre-release Ubuntu 15.10 does show the problem. Could you please verify that you've had the problem with Shim 0.7? Here's a URL to a tarball with Shim 0.7 from a non-updated Ubuntu 15.04:
http://www.rodsbooks.com/shim-u1504.tgz
FWIW, I've discovered that the problem is that the first call to Shim 0.8 from rEFInd succeeds, but subsequent calls always return a failure (as if the binary were not signed by a MOK). Note that if you've got a filesystem driver installed, that first call will be to validate the driver, so it will be impossible to launch anything else after that. If you had no driver installed, you should at least be able to launch GRUB -- but I don't yet know if its calls would succeed or fail.
Hi Rodrick. I too have encountered the exact same symptoms as Valerio Mariani has in this thread. I've been pulling my hair out all day fighting with this, even going to the lenghts of a full wipe-and-reinstall of linux mint 17.2! I was worried that microsoft actually pulled the trigger and perma-revoked all my enrolled keys during my last windows update (win8.1) in some sort of misguided push-everyone-to-win10-only move.
I'm glad to find you were already working on this problem. For the record, my machine is an HP model 2000 laptop, with an EFI v2.31 by INSYDE corporation....if that makes any difference or helps at all. I do know its a late 2012 model.
Anyway, will test your provided shim files. Question, though -- should I put them in /EFI/refind/ or /EFI/ubuntu/ or both?
-edit-
Your 0.7 shim files worked nicely, I'm back up and running. Thank you Rodrick
Last edit: Joel Brown 2015-09-16
I've traced the problem to a change in Shim 0.8. It seems that, to work around a problem with a particular boot path involving the
fallback.efibinary and two Shims, the Shim developers now make Shim essentially unregister itself from the system when the first follow-on program is launched. Thus, rEFInd can authenticate one program and no more. The good news is that I've found a workaround. Here's a test version, in case you care to try it:http://www.rodsbooks.com/refind-bin-0.9.1.1.zip
The source code is in the rEFInd git archive.
This is a preliminary workaround, and it's a bit of an ugly and delicate hack. (It could break again if the Shim developers change their logic.) I've e-mailed the Shim developers about this, so I hope we'll be able to coordinate something better in the long term. Unfortunately, Shim binaries often go without updates for a long time, so rEFInd will have to cope with Shim 0.8 problems for months to come. :-(
This is obviously a high-priority issue, so unless I hear of problems with my workaround or discover a better way to do it, I'll probably push that (perhaps with one or two other minor updates) out as rEFInd 0.9.2 by the end of the week.
Dear Roderick, it works perfectly for me on my two laptops. I am curious, however: if shim unregisters itself, how can a bootloader, for example grub in a normal Ubuntu installation, authenticate the kernels? Also, did you receive an answer from the shim developers?
Thank you again for fixing this!
My workaround takes advantage of the way Shim is coded to trick it into not unregistering itself. (Slightly technical: Launching a program in EFI involves two system calls that Shim intercepts. The first call loads a binary into memory and the second one executes it. Shim unregisters itself on the second of these calls, but only if the program being launched was the last one to be loaded with the first call. My workaround causes rEFInd to load a copy of itself after it loads the program it intends to launch, so Shim's check doesn't match and Shim doesn't unregister itself. This is an ugly solution and it might fail if Shim's developers change the relevant logic, so we could be back at the same point again in the future.)
Yes, I'm in contact with a Shim developer, but there's no perfect solution as of yet. Even if we settle on a long-term solution, there'll still be Shim 0.8 binaries floating around for a long time, so I might need that ugly workaround in rEFInd, at least for a while....
Surely a less ugly solution would be to package refind with the 0.7 shim until a more elegant solution is arrived at?
Joel, I don't think that is possible, if I understand correctly how this works, shim needs to be signed with the Microsoft key, and Roderick does not have access to the signing program. I don't think Roderick can distribute someone else's binary . Plus shim 0.7 has the same problem.
But I wonder, what was the technical reason for this change, do we know? This is not needed to solve the bug, it is already solved. This is just for my curiosity, I like to learn things....
Shim is licensed with the BSD 2-clause license; see (for instance):
http://changelogs.ubuntu.com/changelogs/pool/main/s/shim/shim_0.8-0ubuntu2/copyright
Thus, redistributing the binary should be OK, AFAIK. The trouble is that if I redistribute Distribution A's Shim, then somebody using Distribution B or C will have to register both my (or their, for locally-signed) key and their distribution's key. If the user can use their own distribution's Shim, then they still need to register the rEFInd (or their local) key, but not their distribution's key. I could pay the $99 to get access to the Microsoft signing program, which would enable me to distribute a Shim with rEFInd's key built in, in which case users would have to register their distributions' own keys. This is no advantage to users compared to using a distribution-provided key; and since the signing program requires a lot of bureaucratic hoop-jumping, it's not seemed worth the effort. That's something I'm reconsidering, though.
In my own tests, Shim 0.7 does not have the problem, but I'd appreciate it if Valerio could check that using the Ubuntu Shim 0.7 to which I linked earlier:
http://www.rodsbooks.com/shim-u1504.tgz
Note that some distributions, including Ubuntu 15.04, shipped originally with one Shim but then updated to a later one. That tarball is named for Ubuntu 15.04, but it includes the original 0.7 version of Shim, not the latest version for that distribution.
As to the technical reason for the change, that's described in a comment in the Shim 0.8 source code:
https://github.com/rhinstaller/shim/blob/master/replacements.c
See the start_image() function. Basically, in a boot path that involves Shim -->
fallback.efi--> Shim --> GRUB, you can wind up with GRUB using a mixture of the first and second Shims' features, which is undesirable. This path, it should be noted, is how Red Hat has worked around an EFI bug that's really the bane of the Linux world: Some EFIs forget or ignore the EFI boot order, so Red Hat wrote thefallback.efiprogram to reside in the fallback position (EFI/BOOT/bootx64.efi) to restore the boot order variable and seamlessly continue the boot process. The end result is that we've got a cascading set of problems and workarounds to those problems (Secure boot yields Shim, EFI bugs yieldfallback.efi,fallback.efiyields a Shim "self-destruct" feature, and the Shim "self-destruct" feature yields a crippled rEFInd.). Problems caused by a preceding program require adjustments to follow-on programs because the preceding program can't always be controlled by the person who needs it fixed. At least with rEFInd I have the option of releasing my own Shim, but it will take time and effort for me to do this. There are other possible solutions within rEFInd that I'm investigating, but they've got problems of their own (a lot of extra programming effort, program bloat, or the possibility of crashes, for instance).So the bottom line is that the workaround in the 0.9.1.1 binary is likely to be a temporary band-aid solution at best. Moving forward, options are:
At the moment, it's just too early to say what will happen; I'm still mulling the possibilities, and it's possible that some new technical solution will appear.
Here I am. Sorry for the delay. So, shim-1504 that you posted works. What I mean is that this works with the regular refit in the PPA, not with the updated version you posted for testing
Thanks; that confirms what I thought.
FWIW, I'm releasing 0.9.2 with this workaround now, and will look into a better long-term fix as time permits.