hdbios and dont_scan_volumes

Manu
2014-03-12
2014-04-14
  • Manu
    Manu
    2014-03-12

    Hi,

    I'm using refind 0.7.8 on a MacBookPro (Late 2013) with triple-boot:
    - OS X 10.9.2
    - Windows 7 Pro SP1 x64
    - Clonezilla live 2.2.1-25-amd64

    Which works perfectly, but:

    Now I got another FAT32 partition for data exchange purpose.
    And refind keeps creating a new icon, for that partition.

    Unfortunately Windows uses Bootcamp's default mbr-hybrid partiton table and bios-booting because i didn't get it running directly over EFI. So i can't drop hdbios-scanner...

    Is there any possibility to exclude this partition from hdbios-scanner?

    What I tried:
    - Formatting the partition as NTFS, exFAT (idea was: refind can't read this partitions, FAIL: It recognizes the volume-LABEL)
    - dont_scan_volumes directive (FAIL: -> It only affects the internal/EFI-scanner)

    Thanks for your help,
    Manu

     
  • Could you send me the first 512 bytes of the partition in question? (You can get it with something like sudo dd if=/dev/disk0s5 of=foo.img bs=512 count=1 in OS X, changing /dev/disk0s5 to whatever the appropriate partition identifier is.) As it is, rEFInd explicitly excludes certain known non-bootable FAT signatures, but you may well have some other signature that rEFInd doesn't (yet) recognize as non-bootable.

     
  • Manu
    Manu
    2014-03-14

    Of course, here you are.
    Btw the partition table is created by a program called: iPartition (http://www.coriolis-systems.com/iPartition.php), because it's the only way i found to get a tripple boot on a MacBook Pro (Late 2013) running. It has a magical way to get the partition table right and don't mess up bootcamp.

    I'd say I'm a advanced Linux user but after 2 weeks of trying parted, gdisk, fdisk, gfdisk, not to mess up Bootcamp ... I had to give up and buy this "nice" piece of software.

     
    Attachments
  • Joe van Tunen
    Joe van Tunen
    2014-03-14

    Looks like a German FAT32 Windows BOOTMGR style boot code block.

    You want a boot block with a lot more zeros and a "Non-system disk" type message. Formatting the partition with Disk Utility should do that.

    I think rEFInd needs some Add/Modify/Remove type stanzas that also work with Legacy OS partitions. The verbs will work on items that match a set of specified identifiers (volume name, GPT guid, file system UUID, partition size, partition offset, partition MBR index, partition GPT index, MBR disk identifier, etc). "Add" will add an item that was removed by some other filter. "Modify" will change some information like Display Name (for cases where rEFInd can't get a useful name or the name is a duplicate). "Remove" will hide the item (useful for this case where it's known that the partition is not bootable or the user doesn't want it included for other reasons.

     
  • Could you please try the following test binary:

    http://www.rodsbooks.com/refind_x64_0.7.8.1.efi

    That should do the trick. If not, I'll try again.

    I agree that rEFInd needs ways to better modify the inclusion or exclusion of BIOS-bootable partitions. The trouble is that the existing BIOS-mode code for PCs and Macs is entirely separate, and I want to improve the PC side and merge the two together, if at all possible. The form that these future developments will take will influence the type of solution that may be practical, or even possible, for fine-tuning control options for BIOS-mode boot selection. (A clue, though: You will not get all the options you suggest, Joe.) Thus, I don't want to create a solution that will become unusable in a few releases, necessitating changes to peoples' configuration files.

    A further issue is that, quite honestly, BIOS-mode booting is troublesome to begin with, and I'd rather not have to deal with it as a user or as a software developer. (Yes, I realize that this contradicts what I just wrote about wanting to improve the PC side of the BIOS-mode booting equation. They're equally valid, albeit contradictory, reactions to the same thing.) Unfortunately, Apple and/or Microsoft are keeping mixed-mode booting alive on Macs, and there's a lesser need for it on PCs, but the world would be a better place without it.

     
    • Manu
      Manu
      2014-03-17

      Sorry I left the Laptop at work so i couldn't test the efi-file on the weekend...

      So i changed the /EFI/refind/refind_x64.efi with the one you sent and unfortunately -> nothing changed.

      Is there anything else to do, such as the equivalent of writing mbr?
      Yeah, I'm new to EFI :-)

      The Mac's built in Startup Manager (hold option at startup) doesn't recognize the partition as bootable.
      I only got 3 options here:
      - EFI Boot (refind)
      - Windows (windows partition with bcd bootloader)
      - EFI Boot (Linux grub-EFI)

       
      • Joe van Tunen
        Joe van Tunen
        2014-03-17

        Looks like the test for the "Neustart: Taste drücken" message is missing the letter "c".

        The Mac's built in Startup Manager only displays one Windows option per hard disk for any hard disk that has boot code in the MBR.

        The Windows option starts the Mac BIOS, which runs the boot code in the MBR which then runs the boot code of the partition that is selected as the active partition in the MBR.

        rEFInd does the same thing, but will set the active partition in the MBR to the selected partition before starting the Mac BIOS.

        To make rEFInd hide the partition, you need to remove the boot code from the partition. This is probably most easily done by erasing the partition with Disk Utility.app but you need to backup the files first.

        Another way to do it without erasing the partition is to change the name of the BOOTMGR string in the boot code so that rEFInd can't find it.

        1)
        The following should show "BOOTMGR", if not then don't do the following steps. Make sure you set the disk and partition number correctly.

        sudo dd if=/dev/disk0s5 bs=1 count=1 skip=0x169
        

        2)
        The following overwrites the letter "B" in "BOOTMGR" (make sure you set the disk and partition number correctly otherwise you may damage a partition):

        sudo echo -n "_" | dd of=/dev/disk0s5 bs=1 oseek=0x169 conv=notrunc
        

        3)
        Redo step 1 to make sure the first letter of "BOOTMGR" has been replaced with "_".

         
        Last edit: Joe van Tunen 2014-03-17
  • Joe van Tunen
    Joe van Tunen
    2014-03-15

    What was the problem in the boot block and what does the test binary do to solve it? It seems like a perfectly valid boot block and adding code to exclude it would be wrong unless the code is now searching for the existence of the BOOTMGR file that is referenced by the boot block but that wouldn't work for an NTFS boot block without an NTFS file system driver.

    I don't know how the PC side does BIOS mode booting from EFI. On the Mac side there isn't much of a choice; you set the BootCampHD NVRAM variable to point to a hard disk, and then set the active partition on that hard disk to point to the partition you want to boot, then call a Boot Camp EFI program which does the rest of the work. I don't imagine the PC side would be any different except for the details (what EFI program to run and how to tell the EFI program which hard drive to boot).

     
  • First, understand two points:

    • rEFInd inherits its Mac BIOS-mode boot code from rEFIt. Neither boot manager boots in BIOS mode exactly the way Boot Camp does. In particular, both rEFIt and rEFInd can launch the BIOS-mode boot loader in any partition, regardless of NVRAM entries. This fact means that rEFIt/rEFInd must decide which partition(s) contain a BIOS-mode boot loader.
    • It's not always easy to tell whether a partition holds a BIOS-mode boot loader. If a partition's first sector is entirely empty (holds nothing but "0" values), then it has no boot loader; but almost anything else could be a boot loader. In particular, most tools that create FAT filesystems write out a dummy boot program that does nothing but display a message to the effect that the disk is not bootable. Examples include "Non-system disk" and "This is not a bootable disk."

    In order to deal with these facts, rEFInd, like rEFIt before it, scans the boot sector of every partition. It searches for strings that identify both known BIOS-mode boot loaders (so as to identify what OS they represent) and known non-bootable "dummy" code. Anything that the rEFIt/rEFInd code does not recognize slips through and is assigned a default status, which is bootable in the case of both rEFIt and rEFInd. The boot sector that Manu presented did just this -- it lacked any known signature and so was classified as an unknown OS. The new test binary simply adds part of the German message about the disk being unbootable as another condition for excluding the partition from the boot list. (All of this applies only to Mac-style BIOS-mode booting, BTW. On PCs, rEFInd relies exclusively on NVRAM settings rather than scan partitions for bootability.) This is done in the ScanVolumeBootcode() function in refind/lib.c, if you care to look it over.

    I admit that this isn't a 100% reliable way to exclude partitions from the boot list; as this case illustrates, it's entirely possible that some new or obscure program will include a boot message that's not in the list and therefore slip through. Given the limitations of BIOS-mode boot loaders, though, there's no way to make a 100% reliable determination of bootability, at least not from the boot sector alone. You either assume bootability and accept that some non-bootable partitions will slip through (which is what rEFIt and rEFInd do) or you assume non-bootability and accept that some unknown boot loaders will not be detected. You can call that "wrong," but I call it "making a choice in a no-win scenario." It's one of the reasons that I'm hoping BIOS-mode booting will die out more rather than less quickly.

     
  • Joe van Tunen
    Joe van Tunen
    2014-03-17

    Are you saying rEFInd doesn't write to NVRAM on Mac when booting Legacy partitions? How does the Mac BIOS know which disk to boot then (My Mac Pro 2008 can have up to 4 legacy bootable disks)? It looks like rEFInd is missing the WriteBootDiskHint function from the rEFIt source code. This might explain why rEFInd boots my Windows 7 disk when I select my Windows 8 disk... Note that WriteBootDiskHint was added after the last rEFIt binary (0.14) was created.
    https://sourceforge.net/p/refit/code/HEAD/tree/trunk/refit/refit/main.c

    I wonder if the Mac's built in Startup Manager (hold option at startup) writes to BootCampHD when you select a Windows option. If it does, then it would confirm that it is a necessary step.


    I believe those 3 German error messages are the equivalent of the 3 or 4 error messages that you see in a valid English BOOTMGR boot block. They don't say the disk is unbootable. The Google Translate website does a pretty good job (I guess) of translating them once you figure out the character set is something like IBM437 or IBM850
    http://en.wikipedia.org/wiki/Western_Latin_character_sets_(computing)

    You should probably assume that any boot block containing "NTLDR" or "BOOTMGR" is a valid boot block. If you have a filesystem driver for the disk, then you can search the root directory for those files. This is no more than what you do when searching for EFI files.

    German messages in Manu's boot block:

    Datenträger entfernen = Remove media 
    Medienfehler = Media error
    Neustart: Taste drücken = Restart: Press key
    

    English messages in some boot blocks that I have:

    NTFS_Windows_8_BOOTMGR

    0000180: b40e bb07 00cd 10eb f2c3 0d0a 4120 6469  ............A di
    0000190: 736b 2072 6561 6420 6572 726f 7220 6f63  sk read error oc
    00001a0: 6375 7272 6564 000d 0a42 4f4f 544d 4752  curred...BOOTMGR
    00001b0: 2069 7320 636f 6d70 7265 7373 6564 000d   is compressed..
    00001c0: 0a50 7265 7373 2043 7472 6c2b 416c 742b  .Press Ctrl+Alt+
    00001d0: 4465 6c20 746f 2072 6573 7461 7274 0d0a  Del to restart..
    

    FAT12_BSD_4.4_BOOTMGR

    00001a0: b042 4f4f 544d 4752 2020 2020 0d0a 424f  .BOOTMGR    ..BO
    00001b0: 4f54 4d47 5220 6973 206d 6973 7369 6e67  OTMGR is missing
    00001c0: ff0d 0a44 6973 6b20 6572 726f 72ff 0d0a  ...Disk error...
    00001d0: 5072 6573 7320 616e 7920 6b65 7920 746f  Press any key to
    00001e0: 2072 6573 7461 7274 0d0a 0000 0000 0000   restart........
    

    NTFS_Windows_Vista_BOOTMGR

    0000180: 7409 b40e bb07 00cd 10eb f2c3 0d0a 4120  t.............A 
    0000190: 6469 736b 2072 6561 6420 6572 726f 7220  disk read error 
    00001a0: 6f63 6375 7272 6564 000d 0a42 4f4f 544d  occurred...BOOTM
    00001b0: 4752 2069 7320 6d69 7373 696e 6700 0d0a  GR is missing...
    00001c0: 424f 4f54 4d47 5220 6973 2063 6f6d 7072  BOOTMGR is compr
    00001d0: 6573 7365 6400 0d0a 5072 6573 7320 4374  essed...Press Ct
    00001e0: 726c 2b41 6c74 2b44 656c 2074 6f20 7265  rl+Alt+Del to re
    00001f0: 7374 6172 740d 0a00 8ca9 bed6 0000 55aa  start.........U.
    

    I've updated all my Windows disks to start BOOTMGR which can then chain load NTLDR for Windows XP partitions. Note that older versions of BOOTMGR can't boot Windows 8 (winload.exe). In that case, I have my Windows 7 BCD include a NeoGrub entry (from EasyBCD) that can chain load BOOTMGR on my Windows 8 disk. Maybe it's possible for the Windows 7 BOOTMGR to chain load the Windows 8 BOOTMGR like it does for NTLDR, then I wouldn't have to go through NeoGrub. The Windows 8 BOOTMGR has the annoying feature of restarting the computer before chain loading another loader.

     
    Last edit: Joe van Tunen 2014-03-17
  • Manu, please try the following version:

    http://www.rodsbooks.com/refind_x64_0.7.8.3.efi

    Joe, no, rEFInd does not write to NVRAM when booting in BIOS mode. Scanning partitions for the NTLDR or BOOTMGR file would be unreliable because those files often occur on NTFS partitions, and rEFInd doesn't come with an NTFS driver.

    The bottom line is that, given the way BIOS-mode booting works, there's no way to be 100% accurate in identifying which partitions are actually bootable. I can patch cases that fail when I hear of them, but that's the best I can do. Adding complexity to support what is, essentially, a deprecated boot mode makes little sense to me.

     
    • Manu
      Manu
      2014-03-18

      refind 0.7.8.3 is working!

      After i ran the install.sh script with the new efi-file, it's working!

      Thanks a lot for your fast patch!

       
      Last edit: Manu 2014-03-18
  • Joe van Tunen
    Joe van Tunen
    2014-03-18

    Did you remove WriteBootDiskHint on purpose or did rEFInd get forked from an earlier rEFIt source? It would be really convenient if rEFInd booted the correct disk like the Mac's Startup Manager does.

    I believe searching for NTLDR or BOOTMGR would be 100% reliable when the partition has a file system driver. If there is no file system then falling back on the current method isn't less reliable than it already is.

    I've tried formatting a partition with iPartition and it did not create BOOTMGR boot code. The boot block contained the standard "Non-system disk" and "Press any key to reboot" messages.

    Maybe Manu used a different utility to format the partition? Why would a partition for data exchange be called "INSTALLER"? I might expect a partition that's called "INSTALLER" to have boot code. Was the partition a clone of a bootable installer and then later erased without removing the boot code?

    I think German users with bootable partitions might be surprised that their partitions are no longer bootable with rEFInd.

     
    • Manu
      Manu
      2014-03-18

      Yes, I used iPartion to create it.
      The partition is called INSTALLER, beacuse it contains installer-files for programs.

      This Mac is for training purposes, it contains a blank Windows 7 and a blank OSX Mavericks. And there is an exchange-partiotion called INSTALLER for the software needed in differnt trainings. The forth partition called RESTORE contains an Image and the Clonezilla Linux-live-system to restore the two OSes to it's blank state after the training or what so ever.

      I just looked at the boot block myself and it really contains a BOOTMGR statement, but don't ask me why... :-)
      The partition was created with iPartition with "Visible in Windows" option and afterwards the FAT32-Filesystem was recreated in Windows.

      After all the 0.7.8.3 patch is working so I'm fine, but if it affects other users i could life with your workaround, replaceing the "B" in BOOTMGR. At least I didn't test if this is working.

      Btw: The Windows is booting just fine so 0.7.8.3 does still regonize the BOOTMGR from the BOOTCAMP-partion.

       
      Last edit: Manu 2014-03-18
  • Joe van Tunen
    Joe van Tunen
    2014-03-18

    iPartition created the partition but Windows was responsible for formatting it and setting the boot code. The "Visible in Windows" option puts the partition in the MBR for you so that you don't need to sync the GPT and the MBR manually.

    On the Mac side, the newfs_msdos command line utility is responsible for formatting a FAT32 partition. It has the standard "Non-system disk" and "Press any key to reboot" messages. See bootcode in http://opensource.apple.com/source/msdosfs/msdosfs-198/newfs_msdos.tproj/newfs_msdos.c

    I can't figure out which Windows file contains the "Medienfehler" and other messages. I did a test in Windows 7 which produces the following messages for a new empty FAT32 partition:

    0000160: c300 0266 4049 7594 c342 4f4f 544d 4752  ...f@Iu..BOOTMGR
    0000170: 2020 2020 0000 0000 0000 0000 0000 0000      ............
    0000180: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    0000190: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    00001a0: 0000 0000 0000 0000 0000 0000 0d0a 5265  ..............Re
    00001b0: 6d6f 7665 2064 6973 6b73 206f 7220 6f74  move disks or ot
    00001c0: 6865 7220 6d65 6469 612e ff0d 0a44 6973  her media....Dis
    00001d0: 6b20 6572 726f 72ff 0d0a 5072 6573 7320  k error...Press 
    00001e0: 616e 7920 6b65 7920 746f 2072 6573 7461  any key to resta
    00001f0: 7274 0d0a 0000 0000 00ac cbd8 0000 55aa  rt............U.
    

    These English messages correspond to the German messages even more closely than my previous examples which were from NTFS. If the German boot block is to be excluded (which it shouldn't), then boot blocks of all other languages should be excluded as well. I can't test other languages without reinstalling Windows since I have Windows 7 Professional instead of Enterprise or Ultimate (but I think there are ways to get the other languages to work in Professional).

     
  • Joe, you've convinced me; I'm removing the code that I added.

    Manu, unfortunately this means that you'll need to create a fresh filesystem using a utility that does not create bogus boot code or mangle the boot block you've got now, at least once you update past the test version you're using now.