#2446 32MB+ ISO-9660/El-Torito images, as defined by the UEFI specs, are not supported

[Pete Batard]

Hi Igor,

Because the El Torito specs only allow an image size up to 0xffff virtual 512-byte sectors (i.e 32MB - 512), the UEFI folks added a provision in their own specs for larger images. Per section 13.3.2.1 of the UEFI v2.10 specs, to allow for large ESP images:

If the value of Sector Count is set to 0 or 1, EFI will assume the system partition consumes the space from the beginning of the “no emulation” image to the end of the CD-ROM.

The current 7-zip El-Torito virtual image mapping (as [BOOT]\#-Boot-NoEmul.img) does not currently support such images, and will report them as having size 0 or 1 instead of expanding them to the end of the media, like the UEFI specs described.

It should also be noted that some software, like xorriso, abuses the UEFI specs by not always placing the El Torito image to be expanded as the last, but as an image that may be followed by another El-Torito image (meaning that, a system following the UEFI specs will actually expand the ESP image to include extra data, which is mostly fine but probably not what the UEFI folks intended). So, to contend with such software, 7-zip should probably only expand the UEFI ESP El Torito images to the end of the media, if there doesn't happen to exist another El Torito image starting at a higher LSN (in which case it should only expand it to that LSN).

I am attaching such a problematic xorriso generated image, with a 40 MB (blank) ESP. The [BOOT]\2-Boot-NoEmul.img should be listed by 7-zip as being exactly 40 MB, and allow extraction. But as you will see, 7-zip currently lists that image as being 0-sized and does not allow any operations on it.

[Pete Batard]

For completion, I'll point out that if you want to generate an ISO yourself using xorriso, you can follow these instructions.

[Igor Pavlov]

I don't remember iso format details.
So it's difficult for me to change something in that code.
What way we can detect that we must increase size of file to full image size?
Also is there a way to detect real size of that file?
For example, if we have big image (data stream) that contains many files (like tar), we want to get exact ranges of any iso file in that big stream.

[Pete Batard]

I don't remember iso format details.
So it's difficult for me to change something in that code.

Well, as far as I can see, this should be mostly around this section of the 7-zip code (sorry for using a GitHub mirror, but I couldn't find a repo here).

What way we can detect that we must increase size of file to full image size?

This is something I have recently added to libcdio, most specifically in this section (and while the code is GPLv3 licensed, since I am the author of these changes, I am happy to make the copyright for these specific changes public domain). But please bear in mind that I purposefully took a huge shortcut in the libcdio implementation to avoid implementing full El Torito parsing or processing of anything but NoEmul.

Basically, once you have access to the El Torito boot entries (which you already do) you can find the max LSN of the media from the Primary Volume Descriptor in the Volume Space Size field (which means working on streamed input shouldn't be an issue, since all the data you want is at the beginning). From there, for each Image, you can use a simple loop over all of the other El Torito images to find if their LSN is higher than the current one and lower than the max LSN (i.e. Volume Space Size). Then, if you find an image that matches the UEFI specs (which we don't bother to do in the patch I'm linking to, so that we can also handle software that might have already used 0 as a means to expand images), i.e. with platform id set to 0xef and num sectors set to 0 or 1, and there is more than 0xffff sectors from the LSN of that image to the max/next LSN you obtained above, you can conclude that this image should be expanded in size to that max/next LSN.

Note that the code I pointed to may of course break, if there happens to be additional non El Torito data between or after the images, but that will only be the case for non UEFI compliant images (since UEFI is clear that an extended image will go to the end of the volume), and expanding these is outside of the El Torito specs anyway.

For example, if we have big image (data stream) that contains many files (like tar), we want to get exact ranges of any iso file in that big stream.

Well, since the only new data you really need to compute the exact range is in the PVD of the ISO, which sits at the start, I don't think this should be an issue. Especially, there is no need to read the stream till the end to figure out the size.

[Igor Pavlov]

It's difficult to edit the code for me now. I don't keep ISO details in mind and I don't understand full structure now.
I have VolumeSpaceSize field in CVolumeDescriptor.
But I'm note sure what is based on. Is it Zero based?
And we can have many descriptors.
Do they have different VolumeSpaceSize values?

so I check for CBootInitialEntry be.BootMediaType == NBootMediaType::kNoEmulation && be.SectorCount <= 0

be.SectorCount == 1 can be normal case value, as I suppose.
I don't know what another conditions can be used in my code.
I don't see where platform id can be checked for 0xef in 7-Zip code.

[Pete Batard]

But I'm note sure what is based on. Is it Zero based?

If it's the same field as the one in PVD, it should be the size of the volume, in logical sectors (i.e. in blocks of 2048 bytes). Not sure what you mean by zero based, but no, an ISO volume occupying 10 blocks would not have 9 in that field, it would have 10. So this field can be used like an LSN/RBA to tell us the virtual LSN/RBA of a subsequent El Torito image, if there was any, when expanding per the UEFI specs.

And we can have many descriptors.

Well, I'm pretty sure that the UEFI specs imply that the PVD for the volume in which the El Torito catalog is found is the one that applied. You don't need to bother looking for other descriptors outside the PVD that precedes the El Torito data.

so I check for CBootInitialEntry be.BootMediaType == NBootMediaType::kNoEmulation && be.SectorCount <= 0

The last condition should be && be.SectorCount <= 1, since the UEFI specs also have 1 for sector size that need to be expanded. And yes, it's a bit baffling that they didn't only use 0, which would have made sense (they were defining the specs of UEFI so it's not like there was a precedent to follow), but they treat either a value of 0 or 1 in that field as meaning it should be expanded to the end of the (current) volume (yeah, technically they say end of the CD-ROM, but there's no way they really mean anything like this for multisession, so it's really end of the volume).

be.SectorCount == 1 can be normal case value, as I suppose.

Not if you want to comply with the specs, as explained above.

I don't see where platform id can be checked for 0xef in 7-Zip code.

Yeah, this is a bit of a bother because that field is not to be found in the El Torito section entry, along with the RBA/LSN and sector size, but in a preceding section header entry, which is annoying to parse and the reason why, in the code I pointed to, we don't bother checking for the platform ID at all, but instead determine whether there exist more than 0xffff available virtual sector between the RBA/LSN of the current image and a subsequent image (or the end of the volume), in which case, if we also have 0 or 1 for SectorCount, we assume that we're dealing with an image that requires expansion (which will always be the case if the platform ID was 0xef, and which also makes a lot of sense if SectorCount was set to 0, regardless of the platform ID).