Re: [Gptfdisk-general] GPTPart::GetDescription broken in 1.0.7 on Big Endian systems
Brought to you by:
srs5694
Menu
▾
▴
Re: [Gptfdisk-general] GPTPart::GetDescription broken in 1.0.7 on Big Endian systems
|
From: Erik L. <cat...@gm...> - 2021-06-08 11:33:07
|
Hi Christian, On 8.6.2021 13.47, Christian Ehrhardt wrote: > On Tue, Jun 8, 2021 at 12:20 PM Erik Larsson <cat...@gm...> wrote: >> On 8.6.2021 10.35, Christian Ehrhardt wrote: >>> On Tue, Jun 8, 2021 at 9:24 AM Erik Larsson <cat...@gm...> wrote: <snip> >>> I'd think the fastest way for you could be getting a free s390x VM to >>> work on via >>> https://developer.ibm.com/components/ibm-linuxone/gettingstarted/# >> >> Thanks for the pointer! That was very helpful, I was able to get an >> s390x VM up and running pretty quickly. >> Unfortunately they do not offer Ubuntu specifically but I fired up a >> RHEL8.3 VM to do the testing. > Worst case (if needed) a Ubuntu/Debian container on that RH8.3 should do it. Good point, I could try that. >>>> and find out what's going on. I have no access to an s390x machine but >>>> maybe qemu can help with that... >>> It can, but you'd want to use a rather new qemu and an older >>> Debian/Ubuntu system as too recent instructions always are a problem. >>> As I said the community cloud likely is your quickest way out of this. >> >> Indeed it was the quickest way. I have now compiled gptfdisk from source >> on the s390x instance and compared it against the bundled 'gdisk' on >> RHEL8.3. >> I used a GPT sample image that I created with the 'gpt' utility in >> macOS, just to make sure it's untouched by the gdisk utility on any >> platform. >> It has 8 partitions with the phrase "File system" in various different >> languages as GPT partition labels (for testing Unicode chars). >> >> I uploaded the image here: >> https://catacombae.org/temp/gpttest.img.bz2 >> >> I am however getting the same results as I got on my PowerPC setup. >> >> With the RHEL8.3-provided gdisk 1.0.3: >> -------- >> $ uname -a >> Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17 >> 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux >> $ sudo gdisk /dev/loop1 >> GPT fdisk (gdisk) version 1.0.3 >> >> Partition table scan: >> MBR: protective >> BSD: not present >> APM: not present >> GPT: present >> >> Found valid GPT with protective MBR; using GPT. >> >> Command (? for help): p >> Disk /dev/loop1: 262144 sectors, 128.0 MiB >> Sector size (logical/physical): 512/512 bytes >> Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7 >> Partition table holds up to 128 entries >> Main partition table begins at sector 2 and ends at sector 33 >> First usable sector is 34, last usable sector is 262110 >> Partitions will be aligned on 2-sector boundaries >> Total free space is 98237 sectors (48.0 MiB) >> >> Number Start (sector) End (sector) Size Code Name >> 1 34 20513 10.0 MiB 0700 >> 䘀椀氀攀 猀礀猀琀攀洀 >> 2 20514 40993 10.0 MiB 0700 >> ꌃ촃쌃쐃뜃밃넃 넃섃윃딃꼃줃봃 >> 3 40994 61473 10.0 MiB 0700 蝥ﭼ >> 4 61474 81953 10.0 MiB 0700 >> ␄〄㤄㬄㸄㈄〄伄 䄄㠄䄄䈄㔄㰄〄 >> 5 81954 102433 10.0 MiB 0700 䘆㠆✆䔆 ✆䐆䔆䐆䄆✆⨆ >> 6 102434 122913 10.0 MiB 0700 ⬉㸉܉㈉ 㠉㼉㠉䴉Ἁ⸉ >> 7 122914 143393 10.0 MiB 0700 >> 8 143394 163873 10.0 MiB 0700 >> 锋ꌋ뼋꼋锋촋锋쬋ꨋ촋ꨋ섋 긋섋넋젋긋젋 >> >> Command (? for help): q >> -------- >> >> It's all garbage. However with the patched and freshly compiled gdisk >> 1.0.7 straight from git: >> >> -------- >> $ uname -a >> Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17 >> 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux >> $ sudo ./gdisk /dev/loop1 >> GPT fdisk (gdisk) version 1.0.7 >> >> Partition table scan: >> MBR: protective >> BSD: not present >> APM: not present >> GPT: present >> >> Found valid GPT with protective MBR; using GPT. >> >> Command (? for help): p >> Disk /dev/loop1: 262144 sectors, 128.0 MiB >> Sector size (logical/physical): 512/512 bytes >> Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7 >> Partition table holds up to 128 entries >> Main partition table begins at sector 2 and ends at sector 33 >> First usable sector is 34, last usable sector is 262110 >> Partitions will be aligned on 2-sector boundaries >> Total free space is 98237 sectors (48.0 MiB) >> >> Number Start (sector) End (sector) Size Code Name >> 1 34 20513 10.0 MiB 0700 File system >> 2 20514 40993 10.0 MiB 0700 Σύστημα αρχείων >> 3 40994 61473 10.0 MiB 0700 文件系统 >> 4 61474 81953 10.0 MiB 0700 Файловая система >> 5 81954 102433 10.0 MiB 0700 نظام الملفات >> 6 102434 122913 10.0 MiB 0700 फाइल सिस्टम >> 7 122914 143393 10.0 MiB 0700 טעקע סיסטעם >> 8 143394 163873 10.0 MiB 0700 கணியக்கோப்பு முறைமை >> >> Command (? for help): q >> -------- >> >> With the version compiled from git including my fix it looks fine. All >> characters are decoded properly. >> >> Would you be able to test this image on your Ubuntu s390x instance and >> report back the results? > Sure, I can do that > >> Or provide an image with a counterexample? > And sure again, first a log, then cross tests and eventually the image files > > #1 1.0.6 > > $ dd if=/dev/zero of=/tmp/test1.0.6 bs=1024 count=65536 > 65536+0 records in > 65536+0 records out > 67108864 bytes (67 MB, 64 MiB) copied, 0.154128 s, 435 MB/s > $ gdisk-1.0.6/sgdisk /tmp/test1.6 -o > Warning: Partition table header claims that the size of partition table > entries is 0 bytes, but this program supports only 128-byte entries. > Adjusting accordingly, but partition table may be garbage. > Warning: Partition table header claims that the size of partition table > entries is 0 bytes, but this program supports only 128-byte entries. > Adjusting accordingly, but partition table may be garbage. > Creating new GPT entries in memory. > Warning: The kernel is still using the old partition table. > The new table will be used at the next reboot or after you > run partprobe(8) or kpartx(8) > The operation has completed successfully. > $ gdisk-1.0.6/sgdisk /tmp/test1.0.6 -n 1 -c '1:Linux filesystem' -t 1:8300 > Setting name! > partNum is 0 > Warning: The kernel is still using the old partition table. > The new table will be used at the next reboot or after you > run partprobe(8) or kpartx(8) > The operation has completed successfully. > $ gdisk-1.0.6/gdisk -l /tmp/test1.0.6 > GPT fdisk (gdisk) version 1.0.6 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > Disk /tmp/test1.0.6: 131072 sectors, 64.0 MiB > Sector size (logical): 512 bytes > Disk identifier (GUID): 970AC708-C718-4CDF-8EF4-1DCE94ADF8FB > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 131038 > Partitions will be aligned on 2048-sector boundaries > Total free space is 2014 sectors (1007.0 KiB) > > Number Start (sector) End (sector) Size Code Name > 1 2048 131038 63.0 MiB 8300 Linux filesystem > > > #1 1.0.7 > > dd if=/dev/zero of=/tmp/test1.0.7 bs=1024 count=65536 > 65536+0 records in > 65536+0 records out > 67108864 bytes (67 MB, 64 MiB) copied, 0.190568 s, 352 MB/s > gdisk-1.0.7/sgdisk /tmp/test1.0.7 -o > Creating new GPT entries in memory. > Warning: The kernel is still using the old partition table. > The new table will be used at the next reboot or after you > run partprobe(8) or kpartx(8) > The operation has completed successfully. > gdisk-1.0.7/sgdisk /tmp/test1.0.7 -n 1 -c '1:Linux filesystem' -t 1:8300 > Setting name! > partNum is 0 > Warning: The kernel is still using the old partition table. > The new table will be used at the next reboot or after you > run partprobe(8) or kpartx(8) > The operation has completed successfully. > gdisk-1.0.7/gdisk -l /tmp/test1.0.7 > GPT fdisk (gdisk) version 1.0.7 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > Disk /tmp/test1.0.7: 131072 sectors, 64.0 MiB > Sector size (logical): 512 bytes > Disk identifier (GUID): 14750985-A59C-451B-B0D1-A49614B866E2 > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 131038 > Partitions will be aligned on 2048-sector boundaries > Total free space is 2014 sectors (1007.0 KiB) > > Number Start (sector) End (sector) Size Code Name > 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 > > > #3 cross tests > > #3.1 1.0.6 reading the 1.0.7 image > > $ gdisk-1.0.6/gdisk -l /tmp/test1.0.7 > GPT fdisk (gdisk) version 1.0.6 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > Disk /tmp/test1.0.7: 131072 sectors, 64.0 MiB > Sector size (logical): 512 bytes > Disk identifier (GUID): 14750985-A59C-451B-B0D1-A49614B866E2 > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 131038 > Partitions will be aligned on 2048-sector boundaries > Total free space is 2014 sectors (1007.0 KiB) > > Number Start (sector) End (sector) Size Code Name > 1 2048 131038 63.0 MiB 8300 Linux filesystem > > #3.2 1.0.7 reading the 1.0.6 image > > $ gdisk-1.0.7/gdisk -l /tmp/test1.0.6 > GPT fdisk (gdisk) version 1.0.7 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > Disk /tmp/test1.0.6: 131072 sectors, 64.0 MiB > Sector size (logical): 512 bytes > Disk identifier (GUID): 970AC708-C718-4CDF-8EF4-1DCE94ADF8FB > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 131038 > Partitions will be aligned on 2048-sector boundaries > Total free space is 2014 sectors (1007.0 KiB) > > Number Start (sector) End (sector) Size Code Name > 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 > > > These are the two image files that got created by the commands above: > https://people.canonical.com/~paelzer/test1.0.6.tar.xz > https://people.canonical.com/~paelzer/test1.0.7.tar.xz > > I hope that helps and let me know if you need anything else. Thanks for those images. Now it's much clearer what is going on. Both your images have the on-disk volume label UTF-16 units in big endian, not little endian which is mandated by the EFI specification! Just look at the first 1 KiB + 128 byte (first GPT entry) of the volume how the characters in the string "Linux filesystem" are in big-endian (least significant byte first): -------- $ dd if=tmp/test1.0.6 bs=$[1024+128] count=1 | xxd -a 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ * 000001c0: 0200 ee28 2008 0100 0000 ffff 0100 0000 ...( ........... 000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U. 00000200: 4546 4920 5041 5254 0000 0100 5c00 0000 EFI PART....\... 00000210: 8b05 d554 0000 0000 0100 0000 0000 0000 ...T............ 00000220: ffff 0100 0000 0000 2200 0000 0000 0000 ........"....... 00000230: deff 0100 0000 0000 08c7 0a97 18c7 df4c ...............L 00000240: 8ef4 1dce 94ad f8fb 0200 0000 0000 0000 ................ 00000250: 8000 0000 8000 0000 880a f472 0000 0000 ...........r.... 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................ * 00000400: af3d c60f 8384 7247 8e79 3d69 d847 7de4 .=....rG.y=i.G}. 00000410: 4d33 25ac 07d6 794c 9bcf 50c5 4e86 fc72 M3%...yL..P.N..r 00000420: 0008 0000 0000 0000 deff 0100 0000 0000 ................ 00000430: 0000 0000 0000 0000 004c 0069 006e 0075 .........L.i.n.u 00000440: 0078 0020 0066 0069 006c 0065 0073 0079 .x. .f.i.l.e.s.y 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ 00000460: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000470: 0000 0000 0000 0000 0000 0000 0000 0000 ................ -------- Compare this to the image I provided where the label characters ("File system") are in little endian (most significant byte first): -------- $ dd if=gpttest.img bs=$[1024+128] count=1 | xxd -a 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ * 000001b0: 0000 0000 0000 0000 0000 0000 0000 00ff ................ 000001c0: ffff eeff ffff 0100 0000 ffff 0300 0000 ................ 000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U. 00000200: 4546 4920 5041 5254 0000 0100 5c00 0000 EFI PART....\... 00000210: 7b79 7b6e 0000 0000 0100 0000 0000 0000 {y{n............ 00000220: ffff 0300 0000 0000 2200 0000 0000 0000 ........"....... 00000230: deff 0300 0000 0000 2f50 fea4 64c2 514d ......../P..d.QM 00000240: 920b 3849 bb69 6ba7 0200 0000 0000 0000 ..8I.ik......... 00000250: 8000 0000 8000 0000 6435 b972 0000 0000 ........d5.r.... 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................ * 00000400: a2a0 d0eb e5b9 3344 87c0 68b6 b726 99c7 ......3D..h..&.. 00000410: ceee 9eb1 b179 344b 89dc 7c7c 1b85 379e .....y4K..||..7. 00000420: 2200 0000 0000 0000 2150 0000 0000 0000 ".......!P...... 00000430: 0000 0000 0000 0000 4600 6900 6c00 6500 ........F.i.l.e. 00000440: 2000 7300 7900 7300 7400 6500 6d00 0000 .s.y.s.t.e.m... 00000450: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000460: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000470: 0000 0000 0000 0000 0000 0000 0000 0000 ................ -------- Another indicator that there's something wrong is that GNU parted doesn't print the label: -------- $ losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 1 /home/linux1/gptfdisk/gpttest.img 0 512 /dev/loop2 0 0 0 1 /home/linux1/gptfdisk/tmp/test1.0.6 0 512 /dev/loop0 0 0 0 0 /var/opt/zthin/cfgdrive.iso 0 512 /dev/loop3 0 0 0 1 /home/linux1/gptfdisk/tmp/test1.0.7 0 512 $ sudo parted /dev/loop2 GNU Parted 3.2 Using /dev/loop2 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: Loopback device (loopback) Disk /dev/loop2: 67.1MB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Warning: failed to translate partition name Number Start End Size File system Name Flags 1 1049kB 67.1MB 66.0MB (parted) q $ sudo parted /dev/loop1 GNU Parted 3.2 Using /dev/loop1 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: Loopback device (loopback) Disk /dev/loop1: 134MB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Warning: failed to translate partition name Warning: failed to translate partition name Warning: failed to translate partition name Warning: failed to translate partition name Warning: failed to translate partition name Warning: failed to translate partition name Warning: failed to translate partition name Number Start End Size File system Name Flags 1 17.4kB 10.5MB 10.5MB File system msftdata 2 10.5MB 21.0MB 10.5MB msftdata 3 21.0MB 31.5MB 10.5MB msftdata 4 31.5MB 42.0MB 10.5MB msftdata 5 42.0MB 52.4MB 10.5MB msftdata 6 52.4MB 62.9MB 10.5MB msftdata 7 62.9MB 73.4MB 10.5MB msftdata 8 73.4MB 83.9MB 10.5MB msftdata (parted) q -------- (Clearly GNU parted doesn't like non-ASCII labels, but you can at least see that the first label "File system" is printed.) If sgdisk created these partition layouts, then there's clearly a bug in sgdisk which makes it write labels characters in big-endian format without byteswapping them. Best regards, - Erik >>>> On 8.6.2021 10.12, Christian Ehrhardt wrote: >>>>> Hi, >>>>> I wanted to let you know that the recent try to fix big endian byte >>>>> swapping is actually breaking big endian in 1.0.7. >>>>> >>>>> This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. >>>>> >>>>> When you look at just these logs it seems it is just failing for no >>>>> apparent reason, >>>>> but with -x enabled in gdisk_test.sh one quickly sees there is some >>>>> string-mangling happening: >>>>> >>>>> Number Start (sector) End (sector) Size Code Name >>>>> 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 >>>>> >>>>> Reading the very same file with the old gdisk is good - so likely the >>>>> on-dis content is good as well: >>>>> Number Start (sector) End (sector) Size Code Name >>>>> 1 2048 131038 63.0 MiB 8300 Linux filesystem >>>>> >>>>> This is caused by GPTPart::GetDescription due to this change [3] in 1.0.7 >>>>> >>>>> Reverting this fixes change fixes the problem. >>>>> >>>>> [1]: https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 >>>>> [2]: https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz >>>>> [3]: https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ >>>>> >>> > > |
