Re: [Gptfdisk-general] GPTPart::GetDescription broken in 1.0.7 on Big Endian systems

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi,

Just some minor corrections that I felt compelled to do because of an 
embarassing thought error. They're inlined below...

On 8.6.2021 14.32, Erik Larsson wrote:
> Hi Christian,
>
> On 8.6.2021 13.47, Christian Ehrhardt wrote:
>> On Tue, Jun 8, 2021 at 12:20 PM Erik Larsson <cat...@gm...> 
>> wrote:
>>> On 8.6.2021 10.35, Christian Ehrhardt wrote:
>>>> On Tue, Jun 8, 2021 at 9:24 AM Erik Larsson <cat...@gm...> 
>>>> wrote:
> <snip>
>>>> I'd think the fastest way for you could be getting a free s390x VM to
>>>> work on via
>>>> https://developer.ibm.com/components/ibm-linuxone/gettingstarted/#
>>>
>>> Thanks for the pointer! That was very helpful, I was able to get an
>>> s390x VM up and running pretty quickly.
>>> Unfortunately they do not offer Ubuntu specifically but I fired up a
>>> RHEL8.3 VM to do the testing.
>> Worst case (if needed) a Ubuntu/Debian container on that RH8.3 should 
>> do it.
>
> Good point, I could try that.
>
>>>>> and find out what's going on. I have no access to an s390x machine 
>>>>> but
>>>>> maybe qemu can help with that...
>>>> It can, but you'd want to use a rather new qemu and an older
>>>> Debian/Ubuntu system as too recent instructions always are a problem.
>>>> As I said the community cloud likely is your quickest way out of this.
>>>
>>> Indeed it was the quickest way. I have now compiled gptfdisk from 
>>> source
>>> on the s390x instance and compared it against the bundled 'gdisk' on
>>> RHEL8.3.
>>> I used a GPT sample image that I created with the 'gpt' utility in
>>> macOS, just to make sure it's untouched by the gdisk utility on any
>>> platform.
>>> It has 8 partitions with the phrase "File system" in various different
>>> languages as GPT partition labels (for testing Unicode chars).
>>>
>>> I uploaded the image here:
>>> https://catacombae.org/temp/gpttest.img.bz2
>>>
>>> I am however getting the same results as I got on my PowerPC setup.
>>>
>>> With the RHEL8.3-provided gdisk 1.0.3:
>>> --------
>>> $ uname -a
>>> Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17
>>> 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux
>>> $ sudo gdisk /dev/loop1
>>> GPT fdisk (gdisk) version 1.0.3
>>>
>>> Partition table scan:
>>>     MBR: protective
>>>     BSD: not present
>>>     APM: not present
>>>     GPT: present
>>>
>>> Found valid GPT with protective MBR; using GPT.
>>>
>>> Command (? for help): p
>>> Disk /dev/loop1: 262144 sectors, 128.0 MiB
>>> Sector size (logical/physical): 512/512 bytes
>>> Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7
>>> Partition table holds up to 128 entries
>>> Main partition table begins at sector 2 and ends at sector 33
>>> First usable sector is 34, last usable sector is 262110
>>> Partitions will be aligned on 2-sector boundaries
>>> Total free space is 98237 sectors (48.0 MiB)
>>>
>>> Number  Start (sector)    End (sector)  Size       Code  Name
>>>      1              34           20513   10.0 MiB    0700
>>> 䘀椀氀攀 猀礀猀琀攀洀
>>>      2           20514           40993   10.0 MiB    0700
>>> ꌃ촃쌃쐃뜃밃넃 넃섃윃딃꼃줃봃
>>>      3           40994           61473   10.0 MiB    0700  蝥ﭼ
>>>      4           61474           81953   10.0 MiB    0700
>>> ␄〄㤄㬄㸄㈄〄伄 䄄㠄䄄䈄㔄㰄〄
>>>      5           81954          102433   10.0 MiB    0700 
>>> 䘆㠆✆䔆 ✆䐆䔆䐆䄆✆⨆
>>>      6          102434          122913   10.0 MiB    0700 
>>> ⬉㸉܉㈉ 㠉㼉㠉䴉Ἁ⸉
>>>      7          122914          143393   10.0 MiB    0700
>>>      8          143394          163873   10.0 MiB    0700
>>> 锋ꌋ뼋꼋锋촋锋쬋ꨋ촋ꨋ섋 긋섋넋젋긋젋
>>>
>>> Command (? for help): q
>>> --------
>>>
>>> It's all garbage. However with the patched and freshly compiled gdisk
>>> 1.0.7 straight from git:
>>>
>>> --------
>>> $ uname -a
>>> Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17
>>> 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux
>>> $ sudo ./gdisk /dev/loop1
>>> GPT fdisk (gdisk) version 1.0.7
>>>
>>> Partition table scan:
>>>     MBR: protective
>>>     BSD: not present
>>>     APM: not present
>>>     GPT: present
>>>
>>> Found valid GPT with protective MBR; using GPT.
>>>
>>> Command (? for help): p
>>> Disk /dev/loop1: 262144 sectors, 128.0 MiB
>>> Sector size (logical/physical): 512/512 bytes
>>> Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7
>>> Partition table holds up to 128 entries
>>> Main partition table begins at sector 2 and ends at sector 33
>>> First usable sector is 34, last usable sector is 262110
>>> Partitions will be aligned on 2-sector boundaries
>>> Total free space is 98237 sectors (48.0 MiB)
>>>
>>> Number  Start (sector)    End (sector)  Size       Code  Name
>>>      1              34           20513   10.0 MiB    0700 File system
>>>      2           20514           40993   10.0 MiB    0700 Σύστημα 
>>> αρχείων
>>>      3           40994           61473   10.0 MiB    0700 文件系统
>>>      4           61474           81953   10.0 MiB    0700 Файловая 
>>> система
>>>      5           81954          102433   10.0 MiB    0700 نظام الملفات
>>>      6          102434          122913   10.0 MiB    0700 फाइल सिस्टम
>>>      7          122914          143393   10.0 MiB    0700 טעקע סיסטעם
>>>      8          143394          163873   10.0 MiB    0700 
>>> கணியக்கோப்பு முறைமை
>>>
>>> Command (? for help): q
>>> --------
>>>
>>> With the version compiled from git including my fix it looks fine. All
>>> characters are decoded properly.
>>>
>>> Would you be able to test this image on your Ubuntu s390x instance and
>>> report back the results?
>> Sure, I can do that
>>
>>> Or provide an image with a counterexample?
>> And sure again, first a log, then cross tests and eventually the 
>> image files
>>
>> #1 1.0.6
>>
>> $ dd if=/dev/zero of=/tmp/test1.0.6 bs=1024 count=65536
>> 65536+0 records in
>> 65536+0 records out
>> 67108864 bytes (67 MB, 64 MiB) copied, 0.154128 s, 435 MB/s
>> $ gdisk-1.0.6/sgdisk /tmp/test1.6 -o
>> Warning: Partition table header claims that the size of partition table
>> entries is 0 bytes, but this program  supports only 128-byte entries.
>> Adjusting accordingly, but partition table may be garbage.
>> Warning: Partition table header claims that the size of partition table
>> entries is 0 bytes, but this program  supports only 128-byte entries.
>> Adjusting accordingly, but partition table may be garbage.
>> Creating new GPT entries in memory.
>> Warning: The kernel is still using the old partition table.
>> The new table will be used at the next reboot or after you
>> run partprobe(8) or kpartx(8)
>> The operation has completed successfully.
>> $ gdisk-1.0.6/sgdisk /tmp/test1.0.6 -n 1 -c '1:Linux filesystem' -t 
>> 1:8300
>> Setting name!
>> partNum is 0
>> Warning: The kernel is still using the old partition table.
>> The new table will be used at the next reboot or after you
>> run partprobe(8) or kpartx(8)
>> The operation has completed successfully.
>> $ gdisk-1.0.6/gdisk -l /tmp/test1.0.6
>> GPT fdisk (gdisk) version 1.0.6
>>
>> Partition table scan:
>>    MBR: protective
>>    BSD: not present
>>    APM: not present
>>    GPT: present
>>
>> Found valid GPT with protective MBR; using GPT.
>> Disk /tmp/test1.0.6: 131072 sectors, 64.0 MiB
>> Sector size (logical): 512 bytes
>> Disk identifier (GUID): 970AC708-C718-4CDF-8EF4-1DCE94ADF8FB
>> Partition table holds up to 128 entries
>> Main partition table begins at sector 2 and ends at sector 33
>> First usable sector is 34, last usable sector is 131038
>> Partitions will be aligned on 2048-sector boundaries
>> Total free space is 2014 sectors (1007.0 KiB)
>>
>> Number  Start (sector)    End (sector)  Size       Code  Name
>>     1            2048          131038   63.0 MiB    8300  Linux 
>> filesystem
>>
>>
>> #1 1.0.7
>>
>> dd if=/dev/zero of=/tmp/test1.0.7 bs=1024 count=65536
>> 65536+0 records in
>> 65536+0 records out
>> 67108864 bytes (67 MB, 64 MiB) copied, 0.190568 s, 352 MB/s
>> gdisk-1.0.7/sgdisk /tmp/test1.0.7 -o
>> Creating new GPT entries in memory.
>> Warning: The kernel is still using the old partition table.
>> The new table will be used at the next reboot or after you
>> run partprobe(8) or kpartx(8)
>> The operation has completed successfully.
>> gdisk-1.0.7/sgdisk /tmp/test1.0.7 -n 1 -c '1:Linux filesystem' -t 1:8300
>> Setting name!
>> partNum is 0
>> Warning: The kernel is still using the old partition table.
>> The new table will be used at the next reboot or after you
>> run partprobe(8) or kpartx(8)
>> The operation has completed successfully.
>> gdisk-1.0.7/gdisk -l /tmp/test1.0.7
>> GPT fdisk (gdisk) version 1.0.7
>>
>> Partition table scan:
>>    MBR: protective
>>    BSD: not present
>>    APM: not present
>>    GPT: present
>>
>> Found valid GPT with protective MBR; using GPT.
>> Disk /tmp/test1.0.7: 131072 sectors, 64.0 MiB
>> Sector size (logical): 512 bytes
>> Disk identifier (GUID): 14750985-A59C-451B-B0D1-A49614B866E2
>> Partition table holds up to 128 entries
>> Main partition table begins at sector 2 and ends at sector 33
>> First usable sector is 34, last usable sector is 131038
>> Partitions will be aligned on 2048-sector boundaries
>> Total free space is 2014 sectors (1007.0 KiB)
>>
>> Number  Start (sector)    End (sector)  Size       Code  Name
>>     1            2048          131038   63.0 MiB    8300 
>> 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀
>>
>>
>> #3 cross tests
>>
>> #3.1 1.0.6 reading the 1.0.7 image
>>
>> $ gdisk-1.0.6/gdisk -l /tmp/test1.0.7
>> GPT fdisk (gdisk) version 1.0.6
>>
>> Partition table scan:
>>    MBR: protective
>>    BSD: not present
>>    APM: not present
>>    GPT: present
>>
>> Found valid GPT with protective MBR; using GPT.
>> Disk /tmp/test1.0.7: 131072 sectors, 64.0 MiB
>> Sector size (logical): 512 bytes
>> Disk identifier (GUID): 14750985-A59C-451B-B0D1-A49614B866E2
>> Partition table holds up to 128 entries
>> Main partition table begins at sector 2 and ends at sector 33
>> First usable sector is 34, last usable sector is 131038
>> Partitions will be aligned on 2048-sector boundaries
>> Total free space is 2014 sectors (1007.0 KiB)
>>
>> Number  Start (sector)    End (sector)  Size       Code  Name
>>     1            2048          131038   63.0 MiB    8300  Linux 
>> filesystem
>>
>> #3.2 1.0.7 reading the 1.0.6 image
>>
>> $ gdisk-1.0.7/gdisk -l /tmp/test1.0.6
>> GPT fdisk (gdisk) version 1.0.7
>>
>> Partition table scan:
>>    MBR: protective
>>    BSD: not present
>>    APM: not present
>>    GPT: present
>>
>> Found valid GPT with protective MBR; using GPT.
>> Disk /tmp/test1.0.6: 131072 sectors, 64.0 MiB
>> Sector size (logical): 512 bytes
>> Disk identifier (GUID): 970AC708-C718-4CDF-8EF4-1DCE94ADF8FB
>> Partition table holds up to 128 entries
>> Main partition table begins at sector 2 and ends at sector 33
>> First usable sector is 34, last usable sector is 131038
>> Partitions will be aligned on 2048-sector boundaries
>> Total free space is 2014 sectors (1007.0 KiB)
>>
>> Number  Start (sector)    End (sector)  Size       Code  Name
>>     1            2048          131038   63.0 MiB    8300 
>> 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀
>>
>>
>> These are the two image files that got created by the commands above:
>> https://people.canonical.com/~paelzer/test1.0.6.tar.xz
>> https://people.canonical.com/~paelzer/test1.0.7.tar.xz
>>
>> I hope that helps and let me know if you need anything else.
>
> Thanks for those images. Now it's much clearer what is going on.
> Both your images have the on-disk volume label UTF-16 units in big 
> endian, not little endian which is mandated by the EFI specification!
>
> Just look at the first 1 KiB + 128 byte (first GPT entry) of the 
> volume how the characters in the string "Linux filesystem" are in 
> big-endian (least significant byte first):

This was a thought error. I meant most significant byte, not least. :)

> --------
> $ dd if=tmp/test1.0.6 bs=$[1024+128] count=1 | xxd -a
> 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 000001c0: 0200 ee28 2008 0100 0000 ffff 0100 0000  ...( ...........
> 000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 000001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U.
> 00000200: 4546 4920 5041 5254 0000 0100 5c00 0000  EFI PART....\...
> 00000210: 8b05 d554 0000 0000 0100 0000 0000 0000 ...T............
> 00000220: ffff 0100 0000 0000 2200 0000 0000 0000 ........".......
> 00000230: deff 0100 0000 0000 08c7 0a97 18c7 df4c ...............L
> 00000240: 8ef4 1dce 94ad f8fb 0200 0000 0000 0000 ................
> 00000250: 8000 0000 8000 0000 880a f472 0000 0000 ...........r....
> 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 00000400: af3d c60f 8384 7247 8e79 3d69 d847 7de4 .=....rG.y=i.G}.
> 00000410: 4d33 25ac 07d6 794c 9bcf 50c5 4e86 fc72 M3%...yL..P.N..r
> 00000420: 0008 0000 0000 0000 deff 0100 0000 0000 ................
> 00000430: 0000 0000 0000 0000 004c 0069 006e 0075 .........L.i.n.u
> 00000440: 0078 0020 0066 0069 006c 0065 0073 0079  .x. .f.i.l.e.s.y
> 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........
> 00000460: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000470: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> --------
>
> Compare this to the image I provided where the label characters ("File 
> system") are in little endian (most significant byte first):

This was a thought error. I meant least significant byte, not most. :)

> --------
> $ dd if=gpttest.img bs=$[1024+128] count=1 | xxd -a
> 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 000001b0: 0000 0000 0000 0000 0000 0000 0000 00ff ................
> 000001c0: ffff eeff ffff 0100 0000 ffff 0300 0000 ................
> 000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 000001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U.
> 00000200: 4546 4920 5041 5254 0000 0100 5c00 0000  EFI PART....\...
> 00000210: 7b79 7b6e 0000 0000 0100 0000 0000 0000 {y{n............
> 00000220: ffff 0300 0000 0000 2200 0000 0000 0000 ........".......
> 00000230: deff 0300 0000 0000 2f50 fea4 64c2 514d ......../P..d.QM
> 00000240: 920b 3849 bb69 6ba7 0200 0000 0000 0000 ..8I.ik.........
> 00000250: 8000 0000 8000 0000 6435 b972 0000 0000 ........d5.r....
> 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 00000400: a2a0 d0eb e5b9 3344 87c0 68b6 b726 99c7 ......3D..h..&..
> 00000410: ceee 9eb1 b179 344b 89dc 7c7c 1b85 379e .....y4K..||..7.
> 00000420: 2200 0000 0000 0000 2150 0000 0000 0000 ".......!P......
> 00000430: 0000 0000 0000 0000 4600 6900 6c00 6500 ........F.i.l.e.
> 00000440: 2000 7300 7900 7300 7400 6500 6d00 0000 .s.y.s.t.e.m...
> 00000450: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000460: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 00000470: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> --------
>
>
> Another indicator that there's something wrong is that GNU parted 
> doesn't print the label:
> --------
> $ losetup
> NAME       SIZELIMIT OFFSET AUTOCLEAR RO 
> BACK-FILE                           DIO LOG-SEC
> /dev/loop1         0      0         0  1 
> /home/linux1/gptfdisk/gpttest.img     0     512
> /dev/loop2         0      0         0  1 
> /home/linux1/gptfdisk/tmp/test1.0.6   0     512
> /dev/loop0         0      0         0  0 
> /var/opt/zthin/cfgdrive.iso           0     512
> /dev/loop3         0      0         0  1 
> /home/linux1/gptfdisk/tmp/test1.0.7   0     512
> $ sudo parted /dev/loop2
> GNU Parted 3.2
> Using /dev/loop2
> Welcome to GNU Parted! Type 'help' to view a list of commands.
> (parted) p
> Model: Loopback device (loopback)
> Disk /dev/loop2: 67.1MB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
> Disk Flags:
>
> Warning: failed to translate partition name
> Number  Start   End     Size    File system  Name Flags
>  1      1049kB  67.1MB  66.0MB
>
> (parted) q
> $ sudo parted /dev/loop1
> GNU Parted 3.2
> Using /dev/loop1
> Welcome to GNU Parted! Type 'help' to view a list of commands.
> (parted) p
> Model: Loopback device (loopback)
> Disk /dev/loop1: 134MB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
> Disk Flags:
>
> Warning: failed to translate partition name
> Warning: failed to translate partition name
> Warning: failed to translate partition name
> Warning: failed to translate partition name
> Warning: failed to translate partition name
> Warning: failed to translate partition name
> Warning: failed to translate partition name
> Number  Start   End     Size    File system  Name Flags
>  1      17.4kB  10.5MB  10.5MB               File system  msftdata
>  2      10.5MB  21.0MB  10.5MB                            msftdata
>  3      21.0MB  31.5MB  10.5MB                            msftdata
>  4      31.5MB  42.0MB  10.5MB                            msftdata
>  5      42.0MB  52.4MB  10.5MB                            msftdata
>  6      52.4MB  62.9MB  10.5MB                            msftdata
>  7      62.9MB  73.4MB  10.5MB                            msftdata
>  8      73.4MB  83.9MB  10.5MB                            msftdata
>
> (parted) q
> --------
>
> (Clearly GNU parted doesn't like non-ASCII labels, but you can at 
> least see that the first label "File system" is printed.)
>
> If sgdisk created these partition layouts, then there's clearly a bug 
> in sgdisk which makes it write labels characters in big-endian format 
> without byteswapping them.

Best regards,

-Erik

>>>>> On 8.6.2021 10.12, Christian Ehrhardt wrote:
>>>>>> Hi,
>>>>>> I wanted to let you know that the recent try to fix big endian byte
>>>>>> swapping is actually breaking big endian in 1.0.7.
>>>>>>
>>>>>> This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2].
>>>>>>
>>>>>> When you look at just these logs it seems it is just failing for no
>>>>>> apparent reason,
>>>>>> but with -x enabled in gdisk_test.sh one quickly sees there is some
>>>>>> string-mangling happening:
>>>>>>
>>>>>> Number  Start (sector)    End (sector)  Size       Code Name
>>>>>>       1            2048          131038   63.0 MiB 8300  
>>>>>> 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀
>>>>>>
>>>>>> Reading the very same file with the old gdisk is good - so likely 
>>>>>> the
>>>>>> on-dis content is good as well:
>>>>>> Number  Start (sector)    End (sector)  Size       Code Name
>>>>>>       1            2048          131038   63.0 MiB 8300  Linux 
>>>>>> filesystem
>>>>>>
>>>>>> This is caused by GPTPart::GetDescription due to this change [3] 
>>>>>> in 1.0.7
>>>>>>
>>>>>> Reverting this fixes change fixes the problem.
>>>>>>
>>>>>> [1]: 
>>>>>> https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0
>>>>>> [2]: 
>>>>>> https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz
>>>>>> [3]: 
>>>>>> https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/
>>>>>>
>>>>
>>
>>