Thread: [Gptfdisk-general] GPTPart::GetDescription broken in 1.0.7 on Big Endian systems
Brought to you by:
srs5694
Menu
▾
▴
gptfdisk-general
|
From: Christian E. <chr...@ca...> - 2021-06-08 07:28:49
|
Hi, I wanted to let you know that the recent try to fix big endian byte swapping is actually breaking big endian in 1.0.7. This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. When you look at just these logs it seems it is just failing for no apparent reason, but with -x enabled in gdisk_test.sh one quickly sees there is some string-mangling happening: Number Start (sector) End (sector) Size Code Name 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 Reading the very same file with the old gdisk is good - so likely the on-dis content is good as well: Number Start (sector) End (sector) Size Code Name 1 2048 131038 63.0 MiB 8300 Linux filesystem This is caused by GPTPart::GetDescription due to this change [3] in 1.0.7 Reverting this fixes change fixes the problem. [1]: https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 [2]: https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz [3]: https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ -- Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd |
|
From: Erik L. <cat...@gm...> - 2021-06-08 07:24:32
|
Hi, I submitted this patch and tested it on Ubuntu 16.04 PowerPC (big endian, 32-bit) which had the same string-mangling problem when this fix is *not* applied... so there's something weird going on. In general I prefer to keep data read from disk into memory in the on-disk endianness and transform endianess when retrieving it or setting it. gptfdisk does not do this but instead byteswaps the fields back and forth in the structures that were read from disk. This is error-prone and can lead to on-disk corruption when it's written back. This might be contributing to the problems. I'll check if I can reproduce with e.g. Debian sid ppc64 (big endian) and find out what's going on. I have no access to an s390x machine but maybe qemu can help with that... Best regards, - Erik On 8.6.2021 10.12, Christian Ehrhardt wrote: > Hi, > I wanted to let you know that the recent try to fix big endian byte > swapping is actually breaking big endian in 1.0.7. > > This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. > > When you look at just these logs it seems it is just failing for no > apparent reason, > but with -x enabled in gdisk_test.sh one quickly sees there is some > string-mangling happening: > > Number Start (sector) End (sector) Size Code Name > 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 > > Reading the very same file with the old gdisk is good - so likely the > on-dis content is good as well: > Number Start (sector) End (sector) Size Code Name > 1 2048 131038 63.0 MiB 8300 Linux filesystem > > This is caused by GPTPart::GetDescription due to this change [3] in 1.0.7 > > Reverting this fixes change fixes the problem. > > [1]: https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 > [2]: https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz > [3]: https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ > |
|
From: Christian E. <chr...@ca...> - 2021-06-08 07:36:15
|
On Tue, Jun 8, 2021 at 9:24 AM Erik Larsson <cat...@gm...> wrote: > > Hi, > > I submitted this patch and tested it on Ubuntu 16.04 PowerPC (big > endian, 32-bit) which had the same string-mangling problem when this fix > is *not* applied... so there's something weird going on. Agreed and thanks for the quick reply. > In general I prefer to keep data read from disk into memory in the > on-disk endianness and transform endianess when retrieving it or setting it. > gptfdisk does not do this but instead byteswaps the fields back and > forth in the structures that were read from disk. This is error-prone > and can lead to on-disk corruption when it's written back. This might be > contributing to the problems. > > I'll check if I can reproduce with e.g. Debian sid ppc64 (big endian) I was not seeing the very same on ppc64el which is the ppc64 architecture of Debian/Ubutnu. But then the "el" means little endian so that would make sense. The https://wiki.debian.org/PPC64 for big endian exists but is unofficial and thereby might have some issues. I'd think the fastest way for you could be getting a free s390x VM to work on via https://developer.ibm.com/components/ibm-linuxone/gettingstarted/# > and find out what's going on. I have no access to an s390x machine but > maybe qemu can help with that... It can, but you'd want to use a rather new qemu and an older Debian/Ubuntu system as too recent instructions always are a problem. As I said the community cloud likely is your quickest way out of this. > Best regards, > > - Erik > > On 8.6.2021 10.12, Christian Ehrhardt wrote: > > Hi, > > I wanted to let you know that the recent try to fix big endian byte > > swapping is actually breaking big endian in 1.0.7. > > > > This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. > > > > When you look at just these logs it seems it is just failing for no > > apparent reason, > > but with -x enabled in gdisk_test.sh one quickly sees there is some > > string-mangling happening: > > > > Number Start (sector) End (sector) Size Code Name > > 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 > > > > Reading the very same file with the old gdisk is good - so likely the > > on-dis content is good as well: > > Number Start (sector) End (sector) Size Code Name > > 1 2048 131038 63.0 MiB 8300 Linux filesystem > > > > This is caused by GPTPart::GetDescription due to this change [3] in 1.0.7 > > > > Reverting this fixes change fixes the problem. > > > > [1]: https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 > > [2]: https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz > > [3]: https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ > > -- Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd |
|
From: Erik L. <cat...@gm...> - 2021-06-08 10:20:43
|
Hi Christian, On 8.6.2021 10.35, Christian Ehrhardt wrote: > On Tue, Jun 8, 2021 at 9:24 AM Erik Larsson <cat...@gm...> wrote: >> In general I prefer to keep data read from disk into memory in the >> on-disk endianness and transform endianess when retrieving it or setting it. >> gptfdisk does not do this but instead byteswaps the fields back and >> forth in the structures that were read from disk. This is error-prone >> and can lead to on-disk corruption when it's written back. This might be >> contributing to the problems. >> >> I'll check if I can reproduce with e.g. Debian sid ppc64 (big endian) > I was not seeing the very same on ppc64el which is the ppc64 > architecture of Debian/Ubutnu. > But then the "el" means little endian so that would make sense. Yes, that makes sense. No byteswapping needed there. > The https://wiki.debian.org/PPC64 for big endian exists but is > unofficial and thereby might have some issues. True, but I tested anyway in a chroot just to see if I got any different result but I did not. > I'd think the fastest way for you could be getting a free s390x VM to > work on via > https://developer.ibm.com/components/ibm-linuxone/gettingstarted/# Thanks for the pointer! That was very helpful, I was able to get an s390x VM up and running pretty quickly. Unfortunately they do not offer Ubuntu specifically but I fired up a RHEL8.3 VM to do the testing. >> and find out what's going on. I have no access to an s390x machine but >> maybe qemu can help with that... > It can, but you'd want to use a rather new qemu and an older > Debian/Ubuntu system as too recent instructions always are a problem. > As I said the community cloud likely is your quickest way out of this. Indeed it was the quickest way. I have now compiled gptfdisk from source on the s390x instance and compared it against the bundled 'gdisk' on RHEL8.3. I used a GPT sample image that I created with the 'gpt' utility in macOS, just to make sure it's untouched by the gdisk utility on any platform. It has 8 partitions with the phrase "File system" in various different languages as GPT partition labels (for testing Unicode chars). I uploaded the image here: https://catacombae.org/temp/gpttest.img.bz2 I am however getting the same results as I got on my PowerPC setup. With the RHEL8.3-provided gdisk 1.0.3: -------- $ uname -a Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux $ sudo gdisk /dev/loop1 GPT fdisk (gdisk) version 1.0.3 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Command (? for help): p Disk /dev/loop1: 262144 sectors, 128.0 MiB Sector size (logical/physical): 512/512 bytes Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 262110 Partitions will be aligned on 2-sector boundaries Total free space is 98237 sectors (48.0 MiB) Number Start (sector) End (sector) Size Code Name 1 34 20513 10.0 MiB 0700 䘀椀氀攀 猀礀猀琀攀洀 2 20514 40993 10.0 MiB 0700 ꌃ촃쌃쐃뜃밃넃 넃섃윃딃꼃줃봃 3 40994 61473 10.0 MiB 0700 蝥ﭼ 4 61474 81953 10.0 MiB 0700 ␄〄㤄㬄㸄㈄〄伄 䄄㠄䄄䈄㔄㰄〄 5 81954 102433 10.0 MiB 0700 䘆㠆✆䔆 ✆䐆䔆䐆䄆✆⨆ 6 102434 122913 10.0 MiB 0700 ⬉㸉܉㈉ 㠉㼉㠉䴉Ἁ⸉ 7 122914 143393 10.0 MiB 0700 8 143394 163873 10.0 MiB 0700 锋ꌋ뼋꼋锋촋锋쬋ꨋ촋ꨋ섋 긋섋넋젋긋젋 Command (? for help): q -------- It's all garbage. However with the patched and freshly compiled gdisk 1.0.7 straight from git: -------- $ uname -a Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux $ sudo ./gdisk /dev/loop1 GPT fdisk (gdisk) version 1.0.7 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Command (? for help): p Disk /dev/loop1: 262144 sectors, 128.0 MiB Sector size (logical/physical): 512/512 bytes Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 262110 Partitions will be aligned on 2-sector boundaries Total free space is 98237 sectors (48.0 MiB) Number Start (sector) End (sector) Size Code Name 1 34 20513 10.0 MiB 0700 File system 2 20514 40993 10.0 MiB 0700 Σύστημα αρχείων 3 40994 61473 10.0 MiB 0700 文件系统 4 61474 81953 10.0 MiB 0700 Файловая система 5 81954 102433 10.0 MiB 0700 نظام الملفات 6 102434 122913 10.0 MiB 0700 फाइल सिस्टम 7 122914 143393 10.0 MiB 0700 טעקע סיסטעם 8 143394 163873 10.0 MiB 0700 கணியக்கோப்பு முறைமை Command (? for help): q -------- With the version compiled from git including my fix it looks fine. All characters are decoded properly. Would you be able to test this image on your Ubuntu s390x instance and report back the results? Or provide an image with a counterexample? Best regards, - Erik >> On 8.6.2021 10.12, Christian Ehrhardt wrote: >>> Hi, >>> I wanted to let you know that the recent try to fix big endian byte >>> swapping is actually breaking big endian in 1.0.7. >>> >>> This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. >>> >>> When you look at just these logs it seems it is just failing for no >>> apparent reason, >>> but with -x enabled in gdisk_test.sh one quickly sees there is some >>> string-mangling happening: >>> >>> Number Start (sector) End (sector) Size Code Name >>> 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 >>> >>> Reading the very same file with the old gdisk is good - so likely the >>> on-dis content is good as well: >>> Number Start (sector) End (sector) Size Code Name >>> 1 2048 131038 63.0 MiB 8300 Linux filesystem >>> >>> This is caused by GPTPart::GetDescription due to this change [3] in 1.0.7 >>> >>> Reverting this fixes change fixes the problem. >>> >>> [1]: https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 >>> [2]: https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz >>> [3]: https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ >>> > > |
|
From: Christian E. <chr...@ca...> - 2021-06-08 10:47:43
|
On Tue, Jun 8, 2021 at 12:20 PM Erik Larsson <cat...@gm...> wrote: > > Hi Christian, > > On 8.6.2021 10.35, Christian Ehrhardt wrote: > > On Tue, Jun 8, 2021 at 9:24 AM Erik Larsson <cat...@gm...> wrote: > >> In general I prefer to keep data read from disk into memory in the > >> on-disk endianness and transform endianess when retrieving it or setting it. > >> gptfdisk does not do this but instead byteswaps the fields back and > >> forth in the structures that were read from disk. This is error-prone > >> and can lead to on-disk corruption when it's written back. This might be > >> contributing to the problems. > >> > >> I'll check if I can reproduce with e.g. Debian sid ppc64 (big endian) > > I was not seeing the very same on ppc64el which is the ppc64 > > architecture of Debian/Ubutnu. > > But then the "el" means little endian so that would make sense. > > > Yes, that makes sense. No byteswapping needed there. > > > The https://wiki.debian.org/PPC64 for big endian exists but is > > unofficial and thereby might have some issues. > > > True, but I tested anyway in a chroot just to see if I got any different > result but I did not. > > > I'd think the fastest way for you could be getting a free s390x VM to > > work on via > > https://developer.ibm.com/components/ibm-linuxone/gettingstarted/# > > > Thanks for the pointer! That was very helpful, I was able to get an > s390x VM up and running pretty quickly. > Unfortunately they do not offer Ubuntu specifically but I fired up a > RHEL8.3 VM to do the testing. Worst case (if needed) a Ubuntu/Debian container on that RH8.3 should do it. > >> and find out what's going on. I have no access to an s390x machine but > >> maybe qemu can help with that... > > It can, but you'd want to use a rather new qemu and an older > > Debian/Ubuntu system as too recent instructions always are a problem. > > As I said the community cloud likely is your quickest way out of this. > > > Indeed it was the quickest way. I have now compiled gptfdisk from source > on the s390x instance and compared it against the bundled 'gdisk' on > RHEL8.3. > I used a GPT sample image that I created with the 'gpt' utility in > macOS, just to make sure it's untouched by the gdisk utility on any > platform. > It has 8 partitions with the phrase "File system" in various different > languages as GPT partition labels (for testing Unicode chars). > > I uploaded the image here: > https://catacombae.org/temp/gpttest.img.bz2 > > I am however getting the same results as I got on my PowerPC setup. > > With the RHEL8.3-provided gdisk 1.0.3: > -------- > $ uname -a > Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17 > 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux > $ sudo gdisk /dev/loop1 > GPT fdisk (gdisk) version 1.0.3 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > > Command (? for help): p > Disk /dev/loop1: 262144 sectors, 128.0 MiB > Sector size (logical/physical): 512/512 bytes > Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7 > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 262110 > Partitions will be aligned on 2-sector boundaries > Total free space is 98237 sectors (48.0 MiB) > > Number Start (sector) End (sector) Size Code Name > 1 34 20513 10.0 MiB 0700 > 䘀椀氀攀 猀礀猀琀攀洀 > 2 20514 40993 10.0 MiB 0700 > ꌃ촃쌃쐃뜃밃넃 넃섃윃딃꼃줃봃 > 3 40994 61473 10.0 MiB 0700 蝥ﭼ > 4 61474 81953 10.0 MiB 0700 > ␄〄㤄㬄㸄㈄〄伄 䄄㠄䄄䈄㔄㰄〄 > 5 81954 102433 10.0 MiB 0700 䘆㠆✆䔆 ✆䐆䔆䐆䄆✆⨆ > 6 102434 122913 10.0 MiB 0700 ⬉㸉܉㈉ 㠉㼉㠉䴉Ἁ⸉ > 7 122914 143393 10.0 MiB 0700 > 8 143394 163873 10.0 MiB 0700 > 锋ꌋ뼋꼋锋촋锋쬋ꨋ촋ꨋ섋 긋섋넋젋긋젋 > > Command (? for help): q > -------- > > It's all garbage. However with the patched and freshly compiled gdisk > 1.0.7 straight from git: > > -------- > $ uname -a > Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17 > 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux > $ sudo ./gdisk /dev/loop1 > GPT fdisk (gdisk) version 1.0.7 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > > Command (? for help): p > Disk /dev/loop1: 262144 sectors, 128.0 MiB > Sector size (logical/physical): 512/512 bytes > Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7 > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 262110 > Partitions will be aligned on 2-sector boundaries > Total free space is 98237 sectors (48.0 MiB) > > Number Start (sector) End (sector) Size Code Name > 1 34 20513 10.0 MiB 0700 File system > 2 20514 40993 10.0 MiB 0700 Σύστημα αρχείων > 3 40994 61473 10.0 MiB 0700 文件系统 > 4 61474 81953 10.0 MiB 0700 Файловая система > 5 81954 102433 10.0 MiB 0700 نظام الملفات > 6 102434 122913 10.0 MiB 0700 फाइल सिस्टम > 7 122914 143393 10.0 MiB 0700 טעקע סיסטעם > 8 143394 163873 10.0 MiB 0700 கணியக்கோப்பு முறைமை > > Command (? for help): q > -------- > > With the version compiled from git including my fix it looks fine. All > characters are decoded properly. > > Would you be able to test this image on your Ubuntu s390x instance and > report back the results? Sure, I can do that > Or provide an image with a counterexample? And sure again, first a log, then cross tests and eventually the image files #1 1.0.6 $ dd if=/dev/zero of=/tmp/test1.0.6 bs=1024 count=65536 65536+0 records in 65536+0 records out 67108864 bytes (67 MB, 64 MiB) copied, 0.154128 s, 435 MB/s $ gdisk-1.0.6/sgdisk /tmp/test1.6 -o Warning: Partition table header claims that the size of partition table entries is 0 bytes, but this program supports only 128-byte entries. Adjusting accordingly, but partition table may be garbage. Warning: Partition table header claims that the size of partition table entries is 0 bytes, but this program supports only 128-byte entries. Adjusting accordingly, but partition table may be garbage. Creating new GPT entries in memory. Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. $ gdisk-1.0.6/sgdisk /tmp/test1.0.6 -n 1 -c '1:Linux filesystem' -t 1:8300 Setting name! partNum is 0 Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. $ gdisk-1.0.6/gdisk -l /tmp/test1.0.6 GPT fdisk (gdisk) version 1.0.6 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Disk /tmp/test1.0.6: 131072 sectors, 64.0 MiB Sector size (logical): 512 bytes Disk identifier (GUID): 970AC708-C718-4CDF-8EF4-1DCE94ADF8FB Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 131038 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector) End (sector) Size Code Name 1 2048 131038 63.0 MiB 8300 Linux filesystem #1 1.0.7 dd if=/dev/zero of=/tmp/test1.0.7 bs=1024 count=65536 65536+0 records in 65536+0 records out 67108864 bytes (67 MB, 64 MiB) copied, 0.190568 s, 352 MB/s gdisk-1.0.7/sgdisk /tmp/test1.0.7 -o Creating new GPT entries in memory. Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. gdisk-1.0.7/sgdisk /tmp/test1.0.7 -n 1 -c '1:Linux filesystem' -t 1:8300 Setting name! partNum is 0 Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. gdisk-1.0.7/gdisk -l /tmp/test1.0.7 GPT fdisk (gdisk) version 1.0.7 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Disk /tmp/test1.0.7: 131072 sectors, 64.0 MiB Sector size (logical): 512 bytes Disk identifier (GUID): 14750985-A59C-451B-B0D1-A49614B866E2 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 131038 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector) End (sector) Size Code Name 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 #3 cross tests #3.1 1.0.6 reading the 1.0.7 image $ gdisk-1.0.6/gdisk -l /tmp/test1.0.7 GPT fdisk (gdisk) version 1.0.6 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Disk /tmp/test1.0.7: 131072 sectors, 64.0 MiB Sector size (logical): 512 bytes Disk identifier (GUID): 14750985-A59C-451B-B0D1-A49614B866E2 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 131038 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector) End (sector) Size Code Name 1 2048 131038 63.0 MiB 8300 Linux filesystem #3.2 1.0.7 reading the 1.0.6 image $ gdisk-1.0.7/gdisk -l /tmp/test1.0.6 GPT fdisk (gdisk) version 1.0.7 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Disk /tmp/test1.0.6: 131072 sectors, 64.0 MiB Sector size (logical): 512 bytes Disk identifier (GUID): 970AC708-C718-4CDF-8EF4-1DCE94ADF8FB Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 131038 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector) End (sector) Size Code Name 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 These are the two image files that got created by the commands above: https://people.canonical.com/~paelzer/test1.0.6.tar.xz https://people.canonical.com/~paelzer/test1.0.7.tar.xz I hope that helps and let me know if you need anything else. > Best regards, > > - Erik > > > >> On 8.6.2021 10.12, Christian Ehrhardt wrote: > >>> Hi, > >>> I wanted to let you know that the recent try to fix big endian byte > >>> swapping is actually breaking big endian in 1.0.7. > >>> > >>> This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. > >>> > >>> When you look at just these logs it seems it is just failing for no > >>> apparent reason, > >>> but with -x enabled in gdisk_test.sh one quickly sees there is some > >>> string-mangling happening: > >>> > >>> Number Start (sector) End (sector) Size Code Name > >>> 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 > >>> > >>> Reading the very same file with the old gdisk is good - so likely the > >>> on-dis content is good as well: > >>> Number Start (sector) End (sector) Size Code Name > >>> 1 2048 131038 63.0 MiB 8300 Linux filesystem > >>> > >>> This is caused by GPTPart::GetDescription due to this change [3] in 1.0.7 > >>> > >>> Reverting this fixes change fixes the problem. > >>> > >>> [1]: https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 > >>> [2]: https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz > >>> [3]: https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ > >>> > > > > -- Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd |
|
From: Erik L. <cat...@gm...> - 2021-06-08 11:33:07
|
Hi Christian, On 8.6.2021 13.47, Christian Ehrhardt wrote: > On Tue, Jun 8, 2021 at 12:20 PM Erik Larsson <cat...@gm...> wrote: >> On 8.6.2021 10.35, Christian Ehrhardt wrote: >>> On Tue, Jun 8, 2021 at 9:24 AM Erik Larsson <cat...@gm...> wrote: <snip> >>> I'd think the fastest way for you could be getting a free s390x VM to >>> work on via >>> https://developer.ibm.com/components/ibm-linuxone/gettingstarted/# >> >> Thanks for the pointer! That was very helpful, I was able to get an >> s390x VM up and running pretty quickly. >> Unfortunately they do not offer Ubuntu specifically but I fired up a >> RHEL8.3 VM to do the testing. > Worst case (if needed) a Ubuntu/Debian container on that RH8.3 should do it. Good point, I could try that. >>>> and find out what's going on. I have no access to an s390x machine but >>>> maybe qemu can help with that... >>> It can, but you'd want to use a rather new qemu and an older >>> Debian/Ubuntu system as too recent instructions always are a problem. >>> As I said the community cloud likely is your quickest way out of this. >> >> Indeed it was the quickest way. I have now compiled gptfdisk from source >> on the s390x instance and compared it against the bundled 'gdisk' on >> RHEL8.3. >> I used a GPT sample image that I created with the 'gpt' utility in >> macOS, just to make sure it's untouched by the gdisk utility on any >> platform. >> It has 8 partitions with the phrase "File system" in various different >> languages as GPT partition labels (for testing Unicode chars). >> >> I uploaded the image here: >> https://catacombae.org/temp/gpttest.img.bz2 >> >> I am however getting the same results as I got on my PowerPC setup. >> >> With the RHEL8.3-provided gdisk 1.0.3: >> -------- >> $ uname -a >> Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17 >> 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux >> $ sudo gdisk /dev/loop1 >> GPT fdisk (gdisk) version 1.0.3 >> >> Partition table scan: >> MBR: protective >> BSD: not present >> APM: not present >> GPT: present >> >> Found valid GPT with protective MBR; using GPT. >> >> Command (? for help): p >> Disk /dev/loop1: 262144 sectors, 128.0 MiB >> Sector size (logical/physical): 512/512 bytes >> Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7 >> Partition table holds up to 128 entries >> Main partition table begins at sector 2 and ends at sector 33 >> First usable sector is 34, last usable sector is 262110 >> Partitions will be aligned on 2-sector boundaries >> Total free space is 98237 sectors (48.0 MiB) >> >> Number Start (sector) End (sector) Size Code Name >> 1 34 20513 10.0 MiB 0700 >> 䘀椀氀攀 猀礀猀琀攀洀 >> 2 20514 40993 10.0 MiB 0700 >> ꌃ촃쌃쐃뜃밃넃 넃섃윃딃꼃줃봃 >> 3 40994 61473 10.0 MiB 0700 蝥ﭼ >> 4 61474 81953 10.0 MiB 0700 >> ␄〄㤄㬄㸄㈄〄伄 䄄㠄䄄䈄㔄㰄〄 >> 5 81954 102433 10.0 MiB 0700 䘆㠆✆䔆 ✆䐆䔆䐆䄆✆⨆ >> 6 102434 122913 10.0 MiB 0700 ⬉㸉܉㈉ 㠉㼉㠉䴉Ἁ⸉ >> 7 122914 143393 10.0 MiB 0700 >> 8 143394 163873 10.0 MiB 0700 >> 锋ꌋ뼋꼋锋촋锋쬋ꨋ촋ꨋ섋 긋섋넋젋긋젋 >> >> Command (? for help): q >> -------- >> >> It's all garbage. However with the patched and freshly compiled gdisk >> 1.0.7 straight from git: >> >> -------- >> $ uname -a >> Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17 >> 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux >> $ sudo ./gdisk /dev/loop1 >> GPT fdisk (gdisk) version 1.0.7 >> >> Partition table scan: >> MBR: protective >> BSD: not present >> APM: not present >> GPT: present >> >> Found valid GPT with protective MBR; using GPT. >> >> Command (? for help): p >> Disk /dev/loop1: 262144 sectors, 128.0 MiB >> Sector size (logical/physical): 512/512 bytes >> Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7 >> Partition table holds up to 128 entries >> Main partition table begins at sector 2 and ends at sector 33 >> First usable sector is 34, last usable sector is 262110 >> Partitions will be aligned on 2-sector boundaries >> Total free space is 98237 sectors (48.0 MiB) >> >> Number Start (sector) End (sector) Size Code Name >> 1 34 20513 10.0 MiB 0700 File system >> 2 20514 40993 10.0 MiB 0700 Σύστημα αρχείων >> 3 40994 61473 10.0 MiB 0700 文件系统 >> 4 61474 81953 10.0 MiB 0700 Файловая система >> 5 81954 102433 10.0 MiB 0700 نظام الملفات >> 6 102434 122913 10.0 MiB 0700 फाइल सिस्टम >> 7 122914 143393 10.0 MiB 0700 טעקע סיסטעם >> 8 143394 163873 10.0 MiB 0700 கணியக்கோப்பு முறைமை >> >> Command (? for help): q >> -------- >> >> With the version compiled from git including my fix it looks fine. All >> characters are decoded properly. >> >> Would you be able to test this image on your Ubuntu s390x instance and >> report back the results? > Sure, I can do that > >> Or provide an image with a counterexample? > And sure again, first a log, then cross tests and eventually the image files > > #1 1.0.6 > > $ dd if=/dev/zero of=/tmp/test1.0.6 bs=1024 count=65536 > 65536+0 records in > 65536+0 records out > 67108864 bytes (67 MB, 64 MiB) copied, 0.154128 s, 435 MB/s > $ gdisk-1.0.6/sgdisk /tmp/test1.6 -o > Warning: Partition table header claims that the size of partition table > entries is 0 bytes, but this program supports only 128-byte entries. > Adjusting accordingly, but partition table may be garbage. > Warning: Partition table header claims that the size of partition table > entries is 0 bytes, but this program supports only 128-byte entries. > Adjusting accordingly, but partition table may be garbage. > Creating new GPT entries in memory. > Warning: The kernel is still using the old partition table. > The new table will be used at the next reboot or after you > run partprobe(8) or kpartx(8) > The operation has completed successfully. > $ gdisk-1.0.6/sgdisk /tmp/test1.0.6 -n 1 -c '1:Linux filesystem' -t 1:8300 > Setting name! > partNum is 0 > Warning: The kernel is still using the old partition table. > The new table will be used at the next reboot or after you > run partprobe(8) or kpartx(8) > The operation has completed successfully. > $ gdisk-1.0.6/gdisk -l /tmp/test1.0.6 > GPT fdisk (gdisk) version 1.0.6 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > Disk /tmp/test1.0.6: 131072 sectors, 64.0 MiB > Sector size (logical): 512 bytes > Disk identifier (GUID): 970AC708-C718-4CDF-8EF4-1DCE94ADF8FB > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 131038 > Partitions will be aligned on 2048-sector boundaries > Total free space is 2014 sectors (1007.0 KiB) > > Number Start (sector) End (sector) Size Code Name > 1 2048 131038 63.0 MiB 8300 Linux filesystem > > > #1 1.0.7 > > dd if=/dev/zero of=/tmp/test1.0.7 bs=1024 count=65536 > 65536+0 records in > 65536+0 records out > 67108864 bytes (67 MB, 64 MiB) copied, 0.190568 s, 352 MB/s > gdisk-1.0.7/sgdisk /tmp/test1.0.7 -o > Creating new GPT entries in memory. > Warning: The kernel is still using the old partition table. > The new table will be used at the next reboot or after you > run partprobe(8) or kpartx(8) > The operation has completed successfully. > gdisk-1.0.7/sgdisk /tmp/test1.0.7 -n 1 -c '1:Linux filesystem' -t 1:8300 > Setting name! > partNum is 0 > Warning: The kernel is still using the old partition table. > The new table will be used at the next reboot or after you > run partprobe(8) or kpartx(8) > The operation has completed successfully. > gdisk-1.0.7/gdisk -l /tmp/test1.0.7 > GPT fdisk (gdisk) version 1.0.7 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > Disk /tmp/test1.0.7: 131072 sectors, 64.0 MiB > Sector size (logical): 512 bytes > Disk identifier (GUID): 14750985-A59C-451B-B0D1-A49614B866E2 > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 131038 > Partitions will be aligned on 2048-sector boundaries > Total free space is 2014 sectors (1007.0 KiB) > > Number Start (sector) End (sector) Size Code Name > 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 > > > #3 cross tests > > #3.1 1.0.6 reading the 1.0.7 image > > $ gdisk-1.0.6/gdisk -l /tmp/test1.0.7 > GPT fdisk (gdisk) version 1.0.6 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > Disk /tmp/test1.0.7: 131072 sectors, 64.0 MiB > Sector size (logical): 512 bytes > Disk identifier (GUID): 14750985-A59C-451B-B0D1-A49614B866E2 > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 131038 > Partitions will be aligned on 2048-sector boundaries > Total free space is 2014 sectors (1007.0 KiB) > > Number Start (sector) End (sector) Size Code Name > 1 2048 131038 63.0 MiB 8300 Linux filesystem > > #3.2 1.0.7 reading the 1.0.6 image > > $ gdisk-1.0.7/gdisk -l /tmp/test1.0.6 > GPT fdisk (gdisk) version 1.0.7 > > Partition table scan: > MBR: protective > BSD: not present > APM: not present > GPT: present > > Found valid GPT with protective MBR; using GPT. > Disk /tmp/test1.0.6: 131072 sectors, 64.0 MiB > Sector size (logical): 512 bytes > Disk identifier (GUID): 970AC708-C718-4CDF-8EF4-1DCE94ADF8FB > Partition table holds up to 128 entries > Main partition table begins at sector 2 and ends at sector 33 > First usable sector is 34, last usable sector is 131038 > Partitions will be aligned on 2048-sector boundaries > Total free space is 2014 sectors (1007.0 KiB) > > Number Start (sector) End (sector) Size Code Name > 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 > > > These are the two image files that got created by the commands above: > https://people.canonical.com/~paelzer/test1.0.6.tar.xz > https://people.canonical.com/~paelzer/test1.0.7.tar.xz > > I hope that helps and let me know if you need anything else. Thanks for those images. Now it's much clearer what is going on. Both your images have the on-disk volume label UTF-16 units in big endian, not little endian which is mandated by the EFI specification! Just look at the first 1 KiB + 128 byte (first GPT entry) of the volume how the characters in the string "Linux filesystem" are in big-endian (least significant byte first): -------- $ dd if=tmp/test1.0.6 bs=$[1024+128] count=1 | xxd -a 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ * 000001c0: 0200 ee28 2008 0100 0000 ffff 0100 0000 ...( ........... 000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U. 00000200: 4546 4920 5041 5254 0000 0100 5c00 0000 EFI PART....\... 00000210: 8b05 d554 0000 0000 0100 0000 0000 0000 ...T............ 00000220: ffff 0100 0000 0000 2200 0000 0000 0000 ........"....... 00000230: deff 0100 0000 0000 08c7 0a97 18c7 df4c ...............L 00000240: 8ef4 1dce 94ad f8fb 0200 0000 0000 0000 ................ 00000250: 8000 0000 8000 0000 880a f472 0000 0000 ...........r.... 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................ * 00000400: af3d c60f 8384 7247 8e79 3d69 d847 7de4 .=....rG.y=i.G}. 00000410: 4d33 25ac 07d6 794c 9bcf 50c5 4e86 fc72 M3%...yL..P.N..r 00000420: 0008 0000 0000 0000 deff 0100 0000 0000 ................ 00000430: 0000 0000 0000 0000 004c 0069 006e 0075 .........L.i.n.u 00000440: 0078 0020 0066 0069 006c 0065 0073 0079 .x. .f.i.l.e.s.y 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ 00000460: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000470: 0000 0000 0000 0000 0000 0000 0000 0000 ................ -------- Compare this to the image I provided where the label characters ("File system") are in little endian (most significant byte first): -------- $ dd if=gpttest.img bs=$[1024+128] count=1 | xxd -a 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ * 000001b0: 0000 0000 0000 0000 0000 0000 0000 00ff ................ 000001c0: ffff eeff ffff 0100 0000 ffff 0300 0000 ................ 000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 000001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U. 00000200: 4546 4920 5041 5254 0000 0100 5c00 0000 EFI PART....\... 00000210: 7b79 7b6e 0000 0000 0100 0000 0000 0000 {y{n............ 00000220: ffff 0300 0000 0000 2200 0000 0000 0000 ........"....... 00000230: deff 0300 0000 0000 2f50 fea4 64c2 514d ......../P..d.QM 00000240: 920b 3849 bb69 6ba7 0200 0000 0000 0000 ..8I.ik......... 00000250: 8000 0000 8000 0000 6435 b972 0000 0000 ........d5.r.... 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................ * 00000400: a2a0 d0eb e5b9 3344 87c0 68b6 b726 99c7 ......3D..h..&.. 00000410: ceee 9eb1 b179 344b 89dc 7c7c 1b85 379e .....y4K..||..7. 00000420: 2200 0000 0000 0000 2150 0000 0000 0000 ".......!P...... 00000430: 0000 0000 0000 0000 4600 6900 6c00 6500 ........F.i.l.e. 00000440: 2000 7300 7900 7300 7400 6500 6d00 0000 .s.y.s.t.e.m... 00000450: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000460: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00000470: 0000 0000 0000 0000 0000 0000 0000 0000 ................ -------- Another indicator that there's something wrong is that GNU parted doesn't print the label: -------- $ losetup NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC /dev/loop1 0 0 0 1 /home/linux1/gptfdisk/gpttest.img 0 512 /dev/loop2 0 0 0 1 /home/linux1/gptfdisk/tmp/test1.0.6 0 512 /dev/loop0 0 0 0 0 /var/opt/zthin/cfgdrive.iso 0 512 /dev/loop3 0 0 0 1 /home/linux1/gptfdisk/tmp/test1.0.7 0 512 $ sudo parted /dev/loop2 GNU Parted 3.2 Using /dev/loop2 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: Loopback device (loopback) Disk /dev/loop2: 67.1MB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Warning: failed to translate partition name Number Start End Size File system Name Flags 1 1049kB 67.1MB 66.0MB (parted) q $ sudo parted /dev/loop1 GNU Parted 3.2 Using /dev/loop1 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: Loopback device (loopback) Disk /dev/loop1: 134MB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Warning: failed to translate partition name Warning: failed to translate partition name Warning: failed to translate partition name Warning: failed to translate partition name Warning: failed to translate partition name Warning: failed to translate partition name Warning: failed to translate partition name Number Start End Size File system Name Flags 1 17.4kB 10.5MB 10.5MB File system msftdata 2 10.5MB 21.0MB 10.5MB msftdata 3 21.0MB 31.5MB 10.5MB msftdata 4 31.5MB 42.0MB 10.5MB msftdata 5 42.0MB 52.4MB 10.5MB msftdata 6 52.4MB 62.9MB 10.5MB msftdata 7 62.9MB 73.4MB 10.5MB msftdata 8 73.4MB 83.9MB 10.5MB msftdata (parted) q -------- (Clearly GNU parted doesn't like non-ASCII labels, but you can at least see that the first label "File system" is printed.) If sgdisk created these partition layouts, then there's clearly a bug in sgdisk which makes it write labels characters in big-endian format without byteswapping them. Best regards, - Erik >>>> On 8.6.2021 10.12, Christian Ehrhardt wrote: >>>>> Hi, >>>>> I wanted to let you know that the recent try to fix big endian byte >>>>> swapping is actually breaking big endian in 1.0.7. >>>>> >>>>> This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. >>>>> >>>>> When you look at just these logs it seems it is just failing for no >>>>> apparent reason, >>>>> but with -x enabled in gdisk_test.sh one quickly sees there is some >>>>> string-mangling happening: >>>>> >>>>> Number Start (sector) End (sector) Size Code Name >>>>> 1 2048 131038 63.0 MiB 8300 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 >>>>> >>>>> Reading the very same file with the old gdisk is good - so likely the >>>>> on-dis content is good as well: >>>>> Number Start (sector) End (sector) Size Code Name >>>>> 1 2048 131038 63.0 MiB 8300 Linux filesystem >>>>> >>>>> This is caused by GPTPart::GetDescription due to this change [3] in 1.0.7 >>>>> >>>>> Reverting this fixes change fixes the problem. >>>>> >>>>> [1]: https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 >>>>> [2]: https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz >>>>> [3]: https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ >>>>> >>> > > |
|
From: Erik L. <cat...@gm...> - 2021-06-08 11:39:31
|
Hi, Just some minor corrections that I felt compelled to do because of an embarassing thought error. They're inlined below... On 8.6.2021 14.32, Erik Larsson wrote: > Hi Christian, > > On 8.6.2021 13.47, Christian Ehrhardt wrote: >> On Tue, Jun 8, 2021 at 12:20 PM Erik Larsson <cat...@gm...> >> wrote: >>> On 8.6.2021 10.35, Christian Ehrhardt wrote: >>>> On Tue, Jun 8, 2021 at 9:24 AM Erik Larsson <cat...@gm...> >>>> wrote: > <snip> >>>> I'd think the fastest way for you could be getting a free s390x VM to >>>> work on via >>>> https://developer.ibm.com/components/ibm-linuxone/gettingstarted/# >>> >>> Thanks for the pointer! That was very helpful, I was able to get an >>> s390x VM up and running pretty quickly. >>> Unfortunately they do not offer Ubuntu specifically but I fired up a >>> RHEL8.3 VM to do the testing. >> Worst case (if needed) a Ubuntu/Debian container on that RH8.3 should >> do it. > > Good point, I could try that. > >>>>> and find out what's going on. I have no access to an s390x machine >>>>> but >>>>> maybe qemu can help with that... >>>> It can, but you'd want to use a rather new qemu and an older >>>> Debian/Ubuntu system as too recent instructions always are a problem. >>>> As I said the community cloud likely is your quickest way out of this. >>> >>> Indeed it was the quickest way. I have now compiled gptfdisk from >>> source >>> on the s390x instance and compared it against the bundled 'gdisk' on >>> RHEL8.3. >>> I used a GPT sample image that I created with the 'gpt' utility in >>> macOS, just to make sure it's untouched by the gdisk utility on any >>> platform. >>> It has 8 partitions with the phrase "File system" in various different >>> languages as GPT partition labels (for testing Unicode chars). >>> >>> I uploaded the image here: >>> https://catacombae.org/temp/gpttest.img.bz2 >>> >>> I am however getting the same results as I got on my PowerPC setup. >>> >>> With the RHEL8.3-provided gdisk 1.0.3: >>> -------- >>> $ uname -a >>> Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17 >>> 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux >>> $ sudo gdisk /dev/loop1 >>> GPT fdisk (gdisk) version 1.0.3 >>> >>> Partition table scan: >>> MBR: protective >>> BSD: not present >>> APM: not present >>> GPT: present >>> >>> Found valid GPT with protective MBR; using GPT. >>> >>> Command (? for help): p >>> Disk /dev/loop1: 262144 sectors, 128.0 MiB >>> Sector size (logical/physical): 512/512 bytes >>> Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7 >>> Partition table holds up to 128 entries >>> Main partition table begins at sector 2 and ends at sector 33 >>> First usable sector is 34, last usable sector is 262110 >>> Partitions will be aligned on 2-sector boundaries >>> Total free space is 98237 sectors (48.0 MiB) >>> >>> Number Start (sector) End (sector) Size Code Name >>> 1 34 20513 10.0 MiB 0700 >>> 䘀椀氀攀 猀礀猀琀攀洀 >>> 2 20514 40993 10.0 MiB 0700 >>> ꌃ촃쌃쐃뜃밃넃 넃섃윃딃꼃줃봃 >>> 3 40994 61473 10.0 MiB 0700 蝥ﭼ >>> 4 61474 81953 10.0 MiB 0700 >>> ␄〄㤄㬄㸄㈄〄伄 䄄㠄䄄䈄㔄㰄〄 >>> 5 81954 102433 10.0 MiB 0700 >>> 䘆㠆✆䔆 ✆䐆䔆䐆䄆✆⨆ >>> 6 102434 122913 10.0 MiB 0700 >>> ⬉㸉܉㈉ 㠉㼉㠉䴉Ἁ⸉ >>> 7 122914 143393 10.0 MiB 0700 >>> 8 143394 163873 10.0 MiB 0700 >>> 锋ꌋ뼋꼋锋촋锋쬋ꨋ촋ꨋ섋 긋섋넋젋긋젋 >>> >>> Command (? for help): q >>> -------- >>> >>> It's all garbage. However with the patched and freshly compiled gdisk >>> 1.0.7 straight from git: >>> >>> -------- >>> $ uname -a >>> Linux catacombaetest 4.18.0-305.3.1.el8_4.s390x #1 SMP Mon May 17 >>> 10:16:29 EDT 2021 s390x s390x s390x GNU/Linux >>> $ sudo ./gdisk /dev/loop1 >>> GPT fdisk (gdisk) version 1.0.7 >>> >>> Partition table scan: >>> MBR: protective >>> BSD: not present >>> APM: not present >>> GPT: present >>> >>> Found valid GPT with protective MBR; using GPT. >>> >>> Command (? for help): p >>> Disk /dev/loop1: 262144 sectors, 128.0 MiB >>> Sector size (logical/physical): 512/512 bytes >>> Disk identifier (GUID): A4FE502F-C264-4D51-920B-3849BB696BA7 >>> Partition table holds up to 128 entries >>> Main partition table begins at sector 2 and ends at sector 33 >>> First usable sector is 34, last usable sector is 262110 >>> Partitions will be aligned on 2-sector boundaries >>> Total free space is 98237 sectors (48.0 MiB) >>> >>> Number Start (sector) End (sector) Size Code Name >>> 1 34 20513 10.0 MiB 0700 File system >>> 2 20514 40993 10.0 MiB 0700 Σύστημα >>> αρχείων >>> 3 40994 61473 10.0 MiB 0700 文件系统 >>> 4 61474 81953 10.0 MiB 0700 Файловая >>> система >>> 5 81954 102433 10.0 MiB 0700 نظام الملفات >>> 6 102434 122913 10.0 MiB 0700 फाइल सिस्टम >>> 7 122914 143393 10.0 MiB 0700 טעקע סיסטעם >>> 8 143394 163873 10.0 MiB 0700 >>> கணியக்கோப்பு முறைமை >>> >>> Command (? for help): q >>> -------- >>> >>> With the version compiled from git including my fix it looks fine. All >>> characters are decoded properly. >>> >>> Would you be able to test this image on your Ubuntu s390x instance and >>> report back the results? >> Sure, I can do that >> >>> Or provide an image with a counterexample? >> And sure again, first a log, then cross tests and eventually the >> image files >> >> #1 1.0.6 >> >> $ dd if=/dev/zero of=/tmp/test1.0.6 bs=1024 count=65536 >> 65536+0 records in >> 65536+0 records out >> 67108864 bytes (67 MB, 64 MiB) copied, 0.154128 s, 435 MB/s >> $ gdisk-1.0.6/sgdisk /tmp/test1.6 -o >> Warning: Partition table header claims that the size of partition table >> entries is 0 bytes, but this program supports only 128-byte entries. >> Adjusting accordingly, but partition table may be garbage. >> Warning: Partition table header claims that the size of partition table >> entries is 0 bytes, but this program supports only 128-byte entries. >> Adjusting accordingly, but partition table may be garbage. >> Creating new GPT entries in memory. >> Warning: The kernel is still using the old partition table. >> The new table will be used at the next reboot or after you >> run partprobe(8) or kpartx(8) >> The operation has completed successfully. >> $ gdisk-1.0.6/sgdisk /tmp/test1.0.6 -n 1 -c '1:Linux filesystem' -t >> 1:8300 >> Setting name! >> partNum is 0 >> Warning: The kernel is still using the old partition table. >> The new table will be used at the next reboot or after you >> run partprobe(8) or kpartx(8) >> The operation has completed successfully. >> $ gdisk-1.0.6/gdisk -l /tmp/test1.0.6 >> GPT fdisk (gdisk) version 1.0.6 >> >> Partition table scan: >> MBR: protective >> BSD: not present >> APM: not present >> GPT: present >> >> Found valid GPT with protective MBR; using GPT. >> Disk /tmp/test1.0.6: 131072 sectors, 64.0 MiB >> Sector size (logical): 512 bytes >> Disk identifier (GUID): 970AC708-C718-4CDF-8EF4-1DCE94ADF8FB >> Partition table holds up to 128 entries >> Main partition table begins at sector 2 and ends at sector 33 >> First usable sector is 34, last usable sector is 131038 >> Partitions will be aligned on 2048-sector boundaries >> Total free space is 2014 sectors (1007.0 KiB) >> >> Number Start (sector) End (sector) Size Code Name >> 1 2048 131038 63.0 MiB 8300 Linux >> filesystem >> >> >> #1 1.0.7 >> >> dd if=/dev/zero of=/tmp/test1.0.7 bs=1024 count=65536 >> 65536+0 records in >> 65536+0 records out >> 67108864 bytes (67 MB, 64 MiB) copied, 0.190568 s, 352 MB/s >> gdisk-1.0.7/sgdisk /tmp/test1.0.7 -o >> Creating new GPT entries in memory. >> Warning: The kernel is still using the old partition table. >> The new table will be used at the next reboot or after you >> run partprobe(8) or kpartx(8) >> The operation has completed successfully. >> gdisk-1.0.7/sgdisk /tmp/test1.0.7 -n 1 -c '1:Linux filesystem' -t 1:8300 >> Setting name! >> partNum is 0 >> Warning: The kernel is still using the old partition table. >> The new table will be used at the next reboot or after you >> run partprobe(8) or kpartx(8) >> The operation has completed successfully. >> gdisk-1.0.7/gdisk -l /tmp/test1.0.7 >> GPT fdisk (gdisk) version 1.0.7 >> >> Partition table scan: >> MBR: protective >> BSD: not present >> APM: not present >> GPT: present >> >> Found valid GPT with protective MBR; using GPT. >> Disk /tmp/test1.0.7: 131072 sectors, 64.0 MiB >> Sector size (logical): 512 bytes >> Disk identifier (GUID): 14750985-A59C-451B-B0D1-A49614B866E2 >> Partition table holds up to 128 entries >> Main partition table begins at sector 2 and ends at sector 33 >> First usable sector is 34, last usable sector is 131038 >> Partitions will be aligned on 2048-sector boundaries >> Total free space is 2014 sectors (1007.0 KiB) >> >> Number Start (sector) End (sector) Size Code Name >> 1 2048 131038 63.0 MiB 8300 >> 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 >> >> >> #3 cross tests >> >> #3.1 1.0.6 reading the 1.0.7 image >> >> $ gdisk-1.0.6/gdisk -l /tmp/test1.0.7 >> GPT fdisk (gdisk) version 1.0.6 >> >> Partition table scan: >> MBR: protective >> BSD: not present >> APM: not present >> GPT: present >> >> Found valid GPT with protective MBR; using GPT. >> Disk /tmp/test1.0.7: 131072 sectors, 64.0 MiB >> Sector size (logical): 512 bytes >> Disk identifier (GUID): 14750985-A59C-451B-B0D1-A49614B866E2 >> Partition table holds up to 128 entries >> Main partition table begins at sector 2 and ends at sector 33 >> First usable sector is 34, last usable sector is 131038 >> Partitions will be aligned on 2048-sector boundaries >> Total free space is 2014 sectors (1007.0 KiB) >> >> Number Start (sector) End (sector) Size Code Name >> 1 2048 131038 63.0 MiB 8300 Linux >> filesystem >> >> #3.2 1.0.7 reading the 1.0.6 image >> >> $ gdisk-1.0.7/gdisk -l /tmp/test1.0.6 >> GPT fdisk (gdisk) version 1.0.7 >> >> Partition table scan: >> MBR: protective >> BSD: not present >> APM: not present >> GPT: present >> >> Found valid GPT with protective MBR; using GPT. >> Disk /tmp/test1.0.6: 131072 sectors, 64.0 MiB >> Sector size (logical): 512 bytes >> Disk identifier (GUID): 970AC708-C718-4CDF-8EF4-1DCE94ADF8FB >> Partition table holds up to 128 entries >> Main partition table begins at sector 2 and ends at sector 33 >> First usable sector is 34, last usable sector is 131038 >> Partitions will be aligned on 2048-sector boundaries >> Total free space is 2014 sectors (1007.0 KiB) >> >> Number Start (sector) End (sector) Size Code Name >> 1 2048 131038 63.0 MiB 8300 >> 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 >> >> >> These are the two image files that got created by the commands above: >> https://people.canonical.com/~paelzer/test1.0.6.tar.xz >> https://people.canonical.com/~paelzer/test1.0.7.tar.xz >> >> I hope that helps and let me know if you need anything else. > > Thanks for those images. Now it's much clearer what is going on. > Both your images have the on-disk volume label UTF-16 units in big > endian, not little endian which is mandated by the EFI specification! > > Just look at the first 1 KiB + 128 byte (first GPT entry) of the > volume how the characters in the string "Linux filesystem" are in > big-endian (least significant byte first): This was a thought error. I meant most significant byte, not least. :) > -------- > $ dd if=tmp/test1.0.6 bs=$[1024+128] count=1 | xxd -a > 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > * > 000001c0: 0200 ee28 2008 0100 0000 ffff 0100 0000 ...( ........... > 000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 000001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U. > 00000200: 4546 4920 5041 5254 0000 0100 5c00 0000 EFI PART....\... > 00000210: 8b05 d554 0000 0000 0100 0000 0000 0000 ...T............ > 00000220: ffff 0100 0000 0000 2200 0000 0000 0000 ........"....... > 00000230: deff 0100 0000 0000 08c7 0a97 18c7 df4c ...............L > 00000240: 8ef4 1dce 94ad f8fb 0200 0000 0000 0000 ................ > 00000250: 8000 0000 8000 0000 880a f472 0000 0000 ...........r.... > 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > * > 00000400: af3d c60f 8384 7247 8e79 3d69 d847 7de4 .=....rG.y=i.G}. > 00000410: 4d33 25ac 07d6 794c 9bcf 50c5 4e86 fc72 M3%...yL..P.N..r > 00000420: 0008 0000 0000 0000 deff 0100 0000 0000 ................ > 00000430: 0000 0000 0000 0000 004c 0069 006e 0075 .........L.i.n.u > 00000440: 0078 0020 0066 0069 006c 0065 0073 0079 .x. .f.i.l.e.s.y > 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ > 00000460: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 00000470: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > -------- > > Compare this to the image I provided where the label characters ("File > system") are in little endian (most significant byte first): This was a thought error. I meant least significant byte, not most. :) > -------- > $ dd if=gpttest.img bs=$[1024+128] count=1 | xxd -a > 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > * > 000001b0: 0000 0000 0000 0000 0000 0000 0000 00ff ................ > 000001c0: ffff eeff ffff 0100 0000 ffff 0300 0000 ................ > 000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 000001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U. > 00000200: 4546 4920 5041 5254 0000 0100 5c00 0000 EFI PART....\... > 00000210: 7b79 7b6e 0000 0000 0100 0000 0000 0000 {y{n............ > 00000220: ffff 0300 0000 0000 2200 0000 0000 0000 ........"....... > 00000230: deff 0300 0000 0000 2f50 fea4 64c2 514d ......../P..d.QM > 00000240: 920b 3849 bb69 6ba7 0200 0000 0000 0000 ..8I.ik......... > 00000250: 8000 0000 8000 0000 6435 b972 0000 0000 ........d5.r.... > 00000260: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > * > 00000400: a2a0 d0eb e5b9 3344 87c0 68b6 b726 99c7 ......3D..h..&.. > 00000410: ceee 9eb1 b179 344b 89dc 7c7c 1b85 379e .....y4K..||..7. > 00000420: 2200 0000 0000 0000 2150 0000 0000 0000 ".......!P...... > 00000430: 0000 0000 0000 0000 4600 6900 6c00 6500 ........F.i.l.e. > 00000440: 2000 7300 7900 7300 7400 6500 6d00 0000 .s.y.s.t.e.m... > 00000450: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 00000460: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 00000470: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > -------- > > > Another indicator that there's something wrong is that GNU parted > doesn't print the label: > -------- > $ losetup > NAME SIZELIMIT OFFSET AUTOCLEAR RO > BACK-FILE DIO LOG-SEC > /dev/loop1 0 0 0 1 > /home/linux1/gptfdisk/gpttest.img 0 512 > /dev/loop2 0 0 0 1 > /home/linux1/gptfdisk/tmp/test1.0.6 0 512 > /dev/loop0 0 0 0 0 > /var/opt/zthin/cfgdrive.iso 0 512 > /dev/loop3 0 0 0 1 > /home/linux1/gptfdisk/tmp/test1.0.7 0 512 > $ sudo parted /dev/loop2 > GNU Parted 3.2 > Using /dev/loop2 > Welcome to GNU Parted! Type 'help' to view a list of commands. > (parted) p > Model: Loopback device (loopback) > Disk /dev/loop2: 67.1MB > Sector size (logical/physical): 512B/512B > Partition Table: gpt > Disk Flags: > > Warning: failed to translate partition name > Number Start End Size File system Name Flags > 1 1049kB 67.1MB 66.0MB > > (parted) q > $ sudo parted /dev/loop1 > GNU Parted 3.2 > Using /dev/loop1 > Welcome to GNU Parted! Type 'help' to view a list of commands. > (parted) p > Model: Loopback device (loopback) > Disk /dev/loop1: 134MB > Sector size (logical/physical): 512B/512B > Partition Table: gpt > Disk Flags: > > Warning: failed to translate partition name > Warning: failed to translate partition name > Warning: failed to translate partition name > Warning: failed to translate partition name > Warning: failed to translate partition name > Warning: failed to translate partition name > Warning: failed to translate partition name > Number Start End Size File system Name Flags > 1 17.4kB 10.5MB 10.5MB File system msftdata > 2 10.5MB 21.0MB 10.5MB msftdata > 3 21.0MB 31.5MB 10.5MB msftdata > 4 31.5MB 42.0MB 10.5MB msftdata > 5 42.0MB 52.4MB 10.5MB msftdata > 6 52.4MB 62.9MB 10.5MB msftdata > 7 62.9MB 73.4MB 10.5MB msftdata > 8 73.4MB 83.9MB 10.5MB msftdata > > (parted) q > -------- > > (Clearly GNU parted doesn't like non-ASCII labels, but you can at > least see that the first label "File system" is printed.) > > If sgdisk created these partition layouts, then there's clearly a bug > in sgdisk which makes it write labels characters in big-endian format > without byteswapping them. Best regards, -Erik >>>>> On 8.6.2021 10.12, Christian Ehrhardt wrote: >>>>>> Hi, >>>>>> I wanted to let you know that the recent try to fix big endian byte >>>>>> swapping is actually breaking big endian in 1.0.7. >>>>>> >>>>>> This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. >>>>>> >>>>>> When you look at just these logs it seems it is just failing for no >>>>>> apparent reason, >>>>>> but with -x enabled in gdisk_test.sh one quickly sees there is some >>>>>> string-mangling happening: >>>>>> >>>>>> Number Start (sector) End (sector) Size Code Name >>>>>> 1 2048 131038 63.0 MiB 8300 >>>>>> 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 >>>>>> >>>>>> Reading the very same file with the old gdisk is good - so likely >>>>>> the >>>>>> on-dis content is good as well: >>>>>> Number Start (sector) End (sector) Size Code Name >>>>>> 1 2048 131038 63.0 MiB 8300 Linux >>>>>> filesystem >>>>>> >>>>>> This is caused by GPTPart::GetDescription due to this change [3] >>>>>> in 1.0.7 >>>>>> >>>>>> Reverting this fixes change fixes the problem. >>>>>> >>>>>> [1]: >>>>>> https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 >>>>>> [2]: >>>>>> https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz >>>>>> [3]: >>>>>> https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ >>>>>> >>>> >> >> |
|
From: Christian E. <chr...@ca...> - 2021-06-08 12:35:30
|
... > This was a thought error. I meant most significant byte, not least. :) not a problem, the examples were more important anyway > > -------- > > $ dd if=tmp/test1.0.6 bs=$[1024+128] count=1 | xxd -a ... > > 00000430: 0000 0000 0000 0000 004c 0069 006e 0075 .........L.i.n.u > > 00000440: 0078 0020 0066 0069 006c 0065 0073 0079 .x. .f.i.l.e.s.y > > 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ Yes, thanks for the command I was able to confirm that. ... > > Another indicator that there's something wrong is that GNU parted > > doesn't print the label: oO :-/ ... > > If sgdisk created these partition layouts, then there's clearly a bug > > in sgdisk which makes it write labels characters in big-endian format > > without byteswapping them. Indeed - gdisk exposes the same issue when creating partition labels as the test of gdisk was what initially made me look at this. I guess they share the underlying code anyway. I'm slightly scared as that means the gdisk we have in the field - potentially for quite a while - was labeling the wrong way. I'm somewhat afraid how to handle the disk/images that have already written tables as the fix to the display function will make it show the mangled data. To get a feeling I was checking this through all kinds of past releases and architectures. It seems to only affect s390x, but there all Ubuntu release up until 16.04 are affected the same way. That matches gdisk versions 1.0.1 - 1.0.7 For example: $ lxc exec testkvm-xenial-to -- bash -c "dd if=/dev/zero of=/tmp/mytest bs=1024 count=65536; sgdisk /tmp/mytest -o; sgdisk /tmp/mytest -n 1 -c '1:Linux filesystem' -t 1:8300; dd if=/tmp/mytest bs=$[1024+128] count=1 | xxd -a | grep 't.e.m'" ... 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ Well let us find the root cause and fix first and then think about what we could do to handle past issues any better. Since we don't have any specific delta on top of gdisk I wonder why "our" gdisk would be affected but not the one of RH8.3 that you were testing. > Best regards, > > -Erik > > >>>>> On 8.6.2021 10.12, Christian Ehrhardt wrote: > >>>>>> Hi, > >>>>>> I wanted to let you know that the recent try to fix big endian byte > >>>>>> swapping is actually breaking big endian in 1.0.7. > >>>>>> > >>>>>> This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. > >>>>>> > >>>>>> When you look at just these logs it seems it is just failing for no > >>>>>> apparent reason, > >>>>>> but with -x enabled in gdisk_test.sh one quickly sees there is some > >>>>>> string-mangling happening: > >>>>>> > >>>>>> Number Start (sector) End (sector) Size Code Name > >>>>>> 1 2048 131038 63.0 MiB 8300 > >>>>>> 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 > >>>>>> > >>>>>> Reading the very same file with the old gdisk is good - so likely > >>>>>> the > >>>>>> on-dis content is good as well: > >>>>>> Number Start (sector) End (sector) Size Code Name > >>>>>> 1 2048 131038 63.0 MiB 8300 Linux > >>>>>> filesystem > >>>>>> > >>>>>> This is caused by GPTPart::GetDescription due to this change [3] > >>>>>> in 1.0.7 > >>>>>> > >>>>>> Reverting this fixes change fixes the problem. > >>>>>> > >>>>>> [1]: > >>>>>> https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 > >>>>>> [2]: > >>>>>> https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz > >>>>>> [3]: > >>>>>> https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ > >>>>>> > >>>> > >> > >> -- Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd |
|
From: Christian E. <chr...@ca...> - 2021-06-08 13:00:37
|
On Tue, Jun 8, 2021 at 2:34 PM Christian Ehrhardt <chr...@ca...> wrote: > > ... > > > This was a thought error. I meant most significant byte, not least. :) > > not a problem, the examples were more important anyway > > > > -------- > > > $ dd if=tmp/test1.0.6 bs=$[1024+128] count=1 | xxd -a > ... > > > 00000430: 0000 0000 0000 0000 004c 0069 006e 0075 .........L.i.n.u > > > 00000440: 0078 0020 0066 0069 006c 0065 0073 0079 .x. .f.i.l.e.s.y > > > 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ > > Yes, thanks for the command I was able to confirm that. > > ... > > > > Another indicator that there's something wrong is that GNU parted > > > doesn't print the label: > > oO :-/ > ... > > > > If sgdisk created these partition layouts, then there's clearly a bug > > > in sgdisk which makes it write labels characters in big-endian format > > > without byteswapping them. > > Indeed - gdisk exposes the same issue when creating partition labels > as the test of gdisk was what initially made me look at this. > I guess they share the underlying code anyway. > > I'm slightly scared as that means the gdisk we have in the field - > potentially for quite a while - was labeling the wrong way. > I'm somewhat afraid how to handle the disk/images that have already > written tables as the fix to the display function will make it show > the mangled data. > > To get a feeling I was checking this through all kinds of past > releases and architectures. > It seems to only affect s390x, but there all Ubuntu release up until > 16.04 are affected the same way. > That matches gdisk versions 1.0.1 - 1.0.7 > For example: > > $ lxc exec testkvm-xenial-to -- bash -c "dd if=/dev/zero > of=/tmp/mytest bs=1024 count=65536; sgdisk /tmp/mytest -o; sgdisk > /tmp/mytest -n 1 -c '1:Linux filesystem' -t 1:8300; dd if=/tmp/mytest > bs=$[1024+128] count=1 | xxd -a | grep 't.e.m'" > ... > 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ > > Well let us find the root cause and fix first and then think about > what we could do to handle past issues any better. > Since we don't have any specific delta on top of gdisk I wonder why > "our" gdisk would be affected but not the one of RH8.3 that you were > testing. I didn't have RH8.3, but fedora 34 which also has gdisk 1.0.7 That behaves just as broken as all the Ubuntu systems I've checked. [root@f34-s390x ~]# dd if=/dev/zero of=/tmp/mytest bs=1024 count=65536; sgdisk /tmp/mytest -o; sgdisk /tmp/mytest -n 1 -c '1:Linux filesystem' -t 1:8300; dd if=/tmp/mytest bs=$[1024+128] count=1 | xxd -a | grep 't.e.m' 65536+0 records in 65536+0 records out 67108864 bytes (67 MB, 64 MiB) copied, 0.087469 s, 767 MB/s Creating new GPT entries in memory. Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. Setting name! partNum is 0 Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. 1+0 records in 1+0 records out 1152 bytes (1.2 kB, 1.1 KiB) copied, 3.3789e-05 s, 34.1 MB/s 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ > > Best regards, > > > > -Erik > > > > >>>>> On 8.6.2021 10.12, Christian Ehrhardt wrote: > > >>>>>> Hi, > > >>>>>> I wanted to let you know that the recent try to fix big endian byte > > >>>>>> swapping is actually breaking big endian in 1.0.7. > > >>>>>> > > >>>>>> This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. > > >>>>>> > > >>>>>> When you look at just these logs it seems it is just failing for no > > >>>>>> apparent reason, > > >>>>>> but with -x enabled in gdisk_test.sh one quickly sees there is some > > >>>>>> string-mangling happening: > > >>>>>> > > >>>>>> Number Start (sector) End (sector) Size Code Name > > >>>>>> 1 2048 131038 63.0 MiB 8300 > > >>>>>> 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 > > >>>>>> > > >>>>>> Reading the very same file with the old gdisk is good - so likely > > >>>>>> the > > >>>>>> on-dis content is good as well: > > >>>>>> Number Start (sector) End (sector) Size Code Name > > >>>>>> 1 2048 131038 63.0 MiB 8300 Linux > > >>>>>> filesystem > > >>>>>> > > >>>>>> This is caused by GPTPart::GetDescription due to this change [3] > > >>>>>> in 1.0.7 > > >>>>>> > > >>>>>> Reverting this fixes change fixes the problem. > > >>>>>> > > >>>>>> [1]: > > >>>>>> https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 > > >>>>>> [2]: > > >>>>>> https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz > > >>>>>> [3]: > > >>>>>> https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ > > >>>>>> > > >>>> > > >> > > >> > > > > -- > Christian Ehrhardt > Staff Engineer, Ubuntu Server > Canonical Ltd -- Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd |
|
From: Erik L. <cat...@gm...> - 2021-06-08 13:18:12
|
Hi Christian, On 08-06-2021 15:59, Christian Ehrhardt wrote: > On Tue, Jun 8, 2021 at 2:34 PM Christian Ehrhardt > <chr...@ca...> wrote: >> ... >> >>> This was a thought error. I meant most significant byte, not least. :) >> not a problem, the examples were more important anyway >> >>>> -------- >>>> $ dd if=tmp/test1.0.6 bs=$[1024+128] count=1 | xxd -a >> ... >>>> 00000430: 0000 0000 0000 0000 004c 0069 006e 0075 .........L.i.n.u >>>> 00000440: 0078 0020 0066 0069 006c 0065 0073 0079 .x. .f.i.l.e.s.y >>>> 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ >> Yes, thanks for the command I was able to confirm that. >> >> ... >> >>>> Another indicator that there's something wrong is that GNU parted >>>> doesn't print the label: >> oO :-/ >> ... >> >>>> If sgdisk created these partition layouts, then there's clearly a bug >>>> in sgdisk which makes it write labels characters in big-endian format >>>> without byteswapping them. >> Indeed - gdisk exposes the same issue when creating partition labels >> as the test of gdisk was what initially made me look at this. >> I guess they share the underlying code anyway. >> >> I'm slightly scared as that means the gdisk we have in the field - >> potentially for quite a while - was labeling the wrong way. >> I'm somewhat afraid how to handle the disk/images that have already >> written tables as the fix to the display function will make it show >> the mangled data. >> >> To get a feeling I was checking this through all kinds of past >> releases and architectures. >> It seems to only affect s390x, but there all Ubuntu release up until >> 16.04 are affected the same way. >> That matches gdisk versions 1.0.1 - 1.0.7 >> For example: >> >> $ lxc exec testkvm-xenial-to -- bash -c "dd if=/dev/zero >> of=/tmp/mytest bs=1024 count=65536; sgdisk /tmp/mytest -o; sgdisk >> /tmp/mytest -n 1 -c '1:Linux filesystem' -t 1:8300; dd if=/tmp/mytest >> bs=$[1024+128] count=1 | xxd -a | grep 't.e.m'" >> ... >> 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ >> >> Well let us find the root cause and fix first and then think about >> what we could do to handle past issues any better. >> Since we don't have any specific delta on top of gdisk I wonder why >> "our" gdisk would be affected but not the one of RH8.3 that you were >> testing. > I didn't have RH8.3, but fedora 34 which also has gdisk 1.0.7 > That behaves just as broken as all the Ubuntu systems I've checked. > > [root@f34-s390x ~]# dd if=/dev/zero of=/tmp/mytest bs=1024 > count=65536; sgdisk /tmp/mytest -o; sgdisk /tmp/mytest -n 1 -c > '1:Linux filesystem' -t 1:8300; dd if=/tmp/mytest bs=$[1024+128] > count=1 | xxd -a | grep 't.e.m' > 65536+0 records in > 65536+0 records out > 67108864 bytes (67 MB, 64 MiB) copied, 0.087469 s, 767 MB/s > Creating new GPT entries in memory. > Warning: The kernel is still using the old partition table. > The new table will be used at the next reboot or after you > run partprobe(8) or kpartx(8) > The operation has completed successfully. > Setting name! > partNum is 0 > Warning: The kernel is still using the old partition table. > The new table will be used at the next reboot or after you > run partprobe(8) or kpartx(8) > The operation has completed successfully. > 1+0 records in > 1+0 records out > 1152 bytes (1.2 kB, 1.1 KiB) copied, 3.3789e-05 s, 34.1 MB/s > 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ In my experience the RHEL version of gdisk / sgdisk was just as broken as the Ubuntu ones. I have created a merge request to address the issue with big-endian partition names: https://sourceforge.net/p/gptfdisk/code/merge-requests/26/ Please comment in there and test the state of the pull request to make sure it works for you. We can discuss there what to do about existing volumes. Best regards, - Erik >>>>>>>> On 8.6.2021 10.12, Christian Ehrhardt wrote: >>>>>>>>> Hi, >>>>>>>>> I wanted to let you know that the recent try to fix big endian byte >>>>>>>>> swapping is actually breaking big endian in 1.0.7. >>>>>>>>> >>>>>>>>> This is broken on s390x (=big endian) in Debian [1] and Ubuntu [2]. >>>>>>>>> >>>>>>>>> When you look at just these logs it seems it is just failing for no >>>>>>>>> apparent reason, >>>>>>>>> but with -x enabled in gdisk_test.sh one quickly sees there is some >>>>>>>>> string-mangling happening: >>>>>>>>> >>>>>>>>> Number Start (sector) End (sector) Size Code Name >>>>>>>>> 1 2048 131038 63.0 MiB 8300 >>>>>>>>> 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 >>>>>>>>> >>>>>>>>> Reading the very same file with the old gdisk is good - so likely >>>>>>>>> the >>>>>>>>> on-dis content is good as well: >>>>>>>>> Number Start (sector) End (sector) Size Code Name >>>>>>>>> 1 2048 131038 63.0 MiB 8300 Linux >>>>>>>>> filesystem >>>>>>>>> >>>>>>>>> This is caused by GPTPart::GetDescription due to this change [3] >>>>>>>>> in 1.0.7 >>>>>>>>> >>>>>>>>> Reverting this fixes change fixes the problem. >>>>>>>>> >>>>>>>>> [1]: >>>>>>>>> https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 >>>>>>>>> [2]: >>>>>>>>> https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz >>>>>>>>> [3]: >>>>>>>>> https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ >>>>>>>>> >>>>> >> >> >> -- >> Christian Ehrhardt >> Staff Engineer, Ubuntu Server >> Canonical Ltd > > |
|
From: Christian E. <chr...@ca...> - 2021-06-08 13:38:39
|
... > > 1152 bytes (1.2 kB, 1.1 KiB) copied, 3.3789e-05 s, 34.1 MB/s > > 00000450: 0073 0074 0065 006d 0000 0000 0000 0000 .s.t.e.m........ > > In my experience the RHEL version of gdisk / sgdisk was just as broken > as the Ubuntu ones. > > I have created a merge request to address the issue with big-endian > partition names: > https://sourceforge.net/p/gptfdisk/code/merge-requests/26/ > > Please comment in there and test the state of the pull request to make > sure it works for you. We can discuss there what to do about existing > volumes. I've tested that solution and it works for me. I've commented on that MR and yes we can move the further discussion there. |
|
From: Rod S. <rod...@ro...> - 2021-06-08 14:48:09
|
On 6/8/21 9:17 AM, Erik Larsson wrote: > Hi Christian, > > I have created a merge request to address the issue with big-endian > partition names: > https://sourceforge.net/p/gptfdisk/code/merge-requests/26/ Wow! I woke up to quite the flurry of activity on this. (I'm on the US East Coast.) Anyhow, thanks to both you and Christian for tracking this down and submitting the MR so quickly. I've accepted it, so if subsequent testing shows a need for more changes, please alert me and I'll handle it. There aren't a lot of changes from 1.0.7 to the current state, but given that this is kind of important for big-endian systems, I'll release this change as 1.0.8 soon -- but I'll give it a day or two in case something else crops up. > Please comment in there and test the state of the pull request to make > sure it works for you. We can discuss there what to do about existing > volumes. > > Best regards, > > - Erik > > >>>>>>>>> On 8.6.2021 10.12, Christian Ehrhardt wrote: >>>>>>>>>> Hi, >>>>>>>>>> I wanted to let you know that the recent try to fix big endian >>>>>>>>>> byte >>>>>>>>>> swapping is actually breaking big endian in 1.0.7. >>>>>>>>>> >>>>>>>>>> This is broken on s390x (=big endian) in Debian [1] and Ubuntu >>>>>>>>>> [2]. >>>>>>>>>> >>>>>>>>>> When you look at just these logs it seems it is just failing >>>>>>>>>> for no >>>>>>>>>> apparent reason, >>>>>>>>>> but with -x enabled in gdisk_test.sh one quickly sees there is >>>>>>>>>> some >>>>>>>>>> string-mangling happening: >>>>>>>>>> >>>>>>>>>> Number Start (sector) End (sector) Size Code Name >>>>>>>>>> 1 2048 131038 63.0 MiB 8300 >>>>>>>>>> 䰀椀渀甀砀 昀椀氀攀猀礀猀琀攀洀 >>>>>>>>>> >>>>>>>>>> Reading the very same file with the old gdisk is good - so likely >>>>>>>>>> the >>>>>>>>>> on-dis content is good as well: >>>>>>>>>> Number Start (sector) End (sector) Size Code Name >>>>>>>>>> 1 2048 131038 63.0 MiB 8300 Linux >>>>>>>>>> filesystem >>>>>>>>>> >>>>>>>>>> This is caused by GPTPart::GetDescription due to this change [3] >>>>>>>>>> in 1.0.7 >>>>>>>>>> >>>>>>>>>> Reverting this fixes change fixes the problem. >>>>>>>>>> >>>>>>>>>> [1]: >>>>>>>>>> https://buildd.debian.org/status/fetch.php?pkg=gdisk&arch=s390x&ver=1.0.7-1&stamp=1617791278&raw=0 >>>>>>>>>> >>>>>>>>>> [2]: >>>>>>>>>> https://launchpadlibrarian.net/540613637/buildlog_ubuntu-impish-s390x.gdisk_1.0.7-1_BUILDING.txt.gz >>>>>>>>>> >>>>>>>>>> [3]: >>>>>>>>>> https://sourceforge.net/p/gptfdisk/code/ci/86dd5fea351a5a55bea26b7622eb85ebd6075a60/ >>>>>>>>>> >>>>>>>>>> >>>>>> >>> >>> >>> -- >>> Christian Ehrhardt >>> Staff Engineer, Ubuntu Server >>> Canonical Ltd >> >> > > > _______________________________________________ > Gptfdisk-general mailing list > Gpt...@li... > https://lists.sourceforge.net/lists/listinfo/gptfdisk-general -- Rod Smith rod...@ro... http://www.rodsbooks.com |
|
From: Christian E. <chr...@ca...> - 2021-06-08 14:54:54
|
Hi Rod, > There aren't a lot of changes from 1.0.7 to the current > state, but given that this is kind of important for big-endian systems, > I'll release this change as 1.0.8 soon -- but I'll give it a day or two > in case something else crops up. Agreed, that little wait sounds wise. I've indirectly also pulled IBM into the boat since they are kind of the "master of big-endian things". Maybe (but no promise) they have opinions/hints on this as well. |
|
From: Rod S. <rod...@ro...> - 2021-06-08 15:55:19
|
On 6/8/21 10:54 AM, Christian Ehrhardt wrote: > Hi Rod, > >> There aren't a lot of changes from 1.0.7 to the current >> state, but given that this is kind of important for big-endian systems, >> I'll release this change as 1.0.8 soon -- but I'll give it a day or two >> in case something else crops up. > > Agreed, that little wait sounds wise. > I've indirectly also pulled IBM into the boat since they are kind of > the "master of big-endian things". > Maybe (but no promise) they have opinions/hints on this as well. I've been doing some searching to find a way to identify UCS-2 with swapped bytes, so that the program could fix this automatically when reading corrupted partition tables. (GPT technically uses UCS-2, not UTF-16, although the GPT fdisk code has a lot of references to UTF-16 because at one time I relied on an external UTF-16 library.) So far no luck. -- Rod Smith rod...@ro... http://www.rodsbooks.com |
|
From: Erik L. <cat...@gm...> - 2021-06-08 16:23:09
|
Hi, On 08-06-2021 18:16, Rod Smith wrote: > On 6/8/21 10:54 AM, Christian Ehrhardt wrote: >>> There aren't a lot of changes from 1.0.7 to the current >>> state, but given that this is kind of important for big-endian systems, >>> I'll release this change as 1.0.8 soon -- but I'll give it a day or two >>> in case something else crops up. >> Agreed, that little wait sounds wise. >> I've indirectly also pulled IBM into the boat since they are kind of >> the "master of big-endian things". >> Maybe (but no promise) they have opinions/hints on this as well. > I've been doing some searching to find a way to identify UCS-2 with > swapped bytes, so that the program could fix this automatically when > reading corrupted partition tables. (GPT technically uses UCS-2, not > UTF-16, although the GPT fdisk code has a lot of references to UTF-16 > because at one time I relied on an external UTF-16 library.) So far no luck. This problem is impossible to solve perfectly, however I would think that partition labels with characters outside the ASCII range are very rare in practice so for a start one could check if the first byte is 0 in every UCS-2 byte-pair. If true, then it's almost certain that the label is in big endian (but I would still ask the user to confirm that the big-endian interpretation makes more sense). There are of course also some ranges that are reserved in Unicode that can be used to detect that something might not be right with the endianness and if we're limiting the allowed Unicode range to UCS-2 we can exclude UTF-16 surrogate pair ranges (disallowing for instance the poop emoji as a partition label). One could also check locality (text doesn't usually use characters from all over the place in but stays within a few defined script ranges) and there may also be statistical models based on the frequency that Unicode characters occur in text, but that's probably taking it too far. Best regards, - Erik |
|
From: Rod S. <rod...@ro...> - 2021-06-08 17:16:28
|
On 6/8/21 12:21 PM, Erik Larsson wrote: > Hi, > > On 08-06-2021 18:16, Rod Smith wrote: >> On 6/8/21 10:54 AM, Christian Ehrhardt wrote: >>>> There aren't a lot of changes from 1.0.7 to the current >>>> state, but given that this is kind of important for big-endian systems, >>>> I'll release this change as 1.0.8 soon -- but I'll give it a day or two >>>> in case something else crops up. >>> Agreed, that little wait sounds wise. >>> I've indirectly also pulled IBM into the boat since they are kind of >>> the "master of big-endian things". >>> Maybe (but no promise) they have opinions/hints on this as well. >> I've been doing some searching to find a way to identify UCS-2 with >> swapped bytes, so that the program could fix this automatically when >> reading corrupted partition tables. (GPT technically uses UCS-2, not >> UTF-16, although the GPT fdisk code has a lot of references to UTF-16 >> because at one time I relied on an external UTF-16 library.) So far no >> luck. > > > This problem is impossible to solve perfectly, however I would think > that partition labels with characters outside the ASCII range are very > rare in practice so for a start one could check if the first byte is 0 > in every UCS-2 byte-pair. > If true, then it's almost certain that the label is in big endian (but I > would still ask the user to confirm that the big-endian interpretation > makes more sense). Rather than implement something that stands a chance of getting things badly wrong, I'm leaning toward either doing nothing so that users can manually change partition labels, if desired; or implement an option to let users explicitly select partitions to byte-swap their labels. Either approach enables users to fix problems as they're detected, with the second option providing some help to users in recovering the original intended partition labels. Because most disk tools don't use partition labels for anything important, I don't think it's critical that these problems be corrected automatically. OTOH, automatically "fixing" something that's not broken, because of an algorithm that's prone to false alarms, could be really annoying. > There are of course also some ranges that are reserved in Unicode that > can be used to detect that something might not be right with the > endianness and if we're limiting the allowed Unicode range to UCS-2 we > can exclude UTF-16 surrogate pair ranges (disallowing for instance the > poop emoji as a partition label). > > One could also check locality (text doesn't usually use characters from > all over the place in but stays within a few defined script ranges) and > there may also be statistical models based on the frequency that Unicode > characters occur in text, but that's probably taking it too far. :) > > Best regards, > > - Erik -- Rod Smith rod...@ro... http://www.rodsbooks.com |
|
From: Rod S. <rod...@ro...> - 2021-06-10 00:47:17
|
This is a heads-up that I've released version 1.0.8, which fixes this bug. It also adds a new feature to gdisk and sgdisk that enables correcting a corrupted partition name. To use this feature: * In gdisk, enter the experts' menu and then type 'b'. This will prompt you for a partition number (unless the disk has just one partition), shows the original and transformed names, and asks for verification. After you've fixed all the broken partition names, type 'w' to write your changes and exit. * In sgdisk, use the new -B/--byte-swap-name parameter, which takes a partition number as an option. If valid, changes will be immediately written to disk. This new feature works on any CPU architecture, but it probably won't be necessary on x86 or x86-64. The bug doesn't affect little-endian CPUs like x86 or x86-64, so the bug fix has no effect on them. GPT fdisk 1.0.8 also adds a new partition type code for the Barebox boot loader. On 6/8/21 1:16 PM, Rod Smith wrote: > On 6/8/21 12:21 PM, Erik Larsson wrote: >> Hi, >> >> On 08-06-2021 18:16, Rod Smith wrote: >>> On 6/8/21 10:54 AM, Christian Ehrhardt wrote: >>>>> There aren't a lot of changes from 1.0.7 to the current >>>>> state, but given that this is kind of important for big-endian systems, >>>>> I'll release this change as 1.0.8 soon -- but I'll give it a day or two >>>>> in case something else crops up. >>>> Agreed, that little wait sounds wise. >>>> I've indirectly also pulled IBM into the boat since they are kind of >>>> the "master of big-endian things". >>>> Maybe (but no promise) they have opinions/hints on this as well. >>> I've been doing some searching to find a way to identify UCS-2 with >>> swapped bytes, so that the program could fix this automatically when >>> reading corrupted partition tables. (GPT technically uses UCS-2, not >>> UTF-16, although the GPT fdisk code has a lot of references to UTF-16 >>> because at one time I relied on an external UTF-16 library.) So far no >>> luck. >> >> >> This problem is impossible to solve perfectly, however I would think >> that partition labels with characters outside the ASCII range are very >> rare in practice so for a start one could check if the first byte is 0 >> in every UCS-2 byte-pair. >> If true, then it's almost certain that the label is in big endian (but I >> would still ask the user to confirm that the big-endian interpretation >> makes more sense). > > Rather than implement something that stands a chance of getting things > badly wrong, I'm leaning toward either doing nothing so that users can > manually change partition labels, if desired; or implement an option to > let users explicitly select partitions to byte-swap their labels. Either > approach enables users to fix problems as they're detected, with the > second option providing some help to users in recovering the original > intended partition labels. Because most disk tools don't use partition > labels for anything important, I don't think it's critical that these > problems be corrected automatically. OTOH, automatically "fixing" > something that's not broken, because of an algorithm that's prone to > false alarms, could be really annoying. > >> There are of course also some ranges that are reserved in Unicode that >> can be used to detect that something might not be right with the >> endianness and if we're limiting the allowed Unicode range to UCS-2 we >> can exclude UTF-16 surrogate pair ranges (disallowing for instance the >> poop emoji as a partition label). >> >> One could also check locality (text doesn't usually use characters from >> all over the place in but stays within a few defined script ranges) and >> there may also be statistical models based on the frequency that Unicode >> characters occur in text, but that's probably taking it too far. :) >> >> Best regards, >> >> - Erik > > -- Rod Smith rod...@ro... http://www.rodsbooks.com |
Thanks for helping keep SourceForge clean.
X