freemint-discuss — The official FreeMiNT Mailing List

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

On Montag, 3. Juni 2024 10:24:59 CEST Eero Tamminen wrote:
> I would suggest using Hatari for debugging these.

Maybe next time ;) The problem has been identified and solved.

But in general it is difficult to use Hatari in such scenarios. First off the 
gcc toolchain requires MiNT, and then Hatari is much slower than aranym, even 
when comparing the non-jit version of aranym to Hatari using fast-forward.

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Eero T. <oa...@he...>]

Hi,

On 2.6.2024 15.20, Thorsten Otto wrote:
> - conclusion 2: we need to find some way to test the compiler. Compiling it by
> itself would be a first step, but was not enough in this case (the cross-
> compiler could compile itself, but generates wrong code that caused only
> trouble when running the m68k executable). Obviously there is no easy way to
> test the native compiler, maybe qemu or aranym can help here.

AFAIK:

* System qemu does not support Atari machines
* While m68k user-qemu is very much used by Debian m68k port, it 
supports only Linux kernel, so it would not be much use for Atari testing


Hatari could be used for testing though, although its slower than 
Aranym.  Both offer NatFeats APIs, and Hatari supports also remote 
control, either through given "--cmd-fifo", or "--control-socket".


	- Eero

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Eero T. <oa...@he...>]

Hi,

On 30.5.2024 12.34, Thorsten Otto wrote:
> I've already tried to debug this using gdb, but that's almost impossible: it
> is the cc1 compiler that crashes, and when compiled with debug info, that
> makes ~287MB, and even on aranym it takes about an hour to load it.

I would suggest using Hatari for debugging these.

Extract (relevant set of) demangled C++ symbols from the binary to a 
separate file, load those to Hatari debugger on program startup, enable 
profiling & debugger invocation on exceptions, and you'll get backtrace 
when the program crashes.

At least that works fine with ScummVM, another large C++ program...


	- Eero

Re: [Freemint-discuss] Debugging a problem in gcc-14

[WongCK <won...@ya...>]

 Great work !!Hope you still have sometime left in your weekend to enjoy.... in some other way.
    On Sunday, 2 June 2024 at 08:21:19 pm SGT, Thorsten Otto <ad...@th...> wrote:  
 
 
Phew, i think i found it. Seems the bug was introduced by


https://github.com/th-otto/m68k-atari-mint-gcc/commit/04c9cf5c786b94fbe3f6f21f06cae73a7575ff7a


which implements a new optimization pass. Luckily, that optimization can be turned off by -ffold-mem-offsets. This is only in gcc-14.1.0, not in gcc-13.3.0 (but may eventually appear in the next release).


It also explains why that bug only crashed the native compiler, not the cross-compiler. It has however the consequence that the als the cross-compiler generates wrong code in certain cases (eg. when compiling itself).


In this case, the code around https://github.com/th-otto/m68k-atari-mint-gcc/blob/c9e3ffbc4ef9bcaaee9cbaeeeeed75df8ca459ea/gcc/tree-switch-conversion.cc#L1560-L1569


Was compiled into something that caused the assignment test[k].target_bb = n->m_case_bb to be miscompiled, and the outer loop would not find it, making new entries until the array overflows.


- conclusion 1: we should think about adding --enable-checking=misc to our build scripts. That adds some assert checking, making the compiler slightly slower, but would catch such overflows which may or may not cause a bus-error.


- conclusion 2: we need to find some way to test the compiler. Compiling it by itself would be a first step, but was not enough in this case (the cross-compiler could compile itself, but generates wrong code that caused only trouble when running the m68k executable). Obviously there is no easy way to test the native compiler, maybe qemu or aranym can help here.


I'm currently doing some last tests, then recompiling all gcc-14 versions. Keep fingers crossed.



_______________________________________________
Freemint-discuss mailing list
Fre...@li...
https://lists.sourceforge.net/lists/listinfo/freemint-discuss
  

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

Phew, i think i found it. Seems the bug was introduced by

https://github.com/th-otto/m68k-atari-mint-gcc/commit/
04c9cf5c786b94fbe3f6f21f06cae73a7575ff7a

which implements a new optimization pass. Luckily, that optimization can be 
turned off by -ffold-mem-offsets. This is only in gcc-14.1.0, not in 
gcc-13.3.0 (but may eventually appear in the next release).

It also explains why that bug only crashed the native compiler, not the cross-
compiler. It has however the consequence that the als the cross-compiler 
generates wrong code in certain cases (eg. when compiling itself).

In this case, the code around https://github.com/th-otto/m68k-atari-mint-gcc/
blob/c9e3ffbc4ef9bcaaee9cbaeeeeed75df8ca459ea/gcc/tree-switch-
conversion.cc#L1560-L1569

Was compiled into something that caused the assignment test[k].target_bb = n-
>m_case_bb to be miscompiled, and the outer loop would not find it, making new 
entries until the array overflows.

- conclusion 1: we should think about adding --enable-checking=misc to our 
build scripts. That adds some assert checking, making the compiler slightly 
slower, but would catch such overflows which may or may not cause a bus-error.

- conclusion 2: we need to find some way to test the compiler. Compiling it by 
itself would be a first step, but was not enough in this case (the cross-
compiler could compile itself, but generates wrong code that caused only 
trouble when running the m68k executable). Obviously there is no easy way to 
test the native compiler, maybe qemu or aranym can help here.

I'm currently doing some last tests, then recompiling all gcc-14 versions. 
Keep fingers crossed.

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

On Samstag, 1. Juni 2024 10:05:05 CEST WongCK via Freemint-discuss wrote:
> Sorry to have caused you so much work all of a sudden.

No worries. Maybe i just didn't expect that there are brave people like you 
that actual use gcc on larger projects on real hardware ;)

Re: [Freemint-discuss] Debugging a problem in gcc-14

[WongCK <won...@ya...>]

 Thanks Thorsten for looking into this.Sorry to have caused you so much work all of a sudden.

    On Friday, 31 May 2024 at 09:26:29 pm SGT, Thorsten Otto <ad...@th...> wrote:  
 
 
Just a note: the original executable crashed with a bus-error. While bisecting, i configured both compilers with --enable-checking=yes,types,extra,misc


That already changed the compiler to abort with an assertion instead, at


https://github.com/th-otto/m68k-atari-mint-gcc/blob/d1a1f7e9f0bedea55c558ab95127679bc3e9ff72/gcc/tree-switch-conversion.cc#L1637


That makes sense, since both test cases are switch statements, and if the assertion is disabled, the loop will write beyond the end of the test[] array and corrupt the stack.

_______________________________________________
Freemint-discuss mailing list
Fre...@li...
https://lists.sourceforge.net/lists/listinfo/freemint-discuss
  

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

Just a note: the original executable crashed with a bus-error. While 
bisecting, i configured both compilers with --enable-
checking=yes,types,extra,misc

That already changed the compiler to abort with an assertion instead, at

https://github.com/th-otto/m68k-atari-mint-gcc/blob/
d1a1f7e9f0bedea55c558ab95127679bc3e9ff72/gcc/tree-switch-conversion.cc#L1637

That makes sense, since both test cases are switch statements, and if the 
assertion is disabled, the loop will write beyond the end of the test[] array 
and corrupt the stack.

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

On Donnerstag, 30. Mai 2024 20:28:52 CEST Miro Kropáček wrote:
>I  have quickly verified that it all works without conflicts.

Thanks, thats works indeed better. I still get a few conflicts, but only small 
ones. But it will still take some time, because i have to recompile both 
compilers after each checkout, therefor it takes ~30min before i can tell 
whether a revision is "bad" or "good". And there are about 11000 commits 
between basepoints/gcc-14 and master...

But the good news: the crash is still reproducible with gcc 15.0.0 (the 
version on the master branch).

Bisecting: 5527 revisions left to test after this (roughly 13 steps)

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Miro K. <mir...@gm...>]

On Thu, 30 May 2024 at 14:25, Thorsten Otto <ad...@th...> wrote:

> On Donnerstag, 30. Mai 2024 13:02:03 CEST Miro Kropáček wrote:
>
> Hm, cannot agree:
>
I think you are still doing it wrong (or at least in a different way than I
meant). The most trivial way to do it:

- checkout https://github.com/gcc-mirror/gcc/tree/basepoints/gcc-14 --
that's the tag where gcc-14 development started. That means that at this
point it should still work for you (if not, do the same for
basepoints/gcc-13 and start from there)

- cherry-pick m68k-atari-mintelf commit(s) to this branch and call it e.g.
gcc-14-start

- checkout master into a separate branch, e.g. gcc-master-head

- rebase gcc-master-head on top of gcc-14-start

Result: you can start bisecting at gcc-14-start (good) and gcc-master-head
(bad) while your m68k-mintelf changes are there for all commits.

I have quickly verified that it all works without conflicts.

-- 
http://mikro.atari.org

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

On Donnerstag, 30. Mai 2024 11:34:51 CEST Thorsten Otto wrote:
> - but it crashes with a bus-error (mint's idea of a segmentation fault) when
> using the native gcc 14.1.0 compiler.

Another test done: normally, i compile the native compiler by the same version 
of its cross-compiler. Now the strange thing: native gcc 14.1.0 works, when i 
compile it by gcc 13.3.0 instead. WTF?

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

On Donnerstag, 30. Mai 2024 15:35:47 CEST Mark Duckworth via Freemint-discuss 
wrote:
> Your binary's stack size is sufficient?

Yes, thats always the first thing i try when i get strange errors ;)

And the bug can be reproduced by a simple program like:

int x509_date_is_valid(int mon)
{
    switch (mon) {
        case 1:
        case 3:
        case 5:
        case 7:
        case 8:
        case 10:
        case 12:
            break;
        default:
            return 0x2400;
    }

    return 0;
}

It crashes in GIMPLE: pass switchlower, regardless of optimizations used.

There is a 2nd simple test that also crashes, also involving a switch 
statement:

int mbedtls_to_psa_error(int ret)
{
    switch (ret) {
        case -0x0020:
            return (-134);
        case -0x0060:
        case -0x0062:
        case -0x0064:
        case -0x0066:
        case -0x0068:
            return (-135);
        case -0x006A:
            return (-141);
    }
    return (-132);
}

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

On Donnerstag, 30. Mai 2024 14:24:36 CEST Thorsten Otto wrote:

Doing it the other way round is even worse:

Last commands done (43 commands done):
   pick b153f4e4cae libstdc++: Adjust uses of null pointer constants in docs
   pick d5cd3eada41 Daily bump.
  (see more in file .git/rebase-merge/done)
Next commands to do (9242 remaining commands):

Both branches (https://github.com/th-otto/m68k-atari-mint-gcc/tree/mint/gcc-13 
and https://github.com/th-otto/m68k-atari-mint-gcc/tree/mint/gcc-14 ) are up-
to-date. Could you please give it a quick try, and tell me what i'm doing 
wrong?

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Mark D. <mdu...@at...>]

On 5/30/24 5:34 AM, Thorsten Otto wrote:
> Hi,
> 
> 
> i'm currently trying to diagnose a problem in gcc-14. The symptom is that
> 
> 
> - it works when using the gcc-13.3.0 cross-compiler
> 
> - it works when using the gcc-14.1.0 cross-compiler
> 
> - it works when using the native gcc-13.3.0 compiler on aranym
> 
> - but it crashes with a bus-error (mint's idea of a segmentation fault) 
> when using the native gcc 14.1.0 compiler.
> 
> 

This is super obvious but I've forgotten before.  Your binary's stack 
size is sufficient?

Mark

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

On Donnerstag, 30. Mai 2024 13:02:03 CEST Miro Kropáček wrote:
> Certainly faster than any other alternative.

Hm, cannot agree:

interactive rebase in progress; onto 57888ce604c
Last commands done (157 commands done):
   pick 2b4ca304b16 Daily bump.
   pick d96e4962f53 libstdc++: Implement P2538R1 ADL-proof std::projected
  (see more in file .git/rebase-merge/done)
Next commands to do (554 remaining commands):

And there are still lots of merge conflicts. And those "Daily bumps" are 
driving me mad.

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Miro K. <mir...@gm...>]

On Thu, 30 May 2024 at 12:43, Thorsten Otto <ad...@th...> wrote:

> I think that wont work. Both the mint/gcc-13 branch (which is now at
> gcc-13.3.0) and the mint/gcc-14 branch have common commits from the master
> branch, but with different ids.
>
> I've already tried to create a new branch starting from mint/gcc-13, then
> merge mint/gcc-14 into it. But that gave several hundred merge conflicts.
>
> Or what exact would you try to do?
>
Make it even simpler. :-) It goes both ways: everything in gcc-13 is surely
committed to gcc-14. Assuming that gcc-14 (then-master) wasn't just
horribly broken every second commit you can basically just choose a point
in history (e.g. the fork of master and gcc-13 branch but can be something
later, too) where do you apply our patches and then do the rebase of the
rest of gcc-14.

There's a slight chance you will bump into missing some last-minute fixes
(so you will have to move the rebase to a later commit) but that's easy to
detect and not so probable. Certainly faster than any other alternative.

-- 
http://mikro.atari.org

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

On Donnerstag, 30. Mai 2024 12:32:58 CEST Miro Kropáček wrote:
> Just rebase gcc-14 on top of gcc-13-mint

I think that wont work. Both the mint/gcc-13 branch (which is now at 
gcc-13.3.0) and the mint/gcc-14 branch have common commits from the master 
branch, but with different ids.

I've already tried to create a new branch starting from mint/gcc-13, then 
merge mint/gcc-14 into it. But that gave several hundred merge conflicts.

Or what exact would you try to do?

Re: [Freemint-discuss] Debugging a problem in gcc-14

[Miro K. <mir...@gm...>]

On Thu, 30 May 2024 at 11:35, Thorsten Otto <ad...@th...> wrote:

> So i'm thinking of trying to bisect this, since it obviously is caused by
> some change between 13.3.0 and 14.1.0. But how can i do that? I cannot use
> the master branch, because it does not contain our port changes. And our
> ports of gcc-13 and gcc-14 are on different branches. So how can i bisect
> between different branches?
>
Why complicate your life so much? Just rebase gcc-14 on top of gcc-13-mint
and you're free to bisect from the point of commit "introduce
m68k-atari-mintelf" to latest gcc-14 commit (which is now present on top of
gcc-13-mint)

-- 
http://mikro.atari.org

[Freemint-discuss] Debugging a problem in gcc-14

[Thorsten O. <ad...@th...>]

Hi,

i'm currently trying to diagnose a problem in gcc-14. The symptom is that

- it works when using the gcc-13.3.0 cross-compiler
- it works when using the gcc-14.1.0 cross-compiler
- it works when using the native gcc-13.3.0 compiler on aranym
- but it crashes with a bus-error (mint's idea of a segmentation fault) when 
using the native gcc 14.1.0 compiler.

Its not a problem of aranym, the same fault was also reported when run on real 
hardware.

I've already tried to debug this using gdb, but that's almost impossible: it 
is the cc1 compiler that crashes, and when compiled with debug info, that 
makes ~287MB, and even on aranym it takes about an hour to load it.

So i'm thinking of trying to bisect this, since it obviously is caused by some 
change between 13.3.0 and 14.1.0. But how can i do that? I cannot use the 
master branch, because it does not contain our port changes. And our ports of 
gcc-13 and gcc-14 are on different branches. So how can i bisect between 
different branches?

Re: [Freemint-discuss] Updating compilers for snapshot builds

[Thorsten O. <ad...@th...>]

On Dienstag, 30. April 2024 20:26:15 CEST Miro Kropáček wrote:
> You have forgotten to update the download links in library READMEs 

Ah yes, thanks for reminding.

>Also, as freemint + the tools (Qed, Hypview etc) are still being built with 
m68k-atari-mint-gcc, what was your primary motivation for this change?

Mainly to provide the libraries. They will be required should we decide to use 
the new elf toolchain for the other apps. And switching that is now as easy 
changing the cross_tool name in the build.yml file.

>Btw, maybe it would be better to put the flavours into separate folders, i.e. 
>snapshots/fdlibm/{mint, mintelf, mintelf-fastcall} 

-fastcall isn't a separate archive, only -mint and -mintelf are. But i'll 
think about that.

Re: [Freemint-discuss] latest mintlef failing to build with mintlib

[Miro K. <mir...@gm...>]

On Tue, 30 Apr 2024 at 05:22, Thorsten Otto <ad...@th...> wrote:

> More distracting than before? And its not only the .deps directory that is
> created there, if you run "make check", the test executables are also build
> there.
>
I'm referring only to the asymmetry: the 68000 build has those empty
folders, others don't. It may even look like the other builds didn't
compile properly.

That would require to completely reorganize checkrules, since it currently
> only builds for a single architecture. Or maybe completely get rid of it,
> since it duplicates a lot of things from buildrules. But one thing after
> the other.
>
Maybe for the time being at least include some help into the 'tz' folder,
i.e. how to build tzinit etc for other flavours.

We could build them for a.out too of course, but they would be rather
> useless without a gdb that can handle a.out binaries. I just thought that
> it would be useful to have them build for elf where they actually can be
> used. But of course we can disable that again, to build other apps like qed
> they are not needed.
>
Ah right, the gdb. Ok, that's a fair point although I have never used lib_g
for anything so that's why I asked.

-- 
http://mikro.atari.org

Re: [Freemint-discuss] Updating compilers for snapshot builds

[Miro K. <mir...@gm...>]

On Tue, 30 Apr 2024 at 11:27, Thorsten Otto <ad...@th...> wrote:

> Everything should be in good shape again.
>
> Note that all libraries (except libcmini, until the patches are accepted)
> are now also compiled for mintelf toolchains.
>
Nice, thanks for the tedious work being done.

You have forgotten to update the download links in library READMEs -- they
still point to -latest.tar.bz2 instead of the mint/mintelf filename.

Also, as freemint + the tools (Qed, Hypview etc) are still being built with
m68k-atari-mint-gcc, what was your primary motivation for this change? Not
that I complain. :)

Btw, maybe it would be better to put the flavours into separate folders,
i.e. snapshots/fdlibm/{mint, mintelf, mintelf-fastcall} so the directories
wont become too cluttered. That way one can always use the same archive
name, just with a different path prefix.

-- 
http://mikro.atari.org

Re: [Freemint-discuss] freemint build failing due to libcmini

[Thorsten O. <ad...@th...>]

On Montag, 29. April 2024 23:35:37 CEST Miro Kropáček wrote:
> Thorsten could this be related to your recent mintlib changes?

Could have been some problem when fetching the mintlib, but should work now 
again (the archive now includes the /usr directory).

libcmini's usage of using include <osbind.h> is still wrong though, since such 
a file does not exist. It only works in our builds because both mintlib and 
libcmini are installed.

Re: [Freemint-discuss] Updating compilers for snapshot builds

[Thorsten O. <ad...@th...>]

On Dienstag, 30. April 2024 07:38:15 CEST Thorsten Otto wrote:
> Please be patient, i hope to have all issues fixed soon.

Everything should be in good shape again.

Note that all libraries (except libcmini, until the patches are accepted) are 
now also compiled for mintelf toolchains.

For the few Makefiles were it was missing, the variable DESTDIR is now used to 
specify the installation directory (not PREFIX). This is needed to install 
files into directories like /sbin, /etc, /stguide etc. Consequently, libraries 
that should go into the compilers sys-root directory are now installed to /
usr/${CROSS_TOOL}/sys-root  (not /usr/m68k-atari-mint/sys-root/usr as it was 
before).

[Freemint-discuss] Updating compilers for snapshot builds

[Thorsten O. <ad...@th...>]

Hi,

i'm currently updating the compilers for the snapshot builds. This also 
involves supporting mintelf toolschains for the libraries, and slightly 
modified scripts to fetch the toolchains.

Should you push to any repo, there could be build failures because of this 
until all scripts have been updated.

Please be patient, i hope to have all issues fixed soon.