#896 [z80] Let sdasz80 export 32 bits addresses
The sdasz80 assembler loses the 8 most significant bits of the 32-bit addresses when generating the .REL file. To me this looks like a bug, but maybe it's intentional for some reason I don't know so I'm filing a feature request rather than a bug report.
As explained on this thread, the entire compilation process is compatible with 32-bit addresses (necessary, for example, with ROM mappers with more than 256 segments), except for writing the final .REL file, which is done in 24 bits (and so loses the 8 most significant bits).
The problem is that the number of bytes in the output format is hardcoded in sdasz80 project to 3 (3 bytes = 24 bits), whereas it would have to be 4 to export the .REL file in 32-bit format. In the sdasz80 project, this number is stored in the a_bytes variable and hardcoded to 3 in the minit() function in the z80mch.c file (line 2148). You should either change this value to 4 or, probably better, add an option in sdasz80 (transferable via SDCC) to activate 32-bit mode.
Note that the 8 most significant bits of 32-bit addresses are actually present in .MAP, .NOI and .LK files. However, .SYM files are also 24-bit. From what I understand of the code, allowing a_bytes to be set to 4 should solve the problem for both .REL and .SYM files.
I do not quite understand this request. SDCC does not support > 24 bit banked calls either.
And SDCC does not even support any form of > 16 bit data pointers on top.
Are you using your own home grown 32 bit data pointers maybe?
I develop on MSX where the Z80 memory space is only 16 bits. All the extra addressing bits are only used for ROM mappers to determine the segment number in which static data or compiled code is located. These mappers allow to have a large number of segments, and have mechanisms for determining which segment(s) is/are visible at any given time. So the Z80 can see several hundred kilobytes (or even megabytes) through windows (banks) in its 16-bit memory space (64 KB).
With current support for 24-bit addresses, this gives us 8 bits for segment numbers and therefore a maximum of 256 segments. Conventional mappers use segment sizes of 8 or 16 KB, allowing ROMs of up to 2 or 4 MB.
We're currently working on mapped ROMs that could be up to 32 MB in size, requiring 12 bits for segment numbers.
This is why we need sdasz80 to be able to generate a 32-bit IHX, so as not to lose the 4 high-order bits we need.
Given that sdasz80 generates .MAP files with the correct 32-bit addresses, I imagine that it handles this address size well and that it would therefore suffice to be able to set
a_bytesto 4 for sdasz80 to generate an IHX with untruncated addresses.Does it make sense now?
I understand your request. And what I take from it is that you do not (yet) intend to use the C language to use that memory, but program only in assembler.
My programs are entirely in C. Some functions are in inline assembler for performance reasons, but always wrapped in C functions.
In fact, the Z80 only sees 16 bits, so whether addresses are 24 or 32 bits, it makes no difference (only the lower 16 bits are used for the labels).
All the rest of the address is only used for the bank's segment switching mechanism, which is full device-specific (at least on MSX computers).
Perhaps you are referring to the automatic bank switching mechanism via the
__bankedfunction directive? I don't use this system because it's much safer and more efficient to switch banks manually when needed. And this doesn't at all prevent the use of C data or code in segments. The segment number (whether 8 bits value with a 24-bit address, or 16 bits value with a 32-bit address) has no impact on the C label addresses. The only requirement is that the contents of the segments be compiled at the address (lower 16 bits) where they will be visible to the Z80 once placed in one of the banks at run-time.I hope it's clearer now.
And happy holidays :)
Last edit: Aoineko 2023-12-28
There's one point I didn't explain well: the problem lies in the linker, not the compiler. It's "normal" for the Z80 compiler, which works in 16 bits address space, to generate REL files in 16 bits (currently 24, but obviously bits 16 to 23 are always 0).
The real problem is that the linker seems to use the REL file format generated by the compiler (in the
link_main()function oflkmain.c) to determine a mask on the addresses. At this point, the 24-bit format of the REL file seems to means that the 8 most significant bits of the final relocated addresses (those in segments greater than 256) are lost. I don't know if it's normal for the linker to use the REL file format.So, either the REL files should be 32-bit (so that the linker can generate 32-bit final addresses), or the linker should be able to generate 32-bit final addresses, even if the REL files are in 24 (or 16) bit format.
In my test case, I have 2 segments:
But final address will be:
As a test, I recompiled sdasz80 by replacing
exprmasks(3)withexprmasks(4)at line 2148 of thez80mch.cfile and this solves all my problems.Now the compiler generates 32-bit REL files and the linker generates 32-bit IHX files too (no more loss of the high order 8-bit of virtual addresses).
I'd like to avoid having to distribute a modified version of SDCC with my MSXgl library, so I'd have to see if it's possible to apply this change to the basic version.
Perhaps the best thing would be to add an option on the compiler to choose between 24-bit and 32-bit addresses (in any case, both are specific to the device used, at least on MSX computers).
Last edit: Aoineko 2024-01-02
Hi SDCC team!
I've been using for over 2 weeks a patched version of sdasz80 that keeps 32-bit virtual addresses rather than reducing them to 24-bit when exporting IHX and everything seems to be working as it should (and it allows me to use devices with more than 256 segments).
I've tested all samples of my library and several projects from other developpers.
Having said that, it bugs me to use a patched version rather than the official one.
How can we integrate this feature into SDCC?
So that the default behavior doesn't change (although I'm still having trouble understanding the point of removing the 8-bit most significant bits) would be to add an option to activate 32-bit mode.
To stay in the style of other sdas options, we could add something like a
-hoption to keep the highest bits (32-bit) which would be FALSE by default (so 24-bit).Then, in the
minit()function inz80mch.c, we'd simply have to replace :... by ...
Does that sound right to you?
If so, I could propose a push request.
If not, what do you suggest?
Do we need the option? Would just always using 32 bits break anything for anyone?
Using 32 bits don't break anything for me at least (it solve issue). ^^
I don't know if they where any good reason in a given context to use 24 bits instead of 32.
But, with an option - that can be 24 bits per default - we are sure to don't break anything.
Last edit: Aoineko 2024-01-20
There is one case that some may find problematic, and that's the case of banked functions.
Currently, when a
__bankedfunction is called, the compiler calls trampoline functions that the user must supply:-
get_bank: returns the current bank in register A,-
set_bank: sets the bank given by register A.This system even works with 32-bit virtual addresses, but using the A register means that only the first 8 bits can be used for the bank.
So it wouldn't make any difference to existing programs, although some might like trampolines to be 16-bit compatible too.
Personally, I find the usefulness of trampoline functions very limited, and if I use them in one of my sample programs to show how they work, I've never had a real case where it seemed like a good idea to use them.
Especially since on MSX, for example, all mapper systems operate on several pages (usually 2 or 4), and without sending this information to the trampoline functions, there's no way of knowing which page to change the bank to. In my implementation for my MSXgl library, I chose fixed pages depending on the mapper selected by the user, but this still limits the interest of banked functions.
In any case, it would be a real shame if this non-standard system prevented us from having 32-bit virtual addresses and being able to manage more than 256 segments.
Last edit: Aoineko 2024-01-20
get_bank()/set_bank() is just library stuff. library can be rewritten whatever how, for example, gbdk-2020 has no such functions at all because that is EXTREMELY SLOW, it is all inlined. what is really important is 3 things:
1. amount of bytes reserved on stack when making banked call
2. linker address width
3. calling conventions
there is also performance/size tradeoff, 8-bit systems are limited and it is simply not practical to use 16-bit segments because of lacking registers, slower loads/performance penalties, etc. though 8-bit segment and thus 24-bit addressing is a good solution because 4 megabyte program (assuming 16K page system) is a huge amount.
anyway, systems of the past had support for the memory models: "TINY", "SMALL" and "HUGE". that meant THREE DIFFERENT LIBRARIES. of course that is possible, but the current library is very far from being optimal, maintaining three would be a disaster.
Last edit: Tony Pavlov 2024-01-24
Seems like this change might temporarily break GBDK-2020's parsing of .rel files for z80 platforms until we add XL4 support. We parse them in our automatic bank packer (which allows the user to set source files for automatic, most efficient bank assignment).
If it's just a matter of parsing an addition byte of address space for XL4 then it seems workable. Does the change alter the header for the rel file from
XL3toXL4?Yes. The REL file header is changed accordingly.
In [r14639], in the next branch,
exprmasks(4);is used. The next branch will be merged to trunk after the 4.4.0 release. That should give plenty of time to test this before a possible SDCC 4.5.0 release next year.Related
Commit: [r14639]
I am very disappointed. You just broke the GBDK-2020 compatibility for the SMS, GameGear and MSX of the GBDK-2020 kit, just based on the request of the only user. We even did not have the chance to see this discussion!
Banking support indeed has problems with consistency, and not very well thought from the point of performance across the targets as well, but such random decisions make situation even worse, not better.
Honestly, the method of picking patches into SDCC disappoints me. There are very important production-ready and tested patches pending for YEARS, but the other very dubious breaking changes just merged without any more or less broad discussion.
Not sure this message will be noticed not to say taken in account though.
Last edit: Tony Pavlov 2024-01-24
No chance? This discussion was opened 4 weeks ago!
Further, you state something got broken, but do not specify how. So we also cannot fix it other than by plainly reverting this.
And since this is only in the next branch, it is not even regression tested by our own compile farm. And this ticket is pending, not closed. Your contribution will certainly be taken into account.
yes, because it is lightspeed for this project. we are not monitoring this tracker 24/7.
yes, this change is breaking our bankpack tool, though it is our problem. but what for? for some non-existent in the wild life mapper for the MSX? i understand if it was the part of z80 library and banked ABI overhaul, but not!
you say "linker stuff, nobody cares for the linker internal formats". well, we might not, if this patch, back from 2021, was applied: https://sourceforge.net/p/sdcc/patches/396/ but no.
Last edit: Tony Pavlov 2024-01-24
If you were referring to [patches:#396] when you mentioned “very important production-ready and tested patches pending for YEARS”, it is however not exactly helpful that that patch is still explicitly described as “just preliminary” in the patch ticket.
If it is indeed well tested, production-ready and actually already used in production, that piece of information did not make it here.
Related
Patches: #396
Last edit: Benedikt Freisen 2024-01-24
The change is in the next branch now. After the 4.4.0 release it will be merged to trunk (unless serious problems come up). Then it will be in trunk (and development snapshots). SDCC 4.5.0 will most likely be released sometime 2025.
So we have one year to find problems caused by this change (or adapt other tools, such as gbdk to the change), before the change goes into a SDCC release.
I am aware that making any change to banking stuff has a risk of breaking things for users. While I have some rough ideas about how some of the banking stuff works hardware-wise, neither me nor SDCC will ever know about all the mappers out there, or how users use them.
Any nontrivial change I might make to the banking stuff most likely will be seen as bad by some users. So I tend to be very reluctant to make such changes.
On the other hand, this change just exposes a few more bits of information. The worst that could happen is that someone has to update their parsers for file generated by SDCC. So this change looks much lower-risk to me than other requests to make changes to banking stuff.
I get the feeling that the larger memory space is mostly required for data and not for executable code. That would make it sensible to keep banked funtions in the 24 bit addressable memory and thus the virtual function pointer passed on stack a 24 bit value as well. The linker on the other hand should be able to link in a 32 bit address space. And yes, this might break external tools that rely on sdcc emitting 24 bit .rel files. Maybe those tools can also be made to handle both versions.
If there are really good reasons for SDCC to generate 24-bit virtual addresses rather than 32, I think an option (24-bit by default) is the best way to keep everyone happy.
yes, because it matches Z80 target library and calling conventions. banked call addresses are 24 bit, z80 target reserves 3 bytes on stack performing the banked call (only one byte for the bank), not 4 as on sm83, even by cost of performance penalty! changes should be consistent, not random.
also, changing for z80, why not sm83, where there ARE actual 512-bank mappers (MBC5).
"We're currently working on mapped ROMs that could be up to 32 MB in size, requiring 12 bits for segment numbers." -- your mapper is not even a thing yet!
Last edit: Tony Pavlov 2024-01-24
There's nothing "random" about all this.
If someone has made a choice to have the bank numbers on one byte, well, it is indeed faster than on two (even if the difference is extremely minimal) but it is a totally arbitrary choice. The Z80 has no intrinsic reason to need an 8-bit mapper more than a 16-bit one. These are both hardware devices requirements that are totally independent of the processor.
Once again, I think the best is to add an option to generate virtual addresses in either 24-bit or 32-bit format. With a default value of 24 bits, you're sure not to break anything.
how that is currently implemented for the Z80 target is extremely slow. TWO extra CALLS to get and set the bank on the far call, another extra CALL on return. PUSH AF/INC SP to save byte and DEC SP/POP AF to retrieve it instead of just PUSH AF/POP AF. current library implementation is bad, and decision for the 3 byte stack reservation on the SDCC side is bad (leave calling conventions aside at this point).
it is also bad, that
b<symbol>companion for the banked<symbol>is not visible in C (because of missing underline prefix). that make determining bank very tricky, only possible with some ugly macros, though, God thank you, AT LEAST it is possible. but might be much more convenient.So, @spth, maybe improve this as well? For consistency.
Last edit: Tony Pavlov 2024-01-24