Please don't burn me at the stake for this request. :)
Emulators are now undeniably important development tools for the MSX community, and openMSX plays a major role on this scenario.
Also, keeping strict MSX cross-compatibility was/is always a problem, just like Internet browsers cross-compatibility.
Testing software on a lot of different hardware is both time consuming, costly and ineffective. Akin, for a designer to test its pages on as many browsers as possible is both time consuming, costly and ineffective.
To make things worse, it seems that the recently created community software is getting farther and farther of the MSX concept, by ignoring the BIOS completely and going to direct hardware access in a way that isn't compliant to the standard.
For the Internet browsers, the creation of the Web Acid tests helped to create a standardized way to assure that the interoperability could be quickly tested, exercising the specification to the maximum extend:
The philosophy behind the web AcidTests is: "As with acid tests for gold which produce a quick and obvious assessment of the quality of a piece of metal, the web acid tests were designed to produce a clear indication of a browser's compliance to web standards."
The same philosophy probably can be applied to the MSX standard. I believe that the reason why many MSX programmers don't follow the coding recommendations is because it "just works" on their simple testing setup, exactly as it happened with the browsers before the AcidTests were created: programmers just tweaked their code to work under the limited testing environment they had.
As the MSX standard is all about expansion, the respect this same standard allows the hardware to evolve better and in a cheaper way, even in a hobbyist scenario like we have today. As the MSX community has very limited resources, there should be a way to assure that the most common software compatibility/interoperability errors are quickly detected.
Because of this, I'm asking for the creation of AcidTest-machines under openMSX. Those machines would work on a fashion analog to the web AcidTests.
Each "machine" will have some specific configuration the assure that some aspect of the standard is being respected. As for the web AcidTests, only well behaved software will run successfully. Bad players are expected to fail miserably.
As mentioned, the tests would certify one aspect of the standard. It was grouped this way to make debugging easier:
Acid1 Test: Slot layout test
Acid2 Test: Illegal direct hardware access
Acid3 Test: Incorrect BIOS handling
Acid4 Test: Inter-chip timing Test
Acid5 Test: Speed test
Acid6 Test: Common Mistakes test
Of course the machines don't need to be created in this exact order, since some of the tests will be easier to create (using the existing features) and other will require some more coding on the emulator side.
I have already created the major part of the first two AcidTest machines. But to get them complete I'll need the openMSX team cooperation, as some minor enhancements will have to be coded. It would also be great to have them included as default machines of openMSX.
Detailed description of each test:
1) Acid1Test-slots: This machine is aimed is to detect incorrect slot handling. It will have the weirdest possible slot configuration allowed under the standard, arranged in such way that any false assumption by the programmer about slot handling will just not work. It will off course create gotchas for RAM/extensions detection.
2) Acid2Test-hardware: The MSX Technical Handbook, pg-336 clearly states that programs should not handle the hardware directly and the only direct I/O allowed is to the VDP, by using the ports indicated on addresses 0006h and 0007h of the BIOS. Pg-42 of this same book, on "Notes on I/O Address Assignments", mentions the reason: "Although I/O addresses are defined above, the software must not access those devices directly using the above ports. All I/O accesses must be done using BIOS calls, in order to make the software independent of hardware differences. MSX manufacturers may change some of the hardware from the standard MSX system and maintain software compatibility by rewriting BIOS. The hardware differences would thus be transparent to the software."
This Acid2Test machine is then aimed to detect illegal direct hardware accesses: Any software that tries it will fail to run. It emulates a hypothetic machine that was fully customized by the manufacturer as allowed by the page-42 of the MSX Technical Handbook.
3) Acid3Test-BIOS: This machine is the opposite of the previous one. All hardware is placed at default addresses, but all BIOS routines will be relocated internally. This will catch programs that do direct calls to the routines instead of using the jumptables. Maybe a version of CBIOS without workarounds can be an effective test for this one (I usually tested this on a Sharp Hotbit, but not all routines were relocated and it only tests MSX1 programs).
4) Acid4Test-InterChip: This machine triggers a set of scripts that assure that I/O timing is being respected for the chips that most need it: VDP, OPLL, OPL1, OPL3, OPL4. Currently openMSX already has a script for the VDP timing.
5) Acid5Test-SpeedTest: This one assures that the software has proper timing and don't rely blindly on software-loops for temporization. It assures that turbo machines will be supported correctly. It has the following customizations:
a) Z80 runs at 40MHz
b) V9958 blitter works 100% faster
c) The machine has a built-in SystemTimer
d) There's no way to disable the turbo, which is a common workaround used by programmers (if the turbo could be disabled this test would have no meaning at all: the idea is to test if the algorithms are ok)
6) Acid6Test-CommonMistakes: Similar to Acid4Test, but the scripts do test for common programing mistakes as well as for known MSX quirks (as nobody would like to implement many of those quirks on the emulators), like:
a) VDP register access with interrupts enabled
b) Sending new blitter commands without checking if the previous one finished
c) r#18 in combination with VDP commands corrupts VRAM
d) screensplits+R#18+sprites_disabled causes problems
Here the status of each of the 2 tests I already created:
It's probably complete.
The only feature that's not implemented is a memory-mapper with each of its 4 frames scattered over different slots. It's not implemented because of two reasons:
1) I'm not sure if the standard required that all frames of the same memory-mapper must be on the same slot
2) Even if the standard allows the mapper frames to be scattered, there's no way to implement it that way on the current openMSX.
Yes, I know this idea of scattering the memory-mapper slots is weird, but the if allowed by the standard that would really stress the detection algorithms. :)
Nearly complete, but openMSX have hardcoded addresses for both the Memory-Mapper registers and the expanded-slot register. The memory-mapper ignores the io-base parameter and there's no similar parameter for the expanded-slot register memory address.
ok: a) PPI-8255 on nonstandard I/O port
ok: b) V9958 at 88h I/O port
ok: c) PSG on nonstandard I/O ports
ok: d) YM2413 on nonstandard I/O ports
ok: e) RTC on nonstandard I/O port
ok: f) Y8950 at nonstandard I/O port
not-ok: g) Expanded slot registers at nonstandard address
not ok: h) MemoryMapper registers on nonstandard I/O ports
ok: j) PrinterPort at nonstandard I/O port
Even without being complete, the first two tests gave very interesting results on the set of games I tested:
Acid1Test: It's a very basic test, but it's interesting that many more games than I expected fail on it. I was expecting that only a few games would fail, but quite some games just don't work at all or don't detect extensions properly.
Acid2Test: This is probably the hardest of all tests. I as expecting that almost no game would pass. But to my surprise, some gamehouses really respected the standard! Nearly all Konami games passes with flying colors (*1), proving that no direct hardware is needed to create great games. Casio and ASCII games also have a high success rate. OTOH, nearly all Compile and Falcom games fail miserably here.
*1: The only glitch I could find on Konami games is that the PSG doesn't play on their later games: Pennant Race 2, Space Manbow and Metal Gear 2.