#218 AcidTests for the MSX standard

SD Snatcher

Please don't burn me at the stake for this request. :)

Emulators are now undeniably important development tools for the MSX community, and openMSX plays a major role on this scenario.

Also, keeping strict MSX cross-compatibility was/is always a problem, just like Internet browsers cross-compatibility.

Testing software on a lot of different hardware is both time consuming, costly and ineffective. Akin, for a designer to test its pages on as many browsers as possible is both time consuming, costly and ineffective.

To make things worse, it seems that the recently created community software is getting farther and farther of the MSX concept, by ignoring the BIOS completely and going to direct hardware access in a way that isn't compliant to the standard.

For the Internet browsers, the creation of the Web Acid tests helped to create a standardized way to assure that the interoperability could be quickly tested, exercising the specification to the maximum extend:

The philosophy behind the web AcidTests is: "As with acid tests for gold which produce a quick and obvious assessment of the quality of a piece of metal, the web acid tests were designed to produce a clear indication of a browser's compliance to web standards."

The same philosophy probably can be applied to the MSX standard. I believe that the reason why many MSX programmers don't follow the coding recommendations is because it "just works" on their simple testing setup, exactly as it happened with the browsers before the AcidTests were created: programmers just tweaked their code to work under the limited testing environment they had.

As the MSX standard is all about expansion, the respect this same standard allows the hardware to evolve better and in a cheaper way, even in a hobbyist scenario like we have today. As the MSX community has very limited resources, there should be a way to assure that the most common software compatibility/interoperability errors are quickly detected.

Because of this, I'm asking for the creation of AcidTest-machines under openMSX. Those machines would work on a fashion analog to the web AcidTests.

Each "machine" will have some specific configuration the assure that some aspect of the standard is being respected. As for the web AcidTests, only well behaved software will run successfully. Bad players are expected to fail miserably.

As mentioned, the tests would certify one aspect of the standard. It was grouped this way to make debugging easier:

Acid1 Test: Slot layout test
Acid2 Test: Illegal direct hardware access
Acid3 Test: Incorrect BIOS handling
Acid4 Test: Inter-chip timing Test
Acid5 Test: Speed test
Acid6 Test: Common Mistakes test

Of course the machines don't need to be created in this exact order, since some of the tests will be easier to create (using the existing features) and other will require some more coding on the emulator side.

I have already created the major part of the first two AcidTest machines. But to get them complete I'll need the openMSX team cooperation, as some minor enhancements will have to be coded. It would also be great to have them included as default machines of openMSX.


Detailed description of each test:

1) Acid1Test-slots: This machine is aimed is to detect incorrect slot handling. It will have the weirdest possible slot configuration allowed under the standard, arranged in such way that any false assumption by the programmer about slot handling will just not work. It will off course create gotchas for RAM/extensions detection.

2) Acid2Test-hardware: The MSX Technical Handbook, pg-336 clearly states that programs should not handle the hardware directly and the only direct I/O allowed is to the VDP, by using the ports indicated on addresses 0006h and 0007h of the BIOS. Pg-42 of this same book, on "Notes on I/O Address Assignments", mentions the reason: "Although I/O addresses are defined above, the software must not access those devices directly using the above ports. All I/O accesses must be done using BIOS calls, in order to make the software independent of hardware differences. MSX manufacturers may change some of the hardware from the standard MSX system and maintain software compatibility by rewriting BIOS. The hardware differences would thus be trans­parent to the software."

This Acid2Test machine is then aimed to detect illegal direct hardware accesses: Any software that tries it will fail to run. It emulates a hypothetic machine that was fully customized by the manufacturer as allowed by the page-42 of the MSX Technical Handbook.

3) Acid3Test-BIOS: This machine is the opposite of the previous one. All hardware is placed at default addresses, but all BIOS routines will be relocated internally. This will catch programs that do direct calls to the routines instead of using the jumptables. Maybe a version of CBIOS without workarounds can be an effective test for this one (I usually tested this on a Sharp Hotbit, but not all routines were relocated and it only tests MSX1 programs).

4) Acid4Test-InterChip: This machine triggers a set of scripts that assure that I/O timing is being respected for the chips that most need it: VDP, OPLL, OPL1, OPL3, OPL4. Currently openMSX already has a script for the VDP timing.

5) Acid5Test-SpeedTest: This one assures that the software has proper timing and don't rely blindly on software-loops for temporization. It assures that turbo machines will be supported correctly. It has the following customizations:

a) Z80 runs at 40MHz
b) V9958 blitter works 100% faster
c) The machine has a built-in SystemTimer
d) There's no way to disable the turbo, which is a common workaround used by programmers (if the turbo could be disabled this test would have no meaning at all: the idea is to test if the algorithms are ok)

6) Acid6Test-CommonMistakes: Similar to Acid4Test, but the scripts do test for common programing mistakes as well as for known MSX quirks (as nobody would like to implement many of those quirks on the emulators), like:

a) VDP register access with interrupts enabled
b) Sending new blitter commands without checking if the previous one finished
c) r#18 in combination with VDP commands corrupts VRAM
d) screensplits+R#18+sprites_disabled causes problems
e) etc


Here the status of each of the 2 tests I already created:


It's probably complete.

The only feature that's not implemented is a memory-mapper with each of its 4 frames scattered over different slots. It's not implemented because of two reasons:

1) I'm not sure if the standard required that all frames of the same memory-mapper must be on the same slot
2) Even if the standard allows the mapper frames to be scattered, there's no way to implement it that way on the current openMSX.

Yes, I know this idea of scattering the memory-mapper slots is weird, but the if allowed by the standard that would really stress the detection algorithms. :)



Nearly complete, but openMSX have hardcoded addresses for both the Memory-Mapper registers and the expanded-slot register. The memory-mapper ignores the io-base parameter and there's no similar parameter for the expanded-slot register memory address.

ok: a) PPI-8255 on nonstandard I/O port
ok: b) V9958 at 88h I/O port
ok: c) PSG on nonstandard I/O ports
ok: d) YM2413 on nonstandard I/O ports
ok: e) RTC on nonstandard I/O port
ok: f) Y8950 at nonstandard I/O port
not-ok: g) Expanded slot registers at nonstandard address
not ok: h) MemoryMapper registers on nonstandard I/O ports
ok: j) PrinterPort at nonstandard I/O port


Even without being complete, the first two tests gave very interesting results on the set of games I tested:

Acid1Test: It's a very basic test, but it's interesting that many more games than I expected fail on it. I was expecting that only a few games would fail, but quite some games just don't work at all or don't detect extensions properly.

Acid2Test: This is probably the hardest of all tests. I as expecting that almost no game would pass. But to my surprise, some gamehouses really respected the standard! Nearly all Konami games passes with flying colors (*1), proving that no direct hardware is needed to create great games. Casio and ASCII games also have a high success rate. OTOH, nearly all Compile and Falcom games fail miserably here.

*1: The only glitch I could find on Konami games is that the PSG doesn't play on their later games: Pennant Race 2, Space Manbow and Metal Gear 2.


  • SD Snatcher

    SD Snatcher - 2011-03-17

    As requested, I did add the requests #3218248 and #3218271 as smaller parts of this one.

  • Maarten ter Huurne

    The Web Acid tests provide content (the test) to stress the infrastructure (browser), while these MSX Acid tests provide infrastructure (MSX machines) to test the contents (software). The MSX equivalent of the Web Acid tests would be a set of programs that tests the accuracy of MSX emulators, which would be very useful, but a different project. So the name might be confusing.

    I am not sure the goal of testing compliance with the written standard is really useful to developers writing new MSX software. It is important to be able to test whether software will run on all existing MSX machines, but does it really matter whether software complies with the written spec or not if it runs on every MSX machine? There is a kind of unwritten spec of the common properties of all existing MSX machines. There won't be many new MSX models anymore, but the few that will be made will likely be based on both the written and the unwritten spec.

    My suggestion would be to create two sets of machines: one set that tests compatibility with actually existing MSX machines, but putting all exceptional configurations in only a handful of machines instead of dozens, and one set that tests compatibility with details of the written spec where in practice there is no variation among MSX machines. The first set would be very useful for people developing new MSX games, while the second set would be mostly there for curiosity.

  • SD Snatcher

    SD Snatcher - 2011-03-20

    Hi Maarten!

    The name doesn't matter that much. :)

    The problem with testing with a limited set of machines is, in fact, that there are almost infinite combinations of MSX hardware that are in fact causing problem with the games being developed. This happens exactly because of the great flexibility of the MSX system.

    In examples:

    1) A machine could have 32KB of internal RAM, but have a 4MB memory mapper added on a slot expander. Buggy RAM detection always fall for this

    2) Many games require MSX-DOS1. Isn't it annoying to have to remove cartridges or other extensions from your MSX just to run some software? Or the machine has to have internal workarounds of every kind of bug like this: The MSX-TR had to include both DOS1 and DOS2 BIOS and the workaround to switch between them.

    3) Many software do use the EXTBIOS hook incorrectly. But the bug will not manifest itself under all conditions. It depends on a combination of extensions using the EXTBIOS hook. The best case I can remember is the bug that MEMNAN had on Turbo-R machines.

    In other words, a lot of software is being produced with restrictions on how the user can run them. They run on plain vanilla machines, but have a lot of problems if there are any extensions running. The more extensions an user has, the more problems he will likely have with buggy software, because adding more extensions causes more complexity and subsequently requires more compliance to the MSX spec.

    And restricting the extensions in unwanted on the MSX system. The Nestor's Internet Suite comes to my mind as a wonderful recent extension I wouldn't want to see trashed by buggy software. Yes, there's still only RAM implementations of that spec, but the UNAPI allows an Internet cartridge to be build with the entire implementation in ROM.

    The MSX-AcidTests aim to provide exactly this more complex testing scenario, in a way that if the if the tested software runs on them, it has a much higher degree of chance of running on simpler machines. This is exactly as happens with the WebAcidTests: If a browser is able to run the tests, it certainly will be able to run less complex websites elsewhere.

  • Maarten ter Huurne

    Hi Fabio!

    I fully agree that it is good to offer an easy way for developers to test their programs for potential compatibility problems. But there are two different kinds of compatibility problems:
    A. problems with existing MSX hardware and hardware that is likely to be produced in the future
    B. problems with theoretical MSX hardware that will never be actually produced
    It seems you are currently designing tests for both A + B, while I think developers will only want to test for A.

    The example you mention of 32K unmapped internal RAM + 4MB mapped external RAM is in category A. I once had a slightly similar setup with 128K mapped internal RAM + 512K mapped external RAM and indeed a lot of software didn't work properly with that, which was frustrating. So having a test setup for this is useful for developers.

    An example of category B would be the VDP I/O ports. The official way to access the VDP is either through BIOS routines or by reading the VDP I/O port numbers from address 6/7 of the BIOS and using OUT(C),r and IN r,(C). In practice, hardcoding 0x98..0x9B works on every MSX machine ever built. Being able to use OUT (0x99),A makes code simpler and shorter. So creating a machine with the VDP at a different port and a matching BIOS complies with the MSX spec, but does not help developers in any way; it only makes their life harder.

    That's why I think there should be separate tests for both categories. The tests for category A will be used by people who want to submit a game to for example MSXDEV. The tests for category B will only be used by people who are curious whether some piece of software (not necessarily their own) will work in crazy but spec-compliant machines. If you put all tests in a single test set, I predict that developers will consider it too hard to have their software pass the tests and will stop running the tests altogether.

  • SD Snatcher

    SD Snatcher - 2011-04-23


    You're quite right, and the tests were separated as different machines exactly to fit this idea: each one tests one specific aspect that the programmer might desire, without forcing it to test aspects he don't want.

    The Acid1Test-slots fits on the description of situation-A you described. This situation-A could be best described as "Usual hardware in unusual slot disposition".

    The Acid2Test-hardware fits better on the description of the situation-B you described. This situation-B could be best described as "Unusual hardware in usual slot disposition" :)

    And the same philosophy applies for the rest of the pack:

    Acid3Test-BIOS: "Usual hardware, incorrect BIOS calling" (many incompatibilities with the Sharp Hotbits are catch by this one)

    Acid4Test-Interchip: "Usual hardware, incorrect I/O timing"

    Acid5Test-Speed: "Unusual hardware (turbo), incorrect speed timing routines" (catches incompatibilities with turbo machines and the with OCM"

    Acid6Test-CommonMistakes: "Usual hardware, usual mistakes"