This test sample isn't emulated correctly, albeit it shows somewhat erratic behaviour on real C64 as well.
Load the attached sample program with x64 or x64sc, start with SYS 8160. Now, there's a DMA delay about 8 pixels below the sprites. The behaviour on real C64 (and C128) is that leftmost and rightmost 3 characters are the same; which chars and colors are used depends on individual VIC, sometimes they even flicker. Neither x64 nor x64sc gets this right.
Technical background:
The VIC has an internal buffer for 40 characters and their colors so it needs to fetch them from memory only for the first line of each character row and can retrieve the previously read characters (or colors for bitmap mode) from this buffer for the other 7 lines. Then there's a "pointer" addressing the buffer entry to read or write. It's called VMLI in the well-known VIC article that circulates around the internet in many places and languages, calling it a "6-bit counter with reset input".
In reality it's not a counter but a 40-bit shift register (no proof yet but explains the effects best and die shots also look like a shift register being there). Whenever the VIC starts displaying character data it shifts in a 1; the shift register is shifted by one position in every cycle (not just between the sideborders). And there's no reset entry.
Normally, after the last character has been displayed there's no more 1 bit in the register, thus when characters in the next line start the shift register is all zeroes when the 1 gets shifted in. Furthermore. in idle mode this register is all zeroes, causing reads to also return all zeroes and thus causing idle patterns to be always black.
DMA delay delays the start of the characters in a row and thus the position where the 1 gets shifted in. If that happens during the last 16 visible chars (for PAL timing) there's still a 1 left in the shift register when another one gets shifted in for the next row. If the next row isn't a bad line, the VIC thus fetches 2 cells at once when reading character data and "mixes" their states. When both contain different bits the VIC sometimes uses the first one, the second one, ANDs them or ORs them or flickers between these possibilities (behavior on C64 is unpredictable, highly dependent on the individual VIC; C128 however uses AND most of the time). After reading the VIC writes the values it just read back into the buffer, this also explains why you also see the same strange characters at the right border (since then the second 1 bit already has gone there) and the further character row's lines.
Some special cases to mention:
* If you cause the line following the DMA delay to be a bad line you won't see anything strange. While near the left border the VIC overwrites both the intended buffers and the buffers near the right border still addressed by the 1 bit left by the DMA delay it later on overwrites those erroneously written bytes.
* On C128, when you start DMA delay near the right border but cancel it right away (either by using 2 MHz mode or the test bit), the remainder of that line will be displayed as if it wasn't a bad line and the next line will be idle again. Even more interesting, the 1 still inside the shift register makes the VIC colorize the leftmost fetched idle bytes with character/color buffer contents.
Test case file
Very interesting! This shows some really weird effects on my 6569R3. Thanks for the excellent analysis. Will investigate further.
equivalent test program is now in testprogs/VICII/dmadelay/test4.prg
mmmh last nite i attempted replacing that VMLI variable in the code by a shiftregister that works as described.... with the surprising result that things work exactly like before, ie it didnt fix the problem. i think there must be more to it, or maybe i dont understand the code good enough.... tlr, help! :)
here is the diff for my ugly hack, if anyone wants to look :)
The buffer is not a shift register. It is essentially a 40 word by 12 bit SRAM; there is circuit on the VIC chip that turns on each word line in sequence. The circuit is turned on when a bad line condition is activated. If you time it just right you can trigger a badline via VSP and then again a badline on the beginning of the following line. This will cause two word lines to be activated, where the distance of the word lines is determined by the time in between the two badline pulses.
The behaviour when you turn on two word lines is essentially undefined, and also depends on whatever was on the bus during the previous cycle. The only thing that is consistent is that you will corrupt the 40x12 memory so that a portion on the left is identical to a portion on the right, but what exactly is in these portions is undefined.