#2232 Tokenisation after DATA

[Marc 'BlackJack' Rintsch]

petcat from version 3.9 tokenises after DATA after any colon, even if that colon is in a quoted string. So in…

100 DATA +,":",+

…the first + is stored as PETSCII character and the second + is stored as BASIC token.

I've tried a current SVN version (petcat (VICE 3.10 SVN r46132)) and there it seems worse: anything after DATA is tokenised. Example program:

10 read a$:if a$="" then end
20 print len(a$),asc(a$):goto 10
30 data +,-,print,input,

With petcat at SVN revision r46132 the output looks like this:

run
 1         170
 1         171
 1         153
 1         133
 ```

Expected output:

1 43
1 45
5 80
5 73
```

[gpz]

Darn Rabbit holes :) So more elaborated tests are needed for DATA and REM - and probably the special case handling for them restored (i removed it in https://sourceforge.net/p/vice-emu/code/45960/ - guess that was too much)

If you want to speed it up - testcases in the style of those in https://sourceforge.net/p/vice-emu/code/HEAD/tree/testprogs/petcat/ is what we need :)

[gpz]

should be fixed in r46135 - please test

[Marc 'BlackJack' Rintsch]

Wow, that was fast. 🥰

Works with the test above and also with the BASIC code with the quoted colon I struggled with.

[gpz]

Hehe i was already working on petcat - it should handle (hopefully) all possible quirky situations now (at least for basic v2)