#129 tokenise() is too lax for TOK_PREPROC_ID

[nasm64developer]

The preprocessor's tokenise() function is too
lax when it comes to TOK_PREPROC_ID. Namely it
allows %, %!, %%, or %$ to be followed by an
isidchar(), instead of an isidstart(), and it
doesn't perform a tight check for references
to outer contexts, i.e. %$$id, %$$$id, etc.

[nasm64developer]

Logged In: YES
user_id=804543

Attached find a list of testcases.

Note that they target a version of NASM which
supports the "?:" conditional operator, i.e. a
version in which "?" is not a valid isidstart()
character, but is a valid isidchar() character.

[nasm64developer]

testcases

[nasm64developer]

Logged In: YES
user_id=804543

Attached find a diff -u (against 0.98.38) for a fix.

It shows the required changes in their context.

[nasm64developer]

fix (here: diff -u)

[nasm64developer]

Logged In: YES
user_id=804543

Attached find a diff -w (against 0.98.38) for a fix.

It is identical to the diff -u version.

It merely shows that the required changes are, sans
whitespace differences, rather minimal.

[nasm64developer]

fix (here: diff -w)

[nasm64developer]

Logged In: YES
user_id=804543

The attached fix works for the attached testcases.

Note that one can always return to the old behavior
by using %{...}, as demonstrated in those testcases.

[nasm64developer]

Logged In: YES
user_id=804543

SF RFE #1239930 is related