Fix CHARACTERS_TOKENs that cross buffer boundry
Status: Beta
Brought to you by:
jpcs
Patch to return PREMATURE_END_OF_BUFFER, Instead of returning a partial CHARACTERS_TOKEN when text is split by the end of a buffer.
Example:
First buffer holds:
<?xml version="1.0" encoding="UTF-8"?>
<doc>
<name>long_name_split
Second buffer holds:
_by_end_of_buffer</name>
</doc>
new version of patch.
example program to show the problem.
Updated the patch to correct it. Also attached an example program to show how CHARACTERS_TOKEN's get fragmented.