#580 Fix #2732 - syntax highlighting broken if non-visible lines change
This patch makes LineManager set the firstInvalidLineContext upon insertion or removal of buffer content, instead of just moving it around. This results in the necessary reparsing of changed non-visible (and possibly subsequent) lines in JEditBuffer.markTokens which fixes [#2732]
Since the involved code is somewhat complex, here is a lengthy discussion:
The error
The error can be reproduced with the wrap modes "none" and "hard" and any edit mode that contains multi-line SPANs . When a SPAN is started above the visible lines the syntax highlighting will not be properly updated until the start of the SPAN becomes visible.
(e.g. with the BeanShell Console and python edit mode one could insert / remove a multi-line string start: buffer.insert(0, "'''") / buffer.remove(0,3))
What's going on
After the insertion of the comment characters the visible lines are updated and repainted. This is (in part) being handled by org.gjt.sp.jedit.textarea.ChunkCache which calls org.gjt.sp.jedit.buffer.JEditBuffer.markTokens.
(This can be observed with the help of TOKEN_MARKER_DEBUG and CHUNK_CACHE_DEBUG in org.gjt.sp.jedit.Debug)
JEditBuffer uses a (private) LineManager instance to store line contexts (e.g. whether a line is inside a comment SPAN). LineManager already has a firstInvalidLineContext that is being used only by JEditBuffer.markTokens (if the edit mode is context sensitive): markTokens reparses all lines from the firstInvalidLineContext up to the requested lineIndex (if applicable).
Currently firstInvalidLineContext is set (only):
- by
JEditBuffer.markTokensto the next unmarked line (if the line context has actually changed) - to 0 via
JEditBuffer.loadTextwhich is being called upon instantiation and (re-)loading - to 0 by
JEditBuffer.setTokenMarker
LineManager itself updates firstInvalidLineContext in its contentInserted / contentRemoved methods (which are being called by their counterparts in JEditBuffer) so that it still points to the same line.
What's wrong
Whenever content is inserted or removed from a buffer it warrants an update of the syntax highlighting of the affected lines (as anything can start a SPAN, depending on the edit mode). Therefore the firstInvalidLineContext should be re-set to the first affected line in LineManager.contentInserted / .contentRemoved (just like the firstInvalidFoldLevel)
Why doesn't this happen all the time
Usually changes occur within the visible lines, which are (almost) unconditionally reparsed and redrawn so this error does not affect usual editing.
In soft wrap mode the ChunkCache.getLineSubregionCount (which gets called to determine the number of soft-wrapped lines of a physical line) causes a reparse of the changed lines and thus sets the firstInvalidLineContext if necessary. The error could also be "fixed" by removing the leading if(!textArea.softWrap) return 1; (line 279-280) which is probably not a good idea since this would obscure what's actually going on.
History
I traced the code in question back to commit [r4549] where it is introduced as "various minor optimizations". (At this time, 2003, in org.gjt.sp.jedit.buffer.OffsetManager as lastValidLineContext.) I tried to find any reasoning for the code and could neither find it nor come up with anything myself. As far as I understand it this error has always been present.
I see Eric Le Lay already reviewed this in the comments to one of the other tickets.
Committed 24459.