Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo


#1464 Newlines in LexOthers.cxx aren't colored as default

Neil Hodgson
scintilla (168)
Eric Promislow

The languages in LexOthers.cxx are all colored one line at a time, rather than
as a single stream. The main problem is that it assigns the line-end-sequence
with the same style as the last token, which makes it harder for downstream
code to find boundaries of things like comments.

Is there a reason for this design? Are these lexers due for a major update?


  • Neil Hodgson
    Neil Hodgson

    I don't really understand the problem with comment boundaries.

    Line-oriented lexers are simpler to code and understand. Its unlikely there was any reason to prefer line ends to keep the comment style or return to default. About the only thing I'd like to do is split some of these out of LexOthers.

  • Eric Promislow
    Eric Promislow

    The problem is downstream. Much of the Komodo code uses styles to "parse" files, as it's much faster than processing characters. When calculating indentations after a newline, the code assumes that if a newline is styled as a comment we're in a block comment, and therefore we shouldn't start the next line with the comment-start character (this is with shift-newline, not regular newline).

    The problem I have with these line-oriented lexers is they too often look for one feature in a line, and style the entire line based on that feature, so they aren't set up to style newlines separately. I really don't see the advantage of doing this over having one state machine that runs through the full buffer.

  • Neil Hodgson
    Neil Hodgson

    You can't rely on line comments going to the default state at line end. For example the C++ lexer styles the line ends on line comments as line comments. The implementations may be done this way to make other aspects of lexing easier - I remember some complex issues with C++ preprocessor lexing that depended on how line ends were treated.

    Also, the lexers for VB, Lua, Ada, D, and TCL (out of the ones I could easily test) extend line comment style into the line end.

    I think more consistent results will be produced by understanding the different styles for line and stream comments.

  • Eric Promislow
    Eric Promislow

    It's possible. Currently the code that looks for the start of a comment
    doesn't know whether it's grovelling through block-oriented or line-oriented
    comments. And in Komodo's main languages where this is a factor -- Perl,
    Python, JS, Ruby, and Tcl, the newlines that terminate line-comments
    are always styled as default (we forked the Tcl lexer).

  • Neil Hodgson
    Neil Hodgson

    • labels: --> scintilla
    • status: open --> closed-rejected
    • assigned_to: Neil Hodgson