#1809 Rust lexer does not support current string escapes


The Rust lexer at lexers/LexRust.cxx, particularly the switch case at L401 does not support current string escapes.

To be specific, Rust no longer has a \uNNNN (four Ns) or a \UNNNNNNNN (eight Ns) escape; instead, there is only one \u{NNNNNN} escape that takes 1 to 6 digits N.

Here is relevant section of the Rust language reference.

When such a \u{NNNN} escape is used, the lexer seems to treat any tokens after it as not part of the string. When the lexer encounters the subsequent closing quote ' or ", the lexer treats it as the beginning of another string, which causes large swaths of code to be highlighted as a string.

Here is a screenshot from Geany.


  • Neil Hodgson

    Neil Hodgson - 2016-02-12
    • labels: --> scintilla, lexer, rust
    • status: open --> open-accepted
  • Colomban Wendling

    Attached patch adds support for the escapes \u{123ABC} with 1-6 hex digit. The documentation is not clear on whether or not 0 digits is acceptable, but I went ahead and assumed it was absurd.

    I didn't drop support for \uNNNN or \UNNNNNNNNN in case we wanna keep support for older versions. Not sure what's the policy here on Rust.

    Disclaimer: this is an impelemntation solely based on the OP's request and linked docs. It looks straightforward to implement, but I don't have direct Rust knowledge beyond what I just red on teh docs.

    BTW, the Rust lexer seem a lot stricter than what look really necessary. This might be nice to show syntax errors and all, but it also means it is more work to maintain as it has to follow closer the changes to the syntax (e.g. here if a string was less stict, it wouldn't have broken anyting).

    • Neil Hodgson

      Neil Hodgson - 2016-02-16

      Opinions differ on checking for errors in lexers. For example, there is some recent discussion on the mailing list about detecting more errors in Perl code. I think it is OK to make some syntax errors visible if the code isn't too difficult to maintain. There could be properties to turn on error checking in some cases.

  • Neil Hodgson

    Neil Hodgson - 2016-02-16
    • status: open-accepted --> open-fixed
    • assigned_to: Neil Hodgson
  • Neil Hodgson

    Neil Hodgson - 2016-03-16
    • status: open-fixed --> closed-fixed

Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks