#1112 JS: lexer support for template strings

[monk-time]

Recently approved ECMAScript 2015 (ES6) introduces a new syntax for template strings, which are surrounded with back-ticks and allow embedded expressions inside them, for example:

var a = `${1 + 2}`;
// "3"

See more: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/template_strings

As far as I can see, Scintilla lexer doesn't understand this syntax, and therefore tools based on it cannot highlight template strings properly.

[Neil Hodgson]

The cpp lexer styles backticked strings as SCE_C_STRINGRAW.

[Colomban Wendling]

Unfortunately ES6 supports escaping of the ` in template strings, while current implementation of lexer.cpp.backquoted.strings does not (I guess it's deliberate though). We'd need another option for this (or an additional one to support escape sequences there).

[Neil Hodgson]

Yes, sounds like its more complex.

[monk-time]

Oh, I see. I think this means I should forward (part of) this bug to notepad++ devs, since notepad++ doesn't recognize such strings for js at all.

But still, there probably should be an extra parsing of embedded expressions inside ${} in backticked strings done by Scintilla, if I understand it correctly?

[Neil Hodgson]

If you want additional styles inside the template then you should say what you want. This should probably be optional as others may prefer format strings to just be treated as strings with no internal highlighting.

[monk-time]

What do you mean by additional styles? These internal parts between curly braces can be parsed as normal javascript expressions, using already existing styles.

I'm sorry if I'm asking in the wrong place, if this request is outside of the scope of this project I'll try to move this discussion to npp issue tracker.

[Neil Hodgson]

In the example, which styles go over which ranges?

[monk-time]

Here's how it looks in Firefox's code editor (based on CodeMirror):

Everything inside backticks is styled as a string, excluding contents of curly braces prepended with $, which are styled as a normal expression.

[Thomas Linder Puls]

Has this been addressed?
And what about the HTML lexer which embeds JavaScript/ECMAScript?

[Neil Hodgson]

Not addressed for JavaScript in any lexer.

[Thomas Linder Puls]

OK, thank you

[Corey Farrell]

I've run into an issue related to lack of parsing inside JS template literals. Although syntax highlighting inside the ${} would be nice, a bigger issue occurs when nested template literals are used. The following statement is valid but the backtick immedaitely after ansi.cyan( is interpreted as the end of the template literal, then in this case // is recognized as the start of a comment and the closing parenthesis to match log.info( is not seen.

  log.info(`Listening at ${ansi.cyan(`http://localhost:${port}/app`)}`);

IMO the ${ is like a closing backtick followed by +. The matching } is like a + followed by a new starting backtick. Everything inside ${} is normal JS.

[Neil Hodgson]

There'd have to be a stack so it could back out to literal after the second '}'. Even more difficult if this can be on multiple lines.
This is sufficiently complex that I don't think I'll work on it. Perhaps someone else needs this enough to look into it.

[Corey Farrell]

Thank you for responding. Yes it can be on multiple lines, this would be pretty common in complex templates. It's also possible to stack more than once (template in a template in a template).

Am I correct that these changes would belong in lexers/LexCPP.cxx? Probably would require string properties like lexer.cpp.backquoted.pause and lexer.cpp.backquoted.resume?

I'm not familiar with scintilla code base but I'll try taking a look at this when I have more free time (unlikely in the next couple months).

[Neil Hodgson]

Both LexCPP.cxx and LexHTML.cxx have JavaScript lexers that could be extended with more template functionality. LexCPP, for standa-alone JavaScript files, is more complete and can be extended more easily than LexHTML which handles both client-side and server-side scripts in HTML.

I don't understand what values are expected for lexer.cpp.backquoted.pause and lexer.cpp.backquoted.resume.

[Pba]

Is this problem actually insurmountable? Can we get a definitive "wontfix" or is it that no contributors are interested in this issue?

[Neil Hodgson]

What would be the point of marking this "wontfix"? The set of contributors isn't static and people's interests change.

[Zufu Liu]

I think a standalone lexer for JavaScript, TypeScript and (the dead) ActionScript is better than implement this in LexCPP (which is very complex now), some missing features:

1. Template literal
2. $ and Unicode escape (\u{HHHH} and \uHHHH) in identifier

3. private fields, which is not C/C++ like preprocessor

4. @decorator
5. XML (JSX, TSX) literal and @xmlNode.attribute
6. underscore in number literal