#2121 scintilla perl lexer support for indented here-doc

[SilverDirk]

The Perl lexer does not support indented here-doc, introduced in perl 5.26

I've created a patch, but don't know the workflow for mercurial and you might want to write it differently anyway, so I'm just attaching it to the ticket.

[Kein-Hong Man]

Thank you. I had done updates up to 5.24 but was too busy with other things to focus here. I will look at it this week, learn about 5.26 stuff, and test it. Then when Neil is back from his holiday, he can push it into the repo.

I just quickly scanned perl5260delta.pod and it looks like this indented here-doc is the only thing that need to be added. There does not appear to be anything in perl5280delta.pod that needs lexer updating. But if there are other items like new functions etc, let me know. I will read the deltas in detail later. I think that we can aim for perl 5.26 and 5.28 support for the next release of Scintilla/SciTE. Thanks again.

[Kein-Hong Man]

Here are my notes for perl updates to 5.30. The main breakage is indented here-docs in perl 5.26. There was some minor glitches in 5.24 which I will deal with in another ticket in the future.

This ticket will be for indented here-docs. I will add more test cases while I test the patch. Once this is done, we're in good shape for perl 5.30 code.

[Kein-Hong Man]

The patch works fine, thanks. Attached are some test cases for indented here docs.

The lexer will be more permissive in the kinds of allowed indented here docs compared to perl, not a big deal if the here doc block is edited using the same indentation settings by the coder(s). The coder should be able to fix things if perl barfs on it since the error message is clear.

[SilverDirk]

Yeah, I considered trying to parse it as an error if the leading indent wasn't right, but that would involve a lot of changes to how it is written, and be distracting to the user while in the process of writing a here-doc.

I was also working on a second patch that would implement the "todo" of multiple here-doc that start on the same line, but so far my changes have just been breaking things. I could send you what I have so far if this sounds like something you want to work on, or I can just come back with a new ticket if/when I get it working.

Thanks for working on it!

[Kein-Hong Man]

I also prefer to avoid too much distracting styling changes, so it's good. Generally, the lexer is more permissive (versus parsing the syntax correctly) and does just enough to give a certain level of styling quality. There are plenty of dusty corners cases that breaks highlighting. The lexer's approach is to scan backward and forward to disambiguate things, this avoids having to do traditional parsing especially for instance, when highlighting snippets that is not a complete program.

Please feel free to work on the perl lexer. You can also discuss stuff on the Scintilla mailing list:
https://groups.google.com/forum/#!forum/scintilla-interest
Generally for the perl lexer you probably won't be duplicating work of others; you can also ping the list to avoid duplication of work. For me, I go on a binge of coding and put out something maybe once a year on average, am not coding anything at the moment.

[Kein-Hong Man]

For multiple here docs on the same line, or stacked here docs as some call it, I always got the impression that it sorta works by 'luck' since the lexer remembers the last here doc delimiter. I have only one test sample:

print <<FOO, <<BAR;
some text
FOO
some text
BAR

The end result seemed visually okay to me, but it may do funny things while you are editing it. Being a rare thing, it was not a priority item for me. We can continue discussing improvements to the perl lexer on the mailing list.

[Neil Hodgson]

Committed as [f32c4f]. Added initialisation of StripIndent in HereDocCls constructor to avoid warnings.

Added "SilverDirk" to credits but that can be changed if you would prefer a different name.

[SilverDirk]

Oh, heh, I didn't realize how anonymous I was here. (there doesn't even seem to be a place in the profile to list my real name) Please add me as "Michael Conrad"

[Neil Hodgson]

OK, updated. [df223c]

[Neil Hodgson]

Committed as [f32c4f]. Added initialisation of StripIndent in HereDocCls constructor to avoid warnings.

Added "SilverDirk" to credits but that can be changed if you would prefer a different name.