#1013 Enhance proposal: Pascal lexer: add optional keyword list for more individual highlighting of keywords

[peterme]

For the Pascal lexer/highlighter i find it very restrictive to have only one keyword class = only one single color available for highlighting keywords. Very very poor! I would like to have at least one more color for keywords ...
Below a simple change proposal to allow a little bit more flexibility (works well with scite.exe! Tested).
The idea behind is: leave the regular keyword processing untouched (and so all logic and the current usage untouched!), but provide an additional wordlist. If one or more of the regular defined keywords can be found within this additional wordlist too, you can define a customized color for those keywords (new style id #15).
The changes are no more than follows (would you be so kind as to integrate it??) :

• SciLexer.h: add to the appropriate place:
  #define SCE_PAS_KEYWORDCUSTOM 15

• Scintilla.iface: add to the appropriate place:
  val SCE_PAS_KEYWORDCUSTOM=15

• pascal.properties: integrate:
  # Keyword custom:
  keywordclasscustom.pascal=private protected public published overload override read write message
  keywords2.$(file.patterns.pascal)=$(keywordclasscustom.pascal)
  # or ... simply omit those two settings (= means, that equals same behaviour as before)
  # Keyword custom style:
  style.pascal.15=fore:#FF0000,bold

• LexPascal.cxx:
  ............Modify procedure "ClassifyPascalWord":
  WordList& keywords = *keywordlists[0];
  WordList& keywordscustom = *keywordlists[1];

  if (!ignoreKeyword) {
      if (keywordscustom.InList(s)) {
          sc.ChangeState(SCE_PAS_KEYWORDCUSTOM);
      }
      else {
          sc.ChangeState(SCE_PAS_WORD);
      }
  }
  

  ...........Change procedure pascalWordListDesc:
  "Keywords",
  "KeywordsCustom",

---------> The files are attached

[Neil Hodgson]

This will be examined once 3.3.5 is released in about one week.

[peterme]

Many thanks again for the fast response!
Pls don't hesitate to ask in case of any questions; i'm very interested in this issue (to have at lease a little bit more flexibility in highlighting of keywords).
Of course other solutions are possible (eg. in comparison with LexInno), but that one is expected to be very easy and transparent.
I could provide a sample SciLexer.dll (used from scite.exe), but is very restricted due to other compiler issues i encountered on the NotePad++ side (my usage would be from N++ and would require further changes here).

[Marko Njezic]

Keywords that are defined as "custom" are already defined as regular keywords. As a result this change will enforce one's personal preference of highlighting those keywords differently upon everyone else. Besides this, who can say that only one additional list of keywords is enough? Someone else may come in the future and request a third set of keywords in a third color. Judging by this feature request I guess that the intent was to have support for rainbow identifiers in Pascal lexer and such feature is probably better implemented using Sub Styles framework instead of adding new keyword lists at whim. I would also choose a different name for these additional keywords, something like "Secondary set of keywords" or "User defined keywords" instead of using the word "custom".

[Kein-Hong Man]

Let's just call it word lists, many lexers already have them.

[Marko Njezic]

There are more issues with proposed changes. If you define any of the keywords that are used as fold points in the secondary set of custom keywords, folding will no longer work properly. It is also not very intuitive that you also have to define secondary keywords in the list of regular keywords, for them to be recognized. There's also a question should all keywords be affected by lexer's smart highlighting feature or not.

@KHMan I was talking about textual description of the newly added keyword list.

[peterme]

"Secondary set" or "user defined" instead of "custom" is ok, of course.
One set of this kind should be sufficient and give a little bit more flexibility
(the demand for a 3rd, 4th ... theoretically is possible, but in practice i would not worry about).
-
I did see the point that a candidate for this second set might be an anchor for folding
= impacts on folding --> so i did a modification that deals with that (file attached).
I tested it with scite.exe.
-
The smart highlighting should be untouched because it is queried before the branch
about the secondary set does happen.

[Marko Njezic]

Whether just one additional set of keywords is enough is open for debate. I thought that having only one set of keywords would be enough, as I've modeled the lexer after Delphi IDE, but guess I was wrong.

Regarding the latest version of your modifications, only difference is that you added two lines that attempt to perform style juggling, but as it is typical for all quick hacks, it is incomplete. The latest version also doesn't address my comment that it is not very intuitive (and different from other Scintilla lexers) to have keywords defined in both lists in order for them to be properly recognized.

Since I'm not a big fan of quick hacks and half-baked solutions, like I said in my first comment this feature request for rainbow identifiers in Pascal lexer should be addressed by utilizing Sub Styles framework, which was added to Scintilla for this exact purpose. It would allow users to define arbitrary amount of additional keyword styles, thus catering to different user preferences.

[peterme]

Hi Marko,

i'm fully aware that the method would differ from other lexers; the idea behind is just this not to define a second keyword area (for not to break other logic) but simply give the possibility to define a subset of those keywords that are expected to be highlighted in a different color.
Please keep in mind that the Inno lexer provide 4 keyword sets. The Pascal lexer offers three styles for comments, 2 styles for preprocessor statements. But for keywords - and we're speaking about nearly 110! - it only offers one style, not more. Crazy. I'm not interested in a rainbow palette. I simply would like to see one second coloring possibility. Half-bakened or full-backend .. it should work and be available anyhow. Currently it is not available at all. A lot of other language types a provided with at least two keyword sets although the amount of keywords is not so high.
If it should be really possible already today to use a sub styles framework to fullfill this purpose, and it could be used by NotePad++ without big efforts, please tell me; i would appreciate to have more information how to use this framework.
I'm not a scintially or Notepad coder. I#m simply a user that likes to address an enhancement request and to contribute with a proposal that is leam, without big impact and that was tested before offered here.
If there should be any other solution available i would be very glad.
I understand well the reason for the original decision = the behaviour of the Delphi IDE, but i expect scintialla and Notepa++ to offer a better editor than the Delphi IDE.
Best Regards!
Peter
About the two-liner: it does an adaption in a local context for to avoid code duplication or a series of additional "if"s. It works, but a better coding would be very welcome - if any.

[Marko Njezic]

Having the same keywords in two different lists is simply confusing to the end users. Regarding Inno lexer, out of those keyword sets only one is used for Pascal scripting, so we are actually in the same situation (a better comparison would be Lua lexer, but it's a bad example altogether as it was written before Sub Styles framework was added).

Rainbow identifiers is just a term used to describe multi-colored keywords and it is already implemented in Scintilla (albeit as a provisional feature, but it won't be removed). You can find more information on Scintilla's mailing list ( https://groups.google.com/d/msg/scintilla-interest/Ozi1CcYIr8c/nu37jArY-aIJ ), in Scintilla's documentation ( http://www.scintilla.org/ScintillaDoc.html#SCI_GETSUBSTYLEBASES ) and by looking at CPP lexer, which uses sub styles.

The main reason why I'm against half baked solutions is that once they are added, you are stuck with burden of maintaining backwards compatibility even if the same feature gets implemented properly at some point in the future. In this particular case, it is probably best to use Sub Styles right from the start and allow users freedom of choice regarding keywords coloring.

[peterme]

I think i understand your argumentation, however my point is to have a solution, anyhow.
I would accept any kind or half-bakenend solution or inofficial user exit and - although being an end user - would not be confused.

At that point, as not being a professional C-expert, looking at the sources ie. cpp lexer, i feel that i cannot longer really contribute regarding the coding, so for me nothing else is left than to ask when the full-bakened adaption will be available.

If the half-bakened proposal, although working very well, is not good enough, which kind of proposal may i expect?

[Neil Hodgson]

Originally, there was supposed to just be a single list of keywords for each language and these would be the language's real keywords - that is, words which were syntactically significant so could influence lexing. Some languages combine multiple sublanguages, such as HTML embedding JavaScript, so provision was made for multiple keyword lists.

However, some people wanted to style particular sets of identifiers in different ways, often to differentiate between their own functions and those from a standard library or system API. The keyword mechanism was used to implement this but there were problems since there was no good answer to "how many sets is enough?" and the range of styles had to be shared between different purposes. Thus substyles were implemented to allow this to be implemented more flexibly.

The proposal here is unique in that the 'custom' keyword list does not appear to be wanted for identifiers but to divide the set of real keywords in two. There's no clear reason behind the particular division chosen - perhaps they are seen as more or less important than other keywords. Having some keywords in a second style may cause problems in code that uses the style values, such as automatic indentation.

The substyles feature should be made permanent soon - I wanted to do it for the 3.3.5 release but haven't had time.

[peterme]

Really many thanks from my side for your attention and patience
to all of you who were so kind to involve in this discussion!
(i tried to do my best)
Peter

[Neil Hodgson]

It would be reasonable to add substyles for identifiers to the Pascal lexer. That first requires changing the lexer into an object lexer similar to the cpp lexer, since additional calls have to be made to the lexer which are not available with function lexers.

If there were ever to be substyles added to keywords then it would help applications if there was a call to determine the base style of a style, so the application could easily treat all keywords as one class.