#1380 Add Julia lexer

[getzze]

Add a lexer for Julia.
Tested with geany ( https://github.com/geany/geany/pull/2584 )
Also there: https://sourceforge.net/p/scintilla/code/merge-requests/25/

[Neil Hodgson]

It will not be possible to integrate this until after Scintilla 5.0 is complete.

[Neil Hodgson]

There are some warnings and errors from different compilers and linters - Visual C++, g++, clang++, and cppcheck on Windows 10:

../lexers/LexJulia.cxx: In function 'bool jl_op_suffix_char(uint32_t)':
../lexers/LexJulia.cxx:309:10: error: 'uint' was not declared in this scope; did you mean 'int'?
  309 |     for (uint i = 0; i < opsuffs_len; ++i) {
      |          ^~~~
      |          int
../lexers/LexJulia.cxx:40:20: warning: unused function 'IsJuliaReservedOperator' [-Wunused-function]
static inline bool IsJuliaReservedOperator(int ch) {
                   ^
lexilla\lexers\LexJulia.cxx:310:26: warning: Array 'opsuffs[117]' accessed at index 233, which is out of bounds. [arrayIndexOutOfBounds]
        if (wc == opsuffs[i]) {
                         ^
lexilla\lexers\LexJulia.cxx:804:15: warning: Variable 'style' is assigned a value that is never used. [unreadVariable]
    int style = initStyle;
              ^

uint isn't a standard type . https://en.cppreference.com/w/cpp/language/types

The array length code is weird: the element type is uint32_t but opsuffs_len appears to be assuming uint16_t. If the code is doing the obvious 'elements of an array' then use std::size.

style can be declared inside the loop where its assigned.

[getzze]

Hi,
Thanks for reviewing! This is an old version of the file, here is the updated patch (attached).

[Neil Hodgson]

Thanks for the update. This version has some unused variables and functions. Here are the warnings.

../lexers/LexJulia.cxx: In member function 'virtual void LexerJulia::Lex(Sci_PositionU, Sci_Position, int, Scintilla::IDocument*)':
../lexers/LexJulia.cxx:932:15: warning: unused variable 'max' [-Wunused-variable]
  932 |  Sci_Position max = pos + length;
      |               ^~~
../lexers/LexJulia.cxx:957:10: warning: unused variable 'ishexa' [-Wunused-variable]
  957 |     bool ishexa = false;
      |          ^~~~~~
../lexers/LexJulia.cxx:960:10: warning: unused variable 'isexpon' [-Wunused-variable]
  960 |     bool isexpon = false;
      |          ^~~~~~~
../lexers/LexJulia.cxx:969:10: warning: unused variable 'expr_begin' [-Wunused-variable]
  969 |     bool expr_begin = true;
      |          ^~~~~~~~~~
../lexers/LexJulia.cxx: In member function 'virtual void LexerJulia::Fold(Sci_PositionU, Sci_Position, int, Scintilla::IDocument*)':
../lexers/LexJulia.cxx:1263:7: warning: unused variable 'stylePrev' [-Wunused-variable]
 1263 |   int stylePrev = style;
      |       ^~~~~~~~~
../lexers/LexJulia.cxx:1253:6: warning: variable 'levelMinCurrent' set but not used [-Wunused-but-set-variable]
 1253 |  int levelMinCurrent = levelCurrent;
      |      ^~~~~~~~~~~~~~~
../lexers/LexJulia.cxx: At global scope:
../lexers/LexJulia.cxx:603:13: warning: 'void ScanNumber(Scintilla::StyleContext&)' defined but not used [-Wunused-function]
  603 | static void ScanNumber(StyleContext& sc) {
      |             ^~~~~~~~~~
../lexers/LexJulia.cxx:541:13: warning: 'bool IsExprBeginning(Scintilla::StyleContext&)' defined but not used [-Wunused-function]
  541 | static bool IsExprBeginning(StyleContext& sc) {
      |             ^~~~~~~~~~~~~~~
../lexers/LexJulia.cxx:504:13: warning: 'bool CheckKeywordComprehension(char*)' defined but not used [-Wunused-function]
  504 | static bool CheckKeywordComprehension(char *str) {
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~

lexilla\lexers\LexJulia.cxx:428:26: warning: Array 'opsuffs[117]' accessed at index 233, which is out of bounds. [arrayIndexOutOfBounds]
        if (wc == opsuffs[i]) {
                         ^
lexilla\lexers\LexJulia.cxx:544:9: warning: The scope of the variable 'cc' can be reduced. [variableScope]
    int cc, j;
        ^
lexilla\lexers\LexJulia.cxx:672:9: warning: The scope of the variable 'n' can be reduced. [variableScope]
    int n = 0;
        ^
lexilla\lexers\LexJulia.cxx:688:11: warning: Variable 'n' is assigned a value that is never used. [unreadVariable]
    int n = 0;
          ^
lexilla\lexers\LexJulia.cxx:1253:22: warning: Variable 'levelMinCurrent' is assigned a value that is never used. [unreadVariable]
 int levelMinCurrent = levelCurrent;
                     ^
lexilla\lexers\LexJulia.cxx:1354:20: warning: Variable 'levelMinCurrent' is assigned a value that is never used. [unreadVariable]
   levelMinCurrent = levelCurrent;
                   ^

[getzze]

Hi, here is the new patch with the warnings corrected (and a small addition of a new keyword list).

[Neil Hodgson]

While there are still some issues, Julia.2.patch+Julia.3.patch have been committed as
https://github.com/ScintillaOrg/lexilla/commit/6ee6b086a1920fdafe3af41eecbd865d52162ec7

There are some uses of is* functions from ctype.h that are not protected by IsASCII such as isdigit(sc.chNext) where chNext is a Unicode code point so may be outside the range for isdigit. This can lead to crashes as described by https://drewdevault.com/2020/09/25/A-story-of-two-libcs.html .

There are occurrences of sc.ch != EOF in the lexer but sc.ch should never be EOF (-1). If it ever is then that is a bug.

The x.jl.styled file was not reproduced by this code: on the comment line, it now styles the line end (\n or \r\n) as 0 instead of 1. While this is sensible, its a discrepancy so I wanted to check that it was deliberate.

Updated to match Lexilla 5.x: using namespace Lexilla;, ILexer5, lvRelease5.

There were 2 unused functions that I removed: IsJuliaReservedOperator and isHexa.

Moved SCLEX_JULIA to 133 to avoid clash.

Added other build files, change log entry, warning suppressions, SciTE.properties for example file and git attributes.

Lexilla is now a separate project on GitHub https://github.com/ScintillaOrg/lexilla

[Neil Hodgson]

There's a problem with the properties defined by this lexer. A naming convention prefixes names so the don't clash. Julia lexer properties should start with "lexer.julia." and Julia folding properties should start with "fold.julia.". There are some legacy and shared properties that do not follow this convention but all new properties should.

[getzze]

Hi. thanks for merging the Julia Lexer! Is it better that I open a Pull-Request on github to address the remaining points? I would prefer as I am more familiar with git.

[Neil Hodgson]

Lexilla has moved to https://github.com/ScintillaOrg/lexilla so that is where issues should go.

[honfui]

Is there sample julia.properties to download/use?

[Neil Hodgson]

No properties file for Julia has been contributed.

[honfui]

Thanks. Hope someone can contribute.