#1295 Scintilla: Add support for the DataFlex programming language

[Wil van Antwerpen]

Hi,

Over the past few years me and some friends have been working on an open source code editor called "The Hammer" (see: https://projects.vdf-guidance.com/projects/hammer )
Version 3 of that project switched over to using the scintilla control.
It's a fantastic control, love the functionality of it. Thanks!

Attached is a first version of my implementation for a DataFlex language lexer.
I hope it matches all the code requirements for getting it merged.
If not then please do let me know and I will rework as asked.

The zip also includes a dataflex.properties file that could be used by SciTE for testing/experimenting.
However my main focus for this post is to be able to get DataFlex language support added to the lexer.

[Neil Hodgson]

It looks useful.

The code is mostly OK but there are some minor issues.

New properties are namespaced to avoid collisions. So fold.compilerlist should be fold.dataflex.compilerlist.

To allow automatic documentation extraction, property retrievals (for unique properties) should be preceded by a comment. For the format, see LexErrorList.cxx or LexCSS.cxx. scite/scripts/RegenerateSource.py will extract these into scite/doc/SciTEDoc.html.

There are some type warnings from MSVC converting size_t to int which are more significant in 64-bit builds. The warnings can be silenced by casts but its often better to try to keep variables matching their originating expressions.

..\lexers\LexDataflex.cxx(139): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data
..\lexers\LexDataflex.cxx(382): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data

dataflex.properties associates *.src and *.inc with dataflex but these extensions are also used by other languages. Because of this, and to keep menus small, dataflex will be disabled by default in SciTE distributions but can be enabled by setting a changed imports.exclude property.

[Wil van Antwerpen]

Hello Neil,

Thanks for looking into this and for considering my DataFlex lexer to be useful.

Understood about SciTE coming with DataFlex support disabled due to the file extension collisions. No problem with that as far as I am concerned, I am only supplying the properties file as an alternative for being able to use the lexer without the need to compile and run "The Hammer" project.
FWIW I added a few more common file extensions to the properties file and I'm sure that now there's even more collisions.
If you need an example DataFlex source code file to test with then try the scintilla control wrapper file which can be found here:
http://svn.vdf-guidance.com/TheHammer/trunk/Hammer/Pkg/cSciLexer.pkg

Nice catch on the fold.compilerlist property. I actually did read your suggestion and had fixed this at my internal 3.x scintilla branch, somehow I missed it on merging with the 4.1.6 branch. FWIW, at least I got it right in the properties file. This has been corrected and requested documentation has been added (see attached zip)

I addressed the forementioned type warnings. Then decided to compile with "WAll" and got a bit of a shock. I did not remove all warnings at "Warning All" level as I'm too scared of breaking things, but will look into that over time.
If I am understanding you correctly then your concern was mostly about compiling under 64 bits. Something I had not done yet as DataFlex is currently 32 bits only (64 bits is in the works) Building an x64 version of the DLL now does not trigger any warnings with the default warning level set at level W4.

See updated zip.
Let me know if you have more recommendations or suggestions.

Wil

[Neil Hodgson]

Thanks for the update which builds without warnings. For core code (not lexers from others) I also use the static analyzer in Visual C++ with most of "Microsoft All Rules" turned on although some of its reported issues are more matters of style. cppcheck can also be worth running although, again, not every report is worth fixing.

The only thing left is to add a comment, similar to other lexers, allowing distribution under the Scintilla license.

// The License.txt file describes the conditions under which this software may be distributed.

[Wil van Antwerpen]

Ah yes, of course, quite an important line to be fair.
Also added as well.

Thanks for your swift handling of this feature request.

[Neil Hodgson]

Committed as [ab6954] and [b8f95a].

Some settings were removed from dataflex.properties as they are shared and would cause confusion. Properties files are not scope walls - every setting in every language properties file goes into a single bag of settings.

[Wil van Antwerpen]

Ah right, sorry about that.
Should have cleaned that up, my bad.

Added the removed settings when I was having problems last week getting the properties file to work again in SciTE.
Had used SciTE initially when I started working on the lexer in 2017, but my lexer changed and it took me some efforts to figure out how-to reflect those changes in the properties file. Which as it turns out had nothing to do with those settings. Actually did remove a lot of cruft, but forgot those settings.

Thanks for fixing that up.

[Neil Hodgson]

There's a portability issue with this lexer. It uses strnlen which isn't found in some environments like some releases of MinGW. Its likely it can be replaced by strlen as the result from GetCurrentLowered should always be NUL terminated within the buffer.

[Wil van Antwerpen]

OK, sadly I do not really have time to look at it right away, but can put it on my agenda to look into this shortly (time line: 2 weeks). I saw last week that the lexers moved to Lexilla, so I'm a bit behind on staying up to date with scintilla.